sysbench使用及自定義oltp測試lua腳本


一、sysbench介紹

sysbench是一個模塊化的、跨平台、多線程基准測試工具,主要用於評估測試各種不同系統參數下的數據庫負載情況。
目前sysbench代碼托管在launchpad上,項目地址:https://launchpad.net/sysbench(原來的官網 http://sysbench.sourceforge.net 已經不可用),源碼采用bazaar管理。
注:本文所有的測試都是基於linux操作系統和mysql數據庫的。

二、sysbench安裝

1、依賴包:

  • autoconf
  • automake
  • cdbs
  • debhelper (>= 9)
  • docbook-xml
  • docbook-xsl
  • libmysqlclient15-dev
  • libtool
  • xsltproc

2、編譯安裝:

#./autogen.sh
#./configure
#make

注意事項:./configure命令,sysbench默認是支持MySQL的benchmarking的,如果不加任何選項則要求保證MySQL的安裝路徑都是默認的標准路徑,headfile位於/usr/include目錄下,libraries位於/usr/lib/目錄下。因為我的MySQL是源碼編譯安裝的,安裝路徑是放在/usr/local/mysql下,所以這里要添加相應的選項命令:
./configure --prefix=/usr/local/sysbench --with-mysql=/usr/local/mysql \
--with-mysql-includes=/usr/local/mysql/include/ \
--with-mysql-libs=/usr/local/mysql/lib/

這樣就可以看到/usr/local/sysbench下有一個可執行文件sysbench,這就是sysbench的主程序。

3、功能簡介

sysbench目前可以進行如下幾個方面的性能測試:
- fileio - File I/O test #磁盤io性能
- cpu - CPU performance test #CUP性能
- memory - Memory functions speed test #內存性能
- threads - Threads subsystem performance test #POSIX線程性能
- mutex - Mutex performance test #調度程序性能
- oltp - OLTP test #數據庫性能(OLTP基准測試)

注:在0.4版本的--test選項中是可以直接選用oltp模式,但是在0.4.12.1.1以后oltp測試就轉換成調用lua腳本來進行測試了,腳本主要存放在tests/db目錄下。這樣用戶就可以根據自己的系統定制lua腳本,這樣的測試就能更精確的測試業務的性能。

三、開始進行測試

通用配置

接下來我們來分別看一下各個模式的相關參數、測試方法和結果分析。
sysbench的基本命令格式為:
sysbench –test=< test-name> [options]… < command>
主要分為三個部分:

1、–test=< test-name>

這部分是指定測試類型,基本類型有fileio,cpu,memory,threads,mutex,oltp(或者指定lua腳本)

2、[options]…

這部分包括測試需要的各種選項,有全局的也有每個測試模式自由的選項
(每個測試模式的選項可以用./sysbench –test=< test-name> help來獲取)

3、< command>

控制命令,總共有五個
prepare #准備測試,主要是生成測試數據
run #執行測試,根據選項限制來執行測試
cleanup #清除准備階段生成的測試數據
help #獲取幫助文檔
version #獲取版本信息

幾個重要的全局參數:

–num-threads=N number of threads to use [1] #測試時使用的線程數
–max-requests=N limit for total number of requests [10000] #測試過程最多執行多少次請求
–max-time=N limit for total execution time in seconds [0] #測試過程總共執行多長時間(和–max-requests效果同樣,但是兩個同時限定的時候誰優先還沒有測試)
–report-interval=N periodically report intermediate statistics with a specified interval in seconds. 0 disables intermediate reports [0] #每隔多少秒輸出測試概況(這個過程你可以觀察到mysql redolog的切換情況)
–db-driver=STRING specifies database driver to use (‘help’ to get list of available drivers) #指定需求測試的數據庫類型,默認是mysql
#mysql鏈接選項
–mysql-host=[LIST,…] MySQL server host [localhost] #mysql主機地址
–mysql-port=N MySQL server port [3306] #mysql端口
–mysql-socket=[LIST,…] MySQL socket #mysql socket文件位置,指定這個之后 其他的鏈接選項均可以不指定
–mysql-user=STRING MySQL user [sbtest] #用來測試的mysql用戶名
–mysql-password=STRING MySQL password [] #密碼
–mysql-db=STRING MySQL database name [sbtest] #測試數據庫名 默認sbtest

接下來進入各個測試模式的測試方法

fileio-磁盤io性能

1、主要參數

[root@centostest sysbench]# ./sysbench --test=fileio help
sysbench 0.5: multi-threaded system evaluation benchmark

fileio options:
--file-num=N number of files to create [128]
--file-block-size=N block size to use in all IO operations [16384]
--file-total-size=SIZE total size of files to create [2G]
--file-test-mode=STRING test mode {seqwr, seqrewr, seqrd, rndrd, rndwr, rndrw}
--file-io-mode=STRING file operations mode {sync,async,mmap} [sync]
--file-extra-flags=STRING additional flags to use on opening files {sync,dsync,direct} []
--file-fsync-freq=N do fsync() after this number of requests (0 - don't use fsync()) [100]
--file-fsync-all=[on|off] do fsync() after each write operation [off]
--file-fsync-end=[on|off] do fsync() at the end of test [on]
--file-fsync-mode=STRING which method to use for synchronization {fsync, fdatasync} [fsync]
--file-merged-requests=N merge at most this number of IO requests if possible (0 - don't merge) [0]
--file-rw-ratio=N reads/writes ratio for combined test [1.5]

No help available for test 'fileio'.

參數詳解:
–file-num=N 代表生成測試文件的數量,默認為128。
–file-block-size=N 測試時所使用文件塊的大小,如果想磁盤針對innodb存儲引擎進行測試,可以將其設置為16384,即innodb存儲引擎頁的大小。默認為16384。
–file-total-size=SIZE 創建測試文件的總大小,默認為2G大小。
–file-test-mode=STRING 文件測試模式,包含:seqwr(順序寫), seqrewr(順序讀寫), seqrd(順序讀), rndrd(隨機讀), rndwr(隨機寫), rndrw(隨機讀寫)。
–file-io-mode=STRING 文件操作的模式,sync(同步),async(異步),fastmmap(快速mmap),slowmmap(慢速mmap),默認為sync同步模式。
–file-async-backlog=N 對應每個線程隊列的異步操作數,默認為128。
–file-extra-flags=STRING 打開文件時的選項,這是與API相關的參數。
–file-fsync-freq=N 執行fsync()函數的頻率。fsync主要是同步磁盤文件,因為可能有系統和磁盤緩沖的關系。 0代表不使用fsync函數。默認值為100。
–file-fsync-all=[on|off] 每執行完一次寫操作,就執行一次fsync。默認為off。
–file-fsync-end=[on|off] 在測試結束時執行fsync函數。默認為on。
–file-fsync-mode=STRING文件同步函數的選擇,同樣是和API相關的參數,由於多個操作系統對於fdatasync支持不同,因此不建議使用fdatasync。默認為fsync。
–file-merged-requests=N 大多情況下,合並可能的IO的請求數,默認為0。
–file-rw-ratio=N 測試時的讀寫比例,默認時為1.5,即可3:2。

2、測試實例

測試總大小為5G的10個文件的隨機讀寫性能:

1>先生成測試文件
sysbench --test=fileio --file-num=10 --file-total-size=5G prepare
2>開始測試
sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --max-time=180 --max-requests=100000000 --num-threads=16 --init-rng=on --file-num=10 --file-extra-flags=direct --file-fsync-freq=0 --file-block-size=16384 run
3>結果分析
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.

Random number generator seed is 0 and will be ignored


Extra file open flags: 4000
10 files, 512Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 100000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
(last message repeated 15 times)
Done.

Operations performed: 89247 reads, 59488 writes, 0 Other = 148735 Total
Read 1.3618Gb Written 929.5Mb Total transferred 2.2695Gb (12.888Mb/sec)
824.84 Requests/sec executed

General statistics:
total time: 180.3188s
total number of events: 148735
total time taken by event execution: 2882.8395
response time:
min: 0.08ms
avg: 19.38ms
max: 953.75ms
approx. 95 percentile: 158.81ms

Threads fairness:
events (avg/stddev): 9295.9375/371.20
execution time (avg/stddev): 180.1775/0.07

主要看這行輸出的信息:

Read 1.3618Gb  Written 929.5Mb  Total transferred 2.2695Gb  (12.888Mb/sec)
824.84 Requests/sec executed

這行信息表示:在180秒時間里面總共完成隨機讀取1.3618G數據,寫入929.5Mb數據,平均每秒隨機讀寫的效率為12.888Mb/秒,IOPS為824.84 Requests/sec
因為是虛擬機,所有磁盤的表現還是比較差的。

4>清除測試數據
[root@centostest sysbench]# sysbench --test=fileio --file-num=10 --file-total-size=5G cleanup
sysbench 0.4.12: multi-threaded system evaluation benchmark

Removing test files...

cpu-cpu性能測試

1、主要參數

[root@centostest sysbench]# sysbench --test=cpu help
sysbench 0.4.12: multi-threaded system evaluation benchmark

cpu options:
--cpu-max-prime=N upper limit for primes generator [10000]

參數詳解:
–cpu-max-prime=N 用來選項指定最大的素數,具體參數可以根據CPU的性能來設置,默認為10000

2、測試實例

根據官網的介紹可知:CPU測試使用64位整數,測試計算素數直到某個最大值所需要的時間。

sysbench --test=cpu --cpu-max-prime=20000 run

輸出如下:

[root@centostest sysbench]# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Random number generator seed is 0 and will be ignored


Doing CPU performance benchmark

Primer numbers limit: 20000

Threads started!
Done.


General statistics:
total time: 28.9293s
total number of events: 10000
total time taken by event execution: 28.8916
response time:
min: 2.51ms
avg: 2.89ms
max: 7.46ms
approx. 95 percentile: 3.49ms

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 28.8916/0.00

我們只需要關心測試的總時間(total time)即可(越小越憂)。
CPU性能測試有一個需要注意的地方,上面的測試只使用了一個線程,如果在兩個cpu processor不同的電腦上做比較,這是不公平的。公平的做法是指定合理的線程數,如下所示:

sysbench --test=cpu --num-threads=`grep "processor" /proc/cpuinfo | wc -l` --cpu-max-prime=20000 run

補充知識:
查看CPU核數的方法
查看物理cpu個數

grep "physical id" /proc/cpuinfo | sort -u | wc -l

查看核心數量

grep "core id" /proc/cpuinfo | sort -u | wc -l

查看線程數量

grep "processor" /proc/cpuinfo | sort -u | wc -l

在sysbench的測試中,–num-threads取值為”線程數量”即可,再大的值沒有什么意義,對測試結果也沒有什么影響。

memory-內存分配及傳輸速度

1、主要參數

[root@centostest sysbench]# ./sysbench --test=memory help
sysbench 0.5: multi-threaded system evaluation benchmark

memory options:
--memory-block-size=SIZE size of memory block for test [1K]
--memory-total-size=SIZE total size of data to transfer [100G]
--memory-scope=STRING memory access scope {global,local} [global]
--memory-hugetlb=[on|off] allocate memory from HugeTLB pool [off]
--memory-oper=STRING type of memory operations {read, write, none} [write]
--memory-access-mode=STRING memory access mode {seq,rnd} [seq]

No help available for test 'memory'.

參數詳解:
–memory-block-size=SIZE 測試內存塊的大小,默認為1K
–memory-total-size=SIZE 數據傳輸的總大小,默認為100G
–memory-scope=STRING 內存訪問的范圍,包括全局和本地范圍,默認為global
–memory-hugetlb=[on|off] 是否從HugeTLB池分配內存的開關,默認為off
–memory-oper=STRING 內存操作的類型,包括read, write, none,默認為write
–memory-access-mode=STRING 內存訪問模式,包括seq,rnd兩種模式,默認為seq

2、測試實例

內存測試測試了內存的連續讀寫性能。

./sysbench --test=memory --memory-block-size=8K --memory-total-size=1G  --num-threads=16 run

3、結果分析

輸出結果如下:

[root@centostest sysbench]# ./sysbench --test=memory --memory-block-size=8K --memory-total-size=1G --num-threads=16 run
sysbench 0.5: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Random number generator seed is 0 and will be ignored


Initializing worker threads...

Threads started!

Operations performed: 131072 (626689.49 ops/sec)

1024.00 MB transferred (4896.01 MB/sec)


General statistics:
total time: 0.2091s
total number of events: 131072
total time taken by event execution: 2.3239s
response time:
min: 0.00ms
avg: 0.02ms
max: 50.38ms
approx. 95 percentile: 0.00ms

Threads fairness:
events (avg/stddev): 8192.0000/937.69
execution time (avg/stddev): 0.1452/0.03

threads-POSIX線程性能

1、主要參數

[root@centostest sysbench]# ./sysbench --test=threads help
sysbench 0.5: multi-threaded system evaluation benchmark

threads options:
--thread-yields=N number of yields to do per request [1000]
--thread-locks=N number of locks per thread [8]

No help available for test 'threads'.

參數詳解:
–thread-yields=N 指定每個請求的壓力,默認為1000
–thread-locks=N 指定每個線程的鎖數量,默認為8

2、測試實例

測試線程調度器的性能。對於高負載情況下測試線程調度器的行為非常有用。

sysbench --test=threads --num-threads=64 run

3、結果分析

下面是輸出結果:

[root@centostest sysbench]# sysbench --test=threads --num-threads=64 run
sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64
Random number generator seed is 0 and will be ignored


Doing thread subsystem performance test
Thread yields per test: 1000 Locks used: 8
Threads started!
Done.


General statistics:
total time: 9.4415s
total number of events: 10000
total time taken by event execution: 602.1637
response time:
min: 0.35ms
avg: 60.22ms
max: 479.03ms
approx. 95 percentile: 126.27ms

Threads fairness:
events (avg/stddev): 156.2500/4.82
execution time (avg/stddev): 9.4088/0.02

關注的性能指標也是total time越小越憂。

mutex-調度程序性能

1、主要參數

[root@centostest sysbench]# ./sysbench --test=mutex help
sysbench 0.5: multi-threaded system evaluation benchmark

mutex options:
--mutex-num=N total size of mutex array [4096]
--mutex-locks=N number of mutex locks to do per thread [50000]
--mutex-loops=N number of empty loops to do inside mutex lock [10000]

No help available for test 'mutex'.

參數詳解:
–mutex-num=N 數組互斥的總大小。默認是4096
–mutex-locks=N 每個線程互斥鎖的數量。默認是50000
–mutex-loops=N 內部互斥鎖的空循環數量。默認是10000

2、測試實例

測試互斥鎖的性能,方式是模擬所有線程在同一時刻並發運行,並都短暫請求互斥鎖。

./sysbench --test=mutex --num-threads=16 --mutex-num=2048 --mutex-locks=1000000 --mutex-loops=5000 run

3、結果分析

結果輸出如下:

[root@centostest sysbench]# ./sysbench --test=mutex --num-threads=16 --mutex-num=2048 --mutex-locks=1000000 --mutex-loops=5000 run
sysbench 0.5: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Random number generator seed is 0 and will be ignored


Initializing worker threads...

Threads started!


General statistics:
total time: 0.8856s
total number of events: 16
total time taken by event execution: 13.7792s
response time:
min: 832.79ms
avg: 861.20ms
max: 880.81ms
approx. 95 percentile: 878.94ms

Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.8612/0.01

oltp-數據庫性能測試

1、主要參數

由於sysbench0.5中已經不存在oltp測試模式了,所以使用命令./sysbench --test=oltp help無法獲取好幫助信息,還好0.4和0.5的參數都是兼容的,所以我們這里用0.4的幫助信息來說明:

[root@centostest sysbench]# ./sysbench --help=oltp help
Usage:
sysbench [general-options]... --test=<test-name> [test-options]... command

General options:
--num-threads=N number of threads to use [1]
--max-requests=N limit for total number of requests [10000]
--max-time=N limit for total execution time in seconds [0]
--forced-shutdown=STRING amount of time to wait after --max-time before forcing shutdown [off]
--thread-stack-size=SIZE size of stack per thread [32K]
--init-rng=[on|off] initialize random number generator [off]
--seed-rng=N seed for random number generator, ignored when 0 [0]
--tx-rate=N target transaction rate (tps) [0]
--tx-jitter=N target transaction variation, in microseconds [0]
--report-interval=N periodically report intermediate statistics with a specified interval in seconds. 0 disables intermediate reports [0]
--report-checkpoints=[LIST,...]dump full statistics and reset all counters at specified points in time. The argument is a list of comma-separated values representing the amount of time in seconds elapsed from start of test when report checkpoint(s) must be performed. Report checkpoints are off by default. []
--test=STRING test to run
--debug=[on|off] print more debugging info [off]
--validate=[on|off] perform validation checks where possible [off]
--help=[on|off] print help and exit
--version=[on|off] print version and exit

Log options:
--verbosity=N verbosity level {5 - debug, 0 - only critical messages} [4]

--percentile=N percentile rank of query response times to count [95]

Compiled-in tests:
fileio - File I/O test
cpu - CPU performance test
memory - Memory functions speed test
threads - Threads subsystem performance test
mutex - Mutex performance test
oltp - OLTP test

Commands: prepare run cleanup help version

See 'sysbench --test=<name> help' for a list of options for each test.

參數詳解:
–oltp-test-mode=STRING 執行模式{simple,complex(advanced transactional),nontrx(non-transactional),sp}。默認是complex
–oltp-reconnect-mode=STRING 重新連接模式{session(不使用重新連接。每個線程斷開只在測試結束),transaction(在每次事務結束后重新連接),query(在每個SQL語句執行完重新連接),random(對於每個事務隨機選擇以上重新連接模式)}。默認是session
–oltp-sp-name=STRING 存儲過程的名稱。默認為空
–oltp-read-only=[on|off] 只讀模式。Update,delete,insert語句不可執行。默認是off
–oltp-skip-trx=[on|off] 省略begin/commit語句。默認是off
–oltp-range-size=N 查詢范圍。默認是100
–oltp-point-selects=N number of point selects [10]
–oltp-simple-ranges=N number of simple ranges [1]
–oltp-sum-ranges=N number of sum ranges [1]
–oltp-order-ranges=N number of ordered ranges [1]
–oltp-distinct-ranges=N number of distinct ranges [1]
–oltp-index-updates=N number of index update [1]
–oltp-non-index-updates=N number of non-index updates [1]
–oltp-nontrx-mode=STRING 查詢類型對於非事務執行模式{select, update_key, update_nokey, insert, delete} [select]
–oltp-auto-inc=[on|off] AUTO_INCREMENT是否開啟。默認是on
–oltp-connect-delay=N 在多少微秒后連接數據庫。默認是10000
–oltp-user-delay-min=N 每個請求最短等待時間。單位是ms。默認是0
–oltp-user-delay-max=N 每個請求最長等待時間。單位是ms。默認是0
–oltp-table-name=STRING 測試時使用到的表名。默認是sbtest
–oltp-table-size=N 測試表的記錄數。默認是10000
–oltp-dist-type=STRING 分布的隨機數{uniform(均勻分布),Gaussian(高斯分布),special(空間分布)}。默認是special
–oltp-dist-iter=N 產生數的迭代次數。默認是12
–oltp-dist-pct=N 值的百分比被視為’special’ (for special distribution)。默認是1
–oltp-dist-res=N ‘special’的百分比值。默認是75

2、測試實例

1>生成測試數據,1個表,每個表100000數據
[root@centostest sysbench]# ./sysbench --mysql-host=127.0.0.1 --mysql-port=3166 --mysql-user=root --mysql-password=123456 --test=tests/db/oltp.lua --oltp_tables_count=1 --oltp-table-size=1000000 --rand-init=on prepare
sysbench 0.5: multi-threaded system evaluation benchmark

Creating table 'sbtest1'...
Inserting 1000000 records into 'sbtest1'
……………………
2>開始測試

使用128線程(–num-threads=128)測試60秒(–max-time=60)每10秒輸出一次測試信息(–report-interval=10)

./sysbench --mysql-host=127.0.0.1 --mysql-port=3166 --mysql-user=root --mysql-password=123456 --test=tests/db/oltp.lua --oltp_tables_count=1 --oltp-table-size=1000000 --num-threads=128 --oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=60 --max-requests=0 run
3>結果分析

測試結果輸出如下:

[root@centostest sysbench]# ./sysbench --mysql-host=127.0.0.1 --mysql-port=3166 --mysql-user=root --mysql-password=123456 --test=tests/db/oltp.lua --oltp_tables_count=1 --oltp-table-size=1000000 --num-threads=128 --oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=60 --max-requests=0 run
sysbench 0.5: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 128
Report intermediate results every 10 second(s)
Random number generator seed is 0 and will be ignored


Initializing worker threads...

Threads started!

[ 10s] threads: 128, tps: 63.17, reads: 957.71, writes: 253.70, response time: 6659.14ms (95%), errors: 0.00, reconnects: 0.00
[ 20s] threads: 128, tps: 38.10, reads: 615.56, writes: 169.59, response time: 4702.80ms (95%), errors: 0.00, reconnects: 0.00
[ 30s] threads: 128, tps: 43.10, reads: 627.16, writes: 154.61, response time: 7544.47ms (95%), errors: 0.00, reconnects: 0.00
[ 40s] threads: 128, tps: 27.70, reads: 364.70, writes: 119.80, response time: 7278.28ms (95%), errors: 0.00, reconnects: 0.00
[ 50s] threads: 128, tps: 36.10, reads: 399.28, writes: 135.69, response time: 8400.39ms (95%), errors: 0.00, reconnects: 0.00
[ 60s] threads: 128, tps: 21.80, reads: 434.42, writes: 90.30, response time: 11977.08ms (95%), errors: 0.00, reconnects: 0.00
OLTP test statistics:
queries performed:
read: 33992
write: 9712
other: 4856
total: 48560
transactions: 2428 (39.27 per sec.)
read/write requests: 43704 (706.81 per sec.)
other operations: 4856 (78.53 per sec.)
ignored errors: 0 (0.00 per sec.)
reconnects: 0 (0.00 per sec.)

General statistics:
total time: 61.8331s
total number of events: 2428
total time taken by event execution: 7830.4825s
response time:
min: 210.50ms
avg: 3225.08ms
max: 11983.87ms
approx. 95 percentile: 8024.33ms

Threads fairness:
events (avg/stddev): 18.9688/2.14
execution time (avg/stddev): 61.1756/0.40

主要關注如下幾個值:

    transactions:                        2428   (39.27 per sec.)
read/write requests: 43704 (706.81 per sec.)
other operations: 4856 (78.53 per sec.)

四、oltp-數據庫性能測試中lua腳本分析

在sysbench0.4的后期版本中sysbench已經取消了test中的oltp模式,換而代之的是oltp的lua腳本。這一改變大大的提升了sysbench的靈活性。用戶可以結合業務來定制lua腳本,這樣能更精確的測試出適用於此業務的數據庫性能指標。

這次我們使用sysbench-0.4.12-1.1來看看默認的lua腳本做了哪些工作,以及我們怎么來定制lua腳本。

oltp的測試腳本默認存放在tests/db下,這個目錄下有很多腳本,在oltp基准測試中我們用到比較多的是common.luaoltp.lua

[root@ol5-112 db]# pwd
/root/sysbench-0.4.12-1.1/sysbench/tests/db
[root@ol5-112 db]# ll
total 72
-rw-r--r-- 1 root root 3585 Jul 28 09:32 common.lua
-rw-r--r-- 1 root root 340 Jul 28 09:32 delete.lua
-rw-r--r-- 1 root root 830 Jul 28 09:32 insert.lua
-rw-r--r-- 1 root root 12088 Jul 28 09:32 Makefile
-rw-r--r-- 1 root root 1020 Jul 28 09:32 Makefile.am
-rw-r--r-- 1 root root 11569 Jul 28 09:32 Makefile.in
-rw-r--r-- 1 root root 2925 Jul 28 09:32 oltp.lua
-rw-r--r-- 1 root root 342 Jul 28 09:32 oltp_simple.lua
-rw-r--r-- 1 root root 425 Jul 28 09:32 parallel_prepare.lua
-rw-r--r-- 1 root root 343 Jul 28 09:32 select.lua
-rw-r--r-- 1 root root 3964 Jul 28 09:32 select_random_points.lua
-rw-r--r-- 1 root root 4066 Jul 28 09:32 select_random_ranges.lua
-rw-r--r-- 1 root root 343 Jul 28 09:32 update_index.lua
-rw-r--r-- 1 root root 552 Jul 28 09:32 update_non_index.lua

首先來看common.lua

-- Input parameters
-- oltp-tables-count - number of tables to create
-- oltp-secondary - use secondary key instead PRIMARY key for id column
--
--

-- 創建表,插入測試數據
function create_insert(table_id)

local index_name
local i
local j
local query

if (oltp_secondary) then
index_name = "KEY xid"
else
index_name = "PRIMARY KEY"
end

i = table_id

print("Creating table 'sbtest" .. i .. "'...")
if (db_driver == "mysql") then
query = [[
CREATE TABLE sbtest]]
.. i .. [[ (
id INTEGER UNSIGNED NOT NULL ]]
..
((oltp_auto_inc and "AUTO_INCREMENT") or "") .. [[,
k INTEGER UNSIGNED DEFAULT '0' NOT NULL,
c CHAR(120) DEFAULT '' NOT NULL,
pad CHAR(60) DEFAULT '' NOT NULL,
]]
.. index_name .. [[ (id)
) /*! ENGINE = ]]
.. mysql_table_engine ..
" MAX_ROWS = " .. myisam_max_rows .. " */"

elseif (db_driver == "pgsql") then
query = [[
CREATE TABLE sbtest]]
.. i .. [[ (
id SERIAL NOT NULL,
k INTEGER DEFAULT '0' NOT NULL,
c CHAR(120) DEFAULT '' NOT NULL,
pad CHAR(60) DEFAULT '' NOT NULL,
]]
.. index_name .. [[ (id)
) ]]


elseif (db_driver == "drizzle") then
query = [[
CREATE TABLE sbtest (
id INTEGER NOT NULL ]]
.. ((oltp_auto_inc and "AUTO_INCREMENT") or "") .. [[,
k INTEGER DEFAULT '0' NOT NULL,
c CHAR(120) DEFAULT '' NOT NULL,
pad CHAR(60) DEFAULT '' NOT NULL,
]]
.. index_name .. [[ (id)
) ]]

else
print("Unknown database driver: " .. db_driver)
return 1
end

db_query(query)

db_query("CREATE INDEX k_" .. i .. " on sbtest" .. i .. "(k)")

print("Inserting " .. oltp_table_size .. " records into 'sbtest" .. i .. "'")

if (oltp_auto_inc) then
db_bulk_insert_init("INSERT INTO sbtest" .. i .. "(k, c, pad) VALUES")
else
db_bulk_insert_init("INSERT INTO sbtest" .. i .. "(id, k, c, pad) VALUES")
end

local c_val
local pad_val


for j = 1,oltp_table_size do

c_val = sb_rand_str([[
###########-###########-###########-###########-###########-###########-###########-###########-###########-###########]]
)
pad_val = sb_rand_str([[
###########-###########-###########-###########-###########]]
)

if (oltp_auto_inc) then
db_bulk_insert_next("(" .. sb_rand(1, oltp_table_size) .. ", '".. c_val .."', '" .. pad_val .. "')")
else
db_bulk_insert_next("("..j.."," .. sb_rand(1, oltp_table_size) .. ",'".. c_val .."', '" .. pad_val .. "' )")
end
end

db_bulk_insert_done()


end

-- prepare階段執行的操作
function prepare()
local query
local i
local j

set_vars()

db_connect()


for i = 1,oltp_tables_count do
create_insert(i)
end

return 0
end

-- cleanup階段執行的操作
function cleanup()
local i

set_vars()

for i = 1,oltp_tables_count do
print("Dropping table 'sbtest" .. i .. "'...")
db_query("DROP TABLE sbtest".. i )
end
end

-- 初始化默認值
function set_vars()
oltp_table_size = oltp_table_size or 10000
oltp_range_size = oltp_range_size or 100
oltp_tables_count = oltp_tables_count or 1
oltp_point_selects = oltp_point_selects or 10
oltp_simple_ranges = oltp_simple_ranges or 1
oltp_sum_ranges = oltp_sum_ranges or 1
oltp_order_ranges = oltp_order_ranges or 1
oltp_distinct_ranges = oltp_distinct_ranges or 1
oltp_index_updates = oltp_index_updates or 1
oltp_non_index_updates = oltp_non_index_updates or 1

if (oltp_auto_inc == 'off') then
oltp_auto_inc = false
else
oltp_auto_inc = true
end

if (oltp_read_only == 'on') then
oltp_read_only = true
else
oltp_read_only = false
end

if (oltp_skip_trx == 'on') then
oltp_skip_trx = true
else
oltp_skip_trx = false
end

end

這個腳本主要功能是用來准備測試的表、測試數據、初始化測試中需要的一些默認值、清楚數據。function prepare()會在全局command為perpare時調用,function cleanup()會在全局command為cleanup時調用,而function set_vars()會被引用至自己本身以及其他lua腳本中。

接下來我們來看oltp.lua腳本。

pathtest = string.match(test, "(.*/)") or ""

-- 引入common.lua腳本
dofile(pathtest .. "common.lua")

-- 申明在滿足db_driver == "mysql" and mysql_table_engine == "myisam"條件時測試使用鎖表開頭,解除鎖表結尾。其他全部使用BEGIN開始,commit結束(事務)
function thread_init(thread_id)
set_vars()

if (db_driver == "mysql" and mysql_table_engine == "myisam") then
begin_query = "LOCK TABLES sbtest WRITE"
commit_query = "UNLOCK TABLES"
else
begin_query = "BEGIN"
commit_query = "COMMIT"
end

end

-- 開始測試
function event(thread_id)
local rs
local i
local table_name
local range_start
local c_val
local pad_val
local query

-- 隨機獲取表名后綴
table_name = "sbtest".. sb_rand_uniform(1, oltp_tables_count)
-- 如果指定跳過事務參數為ON 則不調用事務開始標示
if not oltp_skip_trx then
db_query(begin_query)
end
-- 開始執行執行查詢語句
for i=1, oltp_point_selects do
rs = db_query("SELECT c FROM ".. table_name .." WHERE id=" .. sb_rand(1, oltp_table_size))
end

for i=1, oltp_simple_ranges do
range_start = sb_rand(1, oltp_table_size)
rs = db_query("SELECT c FROM ".. table_name .." WHERE id BETWEEN " .. range_start .. " AND " .. range_start .. "+" .. oltp_range_size - 1)
end

for i=1, oltp_sum_ranges do
range_start = sb_rand(1, oltp_table_size)
rs = db_query("SELECT SUM(K) FROM ".. table_name .." WHERE id BETWEEN " .. range_start .. " AND " .. range_start .. "+" .. oltp_range_size - 1)
end

for i=1, oltp_order_ranges do
range_start = sb_rand(1, oltp_table_size)
rs = db_query("SELECT c FROM ".. table_name .." WHERE id BETWEEN " .. range_start .. " AND " .. range_start .. "+" .. oltp_range_size - 1 .. " ORDER BY c")
end

for i=1, oltp_distinct_ranges do
range_start = sb_rand(1, oltp_table_size)
rs = db_query("SELECT DISTINCT c FROM ".. table_name .." WHERE id BETWEEN " .. range_start .. " AND " .. range_start .. "+" .. oltp_range_size - 1 .. " ORDER BY c")
end

-- 如果oltp_read_only=on 則跳過DML語句
if not oltp_read_only then

-- 開始執行DML語句
for i=1, oltp_index_updates do
rs = db_query("UPDATE " .. table_name .. " SET k=k+1 WHERE id=" .. sb_rand(1, oltp_table_size))
end

for i=1, oltp_non_index_updates do
c_val = sb_rand_str("###########-###########-###########-###########-###########-###########-###########-###########-###########-###########")
query = "UPDATE " .. table_name .. " SET c='" .. c_val .. "' WHERE id=" .. sb_rand(1, oltp_table_size)
rs = db_query(query)
if rs then
print(query)
end
end

i = sb_rand(1, oltp_table_size)

rs = db_query("DELETE FROM " .. table_name .. " WHERE id=" .. i)

c_val = sb_rand_str([[
###########-###########-###########-###########-###########-###########-###########-###########-###########-###########]]
)
pad_val = sb_rand_str([[
###########-###########-###########-###########-###########]]
)

rs = db_query("INSERT INTO " .. table_name .. " (id, k, c, pad) VALUES " .. string.format("(%d, %d, '%s', '%s')",i, sb_rand(1, oltp_table_size) , c_val, pad_val))

end -- oltp_read_only

-- 如果指定跳過事務參數為ON,測不執行commit
if not oltp_skip_trx then
db_query(commit_query)
end

end

這個腳本就是默認的oltp測試時使用的測試腳本。結合common.lua中初始化的參數,此腳本每個事務中執行了10次基於主鍵的簡單查詢,1次范圍查詢,一次求和計算,一次排序查詢,一次去重加排序查詢,一次小字段更新,一次長字段更新,一次插入。而且在這個腳本中根據oltp_skip_trx oltp_read_only 這兩個參數的限制不同需要執行的語句塊也不同。

五、自定義lua腳本進行oltp性能測試

那么了解了這兩個腳本之后我們就可以根據業務來定制自己的lua腳本了。
場景:在一個業務中有幾個簡單的主鍵查詢,一個轉賬扣款的事務,一個插入語句語句分別是:

select id from test1 where id=:vid;
select id from test2 where id=:vid;
select id from test3 where id=:vid;

start TRANSACTION
update test4 set k=k-1 where id=:vid;

update test5 set k=k+1 where id=:vid;
commit;

insert into test6 (k,xv) values (:vk,:vxv);

那么我們的腳本就可以這么定制:

pathtest = string.match(test, "(.*/)") or ""

dofile(pathtest .. "common.lua")

function thread_init(thread_id)
set_vars()

if (db_driver == "mysql" and mysql_table_engine == "myisam") then
begin_query = "LOCK TABLES sbtest WRITE"
commit_query = "UNLOCK TABLES"
else
begin_query = "BEGIN"
commit_query = "COMMIT"
end
end

function event(thread_id)
local vid1
local vid2
local vid3
local vid4
local vid5
local vk
local vxv
vid1 = sb_rand_uniform(1,10000)
vid2 = sb_rand_uniform(1,10000)
vid3 = sb_rand_uniform(1,10000)
vid4 = sb_rand_uniform(1,10000)
vid5 = sb_rand_uniform(1,10000)
vk = sb_rand_uniform(10,10000)
vxv = sb_rand_str([[###########-###########-###########-###########-###########]])
rs = db_query("SELECT pad FROM test1 WHERE id=" .. vid1)
rs = db_query("SELECT pad FROM test2 WHERE id=" .. vid2)
rs = db_query("SELECT pad FROM test3 WHERE id=" .. vid3)

db_query(begin_query)
rs = db_query("update test4 set k=k-1 where id=" .. vid4)
rs = db_query("update test5 set k=k+1 where id=" .. vid5)
db_query(commit_query)

rs = db_query("insert into test6 (k,xv) values " .. string.format("(%d , '%s')",vk,vxv));

end

我們另存為mytest.lua
然后我們就可以使用如下命令來進行測試了:

./sysbench --mysql-host=127.0.0.1 --mysql-port=3166 --mysql-user=root --mysql-password=123456 --test=tests/db/mytest.lua --num-threads=128 --report-interval=10 --rand-type=uniform --max-time=600 --max-requests=0 run

思考:
現在多機房熱備的構架已經比較普片了,那么跨機房部署還有一個比較影響性能的因素就是網絡吞吐量,sysbench中是否可以加入網絡吞吐量的測試呢?


注意!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系我们删除。



 
粤ICP备14056181号  © 2014-2020 ITdaan.com