TiDB3.0.1、3.0.2版本的TiKV當機對比測試

                                                               -- 2019-08-20  劉春雷

1、彙總

1.1、概述

為了保障業務高可用，進行 TiKV機器當機測試，版本 3.0.1、 3.0.2

1.2、彙總

2、具體測試

2.1、叢集資訊

3個tidb 機器

8個tikv機器

3個tidb例項上面有域名與TGW，會將流量負載到3個tidb例項

2.2、模擬讀寫

sysbench建立 10張表，單表 1kw資料，與其他資料，共計約 1T資料

sysbench 版本：sysbench-1.0.14

sysbench ./tests/include/oltp_legacy/oltp.lua --mysql-host=xxx   --mysql-port=20000 --mysql-user=dba --mysql-password=xxx  --oltp-test-mode=complex  --oltp-tables-count=10 --oltp-table-size=10000000 --threads=16   --time=5200  --report-interval=1 run

2.3、模擬當機

機房關閉電源：關閉tikv機器 1臺

3、3.0.1版本

3.1、 3.0.1版本的監控情況

3.2、3.0.1版本的sysbench情況

sysbench 穩定情況，總的QPS可以達到  1.5w左右，當機後，QPS降低至 0，具體如下：

當機QPS恢復共計需要時間： 205s

[  295s ] thds: 16 tps: 742.05 qps: 14823.91 (r/w/o: 10381.64/2958.18/1484.09) lat (ms,95%): 29.19 err/s: 0.00 reconn/s: 0.00
[ 296s ] thds: 16 tps: 515.99 qps: 10382.87 (r/w/o: 7262.91/2086.97/1032.99) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
[ 297s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 298s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 299s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 300s ] thds: 16 tps: 2.00 qps: 30.00 (r/w/o: 18.00/8.00/4.00) lat (ms,95%): 3706.08 err/s: 0.00 reconn/s: 0.00
[ 301s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 302s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 303s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 304s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 305s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 306s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 307s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 308s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 309s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 310s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 311s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 312s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 313s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 314s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 315s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 316s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 317s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 318s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 319s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 320s ] thds: 16 tps: 2.00 qps: 42.00 (r/w/o: 30.00/8.00/4.00) lat (ms,95%): 20137.61 err/s: 0.00 reconn/s: 0.00
[ 321s ] thds: 16 tps: 5.00 qps: 111.00 (r/w/o: 81.00/20.00/10.00) lat (ms,95%): 25448.93 err/s: 0.00 reconn/s: 0.00
[ 322s ] thds: 16 tps: 17.00 qps: 327.00 (r/w/o: 226.00/67.00/34.00) lat (ms,95%): 25911.33 err/s: 0.00 reconn/s: 0.00
[ 323s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 324s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 325s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 326s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 327s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 328s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 329s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 330s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 331s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 332s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 333s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 334s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 335s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 336s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 337s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 338s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 339s ] thds: 16 tps: 47.00 qps: 971.03 (r/w/o: 680.02/197.01/94.00) lat (ms,95%): 17752.80 err/s: 0.00 reconn/s: 0.00
[ 340s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 341s ] thds: 16 tps: 3.00 qps: 77.99 (r/w/o: 58.99/13.00/6.00) lat (ms,95%): 44472.74 err/s: 0.00 reconn/s: 0.00
[ 342s ] thds: 16 tps: 9.00 qps: 174.02 (r/w/o: 124.01/32.00/18.00) lat (ms,95%): 20137.61 err/s: 0.00 reconn/s: 0.00
[ 343s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 344s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 345s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 346s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 347s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 348s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 349s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 350s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 351s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 352s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 353s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 354s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 355s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 356s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 357s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 358s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 359s ] thds: 16 tps: 20.00 qps: 370.98 (r/w/o: 251.99/79.00/40.00) lat (ms,95%): 20137.61 err/s: 0.00 reconn/s: 0.00
[ 360s ] thds: 16 tps: 4.00 qps: 80.01 (r/w/o: 56.00/16.00/8.00) lat (ms,95%): 20137.61 err/s: 0.00 reconn/s: 0.00
[ 361s ] thds: 16 tps: 5.00 qps: 85.99 (r/w/o: 57.00/19.00/10.00) lat (ms,95%): 64910.76 err/s: 0.00 reconn/s: 0.00
[ 362s ] thds: 16 tps: 4.00 qps: 64.00 (r/w/o: 44.00/12.00/8.00) lat (ms,95%): 20137.61 err/s: 0.00 reconn/s: 0.00
[ 363s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 364s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 365s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 366s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 367s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 368s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 369s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 370s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 371s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 372s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 373s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 374s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 375s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 376s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 377s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 378s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 379s ] thds: 16 tps: 101.01 qps: 1987.25 (r/w/o: 1397.18/388.05/202.03) lat (ms,95%): 20137.61 err/s: 0.00 reconn/s: 0.00
[ 380s ] thds: 16 tps: 4.00 qps: 83.00 (r/w/o: 55.00/20.00/8.00) lat (ms,95%): 19.29 err/s: 0.00 reconn/s: 0.00
[ 381s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 382s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 383s ] thds: 16 tps: 1.00 qps: 27.00 (r/w/o: 17.00/8.00/2.00) lat (ms,95%): 3511.19 err/s: 0.00 reconn/s: 0.00
[ 384s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 385s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 386s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 387s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 388s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 389s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 390s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 391s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 392s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 393s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 394s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 395s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 396s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 397s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 398s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 399s ] thds: 16 tps: 189.01 qps: 3818.11 (r/w/o: 2672.08/768.02/378.01) lat (ms,95%): 19425.29 err/s: 0.00 reconn/s: 0.00
[ 400s ] thds: 16 tps: 207.01 qps: 4154.11 (r/w/o: 2908.08/832.02/414.01) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 401s ] thds: 16 tps: 136.00 qps: 2739.94 (r/w/o: 1922.96/544.99/271.99) lat (ms,95%): 101.13 err/s: 0.00 reconn/s: 0.00
[ 402s ] thds: 16 tps: 153.99 qps: 3057.78 (r/w/o: 2136.85/612.96/307.98) lat (ms,95%): 56.84 err/s: 0.00 reconn/s: 0.00
[ 403s ] thds: 16 tps: 245.02 qps: 4911.46 (r/w/o: 3438.32/983.09/490.05) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 404s ] thds: 16 tps: 201.00 qps: 4024.91 (r/w/o: 2818.94/803.98/401.99) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 405s ] thds: 16 tps: 201.99 qps: 4041.80 (r/w/o: 2827.86/809.96/403.98) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 406s ] thds: 16 tps: 206.99 qps: 4121.88 (r/w/o: 2884.91/822.98/413.99) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 407s ] thds: 16 tps: 194.01 qps: 3888.14 (r/w/o: 2722.10/778.03/388.01) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
[ 408s ] thds: 16 tps: 209.01 qps: 4181.23 (r/w/o: 2924.16/839.05/418.02) lat (ms,95%): 21.89 err/s: 0.00 reconn/s: 0.00
[ 409s ] thds: 16 tps: 168.00 qps: 3370.97 (r/w/o: 2361.98/672.99/336.00) lat (ms,95%): 50.11 err/s: 0.00 reconn/s: 0.00
[ 410s ] thds: 16 tps: 209.00 qps: 4147.08 (r/w/o: 2901.06/828.02/418.01) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 411s ] thds: 16 tps: 211.00 qps: 4214.94 (r/w/o: 2948.96/843.99/421.99) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 412s ] thds: 16 tps: 180.00 qps: 3625.00 (r/w/o: 2545.00/720.00/360.00) lat (ms,95%): 20.37 err/s: 0.00 reconn/s: 0.00
[ 413s ] thds: 16 tps: 137.00 qps: 2738.03 (r/w/o: 1916.02/548.01/274.00) lat (ms,95%): 51.02 err/s: 0.00 reconn/s: 0.00
[ 414s ] thds: 16 tps: 165.00 qps: 3290.97 (r/w/o: 2300.98/659.99/330.00) lat (ms,95%): 20.74 err/s: 0.00 reconn/s: 0.00
[ 415s ] thds: 16 tps: 157.00 qps: 3154.94 (r/w/o: 2205.96/634.99/313.99) lat (ms,95%): 22.28 err/s: 0.00 reconn/s: 0.00
[ 416s ] thds: 16 tps: 165.00 qps: 3297.98 (r/w/o: 2312.98/655.00/330.00) lat (ms,95%): 20.37 err/s: 0.00 reconn/s: 0.00
[ 417s ] thds: 16 tps: 164.00 qps: 3276.08 (r/w/o: 2286.05/662.02/328.01) lat (ms,95%): 20.74 err/s: 0.00 reconn/s: 0.00
[ 418s ] thds: 16 tps: 161.01 qps: 3195.13 (r/w/o: 2238.09/636.02/321.01) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 419s ] thds: 16 tps: 333.99 qps: 6648.85 (r/w/o: 4663.89/1315.97/668.98) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
[ 420s ] thds: 16 tps: 443.99 qps: 8858.79 (r/w/o: 6194.85/1775.96/887.98) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 421s ] thds: 16 tps: 264.01 qps: 5331.11 (r/w/o: 3735.08/1068.02/528.01) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 422s ] thds: 16 tps: 50.00 qps: 999.98 (r/w/o: 699.99/200.00/100.00) lat (ms,95%): 19.65 err/s: 0.00 reconn/s: 0.00
[ 423s ] thds: 16 tps: 72.00 qps: 1421.98 (r/w/o: 990.99/288.00/143.00) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 424s ] thds: 16 tps: 129.00 qps: 2594.07 (r/w/o: 1815.05/520.01/259.01) lat (ms,95%): 22.28 err/s: 0.00 reconn/s: 0.00
[ 425s ] thds: 16 tps: 161.00 qps: 3218.98 (r/w/o: 2256.99/640.00/322.00) lat (ms,95%): 21.11 err/s: 0.00 reconn/s: 0.00
[ 426s ] thds: 16 tps: 160.00 qps: 3217.01 (r/w/o: 2252.01/645.00/320.00) lat (ms,95%): 21.50 err/s: 0.00 reconn/s: 0.00
[ 427s ] thds: 16 tps: 160.00 qps: 3200.08 (r/w/o: 2237.05/643.02/320.01) lat (ms,95%): 21.11 err/s: 0.00 reconn/s: 0.00
[ 428s ] thds: 16 tps: 160.00 qps: 3174.94 (r/w/o: 2222.96/631.99/319.99) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 429s ] thds: 16 tps: 159.97 qps: 3214.36 (r/w/o: 2254.55/639.87/319.94) lat (ms,95%): 21.50 err/s: 0.00 reconn/s: 0.00
[ 430s ] thds: 16 tps: 158.02 qps: 3163.40 (r/w/o: 2211.28/636.08/316.04) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 431s ] thds: 16 tps: 161.01 qps: 3205.28 (r/w/o: 2243.20/640.06/322.03) lat (ms,95%): 21.89 err/s: 0.00 reconn/s: 0.00
[ 432s ] thds: 16 tps: 181.00 qps: 3628.98 (r/w/o: 2542.99/724.00/362.00) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 433s ] thds: 16 tps: 204.00 qps: 4092.04 (r/w/o: 2864.03/820.01/408.00) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 434s ] thds: 16 tps: 208.98 qps: 4163.67 (r/w/o: 2909.77/835.93/417.97) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 435s ] thds: 16 tps: 192.01 qps: 3853.24 (r/w/o: 2705.17/764.05/384.02) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 436s ] thds: 16 tps: 212.00 qps: 4228.94 (r/w/o: 2956.96/847.99/423.99) lat (ms,95%): 21.89 err/s: 0.00 reconn/s: 0.00
[ 437s ] thds: 16 tps: 209.99 qps: 4211.71 (r/w/o: 2945.80/845.94/419.97) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 438s ] thds: 16 tps: 208.01 qps: 4160.29 (r/w/o: 2910.20/834.06/416.03) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 439s ] thds: 16 tps: 482.00 qps: 9600.91 (r/w/o: 6737.94/1899.98/962.99) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 440s ] thds: 16 tps: 506.92 qps: 10137.37 (r/w/o: 7092.86/2029.67/1014.84) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 441s ] thds: 16 tps: 471.09 qps: 9472.88 (r/w/o: 6642.32/1888.37/942.19) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 442s ] thds: 16 tps: 419.98 qps: 8338.64 (r/w/o: 5826.75/1671.93/839.96) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 443s ] thds: 16 tps: 393.01 qps: 7926.24 (r/w/o: 5555.17/1585.05/786.02) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 444s ] thds: 16 tps: 405.00 qps: 8082.93 (r/w/o: 5658.95/1614.99/808.99) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 445s ] thds: 16 tps: 319.00 qps: 6359.01 (r/w/o: 4452.00/1268.00/639.00) lat (ms,95%): 45.79 err/s: 0.00 reconn/s: 0.00
[ 446s ] thds: 16 tps: 412.99 qps: 8269.72 (r/w/o: 5783.81/1659.94/825.97) lat (ms,95%): 22.28 err/s: 0.00 reconn/s: 0.00
[ 447s ] thds: 16 tps: 366.01 qps: 7292.17 (r/w/o: 5104.12/1456.03/732.02) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 448s ] thds: 16 tps: 327.00 qps: 6552.09 (r/w/o: 4586.06/1312.02/654.01) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 449s ] thds: 16 tps: 262.00 qps: 5249.05 (r/w/o: 3673.03/1052.01/524.00) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 450s ] thds: 16 tps: 262.01 qps: 5215.11 (r/w/o: 3646.08/1045.02/524.01) lat (ms,95%): 22.28 err/s: 0.00 reconn/s: 0.00
[ 451s ] thds: 16 tps: 260.99 qps: 5249.88 (r/w/o: 3676.91/1050.98/521.99) lat (ms,95%): 22.28 err/s: 0.00 reconn/s: 0.00
[ 452s ] thds: 16 tps: 234.99 qps: 4691.88 (r/w/o: 3283.92/937.98/469.99) lat (ms,95%): 33.72 err/s: 0.00 reconn/s: 0.00
[ 453s ] thds: 16 tps: 227.00 qps: 4508.93 (r/w/o: 3156.95/897.99/453.99) lat (ms,95%): 36.24 err/s: 0.00 reconn/s: 0.00
[ 454s ] thds: 16 tps: 256.00 qps: 5122.04 (r/w/o: 3586.03/1024.01/512.00) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 455s ] thds: 16 tps: 256.00 qps: 5140.07 (r/w/o: 3596.05/1032.01/512.01) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 456s ] thds: 16 tps: 261.98 qps: 5208.69 (r/w/o: 3640.79/1043.94/523.97) lat (ms,95%): 22.28 err/s: 0.00 reconn/s: 0.00
[ 457s ] thds: 16 tps: 261.02 qps: 5242.35 (r/w/o: 3678.25/1042.07/522.04) lat (ms,95%): 21.89 err/s: 0.00 reconn/s: 0.00
[ 458s ] thds: 16 tps: 254.98 qps: 5110.58 (r/w/o: 3582.70/1017.92/509.96) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 459s ] thds: 16 tps: 265.02 qps: 5287.45 (r/w/o: 3697.32/1060.09/530.05) lat (ms,95%): 27.66 err/s: 0.00 reconn/s: 0.00
[ 460s ] thds: 16 tps: 326.00 qps: 6525.04 (r/w/o: 4562.03/1311.01/652.00) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 461s ] thds: 16 tps: 273.95 qps: 5476.93 (r/w/o: 3831.25/1097.79/547.89) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 462s ] thds: 16 tps: 312.04 qps: 6243.73 (r/w/o: 4373.51/1246.15/624.07) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 463s ] thds: 16 tps: 308.02 qps: 6143.40 (r/w/o: 4295.28/1232.08/616.04) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 464s ] thds: 16 tps: 307.99 qps: 6166.88 (r/w/o: 4322.92/1227.98/615.99) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 465s ] thds: 16 tps: 314.98 qps: 6300.57 (r/w/o: 4405.70/1265.91/628.96) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 466s ] thds: 16 tps: 311.02 qps: 6229.36 (r/w/o: 4363.26/1243.07/623.04) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 467s ] thds: 16 tps: 357.01 qps: 7129.24 (r/w/o: 4992.17/1423.05/714.02) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 468s ] thds: 16 tps: 385.00 qps: 7729.05 (r/w/o: 5411.03/1548.01/770.00) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 469s ] thds: 16 tps: 453.99 qps: 9082.87 (r/w/o: 6365.91/1808.97/907.99) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 470s ] thds: 16 tps: 454.97 qps: 9088.50 (r/w/o: 6350.65/1827.90/909.95) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 471s ] thds: 16 tps: 458.03 qps: 9128.50 (r/w/o: 6389.35/1823.10/916.05) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 472s ] thds: 16 tps: 455.00 qps: 9124.00 (r/w/o: 6394.00/1820.00/910.00) lat (ms,95%): 22.69 err/s: 0.00 reconn/s: 0.00
[ 473s ] thds: 16 tps: 447.01 qps: 8975.21 (r/w/o: 6285.15/1796.04/894.02) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 474s ] thds: 16 tps: 461.99 qps: 9205.71 (r/w/o: 6441.80/1839.94/923.97) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 475s ] thds: 16 tps: 435.99 qps: 8718.90 (r/w/o: 6098.93/1747.98/871.99) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
[ 476s ] thds: 16 tps: 466.00 qps: 9345.92 (r/w/o: 6546.94/1866.98/931.99) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 477s ] thds: 16 tps: 464.02 qps: 9255.48 (r/w/o: 6480.33/1847.10/928.05) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 478s ] thds: 16 tps: 458.99 qps: 9176.85 (r/w/o: 6424.90/1833.97/917.99) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 479s ] thds: 16 tps: 492.97 qps: 9870.35 (r/w/o: 6909.55/1974.87/985.94) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 480s ] thds: 16 tps: 610.04 qps: 12211.89 (r/w/o: 8538.62/2453.18/1220.09) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 481s ] thds: 16 tps: 694.92 qps: 13922.43 (r/w/o: 9755.90/2776.69/1389.84) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 482s ] thds: 16 tps: 672.08 qps: 13386.68 (r/w/o: 9363.18/2681.34/1342.17) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 483s ] thds: 16 tps: 645.98 qps: 12942.64 (r/w/o: 9055.75/2592.93/1293.96) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 484s ] thds: 16 tps: 572.98 qps: 11473.67 (r/w/o: 8032.77/2294.93/1145.97) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 485s ] thds: 16 tps: 516.02 qps: 10275.40 (r/w/o: 7187.28/2057.08/1031.04) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00
[ 486s ] thds: 16 tps: 440.01 qps: 8795.10 (r/w/o: 6165.07/1749.02/881.01) lat (ms,95%): 31.94 err/s: 0.00 reconn/s: 0.00
[ 487s ] thds: 16 tps: 508.99 qps: 10192.89 (r/w/o: 7135.92/2038.98/1017.99) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 488s ] thds: 16 tps: 487.00 qps: 9773.01 (r/w/o: 6835.00/1964.00/974.00) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 489s ] thds: 16 tps: 496.99 qps: 9915.75 (r/w/o: 6953.83/1967.95/993.98) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 490s ] thds: 16 tps: 510.02 qps: 10156.36 (r/w/o: 7097.25/2040.07/1019.04) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 491s ] thds: 16 tps: 501.96 qps: 10092.17 (r/w/o: 7078.42/2008.83/1004.92) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 492s ] thds: 16 tps: 514.04 qps: 10236.89 (r/w/o: 7149.62/2059.18/1028.09) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 493s ] thds: 16 tps: 507.99 qps: 10196.84 (r/w/o: 7144.89/2035.97/1015.98) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 494s ] thds: 16 tps: 512.95 qps: 10203.01 (r/w/o: 7126.31/2050.80/1025.90) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00
[ 495s ] thds: 16 tps: 501.05 qps: 10075.10 (r/w/o: 7072.78/2000.22/1002.11) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 496s ] thds: 16 tps: 505.99 qps: 10111.88 (r/w/o: 7070.91/2028.98/1011.99) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00
[ 497s ] thds: 16 tps: 465.01 qps: 9304.27 (r/w/o: 6514.19/1860.05/930.03) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 498s ] thds: 16 tps: 451.99 qps: 9058.78 (r/w/o: 6342.84/1811.96/903.98) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 499s ] thds: 16 tps: 671.00 qps: 13409.02 (r/w/o: 9388.02/2680.00/1341.00) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[  500s ] thds: 16 tps: 809.00 qps: 16166.96 (r/w/o: 11314.97/3232.99/1619.00) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 501s ] thds: 16 tps: 799.01 qps: 16007.17 (r/w/o: 11200.12/3210.03/1597.02) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 502s ] thds: 16 tps: 807.00 qps: 16130.00 (r/w/o: 11294.00/3221.00/1615.00) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 503s ] thds: 16 tps: 801.97 qps: 16073.38 (r/w/o: 11257.57/3211.88/1603.94) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
[ 504s ] thds: 16 tps: 807.99 qps: 16132.85 (r/w/o: 11287.89/3229.97/1614.98) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 505s ] thds: 16 tps: 788.01 qps: 15775.16 (r/w/o: 11058.11/3140.03/1577.02) lat (ms,95%): 26.20 err/s: 0.00 reconn/s: 0.00
[ 506s ] thds: 16 tps: 818.05 qps: 16359.98 (r/w/o: 11434.68/3289.20/1636.10) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 507s ] thds: 16 tps: 800.94 qps: 15970.81 (r/w/o: 11178.17/3192.76/1599.88) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 508s ] thds: 16 tps: 803.02 qps: 16129.42 (r/w/o: 11301.29/3220.08/1608.04) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
[ 509s ] thds: 16 tps: 815.02 qps: 16225.49 (r/w/o: 11347.34/3248.10/1630.05) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 510s ] thds: 16 tps: 803.98 qps: 16123.53 (r/w/o: 11282.67/3232.91/1607.95) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 511s ] thds: 16 tps: 807.96 qps: 16139.28 (r/w/o: 11304.50/3218.86/1615.93) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 512s ] thds: 16 tps: 778.01 qps: 15525.17 (r/w/o: 10867.12/3106.03/1552.02) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 513s ] thds: 16 tps: 731.00 qps: 14669.09 (r/w/o: 10273.06/2930.02/1466.01) lat (ms,95%): 29.72 err/s: 0.00 reconn/s: 0.00
[ 514s ] thds: 16 tps: 779.01 qps: 15588.11 (r/w/o: 10912.08/3118.02/1558.01) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 515s ] thds: 16 tps: 791.96 qps: 15803.17 (r/w/o: 11060.42/3159.83/1582.92) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 516s ] thds: 16 tps: 805.96 qps: 16141.26 (r/w/o: 11292.48/3235.85/1612.93) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 517s ] thds: 16 tps: 797.99 qps: 15946.88 (r/w/o: 11161.92/3189.98/1594.99) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 518s ] thds: 16 tps: 761.06 qps: 15269.14 (r/w/o: 10692.80/3053.23/1523.11) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[ 519s ] thds: 16 tps: 791.03 qps: 15776.57 (r/w/o: 11044.40/3151.11/1581.06) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 520s ] thds: 16 tps: 788.01 qps: 15798.21 (r/w/o: 11067.15/3154.04/1577.02) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 521s ] thds: 16 tps: 803.01 qps: 16008.14 (r/w/o: 11192.10/3211.03/1605.01) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
[ 522s ] thds: 16 tps: 766.02 qps: 15347.33 (r/w/o: 10749.23/3065.07/1533.03) lat (ms,95%): 26.20 err/s: 0.00 reconn/s: 0.00
[ 523s ] thds: 16 tps: 735.00 qps: 14681.91 (r/w/o: 10274.94/2937.98/1468.99) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 524s ] thds: 16 tps: 744.86 qps: 14903.15 (r/w/o: 10441.01/2971.43/1490.72) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
[ 525s ] thds: 16 tps: 747.07 qps: 14939.41 (r/w/o: 10456.99/2989.28/1493.14) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 526s ] thds: 16 tps: 764.07 qps: 15293.37 (r/w/o: 10696.96/3067.28/1529.14) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 527s ] thds: 16 tps: 742.97 qps: 14883.41 (r/w/o: 10422.59/2974.88/1485.94) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00
[ 528s ] thds: 16 tps: 757.96 qps: 15128.11 (r/w/o: 10579.37/3032.82/1515.91) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.00
[ 529s ] thds: 16 tps: 761.98 qps: 15238.65 (r/w/o: 10671.75/3043.93/1522.96) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 530s ] thds: 16 tps: 739.09 qps: 14794.83 (r/w/o: 10356.28/2960.37/1478.18) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 531s ] thds: 16 tps: 755.98 qps: 15139.52 (r/w/o: 10604.66/3021.90/1512.95) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 532s ] thds: 16 tps: 745.98 qps: 14933.54 (r/w/o: 10453.68/2987.91/1491.95) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 533s ] thds: 16 tps: 746.01 qps: 14926.30 (r/w/o: 10449.21/2985.06/1492.03) lat (ms,95%): 27.66 err/s: 0.00 reconn/s: 0.00
[ 534s ] thds: 16 tps: 765.00 qps: 15281.09 (r/w/o: 10696.06/3055.02/1530.01) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 535s ] thds: 16 tps: 754.00 qps: 15061.04 (r/w/o: 10544.03/3010.01/1507.00) lat (ms,95%): 25.74 err/s: 0.00 reconn/s: 0.00
[ 536s ] thds: 16 tps: 764.03 qps: 15294.57 (r/w/o: 10705.40/3060.11/1529.06) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00
[ 537s ] thds: 16 tps: 765.00 qps: 15293.03 (r/w/o: 10704.02/3059.01/1530.00) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00
[ 538s ] thds: 16 tps: 749.90 qps: 15000.07 (r/w/o: 10497.65/3002.61/1499.81) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 539s ] thds: 16 tps: 774.08 qps: 15479.54 (r/w/o: 10841.08/3090.31/1548.15) lat (ms,95%): 24.38 err/s: 0.00 reconn/s: 0.00
[ 540s ] thds: 16 tps: 798.99 qps: 15936.88 (r/w/o: 11150.91/3188.98/1596.99) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00

4、3.0.2版本

4.1、3.0.2版本的監控情況

從QPS表，可以看出QPS大約 1分鐘左右 後就開始恢復正常

4.2、3.0.2版本的sysbench情況

sysbench 穩定情況，總的QPS可以達到  1.5w左右，當機後，QPS降低至 0，具體如下：

當機QPS恢復共計需要時間： 25s

[ 570s ] thds: 16 tps: 692.01 qps: 13816.16 (r/w/o: 9644.11/2788.03/1384.02) lat (ms,95%): 35.59 err/s: 0.00 reconn/s: 0.00
[ 571s ] thds: 16 tps: 725.01 qps: 14507.23 (r/w/o: 10170.16/2888.05/1449.02) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[ 572s ] thds: 16 tps: 742.02 qps: 14847.34 (r/w/o: 10392.24/2970.07/1485.03) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00
[ 573s ] thds: 16 tps: 718.99 qps: 14405.76 (r/w/o: 10086.83/2880.95/1437.98) lat (ms,95%): 31.94 err/s: 0.00 reconn/s: 0.00
[ 574s ] thds: 16 tps: 723.01 qps: 14471.16 (r/w/o: 10123.11/2902.03/1446.02) lat (ms,95%): 30.81 err/s: 0.00 reconn/s: 0.00
[ 575s ] thds: 16 tps: 728.94 qps: 14557.71 (r/w/o: 10195.10/2905.74/1456.87) lat (ms,95%): 30.81 err/s: 0.00 reconn/s: 0.00
[ 576s ] thds: 16 tps: 741.06 qps: 14826.24 (r/w/o: 10378.87/2964.25/1483.12) lat (ms,95%): 28.16 err/s: 0.00 reconn/s: 0.00
[  577s ] thds: 16 tps: 738.98 qps: 14702.55 (r/w/o: 10275.68/2948.91/1477.95) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
[ 578s ] thds: 16 tps: 79.00 qps: 1605.01 (r/w/o: 1127.00/320.00/158.00) lat (ms,95%): 33.12 err/s: 0.00 reconn/s: 0.00
[ 579s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 580s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 581s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 582s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 583s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 584s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 585s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 586s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 587s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 588s ] thds: 16 tps: 3.00 qps: 93.99 (r/w/o: 64.00/24.00/6.00) lat (ms,95%): 10917.50 err/s: 0.00 reconn/s: 0.00
[ 589s ] thds: 16 tps: 14.00 qps: 303.99 (r/w/o: 215.99/60.00/28.00) lat (ms,95%): 11317.84 err/s: 0.00 reconn/s: 0.00
[ 590s ] thds: 16 tps: 9.00 qps: 222.00 (r/w/o: 152.00/52.00/18.00) lat (ms,95%): 12384.09 err/s: 0.00 reconn/s: 0.00
[ 591s ] thds: 16 tps: 9.00 qps: 192.00 (r/w/o: 126.00/48.00/18.00) lat (ms,95%): 13550.80 err/s: 0.00 reconn/s: 0.00
[ 592s ] thds: 16 tps: 7.00 qps: 140.00 (r/w/o: 98.00/28.00/14.00) lat (ms,95%): 14302.94 err/s: 0.00 reconn/s: 0.00
[ 593s ] thds: 16 tps: 14.00 qps: 309.00 (r/w/o: 217.00/64.00/28.00) lat (ms,95%): 15371.13 err/s: 0.00 reconn/s: 0.00
[ 594s ] thds: 16 tps: 26.00 qps: 515.04 (r/w/o: 363.03/100.01/52.00) lat (ms,95%): 3639.94 err/s: 0.00 reconn/s: 0.00
[ 595s ] thds: 16 tps: 48.00 qps: 964.99 (r/w/o: 672.99/196.00/96.00) lat (ms,95%): 3639.94 err/s: 0.00 reconn/s: 0.00
[ 596s ] thds: 16 tps: 50.00 qps: 992.00 (r/w/o: 696.00/196.00/100.00) lat (ms,95%): 3040.14 err/s: 0.00 reconn/s: 0.00
[ 597s ] thds: 16 tps: 5.00 qps: 106.00 (r/w/o: 74.00/22.00/10.00) lat (ms,95%): 2198.52 err/s: 0.00 reconn/s: 0.00
[ 598s ] thds: 16 tps: 0.00 qps: 2.00 (r/w/o: 0.00/2.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 599s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 600s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
[ 601s ] thds: 16 tps: 416.00 qps: 8222.98 (r/w/o: 5776.98/1614.00/832.00) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[  602s ] thds: 16 tps: 786.97 qps: 15673.36 (r/w/o: 10962.55/3136.87/1573.94) lat (ms,95%): 26.20 err/s: 0.00 reconn/s: 0.00
[ 603s ] thds: 16 tps: 718.03 qps: 14366.67 (r/w/o: 10049.47/2881.13/1436.07) lat (ms,95%): 30.26 err/s: 0.00 reconn/s: 0.00
[ 604s ] thds: 16 tps: 790.00 qps: 15783.05 (r/w/o: 11055.03/3148.01/1580.00) lat (ms,95%): 28.67 err/s: 0.00 reconn/s: 0.00
[ 605s ] thds: 16 tps: 776.99 qps: 15600.77 (r/w/o: 10927.84/3118.95/1553.98) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 606s ] thds: 16 tps: 789.00 qps: 15766.98 (r/w/o: 11036.98/3154.00/1576.00) lat (ms,95%): 26.68 err/s: 0.00 reconn/s: 0.00
[ 607s ] thds: 16 tps: 792.91 qps: 15845.11 (r/w/o: 11093.67/3163.62/1587.81) lat (ms,95%): 27.17 err/s: 0.00 reconn/s: 0.00
[ 608s ] thds: 16 tps: 783.09 qps: 15672.81 (r/w/o: 10962.27/3145.36/1565.18) lat (ms,95%): 29.19 err/s: 0.00 reconn/s: 0.00
[ 609s ] thds: 16 tps: 768.92 qps: 15401.47 (r/w/o: 10775.93/3086.69/1538.85) lat (ms,95%): 27.66 err/s: 0.00 reconn/s: 0.00

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28823725/viewspace-2654413/，如需轉載，請註明出處，否則將追究法律責任。