FIO的再学习-不同Raid性能验证

背景

发现自己对iodepth的和num_jobs的理解存在偏差
找了一些资料才发现自己很多地方做的不对. 
这里找到一个新资料可以进行模拟数据库的测试

测试配置文件

# R/W: 8/2
# ThreadPool Num: 64
#  IO ThreadNum: 32
 
[global]
runtime=120
time_based
group_reporting
directory=/fiotest
refill_buffers

[innodb-trxlog]
filename=test-innodb.log
bsrange=512-2048
ioengine=sync
rw=write
size=2G
fsync=1
overwrite=1
rate_iops=64
invalidate=1
numjobs=64

 
[mysql-binlog]
filename=test-mysql-bin.log
bsrange=512-1024
ioengine=sync
rw=write
size=12G
sync=1
rw=write
overwrite=1
fsync=100
rate_iops=64
invalidate=1
numjobs=64
 
[innodb-data]
filename=test-innodb.dat
bs=16K
ioengine=psync
rw=randrw
size=20G
direct=1
rwmixread=80
numjobs=32
 
#thinktime=600
#thinktime_spin=200
#thinktime_blocks=2

鲲鹏下Raid卡的性能说明

测试方式

fio patch/to/mysql.conf

鲲鹏下Raid卡的性能说明

配置信息: SAS3508
内存: 2GB
设备类型:LSI SAS3508
RAID5  256KB条带  7.633TB容量
Write back 等写策略
8块 硬盘 SAS 1.2TB 10Krpm
希捷: ST1200MM0009

测试结果验证

不同Raid级别的验证

华为的BMC配置界面

• 注意华为的服务器可以直接在线修改raid卡的设置不需要重启
• 浪潮的服务器需要在bios界面中进行修改. 我验证了下好像修改还存在问题. 性能测试结果不是很正常.

12块盘RAID10 SSD的测试结果为:

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=19168: Mon Jun 26 13:55:14 2023
   read: IOPS=80.5k, BW=1258MiB/s (1319MB/s)(147GiB/120002msec)
    clat (usec): min=81, max=4071, avg=336.16, stdev=114.42
     lat (usec): min=81, max=4071, avg=336.21, stdev=114.42
    clat percentiles (usec):
     |  1.00th=[  184],  5.00th=[  206], 10.00th=[  223], 20.00th=[  245],
     | 30.00th=[  269], 40.00th=[  289], 50.00th=[  314], 60.00th=[  338],
     | 70.00th=[  371], 80.00th=[  412], 90.00th=[  478], 95.00th=[  545],
     | 99.00th=[  693], 99.50th=[  766], 99.90th=[  996], 99.95th=[ 1254],
     | 99.99th=[ 2311]
   bw (  KiB/s): min=34368, max=42400, per=3.12%, avg=40245.47, stdev=781.54, samples=7653
   iops        : min= 2148, max= 2650, avg=2515.31, stdev=48.84, samples=7653
  write: IOPS=26.5k, BW=318MiB/s (334MB/s)(37.3GiB/120032msec)
    clat (nsec): min=1737, max=21641k, avg=874824.91, stdev=1517747.37
     lat (nsec): min=1818, max=21641k, avg=874909.49, stdev=1517778.19
    clat percentiles (usec):
     |  1.00th=[   60],  5.00th=[  135], 10.00th=[  151], 20.00th=[  176],
     | 30.00th=[  200], 40.00th=[  225], 50.00th=[  253], 60.00th=[  289],
     | 70.00th=[  351], 80.00th=[  873], 90.00th=[ 3032], 95.00th=[ 4686],
     | 99.00th=[ 6521], 99.50th=[ 7177], 99.90th=[10290], 99.95th=[12256],
     | 99.99th=[16057]
   bw (  KiB/s): min=   28, max=12160, per=0.62%, avg=2033.61, stdev=4015.57, samples=38329
   iops        : min=   24, max=  760, avg=165.67, stdev=232.07, samples=38329
  lat (usec)   : 2=0.01%, 4=0.06%, 10=0.14%, 20=0.02%, 50=0.02%
  lat (usec)   : 100=0.04%, 250=28.44%, 500=59.71%, 750=6.01%, 1000=0.68%
  lat (msec)   : 2=1.32%, 4=1.85%, 10=1.67%, 20=0.03%, 50=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=1170, max=5787.9k, avg=680462.27, stdev=346724.54
    sync percentiles (usec):
     |  1.00th=[  202],  5.00th=[  285], 10.00th=[  334], 20.00th=[  412],
     | 30.00th=[  478], 40.00th=[  545], 50.00th=[  611], 60.00th=[  685],
     | 70.00th=[  783], 80.00th=[  898], 90.00th=[ 1090], 95.00th=[ 1303],
     | 99.00th=[ 1893], 99.50th=[ 2180], 99.90th=[ 2933], 99.95th=[ 3294],
     | 99.99th=[ 4424]
  cpu          : usr=0.24%, sys=1.79%, ctx=17381473, majf=0, minf=4971
  IO depths    : 1=102.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=9659384,3185632,0,285399 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1258MiB/s (1319MB/s), 1258MiB/s-1258MiB/s (1319MB/s-1319MB/s), io=147GiB (158GB), run=120002-120002msec
  WRITE: bw=318MiB/s (334MB/s), 318MiB/s-318MiB/s (334MB/s-334MB/s), io=37.3GiB (40.1GB), run=120032-120032msec

Disk stats (read/write):
    dm-0: ios=9660460/3275091, merge=0/0, ticks=3119334/703370, in_queue=3879614, util=100.00%, aggrios=9667470/3277750, aggrmerge=0/112, aggrticks=3119830/702969, aggrin_queue=3836192, aggrutil=100.00%
  sda: ios=9667470/3277750, merge=0/112, ticks=3119830/702969, in_queue=3836192, util=100.00%

八块盘Raid5 的结果为

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=3349911: Mon Jun 26 14:05:22 2023
  read: IOPS=45.7k, BW=714MiB/s (749MB/s)(83.7GiB/120002msec)
    clat (usec): min=68, max=15400, avg=462.16, stdev=379.33
     lat (usec): min=68, max=15400, avg=462.27, stdev=379.34
    clat percentiles (usec):
     |  1.00th=[  159],  5.00th=[  184], 10.00th=[  204], 20.00th=[  237],
     | 30.00th=[  269], 40.00th=[  306], 50.00th=[  347], 60.00th=[  396],
     | 70.00th=[  461], 80.00th=[  570], 90.00th=[  824], 95.00th=[ 1205],
     | 99.00th=[ 2147], 99.50th=[ 2507], 99.90th=[ 3195], 99.95th=[ 3458],
     | 99.99th=[ 4113]
   bw (  KiB/s): min=444651, max=1057870, per=99.99%, avg=730974.33, stdev=5257.83, samples=7616
   iops        : min=27777, max=66094, avg=45672.03, stdev=328.64, samples=7616
  write: IOPS=15.1k, BW=182MiB/s (191MB/s)(21.3GiB/120047msec); 0 zone resets
    clat (nsec): min=1652, max=286234k, avg=6657898.82, stdev=13742171.43
     lat (nsec): min=1764, max=286235k, avg=6658109.58, stdev=13742230.93
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[   396], 10.00th=[   453], 20.00th=[   537],
     | 30.00th=[   619], 40.00th=[   709], 50.00th=[   832], 60.00th=[  1057],
     | 70.00th=[  1696], 80.00th=[  5014], 90.00th=[ 28181], 95.00th=[ 36439],
     | 99.00th=[ 59507], 99.50th=[ 68682], 99.90th=[ 93848], 99.95th=[114820],
     | 99.99th=[185598]
   bw (  KiB/s): min=107089, max=280602, per=99.96%, avg=186070.58, stdev=598.45, samples=38080
   iops        : min= 7764, max=23201, avg=14992.59, stdev=37.86, samples=38080
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.08%, 20=0.14%, 50=0.07%
  lat (usec)   : 100=0.01%, 250=18.07%, 500=41.23%, 750=17.55%, 1000=7.23%
  lat (msec)   : 2=7.68%, 4=2.62%, 10=1.05%, 20=0.59%, 50=3.18%
  lat (msec)   : 100=0.46%, 250=0.02%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=2, max=411881, avg=2104.76, stdev=3465.30
    sync percentiles (usec):
     |  1.00th=[   416],  5.00th=[   594], 10.00th=[   701], 20.00th=[   865],
     | 30.00th=[  1012], 40.00th=[  1188], 50.00th=[  1385], 60.00th=[  1631],
     | 70.00th=[  1958], 80.00th=[  2474], 90.00th=[  3785], 95.00th=[  5604],
     | 99.00th=[ 12387], 99.50th=[ 17957], 99.90th=[ 47449], 99.95th=[ 63177],
     | 99.99th=[101188]
  cpu          : usr=0.13%, sys=0.60%, ctx=9090894, majf=0, minf=2337
  IO depths    : 1=102.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5482753,1814540,0,198984 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=714MiB/s (749MB/s), 714MiB/s-714MiB/s (749MB/s-749MB/s), io=83.7GiB (89.8GB), run=120002-120002msec
  WRITE: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=21.3GiB (22.9GB), run=120047-120047msec

Disk stats (read/write):
    dm-2: ios=5474810/1862492, merge=0/0, ticks=2473069/2652106, in_queue=5125175, util=99.96%, aggrios=5502046/1930566, aggrmerge=261/6520, aggrticks=2488430/2851668, aggrin_queue=5340099, aggrutil=99.96%
  sda: ios=5502046/1930566, merge=261/6520, ticks=2488430/2851668, in_queue=5340099, util=99.96%

鲲鹏 8块1.2T HDD raid5 (write back)

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=302295: Mon Jun 26 14:10:12 2023
   read: IOPS=2241, BW=35.0MiB/s (36.7MB/s)(4203MiB/120021msec)
    clat (usec): min=27, max=1504.3k, avg=14220.39, stdev=33655.76
     lat (usec): min=27, max=1504.3k, avg=14220.55, stdev=33655.76
    clat percentiles (usec):
     |  1.00th=[    36],  5.00th=[  1401], 10.00th=[  2147], 20.00th=[  3490],
     | 30.00th=[  4686], 40.00th=[  5800], 50.00th=[  6915], 60.00th=[  8979],
     | 70.00th=[ 11469], 80.00th=[ 15795], 90.00th=[ 25560], 95.00th=[ 40633],
     | 99.00th=[147850], 99.50th=[221250], 99.90th=[446694], 99.95th=[549454],
     | 99.99th=[876610]
   bw (  KiB/s): min=   31, max= 2592, per=3.16%, avg=1131.76, stdev=724.80, samples=7603
   iops        : min=    1, max=  162, avg=70.69, stdev=45.31, samples=7603
  write: IOPS=6989, BW=12.8MiB/s (13.4MB/s)(1532MiB/120032msec)
    clat (nsec): min=1700, max=917451k, avg=1934473.82, stdev=12258242.35
     lat (nsec): min=1890, max=917451k, avg=1934709.17, stdev=12258242.07
    clat percentiles (usec):
     |  1.00th=[     5],  5.00th=[     7], 10.00th=[    37], 20.00th=[   112],
     | 30.00th=[   293], 40.00th=[   537], 50.00th=[   906], 60.00th=[  1450],
     | 70.00th=[  1827], 80.00th=[  2311], 90.00th=[  2900], 95.00th=[  4080],
     | 99.00th=[  8848], 99.50th=[ 34341], 99.90th=[139461], 99.95th=[229639],
     | 99.99th=[574620]
   bw (  KiB/s): min=    7, max= 1056, per=0.64%, avg=83.59, stdev=137.92, samples=37492
   iops        : min=    1, max=  110, avg=44.65, stdev=18.72, samples=37492
  lat (usec)   : 2=0.01%, 4=0.75%, 10=4.16%, 20=0.72%, 50=5.44%
  lat (usec)   : 100=4.37%, 250=6.45%, 500=8.22%, 750=5.95%, 1000=3.94%
  lat (msec)   : 2=18.04%, 4=19.59%, 10=12.97%, 20=5.36%, 50=2.83%
  lat (msec)   : 100=0.66%, 250=0.41%, 500=0.10%, 750=0.02%, 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=990, max=100194k, avg=185808.60, stdev=373956.00
    sync percentiles (usec):
     |  1.00th=[   55],  5.00th=[   73], 10.00th=[   87], 20.00th=[  119],
     | 30.00th=[  133], 40.00th=[  143], 50.00th=[  151], 60.00th=[  163],
     | 70.00th=[  194], 80.00th=[  229], 90.00th=[  285], 95.00th=[  363],
     | 99.00th=[  766], 99.50th=[ 1057], 99.90th=[ 2057], 99.95th=[ 2573],
     | 99.99th=[ 3032]
  cpu          : usr=0.04%, sys=0.27%, ctx=4245205, majf=0, minf=1136
  IO depths    : 1=125.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=268975,839013,0,285399 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=35.0MiB/s (36.7MB/s), 35.0MiB/s-35.0MiB/s (36.7MB/s-36.7MB/s), io=4203MiB (4407MB), run=120021-120021msec
  WRITE: bw=12.8MiB/s (13.4MB/s), 12.8MiB/s-12.8MiB/s (13.4MB/s-13.4MB/s), io=1532MiB (1606MB), run=120032-120032msec

Disk stats (read/write):
    dm-0: ios=270740/770085, merge=0/0, ticks=3826990/58840, in_queue=3885830, util=99.67%, aggrios=270904/770460, aggrmerge=0/182, aggrticks=3832669/63246, aggrin_queue=2604470, aggrutil=99.65%
  sda: ios=270904/770460, merge=0/182, ticks=3832669/63246, in_queue=2604470, util=99.65%

鲲鹏 8块1.2T HDD raid5 (write through)

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=1455700: Mon Jun 26 14:28:00 2023
   read: IOPS=1972, BW=30.8MiB/s (32.3MB/s)(3700MiB/120072msec)
    clat (usec): min=82, max=282886, avg=10274.05, stdev=11258.46
     lat (usec): min=82, max=282886, avg=10274.19, stdev=11258.46
    clat percentiles (usec):
     |  1.00th=[  1172],  5.00th=[  1745], 10.00th=[  2278], 20.00th=[  3326],
     | 30.00th=[  4359], 40.00th=[  5407], 50.00th=[  6390], 60.00th=[  8160],
     | 70.00th=[ 10814], 80.00th=[ 14615], 90.00th=[ 22414], 95.00th=[ 31327],
     | 99.00th=[ 56361], 99.50th=[ 67634], 99.90th=[101188], 99.95th=[114820],
     | 99.99th=[152044]
   bw (  KiB/s): min=  288, max= 1856, per=3.13%, avg=986.52, stdev=212.14, samples=7678
   iops        : min=   18, max=  116, avg=61.62, stdev=13.26, samples=7678
  write: IOPS=498, BW=7851KiB/s (8039kB/s)(926MiB/120845msec)
    clat (usec): min=9, max=31655k, avg=270620.34, stdev=2362223.42
     lat (usec): min=9, max=31655k, avg=270620.47, stdev=2362223.44
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    8], 10.00th=[   10], 20.00th=[   13],
     | 30.00th=[   15], 40.00th=[   18], 50.00th=[   20], 60.00th=[   24],
     | 70.00th=[   28], 80.00th=[   33], 90.00th=[   45], 95.00th=[   58],
     | 99.00th=[12818], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    0, max=  512, per=2.81%, avg=220.72, stdev=102.73, samples=8591
   iops        : min=    1, max=   32, avg=13.96, stdev= 6.00, samples=8591
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.07%, 250=0.14%
  lat (usec)   : 500=0.03%, 750=0.02%, 1000=0.14%
  lat (msec)   : 2=5.53%, 4=15.37%, 10=34.20%, 20=24.74%, 50=17.10%
  lat (msec)   : 100=2.20%, 250=0.17%, 500=0.01%, 750=0.01%, 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (msec): min=30, max=2175, avg=914.03, stdev=463.71
    sync percentiles (msec):
     |  1.00th=[   69],  5.00th=[  224], 10.00th=[  334], 20.00th=[  489],
     | 30.00th=[  592], 40.00th=[  735], 50.00th=[  885], 60.00th=[ 1036],
     | 70.00th=[ 1150], 80.00th=[ 1334], 90.00th=[ 1552], 95.00th=[ 1670],
     | 99.00th=[ 2022], 99.50th=[ 2072], 99.90th=[ 2165], 99.95th=[ 2165],
     | 99.99th=[ 2165]
  cpu          : usr=0.01%, sys=0.01%, ctx=303494, majf=0, minf=1193
  IO depths    : 1=100.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=236803,60298,0,524 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=30.8MiB/s (32.3MB/s), 30.8MiB/s-30.8MiB/s (32.3MB/s-32.3MB/s), io=3700MiB (3880MB), run=120072-120072msec
  WRITE: bw=7851KiB/s (8039kB/s), 7851KiB/s-7851KiB/s (8039kB/s-8039kB/s), io=926MiB (971MB), run=120845-120845msec

Disk stats (read/write):
    dm-0: ios=236809/60734, merge=0/0, ticks=2431830/1618260, in_queue=4050090, util=99.71%, aggrios=236809/60705, aggrmerge=0/55, aggrticks=2431516/1613287, aggrin_queue=2657140, aggrutil=99.69%
  sda: ios=236809/60705, merge=0/55, ticks=2431516/1613287, in_queue=2657140, util=99.69%

鲲鹏 8块1.2T HDD raid5 (write back+direct io + no disk cache)

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=1477293: Mon Jun 26 14:32:11 2023
   read: IOPS=2185, BW=34.1MiB/s (35.8MB/s)(4098MiB/120026msec)
    clat (usec): min=29, max=1423.1k, avg=14527.46, stdev=33170.85
     lat (usec): min=30, max=1423.1k, avg=14527.61, stdev=33170.85
    clat percentiles (usec):
     |  1.00th=[    46],  5.00th=[  1532], 10.00th=[  2147], 20.00th=[  3261],
     | 30.00th=[  4359], 40.00th=[  5473], 50.00th=[  6521], 60.00th=[  8717],
     | 70.00th=[ 11600], 80.00th=[ 16450], 90.00th=[ 27132], 95.00th=[ 43779],
     | 99.00th=[156238], 99.50th=[227541], 99.90th=[413139], 99.95th=[526386],
     | 99.99th=[784335]
   bw (  KiB/s): min=   31, max= 2656, per=3.15%, avg=1100.84, stdev=737.40, samples=7622
   iops        : min=    1, max=  166, avg=68.75, stdev=46.09, samples=7622
  write: IOPS=6976, BW=12.6MiB/s (13.2MB/s)(1507MiB/120032msec)
    clat (nsec): min=1850, max=1390.3M, avg=2069893.13, stdev=16192105.74
     lat (nsec): min=1970, max=1390.3M, avg=2070098.30, stdev=16192104.65
    clat percentiles (usec):
     |  1.00th=[     6],  5.00th=[    34], 10.00th=[    48], 20.00th=[   206],
     | 30.00th=[   404], 40.00th=[   627], 50.00th=[   848], 60.00th=[  1205],
     | 70.00th=[  1565], 80.00th=[  2089], 90.00th=[  2868], 95.00th=[  4359],
     | 99.00th=[ 11863], 99.50th=[ 30802], 99.90th=[219153], 99.95th=[295699],
     | 99.99th=[683672]
   bw (  KiB/s): min=    6, max=  992, per=0.64%, avg=82.70, stdev=136.87, samples=37264
   iops        : min=    1, max=  178, avg=44.82, stdev=19.86, samples=37264
  lat (usec)   : 2=0.01%, 4=0.28%, 10=2.00%, 20=0.68%, 50=5.14%
  lat (usec)   : 100=3.56%, 250=6.55%, 500=8.35%, 750=8.79%, 1000=7.06%
  lat (msec)   : 2=19.66%, 4=15.83%, 10=12.83%, 20=5.05%, 50=2.96%
  lat (msec)   : 100=0.65%, 250=0.45%, 500=0.12%, 750=0.03%, 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=1000, max=277945k, avg=198556.84, stdev=866895.51
    sync percentiles (usec):
     |  1.00th=[   62],  5.00th=[   85], 10.00th=[  109], 20.00th=[  126],
     | 30.00th=[  137], 40.00th=[  147], 50.00th=[  155], 60.00th=[  174],
     | 70.00th=[  208], 80.00th=[  241], 90.00th=[  302], 95.00th=[  371],
     | 99.00th=[  570], 99.50th=[ 1029], 99.90th=[ 2540], 99.95th=[ 2835],
     | 99.99th=[11207]
  cpu          : usr=0.03%, sys=0.25%, ctx=4292295, majf=0, minf=1002
  IO depths    : 1=126.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262275,837396,0,285399 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=34.1MiB/s (35.8MB/s), 34.1MiB/s-34.1MiB/s (35.8MB/s-35.8MB/s), io=4098MiB (4297MB), run=120026-120026msec
  WRITE: bw=12.6MiB/s (13.2MB/s), 12.6MiB/s-12.6MiB/s (13.2MB/s-13.2MB/s), io=1507MiB (1580MB), run=120032-120032msec

Disk stats (read/write):
    dm-0: ios=263736/757311, merge=0/0, ticks=3813920/70820, in_queue=3884740, util=99.76%, aggrios=264200/758124, aggrmerge=0/257, aggrticks=3820039/76217, aggrin_queue=2659180, aggrutil=99.74%
  sda: ios=264200/758124, merge=0/257, ticks=3820039/76217, in_queue=2659180, util=99.74%

鲲鹏 8块1.2T HDD raid5 (write back+cache io + no disk cache)

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=1501287: Mon Jun 26 14:36:49 2023
   read: IOPS=2191, BW=34.2MiB/s (35.9MB/s)(4110MiB/120041msec)
    clat (usec): min=28, max=1318.2k, avg=14510.05, stdev=34303.84
     lat (usec): min=28, max=1318.2k, avg=14510.20, stdev=34303.84
    clat percentiles (usec):
     |  1.00th=[    40],  5.00th=[  1500], 10.00th=[  2114], 20.00th=[  3228],
     | 30.00th=[  4359], 40.00th=[  5473], 50.00th=[  6521], 60.00th=[  8717],
     | 70.00th=[ 11600], 80.00th=[ 16319], 90.00th=[ 26608], 95.00th=[ 42730],
     | 99.00th=[158335], 99.50th=[235930], 99.90th=[446694], 99.95th=[549454],
     | 99.99th=[834667]
   bw (  KiB/s): min=   31, max= 2560, per=3.16%, avg=1106.45, stdev=741.17, samples=7604
   iops        : min=    1, max=  160, avg=69.11, stdev=46.33, samples=7604
  write: IOPS=6977, BW=12.6MiB/s (13.2MB/s)(1509MiB/120041msec)
    clat (nsec): min=1800, max=978028k, avg=2261211.40, stdev=11157179.78
     lat (nsec): min=1940, max=978028k, avg=2261416.15, stdev=11157179.26
    clat percentiles (usec):
     |  1.00th=[     8],  5.00th=[    36], 10.00th=[    60], 20.00th=[   334],
     | 30.00th=[   734], 40.00th=[  1074], 50.00th=[  1434], 60.00th=[  1893],
     | 70.00th=[  2442], 80.00th=[  2966], 90.00th=[  3458], 95.00th=[  4424],
     | 99.00th=[ 11076], 99.50th=[ 32637], 99.90th=[132645], 99.95th=[187696],
     | 99.99th=[408945]
   bw (  KiB/s): min=    7, max=  960, per=0.64%, avg=82.30, stdev=137.38, samples=37513
   iops        : min=    1, max=  114, avg=44.55, stdev=18.71, samples=37513
  lat (usec)   : 2=0.01%, 4=0.12%, 10=1.38%, 20=0.76%, 50=4.93%
  lat (usec)   : 100=2.84%, 250=3.82%, 500=5.01%, 750=4.95%, 1000=5.52%
  lat (msec)   : 2=20.11%, 4=27.97%, 10=13.34%, 20=5.09%, 50=2.98%
  lat (msec)   : 100=0.67%, 250=0.41%, 500=0.10%, 750=0.01%, 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=1040, max=159656k, avg=208481.78, stdev=666477.05
    sync percentiles (usec):
     |  1.00th=[   68],  5.00th=[   94], 10.00th=[  115], 20.00th=[  133],
     | 30.00th=[  143], 40.00th=[  149], 50.00th=[  159], 60.00th=[  176],
     | 70.00th=[  217], 80.00th=[  249], 90.00th=[  318], 95.00th=[  396],
     | 99.00th=[  627], 99.50th=[ 1205], 99.90th=[ 2704], 99.95th=[ 2900],
     | 99.99th=[17695]
  cpu          : usr=0.06%, sys=0.27%, ctx=4375889, majf=0, minf=1002
  IO depths    : 1=125.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=263033,837539,0,285399 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=34.2MiB/s (35.9MB/s), 34.2MiB/s-34.2MiB/s (35.9MB/s-35.9MB/s), io=4110MiB (4310MB), run=120041-120041msec
  WRITE: bw=12.6MiB/s (13.2MB/s), 12.6MiB/s-12.6MiB/s (13.2MB/s-13.2MB/s), io=1509MiB (1583MB), run=120041-120041msec

Disk stats (read/write):
    dm-0: ios=264337/765358, merge=0/0, ticks=3816800/65680, in_queue=3882480, util=99.85%, aggrios=264955/766243, aggrmerge=0/213, aggrticks=3822876/71015, aggrin_queue=2657900, aggrutil=99.85%
  sda: ios=264955/766243, merge=0/213, ticks=3822876/71015, in_queue=2657900, util=99.85%

鲲鹏 8块1.2T HDD raid5 (write back + cache io + enable disk cache + Read ahead)

innodb-trxlog: (groupid=0, jobs=160): err= 0: pid=1519318: Mon Jun 26 14:40:18 2023
   read: IOPS=2232, BW=34.9MiB/s (36.6MB/s)(4224MiB/121077msec)
    clat (usec): min=28, max=1658.1k, avg=14167.62, stdev=37311.51
     lat (usec): min=28, max=1658.1k, avg=14167.77, stdev=37311.51
    clat percentiles (usec):
     |  1.00th=[    40],  5.00th=[  1516], 10.00th=[  2147], 20.00th=[  3326],
     | 30.00th=[  4424], 40.00th=[  5538], 50.00th=[  6587], 60.00th=[  8455],
     | 70.00th=[ 11076], 80.00th=[ 15270], 90.00th=[ 24249], 95.00th=[ 38011],
     | 99.00th=[164627], 99.50th=[256902], 99.90th=[501220], 99.95th=[641729],
     | 99.99th=[943719]
   bw (  KiB/s): min=   31, max= 2624, per=3.20%, avg=1141.98, stdev=751.19, samples=7573
   iops        : min=    1, max=  164, avg=71.31, stdev=46.96, samples=7573
  write: IOPS=6932, BW=12.7MiB/s (13.3MB/s)(1538MiB/121077msec)
    clat (nsec): min=1520, max=1309.9M, avg=2552096.10, stdev=22269813.50
     lat (nsec): min=1640, max=1309.9M, avg=2552301.84, stdev=22269811.90
    clat percentiles (usec):
     |  1.00th=[      5],  5.00th=[     10], 10.00th=[     40],
     | 20.00th=[    117], 30.00th=[    269], 40.00th=[    545],
     | 50.00th=[    873], 60.00th=[   1237], 70.00th=[   1762],
     | 80.00th=[   2343], 90.00th=[   3228], 95.00th=[   4555],
     | 99.00th=[  22152], 99.50th=[  57934], 99.90th=[ 198181],
     | 99.95th=[ 526386], 99.99th=[1149240]
   bw (  KiB/s): min=    2, max= 1088, per=0.66%, avg=85.24, stdev=141.79, samples=36917
   iops        : min=    1, max=  196, avg=45.37, stdev=20.55, samples=36917
  lat (usec)   : 2=0.01%, 4=0.75%, 10=3.23%, 20=0.65%, 50=5.23%
  lat (usec)   : 100=5.25%, 250=7.31%, 500=7.35%, 750=5.84%, 1000=5.46%
  lat (msec)   : 2=16.57%, 4=18.20%, 10=14.78%, 20=5.26%, 50=2.81%
  lat (msec)   : 100=0.64%, 250=0.46%, 500=0.13%, 750=0.04%, 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=990, max=1305.8M, avg=193673.42, stdev=3519837.34
    sync percentiles (usec):
     |  1.00th=[   57],  5.00th=[   77], 10.00th=[   92], 20.00th=[  123],
     | 30.00th=[  135], 40.00th=[  143], 50.00th=[  151], 60.00th=[  161],
     | 70.00th=[  196], 80.00th=[  233], 90.00th=[  289], 95.00th=[  355],
     | 99.00th=[  482], 99.50th=[  562], 99.90th=[ 2606], 99.95th=[ 2900],
     | 99.99th=[ 5669]
  cpu          : usr=0.04%, sys=0.24%, ctx=4136917, majf=0, minf=1002
  IO depths    : 1=125.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=270336,839330,0,285325 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=4224MiB (4429MB), run=121077-121077msec
  WRITE: bw=12.7MiB/s (13.3MB/s), 12.7MiB/s-12.7MiB/s (13.3MB/s-13.3MB/s), io=1538MiB (1613MB), run=121077-121077msec

Disk stats (read/write):
    dm-0: ios=272258/775885, merge=0/0, ticks=3842570/60820, in_queue=3903390, util=99.21%, aggrios=272258/775661, aggrmerge=0/235, aggrticks=3844979/62942, aggrin_queue=2622190, aggrutil=99.02%
  sda: ios=272258/775661, merge=0/235, ticks=3844979/62942, in_queue=2622190, util=99.02%