Flashcache¶

До fashcache

monkey ~ # fio -readonly -name iops -rw=randread -bs=4096 -runtime=20 -iodepth 1 -filename /dev/vda -ioengine libaio -direct=1
iops: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 2.0.7
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [482K/0K /s] [117 /0  iops] [eta 00m:00s]
iops: (groupid=0, jobs=1): err= 0: pid=2941
  read : io=8892.0KB, bw=455156 B/s, iops=111 , runt= 20005msec
    slat (usec): min=8 , max=312 , avg=35.66, stdev=21.26
    clat (usec): min=487 , max=68000 , avg=8957.66, stdev=4331.27
     lat (usec): min=528 , max=68015 , avg=8995.32, stdev=4332.05
    clat percentiles (usec):
     |  1.00th=[ 3440],  5.00th=[ 4512], 10.00th=[ 5024], 20.00th=[ 5920],
     | 30.00th=[ 6816], 40.00th=[ 7776], 50.00th=[ 8640], 60.00th=[ 9536],
     | 70.00th=[10432], 80.00th=[11200], 90.00th=[12096], 95.00th=[12864],
     | 99.00th=[27264], 99.50th=[36608], 99.90th=[52992], 99.95th=[60160],
     | 99.99th=[68096]
    bw (KB/s)  : min=  285, max=  503, per=99.99%, avg=443.95, stdev=49.29
    lat (usec) : 500=0.04%
    lat (msec) : 2=0.18%, 4=1.62%, 10=64.37%, 20=32.07%, 50=1.57%
    lat (msec) : 100=0.13%
  cpu          : usr=0.11%, sys=0.61%, ctx=2230, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2223/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8892KB, aggrb=444KB/s, minb=444KB/s, maxb=444KB/s, mint=20005msec, maxt=20005msec

Disk stats (read/write):
  vda: ios=2197/2, merge=0/7, ticks=19630/103, in_queue=19728, util=98.76%

Холодный кэш

monkey ~ # fio -readonly -name iops -rw=randread -bs=4096 -runtime=20 -iodepth 1 -filename /dev/vda -ioengine libaio -direct=1
iops: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 2.0.7
Starting 1 process
Jobs: 1 (f=1): [r] [95.2% done] [452K/0K /s] [110 /0  iops] [eta 00m:01s]
iops: (groupid=0, jobs=1): err= 0: pid=1497
  read : io=8436.0KB, bw=431901 B/s, iops=105 , runt= 20001msec
    slat (usec): min=8 , max=70032 , avg=59.42, stdev=1524.52
    clat (usec): min=225 , max=133986 , avg=9419.83, stdev=5905.45
     lat (usec): min=289 , max=134035 , avg=9480.52, stdev=6126.76
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   11], 80.00th=[   12], 90.00th=[   13], 95.00th=[   14],
     | 99.00th=[   28], 99.50th=[   43], 99.90th=[   64], 99.95th=[  126],
     | 99.99th=[  135]
    bw (KB/s)  : min=  246, max=  502, per=100.00%, avg=421.10, stdev=63.38
    lat (usec) : 250=0.09%, 500=0.33%, 750=0.05%
    lat (msec) : 2=0.05%, 4=1.38%, 10=61.21%, 20=34.90%, 50=1.71%
    lat (msec) : 100=0.19%, 250=0.09%
  cpu          : usr=0.07%, sys=0.76%, ctx=2116, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2109/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8436KB, aggrb=421KB/s, minb=421KB/s, maxb=421KB/s, mint=20001msec, maxt=20001msec

Disk stats (read/write):
  vda: ios=2139/2, merge=1/4, ticks=20365/732, in_queue=21067, util=98.67%

Горячий кэш

monkey ~ # fio -readonly -name iops -rw=randread -bs=4096 -runtime=20 -iodepth 1 -filename /dev/vda -ioengine libaio -direct=1
iops: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 2.0.7
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [426K/0K /s] [104 /0  iops] [eta 00m:00s]
iops: (groupid=0, jobs=1): err= 0: pid=1502
  read : io=16912KB, bw=865634 B/s, iops=211 , runt= 20006msec
    slat (usec): min=7 , max=163 , avg=19.44, stdev=15.56
    clat (usec): min=172 , max=105321 , avg=4708.36, stdev=5534.23
     lat (usec): min=206 , max=105333 , avg=4728.79, stdev=5540.49
    clat percentiles (usec):
     |  1.00th=[  203],  5.00th=[  211], 10.00th=[  221], 20.00th=[  231],
     | 30.00th=[  241], 40.00th=[  258], 50.00th=[ 2384], 60.00th=[ 6176],
     | 70.00th=[ 7968], 80.00th=[ 9664], 90.00th=[11328], 95.00th=[12352],
     | 99.00th=[18816], 99.50th=[25216], 99.90th=[51968], 99.95th=[57088],
     | 99.99th=[104960]
    bw (KB/s)  : min=  302, max=13928, per=100.00%, avg=854.87, stdev=2191.98
    lat (usec) : 250=34.08%, 500=15.80%
    lat (msec) : 2=0.05%, 4=0.76%, 10=31.86%, 20=16.60%, 50=0.73%
    lat (msec) : 100=0.09%, 250=0.02%
  cpu          : usr=0.18%, sys=0.66%, ctx=4238, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=4228/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=16912KB, aggrb=845KB/s, minb=845KB/s, maxb=845KB/s, mint=20006msec, maxt=20006msec

Disk stats (read/write):
  vda: ios=4205/2, merge=0/7, ticks=19509/68, in_queue=19577, util=98.09%

Сильно разогретый кэш

monkey ~ # fio -readonly -name iops -rw=randread -bs=4096 -runtime=20 -iodepth 1 -filename /dev/vda -ioengine libaio -direct=1
iops: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio 2.0.7
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [469K/0K /s] [114 /0  iops] [eta 00m:00s]  
iops: (groupid=0, jobs=1): err= 0: pid=1543
  read : io=45572KB, bw=2278.5KB/s, iops=569 , runt= 20001msec
    slat (usec): min=8 , max=140 , avg=15.68, stdev=14.67
    clat (usec): min=134 , max=74829 , avg=1736.53, stdev=4006.16
     lat (usec): min=209 , max=74886 , avg=1752.89, stdev=4010.50
    clat percentiles (usec):
     |  1.00th=[  179],  5.00th=[  215], 10.00th=[  223], 20.00th=[  233],
     | 30.00th=[  237], 40.00th=[  241], 50.00th=[  247], 60.00th=[  251],
     | 70.00th=[  258], 80.00th=[  298], 90.00th=[ 8096], 95.00th=[10304],
     | 99.00th=[12864], 99.50th=[16320], 99.90th=[36608], 99.95th=[54016],
     | 99.99th=[72192]
    bw (KB/s)  : min=  328, max=15416, per=100.00%, avg=2324.00, stdev=4584.97
    lat (usec) : 250=56.80%, 500=26.99%
    lat (msec) : 2=0.03%, 4=0.25%, 10=10.30%, 20=5.27%, 50=0.31%
    lat (msec) : 100=0.05%
  cpu          : usr=0.33%, sys=1.20%, ctx=11412, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=11393/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=45572KB, aggrb=2278KB/s, minb=2278KB/s, maxb=2278KB/s, mint=20001msec, maxt=20001msec

Disk stats (read/write):
  vda: ios=11388/2, merge=0/4, ticks=19568/315, in_queue=19882, util=97.67%

Статистика:

kvm320-monkey-a-cache: 0 41943040 flashcache stats: 
    reads(66944), writes(843)
    read hits(35892), read hit percent(53)
    replacement(0), write replacement(0)
    invalidates(2554)
    pending enqueues(0), pending inval(0)
    no room(0)
    disk reads(31053), disk writes(854) ssd reads(35892) ssd writes(23865)
    uncached reads(7188), uncached writes(854), uncached IO requeue(0)
    uncached sequential reads(0), uncached sequential writes(0)
    pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)