ZFS下不同压缩方式的性能测试 (l z4/gzip/zstd)

原创文章，转载请注明出处
https://qiedd.com/

0. 前言

由于 VPS 的硬盘可能不太够，所以需要做一些文件系统上的压缩，因此在 ZFS 和 Btrfs 上取舍，最后选择了 ZFS
但是压缩方式上面让我犯了难，因此我们来测试一下一些比较优秀的压缩方式
https://github.com/openzfs/zfs

环境
腾讯云
系统： Arch Linux
CPU: Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz (2 Cores)
Memory: 4 GB

1. 一些命令

创建一个新的 mountpoint 来作为测试

# 创建mountpoint
zfs create -o mountpoint=/test zroot/data/test

# 查看压缩方式
zfs get compression

# zroot/data/test     compression  zstd            inherited from zroot

# 查看压缩率
zroot/data/test     compressratio  1.00x  -

# 安装iotop和fio
pacman -Sy iotop fio

# 写入测试
fio --name=seqwrite --rw=write --direct=1 --ioengine=psync --bs=128k --iodepth=32 --numjobs=10 --size=1G --group_reporting

# 读取测试
fio --name=seqread --rw=read --direct=1 --ioengine=psync --bs=128k --iodepth=32 --numjobs=10 --size=1G --group_reporting

# 查看进程io
iotop -o

# 生成一个随机的1G文件
dd if=/dev/urandom of=sample.txt bs=64M count=16

2. Lz4

写入

seqwrite: (groupid=0, jobs=10): err= 0: pid=3933293: Tue Jan 18 22:48:55 2022
  write: IOPS=1117, BW=140MiB/s (146MB/s)(10.0GiB/73305msec); 0 zone resets
    clat (usec): min=13, max=65950, avg=8856.39, stdev=1989.56
     lat (usec): min=13, max=65951, avg=8857.77, stdev=1989.60
    clat percentiles (usec):
     |  1.00th=[   19],  5.00th=[ 5604], 10.00th=[ 8291], 20.00th=[ 8979],
     | 30.00th=[ 9110], 40.00th=[ 9110], 50.00th=[ 9241], 60.00th=[ 9241],
     | 70.00th=[ 9241], 80.00th=[ 9372], 90.00th=[ 9634], 95.00th=[10028],
     | 99.00th=[11469], 99.50th=[13304], 99.90th=[25035], 99.95th=[29754],
     | 99.99th=[39584]
   bw (  KiB/s): min=129174, max=827864, per=100.00%, avg=144240.45, stdev=6509.22, samples=1445
   iops        : min= 1006, max= 6464, avg=1126.41, stdev=50.83, samples=1445
  lat (usec)   : 20=1.12%, 50=0.78%, 100=0.02%, 250=0.09%, 500=0.21%
  lat (usec)   : 750=0.23%, 1000=0.05%
  lat (msec)   : 2=0.33%, 4=1.07%, 10=90.76%, 20=5.18%, 50=0.17%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.07%, sys=0.61%, ctx=82572, majf=8, minf=138
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,81920,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=140MiB/s (146MB/s), 140MiB/s-140MiB/s (146MB/s-146MB/s), io=10.0GiB (10.7GB), run=73305-73305msec

读取

seqread: (groupid=0, jobs=10): err= 0: pid=3941480: Tue Jan 18 22:54:13 2022
  read: IOPS=1222, BW=153MiB/s (160MB/s)(10.0GiB/67020msec)
    clat (usec): min=15, max=147529, avg=6833.55, stdev=18975.55
     lat (usec): min=15, max=147529, avg=6833.68, stdev=18975.54
    clat percentiles (usec):
     |  1.00th=[    35],  5.00th=[    46], 10.00th=[    56], 20.00th=[    63],
     | 30.00th=[    64], 40.00th=[    66], 50.00th=[    71], 60.00th=[    77],
     | 70.00th=[    83], 80.00th=[   103], 90.00th=[ 39584], 95.00th=[ 62129],
     | 99.00th=[ 74974], 99.50th=[ 78119], 99.90th=[ 84411], 99.95th=[ 88605],
     | 99.99th=[119014]
   bw (  MiB/s): min= 1863, max= 2270, per=100.00%, avg=1982.82, stdev= 9.51, samples=1111
   iops        : min=14905, max=18163, avg=15861.64, stdev=76.03, samples=1111
  lat (usec)   : 20=0.30%, 50=7.63%, 100=71.39%, 250=5.27%, 500=0.18%
  lat (usec)   : 750=0.10%, 1000=0.12%
  lat (msec)   : 2=0.37%, 4=0.49%, 10=2.79%, 20=0.40%, 50=2.29%
  lat (msec)   : 100=8.65%, 250=0.02%
  cpu          : usr=0.03%, sys=1.16%, ctx=13494, majf=0, minf=456
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=10.0GiB (10.7GB), run=67020-67020msec

3. Gzip

写入

seqwrite: (groupid=0, jobs=10): err= 0: pid=3943106: Tue Jan 18 23:02:32 2022
  write: IOPS=189, BW=23.7MiB/s (24.8MB/s)(10.0GiB/432137msec); 0 zone resets
    clat (usec): min=12, max=98520, avg=52338.47, stdev=11922.03
     lat (usec): min=13, max=98522, avg=52340.13, stdev=11922.07
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[26346], 10.00th=[44827], 20.00th=[52167],
     | 30.00th=[53216], 40.00th=[53740], 50.00th=[54264], 60.00th=[55313],
     | 70.00th=[57410], 80.00th=[58459], 90.00th=[60031], 95.00th=[62129],
     | 99.00th=[65799], 99.50th=[67634], 99.90th=[72877], 99.95th=[74974],
     | 99.99th=[86508]
   bw (  KiB/s): min=17912, max=729856, per=100.00%, avg=24460.45, stdev=2562.57, samples=8565
   iops        : min=  138, max= 5702, avg=190.90, stdev=20.02, samples=8565
  lat (usec)   : 20=0.69%, 50=1.27%, 100=0.02%, 250=0.02%, 500=0.10%
  lat (usec)   : 750=0.18%, 1000=0.06%
  lat (msec)   : 2=0.23%, 4=0.34%, 10=0.43%, 20=0.62%, 50=9.44%
  lat (msec)   : 100=86.58%
  cpu          : usr=0.02%, sys=0.13%, ctx=83526, majf=0, minf=126
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,81920,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=23.7MiB/s (24.8MB/s), 23.7MiB/s-23.7MiB/s (24.8MB/s-24.8MB/s), io=10.0GiB (10.7GB), run=432137-432137msec

读取

seqread: (groupid=0, jobs=10): err= 0: pid=3953413: Tue Jan 18 23:14:01 2022
  read: IOPS=1140, BW=143MiB/s (149MB/s)(10.0GiB/71840msec)
    clat (usec): min=28, max=139673, avg=8034.70, stdev=19823.39
     lat (usec): min=28, max=139673, avg=8034.87, stdev=19823.38
    clat percentiles (usec):
     |  1.00th=[    40],  5.00th=[    53], 10.00th=[    62], 20.00th=[    63],
     | 30.00th=[    66], 40.00th=[    74], 50.00th=[    78], 60.00th=[    82],
     | 70.00th=[    91], 80.00th=[   137], 90.00th=[ 46924], 95.00th=[ 62129],
     | 99.00th=[ 76022], 99.50th=[ 79168], 99.90th=[ 86508], 99.95th=[ 88605],
     | 99.99th=[107480]
   bw (  KiB/s): min=30095, max=1108520, per=100.00%, avg=158744.43, stdev=18988.74, samples=1310
   iops        : min=  234, max= 8657, avg=1239.78, stdev=148.32, samples=1310
  lat (usec)   : 50=4.30%, 100=70.70%, 250=5.73%, 500=0.36%, 750=0.20%
  lat (usec)   : 1000=0.21%
  lat (msec)   : 2=0.52%, 4=0.62%, 10=2.78%, 20=0.88%, 50=4.53%
  lat (msec)   : 100=9.16%, 250=0.02%
  cpu          : usr=0.04%, sys=1.11%, ctx=17249, majf=0, minf=439
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=143MiB/s (149MB/s), 143MiB/s-143MiB/s (149MB/s-149MB/s), io=10.0GiB (10.7GB), run=71840-71840msec

4. Zstd

写入

seqwrite: (groupid=0, jobs=10): err= 0: pid=3955112: Tue Jan 18 23:17:19 2022
  write: IOPS=556, BW=69.6MiB/s (73.0MB/s)(10.0GiB/147117msec); 0 zone resets
    clat (usec): min=13, max=54822, avg=17787.35, stdev=3391.08
     lat (usec): min=13, max=54824, avg=17789.17, stdev=3391.04
    clat percentiles (usec):
     |  1.00th=[   21],  5.00th=[15401], 10.00th=[16712], 20.00th=[17695],
     | 30.00th=[17957], 40.00th=[18220], 50.00th=[18220], 60.00th=[18482],
     | 70.00th=[18744], 80.00th=[19006], 90.00th=[19792], 95.00th=[20317],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[24249], 99.95th=[25560],
     | 99.99th=[30016]
   bw (  KiB/s): min=56576, max=726212, per=100.00%, avg=71873.44, stdev=4721.09, samples=2906
   iops        : min=  442, max= 5672, avg=561.34, stdev=36.88, samples=2906
  lat (usec)   : 20=0.98%, 50=0.93%, 100=0.01%, 250=0.09%, 500=0.15%
  lat (usec)   : 750=0.15%, 1000=0.06%
  lat (msec)   : 2=0.15%, 4=0.37%, 10=0.96%, 20=88.67%, 50=7.47%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.06%, sys=0.41%, ctx=84403, majf=0, minf=119
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,81920,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=69.6MiB/s (73.0MB/s), 69.6MiB/s-69.6MiB/s (73.0MB/s-73.0MB/s), io=10.0GiB (10.7GB), run=147117-147117msec

读取

seqread: (groupid=0, jobs=10): err= 0: pid=3964126: Tue Jan 18 23:21:49 2022
  read: IOPS=1164, BW=146MiB/s (153MB/s)(10.0GiB/70371msec)
    clat (usec): min=27, max=151967, avg=7762.81, stdev=20941.61
     lat (usec): min=27, max=151967, avg=7762.94, stdev=20941.61
    clat percentiles (usec):
     |  1.00th=[    38],  5.00th=[    48], 10.00th=[    61], 20.00th=[    63],
     | 30.00th=[    64], 40.00th=[    65], 50.00th=[    71], 60.00th=[    77],
     | 70.00th=[    84], 80.00th=[   104], 90.00th=[ 50070], 95.00th=[ 67634],
     | 99.00th=[ 80217], 99.50th=[ 83362], 99.90th=[ 89654], 99.95th=[ 94897],
     | 99.99th=[124257]
   bw (  KiB/s): min=20996, max=1533480, per=100.00%, avg=169112.61, stdev=31295.51, samples=1263
   iops        : min=  164, max=11976, avg=1320.63, stdev=244.47, samples=1263
  lat (usec)   : 50=5.66%, 100=73.24%, 250=5.37%, 500=0.12%, 750=0.08%
  lat (usec)   : 1000=0.07%
  lat (msec)   : 2=0.25%, 4=0.39%, 10=2.83%, 20=0.36%, 50=1.57%
  lat (msec)   : 100=10.00%, 250=0.04%
  cpu          : usr=0.04%, sys=1.04%, ctx=13941, majf=0, minf=460
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=10.0GiB (10.7GB), run=70371-70371msec

4. 压缩率

测试文件：PostgreSQL 1G 数据文件

# lz4
zroot/data/test     compressratio  2.54x  -

# gzip
zroot/data/test     compressratio  3.33x  -

# zstd
zroot/data/test     compressratio  3.51x  -

5. 结论

如果对性能有要求，建议在 lz4 和 zstd 中做取舍
zstd 在高压缩率的情况下，写入性能也比 gzip 高，确实不错👍