Understanding the EBS IO1 configuration the right way.

Author：Lau Dai

Date：2026/05/19

Currently, benchmarking the EBS IO1 volume type in the following configuration

Host: c7g.16xlarge (No instance-side volume throttle issue)

|+----------------------------------+------------------+|
|||                 EbsOptimizedInfo                  |||
||+-------------------------------------+-------------+||
|||  BaselineBandwidthInMbps            |  20000      |||
|||  BaselineIops                       |  80000      |||
|||  BaselineThroughputInMBps           |  2500.0     |||
|||  MaximumBandwidthInMbps             |  20000      |||
|||  MaximumIops                        |  80000      |||
|||  MaximumThroughputInMBps            |  2500.0     |||
||+-------------------------------------+-------------+||

Two IO1 volumes configured with 2,000 and 48,000 IOPS, respectively.

fio version: fio-3.32

FIO configuration file

tail -v -n 99 io1-io-testing.cfg
==> io1-io-testing.cfg <==

[global]
clocksource=clock_gettime
randrepeat=0
runtime=600
time_based=1
direct=1
group_reporting=1
norandommap=1
size=1G

[phase1-randwrite-16k]
description=Phase 1: 16KiB Random Write on 2000 IOPS
filename=/dev/nvme1n1
ioengine=psync
rw=randwrite
bs=16k
numjobs=16
do_verify=0

[phase2-randwrite-256k]
stonewall
description=Phase 2: 256KiB Random Write on 2000 IOPS
filename=/dev/nvme1n1
ioengine=psync
rw=randwrite
bs=256k
numjobs=16
do_verify=0

[phase3-randread-16k]
stonewall
description=Phase 3: 16KiB Random Read on 48000 IOPS
filename=/dev/nvme2n1
ioengine=psync
rw=randread
bs=16k
numjobs=4
do_verify=0

[phase4-randread-256k]
stonewall
description=Phase 4: 256KiB Random Read on 48000 IOPS
filename=/dev/nvme2n1
ioengine=psync
rw=randread
bs=256k
numjobs=4
do_verify=0

Encountered an issue where phase3 only reached 8,600 IO/s.

[ec2-user@ip-172-31-30-136 ~]$ date -Isec -u && sudo fio io1-io-testing.cfg
2026-05-19T14:07:10+00:00
phase1-randwrite-16k: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
phase2-randwrite-256k: (g=1): rw=randwrite, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1
...
phase3-randread-16k: (g=2): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
phase4-randread-256k: (g=3): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1
...
fio-3.32
Starting 40 processes
Jobs: 4 (f=4): [_(36),r(4)][100.0%][r=750MiB/s][r=3000 IOPS][eta 00m:00s]
phase1-randwrite-16k: (groupid=0, jobs=16): err= 0: pid=5449: Tue May 19 14:47:11 2026
  Description  : [Phase 1: 16KiB Random Write on 2000 IOPS]
  write: IOPS=2003, BW=31.3MiB/s (32.8MB/s)(18.3GiB/600009msec); 0 zone resets
    clat (usec): min=533, max=18348, avg=7985.87, stdev=1740.89
     lat (usec): min=533, max=18348, avg=7986.05, stdev=1740.89
    clat percentiles (usec):
     |  1.00th=[ 3458],  5.00th=[ 4015], 10.00th=[ 5080], 20.00th=[ 7898],
     | 30.00th=[ 7963], 40.00th=[ 7963], 50.00th=[ 8029], 60.00th=[ 8029],
     | 70.00th=[ 8094], 80.00th=[ 9372], 90.00th=[10028], 95.00th=[10552],
     | 99.00th=[12911], 99.50th=[13042], 99.90th=[13042], 99.95th=[13173],
     | 99.99th=[13304]
   bw (  KiB/s): min=28128, max=82545, per=99.98%, avg=32046.53, stdev=134.74, samples=19184
   iops        : min= 1758, max= 5153, avg=2002.61, stdev= 8.41, samples=19184
  lat (usec)   : 750=0.07%, 1000=0.12%
  lat (msec)   : 2=0.01%, 4=4.19%, 10=84.89%, 20=10.74%
  cpu          : usr=0.02%, sys=0.10%, ctx=1202035, majf=1, minf=154
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1202002,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
phase2-randwrite-256k: (groupid=1, jobs=16): err= 0: pid=5791: Tue May 19 14:47:11 2026
  Description  : [Phase 2: 256KiB Random Write on 2000 IOPS]
  write: IOPS=2000, BW=500MiB/s (525MB/s)(293GiB/600009msec); 0 zone resets
    clat (usec): min=792, max=18620, avg=7994.29, stdev=1221.30
     lat (usec): min=793, max=18621, avg=7995.90, stdev=1221.30
    clat percentiles (usec):
     |  1.00th=[ 5407],  5.00th=[ 5997], 10.00th=[ 6390], 20.00th=[ 7570],
     | 30.00th=[ 7898], 40.00th=[ 7963], 50.00th=[ 7963], 60.00th=[ 8029],
     | 70.00th=[ 8094], 80.00th=[ 8160], 90.00th=[ 8979], 95.00th=[10945],
     | 99.00th=[11469], 99.50th=[11600], 99.90th=[11994], 99.95th=[12125],
     | 99.99th=[13435]
   bw (  KiB/s): min=409600, max=740085, per=100.00%, avg=512284.06, stdev=2600.58, samples=19184
   iops        : min= 1600, max= 2882, avg=2000.71, stdev=10.15, samples=19184
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.04%, 4=0.15%, 10=91.37%, 20=8.43%
  cpu          : usr=0.04%, sys=0.13%, ctx=1200539, majf=0, minf=146
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1200528,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
phase3-randread-16k: (groupid=2, jobs=4): err= 0: pid=6195: Tue May 19 14:47:11 2026
  Description  : [Phase 3: 16KiB Random Read on 48000 IOPS]
  read: IOPS=8631, BW=135MiB/s (141MB/s)(79.0GiB/600001msec)
    clat (usec): min=147, max=19921, avg=463.04, stdev=48.64
     lat (usec): min=147, max=19921, avg=463.09, stdev=48.64
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  429], 10.00th=[  441], 20.00th=[  441],
     | 30.00th=[  441], 40.00th=[  445], 50.00th=[  445], 60.00th=[  449],
     | 70.00th=[  461], 80.00th=[  529], 90.00th=[  529], 95.00th=[  537],
     | 99.00th=[  545], 99.50th=[  553], 99.90th=[  619], 99.95th=[  701],
     | 99.99th=[ 1303]
   bw (  KiB/s): min=134656, max=140320, per=100.00%, avg=138131.36, stdev=155.02, samples=4796
   iops        : min= 8416, max= 8770, avg=8633.12, stdev= 9.69, samples=4796
  lat (usec)   : 250=0.01%, 500=76.87%, 750=23.09%, 1000=0.02%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.13%, sys=0.70%, ctx=5179128, majf=0, minf=54
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5179125,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
phase4-randread-256k: (groupid=3, jobs=4): err= 0: pid=6519: Tue May 19 14:47:11 2026
  Description  : [Phase 4: 256KiB Random Read on 48000 IOPS]
  read: IOPS=3004, BW=751MiB/s (788MB/s)(440GiB/600002msec)
    clat (usec): min=396, max=14135, avg=1330.75, stdev=82.12
     lat (usec): min=397, max=14135, avg=1330.81, stdev=82.12
    clat percentiles (usec):
     |  1.00th=[ 1139],  5.00th=[ 1221], 10.00th=[ 1237], 20.00th=[ 1270],
     | 30.00th=[ 1303], 40.00th=[ 1319], 50.00th=[ 1336], 60.00th=[ 1352],
     | 70.00th=[ 1369], 80.00th=[ 1385], 90.00th=[ 1418], 95.00th=[ 1450],
     | 99.00th=[ 1500], 99.50th=[ 1516], 99.90th=[ 1631], 99.95th=[ 1762],
     | 99.99th=[ 2966]
   bw (  KiB/s): min=754414, max=909312, per=100.00%, avg=769395.83, stdev=3319.61, samples=4796
   iops        : min= 2946, max= 3552, avg=3005.38, stdev=12.97, samples=4796
  lat (usec)   : 500=0.01%, 750=0.02%, 1000=0.08%
  lat (msec)   : 2=99.87%, 4=0.03%, 10=0.01%, 20=0.01%
  cpu          : usr=0.05%, sys=0.57%, ctx=1802984, majf=0, minf=293
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1802982,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=31.3MiB/s (32.8MB/s), 31.3MiB/s-31.3MiB/s (32.8MB/s-32.8MB/s), io=18.3GiB (19.7GB), run=600009-600009msec

Run status group 1 (all jobs):
  WRITE: bw=500MiB/s (525MB/s), 500MiB/s-500MiB/s (525MB/s-525MB/s), io=293GiB (315GB), run=600009-600009msec

Run status group 2 (all jobs):
   READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=79.0GiB (84.9GB), run=600001-600001msec

Run status group 3 (all jobs):
   READ: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=440GiB (473GB), run=600002-600002msec

Disk stats (read/write):
  nvme1n1: ios=141/2402198, merge=0/0, ticks=392/19174041, in_queue=19174433, util=99.99%
  nvme2n1: ios=6981517/0, merge=0/0, ticks=4760872/0, in_queue=4760872, util=100.00%

This does not match what the public documentation states regarding linear increase.

io1 volumes provisioned with more than 32,000 IOPS (up to the maximum of 64,000 IOPS) yield a linear increase in throughput at a rate of 16 KiB per provisioned IOPS. For example, a volume provisioned with 48,000 IOPS can support up to 750 MiB/s of throughput (16 KiB per provisioned IOPS × 48,000 provisioned IOPS = 750 MiB/s).

Ideally, phase 3 IOPS should be 48,000 instead of 8,600.

Deep dive into the root cause:

Let’s deep dive into the latency of the fio requests.

You can see that the Phase 3 latency is only 463 us. This is much lower than other phases, such as 7,985 us in phase 1, 2, and 4.

Phase 3
clat (usec): min=147, max=19921, avg=463.04, stdev=48.64
 lat (usec): min=147, max=19921, avg=463.09, stdev=48.64
---
Phase 1
clat (usec): min=533, max=18348, avg=7985.87, stdev=1740.89
 lat (usec): min=533, max=18348, avg=7986.05, stdev=1740.89

Thus, the IOPS per 4-job thread configuration could be:

1,000,000 us / 463 us = 2159.82721382 per thread

Then four threads would yield around 8,600 IOPS, which aligns with our observation.

The root cause of this issue is the incorrect engine configuration in fio.

 psync
   Basic pread(2) or pwrite(2) I/O. Default on all supported operating systems except for Windows.
 libaio
    Linux native asynchronous I/O. Note that Linux may only support queued behavior with non-buffered I/O (set direct=1 or buffered=0). This engine defines engine specific options.

source

The psync engine enforces iodepth=1 because it is a synchronous engine.

Once you modify the engine to libaio, which is an async engine, you can set a higher iodepth.

How to resolve this issue?

Modify the engine to an async engine such as libaio to send requests asynchronously.

After modifying the configuration, we can see that we achieve the maximum volume performance.

IO latency
avg=5241.51
IOPS
avg=48088.73

[ec2-user@ip-172-31-30-136 ~]$ date -Isec -u && sudo fio fix_8600IOPS-io1-testing.cfg
2026-05-19T15:06:47+00:00
phase3-randread-16k: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.32
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=750MiB/s][r=48.0k IOPS][eta 00m:00s]
phase3-randread-16k: (groupid=0, jobs=4): err= 0: pid=7428: Tue May 19 15:16:47 2026
  Description  : [Phase 3: 16KiB Random Read on 48000 IOPS]
  read: IOPS=48.1k, BW=751MiB/s (788MB/s)(440GiB/600002msec)
    slat (nsec): min=1234, max=3792.4k, avg=82328.73, stdev=246678.82
    clat (usec): min=501, max=28362, avg=5241.51, stdev=3304.00
     lat (usec): min=503, max=30044, avg=5323.84, stdev=3357.75
    clat percentiles (usec):
     |  1.00th=[ 1532],  5.00th=[ 1696], 10.00th=[ 1729], 20.00th=[ 1909],
     | 30.00th=[ 4555], 40.00th=[ 4686], 50.00th=[ 5211], 60.00th=[ 5276],
     | 70.00th=[ 5407], 80.00th=[ 5866], 90.00th=[ 6456], 95.00th=[14222],
     | 99.00th=[17695], 99.50th=[18482], 99.90th=[20317], 99.95th=[21103],
     | 99.99th=[21890]
   bw (  KiB/s): min=332576, max=1781184, per=100.00%, avg=769419.61, stdev=63897.95, samples=4796
   iops        : min=20786, max=111324, avg=48088.73, stdev=3993.62, samples=4796
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=20.67%, 4=3.57%, 10=68.17%, 20=7.41%, 50=0.17%
  cpu          : usr=0.81%, sys=3.06%, ctx=4121230, majf=0, minf=1062
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=28849585,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=440GiB (473GB), run=600002-600002msec

Disk stats (read/write):
  nvme2n1: ios=28844450/0, merge=0/0, ticks=58276760/0, in_queue=58276760, util=100.00%
[ec2-user@ip-172-31-30-136 ~]$ tail -v -n 99 fix_8600IOPS-io1-testing.cfg
==> fix_8600IOPS-io1-testing.cfg <==
[global]
clocksource=clock_gettime
randrepeat=0
runtime=600
time_based=1
direct=1
group_reporting=1
norandommap=1
size=1G

[phase3-randread-16k]
description=Phase 3: 16KiB Random Read on 48000 IOPS
filename=/dev/nvme2n1
ioengine=libaio
iodepth=64
rw=randread
bs=16k
numjobs=4
do_verify=0

Summary:

The system call you invoke will produce different results.

To understand how psync works, please refer to the man 2 [pwrite](https://linux.die.net/man/2/pwrite) manual page.