Tune-Up Your Storage: Why Readahead Matters for High-Performance iSCSI

I spent way too long figuring out why our iSCSI storage was performing terribly. Turns out the default Linux readahead settings are tuned for local disks, and they absolutely murder performance on network storage. After fixing this, our sequential read throughput went from 38 MB/s to 105 MB/s. Same hardware, same network, just one sysctl value.

So here’s what I learned.

What is Readahead Anyway?

When you read data sequentially, Linux tries to be smart. If you’re reading block 100, it figures you’ll probably want blocks 101, 102, 103… so it prefetches them. That way when you ask for the next chunk, it’s already in memory.

Normal Read (no readahead):
Application          Kernel           iSCSI Storage
    |                  |                    |
    |--Request 1MB---->|                    |
    |                  |----Request 1MB---->|
    |                  |                    |
    |                  |<---Send 1MB--------|
    |<--Return 1MB-----|                    |
    |                  |                    |
    |--Request 1MB---->|                    |
    |                  |----Request 1MB---->|
    |    (waiting...)  |    (3-5ms RTT)    |

With readahead, it looks like this:

With Readahead (8MB):
Application          Kernel           iSCSI Storage
    |                  |                    |
    |--Request 1MB---->|                    |
    |                  |----Request 8MB---->|
    |                  |                    |
    |                  |<---Send 8MB--------|
    |<--Return 1MB-----|  (cache: 7MB)     |
    |                  |                    |
    |--Request 1MB---->|                    |
    |<--Return 1MB-----|  (from cache!)    |
    |   (instant!)     |  (cache: 6MB)     |
    |                  |                    |
    |--Request 1MB---->|                    |
    |<--Return 1MB-----|  (from cache!)    |
    |   (instant!)     |  (cache: 5MB)     |

This works great for local SSDs where latency is under 0.1ms. But with iSCSI over a network? Each small read has to go over the wire and back. That’s 2–5ms per request, easy. Readahead can hide all that latency by fetching bigger chunks.

The Problem with Default Readahead setting:

Check your current setting:

cat /sys/block/sdb/queue/read_ahead_kb

You’ll probably see 128 KB. That’s fine for a local disk but way too small for iSCSI.

Here’s why it kills performance:

App reads 1MB with 128KB readahead:

Request 1: [128KB] --network--> (3ms)
Request 2: [128KB] --network--> (3ms)
Request 3: [128KB] --network--> (3ms)
Request 4: [128KB] --network--> (3ms)
Request 5: [128KB] --network--> (3ms)
Request 6: [128KB] --network--> (3ms)
Request 7: [128KB] --network--> (3ms)
Request 8: [128KB] --network--> (3ms)
                        Total: ~24ms just waiting

App reads 1MB with 8MB readahead:

Request 1: [8MB includes your 1MB] --network--> (3ms)
Next 7 reads: served from cache (0ms each)
                        Total: ~3ms

If your app reads 1MB at a time, and readahead is only 128KB, you need 8 separate network requests to get that 1MB (1024 KB ÷ 128 KB = 8). At 3ms per request, that’s 24ms just sitting there waiting. Your app could have read the whole thing in one shot.

Let Me Show You The Difference in Performance with and without Readahead tuning:

I set up a simple test with a 1GB file on an iSCSI volume. Network is plain 1Gbps ethernet, nothing fancy.

Test 1: Default Settings (128KB readahead)

# Make sure we're at defaults
cat /sys/block/sdb/queue/read_ahead_kb
# shows: 128

# Drop cache so we're testing actual disk reads
sync && echo 3 > /proc/sys/vm/drop_caches

# Read the file
dd if=/mnt/iscsi/testfile.dat of=/dev/null bs=1M count=1024

Results:

1073741824 bytes copied, 28.4 seconds, 37.8 MB/s

While that’s running, check iostat:

iostat -x 1 sdb

Device    r/s    rMB/s  rrqm/s  %util  await
sdb      295.0   37.8    0.0    98.2   12.4

See that? 295 requests per second. Each one is a network round-trip. Your disk is “busy” (98% util) but you’re only getting 37 MB/s on a gigabit link that should do 100+ MB/s.

The problem is obvious when you think about it: you’re making 295 tiny requests per second instead of a few big ones.

Test 2: Tuned Settings (8MB readahead)

Now let’s fix it:

echo 8192 > /sys/block/sdb/queue/read_ahead_kb

# Clear cache again
sync && echo 3 > /proc/sys/vm/drop_caches

# Same test
dd if=/mnt/iscsi/testfile.dat of=/dev/null bs=1M count=1024

Results:

1073741824 bytes copied, 10.2 seconds, 105.3 MB/s

Check iostat now:

Device    r/s    rMB/s  rrqm/s  %util  await
sdb       13.2   105.3   52.0   95.1    4.2

Holy crap. Look at those numbers:

• 13 requests per second instead of 295
• 105 MB/s instead of 38 MB/s
• 4.2ms latency instead of 12.4ms

That’s a 2.8x speedup from changing one number. The kernel is now fetching 8MB at a time, so most of your reads come straight from cache.

Lets visualize the comparision:

┌─────────────────────────────────────────────────────────────┐
│                    Default (128KB)                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  App reads → [req][req][req][req][req][req][req][req]       │
│              ↓    ↓    ↓    ↓    ↓    ↓    ↓    ↓           │
│  Network → [========================================]       │
│              295 requests/second                            │
│                                                             │
│  Result: 37.8 MB/s, 12.4ms latency                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                    Tuned (8MB)                              │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  App reads → [req]                                          │
│              ↓                                              │
│  Network → [█████████████]                                  │
│              13 requests/second                             │
│              ↓                                              │
│  Cache → [████████████████████] (feeds next 7 reads)        │
│                                                             │
│  Result: 105.3 MB/s, 4.2ms latency                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Metric Default (128KB) Tuned (8MB) Change Throughput 37.8 MB/s 105.3 MB/s 2.8x faster IOPS 295 13.2 95% fewer requests Latency 12.4 ms 4.2 ms 66% lower

How to Find the Right Value ?

I tested a bunch of different readahead sizes. Here’s what I got:

Performance vs Readahead Size:

120 MB/s ┤
         ┤                        ╭───────
110 MB/s ┤                   ╭────╯
         ┤              ╭────╯
100 MB/s ┤         ╭────╯
         ┤    ╭────╯
 90 MB/s ┤╭───╯
         ┤
 80 MB/s ┤
         ┤
 70 MB/s ┤
         ┤
 60 MB/s ┤
         ┤
 50 MB/s ┤
         ┤
 40 MB/s ┼─╮
         └─┴────┴────┴────┴────┴────┴────┴────
          128  512  1M   2M   4M   8M  16M  32M
           KB   KB

Actual numbers:

128 KB   →  37.8 MB/s  (default, terrible)
256 KB   →  52.3 MB/s
512 KB   →  68.1 MB/s
1 MB     →  79.4 MB/s
2 MB     →  91.2 MB/s
4 MB     →  98.7 MB/s
8 MB     → 105.3 MB/s  (sweet spot)
16 MB    → 106.1 MB/s  (barely better)
32 MB    → 105.8 MB/s  (no real gain)

After 8MB you get diminishing returns. Makes sense — at some point you’re prefetching data you won’t actually use, which just wastes memory.

For my setup, 8MB was the sweet spot. Your mileage may vary depending on your network and workload.

Quick Script to Test Your System

I wrote this to test different values:

#!/bin/bash

DEVICE=/dev/sdb
TEST_FILE=/mnt/iscsi/testfile.dat

for RA in 128 512 1024 2048 4096 8192 16384; do
    echo "Testing ${RA}KB readahead..."

    echo $RA > /sys/block/sdb/queue/read_ahead_kb
    sync && echo 3 > /proc/sys/vm/drop_caches
    sleep 2

    dd if=${TEST_FILE} of=/dev/null bs=1M 2>&1 | grep copied
    echo "---"
done

Run it and see what works best for you.

Making it Stick

These changes don’t survive a reboot, so you need to make them permanent.

Option 1: udev rule (my favorite)

Create /etc/udev/rules.d/60-iscsi-readahead.rules:

ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_BUS}=="scsi", \
  ENV{ID_SCSI_ISCSI}=="1", \
  ATTR{queue/read_ahead_kb}="8192"

Then reload:

udevadm control --reload-rules
udevadm trigger --subsystem-match=block

This automatically sets readahead whenever an iSCSI device appears.

Option 2: systemd service

Create /etc/systemd/system/iscsi-readahead.service:

[Unit]
Description=Set readahead for iSCSI devices
After=iscsi.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo 8192 > /sys/block/sdb/queue/read_ahead_kb'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable it:

systemctl daemon-reload
systemctl enable iscsi-readahead.service

Option 3: Good old rc.local

If you’re old school like me, just add this to /etc/rc.local:

echo 8192 > /sys/block/sdb/queue/read_ahead_kb

Simple and it works.

When This Helps (and When It Doesn’t) ?

This tuning is a huge win for:

• Sequential reads (database scans, large file transfers, video streaming)
• Any workload doing mostly forward reads
• Workloads where you read most of what you fetch

Good for Sequential:
┌──────────────────────────────────────┐
│ [Block 1][Block 2][Block 3][Block 4] │  ← Reading in order
│    ↓        ↓        ↓        ↓      │
│  Readahead prefetches blocks 2-4     │
│  when you read block 1               │
└──────────────────────────────────────┘
Hit rate: ~90%+ (most prefetched data gets used)

This won’t help much for:

• Random reads (OLTP databases doing index lookups)
• Applications that seek around a lot
• Situations where you prefetch data you never use

Bad for Random Access:
┌──────────────────────────────────────┐
│ [Block 1][Block 2][Block 3][Block 4] │
│    ↑                 ↑           ↑   │  ← Jumping around
│    Read Block 1      │           │   │
│         Skip to Block 3          │   │
│              Skip to Block 4         │
│                                      │
│  Prefetched blocks 2 wasted          │
└──────────────────────────────────────┘
Hit rate: ~20% (most prefetched data wasted)

For random I/O, you might actually want to reduce readahead to save memory:

echo 1024 > /sys/block/sdb/queue/read_ahead_kb  # 1MB for random I/O

Some Rough Guidelines

Based on my experience:

For sequential workloads (file servers, backups, video):

echo 8192 > /sys/block/sdb/queue/read_ahead_kb   # 8MB

For mixed workloads:

echo 4096 > /sys/block/sdb/queue/read_ahead_kb   # 4MB

For mostly random I/O (databases):

echo 2048 > /sys/block/sdb/queue/read_ahead_kb   # 2MB

Start with these and adjust based on what you see.

Quick Math

If you want to calculate a starting point:

Readahead ≈ Bandwidth × RTT × 2

Example:
- Network: 100 MB/s
- Round-trip time: 5ms
- Readahead ≈ 100 × 0.005 × 2 = 1 MB minimum

Then multiply by 4-8x to keep the pipeline full.
So aim for 4-8 MB.

Visual representation of why:

Network Pipeline (1MB readahead - too small):

Time →  0ms     5ms     10ms    15ms    20ms
       ┌──┐   ┌──┐   ┌──┐   ┌──┐   ┌──┐
       │RQ│   │RQ│   │RQ│   │RQ│   │RQ│
       └──┘   └──┘   └──┘   └──┘   └──┘
         ↓      ↓      ↓      ↓      ↓
       [wait] [wait] [wait] [wait] [wait]  ← App waiting

Pipeline is never full, lots of idle time

Network Pipeline (8MB readahead - optimal):

Time →  0ms     5ms     10ms    15ms    20ms
       ┌──────────────────────────────┐
       │   Large Request (8MB)        │
       └──────────────────────────────┘
         ↓
       [DATA][DATA][DATA][DATA][DATA][DATA]  ← App reading

Pipeline stays full, minimal waiting

This is just a ballpark. Always test your actual workload.

Checking if It’s Working

Watch your I/O while running:

iostat -x 1 sdb

Look for:

• Lower r/s (requests per second)
• Higher rMB/s (throughput)
• Lower await (latency)

If those aren’t improving, either your workload isn’t sequential or something else is the bottleneck.

Gotchas

Too much readahead can hurt: If you set it to 64MB and you’re doing random I/O, you’ll just waste memory prefetching data you never read. Start reasonable and work up.

Memory pressure: On systems with limited RAM, large readahead values can evict other useful cached data. Watch free -h and tune accordingly.

Network matters: If your iSCSI network is slow or unreliable, no amount of readahead will save you. Fix the network first.

Application-Level Hints

Your application can also give hints to the kernel. In C:

#include <fcntl.h>

int fd = open("/mnt/iscsi/bigfile", O_RDONLY);
posix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL);

Or in Python:

import os

fd = os.open('/mnt/iscsi/bigfile', os.O_RDONLY)
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)

This tells the kernel “hey, I’m going to read this sequentially” and it’ll be more aggressive with readahead.

Final Thoughts

The default 128KB readahead is fine for local disks but completely wrong for network storage. Bump it to 8MB for iSCSI and watch your performance double or triple.

Test it yourself:

# Before
cat /sys/block/sdb/queue/read_ahead_kb
dd if=/your/test/file of=/dev/null bs=1M

# After
echo 8192 > /sys/block/sdb/queue/read_ahead_kb
dd if=/your/test/file of=/dev/null bs=1M

The difference is usually dramatic.

One number, huge impact. That’s the kind of tuning I like.

If you’re serious about passing technical interviews, try PracHub. It helped me structure my preparation with advanced mock sessions. [**Check it out here**] and start practicing. (Disclosure: This is an affiliate link).