SIMD, Zero-Cost Abstractions, and the Hidden Price of Clean Code

This blog is mostly an exploration from the turbopuffer blog. Link in the end.

Imagine you’ve profiled a slow query. The algorithm is correct. The data structures are sound. The code is idiomatic and clean.

This is not a hypothetical. It’s a class of performance bug that hides specifically in well-written, well-abstracted code, and understanding it requires going below the language, below the compiler, all the way down to how the CPU actually executes instructions.

To get there, we need to talk about four ideas: branch prediction, SIMD, zero-cost abstractions, and batching.

The CPU Is Always Guessing

Modern processors don’t execute one instruction, wait for it to finish, then move to the next. They run a pipeline, fetching, decoding, and executing many instructions simultaneously, several stages ahead of where they currently are. This is how a 4 GHz CPU can appear to do far more than 4 billion simple operations per second.

The problem arises with if statements. When the CPU hits a branch, it doesn't know which path to take yet. So it guesses, picks the more likely path based on recent history, and keeps running ahead. This is branch prediction.

When the guess is right (which is most of the time for predictable patterns like loop counters), the pipeline hums along efficiently. When it’s wrong, the CPU has to discard all the speculative work it did and restart down the correct path. On a tight inner loop processing millions of records, this penalty compounds quickly.

The compiler’s fix: predication

For branches that are genuinely hard to predict, say, a condition that depends on random data, the compiler has a trick called predication: converting the branch into pure arithmetic.

Instead of:

if (x % 2 == 0) sum += x;

The compiler emits something like:

mask = -(x % 2 == 0);   // all 1s if true, all 0s if false
sum += x & mask;         // adds x if even, adds 0 if odd

No branch. No prediction needed. No misprediction possible. The CPU just executes a few arithmetic instructions unconditionally. LLVM applies this automatically, so what looks like a conditional in your source code compiles down to four arithmetic instructions with zero branches.

Predication is elegant. But even a perfectly predicated loop can still be far slower than it needs to be.

SIMD: Doing Eight Things at Once

Modern CPUs have wide registers of 256 bits on most x86 machines that can hold eight 32-bit integers simultaneously. A single SIMD instruction (Single Instruction, Multiple Data) can operate on all eight at once, in the same time it would take to operate on one.

Scalar:  [a] + [b] = [c]                         — 1 result per instruction

SIMD:    [a1, a2, a3, a4, a5, a6, a7, a8]
       + [b1, b2, b3, b4, b5, b6, b7, b8]
       = [c1, c2, c3, c4, c5, c6, c7, c8]        — 8 results per instruction

You don’t usually write SIMD manually. The compiler does it for you through auto-vectorization. When it sees a loop over an array, it checks whether iterations are independent of each other. If they are, it rewrites the loop to use SIMD instructions processing 8 (or 16, or 32) elements per cycle instead of one.

The condition for vectorization is simple but strict: the compiler must be able to see the whole loop, and iterations must not depend on each other.

A loop like this is vectorizable:

for i in 0..n {
    output[i] = input[i] * 2;
}

Each iteration is independent. The compiler can freely batch them into groups of 8 and fire SIMD instructions. This is the difference between running at 1/8th of hardware speed and running at full speed.

Zero-Cost Abstractions: The Promise

Rust is built on the idea of zero-cost abstractions. The concept, inherited from C++, means two things:

1. You don’t pay for what you don’t use. A feature that exists in the language but isn’t used in your program adds no overhead.
2. You couldn’t hand-code it better. When you do use an abstraction, the compiled output is equivalent to the best low-level code you could write by hand.

A classic example is Rust’s iterator combinators. When you write:

let sum: i32 = data.iter()
    .filter(|&&x| x % 2 == 0)
    .map(|&x| x * 2)
    .sum();

This doesn’t create intermediate collections. It doesn’t loop three times. The compiler fuses filter, map, and sum into a single tight loop over the data, identical to what you'd write by hand. Zero overhead.

Another form is Zero-Sized Types (ZSTs). You can encode state directly in the type system:

struct Enabled;    // 0 bytes
struct Disabled;   // 0 bytes
struct Input;      // 0 bytes
struct Output;     // 0 bytes

struct Pin<State, Direction> {
    register: u32,
    state: State,         // vanishes at runtime
    direction: Direction, // vanishes at runtime
}

Enabled and Input are real types at compile time, the compiler uses them to prevent invalid operations and catch misuse. But they carry no data, so they occupy zero bytes at runtime. The entire type machinery evaporates, leaving only the register access. Rich compile-time safety, zero runtime cost.

This is the beauty: the compiler does the work, not the CPU.

Zero-Cost: The Hidden Catch

Here is the subtlety that most engineers never need to confront.

Consider a merge iterator, a common pattern in database query engines. You have multiple sorted iterators (each over a different key range), and you need to produce a single sorted, deduplicated stream from all of them. The merge iterator does this by repeatedly asking: “which of my child iterators has the smallest current value?” and advancing the winner.

Built on Rust’s Iterator trait, this is idiomatic and clean. Each call to next():

1. Peeks at the front of every child iterator
2. Finds the smallest current key
3. Advances that child iterator’s internal position
4. Returns the winner

The zero-cost promise holds here: each individual next() call compiles down to exactly the assembly you'd write by hand. LLVM even applies predication inside the loop body. In that sense, the abstraction costs nothing.

But there is a problem the promise doesn’t cover.

Each call to next() mutates internal state that the next call depends on. The result of call N+1 cannot be determined until call N has finished and updated the child iterators' positions. This is a sequential dependency chain — invisible at the abstraction level.

From the compiler’s perspective, the loop looks like:

v1 = next()   ← must complete before
v2 = next()   ← must complete before
v3 = next()   ← must complete before
...

The compiler cannot vectorize this. It cannot batch 8 values and process them with SIMD, because it doesn’t know what values 2 through 8 are until value 1 has been computed and the iterators’ state updated. SIMD is impossible. Unrolling is impossible. The CPU executes one value, waits, executes the next, waits, running at a fraction of its capacity.

The abstraction compiled each call perfectly. It simply couldn’t see, and therefore couldn’t optimize across calls.

“Zero-cost” describes what the abstraction adds. It says nothing about what it prevents.

Batching: Giving the Compiler What It Needs

The fix is a technique as old as databases: batched iterators.

Instead of producing one value per next() call, the iterator fills an array of N values and returns the whole batch at once:

// Before: one value at a time
while let Some(v) = merge_iter.next() {
    process(v);
}

// After: 512 values at a time
while let Some(batch) = merge_iter.next_batch() {
    for v in batch {
        process(v);   // tight inner loop over a plain array
    }
}

This splits the work into two loops:

Outer loop — calls the merge iterator. Still stateful and sequential, as it must be. But it now runs 512× less often, so its cost is amortized to near-zero per element.

Inner loop — processes a plain array. No hidden state. No recursive calls. No dependencies between iterations. This is exactly the shape the compiler needs to vectorize.

The compiler sees the inner loop, confirms iterations are independent, and emits SIMD instructions processing 8 values per instruction instead of 1.

The merge cost didn’t disappear. The algorithm didn’t change. The data didn’t change. The only change was making the loop shape visible to the compiler.

Putting It All Together

These four concepts form a chain that connects high-level code to hardware reality:

Branch prediction keeps the CPU pipeline full by guessing ahead. Mispredictions cost 15–20 cycles — expensive on hot loops processing unpredictable data.

Predication eliminates unpredictable branches by replacing conditionals with arithmetic. No branch, no prediction, no penalty. The compiler applies this automatically when it can.

SIMD processes 8+ values per instruction instead of 1. It’s the difference between running at 1/8th hardware speed and full speed — but only available when the compiler can see a tight loop over independent, contiguous data.

Zero-cost abstractions compile away at the per-call level. But abstraction boundaries can hide the loop structure the compiler needs for SIMD and unrolling. Zero-cost in what they add; potentially costly in what they prevent the compiler from seeing.

Batching restores that visibility. Amortize stateful work across large chunks. Expose the inner loop as plain array iteration. Give the compiler what it needs to go fast.

The Takeaway

Abstractions are how we manage complexity. But now and then, you have to look beneath them.

Reference: https://turbopuffer.com/blog/zero-costhttps://doc.rust-lang.org/beta/embedded-book/static-guarantees/zero-cost-abstractions.html