Distributed Inference

How you actually serve a trained model to thousands of concurrent users.

It’s the natural pair to previous day’s article: training is a batch job you run once, but serving runs forever, under a completely different set of constraints, and honestly it’s where some of the cleverest systems engineering in this field is happening right now.

Reframing the problem: latency now matters

Training only cares about throughput,… total tokens processed per second across the whole cluster over weeks. Nobody’s waiting on any single token. Serving flips that: a real person is staring at a cursor, waiting for the next token to appear. You need low latency per user and high throughput across many simultaneous users, and those two goals actively pull against each other, batching more requests together improves throughput but can delay any individual user’s response.

To understand why serving is architecturally so different from training, you need to understand that generating text isn’t one uniform operation, it’s two, with opposite performance characteristics.

Prefill vs. decode: the split that explains everything else

Prefill happens once per request: the model processes your entire input prompt in one shot, computing attention over all prompt tokens simultaneously. This is highly parallel, it looks like training’s forward pass, and it’s compute-bound: you’re limited by how many FLOPs your GPU can crunch per second.

Decode happens repeatedly: the model generates one token, feeds it back in, generates the next, and so on, inherently sequential, one token at a time. Here’s the counterintuitive part: decode is not compute-bound. Processing a single token barely uses the GPU’s compute capacity. What it actually waits on is memory bandwidth, for every single token generated, the GPU has to read the entire model’s weights (and the growing KV-cache, more on that below) from memory. A modern GPU can do vastly more FLOPs per second than it can move bytes per second, so during decode, your expensive compute cores are mostly sitting idle waiting for data to arrive. This is the single most important fact in LLM serving systems, and almost every serving optimization exists to attack this memory-bandwidth bottleneck.

The KV-cache: inference’s memory wall

During decode, recomputing attention over every previous token from scratch at every step would be absurdly wasteful. So models cache the key and value projections of every token they’ve already processed, the KV-cache, and each new token only needs to compute attention against that cache, not redo the whole sequence.

But that cache isn’t free. Its size scales with 2 × layers × heads × head_dim × sequence_length × batch_size (the 2 is for keys and values). Unlike the model’s weights, which are fixed, the KV-cache grows with every token generated and with every concurrent request you’re serving. For long contexts and large batches, the KV-cache can end up consuming more GPU memory than the model weights themselves. This turns serving into a memory-management problem as much as a compute problem, and that’s exactly where the field’s biggest recent innovation comes in.

PagedAttention: treating GPU memory like an operating system does

Naive KV-cache implementations reserve one big contiguous block of memory per request, sized for the maximum sequence length it might ever reach — even if the request only ends up generating 20 tokens. That’s massive waste, and it fragments memory so badly that GPUs sit at low utilization even though they technically have spare capacity.

vLLM’s PagedAttention borrows a decades-old idea straight from operating systems: virtual memory paging. Instead of one contiguous block per sequence, the KV-cache is split into small fixed-size blocks that can live anywhere in GPU memory, non-contiguous, allocated on demand, freed the instant a request finishes. A lightweight block table maps each sequence to its scattered blocks, the same way a page table maps virtual addresses to physical ones. This single idea — near-zero fragmentation, memory allocated just-in-time, is what let serving throughput jump dramatically, because far more requests can be batched into the same GPU memory at once.

Continuous batching: never let a finished request hold up a full one

Old-school serving used static batching: gather a fixed batch of requests, run them together until every single one in the batch is finished, then start the next batch. The problem is obvious once you say it out loud, if one request in the batch needs 500 tokens and another only needs 20, that second request’s GPU slot sits idle (padded with waste) for the remaining 480 steps, waiting on its slower batchmate.

Continuous batching (also called iteration-level scheduling) fixes this by scheduling at the granularity of a single decode step, not a whole request. After every token generated, the scheduler checks: any requests finished? Evict them, free their KV-cache blocks immediately. Any new requests waiting? Slot them in. The batch composition changes dynamically every single step, so the GPU is never carrying idle padding for a request that’s already done. This, paired with PagedAttention’s flexible memory allocation, is what production serving engines (vLLM, TensorRT-LLM, and similar) are built around.

Speculative decoding: exploiting the fact that decode is “free” compute

Remember that decode is memory-bandwidth-bound, not compute-bound, the GPU’s compute cores are mostly idle during decode, waiting on memory. Speculative decoding exploits that idle compute directly: run a small, fast “draft” model to guess the next several tokens cheaply, then have the big model verify all of those guessed tokens in a single forward pass (which is a prefill-like, parallel operation, and therefore compute-bound territory the big model has spare capacity for). If the draft model guessed correctly, you just got several tokens for close to the price of one decode step. If it guessed wrong partway through, you discard the incorrect tail and fall back to normal decoding from that point, you never sacrifice correctness, only speed. It’s a bet that costs almost nothing when it loses and pays off well when it wins, and it works precisely because it’s spending compute capacity that was otherwise being wasted.

Disaggregated serving: splitting prefill and decode onto different hardware

Here’s a genuinely elegant, fairly recent idea (systems like Splitwise and DistServe popularized it): since prefill is compute-bound and decode is memory-bandwidth-bound, running them on the same GPU pool means each phase is competing for the resource the other phase doesn’t even need, a long prefill can stall decode steps for other users sharing that GPU, hurting latency (this is called a “prefill interference” problem). The fix: run prefill and decode on physically separate pools of GPUs, each provisioned and tuned for its own bottleneck, prefill machines optimized for raw compute throughput, decode machines optimized for memory bandwidth and batch size. After prefill finishes, the resulting KV-cache is transferred over the network to a decode machine, which then handles the token-by-token generation. This decouples the two phases’ scaling entirely, letting you provision each independently based on your actual traffic mix.

The rest of the toolbox, briefly

A few more levers worth knowing exist, without going as deep:

Tensor parallelism at inference serves a different purpose than at training time, it’s not just about fitting a model that’s too big for one GPU, it’s about reducing per-token latency for a single request by splitting the compute of each layer across GPUs, since a user waiting on tokens cares about wall-clock time per token, not aggregate throughput.

Quantization — running weights (and sometimes the KV-cache itself) in int8, fp8, or int4 instead of bf16, directly attacks the memory-bandwidth bottleneck that dominates decode, since fewer bytes moved per token means faster generation, at some cost to numerical precision and, if pushed too far, output quality.

Cache-aware routing — when you have many replicas behind a load balancer, routing a new request to whichever replica already has that request’s prompt prefix sitting in its KV-cache (rather than a random replica) turns a full prefill into a much cheaper partial one. This matters a lot for workloads with repeated system prompts or shared context, like chat interfaces.

Put together, that’s the full inference-time stack: the prefill/decode split that explains every downstream design choice, PagedAttention solving the memory-fragmentation problem, continuous batching solving the scheduling problem, speculative decoding and quantization attacking the memory-bandwidth bottleneck directly, and disaggregated serving taking the prefill/decode split all the way to separate hardware pools.