Photo by author

KV Cache Explained Simply: The Trick That Makes LLMs Fast

Most developers have heard the term. Very few understand what it actually stores, why it runs out of memory, and what happens when it does.

Imagine you are reading a novel. You reach page 300 and want to understand what happens next. A normal reader keeps going from where they left off.

Now imagine you had to re-read all 300 pages from scratch before reading page 301. Then all 301 pages before page 302. Then again, before 303.

That is exactly what a transformer does at inference time without a KV cache. Every single token it generates requires reprocessing everything it has seen before. For a short response, barely noticeable.

For a long conversation, it is computationally unworkable.

KV cache is the fix.

This article explains exactly what it stores, why that costs memory, and what production systems do about it.

If you want more such information about AI, consider subscribing to my newsletter, where you will get noise-free information every week

Link for the newsletter: Newsletter

Photo by author

How transformers generate text

LLMs are autoregressive. That just means: the model generates one token at a time, and each new token depends on every token before it.

Send the prompt “What is the capital of France?” and the model does not output the full answer at once. It generates one token, appends it to the sequence, generates the next, and so on. For each token, it runs a full forward pass through the network. The core of that forward pass is the attention mechanism.

The attention mechanism, without the math

The attention mechanism is how a transformer decides which parts of everything it has seen are relevant to the token currently being generated.

Every token in the sequence gets converted into three vectors: a Query (Q), a Key (K), and a Value (V).

Think of it like a search engine.

• The Query is what you are looking for.
• The Keys are the labels on everything the model has seen.
• The Values are the actual content stored under each label.

When the model generates a new token, it takes that token’s Query vector and compares it against every previous token’s Key vector.

That comparison produces attention weights.

The model then uses those weights to pull a mix of Value vectors, and that mix becomes the context for predicting the next token.

This happens at every transformer layer, for every attention head, for every token generated. Here is where the problem starts.

The recomputation problem

When you generate the second output token, the model attends to all previous tokens again, including the first output token.

So it recomputes K and V for every token it has already processed.

When you generate the hundredth token, it recomputes K and V for all 100 previous tokens. Again.

Without optimization, generating N tokens requires processing the same early tokens O(N) times. Total compute grows quadratically with sequence length.

Without KV cache:
  Step 1: compute K,V for [t1]
  Step 2: compute K,V for [t1, t2]
  Step 3: compute K,V for [t1, t2, t3]
  ...
  Step N: compute K,V for all N tokens  →  O(N²) total work

With KV cache:
  Step 1: compute K,V for [t1]          →  store in cache
  Step 2: compute K,V for [t2] only     →  read t1 from cache
  Step 3: compute K,V for [t3] only     →  read t1, t2 from cache
  ...
  Step N: compute K,V for [tN] only     →  read N-1 from cache  →  O(N) total work

Without KV cache, every token you generate makes the next one more expensive to compute. The model re-reads its own output from scratch, every single step.

What KV cache actually does

Photo by author

The key insight is simple: the K and V vectors for a given token only depend on that token’s position and the model’s weights. They do not change based on what comes after.

So if the model computed K and V for token 1 while generating token 2, those exact same values are valid when generating token 100.

KV cache stores those computed Keys and Values so they are never recomputed.

For each new token, the model computes Q, K, and V only for that one new token, appends K and V to the cache, and reads the rest from memory. The Query is freshly computed because it expresses what the new token is searching for. The Keys and Values of everything before it? Already there.

Depending on sequence length and model size, this reduces generation time by 10x to 50x compared to naive recomputation.

Prefill and decode: two different problems

When you send a prompt, inference splits into two phases with completely different behavior.

Prefill is when the model processes your entire prompt. All prompt tokens are processed in parallel, and every token’s K and V are computed at once. This phase is compute-bound: the GPU is doing dense matrix math fast. By the end of prefill, the KV cache is fully populated for your prompt.

Decode is when the model generates output tokens one at a time. Each step is sequential because each token depends on the previous one. This phase is memory-bound: the GPU is not doing much raw computation; it is mostly reading the KV cache from memory and running a lightweight forward pass for a single new token.

Inference timeline:

[Your prompt]            [Generated output]
  ─────────────────────── ─────────────────────────────────────────
   PREFILL PHASE            DECODE PHASE (one token at a time)
   All tokens in parallel   Sequential, one step per token
   Compute-bound            Memory-bound
   Populates KV cache       Reads from KV cache every step

This distinction matters more than most developers realize. If decode is slow, the problem is usually memory bandwidth, not compute. Throwing more GPU cores at a memory bandwidth bottleneck does not help.

Why is long context expensive?

The KV cache stores Key and Value vectors for every token, at every transformer layer, for every attention head. As the sequence grows, the cache grows linearly.

For a 70B parameter model handling a 200K token context window, the KV cache alone can consume 40 to 80 GB of GPU memory. A single H100 has 80 GB total. That is the entire card, just for context state, before model weights are loaded.

Every token you add to the context window costs more cache memory, and that memory competes directly with batch size. More context per request means fewer simultaneous requests the system can serve.

Long context is a deliberate engineering tradeoff, not a free feature.

See KV Cache in Code

In this chunk of code, you use caching while generating the output by using the parameter use_cache

In this, we used the HuggingFace Transformer Library to implement KV Cache, which is quite easy

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
inputs = tokenizer("What is the capital of France?", return_tensors="pt")

# KV cache is used by default (use_cache=True)
outputs = model.generate(
    **inputs,
    max_new_tokens=20,
    use_cache=True   # default; remove this and generation slows measurably
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

KV cache is on by default, so I would recommend you to experiment by turning it off.

How production systems manage it

Before 2023, most inference systems allocated KV cache memory contiguously, reserving the maximum context length for every request up front. A model supporting 4K tokens would reserve 4K tokens of cache even for a 50-token prompt. Prior systems wasted 60 to 80 percent of allocated KV cache memory this way.

The vLLM team’s PagedAttention fixed this by borrowing virtual memory from operating systems. Instead of one large contiguous block per sequence, it divides the KV cache into small fixed-size pages, allocated on demand as tokens are generated.

When a sequence ends, its pages are freed for new requests. KV cache waste drops to under 4 percent, enabling 2 to 4x throughput improvements on the same hardware.

Two techniques reduce the raw size of the cache itself:

• KV quantization. Keys and Values are typically stored in BF16 (2 bytes per element). Switching to FP8 halves the memory footprint. Attention is tolerant of this precision reduction because softmax normalization averages out small rounding errors. FP8 KV cache is production-ready on H100s and A100s with negligible quality loss for most workloads.
• Grouped Query Attention (GQA). Standard multi-head attention gives every head its own K and V vectors. GQA shares K and V across multiple heads, shrinking the cache proportionally. Llama 2 70B, Mistral, and most modern frontier models use GQA specifically for this reason.

When KV cache does not help

KV cache only benefits the decode phase. Prefill always computes from scratch because you are processing the full prompt in parallel and there is nothing cached yet.

If your workload is mostly prompt processing with minimal output, KV cache saves you very little.

A summarization pipeline that takes a 10,000-token document and outputs a 100-token summary spends almost all its time in prefill. The cache barely enters the picture.

Where KV cache matters most: conversational systems, code completion, anything generating long outputs where output token count is significant relative to the input.

There is also a failure mode worth knowing. In a high-throughput serving system, if the cache is not managed carefully, you exhaust GPU memory before you exhaust compute. The system starts rejecting requests or throttling batch sizes. This is not the model being too slow. It is the cache being over-allocated.

The bottleneck in LLM serving is almost never raw compute. It is memory: model weights, KV cache, and the bandwidth to move them fast enough.

Where to focus based on your situation

Photo by author

The one thing to remember

Most engineers think LLMs are slow because they are big. The weight count is what makes them heavy, so surely that is the bottleneck.

It is not. A loaded model sits in VRAM and does not move. What moves constantly, and grows with every token generated, is the KV cache. The memory it consumes, the bandwidth required to read it on every decode step, and the fragmentation it causes across concurrent requests are what production LLM infrastructure is mostly built around.

If you ever wonder why LLM serving teams obsess over memory rather than FLOPS, this is why. The model is not the bottleneck. The cache is.

The next concepts that build directly on this: Flash Attention (makes prefill faster and more memory-efficient), continuous batching (handles new requests without waiting for existing ones to finish), and speculative decoding (makes the sequential decode phase faster by running some of it in parallel).