The Journey to Multi-Head Latent Attention

Why DeepSeek-V2 invented a strange-looking attention block — and how a tiny algebraic trick made the KV cache 57× smaller without dropping a single point of quality.

Tokens stream in, compress into a single latent vector, then refract back into per-head keys and values — the essence of MLA.

Everyone talks about how transformers scale. Fewer people talk about how transformers remember. And the honest truth is that for any production LLM running today, the KV cache is the boss of you — it determines your context length, your throughput, your GPU bill, and the kind of hardware you can deploy on. Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, is the most elegant attack on that bottleneck I’ve seen.

This is part one of a two-part journey. Here we’ll learn MLA from first principles — starting from a vanilla attention block, watching the KV cache balloon, and then evolving our way through MQA, GQA, and finally the absorption trick that makes MLA both cheaper and, somehow, better. The second part will pick up where this one stops: the decoupled RoPE trick that lets MLA play nicely with rotary positional embeddings.

🪄 A personal note. I sketched the absorption derivation in my notebook — and that scribble (reproduced further down in the article) is honestly the most compact way to see why MLA works. I’ll walk you through it the same way I walked through it for myself.

Contents

1. Where MLA lives inside a transformer
2. The four families of attention
3. The KV cache — and why it exists at all
4. Why the KV cache explodes (with a 70B example)
5. Multi-Query Attention: share the K and V
6. Grouped-Query Attention: a middle ground
7. Multi-Head Latent Attention: cache a latent
8. The absorption trick (the magic)
9. Doing the math: a 57× compression budget
10. What’s next — decoupled RoPE

Where MLA lives inside a transformer

Before we touch a single equation, let’s put MLA on the map. A transformer block is a sandwich: attention + feed-forward, with residual connections and norms gluing it together. MLA is a drop-in replacement for the attention half. Everything else — the embeddings, the FFN, the LayerNorms, the residuals — stays exactly the same.

Figure 1 — MLA is a drop-in replacement for the attention sublayer. Everything else in the transformer is identical.

DeepSeek-V2 introduced MLA alongside two other ideas: a low-rank joint compression of keys and values (the heart of MLA) and a decoupled rotary positional embedding that solves an awkward incompatibility between low-rank compression and RoPE. We’ll cover the first one cover-to-cover here. The second one gets its own follow-up.

The four families of attention

You’ll see four names floating around in modern LLM papers. They all do the same job — compute weighted averages over past tokens — but they differ in how many keys and values they keep around per head. That single choice cascades into the entire memory and throughput profile of the model.

Four flavors of attention, side by side. The journey of this article is the diagonal from top-left to bottom-right.

In most courses these are described almost as a taxonomy — “here are four options, pick one.” But that’s the wrong way to look at it. They are a chain of fixes. Each one exists because the previous one had a problem. To really feel why MLA is needed, we have to start with the bug it was invented to kill: the KV cache.

The KV cache — and why it exists at all

Autoregressive generation is repetitive, and an unhealthy amount of that repetition is wasted work. Let’s see exactly what gets repeated, and where caching enters the picture.

Step 1: Are we repeating calculations?

Imagine a model has just produced the prompt "the dog". It now wants to predict the next token. At step 1 the model computes Q, K, V for both tokens. Fine. Now it samples "bit" and wants the next token.

Naively, step 2 would re-run the attention block on the entire sequence "the dog bit" — recomputing Q, K, V for "the" and "dog" from scratch, even though their input embeddings haven’t changed and Wq, Wk, Wv are frozen after training. Same input × same weight matrix = same output. We’re burning compute on arithmetic we already did.

Same input × frozen weights = same output. The KV cache is just memoization.

Step 2: What do we actually need to predict the next token?

Walk it backwards. To predict the next token, what do we genuinely have to compute?

1. The logits — but only for the last token. We’re not re-predicting old ones.
2. Logits = context vector × Wo. So we need the context vector of the last token.
3. Context vector = attention weights · V. We need the attention row for the last token (length T) and the full matrix V.
4. Attention row = Softmax(Qlast K.T). We need Qlast — *cheap — and the full K matrix.

So the only new compute is the K, V, and Q for the new token. The K and V for every previous token have already been computed — and they never change, because their inputs don’t change. So just keep them around.

Step 3: Quadratic → linear

Without caching, the work at step t is proportional to t.d (full attention over all past tokens), so generating T tokens costs O(T²) . With caching, each step only does the work for the new token interacting with the cached K, V — that’s O(T d)* over the whole sequence. Quadratic becomes linear.

Total work to generate T tokens: quadratic without caching, linear with it.

Why every LLM you’ve used has a KV cache. Without it, doubling the context length quadruples the inference cost. With it, doubling context only doubles cost.

Why the KV cache explodes (with a 70B example)

So KV cache fixes the compute. Great. The catch is that we’ve bought speed with memory. For a vanilla Multi-Head Attention block, the parameters we have to store in the cache are:

where L = number of transformer blocks, T = sequence length, H = number of attention heads, dₕ = head dimension, B = batch size, and the ×2 accounts for both K and V. Multiply by 2 bytes (FP16/BF16) for actual memory:

A concrete example — LLaMA-2–70B at 32k context

LLaMA-2–70B has L = 80 blocks, H = 64 heads, dₕ = 128 . Take a single sequence (B = 1) at T = 32,768 tokens, FP16:

86 gigabytes — just for the KV cache of a single 32k-token request. That’s more than the model weights themselves, and it’s why even an H100 (80 GB VRAM) chokes on long-context inference for big dense MHA models. Higher memory means higher cost, more swapping, and slower per-token latency.

Every flavor of attention invented after vanilla MHA — MQA, GQA, MLA, sliding-window attention, paged attention — exists to attack this single number.

Multi-Query Attention: share the K and V

The first surgical fix came from Noam Shazeer in 2019. In vanilla MHA, every head learns its own Wₖ and Wᵥ — that’s H separate keys and H separate values per token, all of which we store. But what if all heads shared the same K and V, and only the queries were per-head?

MQA collapses K and V down to a single head shared by all queries.

MQA shrinks the per-token KV cache by a factor of 1/ H — for an 8-head model, an 8× memory saving. The cost: a noticeable drop in quality, because forcing all heads to share a single K and V over-constrains the attention pattern.

Grouped-Query Attention: a middle ground

GQA (Ainslie et al., 2023) is the obvious compromise: pick G groups, where 1 ≤ G ≤ H. Each group shares one K and one V across its heads. When G = 1, GQA collapses to MQA; when G = H, it collapses to MHA. LLaMA-2–70B uses G = 8, giving an H/G = 8 *saving with much less quality damage.

MQA buys you the biggest discount but at the steepest quality price. GQA buys you a decent discount at a small price. Both are still on the same axis: reduce the number of distinct K, V vectors per token. What if there’s an entirely different axis?

Multi-Head Latent Attention: cache a latent

Here’s the leap. MHA, MQA, and GQA all assume we have to cache something shaped like K and V. MLA breaks that assumption. It asks:

What if we cache one small matrix per token — not K, not V, but a compact representation that both can be reconstructed from on demand?

The setup is shockingly simple. We introduce a down-projection that takes the input embedding and produces a latent matrix dimension (lesser than attention head * head dim):

Then we use two up-projections Wᵤₖ and Wᵤᵥ to reconstruct K and V on the fly when we actually need them inside attention:

Queries stay regular (or you can apply the same trick to Q for training-memory savings — DeepSeek-V2 does both).

The MLA pipeline: project input into a small latent, cache that, expand into K and V only when attention runs.

But wait — we just added matrices (Wdₖᵥ, Wᵤₖ , Wᵤᵥ). How does adding more stuff make the system smaller?

Because the new matrices are weights (loaded once with the model), and what we used to store per token was activations (one fresh K and V for every token, forever). We swapped a growing thing for a fixed thing.

Think of it like packing a suitcase. Instead of stuffing your full wardrobe (K and V) into every hotel room you visit, you carry a tiny travel cube (Cₖᵥ) and unpack the outfit you need only when you walk in the door. Same wardrobe, way less luggage.

The absorption trick (the magic)

Let’s look at the standard attention score for query token t against key token s:

Expand and re-associate the matrix product (it’s all just multiplication of fixed matrices on the right side):

Read that again. The product is between two matrices that never change after training. So we can fold them together once offline, into a single matrix — call it the absorbed query projection . At inference time the score becomes:

We never have to reconstruct K at all. We never even materialize Wᵤₖ at runtime. We just multiply the new query against the cached latent.

The same trick works on the value path. The output of attention for token t is:

Substitute V and again re-associate Wᵤᵥ and Wₒ— both fixed:

So the entire forward pass uses only the latent cₖᵥ, plus two new “absorbed” matrices and that we fold once at load time. The original Wᵤₖ and Wᵤᵥ vanish from the runtime path. This is the absorption trick.

Here’s how it looked in my notebook

I want to share the page where this clicked for me, because honestly it’s clearer than any polished diagram. Three rows: the attention score, the context vector, and the logits — each rewritten to absorb the fixed matrices. Whatever survives at the bottom of each line is either folded at training time or cached at inference time.

My notebook page on the absorption trick — yellow ink, dark page, the works. The red boxes are the parts that get folded once at training time.

Why this is so powerful. Notice what the absorption trick does to costs: it eliminates the per-token K/V reconstruction at inference, removes Wᵤₖ and Wᵤᵥ from the hot path, and leaves us with the smallest possible thing to cache. We compressed the cache and saved compute.

Doing the math: a 57× compression budget

Let’s plug DeepSeek-V2’s numbers in. For a single token, the MHA cache holds K and V across all heads:

The MLA cache holds just the latent (plus a tiny decoupled-RoPE channel — more on that next time):

With DeepSeek-V2’s :

That’s the famous “57× cache reduction” headline. And critically — because MLA still gives each head its own up-projection Wᵤₖ and Wᵤᵥ— the expressiveness of multi-head attention is preserved. We didn’t collapse heads like MQA or group them like GQA; we changed what we store, not how many heads we have.

Two birds, one stone

• Cache size: down ~57×. Suddenly the 86 GB cache we computed for a 70B-class model collapses to a couple of GB.
• Quality: DeepSeek-V2’s ablations show MLA matches or beats vanilla MHA on downstream tasks, while crushing MQA/GQA.
• Compute: the absorption trick removes the K/V reconstruction at inference. We saved memory and FLOPs (floating point operation per sec).

Every previous attention variant traded quality for memory. MLA refuses the trade. That’s why it’s the default in DeepSeek-V2, DeepSeek-V3, and the entire DeepSeek-Coder family, and why it’s being copied into the rest of the open-source ecosystem fast.

What’s next — decoupled RoPE

There is one elephant in the room I’ve carefully tiptoed around: rotary positional embeddings. RoPE rotates the K and Q vectors based on absolute position and index before the dot product. But MLA never materializes K — and that means the rotation can’t be applied the usual way.

DeepSeek-V2’s answer is a small, beautiful hack called decoupled RoPE: a tiny side-channel of K and Q that carries position information, kept outside the absorbed path, then concatenated to the main attention. It’s the perfect topic for the next article — short, sharp, and very visual. Stay tuned.

Position-aware attention without breaking the absorption trick. That’s the part two cliffhanger.

If you remember three things from this article

1. The KV cache turns generation quadratic from into linear.
2. MQA, GQA, and MLA are not a taxonomy — they’re a chain of fixes targeting the same memory bottleneck, each more aggressive than the last.
3. MLA’s absorption trick folds fixed matrices together at training time, leaving a tiny latent as the only thing you cache. ~57× smaller (DeepSeek-V2), no quality loss.

Thanks for sticking around for the whole walk. MLA is one of those ideas that looks intimidating from a distance and obvious from up close — and once you’ve seen the absorption trick, you can’t un-see it. The next post picks up exactly where this one stops: decoupled RoPE, the small but clever workaround that lets all of this coexist with rotary positions. If you’d like a nudge when it lands, the clap below is the easiest way to tell me you’re in for round two. 👏