How DeepSeek exactly implemented Latent Attention | MLA + RoPE

DeepSeek’s Multi-Head Latent Attention (MLA) is one of the most important innovations behind DeepSeek-V2 and DeepSeek-V3. Instead of storing massive Key-Value caches like traditional Transformers, MLA compresses attention information into a compact latent representation, dramatically reducing memory usage while maintaining performance. In this article, we’ll explore how MLA works, how RoPE is integrated, and why this design makes large-scale language models more efficient.

Normal MLA (without RoPE)

Why MLA Works Without RoPE

Attention scores are computed as:

QKᵀ

= (XW_q)(WᵀᵤₖWᵀ_dkvXᵀ)

= X(W_qWᵀᵤₖ)(XW_dkv)ᵀ

Notice that:

W_qWᵀᵤₖ

depends only on model weights and is fixed after training.

Therefore, it can be absorbed into a single matrix:

W_absorbed = W_qWᵀᵤₖ

As a result, only:

XW_dkv

needs to be cached during inference.

✅ Smaller KV cache

✅ Lower memory usage

✅ Faster inference

What Changes When RoPE Is Added?

RoPE applies a position-dependent rotation:

RoPE(x₁, x₂, x₃, x₄)

↓

(x₁′, x₂′, x₃′, x₄′)

Since RoPE depends on token position:

Rₚₒₛ₁(x) ≠ Rₚₒₛ₂(x)

Without RoPE vs With RoPE Attention Scores

the attention score becomes:

Rₚₒₛ(XW_q) ∗ Rₚₒₛ(Wᵀᵤₖ(XW_dkv)ᵀ)

Now the projection matrix can no longer be absorbed because:

Rₚₒₛ(WᵀᵤₖA)

≠

WᵀᵤₖRₚₒₛ(A)

The rotation is applied after the projection and varies with position.

Therefore:

Wᵀᵤₖ cannot be merged into W_q.

Consequence

Without RoPE:

W_qWᵀᵤₖ

is computed once and reused.

With RoPE:

Rₚₒₛ(Wᵀᵤₖ(XW_dkv)ᵀ)

must be reconstructed for every token position.

↓

We need to recompute the keys for all tokens during inference.

↓

More computation

↓

More memory traffic

↓

Lower inference throughput

↓

This significantly hinders inference efficiency.

How to solve this ?

DeepSeek said if I can’t absorb Rₚₒₛ(Wᵀᵤₖ(XW_dkv)ᵀ) while using RoPE why don’t I break down latent attention in 2 parts.

DeepSeek innovation

Here, I break Query into 2 parts Qc and Qr where Qc is without RoPE and Qr is with RoPE and vice versa to Keys.

In this way, I can apply absorption trick and retain old magic of MLA in some part and on other part I perform computation. This idea is called Decoupled RoPE explained below

DeepSeek’s Key Idea: Decoupled RoPE

Instead of applying RoPE to the entire Query and Key vectors, DeepSeek splits them into two components:

Query = [Qᶜ, Qʳ]

Key = [Kᶜ, Kʳ]

where:

• Qᶜ, Kᶜ → Content vectors (no RoPE applied)
• Qʳ, Kʳ → Positional vectors (RoPE applied)

The attention score becomes:

(QᶜKᶜᵀ) + (QʳKʳᵀ)

The first term contains no positional encoding, so the MLA absorption trick still works:

QᶜKᶜᵀ

= X(W_qᶜWᵀᵤₖ)(XW_dkv)ᵀ

Since the projection matrices can still be absorbed, only the compressed latent representation needs to be cached.

For the second term:

QʳKʳᵀ

RoPE is applied normally to preserve positional information.

As a result, DeepSeek gets the best of both worlds:

✅ Retains MLA’s KV-cache compression

✅ Preserves RoPE’s positional awareness

✅ Avoids reconstructing full keys for every token

✅ Maintains high inference efficiency

In other words, DeepSeek keeps the “old MLA magic” on the content component while applying RoPE only to a small positional component.

How Decoupled RoPE works?

Normal latent attention works like this. Look it carefully, deepseek made changes in this why multiplying with Wq(8,4) which we will discuss. They first down projected (WDq) to get Cq and again upprojected(Wu₂) again to get Qc matrix

WDq → Cq → Wu₂ → Qc.

Normal latent attention

Why they did down projection and up projection?

They mention in paper that it saves the activation memory during training time leading to better performance.

Implementation Decoupled RoPE of consist of two part:

Part 1- Without RoPE:

Without RoPE

Part 2- With RoPE:

With RoPE

With RoPE

See Qr and Kr both have dimension 4,8 i.e (nh dhR) but see the difference how Qr and Kr is formed. Kr is formed by expanding/sharing over head whereas in Qr there are different for different head because dimension of Wqr is (4,8) but Wkr is (8,4).*

Notice that both Qʳ and Kʳ have the same final dimension:

(nₕ × dₕᴿ) = (4, 8)

However, they are constructed very differently.

For Qʳ, the projection matrix W_qr is head-specific. Since W_qr ∈ ℝ⁴ˣ⁸, each attention head learns its own positional query representation:

Qʳ₁, Qʳ₂, Qʳ₃, ..., Qʳₙₕ

This allows different heads to attend to positional information in different ways.

In contrast, Kʳ is generated using a shared projection W_kr ∈ ℝ⁸ˣ⁴. A single RoPE key vector is first produced and then expanded (shared) across all heads.

Kʳ_shared
      ↓
Head₁
Head₂
Head₃
...
Headₙₕ

Therefore:

• Qʳ → Multi-head, head-specific positional queries.
• Kʳ → Single shared positional key replicated across heads.

This design is important because it preserves the expressive power of multi-head attention while storing only one positional key representation in the KV cache, significantly reducing memory usage.

All of these variable can be seen in deepseek paper.

d= original dimension

dc=latent dimension

dc`= query down projected dimension.

dhR=RoPE dimension

nh= no of attention head

dh= original dimension

DeepSeek paper

DeepSeek Paper

DeepSeek Paper

Till now, we can now understand 9–15 equation.

Now understand other remaining equation.

Attention Scores

Now, you can understand this diagram for Deepseek.

DeepSeek: Multi Head Latent Attention

You see along with Ckv matrix, kR matrix is also cached as we have additional computatio here.

So what exactly happens when a new token comes in?

Two path for next token

Attention scores for Path 1 is given by:

Attention scores for Path 2 is given by:

Merging path1 and path 2

Merging

Journey of Token in DeepSeek MLA

Note

The term nₕdₕᴿ corresponds to storing a separate RoPE key for each attention head. DeepSeek avoids this overhead by sharing a single RoPE key Kʳ across all heads.

As a result, the KV cache stores:

dc + dₕᴿ

instead of:

dc + nₕdₕᴿ

This preserves MLA’s KV-cache compression while significantly reducing memory usage.

All of these variable can be seen in deepseek paper.

d= original dimension

dc=latent dimension

dc`= query down projected dimension.

dhR=RoPE dimension

nh= no of attention head

dh= original dimension

KV Cache Memory Size Comparision

• l = number of layers
• b = batch size
• s = sequence length
• h = number of heads
• dₕ = per-head dimension
• g = number of KV groups
• The first 2 = Key + Value
• The second 2 = fp16 (2 bytes)

KV cache Memory Size Comparision

MHA vs (MLA+RoPE)

Even after adding RoPE, MLA only needs to cache d_c + dₕᴿ instead of storing full multi-head keys and values. As a result, MLA + Decoupled RoPE reduces KV-cache memory by about 57× compared to standard MHA.

Memory Comparision

Performance Comparision

Architecturally, MLA achieves up to ~57× theoretical KV-cache compression compared to standard MHA. In practice, DeepSeek reports about ~25× KV-cache reduction while matching or outperforming MHA across major benchmarks such as BBH, MMLU, C-Eval, and CMMLU.