Understanding DeepSeek Architecture: Rotary Positional Embeddings (RoPE)

I wrote this blog for my own edification

All images generated in Python if not stated otherwise

Photo by Solen Feyissa on Unsplash

If you have been blown away by the DeepSeek models, then maybe you would like to know how it works under the hood. In this article, we dive deep into Rotary Positional Embeddings (RoPE), which is one key improvement that the researchers proposed.

First, let's understand positional encodings.

What is Positional Encoding and Why Do We Need It?

Imagine you have a sentence like “The cat sat on the mat.” In many language models, especially Transformers, the model processes all words in the sentence simultaneously (or nearly so). If we just feed the word embeddings (numerical representations of words) into the model, it has no inherent idea of the order of the words. It wouldn’t know that “cat” comes before “sat,” or that “the” appears twice but in different positions.

Positional Encoding is a technique to give the model information about the position of each token (i.e. word or sub-word) in a sequence.

Why is word order important?

• “The cat chased the dog” means something different from “The dog chased the cat.” The embedding for “dog” or “cat” would always be the same regardless of where it is in the sentence (shown below).

• Grammar and syntax depend heavily on word order.

Early methods of positional encoding involved adding a fixed vector (based on the position) to the word embedding.

RoPE offers a different, and often more effective, approach.

Introducing Rotary Positional Encoding (RoPE): The Core Idea

Instead of adding a positional vector to the word embedding, Rotary Positional Encoding (RoPE) rotates the word embedding based on its position.

Imagine each word embedding as a point (or a vector) in a 2D space (we’ll generalize to higher dimensions later, but 2D is easiest to visualize).

• If a word is at position 0, its embedding might point in a certain direction.
• If the same word appears at position 1, RoPE will take its original embedding and rotate it by a specific angle.
• If it’s at position 2, it will be rotated by a larger angle, and so on.

The key insight of RoPE is that the relative position between two words can be captured by the difference in the angles by which their embeddings are rotated. This is important because attention mechanisms (which we’ll touch on later) often rely on how related two words are, and their relative positions are a big part of that relationship.

Think of it like hands on a clock:

• The position of the minute hand tells you something absolute (e.g., 15 minutes past the hour).
• The angle between the minute hand and the hour hand tells you something relative about their positions.

RoPE aims to encode positional information in a way that is naturally compatible with the dot-product operations used in the attention mechanism of Transformers. The rotation ensures that the dot product between two rotated vectors depends on their original values and their relative positions.

Representing Token Embeddings for Rotation (Using Pairs)

RoPE doesn’t rotate the entire high-dimensional word embedding vector as a single unit with one complex rotation. Instead, it groups the dimensions of the embedding vector into pairs.

Let’s say your word embedding vector x has d dimensions:

RoPE considers these dimensions in pairs:

Each of these pairs can be thought of as a 2D vector or, equivalently, as a complex number.

• As a 2D vector:

(where k=i/2)

• As a complex number:

(where j is the imaginary unit)

Why pairs? The fundamental operation in RoPE is a 2D rotation, and 2D rotations are naturally applied to 2D vectors or complex numbers. By breaking the d-dimensional embedding into d/2 pairs, we can apply d/2 independent 2D rotations.

Example: Suppose our word embedding dimension d = 4. Our embedding vector is x = [x₀​,x₁​,x₂​,x₃​]. We split this into two pairs:

• Pair 1: (x₀​,x₁​​)
• Pair 2: (x₂​,x₃)

Each of these pairs will be rotated independently, but the rotation angle applied to each pair will be different (we’ll see why in a later step). This use of different rotation frequencies for different pairs allows the model to capture positional information at various scales.

So, the core idea is that for a token at a specific position m, we take its original embedding xₘ​, split it into these pairs, and then rotate each pair. The amount of rotation will depend on the position m and which pair it is.

The Rotation Mechanism (2D Rotation Matrix or Complex Multiplication)

For each pair (xᵢ​,xᵢ₊₁) from our embedding, and for a given position m, we want to rotate it by an angle, let’s call it θₘ​.

There are two equivalent ways to represent this 2D rotation:

1. Using a 2D Rotation Matrix:

If we think of our pair as a 2D column vector *v = (xᵢ​,xᵢ₊₁​​), we can rotate it by an angle θ*ₘ​ by multiplying it with a 2D rotation matrix:

The rotated vector v′ is then:

So, the new components are:

2. Using Complex Number Multiplication:

If we think of our pair as a complex number

(where j is the imaginary unit), we can rotate it by multiplying it with another complex number that represents the rotation. This rotation complex number is

The rotated complex number c′ is:

Since j ² = −1:

The real part of c′ is our new xᵢ′​. The imaginary part of c′ is our new xᵢ₊₁′​.

Both methods give the exact same result, which is a hallmark of a well-defined mathematical operation! The complex number formulation is often more concise in papers, but the matrix form clearly shows the transformation on the vector components.

Example: Let’s say for a specific pair (x₀​,x₁​​) = (1,0) and the rotation angle for its position is θₘ = π/2 (90 degrees) and cos(π/2)=0 and sin(π/2)=1.

Using the matrix:

So, the vector (1,0) (pointing along the x-axis) rotates to (0,1) (pointing along the y-axis).

This rotation is applied to each pair of dimensions in the embedding. The crucial part, which we’ll cover next, is how θₘ​ is defined and why it might be different for different pairs.

Defining the Rotation Angle θₘ​

Now we need to figure out what this rotation angle θₘ actually is. It needs to depend on two things:

1. The position of the token, m.
2. Which pair of dimensions (xᵢ​,xᵢ₊₁) we are currently rotating.

The formula for the angle θₘ,ᵢ (where m is the position and i indicates the dimension pair index) is:

Let’s break this down:

• m: This is the absolute position of the token in the sequence (e.g., 0, 1, 2, …). As the position m increases, the angle (m⋅something) also tends to increase, meaning more rotation for tokens further down the sequence.
• d: This is the total dimensionality of the word embedding (e.g., 512, 768).
• i: This is the index of the first dimension in the pair we’re considering. So, for the pair (x₀,x₁​), i=0. For (x₂​,x₃​), i=1, and so on, up to d−2.
• B: This is a large constant, often chosen to be 10000. It’s a hyperparameter.

What does the term 1/[ B^(2i/d) ]​ do?

This term acts like a “frequency” or “wavelength.”

• When i is small (e.g., i=0, for the first pair (x₀​,x₁​​) ):
• 2i/d = 0
• B⁰ = 1
• The term becomes 1/1=1.
• So, for the first pair, θₘ,₀ ​= m⋅1 = m. The angle changes linearly with position m.
• When i is large (e.g., i ≈ d−2, for the last pairs):

the term becomes 1/B² = 1/10000². This is a very small number.

So, for the later pairs, θ ​= m⋅(very small number). The angle changes very slowly with position m.

Analogy: Think of a set of gears or spinning discs.

• The first pair (x₀,x₁​) rotates by m radians.
• The next pair (x₂​,x₃​) rotates by m⋅( 1/B^(4/d) ) radians, which is slower.
• Subsequent pairs rotate even more slowly.

This is similar to how sines and cosines of different frequencies are used in traditional sinusoidal positional embeddings. The higher frequency (faster changing angle) parts capture fine-grained relative position information for nearby words, while lower frequency (slower changing angle) parts capture coarser-grained information over longer distances.

So, for a token at position m, its embedding x=[x0​,x1​,…,xd−1​] is transformed as follows:

The pair (x₂ₖ​, x₂ₖ₊₁​) is rotated by angle m⋅ θₖ​, where θₖ ​= 1/B^(2k/d).

…and so on.

This means different parts of the embedding vector are rotated at different “speeds” based on their index i and the token’s position m.

Applying RoPE to Query and Key Vectors

In the core of a Transformer lies the self-attention mechanism. For each word, the model generates three vectors: a Query vector, a Key vector, and a Value vector.

• Query (Q): Think of this as the current word asking, “Who should I pay attention to?”
• Key (K): Think of this as every other word in the sequence having a label that says, “Here’s what I’m about.”
• Value (V): This vector holds the actual content or meaning of a word that gets passed along if attention is paid to it.

The attention score between two words is calculated by taking the dot product of the first word’s Query (Q) vector and the second word’s Key (K) vector.

This is where RoPE comes in. Instead of applying the rotation to the initial word embeddings, RoPE is applied independently to the Query and Key vectors just before they are used to calculate the attention scores.

So, the process looks like this:

1. Start with the word embedding for a token at position m.

2. Create the Query vector qₘ​ and Key vector kₘ​ from this embedding (this is done via standard linear transformations, just like in a normal Transformer).

3. Apply RoPE:

• Rotate the Query vector qₘ​ according to its position m to get a new vector, qₘ ′
• Rotate the Key vector kₘ​​ according to its position m to get a new vector, kₘ​′​

Calculate the attention score between a token at position m and another at position n using the dot product of the rotated vectors: score = qₘ ′​. kₙ′​

Crucially, the Value (V) vector is NOT rotated. The rotation’s purpose is to modify the relationship score (the dot product) to include positional information. The Value vector’s job is to carry the actual content, which should not be altered by its position.

The Dot Product

This is where all the previous steps come together to achieve the final goal. We’ve established that we rotate the Query vector​ from position m and the Key vector from position n. The attention score between them is their dot product.

Let’s see why this is so special.

The core property of RoPE is that the dot product of the two rotated vectors, qₘ ′​. kₙ′​, simplifies to a form that only depends on the original vectors and their relative position, m−n.

Let’s look at a single 2D pair to understand this.

• Original Query pair: (qᵢ,qᵢ₊₁​)
• Original Key pair: (kᵢ,kᵢ₊₁​)
• Rotation angle for position m: θₘ,ₖ ​= m ⋅ θₖ​
• Rotation angle for position n: θₙ,ₖ​ = n⋅ θₖ​

After performing the rotations and calculating the dot product for just this one pair, the math simplifies to this elegant result:

This equation is the heart of RoPE. Let’s break it down:

• The terms like

depend only on the original, un-rotated Query and Key vectors. They capture the content or meaning of the words.

• The terms

depend only on the relative distance, m−n.

Notice that the absolute positions m and n have completely disappeared! Only their difference, m−n, remains.

What This Achieves

1. Pure Relative Positional Information: The attention score between two words becomes a function of their content (the original q and k) and their relative position. This is exactly what we want. The relationship between “cat” and “sat” should be the same whether the sentence is “The cat sat” or “Yesterday, the big fluffy cat sat on the mat.” RoPE achieves this naturally.
2. No Information Contamination: Unlike methods that add a positional vector to the word embedding, RoPE uses rotation. This means the length (or magnitude) of the Query and Key vectors doesn’t change. The positional information is encoded purely in the vector’s “direction.” This is a cleaner way to inject positional data without altering the vector’s original magnitude.
3. Handles Long Sequences: The use of sines and cosines means the positional signal has a natural decay over long distances. As the relative distance m−n gets very large, the positional signal continues to provide information without growing infinitely, which can be a problem for other methods.

This is the core idea of Rotary Positional Encoding. It’s a mathematically elegant solution that uses rotations to weave relative positional information directly into the self-attention mechanism, making it a very effective and widely used technique in modern language models.

Conclusion: The Power of RoPE

Rotary Positional Encoding (RoPE) is a clever and elegant method for teaching Transformer models about the order of words in a sentence. Instead of adding positional numbers to word embeddings, RoPE encodes positional information by rotating the Query (q) and Key (k) vectors in the attention mechanism.

The core process involves:

1. Breaking the Query and Key vectors into pairs of dimensions.
2. Viewing each pair as a 2D vector (or complex number).
3. Rotating each pair by an angle that depends on its absolute position (m) and its dimension index (i). Different pairs are rotated at different speeds, allowing the model to understand both short-range and long-range relationships.

The true magic happens during the attention calculation. The dot product between a rotated Query and a rotated Key​ simplifies mathematically so that it only depends on two things:

• The original, un-rotated vectors.
• The relative position between them (m−n).

This means the absolute positions disappear from the attention score, leaving only the relative distance that truly matters for understanding context.

Key Advantages

• Relative Position by Design: It directly embeds relative positional information into the self-attention mechanism, which is often more useful for language tasks than absolute position.
• Long Sequence Handling: Its sinusoidal nature allows it to effectively handle very long sentences without the positional signal becoming unstable.
• Preserves Magnitude: Because it’s a pure rotation, it doesn’t change the length (norm) of the vectors, preventing the “meaning” of the word embedding from being altered by the positional encoding.
• No Learnable Parameters: RoPE is a fixed transformation, adding no extra parameters to be trained, making it computationally efficient.