A Visual Dissection of X’s Recommendation Algorithm: Understanding the Transformer Behind It

In the previous article, we focused on analyzing the three retrieval approaches used in X’s older recommendation algorithm. In this piece, we turn to the new system and examine how, during the retrieval stage, a Transformer is used to train the user tower. After reading this article, you should have an intuitive understanding of how Transformer training actually works. If you are not familiar with what the “user tower” or the “retrieval stage” means, it would be best to review the previous article first.

Overview

The new retrieval system still uses a two-tower architecture, but an important difference is that the user tower is now trained with a Transformer (from the same technical lineage as Grok). First, consider the overall diagram:

The goal of training, summarized in one sentence, is to find an optimal function mapping (i.e., parameter matrices) so that the network can produce expected outputs for specific inputs. The core mechanism that enables parameter matrices to continually improve is the concept of “backpropagation.” All parameter matrices — such as the embedding tables, the Query matrix, and the Key matrix introduced later — are initialized with random values. During training, we define a “judge,” namely the loss function, which calculates the error between the ground-truth labels and the predicted labels produced in each training round. This error then guides how each matrix should adjust its values until the prediction error becomes very small. Although this concept will not be mentioned again later, whenever you see a parameter matrix, remember that it is gradually learned step by step.

Preparing User Embedding Vectors

The model’s input consists of vectorized user historical behaviors, ordered by time.

For example, suppose user Alice (ID: 101) yesterday liked a tweet about cats posted by Bob (ID: 303) on the Home timeline (Post ID: 202). On the server, the log might be recorded as:

{User: 101, Post: 202, Author: 303, Action: Like, Context: Home}

The model cannot directly consume this event sequence. It must first be converted into a multi-dimensional real-valued vector. This is done by looking up different embedding tables — such as the user embedding table and the tweet embedding table — to obtain vectors of possibly different dimensions, concatenating them, and then projecting them linearly into a vector with the model’s standard dimension. These embedding tables are concrete examples of the parameter matrices mentioned earlier.

Why must we convert the data into vectors, and specifically multi-dimensional real-valued ones? The feature-fusion process (shown in the below figure) reveals the reason: the system cares about many features, and each feature requires at least one dimension to represent it, so multi-dimensional vectors are a natural choice. Compared with integer vectors, real-valued vectors have stronger expressive power and can be optimized smoothly during training, allowing fine-grained adjustments.

A prepared tensor might look like this (numbers are random):

Input_Tensor = [
  [
    [1.0, 0.2],  # User token (Alice herself)
    [0.9, 0.1],  # History 1 (Alice liked Bob’s cat tweet)
    [0.1, 0.9]   # History 2 (Alice retweeted David’s dog tweet)
  ]
]

Token segmentation has some flexibility. We could treat multiple historical behaviors as one token, but for intuitive understanding — and commonly in practice — each historical behavior is treated as one token.

Note also that tokens must be ordered by time. The reason will become clear later.

Rotary Positional Encoding

Next comes positional encoding. In a Transformer, a token’s position itself contains significant information. Sequence order is crucial for determining meaning and behavior. For example, “Today Bob rescued a cute cat” and “Today a cute cat rescued Bob” contain the same tokens but entirely different meanings.

In X’s recommendation system, the order of user behaviors matters as well, such as “the user first looked at a phone, then looked at a phone case.”

There are many positional encoding methods. X uses Rotary Position Embedding (RoPE). Compared with absolute positions, it emphasizes relative positions. Instead of knowing “this is the 1000th action,” it is more useful to know “after the 998th action, another action occurred before this one.”

After RoPE, the tensor becomes:

RoPE_Output = [
  [
    [ 1.000000,  0.200000],
    [ 0.402125,  0.811354],
    [-0.859982, -0.283602]
  ]
]

“Sandwich” Normalization

To prevent certain values from hijacking the neural network during training, the model performs normalization before the attention sublayer. Normalization converts uncontrolled magnitudes into stable, manageable scales.

X uses Sandwich RMSNorm: normalization both before (Pre-Norm) and after (Post-Norm) each sublayer. This ensures not only that inputs entering the sublayer are well-behaved, but also that outputs are stable — hence the “sandwich” analogy. Compared with LayerNorm, RMSNorm does not subtract the mean; it simply rescales, better preserving the direction of the vector.

After applying pre-sub-layer normalization (note that this step occurs before and after every attention and FFN sub-layer, and will not be repeated later), our tensor now looks like this:

RMSNorm_Output = [
  [
    [ 1.38,  0.27 ],
    [ 0.62,  1.26 ],
    [-1.34, -0.44 ]
  ]
]

You can see that the direction of each vector has not changed — they have only been uniformly “stretched or shrunk.” This is the input to the sub-layer.

The Attention Sublayer

The goal of this layer is to let each time-series token learn “how other tokens influence me,” because in a recommendation system, an isolated event usually does not carry much meaning — only by observing the full context can we more accurately profile a user. This is easy to understand: among people who like pets, someone who clicked on 10 pet-related posts is more likely to be a pet enthusiast than someone who clicked on just one.

This layer has three key parameter matrices: WQ, WK, WV, randomly initialized and learned during training.

WQ (Query Matrix) learns how to ask: “Find content related to cats.” WK (Key Matrix) learns how to be found: “I contain cat-related information.” WV (Value Matrix) learns how to transmit the answer: “Here is the detailed information.”

Both WK and WV are related to answering, but the difference is that WK operates at the control level — it exists to “be found” (like a book’s index number), while WV operates at the information level — it exists to be read (like a book’s content).

Multiplying each matrix with the input tensor produces three matrices: Q, K, V.

Each row in Q, K, and V is the projection of the input embedding vector onto a particular coordinate space (here we use row-vector notation rather than column-vector notation).

Using the attention formula:

Attention(Q,K,V) = softmax(QKᵀ / √dk) V

we obtain the final attention output.

How should we understand this formula?

QKᵀ is the similarity score matrix. Mathematically, Qi·Kj measures how aligned two vectors are. Higher values indicate stronger relationships between tokens.

Dividing by √dk rescales the values to stabilize softmax and prevent gradient issues.

Applying softmax to each row converts the scores into attention weights between 0 and 1. For a given Query, it distributes attention across all Keys.

Element (i, j) in the matrix answers: “For question i, can token j provide a relevant answer?” The closer the value is to 1, the higher the probability of finding an answer.

In the context of a recommendation system, you can think of (i) as “I am about to recommend posts; what historical actions should I pay attention to?” and j as “Pay attention to me, because I represent retweeting about dogs, indicating a strong pet interest tendency.”

Finally, multiplying this weight table by V produces a (T × dv)-dimensional matrix. V represents the actual recommendation direction adjustment: “You should recommend more posts about dog food / pet supplies.” The larger the weight in the weight table, the more valuable that historical action is as a reference.

Since this is a directional adjustment (increment), adding it to the original token vector produces a new vector — this step is called the residual connection.

In the weight diagram above, all tokens are fully visible to each other, meaning that later actions in the time series can also influence earlier actions. Like many other applications, X’s recommendation algorithm adds a causal mask, so that an action is only influenced by earlier actions in the time series. That is, the mask prevents q1 from seeing k2 and beyond, q2 from seeing k3 and beyond, and so on.

Why use a causal mask? From an architectural perspective, because the Transformer we use is a decoder-only model, which is autoregressive: the present me can only be derived from the past me. Mathematically, predicting the output xt at time t can only depend on inputs x{<t} before time t.

In the context of a recommendation system, when we “ask” what other sequences should q2 pay attention to, only the k1 sequence has already occurred. The prerequisite for this mask to work is that our input tokens are already sorted by time. Note that even with the mask, the sum of each row must still equal 1.

The examples we have discussed so far treat the input vector dimension (d_model) as a single complete space. However, in many engineering practices, including X’s recommendation algorithm, Multi-Head Attention is used. It splits d_model into h subspaces and computes attention in parallel, with each subspace learning different relationships, such as a long-term interest subspace, a short-term intent subspace, a behavioral intensity subspace, etc. Each head computes attention independently, and the results are then concatenated and projected back to the d_model dimension. This projection step uses a WO matrix to perform a linear mixing of the concatenated vector, learning the collaborative relationships between heads.

The Feed-Forward Network Sublayer

Attention operates between tokens, allowing token2 (user likes cats) to absorb information from token3 (user likes dogs), and the Feed-Forward Network (FFN) introduced here is for internal digestion and reorganization within a token, transforming it into “the user likes both cats and dogs, is a pet enthusiast, and should be recommended more pet-related content.”

The standard FFN is:

FFN(x) = ReLU(x·W_up + b1)·W_down + b2

ReLU is a nonlinear activation function that acts as a gate to filter features. W_up and W_down are two matrices: W_up is responsible for expanding dimensions — decomposing, amplifying, and finding patterns in features; W_down is responsible for reducing dimensions — reassembling and translating patterns into new higher-level features.

For example (bias terms omitted here):

               [Biological]  [Mechanical]
Input Token x: [1.5,         -1.5]

In the input vector, each column is a “projection” onto a semantic axis. For instance, the first column represents “biological-ness,” and the second column represents “mechanical-ness.” At this stage, the semantics are still highly entangled and ambiguous.

The W_up matrix:

[Pet]  [Car]  [Noise]
[1.0,  -2.0,   0.5]
[-0.5,  2.0,   0.5]

Compute x · W_up:

x · W_up = [2.25, -6.00, -0.00]

The column dimension (d_ff) of (W_up) is larger than the input dimension (d_model). Each column of (W_up) acts like a feature detector operator, searching for a specific semantic pattern. For example, its first column can take the “biological-ness” component in the input embedding vector and make it more concrete as “pet.” At this point, the originally entangled signal is mapped into a higher-dimensional intermediate space — this is the process of disentangling (decoupling) features.

ReLU: f(z) = max(0, z)
ReLU(x · W_up) = [2.25, 0.00, -0.00]

You can see that ReLU performs a signal cutoff here: the -6.0 on the “car” dimension is discarded. This ensures that the subsequent (W_down) reconstructs the representation only based on truly useful features. This kind of nonlinear truncation allows the FFN to select genuinely meaningful logical operators from an enormous space of linear combinations.

The W_down matrix:

Pet    [1.0,  -0.8]
Car    [0.6,   1.5]
Noise  [0.2,   0.1]

Here each row corresponds to one intermediate feature: Pet, Car, Noise; and the columns map back to the original two semantic axes of the input embedding vector.

Compute ReLU(x · W_up) · W_down:

ReLU(x · W_up) · W_down = [2.25, -1.80]

Each row of (W_down) and each column of (W_up) form a paired “semantic pair,” defining how the model expresses features. When the weight produced by ReLU(x·W_up) is large, then by the nature of matrix multiplication, the corresponding row in (W_down) contributes more to the reconstructed vector:

ReLU(x · W_up) · W_down
= 2.25 * [1.0, -0.8] + 0.00 * [0.6, 1.5] + -0.00 * [0.2, 0.1]
= [2.25, -1.80]

You can see that the number of columns in (W_down) matches the number of columns in the original input embedding vector. In the end, the disentangled features are merged again and projected back into the original space. The token changes from ([1.5, -1.5]) to ([2.25, -1.80]), effectively “learning” a summary like: this token is more likely to represent something biological, and less likely to represent something mechanical.

This example also explains why we need the ReLU function. If there were no ReLU, the computation would look like this:

x · W_up = [2.25, -6.00, -0.00]
(x · W_up) · W_down = [-1.35, -10.80]

See what it “learns” now: it concludes that this token has a very small probability of being biological, and an even smaller probability of being mechanical. That contradicts what we saw in (W_up), where the “pet” score was actually quite high. Why does this happen? Because the negative value -6.0 in the “car” dimension affected the final result — almost as if the model were saying: “Since it’s very unlike a car, it’s also very unlike a cat!” Without ReLU, the “negation” on the car dimension keeps propagating downward and leads to an incorrect conclusion. ReLU effectively cuts off those connections in the matrix that “shouldn’t conduct,” ensuring that only the concepts that are activated are allowed to influence the output.

Of course, there are other activation functions that allow negative values to pass through. But I hope this example helps you feel the power of activation functions directly: without them, no matter how many matrix multiplications you stack, you still only get linear transformations. The role of an activation function is to introduce nonlinearity into high-dimensional space, enabling the model to handle complex logical relations such as XOR. ReLU’s “truncation” is essentially a nonlinear partition of the feature space, giving deep networks powerful fitting capacity beyond simple linear regression.

What we just described is the standard feed-forward sublayer process as presented in the classic paper Attention Is All You Need. Now let’s look at X’s specific algorithm: it actually uses an enhanced FFN called GeGLU (GeLU-Gated Linear Unit), which can dynamically adjust inference strength more effectively:

FFN_GeGLU(x) = [ GeLU(X * W_up_gate) * (X * W_up_value) ] * W_down

GeGLU differs from the standard FFN in two ways:

1. It uses GLU (Gated Linear Unit) to replace ReLU. Unlike a single linear transformation, GLU applies two linear transformations to the input: one is the content path (value), and the other is the gating path (gate). What is the benefit? It treats “the feature itself” and “how much of the feature to keep” as separate things. In the gating path, the model can learn — based on context — exactly how much feature to retain, rather than relying on a passive, fixed function like ReLU.
2. Its activation function is GeLU, a smooth activation function based on Gaussian distribution probability properties, which is more suitable for deep Transformers than ReLU. In the region (x < 0), it has a nonlinear curve and allows small negative values to pass through, which helps gradients flow more easily.

After one feed-forward sublayer, there will be another attention sublayer, then another feed-forward sublayer… stacked layer by layer, nested repeatedly, forming a deep neural network.

Aggregation and Output

We finally reach the last step of training the user tower. Each user has multiple historical tokens, but in the two-tower model we ultimately need a single user vector. We must compress the FFN output (T × dmodel) into (1 × dmodel). X uses masked mean pooling, averaging only meaningful history.

What counts as meaningful history?

When the input samples contain “zero-padding” — for example, if we require each user to use 3 historical time sequences for training, but a particular user only has 2 historical actions, the last dimension will be zero-padded. The vector from this padding ends up as noise after training and it NOT a meaningful history.

Can’t we just use 2 historical actions in that case? you might ask. The reason for zero-padding is that GPU parallel computation requires the input data to be a regular rectangle for efficient computation. Here is an example:

Example:

FFN output = [
  [1.38, 0.27],   # real
  [0.62, 1.26],   # real
  [-1.34,-0.44]   # padding noise
]

After mask:
[
  [1.38,0.27],
  [0.62,1.26],
  [0.00,0.00]
]

Sum = [2.00,1.53]
Output = [1.00,0.765]

This way, from a variable-length input tensor of user historical event sequences, through training, we obtain a fixed-length user feature vector.

Although this “mean pooling” concept is simple, we can think about why masked mean pooling is used instead of other aggregation methods. Keep in mind that we’re using a decoder-only architecture (like GPT), where each token’s output already contains all information from the beginning up to that token. So to represent a user’s current interest profile, the most natural approach would be to take the last token’s vector as the user’s final vector. If we average across all historical tokens, it’s like forcibly averaging “the user’s interest state from a year ago” with “the user’s current interest state.”

And this might actually be what the recommendation system values more. Because compared to a pure text prediction model, the user interest vector that a recommendation system ultimately outputs cares more about the diversity and comprehensiveness of history. When writing articles or having conversations, logical coherence is crucial — the last few words often determine what the next word is. But in recommendation systems, user behavior tends to have much more randomness. Just because I clicked on a funny video one second ago (maybe it was an accidental tap, or I just wanted to take a peek) doesn’t mean I only want to see funny videos the next second. Using average pooling avoids the token being misled by the randomness of the last token (noise resistance) and prevents other tokens’ information from being overly diluted (anti-forgetting, better diversity).

After average pooling, as a final step, we further normalize the user interest vector to get [0.794, 0.608], making the vector’s length equal to 1. This is because when ultimately computing similarity with the item tower, we only care about similarity, not magnitude.

With this, we’ve completed the training from the input user time series to the output user interest vector (user profile features).

And that concludes this article! Using the opportunity to analyze X’s new algorithm, we walked through the Transformer in detail. I’ll write an article on how this user tower is then used in the retrieval process, as well as how Transformer is used in the ranking process soon.