The Transformer: A Beginner’s Deep Dive Into the Architecture That Changed AI Forever

*Now let’s zoom out and see how self-attention embeddings, Query/Key/Value vectors, scaled dot products the pieces fit into the full Transformer architecture that powers GPT, BERT, and every large language model you’ve heard of.**

The Problem: Why We Needed Something New

Before the Transformer, the dominant tools for language tasks were Recurrent Neural Networks (RNNs) specifically LSTMs and GRUs. The idea was intuitive: read a sentence word by word, left to right, carrying a “memory” (a hidden state) forward at each step.

This worked reasonably well for short sentences. But it had two fatal flaws.

1. Vanishing gradients. When you train an RNN on a long sentence, the signal used to update the network’s weights has to travel back through every single time step. By the time it reaches the beginning of the sentence, it’s practically zero. The network simply forgets what happened early on. Imagine trying to remember the subject of a sentence by the time you’ve read a paragraph your brain is fine with it, but an RNN struggles badly.

2. No parallelism. Because RNNs process words one at a time (step 2 depends on step 1, step 3 depends on step 2…), you can’t parallelize training across a long sequence. This made RNNs brutally slow to train at scale.

There was also a deeper problem with fixed word embeddings. The word “apple” has one vector in traditional embedding spaces whether you’re talking about the fruit or the tech company. Language is context-dependent, and older approaches couldn’t capture that.

The Transformer, introduced in the 2017 paper ”Attention Is All You Need” by Vaswani et al. at Google, solved all three problems at once.

The Transformer — model architecture

Self-Attention: Every Word Looks at Every Other Word

Self-attention is the heart of the Transformer. Let’s build intuition from the ground up.

Take the sentence: ”I bought apple to eat.”

When the model processes the word apple, it needs to figure out which meaning applies. It does this by comparing apple to every other word in the sentence and deciding which words are most relevant.

Here’s how it works mechanically:

Query, Key, and Value Vectors

Each word’s embedding is projected into three separate vectors using learned weight matrices:

• Query (Q): “What am I looking for?” this is the word asking a question.

• Key (K): “What do I represent?” every word announces itself.

• Value (V): “What information do I carry?” the actual content to pass along.

Think of it like a library search system. Your Query is the search term you type in. Each book has a Key a title and description that gets compared to your query. The closer the match, the higher the relevance score. The Value is the actual content of the book you retrieve.

Scaled Dot-Product Attention

The relevance score between a query word and a key word is just their dot product a single number measuring how similar two vectors are. We compute this for every pair of words in the sentence, giving us a matrix of raw scores.

Then two things happen:

1. We scale the scores by dividing by √dₖ (the square root of the key vector dimension). This prevents scores from growing so large that the softmax function later would saturate and produce near-zero gradients.

2. We apply softmax to turn the scores into probabilities that sum to 1.

These softmax weights tell us: how much should this word attend to each other word?

Finally, we take a weighted sum of the Value vectors, weighted by those attention scores. The result is a new, context-enriched representation for the word.

The formula is elegant:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

For ”apple” in our sentence, the attention weights might land heavily on ”eat”, signaling that this is food-related not a tech product. This is context-dependence, solved.

Multi-Head Attention: Eight Perspectives Are Better Than One

Running attention once gives you one lens on the sentence. But language has multiple types of relationships simultaneously:

• Syntactic: which word is the subject?

• Semantic: which words share meaning?

• Coreference: what does “it” refer to?

• Positional: what follows what?

The Transformer runs attention 8 times in parallel, each time with different learned weight matrices for Q, K, and V. Each of these is called an attention head.

Think of it like eight specialists examining the same sentence, each trained to notice different things. One head might focus on grammatical dependencies; another on long-range semantic connections; another on adjacent words.

The 8 outputs (each a vector) are concatenated and passed through one final linear projection to produce the combined result.

In the original paper, with a model dimension of 512, each of the 8 heads works in a subspace of dimension 64 (512 ÷ 8 = 64). The total computation stays the same as single-head attention you get richer representations for free.

Positional Encoding: Teaching the Model About Order

Here’s a subtle but critical problem: the attention mechanism we just described is permutation-invariant. Feed it “dog bites man” or “man bites dog” the same words and without additional information, it produces the same result. Word order doesn’t matter to raw attention.

But order does matter in language. Enormously.

The Transformer’s solution: inject position information directly into the input embeddings before they enter the network.

For each position pos in the sequence and each dimension i of the embedding:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))

PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

This might look mysterious, but the intuition is beautiful. Think of it like a clock with many hands, each rotating at a different speed. Short-wavelength sinusoids capture fine-grained, nearby position differences. Long-wavelength sinusoids capture coarse, large-scale position structure.

The key property: for any fixed offset k, the positional encoding of position pos + k can be expressed as a linear transformation of the encoding at pos. This means the model can easily learn to attend by relative position not just absolute position which is exactly what grammar requires.

These positional encodings are added element-wise to the word embeddings before any processing begins.

The Encoder: Building a Rich Understanding

The encoder’s job is to take the input sequence say, an English sentence and produce a deeply contextualised representation of it.

It’s composed of 6 identical layers stacked on top of each other. Each layer has two sub-layers:

1. Multi-Head Self-Attention every token attends to every other token in the input.

2. Position-wise Feed-Forward Network a two-layer MLP applied independently to each position, with a hidden dimension of 2048 and a ReLU activation in between.

Each sub-layer is wrapped with two stabilising mechanisms we’ll discuss shortly: a residual connection and layer normalisation.

As information flows through all 6 encoder layers, representations become progressively richer. Early layers might capture surface syntax; deeper layers start encoding abstract semantic relationships. By the time we exit the encoder, each token’s vector reflects its meaning in the full context of the sentence.

The Decoder: Generating the Output

The decoder generates the target sequence say, an Italian translation one token at a time.

It’s also a stack of 6 identical layers, but each layer has three sub-layers instead of two:

1. Masked Multi-Head Self-Attention the decoder attends to the tokens it has already generated. The “masked” part is crucial: it prevents any token from attending to future positions. During training, we feed the whole target sentence at once, but we mask out future tokens so the model can’t cheat by looking ahead.

2. Cross-Attention (Encoder-Decoder Attention) this is where the magic of translation happens. The decoder’s queries come from its own previous layer, but the keys and values come from the encoder’s output. This allows every position in the decoder to attend to every position in the encoder so when generating the Italian word for “apple”, the decoder can directly look at the full English context.

3. Position-wise Feed-Forward Network same as in the encoder.

Cross-attention is the bridge between the two languages. It lets the decoder ask: “Given everything the encoder understood about the English sentence, what should I generate next in Italian?”

## Residual Connections and Layer Normalisation: The Stabilisers

With 6 layers deep and gradients needing to flow backwards through all of them during training things can go wrong. Gradients can explode or vanish. Training becomes unstable.

Two techniques keep things smooth:

### Residual Connections

After each sub-layer, the original input is added back to the sub-layer’s output:

output = LayerNorm(x + Sublayer(x))

This is the same idea as in ResNet (residual networks for vision). The intuition: even if the sub-layer learns nothing useful, the signal can still flow through unchanged via the shortcut. Gradients have a direct path back through the addition, preventing vanishing. It’s like having a highway alongside a winding road traffic can always take the fast route if needed.

### Layer Normalisation

After the residual addition, the values are normalised across the feature dimension shifted and scaled so their mean is 0 and variance is 1 (with learned parameters to allow the model to rescale as needed).

This prevents the activations from growing uncontrollably large or small as they pass through deep networks, keeping training stable and fast.

Together, residual connections + layer norm are what make training a 6-layer (or 96-layer) Transformer possible at all.

## The Full Picture: English to Italian

Let’s walk through a complete translation of ”I bought apple to eat” → ”Ho comprato una mela da mangiare”.

Step 1 Tokenise and Embed. The English sentence is split into tokens and converted to 512-dimensional vectors via learned embeddings.

Step 2 Add Positional Encodings. Sinusoidal signals are added to each embedding, giving the model a sense of word order.

Step 3 Encode. The enriched embeddings pass through all 6 encoder layers. Multi-head self-attention lets every English word look at every other. After 6 layers, each token carries a deep, context-aware representation. “Apple” now strongly encodes its food-related meaning, not its tech meaning, because the attention mechanism spotted “eat.”

Step 4 Begin Decoding. The decoder starts with a special <START> token. It attends to what it has generated so far (masked self-attention), then cross-attends to the full encoder output, then passes through the feed-forward network. A final linear layer and softmax produce a probability distribution over the entire vocabulary.

Step 5 Generate Token by Token. The highest-probability token is chosen say, “Ho”. This gets appended to the decoder’s input, and the whole decoder runs again to produce “comprato”, then “una”, then “mela”, and so on, until a <END> token is generated.

Step 6 Output. “Ho comprato una mela da mangiare.” ✓

## Why This Was Revolutionary

The Transformer’s advantages over RNNs were immediate and dramatic:

• Parallelism. Because self-attention processes all positions simultaneously (not sequentially), the entire training computation can run in parallel on GPUs. Training that would have taken weeks with RNNs took hours.

• Long-range dependencies. Every word can directly attend to every other word in a single step, regardless of distance. No more vanishing gradient problem across long sentences.

• Scalability. Stack more layers, add more heads, widen the model dimension performance keeps improving. This scaling property is what eventually gave us GPT-3, GPT-4, and their cousins.

The original Transformer achieved a BLEU score of 28.4 on English-to-German translation surpassing all prior models, including ensembles, in a fraction of the training time.

## Closing Thought

Every large language model you interact with today ChatGPT, Claude, Gemini, LLaMA is built on some version of the architecture described in this article. The specific details vary: some are encoder-only (BERT), some are decoder-only (GPT), some use different normalisation strategies or positional encodings. But the core idea replace recurrence with self-attention, stack it deep, stabilise with residuals and LayerNorm has remained the foundation.

Understanding the Transformer isn’t just academic. It’s understanding the engine underneath modern AI.

Further reading: “Attention Is All You Need” (Vaswani et al., 2017) the original paper is remarkably readable and freely available on arXiv.