Attention Is All You Need

A comprehensive breakdown of the Transformer architecture — the research paper that changed everything in artificial intelligence.

The Revolution

In 2017, eight Google Brain and Google Research engineers submitted a paper to NeurIPS with a deceptively simple title: “Attention Is All You Need.” The paper introduced the Transformer — an architecture built entirely on attention mechanisms, discarding recurrence and convolution entirely.

The biggest benefit comes from how the Transformer lends itself to parallelization — making it possible to train on unprecedented scales of data.

Within a few years, the Transformer became the backbone of virtually every significant AI system: BERT, GPT, T5, PaLM, and ultimately the large language models powering the AI revolution today. Understanding this architecture is not merely academic — it is foundational to understanding modern AI.

Paper Reference

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, 30.

The Problem Before Transformers

Before 2017, the dominant approaches to sequence modeling were Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs). These models process tokens one at a time, maintaining a hidden state that carries information forward through the sequence.

The Core Problems of RNNs

Sequential processing meant training could not be parallelized. On long sequences, RNNs notoriously suffered from the vanishing gradient problem — distant relationships in a sentence were effectively forgotten. LSTMs mitigated this but did not eliminate it.

Even seq2seq models with attention (Bahdanau et al., 2015) still relied on RNN encoders and decoders as their spine. The Transformer paper asked: what if attention was not supplementary — but the entire architecture?

The Architecture: A High-Level View

At the highest level, the Transformer is an encoder-decoder architecture. Given an input sequence (say, a French sentence), the encoder reads and represents it, and the decoder generates an output sequence (the English translation) one token at a time.

Figure: The complete Transformer architecture. The encoder (left) produces a rich representation; the decoder (right) attends to it while generating output.

The original paper stacks 6 encoders and 6 decoders, though this number is a hyperparameter. All encoders are structurally identical but do not share weights. The same applies to decoders.

Key Insight

Unlike RNNs, all positions in the encoder process simultaneously. This is the core reason the Transformer can be trained on modern GPU hardware at previously impossible scales.

Input Embeddings

A neural network cannot natively understand words — it operates on numbers. Every token in the input sentence is first converted into a dense vector called an embedding.

In the Transformer, each token is embedded into a vector of 512 dimensions (denoted d_model = 512 in the paper). These embeddings are learned during training, capturing semantic relationships between words — so “king” and “queen” end up near each other in this high-dimensional space.

Importantly, embedding only happens once — at the bottom of the encoder stack. All subsequent encoder layers receive the 512-dimensional output of the layer below them.

Positional Encoding

Here we arrive at the Transformer’s first clever trick. Because all tokens are processed in parallel — not sequentially — the model has no inherent sense of word order. “The cat sat on the mat” and “The mat sat on the cat” would be identical to a naive Transformer.

The solution: inject positional information directly into the embeddings. Before entering the encoder, each embedding vector has a positional encoding vector added to it — a unique signature for each position in the sequence.

Even dimensions use sine, odd dimensions use cosine, each at a different frequency. This gives each position a unique pattern of values across the 512 dimensions. The sinusoidal pattern has a beautiful property: relative positions can be computed via linear transformations, so the model learns to exploit positional relationships naturally.

Why Sinusoids?

The sinusoidal approach allows the model to generalize to sequence lengths longer than those seen during training — a fixed positional embedding table cannot do this.

Self-Attention: The Core Mechanism

This is the beating heart of the Transformer. Self-attention allows each word in the sequence to look at every other word and decide how much attention to pay to each — all in a single, parallelizable operation.

Consider the sentence: “The animal didn’t cross the street because it was too tired.” When encoding “it”, the model needs to understand that “it” refers to “animal”, not “street”. Self-attention makes this possible by letting “it” attend heavily to “animal”.

Query, Key & Value Vectors

For each token, self-attention creates three learned projections, each of 64 dimensions (d_k = 64):

These vectors are produced by multiplying the token’s embedding by three separate learned weight matrices: W_Q, W_K, and W_V, each of shape (512 × 64).

Scaled Dot-Product Attention

With Q, K, V in hand, attention is computed in four steps:

1.Score

Dot product of the query with every key: Q · Kᵀ. High dot product = high relevance between tokens.

2.Scale

Divide by √d_k (= √64 = 8). Prevents the dot products from growing too large, which would push softmax into regions with tiny gradients.

3.Softmax

Apply softmax to normalize scores into a probability distribution (all positive, summing to 1). This is the attention weight.

4.Aggregate

Multiply each value vector by its attention weight, then sum. The result is a context-rich representation of the current token.

In matrix form, all tokens are processed simultaneously. The input matrix X is multiplied by the three weight matrices to produce Q, K, and V all at once — making the operation massively parallelizable on modern hardware.

Multi-Head Attention

A single self-attention operation captures one type of relationship at a time. But language is rich with multiple simultaneous dependencies — a word might relate to its grammatical subject, its antecedent pronoun, and a nearby modifier all at once.

The paper’s solution: run 8 independent attention heads in parallel, each with its own learned Q, K, V projection matrices. Each head can specialize in a different type of relationship.

Each head operates in a 64-dimensional subspace (512 / 8 = 64), so the total computation remains constant. After all 8 heads compute their outputs (Z₁…Z₈), they are concatenated into a single matrix and multiplied by a learned output weight matrix W_O to produce the final multi-head output.

Empirical Finding

Visualization studies show different attention heads genuinely specialize: one might track syntactic dependencies, another coreference resolution, another positional proximity. This emergent specialization is learned — not programmed.

The Encoder Block

Each of the 6 encoder layers is structurally identical, composed of two sub-layers with a critical design pattern wrapping each:

Sub-layer 1: Multi-Head Self-Attention

All input tokens attend to one another simultaneously. The output is a rich, context-aware representation of each token — “animal” now has information about “tired” and “cross” baked in.

Sub-layer 2: Position-wise Feed-Forward Network

A two-layer fully connected network applied independently to each position. In the paper, this is: Linear(512 → 2048) → ReLU → Linear(2048 → 512). This step transforms the attention-mixed representations into richer features.

Add & Normalize (Residual Connections)

Both sub-layers are wrapped in a residual connection followed by Layer Normalization:

Residual connections (borrowed from ResNets) allow gradients to flow directly through deep stacks without vanishing. Layer Normalization stabilizes training by normalizing activations within each layer. Together, they make deep Transformers trainable.

The Decoder Block

The decoder is structurally similar to the encoder but with a crucial addition — it has three sub-layers:

1.Masked Self-Attention

Like encoder self-attention, but future positions are masked to −∞ before softmax. During training, this prevents the decoder from “cheating” by looking ahead at the answer.

2.Cross-Attention

Queries come from the decoder; Keys and Values come from the encoder output. This is how the decoder “reads” the encoded input — attention bridges the two stacks.

3.Feed-Forward Network

Identical in structure to the encoder’s FFN — position-wise transformation of the combined representations.

The decoder generates output autoregressively: it produces one token at a time, feeding each generated token back as input for the next step. At inference time, generation continues until a special <EOS> (end-of-sequence) token is produced.

Key Asymmetry

During training, the decoder can process all target tokens in parallel thanks to teacher forcing (feeding the correct previous tokens). At inference, it must generate sequentially — but the encoder only runs once per input.

The Final Output Layer

The decoder stack outputs a 512-dimensional vector for each generated position. To turn this into a predicted word, two final layers are applied:

Linear Projection

A fully connected layer projects the 512-dim vector up to the full vocabulary size — a vector of logits, one per possible output word. If the vocabulary has 37,000 tokens, the output is a 37,000-dimensional logit vector.

Softmax

Softmax converts the logits into a probability distribution over all vocabulary tokens. The token with the highest probability is selected as the model’s prediction for that position.

Training the Transformer

The Transformer is trained end-to-end via standard backpropagation with the Adam optimizer. The loss function is cross-entropy between the predicted probability distribution and the one-hot true label.

Learning Rate Schedule

The paper introduced a custom learning rate warmup schedule — rates increase linearly for the first 4,000 steps, then decay proportionally to the inverse square root of the step number. This warmup was critical for stable training.

Regularization

The paper applied dropout (rate 0.1) to the output of each sub-layer (before the residual add), to attention weights, and to the embedding sums. Additionally, label smoothing (ε = 0.1) replaced hard one-hot targets with soft distributions, improving calibration even at the cost of perplexity.

Results & Benchmarks

On the WMT 2014 English-to-German translation task, the Transformer Big achieved 28.4 BLEU — more than 2 BLEU points above the previous state-of-the-art (an ensemble of models), at a fraction of the training cost.

The Transformer also demonstrated superior generalization: trained exclusively on translation, it transferred effectively to constituency parsing — outperforming task-specific models with minimal modification.

The Legacy: What Came After

The Transformer did not merely advance the state of the art — it restructured the entire research landscape of AI. Every major model of the subsequent era is either a direct Transformer variant or built on its attention mechanisms.

1.BERT (2018)

Encoder-only Transformer pretrained bidirectionally on masked language modeling. Dominated NLU tasks for years.

2.GPT Series

Decoder-only Transformers trained on next-token prediction. GPT-3 (175B params) demonstrated emergent few-shot abilities.

3.T5 (2020)

Text-to-Text Transfer Transformer. Unified all NLP tasks into a single seq2seq format, exploring the full encoder-decoder design.

4.Vision Transformer

Applied the Transformer to image patches (2020), achieving competitive performance with CNNs — proving attention is architecture-agnostic.

The Transformer is not just a model — it is the operating system of modern AI. Everything from ChatGPT to AlphaFold, from image generation to code synthesis, runs on its foundations.

Modern Innovations Beyond the Original Paper

The original architecture has been extended in numerous ways: Rotary Positional Embeddings (RoPE) replace sinusoidal encodings for better length extrapolation. Multi-Query Attention reduces memory bandwidth at inference. Flash Attention rewrites the attention computation for IO efficiency. Grouped Query Attention balances the tradeoffs. The research community has spent seven years optimizing, scaling, and reinventing this architecture — yet the core scaled dot-product attention equation remains unchanged.

The Paper’s Own Words

The authors concluded: “We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text.” — A remarkably understated prediction of what followed.

References

1. Attention Is All You Need (Original Paper) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS 2017).
2. NeurIPS Official Publication: Attention Is All You Need Official conference publication of the Transformer architecture paper.
3. Ashish Vaswani et al. (2017). Introduced the Transformer architecture based entirely on self-attention mechanisms.
4. The Illustrated Transformer by Jay Alammar One of the most widely used visual explanations of the Transformer architecture.
5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Introduced BERT, an encoder-only Transformer model.
6. Language Models are Unsupervised Multitask Learners (GPT-2) Demonstrated the power of large-scale decoder-only Transformers.
7. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) Raffel, C. et al. (2019). Unified NLP tasks into a text-to-text framework.
8. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Vision Transformer) Dosovitskiy, A. et al. (2020). Extended Transformers to computer vision.