The Secret Lives of Attention Heads: How a Transformer Quietly Teaches Itself to Specialize

Nobody tells the heads what to do. So why do they all end up with jobs?

There’s a strange and beautiful thing that happens inside every large language model, and almost nobody notices it.

When a Transformer is born — the instant it’s initialized — all of its attention heads are identical twins. Not similar. Not “kind of alike.” Statistically interchangeable. If you peeked inside, every single head would look like the same shapeless cloud of random noise. There is no Head of Grammar. No Head of Names. No Head of Punctuation. There is nothing but undifferentiated static.

And yet, by the time training finishes, those same heads have grown into specialists. One quietly tracks subjects and verbs. Another follows names through a paragraph. Another keeps an eye on matching brackets. Another copies words from earlier in the text. They’ve developed personalities — and no one assigned them.

How does that happen? How do identical pieces of math, all chasing the exact same goal, end up dividing the labor between them like a well-run team?

That question turns out to be one of the most fascinating in modern deep learning — and the answer is genuinely lovely. Let’s walk through it slowly.

First, let’s kill a common myth

A lot of people assume the architecture forces heads to specialize. That somewhere in the Transformer’s blueprint there’s a rule like:

Head 1 handles syntax. Head 2 handles coreference. Head 3 handles position.

There isn’t. Nothing of the sort exists anywhere in the design.

Multi-head attention simply gives you several heads and lets each one learn its own way of looking at the data. That’s the entire deal. It hands out a set of identical blank slates and walks away. There is no instruction sheet, no division of roles, no hidden constraint nudging Head 1 toward grammar.

So if the architecture doesn’t cause specialization, and all the heads start out the same and chase the same objective… why on earth don’t they all just learn the exact same thing?

This is where it gets interesting.

A tale of eight engineers

Picture eight engineers hired on the same day to build a product together.

They all walked in with the same education, the same tools, and the same goal: ship something great. No manager handed out titles. Nobody said “you’re the database person, you’re the frontend person.”

Come back a year later, and something has happened. One of them has become the infrastructure wizard. One owns the database. One lives in the frontend. The roles crystallized on their own.

Why? Because specialization made the team better. The moment one person got slightly ahead on databases, it became more efficient for them to keep going deeper there while someone else covered other ground. Little early differences snowballed into distinct expertise.

A Transformer does precisely the same thing — except instead of ambition and coffee, the engine driving it is gradient descent.

Symmetry breaking: the tiny accident that changes everything

Here’s the mathematical heart of it.

At the start of training, Head 1 and Head 2 are symmetric. You could swap them entirely — copy Head 1 into Head 2’s slot and vice versa — and the network would behave almost identically. They’re mirror images doing the same job. In physics and math, this kind of “you can swap these and nothing changes” property is called a symmetry.

But here’s the crack in the mirror: random initialization is never perfectly identical. The heads start out as random noise, and random noise has tiny, meaningless differences baked in. Maybe, purely by chance, Head 1 begins life a hair more sensitive to relationships between nouns than Head 2 is.

That difference is microscopic. It means nothing on its own. But it’s enough.

Because Head 1 is now slightly useful for noun relationships, the gradients — the forces that adjust the model to reduce its errors — start pushing Head 1 a little further in that direction. Which makes it more useful for nouns. Which makes the gradients push it even harder. Which makes it more useful still.

A positive feedback loop ignites. A meaningless random fluctuation gets amplified into a genuine specialty.

This is called symmetry breaking, and it shows up everywhere in nature and in deep learning. A perfectly balanced pencil standing on its tip is symmetric — it could fall in any direction. But it will fall, and the instant it tips even a hair one way, that direction wins completely. The symmetry doesn’t survive contact with reality. Neither does the sameness of attention heads.

What specialization actually looks like: many worlds, one word

To really understand what each head becomes, you have to think geometrically — and this is the part I find genuinely beautiful.

Every head has its own set of learned projections (the famous WQ, WK, and WV matrices). In plain terms: each head learns its own private way of re-drawing the map of word meanings. Each head builds its own coordinate system, its own notion of what makes two words “close.”

So take a single token — say, the word Apple — sitting in a 768-dimensional space.

In one head’s geometry, “Apple” lands right next to Microsoft, Google, and Amazon. To that head, Apple means big tech company.

In another head’s geometry, the same word “Apple” sits near CEO, company, revenue — that head sees Apple in terms of corporate concepts.

In yet another head, “Apple” is just cozying up to whatever word came right before it and right after it — that head doesn’t care about meaning at all; it cares about position.

Same word. Same input. But living simultaneously in multiple geometries, each with a completely different idea of what “similar” means.

This is one of the deepest ideas in the whole architecture: a Transformer doesn’t build one representation of language. It builds many parallel coordinate systems at once, and lets each one specialize in a different kind of relationship.

A subtle but crucial point: it’s the projections, not the softmax

It’s tempting to think the specialization lives in the attention pattern — the famous softmax(QKᵀ) distribution that decides which words pay attention to which.

It doesn’t. Not directly.

The attention distribution is a consequence, not a cause. The real source of a head’s identity is its learned projections — its private geometry. The causal chain runs like this:

Different WQ / WK / WV

↓

Different geometry

↓

Different notions of similarity

↓

Different attention patterns

↓

Different jobs

So, when you watch a head light up beautifully on, say, matching quotation marks, you’re watching the last link in that chain. The actual decision — the personality — was forged upstream, in the geometry the head learned. The SoftMax just faithfully reports what the geometry already decided.

Why training actively pushes heads apart

So far we’ve explained how specialization can start (symmetry breaking) and what it looks like (parallel geometries). But there’s still a force pulling all of this forward: the relentless pressure of training to reduce loss.

Imagine two heads that both ended up tracking syntax. Identical jobs. From the model’s point of view, the second one is dead weight — pure redundancy. The model spent precious capacity learning the same trick twice and got nothing extra for it.

Loss doesn’t reward that. Loss rewards capturing new useful information. A head that goes off and discovers something nobody else is doing — subject-verb agreement, or tracking who “he” refers to, or noticing indentation structure in code — reduces the error more than a copycat ever could. So gradients quietly favor the heads that branch out.

If this reminds you of Principal Component Analysis (PCA), you’ve got good instincts. In PCA, the first component grabs the biggest chunk of variation in the data. The second component is forced to capture something different — because capturing the same thing again would be pointless. Each component earns its place by being unlike the others.

Attention heads feel the same pressure. Diversity is never explicitly rewarded. There’s no line in the loss function that says “be different.” Heads diversify simply because being different is what lowers the loss. Usefulness and novelty turn out to be the same thing here, and the optimizer chases usefulness without ever knowing it’s creating a team of specialists.

An information-theoretic way to see it

Here’s another lens that makes the whole thing click.

Think of each head as contributing some information gain toward predicting the next word. If two heads learn the same pattern, they share a lot of mutual information — they’re telling the model the same thing twice, and the second telling adds almost nothing.

A head that finds a genuinely new predictive signal — coreference, quotation structure, code indentation, agreement between subject and verb — adds information the model didn’t have. That’s worth more. That cuts the loss harder.

So the system drifts, all on its own, toward a set of heads that each carry distinct, non-overlapping information. Not by design. By the plain arithmetic of what reduces error.

The proof: cracking open real models

This isn’t just a nice story. Researchers in the field of mechanistic interpretability have literally pried open trained models — GPT-2 among them, along with models studied inside Anthropic — and gone looking for these specialists. They found them. With names.

Induction heads. These spot a pattern that already happened and continue it. If the model has seen A B C earlier, and now sees A B, an induction head predicts C. It's the circuit behind a model's uncanny ability to pick up on and repeat structure it just encountered — a foundational building block of in-context learning.

Name mover heads. These track names through a passage. If “John” was just introduced and the sentence needs the right name again, these heads make sure John shows up where John belongs.

Copy heads. These reach back into earlier context and bring forward information verbatim — useful any time the right answer is to repeat something already said.

Delimiter heads. These obsess over matching pairs — parentheses, braces, brackets — keeping track of what opened and what still needs to close.

Nobody coded any of these. No engineer wrote a “delimiter head” module. They emerged — grown entirely by gradient descent out of that original field of random noise. We didn’t build these circuits; we discovered them, the way a biologist discovers an organ by dissecting an animal that evolution already assembled.

Not every head makes it: the ones that fade

Now for the bittersweet part, because the story isn’t all triumph.

Specialization is not guaranteed. Some heads never find a useful niche. They drift through training and end up nearly inactive — or worse, redundant, doing a watered-down version of what some other head already does better.

We know this because researchers have pruned heads — surgically deleted them from large trained models — and watched the performance barely flinch. For a meaningful fraction of heads, you can simply remove them and the model carries on almost unchanged.

That’s a striking thing to sit with. It tells us specialization is an emergent outcome, not a built-in promise. The architecture opens the door to specialization; the training process makes specialization rewarding; but whether any particular head walks through that door is left to the messy dynamics of optimization. Some heads find their calling. Some never do. Just like our eight engineers — except a few of them, it turns out, never quite figured out what they were for.

The deepest way to see it

Let me try to compress the whole idea into a single image.

A Transformer layer is not learning one representation of your text. It’s learning many parallel coordinate systems over the same words — a whole stack of different lenses, each redrawing the map of meaning in its own way. Training is the process of asking, over and over:

Which of these geometries actually helps predict what comes next?

The geometries that help, survive and sharpen. The geometries that don’t, fade into redundancy and get pruned away. What’s left is a collection of finely-tuned lenses, each excellent at one specific kind of relationship.

This is why modern interpretability researchers have stopped thinking of an attention head as “just a mechanism for weighting words.” They’ve started thinking of each head as something richer:

A learned retrieval circuit operating inside its own private semantic space.

A little specialist. With its own worldview. That taught itself the job.

And this is only half the story

Here’s the cliffhanger.

Everything we’ve talked about — all this gorgeous specialization — is about attention. And attention answers exactly one question:

Which information should I go fetch?

That’s retrieval. It’s about locating and pulling together the right pieces from across the text. But once you’ve gathered the right ingredients, a completely different problem is still sitting there, untouched:

Now what? How do I actually transform and compute with what I just retrieved?

Attention doesn’t do that part. A lot of engineers assume it does both — that attention is where all the thinking happens — but it isn’t. Retrieval and computation are two genuinely different jobs, and the Transformer splits them on purpose. The second job belongs to a different component entirely: the Feed-Forward Network, the unsung place where an enormous share of the model’s actual computation — and its stored knowledge — quietly lives.

The separation between retrieval (attention) and computation (FFN) is one of the most important conceptual splits in the entire architecture. And it’s exactly where we’ll go next.

Because if attention is how a Transformer decides what to look at, the Feed-Forward Network is how it decides what to think.

If this gave you a new way of seeing what’s happening inside these models, the FFN deserves the same kind of close look — and it tells an equally surprising story about where a model’s knowledge actually lives.