Ilya’s Papers, Part 13: The Dawn of Attention

Ilya’s Papers, Part 12: The Dawn of Attention

It was a quieter time, a simpler time. Dinosaurs roamed the earth. “Uptown Funk” by Bruno Mars was at the top of the charts, and no one had heard of ChatGPT. I am referring, of course, to the ancient year of 2015, which is basically prehistory in AI.

Transformers did not exist yet. “Attention Is All You Need” was still in the future. But even before transformers, researchers had already started exploring an important idea: attention.

I have written about RNNs before, and this post will make a lot more sense if you already know the basics. Still, I’ll try to explain this one as painlessly as possible, because RNN papers have a special talent for making people sad.

The basic problem

Imagine you want an AI system to translate a sentence from one language into another.

Back then, a common approach was this:

• One RNN reads the whole sentence in the source language

• It compresses everything it learned into one final hidden state vector

• A second RNN uses that vector to generate the translated sentence

In simple terms, the first RNN reads the sentence and hands the second RNN a single bundle of notes.

That sounds fine… until the sentence gets long.

Because now the entire meaning of the sentence has to fit into one vector. That is a lot to ask. It is like hearing a ten-minute lecture and being told, “Great, now summarize all of that in one sticky note.”

Sometimes the sticky note is not enough.

The big idea of attention

The solution in this paper is simple and powerful:

Instead of forcing the encoder to squeeze the whole sentence into one final vector, we save the encoder’s hidden states from every step.

So if the encoder reads a three-word sentence, we keep three hidden state vectors, one for each word position.

And because this is a bidirectional encoder, each hidden state is not just about one word alone. Each one also contains some information about the surrounding words, because the sentence is processed both left-to-right and right-to-left.

So now, instead of giving the decoder one tiny sticky note, we give it access to a whole stack of notes.

Much better.

What the decoder uses

At each step, the decoder uses three things:

1. the most recently generated word

2. the decoder’s current hidden state

3. a context vector, which we can think of as the attention vector

That attention vector is the new ingredient.

Its job is to tell the decoder, “Here is the most relevant information from the input sentence for what you are trying to write right now.”

How the attention vector is built

Here is the core idea.

At a given decoding step, we take the decoder’s current hidden state and compare it to each encoder hidden state we saved earlier.

For each encoder hidden state, we ask:

How relevant is this encoder note to what the decoder is trying to produce right now?

A small neural network computes a score for each one.

So if we saved three encoder hidden states, we get three scores.

We put the current hidden decoder state vector and one of the encoder’s state vectors into the small scoring network to get a score about how relevant that encoder vector is to the decoder vector. It’s like the decoder is saying, I want to translate the word “dog”. Are you the word “dog”?

For example:

• encoder hidden state 1 gets score 2

• encoder hidden state 2 gets score 5

• encoder hidden state 3 gets score 3

The total score is 10.

So we turn those into weights:

• 20% for hidden state 1

• 50% for hidden state 2

• 30% for hidden state 3

Then we create a weighted combination of those encoder hidden states.

That weighted combination becomes the attention vector.

And that vector is passed into the decoder along with the decoder’s hidden state and the previous output word.

An intuition for what is happening

You can think of it like this.

A student is taking a math test. He has a backpack full of notes from different classes.

He looks at his history notes. Not very helpful right now. Low score.

He looks at his math notes. Very helpful. High score.

He looks at some science notes. A little helpful. Medium score.

Instead of picking just one notebook, he builds a quick mental summary weighted toward the most relevant notes.

That summary is what the attention vector is doing.

The decoder is basically saying:

“I am trying to generate the next word. Which parts of the input sentence matter most right now?”

Why this was a big deal

Without attention, the decoder only gets one final encoder vector, which may lose important details.

With attention, the decoder can look back at all the encoder hidden states and focus more on the useful ones for the current step.

That makes translation much easier, especially for longer sentences.

It is no longer relying on one overworked summary vector that is doing the computational equivalent of carrying a piano up the stairs alone.

Why this matters for transformers

If this sounds a little like transformers, that is because it is pointing in the same direction.

Transformers also compute relevance scores, use those scores as weights, and build weighted combinations of vectors.

The details are different, but the basic instinct is similar:

look at all the available context, decide what matters most, and build a useful summary from it.

That is why this paper matters. It helped introduce the core idea that later became central to modern AI.

Thank you for reading, and I will see you in Number 14!