Birth of Transformers(Part 4)Attention: The Idea That Changed AI Forever

TL;DR

In the previous blog, we saw how Encoder-Decoder models improved machine translation by compressing an entire input sentence into a single context vector. While this worked for short sentences, it struggled with longer ones because important information was often lost in the bottleneck.

This article introduced the breakthrough that changed sequence modeling: the Attention mechanism.

Instead of forcing the Decoder to rely on one fixed summary of the whole sentence, Attention lets it look back at the Encoder’s hidden states whenever it needs relevant information. We built this idea from first principles and then explored Bahdanau Attention, where the Decoder computes alignment scores for each source word, converts them into attention weights using Softmax, and uses those weights to form a dynamic context vector.

Through mathematical derivations, numerical examples, and translation-based visualizations, we saw that Attention works like a soft lookup mechanism. Rather than storing everything in one vector, the model retrieves the most relevant information for each output word as it is generated. Although this greatly improved machine translation and solved the bottleneck problem, one limitation still remained: the Encoder and Decoder were still based on recurrent neural networks, which made computation sequential and difficult to parallelize.

That leads to the next big question:

If Attention is doing most of the work, do we still need recurrent neural networks at all?

The answer led to one of the most influential papers in AI history: Attention Is All You Need.

In the previous article, we explored the Encoder-Decoder architecture and the challenge of compressing an entire sentence into one vector. The Encoder reads the source sentence and turns it into a single context vector. The Decoder then uses that vector to generate the translation one word at a time. This was a major breakthrough — it removed the need for hand-built translation rules and allowed neural networks to learn translations directly from sentence pairs. But the design had a serious flaw: all the information in the input sentence had to pass through one fixed-size vector. For short sentences, this worked well. For longer ones, important details were lost.

A Quick Journey Back to Machine Translation

Let’s return to the task we explored in the previous article. Suppose we want to translate the English sentence:

The cat sat on the mat.

into French:

Le chat était assis sur le tapis.

The Encoder-Decoder architecture solved this by dividing the task into two parts. The Encoder reads the entire English sentence and gradually builds an internal representation of its meaning. Once it reaches the final word, it produces a single vector called the context vector. The Decoder then receives this context vector and generates the French sentence one word at a time.

Think carefully about what the Encoder is actually doing. It doesn’t send individual words to the Decoder. It doesn’t allow the Decoder to revisit the original sentence. After processing the final word, it produces exactly one vector — and that single vector is expected to contain everything the Decoder will ever need. Every noun, every verb, every grammatical relationship, every piece of meaning in the sentence must survive inside that one fixed-size numerical representation.

Fig. : Original Encoder Decoder Pipeline

For short sentences, this approximation worked surprisingly well. For longer ones, it began to break down.

The Bottleneck Gets Worse

Consider these two sentences. The first is simple:

The cat sat on the mat.

The Encoder only needs to capture a small amount of information. Now consider something more realistic:

The cat that had been rescued from a shelter several years ago and had spent most of its life living with an elderly couple in a quiet village near the coast sat peacefully on the mat beside the fireplace while watching the children play.

This sentence contains far more information — additional descriptions, multiple clauses, relationships between several different ideas. Yet despite this increase in complexity, the Encoder still produces exactly the same thing: one context vector. The amount of information has grown enormously, but the size of the communication channel has not. This is why researchers began referring to it as the information bottleneck.

Fig. : Information Bottleneck

LSTMs and GRUs had dramatically improved memory, preserving information across much longer sequences than traditional RNNs. But memory alone was no longer the limiting factor. The Decoder still depended entirely on whatever information survived inside the context vector. If a detail was lost during compression, the Decoder had no way to recover it — because once the Encoder finished reading the sentence, the original words were simply gone. The Decoder had no access to them.

A Question That Changed Artificial Intelligence

Eventually, someone asked a question so simple that it almost seems obvious today.

Instead of forcing the Decoder to remember everything, why can’t it simply look back?

The Encoder had already processed every word. The information already existed. Why compress the entire sentence into one vector when the Decoder could revisit the original sentence whenever it needed more context? Suppose the Decoder is generating the French word corresponding to cat. Wouldn’t it be helpful if it could look directly at the English word cat? And when translating mat, shouldn’t it be able to focus on mat instead of relying on a compressed memory created several steps earlier?

This single idea challenged one of the fundamental assumptions behind sequence modeling. Instead of asking neural networks to remember everything, researchers began exploring a completely different philosophy: perhaps the answer wasn’t better memory, but the ability to look back.

The Revolutionary Idea — Looking Back Instead of Remembering

Imagine you’re taking an open-book exam. The examiner asks about Newton’s Second Law. In the first scenario, you read the textbook once before the exam, and then the book is taken away — every answer must come from memory. In the second, the textbook remains on your desk and you can verify anything you’re unsure about at any moment. The answer to which situation you’d prefer is obvious. If you have access to the original source of information, there is no reason to rely entirely on memory.

Yet for many years, this was exactly how neural machine translation systems operated. The Encoder read an entire sentence, then closed the book. Researchers had spent years building better memory through RNNs, LSTMs, and GRUs. But perhaps they were solving the wrong problem. The Decoder didn’t need better memory — it needed permission to look back. That single shift in perspective would completely change the future of artificial intelligence.

From Memory Retrieval to Information Retrieval

To appreciate why this idea was so powerful, let’s revisit our translation example. An Encoder-Decoder model processes the English sentence and converts it into a single context vector. The Decoder begins generating the French translation and predicts Le as the first word. Now it needs to generate the next word — ideally chat. In the original architecture, the Decoder has only one source of knowledge: the context vector. It cannot inspect individual words, cannot revisit the sentence, cannot verify where cat appeared. It simply hopes the context vector preserved enough information.

Now imagine a different approach. Instead of relying entirely on memory, the Decoder briefly looks back at the original sentence before generating each word. When generating chat, it focuses primarily on the English word cat. When generating tapis, it focuses on mat. Instead of trying to remember every detail, the Decoder retrieves the information it needs at exactly the moment it needs it. That is the central idea behind Attention.

Fig. : Remembering vs Looking Back

The Decoder Doesn’t Need Everything

At first glance, it might seem that the Decoder needs to remember the entire source sentence while generating every output word. But that’s rarely true. Suppose the Decoder has already generated

Le chat était assis.

The next word should be

sur

To generate this word, does the Decoder really need to remember every detail of the sentence?

Does it need to think about cat?

Does it need to think about the?

Does it need to think about the beginning of the sentence?

Probably not.

At this particular moment, the most useful information is the phrase:

on the

Similarly, when the Decoder eventually generates:

tapis

the most relevant English word is:

mat

Everything else becomes less important.This reveals something profound.

The Decoder does not need access to all information equally.

It needs access to the right information at the right time.

Humans process language in exactly the same way.

When translating a sentence, your attention naturally shifts from one word or phrase to another.

You do not constantly think about every word simultaneously.

Instead, your focus moves dynamically as you construct the translation.

Researchers realized that neural networks should work the same way.

Fig. : Attention Moves Across the Sentence

A Dynamic Context Instead of a Fixed Context

The original Encoder-Decoder architecture relied on a single context vector that never changed. Whether the Decoder was generating the first word or the last word, it always received the same compressed summary of the source sentence.

Attention introduced a radically different idea: instead of one fixed context vector, the Decoder would construct a new context vector for every output word.

When translating chat, the context vector emphasizes cat.

When translating assis, it emphasizes sat.

When translating tapis, it emphasizes mat.

The context was no longer static — it became dynamic, changing continuously throughout the translation process. Attention didn’t eliminate the context vector. It transformed it from a fixed summary into a representation that adapts to the Decoder’s current needs.

Fig. : Static vs Dynamic Context

One important question still remained. If the Decoder can look back at the entire sentence, how does it decide where to look? Should every word receive equal importance? How does the model know that cat is more relevant than the when generating chat? The answer lies in one of the most elegant ideas in deep learning: the model learns to assign a relevance score to every word. Those scores determine exactly where the Decoder should focus.

How Attention Actually Works — Teaching a Model Where to Look

A neural network cannot simply “look back” because we tell it to. It needs a mathematical procedure to answer a very specific question every time it generates a new word: among all the words in the source sentence, which ones are the most relevant right now? Everything else — alignment scores, attention weights, context vectors, and eventually Transformers — is built upon the answer to that question.

Every Word Competes for the Decoder’s Attention

Suppose the Decoder has already generated Le and now needs to produce chat. If you were translating this sentence yourself, your eyes would naturally move toward the word cat. You wouldn’t spend much time looking at the, on, or mat. Researchers wanted neural networks to develop the same ability. Instead of treating every source word equally, the Decoder should learn which words deserve more attention and which can be safely ignored.

Every Encoder hidden state is essentially trying to answer one question:

Am I useful for predicting the next output word?

The most useful hidden states receive higher scores. The less useful ones receive lower scores.

Fig. : Every Encoder State Competes for Attention

Introducing the Alignment Score

Bahdanau and his colleagues introduced a quantity called the alignment score to formalize this competition. The name is intuitive — it measures how well the Decoder’s current situation aligns with each Encoder hidden state.

Every symbol here has a direct meaning.

Let’s understand every symbol one by one.

• eij​ is the alignment score between the Decoder’s current state and the jth Encoder hidden state.
• si−1​ represents the Decoder’s hidden state just before generating the next output word.
• hj​ is the hidden state produced by the Encoder for the jth input word.
• a(⋅) is a small neural network whose job is to determine how compatible these two vectors are.

The equation simply asks: given what the Decoder is trying to generate right now, how relevant is this particular Encoder hidden state?

Understanding the symbols with our translation example.

Our English sentence maps to:

| Word | Encoder Hidden State |
| ---- | -------------------- |
| The  | (h_1)                |
| cat  | (h_2)                |
| sat  | (h_3)                |
| on   | (h_4)                |
| the  | (h_5)                |
| mat  | (h_6)                |

Now suppose the Decoder is about to generate:

chat

Its current hidden state is represented by:

s1

The Decoder now compares s1​ with every Encoder hidden state.

It computes:

for cat,

for sat,

for mat,

and similarly for every other word.

Each comparison produces one numerical score.

The larger the score, the more relevant that Encoder hidden state is for generating the next French word.

Notice what has changed.

The Decoder is no longer relying on one compressed vector.

Instead, it evaluates every Encoder hidden state individually.

This is the first major departure from the original Encoder-Decoder architecture.

Fig. : Computing Alignment Scores

What Is the Alignment Function?

You might now wonder:

What exactly is this mysterious function a(⋅)?

Is it a hand-written rule?

Does someone manually specify which words should match?

Not at all.

The alignment function is itself a small neural network.

During training, it learns how to compare the Decoder’s hidden state with each Encoder hidden state.

Bahdanau proposed the following formulation:

Let’s build it piece by piece.

First, the Decoder state si−1 is multiplied by a weight matrix:

At the same time, the Encoder hidden state is transformed using another weight matrix:

These two transformed vectors are then added together:

This combined representation captures information from both the Decoder and the Encoder.

Next, the model applies the tanh activation function.

This introduces non-linearity, allowing the model to learn more complex relationships than simple linear comparisons.

Finally, the resulting vector is multiplied by another learned parameter:

This converts the vector into a single scalar value.

That scalar is the alignment score.

In other words, the alignment function is simply a small neural network whose output is one number indicating how relevant an Encoder hidden state is for the Decoder’s current prediction.

Every parameter in this equation — Wa​, Ua, and va ​ — is learned automatically during training.

The model gradually discovers which patterns lead to good translations and adjusts these weights accordingly.

Fig. : Inside the Alignment Function

A Small Numerical Example

Suppose after the alignment function processes each Encoder hidden state, it produces the following scores:

| Source Word | Alignment Score |
| ----------- | --------------- |
| The         |             0.4 |
| Cat         |             4.8 |
| Sat         |             1.2 |
| On          |             0.5 |
| The         |             0.3 |
| Mat         |             0.9 |

These are raw scores — not probabilities, and they don’t add up to one.

One thing is immediately clear: the hidden state corresponding to cat has received the highest alignment score. The neural network has correctly identified that cat is the most relevant part of the English sentence for generating chat. But these raw scores cannot yet be used directly, and that’s where the next tool enters the story.

From Scores to Focus — How Attention Learns Where to Look

Why Raw Scores Are Not Enough

Imagine six students applying for one scholarship.

After evaluating them, a committee assigns the following scores:

| Student | Score |
|  ---    | ---   |
| A       |  55   |
| B       |  91   |
| C       |  64   |
| D       |  52   |
| E       |  50   |
| F       |  58   |

These numbers tells us who performed better.

But they don’t directly answer questions like:

• What percentage of the scholarship should each student receive?
• How much more deserving is Student B compared to Student C?
• How should we compare all six students fairly?

To answer these questions, we first need to normalize the scores.

Attention faces the same challenge.

The alignment scores tell us which Encoder hidden states are more relevant.

But the Decoder needs something much more precise.

It needs values that behave like probabilities.

It needs numbers that tell it exactly how much importance to assign to every source word.

That is precisely what the Softmax function does.

Fig. : Raw Scores Need Normalization

The Softmax Function

Mathematically, the Softmax function converts every alignment score into an attention weight.

It is defined as:

At first glance, this equation may look intimidating.

Let’s understand it one piece at a time.

What Does Each Symbol Mean?

• aij is the attention weight assigned to the jth source word while generating the ith target word.
• eij is the raw alignment score that we computed in the previous section.
• exp(.) is the exponential function.
• The denominator sums the exponentials of all alignment scores.

This final step is what makes Softmax special.

Instead of treating each score independently, it compares every score against all the others.

As a result:

• Every attention weight lies between 0 and 1.
• All attention weights add up to 1.

In other words, the Decoder now has a probability distribution over the entire source sentence.

Instead of asking,

“Which word is important?”

it now asks,

“How much attention should I give to each word?”

This is a much richer and more useful representation.

Why Use the Exponential Function?

A natural question arises.

Why don’t we simply divide each alignment score by their total?

Why introduce the exponential function at all?

The answer lies in how Softmax emphasizes differences.

Suppose two words receive alignment scores of:

2.0

and

4.0

The second word is only twice as large numerically.

After applying the exponential function:

Now the difference becomes much larger.

Softmax amplifies high scores while suppressing lower ones.

This helps the model concentrate its attention on the most relevant words instead of spreading attention almost equally across the sentence.

You can think of it as adjusting the focus of a camera.

Instead of keeping the entire scene equally sharp, the camera focuses clearly on the subject while gently blurring the background.

Attention works in a remarkably similar way.

Fig. : Effect of the Exponential Function

A Complete Worked Example

Let’s apply Softmax to the alignment scores we computed earlier.

Suppose the Decoder has produced the following raw scores.

| Source Word | Alignment Score |
| ----------- | --------------- |
| The         |             0.4 |
| Cat         |             4.8 |
| Sat         |             1.2 |
| On          |             0.5 |
| The         |             0.3 |
| Mat         |             0.9 |

Step 1: Compute the Exponential

Using the exponential function, we obtain approximately:

| Source Word | e^{score} |
|  ---        |  ---      |
| The         | 1.49      |
| Cat         | 121.51    |
| Sat         | 3.32      |
| On          | 1.65      |
| The         | 1.35      |
| Mat         | 2.46      |

Step 2: Compute the Total

Adding these values gives approximately:

1.49 + 121.51 + 3.32 + 1.65 + 1.35 + 2.46 = 131.78

Step 3: Normalize

Now divide each exponential by the total.

For cat,

Similarly,

| Source Word| Attention Weight |
| ---        | ---              |
| The        | 0.011            |
| Cat        | 0.922            |
| Sat        | 0.025            |
| On         | 0.013            |
| The        | 0.010            |
| Mat        | 0.019            |

Now the numbers have a completely different interpretation.

Instead of arbitrary scores, they represent probabilities.

The Decoder should devote approximately 92% of its attention to the word cat while generating chat.

The remaining words receive only small amounts of attention.

This matches our intuition perfectly.

Fig. : Softmax Calculation Pipeline

Attention Is Not Choosing One Word

At this point, it’s tempting to think the Decoder simply selects the word with the highest attention weight. That isn’t what happens, and this distinction is one of the most important ideas in the entire mechanism. The Decoder does not choose a single Encoder hidden state. Instead, it distributes its attention across all hidden states — some receive more, some receive less, some receive almost none, but every Encoder hidden state still contributes.

This is why the mechanism is called soft attention rather than hard attention. Hard attention forces the model to choose exactly one word. Soft attention allows the model to consider every word simultaneously while assigning different levels of importance. This makes learning much smoother, allows the entire architecture to be trained using gradient descent, and mirrors how humans read — your attention concentrates on the most relevant words while still taking surrounding context into account.

The Final Piece of the Puzzle

We now have everything we need to answer an important question.

The Decoder has:

• Calculated alignment scores.
• Converted them into attention weights.
• Learned exactly how much attention each source word deserves.

But one mystery still remains.

How do these attention weights actually help generate the next word?

The Decoder cannot generate text using probabilities alone.

It still needs a meaningful representation of the source sentence.

The answer is beautifully elegant.

Instead of using one fixed context vector for the entire translation, the model builds a new context vector every time it generates a word.

And it builds that vector using the attention weights we have just computed.

That single idea completes the Attention mechanism.

Dynamic context vector

The Dynamic Context Vector — Where Everything Comes Together

By this point, our Decoder has learned something remarkable.

Instead of depending on one fixed context vector for the entire translation, it now knows how important every source word is for predicting the next target word.

For our running example, suppose the Decoder is generating the French word:

chat

After applying the Softmax function, we obtained the following attention weights:

| Source Word| Attention Weight |
| ---        | ---              |
| The        | 0.011            |
| Cat        | 0.922            |
| Sat        | 0.025            |
| On         | 0.013            |
| The        | 0.010            |
| Mat        | 0.019            |

These numbers tell us exactly where the Decoder should focus.

But they don’t yet tell us what information should actually be passed to the Decoder.

Knowing that cat deserves 92% of the attention is useful.

The Decoder still needs a meaningful representation of that information.

This leads us to the final step of the Attention mechanism.

The model builds a new context vector.

Not once.

But every time it generates a word.

This single idea transformed sequence modeling forever.

From Attention Weights to Information

Let’s return to our open-book exam analogy.

Imagine you’re answering a question about Newton’s Second Law.

Your attention naturally shifts toward the chapter discussing Newtonian mechanics.

But simply looking at the page isn’t enough.

You still need to read the relevant information and use it to write your answer.

Attention in neural networks works exactly the same way.

The attention weights determine where to look.

The context vector determines what information to retrieve.

In other words:

Attention Weights ➡️ Tell us where to focus.

Context Vector ️ ️➡️ Collects the useful information from those locations.

This distinction is subtle but extremely important.

Many think that attention weights themselves are the information.

They are not.

They are simply instructions telling the model how much information should be collected from each Encoder hidden state.

Fig. : Attention Chooses, Context Collects

Building the Dynamic Context Vector

Now we arrive at one of the most elegant equations in deep learning.

The context vector is computed as:

Although this equation looks compact, it captures the entire philosophy of Attention.

Let’s understand every symbol carefully.

• (c_i) is the context vector used to generate the ith target word.
• (h_j) represents the Encoder hidden state for the jth source word.
• aij is the attention weight assigned to that hidden state.
• The summation means we combine information from every Encoder hidden state.

This equation says something incredibly simple.

Multiply every Encoder hidden state by its attention weight.

Then add them together.

That’s it.

No complicated lookup tables.

No handcrafted rules.

Just a weighted average.

This is why many researchers describe Attention as a soft lookup mechanism.

Instead of selecting one location, the model softly combines information from many locations according to their importance.

Fig. : Weighted Average Intuition

A Numerical Example

Let’s use a simplified example.

Suppose each Encoder hidden state contains only two numbers instead of hundreds.

Word    Hidden State
The [0.2, 0.1]
Cat [0.9, 0.8]
Sat [0.4, 0.5]
On  [0.1, 0.2]
The [0.3, 0.2]
Mat [0.6, 0.7]

And suppose the attention weights are:

| Source Word| Attention Weight |
| ---        | ---              |
| The        | 0.011            |
| Cat        | 0.922            |
| Sat        | 0.025            |
| On         | 0.013            |
| The        | 0.010            |
| Mat        | 0.019            |

Now we multiply each hidden state by its corresponding attention weight.

For example,

For cat, 0.922 * [0.9,0.8] = [0.828,0.736]

For mat, 0.02 * [0.6,0.7] = [0.012,0.014]

For the, 0.01 * [0.2,0.1] = [0.002,0.001]

Repeating this for every word and then adding all the resulting vectors gives approximately:

c_i = [0.856, 0.767]

This final vector is not identical to any single Encoder hidden state.

Instead, it is a carefully constructed summary that emphasizes the most relevant information while still retaining a small contribution from the rest of the sentence.

That is exactly what the Decoder needs.

Fig. : Numerical Construction of the Context Vector

Why Every Output Word Gets Its Own Context

Now comes the most profound consequence of the Attention mechanism.

Suppose the Decoder is generating:

chat

Its attention is concentrated on cat.

Therefore, the context vector is dominated by information related to cat.

A few moments later, the Decoder wants to generate:

tapis

Should it use the same context vector?

Of course not.

Now the relevant source word is mat.

The Decoder recomputes the alignment scores.

It applies Softmax again.

New attention weights are produced.

A completely new context vector is constructed.

This means the context vector is no longer fixed.

It changes continuously throughout the translation process.

Every output word receives its own personalized summary of the source sentence.

This seemingly small change solved the bottleneck problem we discussed in the previous blog.

Instead of forcing the Decoder to rely on one compressed representation for the entire sentence, Attention allows it to build a fresh representation whenever it needs one.

Fig. : Dynamic Context Vectors

Attention Is a Soft Lookup Mechanism

At this point, we can finally describe Attention in one sentence.

Attention is a soft lookup mechanism that retrieves the most relevant information from the Encoder whenever the Decoder needs it.

Notice the word soft.

The Decoder does not jump to exactly one Encoder hidden state.

Instead, it consults all of them.

Some contribute a lot.

Some contribute a little.

Some contribute almost nothing.

This makes Attention both flexible and differentiable, allowing it to learn entirely through gradient descent.

More importantly, it changes how we think about neural networks.

The Decoder is no longer trapped inside its own memory.

It has learned how to retrieve information intelligently.

Memory is no longer static.

It has become dynamic.

And that simple shift — from storing information to retrieving it — would soon inspire researchers to ask an even bigger question.

If Attention can retrieve information so effectively…

Do we still need recurrent neural networks at all?

That question would ultimately lead to one of the most influential papers ever published in artificial intelligence:

Attention Is All You Need.

Why Attention Changed Artificial Intelligence Forever

To appreciate the full impact, consider what changed. Before Attention, the Decoder relied on a single fixed context vector for the entire translation. After Attention, it builds a new context vector for every output word. That single difference solved the bottleneck problem that had limited Encoder-Decoder models for years.

The original Bahdanau Attention paper demonstrated something striking. As sentence length increased, traditional Encoder-Decoder models rapidly lost translation quality — important details disappeared, relationships between distant words were forgotten, grammatical structures became less accurate. After introducing Attention, these problems were significantly reduced. The Decoder could dynamically retrieve information from the most relevant parts of the source sentence, behaving much more like a human translator who repeatedly looks back rather than translating from a memorized summary.

Fig. : Moving Focus During Translation

Throughout the history of neural networks, researchers had been trying to answer one question: how can we build better memory? RNNs introduced hidden states. LSTMs added memory cells and gates. GRUs simplified those mechanisms. Each architecture focused on making memory stronger. Attention asked a different question: instead of continuously improving memory, why not improve retrieval? That shift — from memorization to retrieval — became one of the defining ideas behind modern AI.

Fig. : Memory vs Retrieval

The Limitation That Still Remained

By this point, it might seem as though the sequence modeling problem had finally been solved. The bottleneck was gone, long sentences translated well, and the Decoder no longer depended entirely on a fixed memory. So why didn’t researchers stop here?

Because one major limitation remained hidden inside the architecture. The Encoder still reads one word at a time, and the Decoder still generates one word at a time. Every output word depends on the previous one, which means nothing can happen in parallel. No matter how powerful Attention becomes, the underlying architecture is still built on recurrent neural networks, and this creates two expensive problems: training is slow because every word must wait for the previous word to be processed, and recurrent computation is difficult to parallelize on modern GPUs.

Fig. : Sequential Processing Bottleneck

Researchers noticed something fascinating. The most important computations were no longer happening inside the RNN — they were happening inside the Attention mechanism itself. The Decoder wasn’t succeeding because it had better memory. It was succeeding because it knew where to look. If Attention was already solving the hardest part of sequence modeling, perhaps recurrence wasn’t necessary after all. Perhaps neural networks didn’t need to process language one word at a time.

Looking Ahead

The Attention mechanism replaced a fixed memory with dynamic retrieval, allowed models to focus on the most relevant parts of a sentence instead of compressing everything into one vector, and transformed machine translation while inspiring an entirely new way of thinking about sequence modeling. Yet despite all its success, Attention still lived inside an architecture built around recurrent neural networks. The Encoder was still an RNN. The Decoder was still another RNN. Training was still sequential.

Researchers had discovered something powerful. Now they wanted to know just how far that idea could go.

What would happen if Attention became the entire architecture — if the Encoder and Decoder no longer depended on recurrence at all?

In 2017, a team of researchers at Google answered those questions with a paper whose title sounded almost provocative:

Attention Is All You Need.

As we’ll discover in the next article, that bold statement didn’t just introduce another neural network architecture. It marked the beginning of the Transformer era — the foundation of ChatGPT, Claude, Gemini, Llama, and nearly every large language model used today.

Frequently Asked Questions

Why does the alignment function need to be a neural network? Couldn’t simpler methods like dot products work?

The alignment function needs to capture compatibility between two vectors that live in different representational spaces — the Decoder’s hidden state and an Encoder hidden state. Bahdanau used an additive neural network because the two vectors could have different dimensionalities and the relationship between them can be highly non-linear. Later work, particularly by Luong et al., showed that a simple dot product between the two vectors also works well when their dimensions match, and it’s significantly cheaper to compute. The dot-product approach became the foundation of the scaled dot-product attention used in Transformers, so both formulations matter in the broader story.

What happens to attention weights at the very first decoding step when the Decoder has no previous output?

At the first step, the Decoder receives a special start token, commonly written as <START> or <BOS> (beginning of sequence), as its initial input. Its hidden state s_0 is typically initialized to the Encoder's final hidden state — the same context vector that the original Encoder-Decoder architecture relied on entirely. From that point forward, the Decoder generates words one at a time, using its own previous output at each step, and the attention mechanism adapts continuously as the hidden state evolves.

If soft attention considers all words simultaneously, does the model ever focus sharply on just one word?

Yes, it can, and it often does. When one alignment score is dramatically higher than the others, the Softmax function amplifies that difference further, pushing the attention weight for that word close to 1 and collapsing everything else toward 0. In practice, researchers visualize attention weight matrices as heatmaps, and these heatmaps often show sharp, focused peaks when a single source word is strongly relevant — for example, when generating a proper noun that has a direct counterpart in the target language.

How does Bahdanau Attention relate to the self-attention used in Transformers?

Bahdanau Attention computes relationships between two different sequences — the source sentence (Encoder) and the target sentence (Decoder). Self-attention applies the same core idea but within a single sequence, computing relationships between every word and every other word in the same sentence. This allows a model to understand how words relate to each other regardless of distance, without any recurrent structure at all. Self-attention is the mechanism that makes Transformers possible, and it descends directly from the ideas introduced in Bahdanau’s work.

Why is it called “soft” attention specifically? What would “hard” attention look like?

Hard attention selects exactly one Encoder hidden state at each decoding step rather than taking a weighted combination. The model would attend to cat with weight 1.0 and give everything else a weight of exactly 0. The problem is that selecting one discrete item is not differentiable, so the model cannot be trained with standard backpropagation. Hard attention requires reinforcement learning techniques or other approximations, which makes training significantly more difficult and less stable. Soft attention, by distributing weight continuously across all hidden states, keeps the entire computation differentiable end-to-end.

Did Attention immediately replace all previous approaches to machine translation?

Not immediately, but the transition was fast by research standards. Bahdanau Attention was published in late 2014, and by 2016 it had become a standard component in competitive machine translation systems. Google’s production translation system adopted attention-based neural machine translation in 2016, replacing years of statistical methods. Within roughly three years of the original Attention paper, the field had moved further still — the 2017 Transformer paper proposed removing recurrence entirely, and within another two years, Transformer-based models had displaced recurrent architectures across nearly every area of natural language processing.

Further Reading

Foundational Research Papers

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho & Bengio, 2014) — The original paper that introduced the Attention mechanism and solved the Encoder-Decoder bottleneck.

Practical Explanations

Jason Brownlee, Machine Learning Mastery — The Bahdanau Attention Mechanism, What Is Attention?, The Attention Mechanism from Scratch

In the next blog, we’ll study one of the most influential research papers in the history of artificial intelligence.

Attention Is All You Need.

We’ll understand why researchers abandoned recurrent neural networks, how self-attention replaced recurrence, and why the Transformer became the architecture that powers today’s large language models.