“How ‘Attention Is All You Need’ sparked my deep dive into neural networks — and why Transformers changed everything”

All it took was a single research paper that made me dive deep into Neural Networks — “Attention Is All You Need” — which describes Transformers. And honestly? It all started with just one question in my head: “How does the human brain work?”

We know the answer, right? It works through neurons! Neurons communicate with each other, pass information, and help us make decisions and do all kinds of tasks. So the thought was — why can’t we try the same thing in machines? And that is exactly how the Perceptron came to life.

The Perceptron is the simplest building block of a neural network. I know we all know about perceptron already, so instead of spending time on that, let’s jump straight into something more interesting — Multilayer Perceptrons and Feedforward Networks!

Multilayer Perceptron (MLP) and Feedforward Networks

So, an MLP expands on the single-layer perceptron to solve highly complex, non-linear problems. It is defined by three major traits:

Layer Structure: It contains at least three layers — an input layer, one or more hidden layers, and an output layer.

Full Connectivity: Every neuron in one layer connects to every single neuron in the next layer. These are often called dense layers.

Non-Linear Activation: It uses non-linear activation functions like ReLU, Sigmoid, or Tanh in the hidden layers. Without these, stacking multiple layers would just collapse into a single linear function and trust me, that defeats the whole purpose!

Now, in a feedforward network, information travels only in the forward direction no cycles, no loops, no memory states unlike RNNs (we will get there, don’t worry!). So here is how the whole thing works:

Forward Propagation: Data enters the input layer, gets multiplied by weights, adds a bias, passes through activation functions, and exits as a prediction at the output layer.

Loss Calculation: The network looks at its prediction and compares it to the actual answer using a loss function basically asking itself, “okay, how wrong was I?”

Backpropagation: Now here comes the interesting part! The network passes the error gradient backward through all the layers to figure out how much each weight was responsible for the mistake.

Weight Optimization: Then an optimization algorithm like Stochastic Gradient Descent (SGD) updates the weights to do better next time. And the formula for that is:

New Weight = Old Weight − (Learning Rate × Gradient)

So we are calculating the new weight from the old one we are nudging it step by step in the direction that reduces the error. Simple, right?

Convolutional Neural Networks (CNNs)

Okay so next up CNNs! A CNN is a specialized type of feedforward neural network designed mainly to process structured grid data like images. Unlike standard MLPs where every single input pixel connects to every neuron, CNNs use a mathematical operation called convolution to automatically extract local features like edges, shapes, and textures while sharing weights across the input. Pretty smart!

CNNs work on three core design principles:

• Local Receptive Fields : each neuron only looks at a small region of the input, not the whole thing at once.
• Shared Weights :the same filter gets applied across the entire input, which reduces the number of parameters dramatically.
• Spatial Invariance : the network can recognize a feature no matter where it appears in the image.

Now, CNNs are brilliant at what they do. But wait :I think I had a doubt here, and I am sure you had the same one too!

Recurrent Neural Networks (RNNs)

“CNN is working efficiently and giving great outputs ,then why do we even need RNN?”

Right? That was exactly my question! So let me explain.

The thing is, CNNs and MLPs are just not built for sequential data of variable lengths things like text, audio, time-series, or DNA sequences. A CNN treats its input like a static snapshot. It has no way to remember what it saw before. It cannot understand that the word “it” in a sentence refers to something said three words ago. It just… doesn’t know.

So, RNNs came to solve exactly this! A **Recurrent Neural Network is a deep learning model trained to process sequential data and convert it into a specific sequential output. These networks capture long-term dependencies** in sequential data by maintaining a hidden state that carries information from one step to the next like a little memory!

Now, when it comes to learning from mistakes, RNNs use something called Backpropagation Through Time (BPTT). Let me put it simply ,regular backpropagation works across layers of a network, but BPTT works across time steps. It unrolls the network through the whole sequence and figures out how much each past step contributed to the current error. The core math is still the same chain rule and gradient descent just applied through time!

But here is the problem. As the error travels back through many many time steps, the gradients keep getting smaller and smaller ,sometimes almost vanishing to zero. And when that happens, the network just stops learning from things that happened far back in the sequence. It basically forgets. This is called the vanishing gradient problem, and it was a big headache.

And that is exactly why GRU and LSTM came into the picture!

Gated Recurrent Unit (GRU)

GRU is a highly efficient variant of RNN used to process sequential data. It solves the traditional RNN memory problem using just two gates that control exactly when to remember and when to forget:

• Update Gate :decides how much of the past information to carry forward into the future.
• Reset Gate :decides how much of the past to throw away when computing the new state.

GRU is simpler and faster to train, and in many tasks it performs just as well as LSTM. So why LSTM then? Let’s see!

Long Short-Term Memory (LSTM)

LSTM is a specialized RNN architecture designed to process sequential data like text or time series. It directly tackles the vanishing gradient problem, allowing the model to remember or forget context over thousands of timesteps capturing genuinely long-term dependencies.

LSTM does this with three gates:

• Forget Gate : looks at what is in memory and decides what to throw away.
• Input Gate : decides what new information to store into the memory.
• Output Gate : decides what part of the memory to actually output as the hidden state.

So both GRU and LSTM were massive improvements over plain old RNNs. But ,and here is the thing they still had one big limitation. They process data one step at a time, sequentially. Step 1, then step 2, then step 3… You cannot skip ahead. This made training on long sequences really, really slow.

And this is the exact problem Transformers came to blow up. But before we get there, we absolutely need to understand the mechanism that makes Transformers tick the Attention Mechanism!

Attention Mechanism

Okay so the **Attention Mechanism** is honestly one of my favourite things to explain. It is a technique that makes a model selectively focus on the most important parts of the input not ignoring everything else, but deciding how much attention to give to each part. Like how when you are reading a long paragraph, your brain does not give equal focus to every single word some words just matter more in that moment!

Here is a better way to feel it. Consider this sentence:

“The animal didn’t cross the street because it was too tired.”

Your brain instantly knows “it” refers to the animal, not the street. How? Because without even thinking, you are weighing every word against every other word and figuring out which ones are most relevant for understanding “it.” That is literally attention at work!

Mathematically, attention works through three things : Query (Q), Key (K), and Value (V):

• The Query is the word trying to understand its context. It is basically asking “which parts of this sequence should I focus on?”
• The Key is what every other word in the sequence puts out about itself like a label saying “here is what I represent.”
• The Value is the actual content each word carries “here is what I will contribute if you choose to attend to me.”

So the attention score is calculated by comparing the Query against all the Keys. Words whose Keys closely match the Query get a higher score. Those scores become weights, and those weights are applied to the Values giving every word a rich, context-aware representation. Every word gets to look at every other word directly. No fading memory. No sequential steps. Just pure, parallel attention!

Transformers

Okay, now we are here! The thing that started this whole journey for me!

A **Transformer is a neural network architecture that transforms an input sequence into an output sequence. And here is the big thing unlike RNNs which process data one element at a time in sequential steps, Transformers process the entire sequence simultaneously, in parallel.** This makes training so much faster and removes all those memory bottlenecks we had with RNNs.

But just like how CNN works on spatial data like images, Transformers work specifically on sequential data.

Now, traditional sequence models used an Encoder-Decoder approach:

• Encoding is the process where the encoder reads and processes the entire input sequence and compresses its meaning, context, and relationships into a mathematical representation a hidden state or a vector.
• Decoding is where the decoder takes that compressed representation and generates the output, one step at a time.

The problem with this? You are compressing an entire sentence or a paragraph, or even a whole document into a single fixed-size vector. That is a LOT of information forced into a small space. The model starts losing context, especially on longer sequences.

Transformers said forget that approach. Instead of compressing everything into one vector, let every position in the sequence directly attend to every other position using Self-Attention. No compression. No bottleneck. No forgetting.

And that is why Transformers gave more efficient and accurate outputs than RNN, GRU, and LSTM not by making them better, but by replacing them entirely with something smarter!

So, Where Does This Leave Us?

Okay so let us come back to where this whole thing started one paper, one question.

“Attention Is All You Need.” And “How does the human brain work?”

We went from the Perceptron ,the simplest possible imitation of a neuron all the way to the Transformer, which today powers GPT, BERT, Gemini, and basically every modern AI system you interact with. The entire era of large language models that the world is going crazy about right now? It runs on the architecture we just walked through together.

And here is the thing that genuinely gets me the human brain still uses something we do not fully understand. But the Attention Mechanism is probably the closest mathematical approximation we have built so far. Selectively weighting what matters, letting go of what does not, and building understanding from relationships rather than rigid order.

We are not done though! The next step is understanding how Transformers are actually structured inside Positional Encoding (because attention alone has no sense of word order), Multi-Head Attention (running multiple attention mechanisms at the same time), and how BERT and GPT use the same architecture in completely opposite ways.

The rabbit hole goes much deeper. And honestly? I cannot wait to go further.

“What’s your favourite deep learning architecture? Have you worked with Transformers? Let me know in the comments. I’d love to hear your journey.

Further Reading

• Attention Is All You Need — original Transformer paper
• IBM: Attention Mechanism — simple explanation of attention
• AWS: What is an RNN? — quick refresher on sequential models
• AWS: What are Transformers in AI? — high-level overview

— moon