Overview of Transformers Architecture

What is a transformer?

A Transformer is an AI architecture that understands relationships between words to process and generate language.

It is mainly built using these components:

• Embedding
• Positional Encoding
• Self-Attention
• Masked Self-Attention
• Cross Attention
• Feed Forward Neural Network
• Multi-Head Attention

1. Embedding

Embedding is the process of converting words into vectors (numbers).

Why do we need Embeddings?

Computers cannot directly understand words like:

So, embeddings convert words into numerical vectors:

• Hello → [1, 2, 4]
• Good → [3, 1, 7]

How do we get vector numbers like these?

We already have trained models like BERT, developed by Google, that are trained on large amounts of text data to learn vector representations of words.

These models are trained in such a way that:

• Words with similar meanings, like “Hello” and “Hi”, get similar vector values.
• Words with very different meanings, like “Hello” and “Bye”, get very different vector values — meaning they are far apart in vector space.

2. Positional Encoding

Positional Encoding adds position information to embeddings.

Why do we need Positional Encoding?

Example: We can write “I LOVE AI” in different ways

• “I LOVE AI”
• “LOVE AI I”
• “LOVE I AI”

The position of each word in a sentence is very important because changing the order can change the meaning of the sentence.

Problem: Transformers process all words in parallel.

So the model sees all words at the same time.

I
LOVE
AI

Without positional information, the Transformer would not know:

• which word came first
• which word came second

How do we add Positional Encoding?

We use sine and cosine curves to generate positional values.

These positional values are added to the word embeddings to preserve the order of words in a sentence.

Why don’t we use only the sine curve?

If we use only the sine curve, the values repeat after some interval because sine is a periodic function.

This can make different positions look similar to the model.

So, Transformers use a combination of both sine and cosine curves to create more unique positional patterns.

Why do we use different curve lengths (frequencies)?

Different sine and cosine curves change at different speeds.

• Some curves change very quickly
• Some curves change slowly

This helps the Transformer capture both:

• nearby word positions
• long-distance word relationships

The combination of all these curves creates a unique positional pattern for each word position.

3. Self Attention:

It helps the Transformer understand how words in a sentence are related to each other.

It converts normal embeddings into contextual embeddings by using information from related words.

For Example: “I LOVE AI”

The model calculates relationships between all words and also with themselves

I ↔ LOVE
I ↔ AI
LOVE ↔ AI

This creates attention scores.

Higher score:

• stronger relationship

Lower score:

• weaker relationship

How Self-Attention Actually Works

Each embedding is transformed into 3 vectors:

• Query (Q) → what this word is searching for
• Key (K) → what this word offers
• Value (V) → actual information/content of the word

Example: For the word “LOVE”:

Query asks: “Which words are relevant to me?”

Key says: “Here is the type of information I contain.”

Value contains: actual contextual information

What happens next?

1. Softmax converts scores into attention weights
2. Attention weights multiply the Value vectors
3. Weighted vectors are combined

This creates contextual vectors.

Now each word understands:

• surrounding words
• context
• meaning

Explanation:

4. Multi-Head Attention

Instead of using only one attention calculation, Transformers use multiple attention heads.

Each head learns different types of relationships.

Example

Sentence:

"I LOVE AI"

Different heads may learn:

• grammatical relationships
• positional relationships
• emotional relationships
• subject-object relationships

Each head has separate:

• Q matrices
• K matrices
• V matrices

Outputs from all heads are combined together.

This helps the Transformer understand language from multiple perspectives.

5. Add & Normalize

After attention, the original input is added back to the output.

This is called a Residual Connection.

Why do we add the original input?

Self-attention updates the word vectors using contextual information from other words.

But sometimes the attention layer may over-change the representation and important original information could get weakened.

So, we add the original input back to preserve the original meaning while also keeping the new contextual understanding.

This helps the model:

• preserve important information
• avoid losing the original context
• train deep networks more effectively

This is called a Residual Connection.

Why do we normalize?

After addition, some vector values may become too large or unstable.

Normalization balances the values and keeps training stable.

This helps the Transformer:

• train faster
• learn better
• avoid unstable values across layers

Masked Self-Attention is used in the decoder part of the Transformer.

Its main purpose is to prevent the model from seeing future words while predicting the next word.

6. Feed Forward Neural Network (FFN)

After attention, each contextual vector passes through a small neural network independently.

Purpose:

• learn deeper patterns
• improve representation quality
• increase model capacity

FFN Formula

This is applied separately to every token.

Encoder Workflow

The Encoder mainly performs:

1. Embedding
2. Positional Encoding
3. Multi-Head Self-Attention
4. Add & Normalize
5. Feed Forward Network
6. Add & Normalize

Purpose:

• deeply understand the input sentence

The encoder produces contextual representations.

7. Mask Attention:

Masked Self-Attention is used in the Decoder.

Its purpose is to prevent the model from seeing future words.

It is similar to attention but we just use masking before softmax to make sure to remove the future words from the context

Why do we need masking?

During training, the model should learn how to predict the next word step-by-step.

So while predicting the current word, the decoder is only allowed to see:

• previous words
• current position

but NOT future words.

Why can’t we pass the full sentence directly?

If we allow the decoder to see future words, the model may simply copy the answers instead of learning how to predict them.

For example: I LOVE AI => Translate => “Naaku AI ante istam”

Target sentence:“Naaku AI ante istam”

Suppose the model is trying to predict:“AI”

If the decoder already sees: “AI ante istam” then prediction becomes too easy.

The model would basically “cheat” during training.

So we use a mask to hide future words.

How does training happen?

The decoder predicts one word at a time.

Example:

Step 1

Input:*<SOS>*

Predict:“Naaku”

Step 2

Input:*<SOS> Naaku*

Predict:“AI”

Step 3

Input:*<SOS> Naaku AI*

Predict:“ante”

Step 4

Input:*<SOS> Naaku AI ante*

Predict:“istam”

This continues until the model predicts:*<EOS>*

(End Of Sentence)

Important Point

Even though all words are processed in parallel internally, masking ensures that each position can only attend to previous words, not future words.

So the model learns true next-word prediction.

Note

At every step, the model may have many possible next-word choices.

Masked self-attention helps the model learn which next word is most probable based on previous context.

8. Encoder-Decoder Attention (Cross Attention)

In Cross Attention:

• Query (Q) comes from the decoder
• Key (K) comes from the encoder
• Value (V) comes from the encoder

Why do we need this?

The decoder already knows:

• previously generated words
• target-side context

But it also needs information from the input sentence.

So the decoder uses Query vectors to search relevant information from the encoder outputs.

Example

Input English sentence:“I LOVE AI”

Encoder creates contextual vectors for:

• I
• LOVE
• AI

Now suppose decoder currently generated: “Naaku” and wants to predict the next Telugu word.

The decoder Query searches the encoder context vectors to find which input words are most relevant.

For example:

• “LOVE”
• “AI”

may receive higher attention scores.

What happens internally?

Step 1

The decoder compares: Q⋅KT to calculate similarity scores.

This creates an attention similarity matrix.

Step 2

Softmax is applied to convert similarity scores into attention weights.

Higher score: → higher importance

Lower score: → lower importance

Step 3

These attention weights are multiplied with the encoder Value (V) vectors.

The weighted vectors are then combined to create a new context-aware decoder representation.

Now the decoder better understands:

• which input words are important
• what context should be used for prediction

9. How is the next word predicted?

Cross Attention does NOT directly predict words.

It only gathers contextual information from the encoder.

After this, the decoder produces a final contextual vector.

Step 1: Linear Layer

The Linear Layer converts the decoder vector into scores for every word in the vocabulary.

Example:

AI     → 8.5
ante   → 2.1
istam  → 1.3

These scores are called logits.

Higher score:

• more likely word

Lower score:

• less likely word

Step 2: Softmax Layer

Softmax converts logits into probabilities.

Example:

AI     → 0.75
ante   → 0.15
istam  → 0.10

Now all probabilities sum to:

The word with the highest probability is selected as the next predicted word.

So the decoder predicts:

"AI"

Example:

AI =>0.75, ante => 0.15, istam => 0.10

The word with the highest probability becomes the next predicted word.

Here we pick “AI” => full sentence=> Naaku “AI”

Decoder Workflow

The Decoder mainly performs:

1. Output Embedding
2. Positional Encoding
3. Masked Multi-Head Attention
4. Add & Normalize
5. Cross Attention
6. Add & Normalize
7. Feed Forward Network
8. Add & Normalize
9. Linear Layer
10. Softmax

Purpose:

• generate output one word at a time

Note: This blog is written based on my current understanding of Transformer architecture. If you find any mistakes or inaccuracies, please feel free to let me know.