Explain BERT architecture.

BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model introduced by Google in 2018 that revolutionized natural language processing. Its architecture is based on the Transformer encoder from the paper “Attention Is All You Need”.

1. High-level idea

BERT reads text bidirectionally — it looks at both left and right context simultaneously. This is different from older models that read text only left-to-right or right-to-left.

2. Input Representation

Before text enters BERT, it’s converted into three types of embeddings:

• Token embeddings — represent each word/subword
• Segment embeddings — distinguish sentence A vs sentence B
• Position embeddings — encode word order

These are summed to form the final input vectors.

3. Transformer Encoder Stack

BERT is built from multiple identical layers (called encoder blocks).

• BERT Base: 12 layers
• BERT Large: 24 layers

Each layer has two main components:

a. Multi-Head Self-Attention

• Every word attends to every other word in the sentence
• Captures contextual relationships
• Multiple “heads” allow learning different types of relationships

Example: In “The bank is near the river,” attention helps determine that bank refers to a riverbank, not a financial institution.

b. Feed-Forward Neural Network

• A simple fully connected network applied to each token
• Adds non-linearity and deeper representation **learning**

c. Add & Normalize

• Residual connections + layer normalization improve stability and training

4. Special Tokens

BERT uses special markers:

• [CLS] → added at the beginning; used for classification tasks
• [SEP] → separates sentences

The final hidden state of [CLS] acts as a summary of the whole input.

5. Pretraining Objectives

BERT is trained using two key tasks:

a. Masked Language Modeling (MLM)

• Random words are masked
• Model predicts missing words
• Enables bidirectional understanding

Example: “The cat sat on the [MASK]” → predicts “mat”

b. Next Sentence Prediction (NSP)

• Determines if sentence B logically follows sentence A
• Helps with tasks like QA and entailment

6. Why BERT Works Well

• Deep bidirectional context understanding
• Scales with more layers and data
• Pretrained once, fine-tuned for many tasks

7. Typical Applications

• Sentiment analysis
• Question answering
• Named entity recognition
• **Text classification**
• Search and ranking

8. Simple Architecture Flow

Input Text
   ↓
Token + Position + Segment Embeddings
   ↓
Transformer Encoder Layers (stacked)
   ↓
Contextualized Word Representations
   ↓
Task-specific Output Layer