Tech Blog #5 — BERT

Bidirectional Encoder Representations from Transformers — October 2018 — By Google

The Problem BERT Solved

Before BERT, if you wanted an AI to do Sentiment Analysis, you built a custom model. If you wanted it to do Question Answering, you built a different custom model. There was no “general purpose” understanding engine. Existing models (like LSTMs or early GPT) were also mostly unidirectional (reading left-to-right), meaning they couldn’t use the full context of a word.

The BERT Architecture

• The Chassis: They took the Encoder half of the 2017 Transformer and stacked it either 12 times (BERT-Base: 110M parameters) or 24 times (BERT-Large: 340M parameters).
• No Decoder: Remember our rule: BERT has no Decoder because it doesn’t generate text; it only understands it.

The Two “Games” (Pre-Training Objectives)

This is the actual genius of the paper. They trained the model using two simultaneous games:

MLM (Masked Language Modeling): Hide 15% of words and force the model to guess them using left and right context.

Of the 15% masked tokens: 80% are replaced with [MASK] token, 10% are replaced with a RANDOM word & 10% are LEFT UNCHANGED

Why not mask all 15% with [MASK]?

• Because during inference, the model never sees [MASK] tokens in real sentences. If it only trained on [MASK], it would be confused by real words. The random and unchanged words force the model to develop a better general understanding of ALL words, not just masked ones.

NSP (Next Sentence Prediction): Feed the model two sentences (A and B). 50% of the time, B actually follows A.

50% of the time, B is a random sentence from elsewhere. The model must guess: IsNext or NotNext. (This forces the model to learn relationships between sentences, not just words).

The [CLS] Token Invention

BERT adds a completely fake, invisible word at the very beginning of every input called [CLS] (Classification).

During training, every token ATTENDS to all other tokens through self-attention and its resulting vector REPRESENTS the sentence.

Think of it as the designated note-taker in your team meeting. Their job is to absorb everything said by everyone and produce a summary.

The [SEP] Token Invention

Added at the END of each sentence, separating Sentence A from Sentence B during NSP training. Also marks the end of the input.

The BERT Input Anatomy (Different from GPT)

GPT INPUT:    [ "The", "cat", "sat" ]
BERT INPUT:   [ [CLS], "The", "cat", "sat", ".", [SEP] ]

BERT also has a "Segment Embedding" ([SEP] token) to tell it if words belong to Sentence A or B:

Input:   [CLS] The cat sat [SEP] It was tired [SEP]
Segment: [  0 ]   0   0   0 [ 0 ]  1   1   1  [ 1 ]
          ^^^^^^^^^^^^^^^^^^^^   ^^^^^^^^^^^^^^^^^
          Sentence A              Sentence B

The Two Pre-Training Games Visualized

=============================================================
GAME 1: MLM (Masked Language Modeling) - Word Level
-------------------------------------------------------------
Input:  "The man went to the [MASK] to buy milk."
Target: Guess "store". (Learns grammar and word context)

=============================================================
GAME 2: NSP (Next Sentence Prediction) - Sentence Level
-------------------------------------------------------------
Input A: "My dog is very energetic."
Input B: "He loves to play fetch."     -> Label: [IsNext]
Input B: "The stock market crashed."   -> Label: [NotNext]
Target: Guess the relationship. (Learns logical flow between sentences)

The “Before & After” Benchmark (The GLUE Score)

When this paper dropped, it destroyed every record on the GLUE benchmark (a test of 11 different NLP tasks).

The Layman Explanation — The “College Graduate” Analogy

Imagine you are running a massive corporation (Google).

In the old days, every time a new department needed an employee, you had to hire someone off the street and train them from scratch for that specific job. (Task-specific models).

Then, you decide to create an internal “University” (Pre-training). You put a student through a brutal 4-year program where they play two specific games:

1. Game 1 (MLM): You give them articles with missing words and tell them to fill in the blanks. (They learn how to read perfectly).
2. Game 2 (NSP): You give them two paragraphs and ask, “Does the second paragraph logically follow the first?” (They learn how to reason).

After 4 years, this student is a BERT-Base Graduate. They don’t know how to do any specific job yet, but their reading comprehension and logic are off the charts.

Now, a department needs a Sentiment Analyst. Instead of training from scratch, you take the BERT Graduate, sit them at a desk, and give them a 1-day workshop (Fine-Tuning). You say, “Look at movie reviews. If it’s good, output a 1. If it’s bad, output a 0.”

Because the student is already a genius at reading, they master the 1-day workshop instantly and become the best Sentiment Analyst you’ve ever had.

That is why BERT changed the world. It proved that “Pre-training + Fine-tuning” is infinitely better than training from scratch.

The Only Math You Need to Read the BERT Paper

=============================================================
FORMULA 1: BUILDING THE INPUT (Addition)
-------------------------------------------------------------
Word "Apple"  = [0.5, 0.2, 0.8]  (Token)
Position "2"  = [0.1, 0.1, 0.1]  (Position)
Sentence "A"  = [0.0, 0.9, 0.0]  (Segment)
-----------------------------------
FINAL INPUT   = [0.6, 1.2, 0.9]  (Just add them up!)

=============================================================
FORMULA 2 & 3: CALCULATING THE ERROR (Subtraction/Division)
-------------------------------------------------------------
Both MLM and NSP just use standard "Log Loss". 
It's just the mathematical way to measure the "Error" we talked about!

Total Error = (Error from guessing the I) + (Error from guessing IsNext)

Formula Used in Paper

The Layman Explanation of Formulae (The “Building a Sandwich” Analogy)

If the 2017 Transformer paper was a complex chemistry textbook explaining how to bake bread from molecular flour (the Q/K/V math), the BERT paper is a recipe card that says: “Go buy the bread from the store.”

• Formula 1 is the recipe: “Take one slice of bread (Token), add a slice of cheese (Position), and add a slice of ham (Segment).” It’s just assembly instructions.
• Formulas 2 & 3 are the taste test: “Did the sandwich taste good (did we guess the right word)? If not, calculate how bad it tasted (Log Loss) so we know what ingredients to change next time.”

There is no complex physics or calculus in this paper. It is purely a “systems engineering” paper. They took pre-existing math blocks, stacked them up, and figured out the best way to train them.

What This Means for YOU in Human Terms:

• If somebody writes [SEP]: It's just a period/punctuation mark that tells the model "End of Sentence A, start of Sentence B."
• If somebody writes [CLS]: Remember the "Team Manager." All the other words do the hard work of reading, but they pass their notes up to [CLS]. When you build a classifier in Python using HuggingFace, you literally write: outputs = model(input)[0][:, 0, :] (which means: give me the numbers from the 0th token, which is CLS).
• Why BERT eventually “lost” to GPT: BERT is an incredible reader, but it is a terrible writer. You can’t have a conversation with BERT. You can only ask it to classify or fill in blanks. OpenAI realized that while understanding is great, humans actually want to talk to AI. So they took the Decoder (GPT), made it massive, and left BERT behind in the dust for consumer products. (Though BERT is still heavily used in the background of Google Search to this day!).

Remember how FastText and GloVe still couldn’t solve Polysemy? ‘Apple’ always got one vector regardless of context?

BERT solves this completely. Because BERT is bidirectional and context-aware, Apple’ in ‘I ate an Apple’ gets a completely DIFFERENT vector from ‘Apple’ in ‘Apple launched a new iPhone’. Same word, different meaning, different vector.

This is the breakthrough that changed everything.