Tech Blog #10 — Byte Pair Encoding (BPE)

I understand everything, I can decode your typos

Courtesy: Google Images

The Core Problem: The “Unknown Word” Trap

In the old days (Word2Vec, GloVe), AI split text strictly by spaces.

• The Flaw: The word “Hamburger” and a typo like “Hamburge” were treated as two completely different, unrelated words. If the AI saw “Hamburge” in a sentence, it had zero idea what to do because it wasn’t in its dictionary (this is called an Out-of-Vocabulary or OOV error).
• The Second Flaw: “Low”, “Lower”, and “Lowest” were three totally different dictionary entries. The AI didn’t inherently know they shared the same root word.

The BPE Solution

Byte Pair Encoding is a data compression algorithm that AI researchers borrowed to build a better dictionary. Instead of starting with a dictionary of whole words, BPE starts with a dictionary of single characters (a, b, c, z, spaces, punctuation). It then scans millions of documents, finds the two characters that sit next to each other most frequently, and glues them together forever into a new token. It repeats this until it hits a desired dictionary size (e.g., 50,000 tokens).

The Result

“Low”, “Lower”, and “Lowest” are no longer strangers. They are broken down into sub-word pieces (e.g., low + er + est). The AI learns the concept of low once, and applies it everywhere. And if you type "Hamburge" (missing the 'r'), the AI just breaks it into Ham + burge, and still understands what you mean because the pieces are in its dictionary!

Text-Based Infographics

The “Low, Lower, Lowest” BPE Factory

Watch how BPE builds its vocabulary step-by-step by looking at a training corpus:

STEP 1: The Character Dictionary (Size: 26 letters + space)
-------------------------------------------------------------
Dictionary: [a, b, c... l, o, w, e, r, s, t, _]
Text in corpus: "low", "lower", "lowest"

STEP 2: The First Scan (Find most frequent character pair)
-------------------------------------------------------------
The pair "l" and "o" appear next to each other ALL THE TIME.
Action: Glue them together!
Dictionary: [..., lo, w, e, r, s, t, _]
Text is now: "lo w", "lo w er", "lo w er est"

STEP 3: The Second Scan (Find next most frequent pair)
-------------------------------------------------------------
The pair "lo" and "w" appear often.
Action: Glue them together!
Dictionary: [..., low, e, r, s, t, _]
Text is now: "low", "low er", "low er est"

STEP 4: The Third Scan
-------------------------------------------------------------
The pair "low" and "er" appear often.
Action: Glue them together!
Dictionary: [..., lower, s, t, _]
Text is now: "low", "lower", "lower est"

STEP 5: The Final Scan
-------------------------------------------------------------
The pair "lower" and "est" appear.
Action: Glue them together!
Dictionary: [..., lowest, _]
Text is now: "low", "lower", "lowest"
THE MAGIC: 
The AI doesn't have 3 separate words anymore. 
It understands "lower" is just [low] + [er].
It understands "lowest" is just [low] + [er] + [est].

Old Way vs. BPE Way

INPUT: "Unbelievably good"

OLD WAY (Space Splitting):
Tokens: ["Unbelievably", "good"]
Problem: What if the user types "unbeliveably" (typo)? 
         -> [UNK] [good] (AI breaks).

BPE WAY (Sub-word Splitting):
Tokens: ["Un", "believ", "ably", "good"]
Typo: "unbeliveably" 
Tokens: ["un", "belive", "ably", "good"]
Result: The AI still understands 80% of the word because "Un", 
        "ably", and "good" are in its dictionary!

The Layman Explanation

The “Lego Brick” Analogy:

Imagine you are an architect, and your supplier only ships you single, tiny Lego blocks (individual letters).

Building a word like “Hamburger” takes 9 tiny blocks. It works, but it’s tedious.

One day, you look at your orders and realize, “Wow, I build the sequence ‘ham’ and ‘bur’ and ‘ger’ almost every single day.”

So, you go to your factory and use superglue to permanently fuse those tiny blocks together into larger, custom Lego bricks labeled [ham] and [bur] and [ger].

Now, your dictionary doesn’t just have letters; it has these custom sub-words.

• When someone orders a “Hamburger”, you grab your [ham], [bur], and [ger] bricks.
• If someone orders a “Hamster”, you grab [ham] and [ster].
• If someone makes a typo and orders a “Hamburge”, you just grab [ham], [bur], and [ge]. You don't have the exact brick, but you can still build 90% of the toy using your custom pieces!

That is Byte Pair Encoding. It dynamically creates a custom dictionary of “sub-word bricks” based on what appears most often in the data, ensuring it almost never encounters a word it can’t break down.

What This Means for US in Human Terms

This is the secret behind ChatGPT’s weird counting.

Have you ever noticed that if you paste a massive paragraph into ChatGPT and ask it to summarize it, ChatGPT might say, “That was 85 tokens” even though it was only 60 words?

Now you know why.

• Common words like “The” or “apple” might be exactly 1 token.
• A complex word like “Unbelievably” might be broken into 3 tokens: [Un], [believ], [ably].
• A massive word like “Supercalifragilisticexpialidocious” might be broken into 8 tokens!
• Even spaces and punctuation are tokens.

Why GPT uses BPE instead of whole words:

1. No Typos: As we saw, typos don’t break the AI anymore. It just reads the sub-words.
2. No Unknown Words: You can feed GPT-4 a made-up chemical formula, a piece of code, or Klingon, and it won’t return an error. It will just break it down into characters or syllables and try to process it.
3. Multilingual Magic: BPE doesn’t care about English. It will see that “chat” and “chien” (French for dog) both have “ch”, and it will glue “ch” into a token. It inherently learns cross-language roots without being explicitly taught!

Quick Summary

• Origins: Originally invented in 1994 for data compression (making files smaller).
• Adopted by AI: Popularized for NLP in 2016 by Rico Sennrich.
• GPT Variants: OpenAI doesn’t use pure vanilla BPE for ChatGPT; they use a slightly upgraded version called TikToken (Byte-Pair Encoding mixed with byte-level fallbacks), but the core logic is identical.
• The Trade-off: A vocabulary size of 50,000 tokens is standard. If it’s too small (1,000), every word gets chopped into tiny pieces and the AI is slow. If it’s too big (1,000,000), the model requires massive amounts of RAM to hold the dictionary. 50,000 is the sweet spot.

What is the Scanning mentioned in the BPE Process?

The “Who”

• Not the Transformer: The GPT model (with its massive Q, K, V matrices) has absolutely nothing to do with this process.
• The Engineers: A human data scientist at OpenAI writes a simple script.
• The Actor: A standard Python for loop running on a regular CPU.

The “What” (What the script actually does)

It is literally just a Find and Replace function combined with a Frequency Counter.

1. Count: The script opens a giant text file (like all of Wikipedia). It slides a window over every two characters and counts them in a database. (e.g., “l-o” appears 500,000 times; “q-z” appears 2 times).
2. Find: It finds the pair with the highest score (“l-o”).
3. Replace: It uses a basic text editor command: text.replace("l o", "lo").
4. Repeat: It does this a few thousand times until it has built a dictionary of 50,000 sub-words.
5. Save: It saves that dictionary to a file and throws the script away.

The “When”

This entire process happens only once, months before the AI is ever trained. It does not happen while you are chatting with ChatGPT.

Text-Based Infographics

The Factory Floor Analogy (Who is doing it?)

THE AI (Transformer Brain): 
I only understand numbers (Token IDs). I have no idea what letters are.

THE DATA SCIENTIST:
I need to turn our English text into those Token IDs for the AI. 
I am going to run a Python script to build a translation dictionary.

THE PYTHON SCRIPT (The "Scanner"):
I am a dumb algorithm. I don't understand English. 
I just have a clipboard and a Find/Replace tool.
Step 1: I see 'l' next to 'o' 500,000 times. I'm writing 'lo' in my dictionary.
Step 2: I see 'lo' next to 'w' 200,000 times. I'm writing 'low' in my dictionary.
Step 3: Done. Here is the final dictionary file.

THE AI:
Great, I will memorize this dictionary. From now on, if you ever 
give me the word 'lower', I will look it up and know it is Token #402.

The Exact Timeline of a Word

PHASE 1: SETUP (Months before training)

Raw Text: "low lower lowest"
  │
  ▼
[ PYTHON SCRIPT RUNS HERE ]
  │ (Just counting characters and gluing them together)
  ▼
Dictionary File Created: { "low": 402, "er": 104, "est": 89 }

PHASE 2: TRAINING (During GPT training)

The AI reads books, but it reads them as numbers using the Dictionary File.
PHASE 3: INFERENCE (When you type in ChatGPT)
You type: "I feel low"
  │
  ▼
[ TOKENIZER LOOKS UP DICTIONARY ]
  │ "I" -> 45  |  "feel" -> 1200  |  "low" -> 402
  ▼
AI receives: [45, 1200, 402]

The Layman Explanation

The “Post Office Sorting Machine” Analogy:

Imagine a post office that needs to sort mail by city.

The Postmaster (the engineer) buys a giant, mechanical sorting machine (the Python script). He turns it on, dumps 10 million letters into it, and lets it run for 3 days. The machine doesn’t read the letters. It just looks at the envelopes and physically sorts them into different bins based on the zip code.

Once the 3 days are over, the machine is turned off.

Now, when a new letter arrives at the post office, the mailman (the AI) doesn’t need the giant sorting machine anymore. The mailman just looks at the zip code on the envelope, looks at the sign above the bin (the Dictionary File), and puts the letter in the right box.

This is exactly how BPE works. The “scanning” isn’t an AI process. It’s just the mechanical sorting machine running during setup to build the dictionary. Once the dictionary is built, ChatGPT never “scans” for character pairs ever again. It just looks up the pre-calculated results.

What This Means for US in Human Terms

Why this distinction is crucial for your understanding:

• BPE has zero “intelligence”: BPE doesn’t know that “er” is a suffix. It doesn’t know what “low” means. It just blindly counted that “e” and “r” sit next to each other a lot. It is pure statistics, not neural network magic.
• Why ChatGPT’s tokenizer feels “dumb” sometimes: Have you ever noticed that if you type “apple” in ChatGPT, it’s one token, but if you type “Apple” (capital A), it might be split into two weird tokens? That’s because the BPE script ran on mostly lowercase internet text. It glued lowercase “a-p-p-l-e” together, but it rarely saw uppercase “A-p-p-l-e”, so it never glued them. The script wasn’t smart enough to know they were the same word; it just counted characters.
• The “Tiktoken” library: When OpenAI built ChatGPT, they open-sourced their BPE script. It’s a publicly available Python library called **tiktoken**. You can actually download it, run it on your laptop in 1 second, and see exactly how it chops up words using the exact same dictionary ChatGPT uses.

So, when you hear “BPE,” don’t picture a brain thinking. Picture an Excel spreadsheet running a **COUNTIF** formula!