How LLMs Really Read Text: A Hands-On Guide on BPE Tokenizer

We interact with Large Language Models (LLMs) like GPT-4 and Llama 3 every day, but their inner workings can feel like magic. Before any complex reasoning happens, every LLM must perform a crucial first step: tokenization, the process of converting human text into numbers it can understand.

In this guide, we’ll demystify this foundational process by building a Byte-Pair Encoding (BPE) tokenizer from scratch. You’ll see exactly how text is methodically broken down and compressed. By the end, you’ll even load the official GPT-2 vocabulary, proving your from-scratch tool can stand up to an industry giant.

The Problem: Why Not Just Use Characters?

The simplest way to turn text into numbers is to encode each character into its numerical byte value.

text = "This is some text"
byte_ary = bytearray(text, "utf-8")
ids = list(byte_ary)

print(ids)
# [84, 104, 105, 115, 32, 105, 115, 32, 115, 111, 109, 101, 32, 116, 101, 120, 116]

This approach is straightforward, but it’s highly inefficient. A short 17-character sentence becomes a sequence of 17 distinct token IDs.

This creates significant problems:

• Long, Unwieldy Sequences: Even moderately long texts result in massive input sequences for the model.
• Difficulty Learning Context: It becomes harder for the LLM to learn relationships and patterns across long distances.
• Computational Inefficiency: Processing these long sequences is slow and resource-intensive.

The Solution: Byte-Pair Encoding for Smart, Compact Tokens

Now, let’s see what a BPE tokenizer does with the same sentence. When run through GPT-2’s tokenizer, you get this:

“This is some text” → [1212, 318, 617, 2420]

Just 4 tokens instead of 17!

BPE finds the perfect middle ground by identifying and merging the most common character sequences into single, new tokens. It avoids the pitfalls of two other extremes:

• Character-level tokenization: Creates sequences that are too long.
• Word-level tokenization: Creates a brittle, gigantic vocabulary that can’t handle typos or new words.

The BPE Algorithm: How Does It Actually Work?

The beauty of BPE is its iterative simplicity. It starts with a basic set of characters and builds a rich vocabulary by repeatedly merging the most common adjacent pairs.

The High-Level Steps

1. Initialize with Characters: Start with a base vocabulary containing all 256 individual bytes.
2. Find the Most Frequent Pair: Scan your training text and count the occurrences of all adjacent token pairs to find the most common one.
3. Merge and Create a New Token: Replace every instance of that pair with a single, new token ID and add this new token to your vocabulary.
4. Repeat: Go back to step 2 and continue merging until you reach your desired vocabulary size.

A Detailed Mini-Example

Let’s trace this process with the training text: “the cat in the hat”.

Iteration 1: Merge (“t”, “h”)

1. Identify: The most frequent adjacent character pair is (“t”, “h”), which appears twice.
2. Replace & Record: We create a new token ID, 256, to represent the string “th”. We then replace all “th” sequences in our text.

• New Text: <256>e cat in <256>e hat
• Vocabulary Update: We add {256: “th”} to our vocabulary.

Iteration 2: Merge (<256>, “e”)

1. Identify: In our new text, the most frequent adjacent pair is the token <256> followed by the character “e”. This pair also appears twice.
2. Replace & Record: We create token ID 257 to represent this new sequence.

• New Text: <257> cat in <257> hat
• Vocabulary Update: We add {257: “<256>e”}. Note that this new token decodes back to “the”.

Iteration 3: Merge (<257>, “ “)

1. Identify: The most frequent pair is now the token <257> followed by a space “ “.
2. Replace & Record: We create token ID 258.

• New Text: <258>cat in <258>hat
• Vocabulary Update: We add {258: “<257> “}, which decodes to “the “.

This iterative process continues, building up a vocabulary of common subwords and, eventually, entire words from the training data.

Loading GPT-2’s vocabulary and merges

Libraries such as:

• **tiktoken** by OpenAI
• **Hugging Face tokenizers**

provide official vocabularies (encoder.json) and merge rules (vocab.bpe) for GPT-2.

By loading these files into your tokenizer (or simply using these libraries), you can check:

• Whether the same text produces the same token IDs.
• Whether decoding returns to the original text.

For example:

from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
input_text = "This is some text"
token_ids = tokenizer.encode(input_text)

print(token_ids)
# [1212, 318, 617, 2420]

Why This Matters

• It’s Not a Black Box: You now know the actual logic of how an LLM reads and compresses text.
• Better Troubleshooting: When a model gives a weird output, you’ll have a better gut feeling if tokenization is the problem.
• Real Engineering, Not Magic: You can see that powerful libraries are just smart applications of a simple, effective algorithm.

Conclusion

Tokenization is the essential first step for any LLM. BPE is the method that makes it work, finding a balance between using single characters (too long) and whole words (too rigid).

You now understand the core process: find the most common pair, merge it, and repeat. This isn’t just theory — it’s exactly how production models like GPT-2 were built. Now you know the first, critical step behind every LLM interaction.