Tokenization Deep Dive : The Hidden System Behind Every LLM

A beginner-friendly guide to the invisible step that happens before every LLM understands your text.

Hey everyone! 👋

Nonmembers click here

Have you ever wondered why an AI can write code, explain complex topics, and even create stories but sometimes struggles to count letters in a simple word or solve basic arithmetic?

It might sound surprising, but the answer starts before the AI even reads your text.

Welcome to the world of tokenization.

Tokenization is the very first step that happens before a language model processes anything you type. Instead of reading text the way humans do, the model first breaks your sentence into smaller pieces called tokens.

This one step influences almost everything that happens afterward. It affects how the model understands language, why it sometimes makes unexpected mistakes, how much context it can remember, and even how much you pay for every API request.

In other words, tokenization is the invisible foundation behind every Large Language Model (LLM). Once you understand how it works, many AI behaviors that seem confusing suddenly start to make sense.

So, let’s dive in and explore one of the most important but often overlooked concepts in modern AI.

LLMs Don’t Read Words, They Read Tokens

When we read a sentence, we see words. But a language model does not read text exactly like humans do. Before it can understand anything, your text is first broken into smaller pieces called tokens. A token can be a full word, part of a word, a symbol, a space, or even punctuation.

For example, a sentence like:

“I love tokenization.”

may be broken into pieces like:

I | love | token | ization | .

After this, each token is converted into a number. The model does not directly process the word “token.” It processes the number that represents that token. This step matters a lot.

If we made every single character a token, the text would become too long. Even a small paragraph could turn into hundreds of tokens, which would make the model slower and more expensive to run.

But if we made every full word a token, the model would need a huge vocabulary. It would also struggle with new words, names, spelling mistakes, programming code, and languages that were not seen often during training.

That is why modern LLMs use something in the middle: subword tokenization. Subword tokenization breaks text into pieces that are bigger than characters but smaller than full words. Common words like “the” may stay as one token.

But rare or longer words like “tokenization” may be split into smaller parts like:

token + ization

This gives the model a flexible way to handle both common and uncommon text. So, before an LLM writes an answer, explains an idea, or generates code, it first sees your text as tokens. Tokens are the small building blocks that every language model depends on.

Byte-Pair Encoding: How LLMs Learn Token Pieces

Now let’s talk about Byte-Pair Encoding, or simply BPE. BPE is one of the most common methods used to create tokens for language models.

The idea is simple:

Start with very small pieces, then keep joining the pieces that appear together most often. For example, if the letters t and h appear together many times, BPE may merge them into one piece: th. Then if th and e appear together often, it may merge them again into the. So instead of storing every possible word, BPE slowly learns useful text pieces from real data.

The process looks like this:

1. Start with small units like characters or bytes
2. Count which pieces appear together most often
3. Merge the most common pair into one new token
4. Repeat this many times

After training, these merge rules are saved. Later, when you type new text into an LLM, the tokenizer applies the same learned rules to break your text into tokens. This is why common words usually stay as one token, while rare or longer words may be split into smaller parts.

In simple words:

BPE teaches the model how to break language into useful pieces.

Here is a simple BPE training example in Python. We use a tiny text corpus with repeated words and watch how BPE slowly learns useful subword pieces step by step :

from collections import Counter

sample_text = (
    "low low low low low "
    "lowest lowest "
    "newer newer newer newer newer newer "
    "wider wider"
)

words = sample_text.split()
word_counts = Counter(words)

bpe_vocab = {}

for word, count in word_counts.items():
    pieces = tuple(word) + ("</w>",)
    bpe_vocab[pieces] = count

def collect_adjacent_pairs(vocab):
    pair_counts = Counter()

    for pieces, count in vocab.items():
        for left, right in zip(pieces, pieces[1:]):
            pair_counts[(left, right)] += count

    return pair_counts

def apply_merge(vocab, target_pair):
    merged_token = "".join(target_pair)
    updated_vocab = {}

    for pieces, count in vocab.items():
        new_pieces = []
        index = 0

        while index < len(pieces):
            can_merge = (
                index < len(pieces) - 1
                and pieces[index] == target_pair[0]
                and pieces[index + 1] == target_pair[1]
            )

            if can_merge:
                new_pieces.append(merged_token)
                index += 2
            else:
                new_pieces.append(pieces[index])
                index += 1

        updated_vocab[tuple(new_pieces)] = count

    return updated_vocab

print("Word counts:")
print(dict(word_counts))
print()

print("Starting vocabulary:")
for pieces, count in sorted(bpe_vocab.items(), key=lambda item: -item[1]):
    print(f"  {' '.join(pieces):25s} x{count}")

print()

learned_merges = []

for round_no in range(1, 11):
    pair_counts = collect_adjacent_pairs(bpe_vocab)

    if not pair_counts:
        break

    most_common_pair, frequency = pair_counts.most_common(1)[0]

    bpe_vocab = apply_merge(bpe_vocab, most_common_pair)
    learned_merges.append(most_common_pair)

    merged_text = "".join(most_common_pair)

    print(
        f"Merge {round_no:2d}: "
        f"{most_common_pair[0]!r} + {most_common_pair[1]!r} "
        f"-> {merged_text!r}  (frequency={frequency})"
    )

print()

print("Vocabulary after 10 merges:")
for pieces, count in sorted(bpe_vocab.items(), key=lambda item: -item[1]):
    print(f"  {' '.join(pieces):25s} x{count}")

print()

print("Encoding a new word: lowest")
encoded_word = list("lowest") + ["</w>"]

print("Start:", encoded_word)

for merge_pair in learned_merges:
    merged_token = "".join(merge_pair)
    encoded_result = []
    index = 0

    while index < len(encoded_word):
        can_merge = (
            index < len(encoded_word) - 1
            and encoded_word[index] == merge_pair[0]
            and encoded_word[index + 1] == merge_pair[1]
        )

        if can_merge:
            encoded_result.append(merged_token)
            index += 2
        else:
            encoded_result.append(encoded_word[index])
            index += 1

    encoded_word = encoded_result

print("Final:", encoded_word)

What the BPE Output Shows

Notice what happened here.

The word “newer” appeared many times, so BPE learned it as one strong token. The word “low” also became one familiar piece. Because of that, when BPE sees “lowest”, it does not treat the whole word as unknown. It breaks it like this:

[“low”, “e”, “s”, “t”]

So the model already understands “low” as a known piece, then reads the remaining letters separately. This is the basic idea behind real tokenizers too: they handle rare words by breaking them into smaller familiar parts.

Imp note : BPE is basically a learned compression system. It learns which text pieces should be merged based on the data used to train the tokenizer. But one important thing to remember: the tokenizer’s training data and the model’s training data are not always exactly the same. This small mismatch can create strange behavior, including something called glitch tokens, which we will discuss later.

Byte-Level BPE: Why Unknown Tokens Almost Disappeared

The first version of BPE worked mostly with characters. That was useful, but it still had one problem: what happens when the tokenizer sees a symbol, emoji, or character it has never seen before?

Earlier systems often used an unknown token, usually written as **<UNK>**.

*In 2019, the GPT-2 paper by Radford et al. introduced Byte-Level BPE, which solved this problem in a much smarter way.*

Byte-level BPE solved this problem in a smarter way. Instead of starting from characters, it starts from raw bytes. A byte-level tokenizer begins with 256 basic byte values. Since almost any text can be converted into bytes using UTF-8, the tokenizer can represent almost anything. English text, Hindi text, Chinese text, Arabic text, emojis, code, symbols everything can be broken into bytes first. That means the tokenizer does not need to panic when it sees something new.

It can always break the input into smaller byte pieces. Modern tokenizers also use a step called pre-tokenization. This step lightly separates text before BPE starts merging pieces. For example, it may keep contractions, numbers, spaces, symbols, and different language scripts separate in a cleaner way.

So in simple words:

Byte-level BPE made tokenizers more flexible, and pre-tokenization made the splitting process cleaner.

Three Tokenizer Families Commonly Used in Production LLMs

Many production LLMs commonly rely on one of these major tokenizer ecosystems. Understanding them helps when you are picking models, estimating API cost, or debugging unexpected token behavior.

This matters because tokenizers affect many practical things, like:

• how fast text is processed
• how API cost is calculated
• how different languages are handled
• why some text gets split in strange ways

1. tiktoken

tiktoken is OpenAI’s tokenizer library. It is very fast and is mainly used for inference, not for training a new tokenizer from scratch. It powers OpenAI models and is also used in some other modern model ecosystems.

2. SentencePiece

SentencePiece is widely used by Google model families like Gemini and Gemma. One interesting thing about SentencePiece is that it treats text more like a raw stream instead of depending heavily on spaces between words. That makes it useful for languages where word boundaries are not always clear, like Chinese, Japanese, and Thai.

3. Hugging Face Tokenizers

Hugging Face Tokenizers is one of the most flexible options. It supports different methods like BPE, WordPiece, and Unigram. It also supports training your own tokenizer. That is why many open-source models use it.

Simple way to remember it

tiktoken is great for speed. SentencePiece is great for multilingual text. Hugging Face Tokenizers is great for flexibility.

Imp note: According to Kudo’s 2018 Unigram language model tokenizer work, Unigram works differently from BPE. BPE starts with small pieces and keeps merging them, while Unigram starts with many possible token pieces and removes the least useful ones step by step. Because of this, Unigram can often preserve meaningful word endings like -ing, -ly, and -tion more clearly, which makes it useful for languages with complex word forms.

Vocabulary Sizes Are Getting Bigger

Tokenizer vocabularies have become much larger in newer LLMs.

The reason is simple: a larger vocabulary can represent more text pieces directly, so the same sentence may need fewer tokens. Fewer tokens can reduce the work needed during attention and can also improve multilingual coverage. For example, OpenAI’s o200k_base encoding supports around 200k tokens, while Gemma 3 uses a 262k SentencePiece vocabulary that Google says is better balanced for non-English languages. The tradeoff is that larger vocabularies also need larger embedding tables, so they are not “free.”

In short, vocabulary size has grown a lot.

Older models like Llama 2 used around 32K tokens, while newer model families like Gemma 3 / Gemini-style tokenizers use around 262K tokens. This growth is not random. A 2024 NeurIPS paper by Tao et al. found that larger models often need larger vocabularies for better compute efficiency. Their study predicted that while 32K was reasonable for smaller Llama 2 models, the 70B version could have benefited from a vocabulary of at least 216K tokens.

So the simple takeaway is:

As models get larger and more multilingual, bigger vocabularies can help reduce token length and improve training efficiency — but they also increase embedding size.

The Vocabulary Size Tradeoff

Choosing vocabulary size is a big decision when building an LLM. A larger vocabulary means the model has more token options to choose from every time it predicts the next token.

In simple terms:

A model with a 262K vocabulary has to compare many more possible tokens than a model with a 32K vocabulary.

So yes, bigger vocabulary can make the final prediction step heavier. But there is another side. A larger vocabulary can also represent text using fewer tokens. That means the sentence becomes shorter after tokenization. And shorter token sequences reduce the work inside the attention mechanism, because attention becomes expensive very quickly as the sequence gets longer.

So the tradeoff is:

Bigger vocabulary = more choices per token Shorter sequence = less attention work

In practice, larger vocabularies can often help overall efficiency, especially for large and multilingual models. The model may spend more work choosing from a bigger vocabulary, but it can save compute because fewer tokens need to move through the transformer.

Five Ways Tokenization Can Break Your Model

Tokenization looks simple, but it is not perfect. The way text gets split into tokens can create real problems for a language model. It can affect accuracy, cost, speed, and even how fairly the model handles different languages or writing styles. In this section, let’s look at five common ways tokenization can cause unexpected behavior.

Arithmetic and Number Tokenization

Tokenization can also affect basic math.

For example, an LLM may struggle with a problem like:

1,234 + 5,678

Not because the model has never seen addition, but because numbers are not always split in a clean digit-by-digit way. One number may become a single token, while another number may be split into different pieces. Because of this, the model does not always see digits lined up the way humans do when we do column addition.

A 2024 paper by ***Singh and Strouse* showed that number tokenization has a real impact on arithmetic performance. They found that right-to-left number grouping**, often created by adding commas, can significantly improve arithmetic accuracy for GPT-3.5 and GPT-4.

Simple takeaway: tokenization can make math easier or harder for an LLM depending on how numbers are split.

The Multilingual Token Tax

The same sentence can cost different amounts in different languages.vEnglish is usually tokenized very efficiently because most tokenizers are heavily optimized around English and high-resource languages. But many low-resource or morphologically rich languages can be split into many more tokens for the same meaning.

This creates a multilingual token tax.

More tokens means:

• more context window usage
• more compute
• more latency
• higher API cost

It can also affect quality. Research by **Lundin et al.** on African languages found that higher token fertility, meaning more tokens per word, is linked with lower model accuracy. In simple terms, when a language needs more tokens to say the same thing, the model may pay more compute and still perform worse.

Glitch Tokens: When One Token Confuses the Model

Sometimes a tokenizer learns a token that the model itself does not understand well. A famous example is “SolidGoldMagikarp.” In early 2023, researchers Jessica Rumbelow and Matthew Watkins found that this token could make GPT models behave strangely. The likely reason was a mismatch: the tokenizer had seen this text often enough to store it as one token, but the language model had not learned a strong meaning for it during training. So when the model saw that token, its embedding was weak or noisy.

This problem is called a glitch token.

Recent research also shows that glitch tokens are still a real issue. The **GlitchMiner** paper introduced a method for finding these tokens by searching for inputs that create high uncertainty in model predictions.

Code Formatting Waste

Tokenization can also waste tokens in code. Spaces, indentation, and new lines may look small to us, but they still become tokens. In programming languages, this formatting can take a noticeable part of the total token budget. This becomes even bigger with structured outputs like JSON, where quotes, brackets, commas, and indentation also consume tokens. A 2025 study by **Pan et al.** found that formatting overhead can be significant in code, with languages like Java and C# losing a measurable percentage of tokens to pure formatting.

Token Boundary Misalignment

Sometimes the model expects text to be split in one way, but the prompt creates token boundaries in a different way. This is called token boundary misalignment. When this happens, the model may become less confident about the next token, and its output quality can drop. This problem is especially noticeable in languages like Chinese, where words are not separated by spaces in the same way as English. To reduce this issue, some tools use a technique called token healing. For example, Microsoft’s Guidance library can move back from a partial token and continue generation from a cleaner token boundary.

The Rise of Byte-Level Models

Researchers are now exploring a new idea: what if language models did not need traditional tokenization at all?

Instead of breaking text into words or subwords, these models work directly with raw bytes.

The goal is simple: remove many of the limitations caused by tokenization.

If successful, byte-level models could reduce problems like inconsistent number splitting, multilingual token inequality, glitch tokens, and token boundary issues. While this is still an active area of research, it shows that tokenization is still evolving, and future language models may rely much less on the tokenizers we use today.

🔹 ByT5 by **Xue et al. (2022) showed that this idea can work. Instead of using normal tokens, ByT5 processes text directly as byte sequences**. This proved that byte-level models can compete with token-based models.

But there is one big problem:

Byte sequences are usually much longer than token sequences, often around 4–5x longer. That makes attention much more expensive, because the model has to process many more positions.

🔹 MEGABYTE, introduced by **Yu et al. (2023) from Meta, took a different approach to solving the long byte sequence problem. Instead of using one transformer for all bytes, it uses two transformers**:

• A global transformer first reads large chunks (patches) of bytes.
• A local transformer then processes the individual bytes inside each chunk.

This two-level design lets the model handle very long byte sequences much more efficiently than a standard transformer.

🔹 SpaceByte, introduced by Slagle (2024) at NeurIPS 2024, improved this idea in a smarter way. Instead of splitting bytes into fixed-size chunks, SpaceByte uses spaces as natural boundaries. In simple terms, it gives more attention after space characters, because spaces often mark where new words begin. This helped SpaceByte perform close to normal subword-based transformers on English text and code.

Byte Latent Transformer (BLT) by **Pagnoni et al. (2024)** is one of the most important recent ideas in this direction.

Instead of using a fixed tokenizer, BLT works directly on raw bytes. But it does not treat every byte equally. It uses a small byte-level model to decide where the text is easy or hard to predict. Easy parts, like common words, are grouped into larger patches. Hard parts, like rare words, code, numbers, or spelling variations, get smaller patches and more attention.

This makes BLT more flexible than normal tokenizers because there is no fixed vocabulary. It can handle typos, new words, and unusual text more naturally.

The 2025–2026 Frontier in Tokenization Research

Tokenization is still evolving. While Byte-Level models are gaining attention, researchers are also improving existing subword tokenization methods. New techniques aim to make tokenizers faster, more efficient, fairer across languages, and better at handling code, numbers, and long contexts. In this final section, we’ll look at some of the most interesting tokenization research from 2025–2026 and see where the future of LLM tokenization is heading.

🔹SuperBPE, introduced at COLM 2025, improves normal BPE by adding a second pass. First, it learns regular subword tokens. Then, it learns larger “superword” tokens that can cross spaces, such as common phrases or word groups. This helps the same text fit into fewer tokens. The paper reports up to 33% fewer tokens, around 27% less inference compute, and a +4.0% average improvement across 30 tasks, including +8.2% on MMLU.

🔹 BoundlessBPE, introduced at **COLM 2025, removes one limitation of normal BPE: it does not force token merges to stop at word boundaries. This means it can merge useful pieces even across spaces when that improves compression. The result is fewer tokens for the same text, with reported gains of up to 15% better bytes per token and 3–5% higher Rényi efficiency**.

🔹 LiteToken, introduced in **February 2026, is more like a cleanup method for existing BPE tokenizers. It finds tokens called intermediate merge residues **tokens that were useful during BPE training but are rarely used in the final tokenized output. The paper reports that these residue tokens waste vocabulary space and can make tokenizers weaker on noisy or misspelled text. LiteToken removes these low-use tokens without needing full retraining, helping reduce fragmentation and improve robustness.

🔹 Dynamic tokenization is also becoming an important research direction. Instead of using one fixed tokenizer forever, dynamic methods try to adjust tokenization based on the model, input, or training feedback.

For example, **ADAT from NeurIPS 2024 refines the vocabulary using model feedback during training. Another ACL 2025 work, Retrofitting LLMs with Dynamic Tokenization**, shows that existing language models can use more flexible token boundaries after training, reducing sequence length while keeping performance mostly stable.

Researchers are also studying tokenization from a theoretical perspective. A NeurIPS 2024 paper by **Rajaraman et al.** showed that tokenization is not just a compression trick. Their work suggests that tokenization can actually make certain language patterns easier for transformers to learn. In some cases, the model could not efficiently learn these patterns without tokenization, but it could once tokenization was introduced.

When to Use Each Tokenization Strategy

The best tokenizer depends on what you are building.

Imp tip: Prompt caching can reduce cost when you send the same large prompt prefix again and again, such as system prompts, tool instructions, or long context blocks. OpenAI says prompt caching can reduce input token cost by up to 90%, and Google also offers discounted cached tokens. So, when you combine caching with a tokenizer that produces fewer tokens, production workloads can become much cheaper.

Practical Cost Implications

Tokenization also affects your API bill.

Most LLM providers charge based on tokens. So even if two models receive the same text, the final cost can be different because their tokenizers may split that text differently.

For example, around 1,000 English words may become roughly 1,300 tokens with a large modern tokenizer, but 1,500+ tokens with a smaller vocabulary tokenizer.

For non-English text, the difference can be much bigger. The same Arabic or Hindi paragraph may use far more tokens with one tokenizer than another.

This matters because output tokens are usually more expensive than input tokens. So a better tokenizer can save money on both sides: the text you send and the text the model generates.

If this story helped you, you might enjoy my collection of beginner-friendly AI and System Design articles. You can explore them here. 👇

https://medium.com/@Deep-concept/list/core-software-engineering-concepts-explained-in-simple-words-a11e143c8bc4

Conclusion

Tokenization is one of the most important parts of the LLM stack, but most people ignore it.

Many common LLM problems start here: math mistakes, letter-counting errors like “how many r’s in strawberry,” high API costs for non-English text, strange glitch tokens, and unexpected model behavior.

The field is now changing fast. BPE has been useful for years, and newer ideas like SuperBPE and LiteToken are improving it further. At the same time, byte-level models like Meta’s BLT show that future models may work with raw bytes instead of fixed tokenizers.

For builders, the takeaway is simple:

Tokenization is not just a preprocessing step. It is a real system design choice.

The tokenizer you choose affects accuracy, cost, multilingual fairness, speed, and reliability. So if you are working with LLMs, understanding tokenization is no longer optional.

Thanks for Reading! ❤️

If you made it this far, thank you for spending your time with me. I hope this story helped you understand tokenization in a simple and beginner-friendly way.

If you found this article useful, please consider leaving a few claps, sharing it with others, and reposting it so more people can learn from it.

And if you’d like to see more beginner-friendly stories on AI, LLMs, System Design, RAG, Agents, and Software Engineering, don’t forget to follow me.

I’d also love to hear your thoughts! If you have any questions, suggestions, or if you think I explained something incorrectly, please leave a comment below. Your feedback helps me improve and create better content for everyone.

See you in the next story! 🤗

Editor’s: This story is based on my own learning, research, and understanding. I also used AI for grammar improvements and creating some illustrations to make the content easier to understand. The primary research resource that inspired this **paper**.