I Built 3 Tokenizers From Scratch (And Finally Understand How LLMs Read Your Prompt)

I had heard the word a hundred times. I even used tokenizer.encode() without thinking about it. But this week, I sat down and build three tokenizers in Python from scratch. And it really changed the way I think about how language models work.

Here’s everything I learned, with code.

What Even Is Tokenization?

Before a language model can process text, it needs to convert words into numbers. Computers don’t understand “hello”. they understand 15496. Tokenization is that conversion process.

Think of it like a translation layer:

"Punit sharma" → [encode] → [26, 46, 40, 35, 46, 1, 44, 33, 28, 43, 39, 28] → [decode] → "Punit sharma"

Every LLM has a vocabulary, a fixed dictionary of tokens it knows. When text comes in, it gets mapped to IDs from that vocabulary. When output comes out, those IDs get mapped back to text.

Simple concept. Wild implications.

Approach 1: Character-Level Tokenization

The most granular approach. Every single character is a token.

import string

letters = list(string.ascii_lowercase + string.ascii_uppercase + " ")
special = ["[PAD]", "[UNK]"]
vocab = special + letters

char2id = {char: idx for idx, char in enumerate(vocab)}
id2char = {idx: char for idx, char in enumerate(vocab)}

def encode(text):
    ids = []
    for char in text:
        if char in char2id:
            ids.append(char2id[char])
        else:
            ids.append(char2id["[UNK]"])
    return ids

def decode(ids):
    chars = []
    for id in ids:
        if id in id2char:
            chars.append(id2char[id])
        else:
            chars.append(id2char["[UNK]"])
    return "".join(chars)

How it works:

• Build a vocab of all lowercase + uppercase letters + space
• Add two special tokens: [PAD] (padding) and [UNK] (unknown)
• Each character gets a unique integer ID
• encode("Punit") → list of IDs, decode(ids) → back to string

Pros:

• Tiny vocabulary (just ~54 tokens in this case)
• Never hits an unknown word as every character exists
• Works across languages if you extend the alphabet

Cons:

• Sequences get very long “hello” becomes 5 tokens
• No semantic meaning, the model has to learn everything from scratch
• Bad at understanding word boundaries

Where it’s used: Some early NLP models, speech recognition systems, handwriting recognition

Approach 2: Word-Level Tokenization

Now we go the other direction - each word is a token.

corpus = [
    "The quick brown fox jumps over the lazy dog.",
    "Tokenization converts text to numbers",
    "Large language models predict the next token"
]

PAD, UNK = "[PAD]", "[UNK]"

words = set()
for sentence in corpus:
    words.update(sentence.lower().split())

vocab = [PAD, UNK] + list(words)

word2id = {w: i for i, w in enumerate(vocab)}
id2word = {i: w for i, w in enumerate(vocab)}

def encode(sentence):
    ids = []
    for word in sentence.lower().split():
        if word in word2id:
            ids.append(word2id[word])
        else:
            ids.append(word2id[UNK])
    return ids

def decode(ids):
    return " ".join(id2word.get(id, UNK) for id in ids)

How it works:

• Scan your corpus and collect every unique word into a set
• Assign an ID to each word
• Unknown words (not in training corpus) get mapped to [UNK]

Pros:

• Sequences are short, one token per word
• Preserves word level meaning
• Works across languages if you extend the alphabet

Cons:

• Vocabulary explodes because English has 170,000+ words
• [UNK] problem: any new word is a black box
• “run”, “running”, “ran” are completely unrelated tokens
• Terrible on names, slang, code, or multilingual text

Where it’s used: Older NLP systems like Word2Vec, early sentiment classifiers

Approach 3: Subword Tokenization (The One LLMs Actually Use)

This is where it gets interesting.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

def encode(text):
    return tokenizer.encode(text)

def decode(ids):
    return tokenizer.decode(ids)

print(encode("Punit sharma"))
# → [47, 20992, 427, 9998, 1689, 559]  (example output)

Yes, that’s it. One import, and you’re using the same tokenizer GPT-2 was trained on.

But the magic is what’s happening under the hood: GPT-2 uses Byte Pair Encoding (BPE).

How BPE works (conceptually):

1. Start with individual characters as your vocabulary
2. Count the most frequent pairs of adjacent tokens in your corpus
3. Merge the most frequent pair into a new token
4. Repeat until you hit your vocabulary size limit (GPT-2 uses ~50,000 tokens)

So a word like "tokenization" might become ["token", "ization"] , two subword pieces. You get the benefits of both worlds:

• Common words like "the" stay as single tokens (efficiency)
• Rare words like "tokenization" get split into known pieces (flexibility)
• No true unknowns, you can always fall back to characters

Pros:

• Handles rare and unknown words gracefully
• Vocabulary stays manageable (50k is the sweet spot for most models)
• Works across languages
• Preserves morphological structure (“un” + “happy” = “unhappy”)

Cons:

• Less interpretable token boundaries don’t always make linguistic sense
• Tokenization is model-specific, GPT-2 and Llama use different tokenizers

Where it’s used: GPT-2, GPT-3, GPT-4, LLaMA, BERT (uses WordPiece, a BPE variant), almost every modern LLM

The Bigger Picture

Here’s what hit me after building all three:

Tokenization is where language meets math. Before any attention mechanism, any transformer layer, any embedding, the model has to agree on what a unit of text even is. That decision shapes everything downstream.

GPT-4’s behavior on code vs prose? Partially tokenization. Why LLMs struggle with counting letters in words? Tokenization. Why some languages are more expensive to process? Tokenization.

What I’m Building On Top of This

I’m building toward a full LLM Playground project. The tokenization piece feeds into understanding embeddings, then attention, then the full transformer stack.

If you’re on a similar journey, learning AI engineering from a software dev background, I’ll keep posting what I build and learn each week.

Drop a comment if you’ve got questions, or if you’re doing something similar. Always down to connect with people building in this space

It’s the foundation, and most people (including past me) skip right over it

Find me on X: punitmudgal_