Working with Tokenizers

A tokenizer is the first step in how an AI language model understands text. It breaks a sentence into smaller units called tokens, which may represent whole words, parts of words, punctuation, or even spaces. These tokens are then converted into numerical representations that the model can process, making tokenization a fundamental part of how modern AI systems read, interpret, and generate human language. A model also contains a vocabulary that includes special tokens to indicate the LLM start of a prompt, or an end, etc. The tokenization approach of different models could be different, i.e., they all don’t need to follow the same approach.

Before using a tokenizer, we need to import a few libraries that help us authenticate with Hugging Face and load a pre-trained tokenizer.

from google.colab import userdata
from huggingface_hub import login
from transformers import AutoTokenizer

• **userdata**: Accesses securely stored secrets (such as Hugging Face tokens) in Google Colab.
• **login**: Authenticates your notebook with your Hugging Face account.
• **AutoTokenizer**: Automatically loads the correct tokenizer for a specified pre-trained model.

Next, we retrieve the Hugging Face access token stored securely in Google Colab and use it to log in to the Hugging Face Hub.

hf_token = userdata.get('HF_TOKEN')
login(hf_token, add_to_git_credential=True)

• **userdata.get('HF_TOKEN')** retrieves the saved Hugging Face access token.
• **login()** authenticates your session, allowing you to access Hugging Face models and repositories. The add_to_git_credential=True option also stores the credentials for Git operations.

Creating the Llama-3.1–8B tokenizer

Now, we load the tokenizer for the Meta-Llama-3.1–8B model and use it to convert a sentence into tokens.

tokenizer = AutoTokenizer.from_pretrained(
    'meta-llama/Meta-Llama-3.1-8B',
    trust_remote_code=True
)

text = "This is a blog to understand how tokenizers work."
tokens = tokenizer.encode(text)
print(tokens)

Output:

[128000, 2028, 374, 264, 5117, 311, 3619, 1268, 4037, 12509, 990, 13]

The encode() method converts the input text into a sequence of token IDs. Each number represents a token from the model's vocabulary. Notice that the first ID (128000) is a special token added by the tokenizer, while the remaining IDs correspond to the words and punctuation in the sentence. These token IDs are what the language model actually processes, not the original text.

To convert the token IDs back into readable text, we use the decode() method.

tokenizer.decode(tokens)

Output:

<|begin_of_text|>This is a blog to understand how tokenizers work.

The decode() method reconstructs the original text from the token IDs. Notice the special token <|begin_of_text|> at the beginning. This is automatically added by the tokenizer to indicate the start of the input sequence. While the model uses this special token during processing, it is not part of the original text written by the user.

Instead of converting the entire sequence back into a single string, we can decode each token individually using batch_decode().

tokenizer.batch_decode(tokens)

Output:

[
 '<|begin_of_text|>',
 'This',
 ' is',
 ' a',
 ' blog',
 ' to',
 ' understand',
 ' how',
 ' token',
 'izers',
 ' work',
 '.'
]

This output shows exactly how the tokenizer split the sentence into individual tokens. Notice that “tokenizers” is divided into two tokens — " token" and "izers"—demonstrating that tokenizers often split words into smaller subwords. This approach helps language models efficiently represent both common and rare words.

A peek into special tokens

The tokenizer also provides a list of all the special tokens it adds to the model’s vocabulary.

tokenizer.get_added_vocab()

Output (truncated):

{
    '<|begin_of_text|>': 128000,
    '<|end_of_text|>': 128001,
    '<|start_header_id|>': 128006,
    '<|end_header_id|>': 128007,
    '<|eot_id|>': 128009,
    ...
}

The get_added_vocab() method returns a dictionary of special tokens and their corresponding token IDs. These tokens are not regular words—they serve specific purposes, such as marking the beginning or end of text, separating messages, or formatting conversations. For example, the token <|begin_of_text|> with ID 128000 is the same special token that appeared when we decoded our input.

Length of vocabulary

We can also check the size of the tokenizer’s vocabulary using the vocab attribute.

len(tokenizer.vocab)

Output:

128256

This tells us that the tokenizer has a vocabulary of 128,256 tokens. Each token in this vocabulary has a unique ID, allowing the model to efficiently convert text into numerical representations for processing.

Tokenizer using Instruct variants

Many language models also provide an Instruct variant, which is specifically trained to follow instructions and engage in conversations. These models expect prompts in a structured chat format containing system, user, and assistant messages.

The apply_chat_template() method automatically converts a list of messages into the exact prompt format expected by the model.

tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Tell an interesting fact about the  universe."}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

print(prompt)

Output (formatted):

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

You are a helpful assistant<|eot_id|>
<|start_header_id|>user<|end_header_id|>

Tell an interesting fact about the universe.<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

Notice how the tokenizer automatically inserts the required special tokens to separate the system, user, and assistant messages. Since add_generation_prompt=True is used, the prompt ends with the assistant header, signaling the model to generate the assistant's response next.

Conclusion

Tokenizers are a fundamental component of every large language model, acting as the bridge between human-readable text and machine-understandable numbers. They split text into tokens, map them to unique IDs, and format prompts in the structure expected by the model. Understanding how tokenizers work provides valuable insight into how modern AI systems process language and generate intelligent responses.