LLMs (Part-02): Before the Encoder Stack

Understanding Tokenization, Embedding & Positional Encoding in Transformers

Greetings from the author! If this is the first story that you have come across on transformers, don’t miss out on the entire Large Language Models (LLMs) series — explaining in great detail the transformer architecture, encoder & decoder stacks, blocks and layers, embedding & un-embedding layers, mathematical operations inside LLMs, feature extraction, feed forward mechanism — and a lot more. We also have a great multi-part Deep Learning series as well; don’t forget to check it out too!

Article Outline

— Part-01: Tokenizers — Part-02: The Embedding Layer — Part-03: The Positional Encoding Layer

Part-01: Tokenizers

— A Standalone Service Engine, Sitting in Front of the Transformer —

Tokenizer is neither a part of LLMs / transformers nor an intelligent component itself. It is a rule-based text-processing service that performs two actions on the incoming user input sequence before it is forwarded to the LLM / transformer.

How Tokenizers Work — 0s & 1s- CGvDDG

( a ) Processing Steps of a Tokenizer

A tokenizer:

• Chops up the given sequence of words / image / video / audio waveforms into chunks — where it can even split one word into multiple tokens.\
• Looks up the chunk-to-ID hash map and maps the chunks to whole number IDs.

The hash map is generated during the software development process of the tokenizer. The final output of the tokenizer is a vector — a 1D matrix — which contains IDs for the tokens. For example:

• Tokenizer # 1: “Hello World” → [“Hello”, ”World”] → [101, 202]
• Tokenizer # 2: “Hello World” → [“Hel”, ”lo”, ”World”] → [55, 67, 202]

So, depending upon the tokenizer rules, we may get slightly different vector lengths and vastly different numerical values inside the 1D matrix.

( b ) Tokenizer Similarity Across Training & Inference

It is absolutely crucial that we use the same tokenizer across training and inference.

The problem is that one tokenizer may generate the ID 100 for the word “hello” and the other one may generate the ID 100 for the word “world”. In this case, in training, the LLM will learn that the token ID 100 corresponds to an embedding which means “hello”, whereas in production, the ID 100 will correspond to the word “world” in its embedding form for the LLM, and the similar embedding now will have a completely different meaning inside the LLM (hello vs. world).

( c ) Case Sensitivity of Tokenizers

Tokenizers may or may not disregard the capitlization of the letters.

• One tokenizer may generate an ID of 678 for “Foo” as well as “foo”
• Another one might generate an ID of 987 for “Foo” and 123 for “foo”

So, depending upon the rules inside the source code of the tokenizer engine, we may get same or different embeddings in such cases. Then, inside the LLM / transformer, the embedding layer will behave accordingly too, based on the tokenizer’s behaviour.

( d ) Tokenizer Hash Table Development During Training

The tokenizer’s hash table is also developed based on the data coming in from the training data set. Each unique chunk is assigned a unique token ID inside the hash map if it already doesn’t exist. If it does, we skip updating it.

In short, tokenizer only “knows” tokens that it has seen in the training data.

Part-02: The Embedding Layer

— The First Layer Inside the Transformer —

An embedding is basically a vector of length “d” called its dimension. This vector essentially has mathematical numbers present in it, separated by comas.

An embedding vector, X, might look like:

Xemb = [8.7625, 9.3546, 7.3483, …., 7.3834 ]

( a ) Embeddings: Transformer Vs. Vector DBs

• Transformer / LLMs convert the incoming sequence to embeddings via their embedding layer. They generate one high-dimensional vector per token.
• On the other hand, embedding models pick up user query and generate a single high-dimensional vector which can be used for similarity search inside a vector DB — not multiple vectors per word in a chunk of text.

( b ) Initial Embedding Don’t Capture Contextual Meaning

The initial embedding will always be the same for a given word’s token representation inside the same transformer. It is just that when the words are combined with a specific sequence and “attention” is given to them in the light of others and attention is given to others in their light that we figure out the contextual relevance of the embedding.

For example, in “Apple makes the iPhone” and “Apple is the king of all fruits”, we will have exactly identical embeddings for the word Apple at the embedding layer, but during self-attention, we will understand the difference in both the meanings via considering their context which comes from the neighbouring tokens.

( c ) Embeddings are Deterministic per Model

If the tokenizer remains the same, which means the same token IDs will be generated for a word (“rug”) in each user query, then:

• Intra-model Queries: All user queries coming into the same model (say Gemini 2.5 Pro) will have same embeddings for the word “rug”.
• Inter-model Queries: But some other model (Claude Haiku 4.5) will generate embedding for the word “rug” that is different from the Gemini model’s embedding of “rug”. But all user queries that come into Claude Haiku 4.5 will generate same embedding each time for the word “rug” — provided that the tokenizer also remains the same.

So, emeddings vary from model to model but remain the inside a given model.

( d ) Embedding Have Fixed Dimension Per Model

During training, the embedding dimension d (which is the length of the 1D vector) is fixed for the whole model. So, every token ID in each user query maps to a vector of the exact same length (d).

( e ) Mathematical Equation of Embedding

For a token vector Vt, we go over each token ID and generate an embedding vector et for that particular token. Then, combining all these vectors, we get the overall embedding 2D matrix. For example:

Input = Hello World

Vt = [2435, 8237] // output of the tokenizer

e0 = [2.232378, 6.27323, ….. 8.12837 ] // embedder output for e0 based on the input Vt[0] (Hello)

e1 = [6.92371493, 11.239847, ….. 27.239847 ] // embedder output for e1 based on the input Vt[1] (World)

Xemb = [e0, e1,] // A 2D matrix of 1D matrices (vectors)

The Xemb matrix (2D) has dimensions:

n d*

…. where:

• n = total token IDs (rows coming from the length of the tokenizer vector)
• d = length of the embedding vector (meaning total coma-separated numbers in it)

( f ) Embedding Lookup Table

To convert the token IDs to distinct d-dimensional embedding vectors, we use the embedding lookup table at inference time. It is a pre-developed lookup table from training that maps from token IDs to vectors of d dimension.

The table dimensions are:

Vocabulary size d*

…. where:

• d = dimension of the vector for each token’s embedding
• Vocabulary Size = total distinct token IDs (and therefore distinct embeddings) seen by the model given to it by the tokenizer during training in all the training data combined.

How Embedding Lookup Works in LLMs / Transformers — 0s & 1s- CGvDDG

Part-03: The Positional Encoding Layer

— The Second Layer Inside the Transformer —

( a ) Positional Encoding — The Need

The problem is that transformers take in the entire input sequence all at once into their layers. Hence, when all the tokens come in parallel, the transformers are unable to know the position of the words. Hence, they have no idea what the correct sequence looks like. This is why we have to use positional encoding.

( b ) Problem of Having No Positional Encoding

Basically, we have two choices:

• Serial Computation: Do attention compute token-by-token in a serial way, so the model knows the position. This reduces performance a LOT.
• Parallel Computation: Do attention computation in parallel for all tokens, but at first add positional encoding so the model knows positions as well

For example:

• man loves car
• car loves man
• loves car man
• loves man car

… will appear the same to the LLM and its attention layers unless we add at first positional encoding to the per-token embedding vector.

NOTE: The positional encoding must have the same dimension as the token embedding so you can add them.

( c ) Parallel Processing Meaning

There are two different kinds of “parallel.”

1) Heads in Parallel: Each attention head learns a different pattern or feature. Example: one head may focus on syntax, another on long-range dependencies, another on nearby words. These heads run at the same time.

2) Computations Within a Head in Parallel: Inside one head, the model computes attention scores from one token to all other tokens at once. So token 1 can compare to tokens 2, 3, 4… simultaneously. That’s the n 2 n 2 -style matrix computation.

The second type of parallelization is the one for which we need positional encoding to be added to the per-token embedding vector.

( d ) Mathematically Representing Positional Encoding

Say we have 3 tokens:

• Inputs: “I”, “love”, “football”
• IDs: Token IDs might be [987, 876, 5708]
• Embeddings: Token embeddings might be:

— e1 = [ 1 , 0 ] — e2 = [ 0 , 1 ] — e3 = [ 1 , 1 ]

… where embedding dimension is 2 (d = 2);

• Positional Encoding: Positional encodings might be:

— p1 = [ 0.1 , 0.2 ] — p2 = [ 0.3 , 0.4 ] — p3 = [ 0.5 , 0.6 ]

… where the p vectors have same dimensions as the e vectors (= d).

( e ) Positional Encoding Lookup Table

In many transformers, there is a learned positional embedding table. Its shape is usually:

max_seq_len × d max_seq_len×d; where

• max_seq_len = maximum sequence length the model supports as input in one message
• d = embedding dimension

If the input sequence inputted at run-time is longer than the input sequence allowed, we truncate it, and, therefore, never go out of the bounds of our learned positional encoding table.

In this table, we store one vector per row of length d. For calculating the positionally-encoded embedding vector for input token number 101, we add the two things:

• The vector at the 100th row (starting from 0th index) of the positional encoding 2D matrix
• The embedding vector itself for the 100th token ID (starting from 0th index)

That’s how we calculate the positionally encoded embedded vector. Now, the positionally encoded embeddings look like:

— x1 = e1 + p1 = [ 1.1 , 0.2 ] — x2 = e2 + p2 = [ 0.3 , 1.4 ] — x3 = e3 + p3 = [ 1.5 , 1.6 ]

And the overall positionally encoded 2D embedding matrix containing all the 1D vectors becomes:

Xemb = [x1, x2, x3] // Xemb has per-token embedding vectors with positional encoding in them

How Positional Encoding Works in LLMs / Transformers — 0s & 1s- CGvDDG

Part-04: The Three Lookup Tables

— Tokenizer, Embedding & Positional Encoding Tables —

( a ) Storage Place for Lookup Tables

They’re stored as model/runtime data, but in different places:

• Tokenizer Lookup Table: stored with the tokenizer files/config, outside the model weights. It contains the vocabulary and rules for mapping text pieces to token IDs.
• Embedding Lookup Table: stored inside the model parameters as a learned weight matrix ( V * d ).
• Positional Encoding Table: If learned, also stored inside the model parameters as another learned weight matrix ( max_seq_len × d).

( b ) Development Process for the Lookup Tables

• Tokenizer Table (Computed in Training): It is built from the training data by choosing a key of token pieces and assigning each piece a unique ID as value in the hash map. Techniques like Byte-pair Encoding can be Used for developing tokenizers.
• Embedding Table (Learned in Training): It starts as random vectors in a ∣V∣×d matrix and is learned by backpropagation so each token ID selects a row whose values are updated from training. Techniques like GLoVe and Word2Vec are used for making embedding tables.
• Positional Table (Learned / Computed in Training): There are two possibilites for this: — ( a ) Learned: If learned, it starts as random position vectors in a max_seq_len × d matrix and is updated by backpropagation. — ( b ) Computed: If sinusoidal, each vector is computed directly from the position using sine and cosine formulas.

Tokenizer Table Vs. Embedding Lookup Table Vs. Positional Encoding Table — 0s & 1s — CGvDDG

Until Next Time,

0s & 1s