Transformers Part 2: Input and Positional Encoding

In Part 1, we explored the high-level concept of transformers—why they revolutionized deep learning and how the attention mechanism made recurrence obsolete.

1. Input Embeddings

Input embeddings are the numerical vector representations of words (or subword tokens) in a way that captures their meaning and relationships with other words.

Why do we need them?

Neural networks can’t process raw text like “apple” or “robot” directly—they require numerical input. To achieve this, each token (typically a word or subword) is converted into a fixed-size vector of real numbers, known as an embedding vector. In Transformer models, the input sequence is usually limited to a maximum of 512 tokens, with each token represented by its corresponding embedding.

Steps:

1. Using a typical BERT tokenizer (e.g., WordPiece), the above sentence would be split into tokens like

transformers, have, revolutionized, natural, language, processing, by, enabling, models, to, learn, contextual, representations, .

2. Either we do padding or truncating if the length of the token is too large. Here it is 15, so we do padding. We also add special tokens [CLS] at the beginning and [SEP] at the end.

[CLS] transformers have revolutionized natural ... representations . [SEP] [PAD] [PAD] ... [PAD]

In a Transformer model, the input embedding size refers to the dimensionality of the vector representation for each input token (such as a word or subword). This size is typically denoted as **d_model, which in the original Transformer paper ("Attention is All You Need" by Vaswani et al., 2017) is set to 512. Each token is mapped to a 512-dimensional vector, and this dimension remains consistent throughout the model. As a result, an input sequence with N tokens is represented as a matrix of shape [N, d_model]**.

[Token_1_embedding]
[Token_2_embedding]
...
[Token_512_embedding]   ← final matrix of shape (512 × 512)

2. Positional Encoding

Transformers process input tokens in parallel, unlike RNNs or LSTMs, which process tokens sequentially. This means transformers have no sense of order—so we need to inject positional information manually. It adds information about the position of each token in the sequence to its embedding.

Positional encoding is a set of vectors—one for each position in the sequence. Each vector is of the same dimension as the embedding (e.g.,**d_model 512). These vectors are added to the input embedding** matrix—element-wise.

This means

• Even indices (0, 2, 4, …) → sine functions
• Odd indices (1, 3, 5, …) → cosine functions

This allows the model to learn relative and absolute positions through periodic patterns.

• pos: The position of the token in the sequence (0, 1, 2, …, max_seq_length)
• i: The dimension index (0 to d_model−1)
• d_model: The embedding size (e.g., 512)

Assume:

["apple", "ball"]

| Token | Word Embedding (4D)    |
| ----- | -----------------------|
| apple | [0.4, -0.2, 0.6, 0.1]  |
| ball  | [0.7,  0.3, -0.1, 0.8] |

| Position | Positional Encoding                                                |
| -------- | -------------------------------------------------------------------|
| 0        | [sin(0), cos(0), sin(0), cos(0)] = [0.0, 1.0, 0.0, 1.0]            |
| 1        | [sin(1), cos(1), sin(0.01), cos(0.01)] ≈ [0.84, 0.54, 0.01, 0.999] |

"apple"
[0.4, -0.2, 0.6, 0.1] + [0.0, 1.0, 0.0, 1.0] = [0.4, 0.8, 0.6, 1.1]

"ball"
[0.7, 0.3, -0.1, 0.8] + [0.84, 0.54, 0.01, 0.999] ≈ [1.54, 0.84, -0.09, 1.799]

Now "apple" has one combined vector (semantic + positional), and "ball" has another. Even if "ball" appears later in another sentence, its positionally encoded embedding will be different, allowing the Transformer to distinguish "apple ball" from "ball apple".

• Input embedding: captures what the token is.
• Positional encoding: captures where the token is.

Please check into my next part about Encoders.

References

[1] Ashish Vaswani, et al. and team, Attention is all you need, 2017.

[2] https://towardsdatascience.com/transformers-explained-visually-not-just-how-but-why-they-work-so-well-d840bd61a9d3/