Day 17/100: Embedding Layers — Turning Tokens Into Meaningful Vectors

Welcome to Day 17 of our “100 Days of Deep Dive into Machine Learning” series! Yesterday, we explored Tokenization — the process of breaking raw text into smaller units a model can process. Today, we move to the next crucial step in the pipeline: Embedding Layers, where those tokens are transformed into dense vector representations that models can learn from.

🎯 Why Should You Care About Embedding Layers?

Machine learning models don’t work with text — they work with numbers. But not just any numbers.

Embedding layers allow us to represent words, subwords, or tokens as dense, learnable vectors that capture:

• Semantic meaning
• Contextual similarity
• Syntax patterns

They're the bridge between symbolic language and numerical learning.

🔎 What Are Embedding Layers?

An embedding layer maps each token ID to a dense vector of real numbers (usually 100–1024 dimensions). These vectors are:

• Initialized randomly or with pretrained values
• Learned during training (if trainable)
• Stored in an embedding matrix (token_id → embedding vector)

# Pseudo-code in PyTorch
embedding = nn.Embedding(num_embeddings=vocab_size, embedding_dim=hidden_size)

🧠 Why Embeddings Work

They allow the model to:

• Understand similarity between words (e.g., "king" and "queen" are close)
• Capture contextual relationships
• Reduce the input dimensionality (vs. one-hot encoding)

Instead of treating every token as unrelated, embeddings cluster semantically related tokens closer together in vector space.

⚙️ Types of Embeddings

1️⃣ Static Embeddings

Each word has one fixed vector (no matter the context)

Examples:

• Word2Vec
• GloVe
• FastText

✅ Pros:

• Fast, simple
• Useful in classical ML and shallow models

❌ Cons:

• Can’t distinguish "bank" (river) vs. "bank" (finance)

2️⃣ Contextual Embeddings

Word vectors change based on context in the sentence

Examples:

• ELMo
• BERT
• GPT

✅ Pros:

• Capture polysemy and nuance
• Better for downstream tasks

❌ Cons:

• Require full model inference
• Heavier computational cost

📊 Real-World Analogy — Word Meaning in a Dictionary

Think of embeddings as definitions for each word, but encoded in numbers.

• Static embeddings: One definition per word
• Contextual embeddings: Multiple definitions depending on the sentence

🔧 When to Use Trainable Embeddings

✅ Use trainable embeddings if:

• You have enough data
• You want the model to adapt representations for your specific task

✅ Use pretrained embeddings if:

• You’re working with a small dataset
• You want to leverage transfer learning

You can also freeze pretrained embeddings or fine-tune them during training.

🌱 Embeddings in Transformers

In Transformer models:

• Tokens are first mapped to embeddings
• Then added to positional encodings
• Then passed into the encoder/decoder stack

input_embedding = token_embedding + positional_encoding

🧪 Bonus: Embedding Visualizations

Tools like TensorBoard, projector.tensorflow.org, or UMAP can project high-dimensional embeddings into 2D/3D space.

Use these to:

• Explore relationships
• Debug token issues
• Visualize training progress

📖 What’s Next? Sneak Peek into Day 18

Tomorrow, we’ll explore Encoder vs. Decoder — the two halves of the Transformer architecture and how they work together in language modeling and translation.