Fully Transparent Mini Transformer: Complete Numerical Walkthrough with Positional Encoding — The World’s First??

A Step-by-Step Educational Model on the Sentence “The animal didn’t cross the street because it was too tired.” (d_model=4)

Mini Transformer: An Educational Self-Attention Model

Sentence: “The animal didn’t cross the street because it was too tired.”

Model Specifications

• Dimension (d_model): 4
• Heads: Single Head
• Vocabulary: 11 tokens
• Positional Encoding: Simplified Sinusoidal (scaled for stability)
• Purpose: To clearly demonstrate how Self-Attention works with realistic linguistic phenomena

1.Token Embeddings (Final Adjusted)

2. Positional Encoding (Simplified Sinusoidal)

How to use: Add the Positional Encoding vector to the Token Embedding before computing Q, K, and V. Final Input = Token Embedding + Positional Encoding

Positional Encoding Table (scaled for this educational model):

3. Weight Matrices

W_q (Query)

W_k (Key)

W_v (Value): Nearly Identity matrix (preserves original information)

4. Attention Scores (Top 5 for Each Query)

5. Notable Achievements of This Model

• Clear distinction between “The” and “the”: “The” (position 0) focuses on “animal”, while “the” (position 4) correctly focuses on “street”. This demonstrates the power of positional encoding.
• Strong syntactic phrase detection: “didn’t ↔ cross” and “cross ↔ street” show robust detection of verb phrases.
• Coreference resolution: “animal” and “it” strongly attend to each other.
• Semantic focus: The reason clause (“because it was too tired”) correctly converges on “tired”.
• Realistic functional vs content word behavior: Articles (“The”, “the”) have lower overall scores but focus on the most relevant nouns.

This small-scale model (d_model=4) successfully demonstrates core Transformer mechanisms — positional awareness, syntactic binding, and semantic focus — in an understandable way.

Created in collaboration with Grok (xAI) This is an educational miniature model designed to help students and developers understand how Transformer attention actually works.

This is kind of half a joke, honestly — the numbers are definitely hand-picked for explanation purposes. But I think it still does a decent job of showing what Wq, Wk, and positional encoding are actually doing.