Why Does “Bank” Always = Token 750? Embeddings Explained

From word to number: how “bank” becomes token ID 750, and then a 768-dimensional vector the model can actually do math with.

A complete, question-driven walkthrough of embeddings, token IDs, tokenization, and cosine similarity — from “does every token have 768 dimensions?” to superposition, RAG, and vector search.

Most “What are embeddings?” articles read like a lecture — they explain the concept top-to-bottom and assume you’ll magically absorb it. But that’s not how real understanding works. When you’re actually learning this stuff, your brain doesn’t ask for a definition — it fires off questions: “Wait, does every token really have 768 dimensions?” “Is ‘royal’ literally one of those dimensions?” “If the model already knows bank = 750, how does my brand-new query get the same ID?"

That’s exactly why this article is built as a question-and-answer walkthrough instead of a straight essay. Each concept is introduced the moment a natural doubt pops up — so you’re never reading an answer to a question you haven’t thought of yet. We start from the absolute basics (what a vector even is) and climb all the way to expert-level ideas like superposition, anisotropy, and embedding drift — one honest question at a time.

By the end, you won’t just know what embeddings are. You’ll have cleared the exact doubts that trip up 90% of people — because we asked them out loud, in the right order, and answered each one properly.

TL;DR (key takeaways)

• A token ID is just an address; the embedding is the vector stored at that address.
• Embedding dimensions are shared across all tokens — one coordinate system, many points.
• Base embeddings are context-free; hidden states (after attention) are where contextual meaning lives — that’s why “bank” can mean finance or river edge.
• Cosine similarity compares direction, and every dimension contributes via the dot product.
• Token IDs come from a deterministic tokenizer, so your typed “bank” maps to the same ID every time.
• Expert layer: superposition, anisotropy, RoPE, RAG, vector databases, and embedding drift are what separate “using” embeddings from truly understanding them.

Part 1 — What an Embedding Actually Is

Q1: What is an embedding, in plain English? An embedding is a vector — a list of numbers — that represents a token (or token position) so the model can do math with it. Humans store meaning in words; models store “meaning-ish” information in numbers, because math is their native language. When people say “embeddings represent meaning,” they mean: similar tokens end up with vectors pointing in similar directions in a high-dimensional space.

Q2: What does “768 dimensions” actually mean? Each token position is represented by 768 numbers (for models with hidden size 768). In 2D a point needs (x, y); in 768D it needs 768 coordinates. Same idea — vastly more room.

Q3: If embeddings are high-dimensional, why do people show 2D plots? Because we can’t visualize 768D directly, so we project down using UMAP or t-SNE. Crucial warning: a 2D plot is a shadow of the real structure — useful, but it can distort distances and neighborhoods.

Q4: Does every token really have 768 dimensions? In a model with hidden size 768, yes — every token position is a 768-number vector at that stage.

Q5: Are the dimensions the same for all tokens, or does each token get its own? Shared. There is one coordinate system: dim1 … dim768. Every token is a different row (point) in that same space.

Q6: Is there a simple real-world analogy? Picture a control panel with 768 sliders. The sliders are fixed (shared dimensions); each token just sets them to different positions (different values).

Q7: Are dimensions literally interpretable? Is “Royal” a dimension? In teaching examples we pretend dimensions have names (“Royal,” “Fruit,” “Finance”) to build intuition. In real LLMs, concepts are distributed — “royal-ness” is a pattern across many dimensions, not one clean axis.

Part 2 — Token IDs vs Embeddings (the Biggest Confusion)

Q8: If “king = 102” (a token ID), is that the embedding? No. A token ID is an address (an index into a table). The embedding is the row of numbers stored at that address.

Q9: Can I measure similarity using ID distance? Never. ID assignment is essentially arbitrary indexing. Semantic similarity comes from the vectors, not the ID numbers. “Close IDs” ≠ “close meaning.”

Part 3 — When Are Dimensions and Embedding Tables “Created”?

Q10: Are dimensions created when a token is received? No. The model designer chooses the hidden size (e.g., 768) up front. Training learns the values; inference just uses them.

Q11: Does the embedding table change when I type a query? Not during inference. Training updates weights; inference freezes them. Yet meaning still shifts at inference, because the model computes different hidden states from the same frozen weights.

Part 4 — The “Bank” Question (Contextual Meaning)

Q12: If “bank” is the same token, how can it mean finance or river edge? Because there are two different things:

• Token embedding (base lookup): e_bank
• Contextual hidden state (after transformer layers): h_bank

Even if e_bank starts identical, the transformer uses context to produce a different h_bank. Analogy: token embedding = dictionary entry; hidden state = what the word means in this specific sentence.

Part 5 — Cosine Similarity and Why Features Decide It

Q13: Do the embedding features decide cosine similarity? Yes — cosine similarity is computed from the vector components. It measures the angle between vectors, not raw distance.

cos(a, b) = (a · b) / (‖a‖ · ‖b‖)

Q14: Does every dimension contribute to similarity? Yes. The dot product is literally a sum over dimensions.

Part 6 — The Toy Embedding Exercise

Q15: Is “king − man + woman ≈ queen” actually possible? In a toy space, yes — and it’s a great intuition-builder. Expert caveat: real models don’t have clean “Gender” or “Royal” knobs, but the idea (directions capture attributes) still holds.

Q16: Why did apple and banana become identical in the toy example? Because the toy space didn’t have enough dimensions to distinguish them. Low-dimensional representations collapse distinct items into the same point.

Q17: How do we fix that? Add a dimension that captures the missing difference. This is the cleanest way to explain why real models use large embedding sizes: more representational room.

Part 7 — Tokenization and the “Bank = 750” Matching Question

Q18: Where do token IDs come from? Are they created when I type? IDs come from the tokenizer, assigned when the tokenizer is built/trained — not when you type. BPE, WordPiece, and SentencePiece start from small pieces, learn merges from data, build a fixed vocabulary, and assign each token an index.

Q19: Can you break down “king is royal person” token-by-token? The split depends on the tokenizer and its training data — the same sentence can split in multiple valid ways.

Q20: What happens at runtime once tokens exist? Always: text → token pieces → token IDs → embedding lookup → transformer computation.

Q21: If the model has bank = 750 internally, how does my external query also get 750? Because the same tokenizer processes your input, and tokenization is deterministic: same tokenizer + same token piece ⇒ same ID every time. Two caveats: (1) whitespace markers can change the token ("bank" vs " bank", shown as Ġ or ▁); (2) different tokenizers assign different IDs — "bank = 750" is tokenizer-specific, not universal.

Q22: If I type a brand-new sentence, does the model create new IDs? No. The vocabulary is fixed. New sentences are built from existing pieces — like building new structures from a fixed LEGO set.

Q23: What if I type a word the vocabulary doesn’t have? The tokenizer splits it into smaller known pieces (still existing IDs). No new IDs are ever created at runtime.

Part 8 — Going Deeper: Intermediate to Expert

This is where most tutorials stop — and where real understanding of vector embeddings begins.

Q24: What’s the difference between the input embedding table and the output (unembedding) layer? The input embedding maps token ID → vector at the start. The output layer (the LM head) maps the final hidden state back to a probability over the whole vocabulary. Many models tie these weights (share one matrix for input and output) to save parameters — this is weight tying.

Q25: If token embeddings are context-free, where does context actually get injected? Through self-attention. The base embedding e_bank is identical everywhere, but each transformer layer lets a token "look at" other tokens and mix their information in. After a few layers, h_bank in "river bank" and "savings bank" point in very different directions — same start, different journey.

Q26: How does the model know word order? Aren’t embeddings just a bag of vectors? Raw token embeddings carry no position info — attention is permutation-invariant by default. So we add positional information: learned positional embeddings, sinusoidal encodings, or modern schemes like RoPE (rotary) and ALiBi. Without them, “dog bites man” and “man bites dog” would look identical.

Q27: Why cosine similarity and not plain Euclidean distance? In high dimensions, direction carries meaning more robustly than magnitude, and embedding norms can vary for reasons unrelated to semantics (e.g., token frequency). Cosine normalizes away magnitude and compares orientation. Some systems deliberately use dot product instead — it’s a design choice, not a universal law.

Q28: What is anisotropy, and why do “random” tokens sometimes look similar? In many trained models, embeddings occupy a narrow cone rather than spreading evenly — this is anisotropy. The side effect: even unrelated tokens can show deceptively high cosine similarity. Fixes include whitening, mean-centering, or contrastive training objectives that spread representations out.

Q29: The “curse of dimensionality” says high dimensions are sparse — so why do 768+ dimensions help? Both are true at once. High dimensions give the model room to encode many independent attributes (the benefit). But naive distance metrics get less discriminative, which is exactly why we lean on cosine, learned metrics, and training objectives that shape the space. Dimensionality is a resource and a hazard.

Q30: Roughly how many “concepts” can a 768D space hold — surely not just 768? Far more. Because of superposition, models pack many more features than dimensions by using nearly-orthogonal directions instead of perfectly orthogonal axes. The trade-off is interference (features slightly bleed into each other), tolerated because most features are rarely active at once. This is a core idea from mechanistic interpretability.

Q31: Is there a single “royal direction” I could actually find and steer? Sometimes, approximately. Techniques like probing and sparse autoencoders (SAEs) can extract interpretable directions, and you can nudge activations along them (“activation steering”). But it’s messy — features are distributed, entangled, and layer-dependent, so a clean universal “royal knob” usually doesn’t exist.

Q32: Static embeddings (Word2Vec/GloVe) vs contextual embeddings (BERT/GPT) — what’s the real difference? Static: one fixed vector per word regardless of context — “bank” always the same. Contextual: the vector is computed on the fly from the whole sentence, so “bank” changes with context. Static embeddings are the base lookup; contextual embeddings are hidden states after attention. Modern LLMs are contextual all the way down.

Q33: Why do LLM embeddings vary by layer? Which layer should I use for similarity search? Early layers stay close to surface/lexical features; middle layers often carry the richest semantics; late layers specialize toward next-token prediction. For retrieval or semantic search, middle-to-late layers (or a purpose-built embedding model) usually work best — the very last layer is optimized for generation, not general-purpose similarity.

Q34: For sentence embeddings, can I just average the token vectors? You can (mean pooling) — it’s a fine baseline. But naive averaging is dominated by frequent/filler tokens and ignores that these models weren’t trained for it. Better: models fine-tuned with a contrastive objective (Sentence-BERT style), which produce far more useful sentence-level geometry.

Q35: BPE vs WordPiece vs SentencePiece — does the choice actually matter? Yes. They differ in how merges/splits are learned and how whitespace is handled (SentencePiece treats text as a raw stream with ▁ markers, making it language-agnostic and reversible). The choice affects vocabulary size, word fragmentation, sequence length, and even multilingual fairness — some languages get chopped into far more tokens than others.

Q36: Why does “ bank” (with a leading space) sometimes get a different ID than “bank”? Byte-level BPE tokenizers (GPT-style) fold the leading space into the token, marking it with Ġ. So Ġbank and bank are genuinely different vocabulary entries with different IDs and different embeddings. This is why prompt spacing can subtly change behavior — a classic real-world gotcha.

Q37: What determines vocabulary size, and what’s the trade-off? It’s a tuned hyperparameter (often ~30k–250k+). Bigger vocab → shorter sequences (cheaper attention) but a larger embedding table and rarer tokens. Smaller vocab → tiny table but longer sequences and more fragmentation. A compute-vs-coverage balancing act.

Q38: How are embeddings for rare tokens learned well if they barely appear? Often they aren’t — rare tokens get under-trained embeddings, and truly unseen strings are split into subwords so their meaning is composed from better-trained pieces. This is a known weakness, sometimes exploited by “glitch tokens” that produce bizarre outputs because their embeddings were essentially never trained.

Q39: Do embeddings encode bias, and can it be measured? Yes. Because embeddings mirror training-data statistics, social biases show up as directions in the space (the classic WEAT tests, and analogies like “man:programmer :: woman:homemaker”). You can measure them via association tests and attempt debiasing by neutralizing the bias direction — though fully removing it is unsolved.

Q40: Can I mix embeddings from two different models — say, cosine between a BERT vector and a GPT vector? No — that’s meaningless. Each model learns its own coordinate system with no shared axes, orientation, or scale. Comparing across models requires an explicit alignment/mapping learned between the two spaces. Raw cross-model cosine is nonsense.

Q41: In RAG systems, why must the query and documents use the same embedding model? Same reason as Q40 — similarity only makes sense within one space. If you index documents with model A and embed queries with model B, the vectors live in incompatible spaces and retrieval collapses. Query and corpus must share the exact same encoder (and ideally the same preprocessing).

Q42: How does approximate nearest-neighbor (ANN) search scale similarity to billions of vectors? Exact cosine over billions is too slow, so vector databases use ANN indexes — HNSW graphs, IVF partitioning, or product quantization (PQ). They trade a little recall for massive speed/memory gains, letting you query huge corpora in milliseconds. This is the engine behind production semantic search and RAG.

Q43: What is embedding quantization, and what does it cost? Storing vectors in lower precision (fp16, int8, or even binary) to shrink memory and speed up search. int8 is often nearly lossless for retrieval; binary is extremely compact but sacrifices accuracy. It’s the standard way to make billion-scale vector search affordable.

Q44: Do embedding spaces “drift” over time or when the model is fine-tuned? Yes — embedding drift. Fine-tuning or retraining reshapes the space, so vectors from old and new model versions are no longer comparable. In production, this means you must re-embed your entire corpus when you upgrade the model; mixing old and new vectors silently degrades retrieval.

Q45: Isotropy fixes, whitening, dimensionality reduction — when should I actually touch the raw vectors? When measurements tell you to. If retrieval quality is poor and baseline similarity is high everywhere (anisotropy), try mean-centering/whitening. If storage/latency is the bottleneck, try PCA/quantization. Rule of thumb: don’t post-process blindly — a well-trained embedding model often needs none of it, and careless reduction destroys signal.

Q46: What’s the single mental model that ties all of this together? Three distinct layers — don’t conflate them: (1) Token ID = a fixed address from a deterministic tokenizer; (2) Base embedding = a learned, context-free vector at that address; (3) Hidden state = a context-dependent vector computed by attention at inference. IDs are plumbing, base embeddings are the starting point, and hidden states are where meaning actually lives.

If this helped the pieces finally click, a clap or two genuinely helps it reach more people. Got a question I didn’t cover — or one that stumped you? Drop it in the comments; the best articles come from the sharpest questions.