NLP from Scratch : Word2Vec, GloVe, and FastText (Teaching Machines What Words Actually Mean)

This is Part 2 of the “BoW to Transformers” series. In Part 1, we built Bag of Words and TF-IDF by hand — and ended on a confession: neither of them understands a single word. They just count. This part fixes that.

The Problem We Left Unsolved

Quick recap of where Part 1 left us stuck. In TF-IDF, “good” and “great” are two completely unrelated columns in a matrix. So are “king” and “queen.” So are “Delhi” and “Mumbai.” To TF-IDF, every word is an island — equally distant from every other word, no matter how related they actually are in meaning.

That’s not a small gap. That’s the entire problem of meaning missing from the model.

So here’s the real question this part answers:

Can we represent a word as a vector such that words with similar meaning end up close together in space?

The answer arrived in 2013, and it changed NLP forever: Word2Vec.

The Idea That Changed Everything

Word2Vec is built on one deceptively simple linguistic idea, first stated by linguist J.R. Firth:

“You shall know a word by the company it keeps.”

In other words — words that appear in similar contexts tend to have similar meaning. “Coffee” and “tea” show up near words like “drink,” “cup,” “morning,” “hot.” So a model that learns to predict a word from its neighbors, or neighbors from a word, ends up learning meaning as a side effect.

That’s the whole trick. Word2Vec doesn’t try to “understand” language. It just plays a very simple prediction game, millions of times, and meaning emerges from the pattern.

Two Flavors of Word2Vec

CBOW (Continuous Bag of Words)

Given the surrounding words, predict the middle word.

Sentence: “The quick brown fox jumps over the lazy dog”

Window size = 2

Context: [quick, brown, jumps, over] → Predict: “fox”

Skip-Gram

The reverse — given the middle word, predict the surrounding words.

Input: “fox” → Predict: [quick, brown, jumps, over]

Rule of thumb: CBOW is faster and works better with frequent words. Skip-Gram is slower but shines on rare words and smaller datasets — which is why Skip-Gram tends to be the more popular choice in practice.

What’s Actually Happening Under the Hood

Here’s the part most explanations skip over, and the part that actually matters if you want to use this properly, not just quote it in an interview.

Word2Vec is a shallow neural network — one hidden layer, no activation function on it, just a linear projection. The trick isn’t the network architecture. The trick is what you throw away after training.

You train the network to predict context words. Once training is done, you discard the output layer entirely. What you keep is the hidden layer’s weight matrix — and each row of that matrix becomes the vector representation for one word in your vocabulary.

W is a V × d matrix — V rows (one per vocabulary word) and d columns (the embedding size, commonly 100–300). Each row wᵢis a dense, low-dimensional vector — a massive downgrade from TF-IDF’s sparse 50,000-length vectors, and yet it captures far more.

The Softmax Bottleneck (and Why You’ll Hear “Negative Sampling”)

Training this naively means computing a softmax over the entire vocabulary for every single training step:

With a 100,000-word vocabulary, that denominator sum is brutal to compute millions of times. Word2Vec’s actual innovation — the reason it could train at scale in 2013 — was negative sampling: instead of updating weights against the entire vocabulary, update against the true context word plus a handful (5–20) of randomly sampled “negative” words that aren’t in the context. This turns an expensive multi-class problem into a cheap set of binary classification problems, and it’s the detail that made Word2Vec fast enough to train on billions of words.

The Famous Result: Vector Arithmetic

This is the moment that made the NLP world sit up in 2013:

king — man + woman ≈ queen

Read that again. Nobody told the model that gender is a concept. Nobody labeled anything. The model just predicted context words over and over — and out of that, directions in vector space started encoding relationships: gender, tense, country-capital, plural-singular. Meaning became geometry.

Code (Gensim)

from gensim.models import Word2Vec

sentences = [[“i”, “love”, “nlp”], [“i”, “love”, “deep”, “learning”], [“nlp”, “is”, “a”, “part”, “of”, “deep”, “learning”], [“king”, “is”, “a”, “man”], [“queen”, “is”, “a”, “woman”]]

model = Word2Vec(sentences, vector_size=50, window=2, min_count=1, sg=1) # sg=1 -> Skip-Gram

print(model.wv[“nlp”]) # dense 50-dim vector

print(model.wv.most_similar(“nlp”)) # closest words by cosine similarity

On tiny toy data like this, the vectors won’t show much — Word2Vec needs real scale (Wikipedia-sized corpora) to produce those famous analogies. That itself is an important, honest caveat: Word2Vec is data-hungry.

GloVe: A Different Philosophy, Same Goal

Word2Vec learns from local context windows — it never sees the corpus as a whole, only small sliding windows. GloVe (Global Vectors, Stanford, 2014) asks: why throw away global statistics?

GloVe starts by building a co-occurrence matrix X, whereXᵢⱼ counts how often word i appears near word j across the entire corpus. Then it learns vectors such that their dot product approximates the log of that co-occurrence count:

wᵢᵀw̃ⱼ + bᵢ + b̃ⱼ = log(Xᵢⱼ)

The intuition: if “ice” and “solid” co-occur far more than “ice” and “gas,” that ratio should be baked directly into the vectors — not learned indirectly through prediction, but fit directly to global co-occurrence statistics.

Word2Vec vs GloVe, in one line: Word2Vec is predictive (learns by guessing context), GloVe is count-based (learns by fitting to global statistics). In practice, their resulting embeddings behave fairly similarly — this is more a difference in training philosophy than in output quality.

FastText: Fixing the Biggest Blind Spot

Here’s a question that breaks both Word2Vec and GloVe: what happens when you meet a word you’ve never seen during training?

Answer: nothing good. Both treat “unbelievable” and “unbelievably” as two completely unrelated tokens, each needing its own training exposure. And any truly unseen word — a typo, a rare name, a new slang term — has no vector at all. This is the out-of-vocabulary (OOV) problem, and for morphologically rich languages (Hindi, Telugu, Finnish, German) it’s a serious weakness.

FastText (Facebook, 2016) fixes this with one elegant change: instead of learning a vector per whole word, it learns vectors for character n-grams, and represents a word as the sum of its n-gram vectors.

Word: “learning” (with boundary symbols, n=3)

n-grams: “<le”, “lea”, “ear”, “arn”, “rni”, “nin”, “ing”, “ng>”

vector(“learning”) = sum of vectors of all these n-grams

The payoff: now “learning” and “learn” and “learner” share n-grams, so they share meaning components automatically. And a brand-new, never-seen-before word can still get a reasonable vector, built from n-grams the model has seen — no OOV problem.

Code (Gensim FastText)

from gensim.models import FastText

model = FastText(sentences, vector_size=50, window=2, min_count=1, sg=1)

print(model.wv[“learning”])

print(model.wv.most_similar(“learning”))

The real test — a word NEVER seen during training still gets a vector:

print(model.wv[“learnable”])

Comparison Table — Which One Do You Actually Reach For?

Aspect | Word2Vec | GloVe | FastText

Learns from | Local context window | Global co-occurrence stats | Character n-grams

Handles unseen (OOV) words | No | No | Yes

Good for morphologically rich languages | Weak | Weak | Strong

Training speed | Fast | Moderate (needs co-occ. matrix) | Slower (n-gram overhead)

Captures word analogies | Yes | Yes | Yes, slightly less crisp

Best use case | General-purpose, large clean corpora | Static, pre-trained embeddings for downstream tasks | Any real-world text with typos, slang, rare/compound words

If you’re working with clean English text and a huge corpus — Word2Vec or GloVe both do the job. If you’re working with noisy, real-world, or morphologically rich text (which, if you’ve built anything like a JD–skill matcher on real job postings, you already know is most real text) — FastText’s OOV handling alone can be the deciding factor.

The Limitation That Sets Up Part 3

Here’s the catch none of these three fully solve, and it’s the one that matters most:

Every one of these gives a word exactly one fixed vector — no matter the sentence it’s in.

“I sat by the river bank.”

“I withdrew cash from the bank.”

Same word, wildly different meaning — but Word2Vec, GloVe, and FastText all hand you back the same vector for “bank” both times. These are called static embeddings, and this is precisely the ceiling they can’t break through. Meaning here still isn’t context-aware; it’s fixed the moment training ends.

Solving that — vectors that change depending on the sentence around them — is what pulled the field toward RNNs, then Attention, then Transformers. That’s Part 3.

Where We Are in the Journey

BoW → TF-IDF → Word2Vec / GloVe / FastText → RNN → Attention → Transformers → LLMs

You are here: right after static word embeddings, right before sequence models.

We’ve gone from “just count words” to “words that live in meaningful space.” That’s a real leap. But space isn’t enough — the next leap is context, and that’s where things get genuinely interesting.

Part 3 picks up with RNNs and the sequence problem — why order matters, why static vectors aren’t enough, and how attention finally cracked context. Follow along for the rest of the series.