The Day We Stopped Counting Words: A Look at Semantic Alchemy

Imagine trying to learn a language by reading a dictionary, but the definitions are missing. You only know that “dog” appears on page 45 and “cat” appears on page 102. You know they exist, but you have no idea that they are both furry, four-legged pets.

For decades, this is exactly how computers understood language. We represented words as simple ID numbers or tally marks. A computer knew that “good” and “excellent” were different words, but it had no mathematical way to know they were synonyms. They were just distinct scratches on a digital surface.

That changed in the early 2010s with a concept that we can call Semantic Alchemy, popularly known as Word2Vec.

The “Fake” Task that Changed Everything The genius of Word2Vec is that it is unsupervised. You don’t need humans to label data. You just feed the model raw text — billions of words of it — and give it a “fake” job to do.

The model isn’t trying to understand Shakespeare; it’s just trying to predict the next word in a sentence. It’s a game. But to win this game, the model is forced to learn the relationships between words. It realizes that “dog” and “cat” are often found in the exact same sentences (near “pet,” “food,” “vet”), so it pushes their mathematical representations closer together.

The Introvert (CBOW) vs. The Extrovert (Skip-Gram) In my full breakdown, I explore the two distinct personalities of this architecture:

1. CBOW (Continuous Bag of Words): Think of this as the careful introvert. It looks at all the surrounding clues (“The ____ sat on the mat”) and averages them out to make a safe, highly probable guess (“cat”). It is fast and efficient.
2. Skip-Gram: This is the chaotic extrovert. You give it one word (“sat”), and it tries to predict every single word that might be standing near it. It’s a harder task, but because it works so hard, it learns much finer details about specific, rare words.

Why It Matters This technology gave us the famous equation: King - Man + Woman = Queen. It proved that we could map the "soul" of a word into a list of numbers (a vector). While we have moved on to massive Transformers like GPT-4, those models stand on the shoulders of these static embeddings. You cannot understand the skyscraper if you don't understand the foundation.

I’ve written a comprehensive guide that strips away the jargon and dives into the architecture, the “Negative Sampling” efficiency hack, and the mathematics that make this alchemy possible.

Read the full Semantic Alchemy deep dive here: https://vectorsandverbs.com/posts/word-embeddings/