Embedding in LLM :

How AI Understands Meaning: The Hidden Story of Embeddings

Imagine asking an AI:

“I love learning about artificial intelligence.”

The AI responds naturally. It understands your question. It even connects ideas together.

But here’s the surprising part:

The AI does NOT understand words directly.

Step 1: AI Doesn’t Read Words

Before anything else happens, your sentence is broken down into smaller pieces called tokens.

So instead of:

“I love learning about artificial intelligence

The AI might see something like:

“I” | “love” | “learning” | “about” | “artificial” | “intelligence”

If you want to understand tokens and the tokenization process in detail, follow this Medium blog of mine: Click here.

After tokenization, the model still only sees numbers.

Embeddings :

Embedding = converting tokens into numerical vectors. Embeddings are one of the most important ideas in modern AI.

Step 1: Tokens Alone Mean Nothing

After tokenization, your sentence becomes something like:

“I” | “love” | “learning” | “about” | “AI”

Then each token becomes an ID:

[102, 532, 876, 221, 999]

But these IDs are meaningless.

Why?

Because:

999 ≠ intelligent

532 ≠ emotion

They are just labels.

Think of token IDs like:

• Roll numbers in school
• Phone numbers
• Product IDs

The number itself carries no meaning.

So the model needs another step.

It needs a way to represent meaning mathematically.

That is where embeddings begin.

Step 2: Tokens Become Vectors

Every token ID is converted into a vector.

A vector is simply:

👉 A list of numbers

Example:

“cat”

→ [0.21, -0.88, 0.45, 0.91, …]

Another word:

“dog”

→ [0.19, -0.82, 0.40, 0.87, …]

Notice something interesting?

The numbers are similar.

That is NOT random.

The model learns:

👉 Words with similar meanings should have similar vectors.

This mathematical representation is called an: Embedding Vector

What Is an Embedding?

An embedding is:

A dense numerical representation of meaning.

In simpler words:

👉 Embeddings convert language into coordinates inside a mathematical space.

Words with similar meanings end up close together.

Imagine a Giant Meaning Map

Think of embeddings like locations on a map.

Example:

cat → near dog

pizza → near burger

car → near truck

planet → far from sandwich

This space is called:

Vector Space

Every word becomes a point inside this giant multidimensional map.

Humans see:

“king”

The model sees:

[0.82, -0.11, 0.73, …]

But the position of that vector captures meaning.

Step 3: Similar Meaning = Nearby Vectors

This is the BIG idea.

Words that appear in similar contexts learn similar embeddings.

Example:

The cat sat on the mat

The dog sat on the mat

The model notices:

• “cat” and “dog” appear in similar environments
• Both are animals
• Both interact similarly in sentences

Over time:

👉 Their vectors move closer together.

This is how AI learns semantic relationships.

The Famous Example

One of the most famous embedding discoveries:

king — man + woman ≈ queen

Why does this happen?

Because embeddings learn relationships.

The model discovers patterns like:

male ↔ female

royalty ↔ ordinary

country ↔ capital

So mathematically:

king

minus masculine features

plus feminine features

≈ queen

This shocked researchers when embeddings first became popular.

Because:

👉 The model learned meaning from patterns alone.

Nobody manually programmed these relationships.

Step 4: How Embeddings Are Learned

Now comes the interesting question.

How does the model actually learn these vectors?

It starts randomly.

Initially:

cat → random numbers

dog → random numbers

sun → random numbers

Nothing meaningful yet.

Then training begins.

The model repeatedly predicts missing or next words.

Example:

The cat drank ___

The correct answer might be:

milk

If the prediction is wrong:

• the model updates weights
• embeddings slightly change
• related words move closer or farther

After billions of examples:

👉 Meaning starts emerging naturally.

Step 5: Dimensions in Embeddings

Embeddings are not just 2D or 3D.

Real models use huge dimensions.

Examples:

ModelEmbedding SizeWord2Vec300BERT Base768GPT modelsthousands

That means each word may be represented by:

[0.12, -0.44, 0.98, … thousands more numbers]

Each dimension captures hidden patterns.

Not manually designed.

Learned automatically.

Some dimensions may loosely represent:

• gender
• tense
• emotion
• size
• formality
• technicality

But in reality:

👉 Meaning is distributed across many dimensions.

Word Embeddings vs Contextual Embeddings

This is extremely important.

Older models used:

Static Embeddings

Example:

bank

Always had the SAME vector.

Problem?

“bank” can mean:

• river bank
• financial bank

The embedding could not adapt.

Modern transformers use:

Contextual Embeddings

Now the vector changes depending on context.

Example:

I deposited money in the bank

vs

We sat near the river bank

Same word. Different embeddings.

This was a massive breakthrough.

Transformers made this possible.

How Transformers Use Embeddings

Now let’s connect everything.

Full pipeline:

Step 1: Input Text

“I love learning AI”

Step 2: Tokenization

“I” | “love” | “learning” | “AI”

Step 3: Token IDs

[102, 2456, 8761, 332]

Step 4: Embeddings

[102] → [0.12, -0.98, …]

Step 5: Positional Encoding

The model also learns word order.

Because:

Dog bites man

is different from:

Man bites dog

Step 6: Transformer Layers

Now attention mechanisms process these vectors.

This is where:

• context understanding
• reasoning
• prediction
• language generation

actually happen.

Sentence Embeddings

Embeddings are not limited to words.

Entire sentences can become vectors.

Example:

“I love pizza”

→ one vector

[0.91, -0.22, 0.78, …]

Now similar sentences end up near each other.

Example:

“I enjoy pizza”

will be close to:

“I love pizza”

This powers:

• semantic search
• recommendation systems
• AI retrieval systems
• vector databases
• RAG systems

Embeddings Power Semantic Search

Traditional search:

search = keyword matching

Embeddings search:

search = meaning matching

Example:

Search query:

“best places to study”

The AI may retrieve:

“quiet libraries for students”

Even though exact words differ.

Because embeddings capture meaning.

This is called:

Semantic Similarity

Vector Databases

Once text becomes embeddings, AI systems store them in:

Vector Databases

Examples:

• Pinecone
• Weaviate
• FAISS
• Chroma
• Milvus

These databases search vectors efficiently.

Instead of:

exact words

they search:

nearby meanings

Why Embeddings Changed AI Forever

Before embeddings:

Computers treated words like isolated symbols.

After embeddings:

AI began understanding relationships.

This changed:

• NLP
• search engines
• recommendation systems
• chatbots
• retrieval systems
• multimodal AI

Modern AI became possible because language gained mathematical structure.

The Fascinating Part

Humans think in:

• ideas
• emotions
• meanings

Machines think in:

• vectors
• matrices
• geometry

Embeddings are the bridge between those two worlds.

That’s why they are one of the most beautiful concepts in AI.

Final Thought

Next time you type something into ChatGPT, remember:

The AI is not seeing words the way you do.

It is seeing:

• tokens
• vectors
• positions in mathematical space

And from those vectors, it builds meaning.

That hidden geometric world is what allows modern AI to feel intelligent.