Embedding Models Explained: From TF-IDF to Transformers and OpenAI Embeddings

A practical guide for engineers building search, RAG, recommendation, and semantic systems

When we build software, we usually deal with exact things: IDs, strings, enums, timestamps, JSON fields, and database rows.

But human language is messy.

Two users can ask the same thing in completely different ways:

“How do I reset my password?” “I forgot my login credentials.” “Can’t access my account.”

To a normal keyword search system, these may look different.To a good embedding model, they are close in meaning.

That is the core idea of embeddings.

An embedding model converts text into a vector, which is just an array of numbers. The goal is simple: texts with similar meanings should produce vectors that are close to each other.

For example:

“payment failed” “transaction did not go through” “card was declined”

A good embedding model should place these near each other in vector space.

This is why embeddings are one of the most important building blocks behind modern GenAI systems, especially semantic search, recommendation systems, clustering, duplicate detection, and Retrieval-Augmented Generation, also known as RAG. OpenAI’s embedding documentation describes embeddings as numerical representations that measure relatedness between text strings and are useful for search, clustering, recommendations, anomaly detection, classification, and related tasks.

But not all embeddings are the same.

The evolution goes roughly like this:

One-hot / Bag of Words ↓ TF-IDF / BM25 ↓ Word2Vec / GloVe / FastText ↓ BERT / Transformer-based contextual embeddings ↓ Sentence/document embeddings ↓ General-purpose embedding APIs ↓ Task-specific and domain-specific embeddings

Let’s walk through each layer like engineers, not hype merchants.

1. Why do we need embedding models?

Computers do not naturally understand words.They understand numbers.

So before we can build NLP or GenAI systems, we need a way to convert text into something mathematical.

That conversion helps us answer questions like:

Are these two sentences similar? Which document is closest to this user query? Which support tickets are duplicates? Which products are similar? Which code snippet matches this requirement? Which internal wiki page should be passed to an LLM?

Without embeddings, we mostly rely on exact keyword matching.

With embeddings, we can compare meaning.

That is the big shift.

2. Basic lexical vectors: one-hot encoding and Bag of Words

Before neural embeddings, we had simple lexical representations.

Imagine we have this vocabulary:

[“cat”, “dog”, “bank”, “loan”, “river”]

A one-hot vector for “bank” could look like this:

cat dog bank loan river 0 0 1 0 0

The word is represented by its position in the vocabulary.

For a document, Bag of Words counts how many times each word appears.

Example:

Document: “cat cat dog” cat dog bank loan river 2 1 0 0 0

What problem does this solve?

It gives us the first useful numerical form of text.

Now we can feed text into traditional ML models like logistic regression, Naive Bayes, SVMs, and simple search systems.

What is the problem?

There is no real meaning.

The model does not know that:

car ~ automobile doctor ~ physician payment failed ~ transaction declined

It only knows exact words.

Also, the vector is very sparse. If your vocabulary has 100,000 words, your vector has 100,000 dimensions, and most values are zero.

Best use

Use basic lexical vectors when you need a quick baseline, simple text classification, or explainable word-count features.

They are not great for semantic understanding.

3. TF-IDF and BM25: weighted sparse lexical models

TF-IDF is a smarter version of word counting.

It asks:

“Is this word important in this document, compared to the whole corpus?”

TF-IDF stands for:

Term Frequency x Inverse Document Frequency

In simple words:

Words that appear often in one document get higher weight. Words that appear everywhere get lower weight.

So words like:

the, is, and, of

get low importance.

But words like:

embedding, transformer, Kubernetes, payment, authentication

may get higher importance depending on the corpus.

Scikit-learn describes TF-IDF as term frequency multiplied by inverse document frequency, commonly used in information retrieval and document classification, with the goal of reducing the impact of very frequent and less informative tokens.

BM25 is another classic ranking algorithm used heavily in search engines. It improves over plain TF-IDF-style scoring by considering term frequency saturation and document-length normalization. Elastic’s BM25 explanation describes how BM25 uses query terms, inverse document frequency, term frequency, and field length normalization to score relevance.

What problem does this solve?

TF-IDF and BM25 solve one big problem:

Not every word is equally important.

This makes search and classification much better than raw word counts.

For example, in a document about Java backend services, the word “the” is useless. But “Spring Boot”, “JVM”, “Kafka”, or “thread pool” may be meaningful.

What is still missing?

TF-IDF and BM25 are still mostly lexical.

They are good at matching words, not meaning.

For example:

Query: “car insurance” Document: “vehicle coverage policy”

A human sees the connection.

TF-IDF may struggle because the exact words are different.

Best use

TF-IDF and BM25 are excellent for:

keyword search log search legal search documentation search exact term matching search with product IDs, error codes, API names, ticket IDs

When should you use them?

Use TF-IDF or BM25 when exact words matter.

For example:

“ORA-00942” “NullPointerException” “payment-service-prod” “invoice_id” “GPT-4.1”

In production systems, BM25 is still extremely useful. Do not throw it away just because embeddings exist.

A very strong search architecture is often:

BM25 + dense embeddings + reranker

BM25 catches exact matches.

Embeddings catch semantic matches.

The reranker improves the final ranking.

That combo is boringly effective, which is the best kind of effective.

4. Static word embeddings: Word2Vec and GloVe

Now we move from sparse keyword vectors to dense semantic vectors.

This is where embeddings start to feel more like “meaning.”

Instead of representing each word as a huge sparse vector, models like Word2Vec and GloVe represent each word as a smaller dense vector.

Example:

king = [0.21, -0.45, 0.88, …] queen = [0.19, -0.40, 0.91, …] apple = [-0.77, 0.12, 0.34, …]

The values themselves are not directly human-readable.

But the geometry matters.

Words used in similar contexts get placed near each other.

Word2Vec proposed efficient model architectures for learning continuous vector representations of words from very large datasets, showing strong results on syntactic and semantic word similarity tasks.

GloVe, from Stanford, is also a static word embedding method. It trains on global word-word co-occurrence statistics from a corpus and produces word vectors with useful linear structure.

How does Word2Vec learn meaning?

The intuition is:

A word is known by the company it keeps.

For example, the word “king” may appear near:

royal, palace, crown, throne, kingdom

The word “queen” may appear near:

royal, palace, crown, throne, kingdom

Because they appear in similar contexts, their vectors become close.

This also explains the famous vector relationship:

king — man + woman ~ queen

This is not because the model “understands monarchy” like a human.

It learns statistical patterns from text.

What problem does this solve?

Static word embeddings solve the semantic gap that TF-IDF cannot handle well.

They can understand that:

car ~ vehicle doctor ~ physician king ~ queen

They are dense, compact, and capture useful semantic relationships.

What is the problem?

The big limitation is that every word has one fixed vector.

The word “bank” has the same vector in both sentences:

I deposited money in the bank. I sat near the river bank.

That is a problem.

The meaning changed, but the vector did not.

This is why we call them static embeddings.

Best use

Static word embeddings are useful for:

word similarity legacy NLP pipelines simple recommendation features clustering words lightweight ML models low-resource environments

When should you use them?

Use Word2Vec or GloVe when you need word-level semantic features and do not need sentence-level context.

For most modern GenAI or RAG systems, you usually move beyond this layer.

5. Improved static embeddings: FastText

FastText improves static word embeddings by looking inside the word.

Instead of treating a word as a single atomic unit, FastText breaks words into character n-grams.

For example:

embedding

could be broken into chunks like:

emb mbe bed edd ddi din ing

This helps the model understand morphology and handle rare or unseen words.

The FastText paper explains that many earlier word representation models assign a distinct vector to each word, which is limiting for rare words and morphologically rich languages. FastText represents words as bags of character n-grams, allowing it to compute representations even for words not seen during training.

What problem does this solve?

FastText handles rare words better.

For example:

run running runner runs

These words share subword patterns.

FastText can use those patterns to build better representations.

It also helps with misspellings and domain-specific variants.

What is still missing?

It is still static.

The word “bank” still has the same representation regardless of whether we mean a financial institution or a river bank.

Best use

FastText is good for:

rare words misspellings morphologically rich languages word-level NLP lightweight semantic features

When should you use it?

Use FastText when subword structure matters and you still want a lightweight model.

For example, it can be useful in multilingual systems, noisy user-generated text, or low-resource NLP tasks.

6. Contextual token embeddings: BERT and transformers

This is where attention enters the party.

With transformer-based models like BERT, each token gets a vector based on the full context around it.

So the word “bank” will have different embeddings in these two sentences:

I deposited money in the bank. I sat near the river bank.

In the first sentence, “bank” is close to finance.

In the second sentence, “bank” is close to geography or nature.

BERT introduced deep bidirectional representations by conditioning on both left and right context in all layers, making it powerful for language understanding tasks like question answering and inference.

What problem does this solve?

Contextual embeddings solve the biggest limitation of static embeddings:

The same word can mean different things in different contexts.

Transformers use attention so each token can “look at” other tokens in the sentence.

For example:

The bug was found near the river bank. The bug was found in the payment service.

Even the word “bug” can change meaning depending on context.

It could mean an insect.

It could mean a software defect.

Contextual models are much better at handling this.

What is the problem?

BERT-like models usually produce token-level embeddings.

For a sentence with 10 tokens, you may get 10 vectors.

But for many real-world applications, we want one vector for the whole sentence or document.

Example:

Input: “How do I reset my password?” Output needed: one vector representing the whole sentence

Raw BERT does not automatically solve sentence similarity perfectly. You need pooling, fine-tuning, or a model specifically trained for sentence embeddings.

Also, transformer models are heavier than TF-IDF or Word2Vec. They need more compute, more memory, and more careful deployment.

Best use

Contextual token embeddings are best for:

named entity recognition question answering token classification text classification fine-tuned NLP tasks language understanding

When should you use them?

Use BERT-style models when context matters at the token or sentence level, especially when you are fine-tuning for a specific NLP task.

For pure semantic search, you usually want sentence/document embedding models instead.

7. Sentence and document embeddings

Now we get to the models most engineers use for semantic search and RAG.

Instead of producing one vector per word or token, sentence embedding models produce one vector for a full sentence, paragraph, or document chunk.

Example:

Input: “How do I reset my password?”

Output: [0.12, -0.45, 0.78, …]

One input text.

One vector.

That vector captures the overall meaning.

Sentence-BERT, also called SBERT, modified pretrained BERT networks using siamese and triplet network structures to produce semantically meaningful sentence embeddings that can be compared using cosine similarity. The SBERT paper specifically notes that vanilla BERT is inefficient for large-scale semantic similarity search and that SBERT drastically reduces the comparison cost.

What problem does this solve?

Sentence embeddings solve the practical search problem.

Suppose you have 1 million internal documentation chunks.

You want to answer:

“How do I rotate database credentials?”

You embed the query.

You search for nearby document vectors.

You retrieve the most relevant chunks.

That is the backbone of many RAG systems.

What is still missing?

Sentence embeddings are great, but they are not magic.

They can still struggle with:

very long documents poor chunking domain-specific jargon numbers and exact values fresh information ambiguous queries negation complex reasoning

For example, these two sentences may look semantically close:

The service supports OAuth. The service does not support OAuth.

The difference is just one word, but that one word changes everything.

Embedding models can sometimes miss this kind of precision.

Best use

Sentence/document embeddings are excellent for:

semantic search RAG retrieval FAQ matching duplicate ticket detection recommendation systems document clustering similarity matching support automation knowledge base search

When should you use them?

Use sentence embeddings when you want to compare pieces of text by meaning.

This is the default choice for modern semantic search and RAG.

8. General-purpose embedding APIs: OpenAI and similar models

General-purpose embedding APIs are production-friendly sentence/document embedding models exposed as a service.

You send text.

You get a vector.

Example:

Input: “Explain Kubernetes pod autoscaling.”

Output: [0.018, -0.221, 0.734, …]

OpenAI’s embedding guide lists text-embedding-3-small and text-embedding-3-large as embedding models and describes embeddings as useful for search, clustering, recommendations, classification, and related use cases.

Conceptually, these are similar to sentence/document embeddings.

The difference is mostly practical:

You do not train the model. You do not host the model. You do not manage GPUs. You call an API.

What problem does this solve?

It solves the engineering burden.

You do not need to:

collect training data train embedding models maintain model infrastructure optimize inference benchmark multiple open-source models handle scaling manually

For many teams, this is a big deal.

Especially when building an MVP, internal assistant, RAG system, search engine, or recommendation prototype.

What is the problem?

The trade-offs are real:

API cost network latency vendor dependency data privacy concerns rate limits less control over training behavior possible re-embedding when models change

Also, general-purpose embeddings may not fully understand your company’s internal jargon, product names, code names, or domain-specific terminology.

Best use

General-purpose embedding APIs are best for:

RAG systems semantic search internal knowledge assistants chatbot retrieval document similarity clustering classification features recommendation prototypes production apps where speed matters

When should you use them?

Use general-purpose embedding APIs when you want strong quality quickly and do not want to manage embedding infrastructure yourself.

This is often the best default for product teams.

9. Task-specific and domain-specific embeddings

General embeddings are like good generalist engineers.

Domain-specific embeddings are like specialists.

They are trained or fine-tuned for a specific domain or task.

Examples:

medical embeddings legal embeddings financial embeddings code embeddings e-commerce product embeddings customer support embeddings security log embeddings scientific paper embeddings

A medical embedding model should understand that:

myocardial infarction ~ heart attack

A code embedding model should understand that:

“sort array in Java”

is related to:

Arrays.sort(arr);

A support-ticket embedding model should understand that:

“login loop after SSO redirect”

is different from:

“forgot password”

even though both involve authentication.

What problem does this solve?

Domain-specific embeddings solve the knowledge mismatch problem.

A general embedding model may not understand:

internal service names medical abbreviations legal clauses financial terminology code semantics company-specific acronyms log messages error codes

A domain-tuned model can perform better because it has learned the language of that domain.

What is the problem?

Specialization costs money and time.

You may need:

training data evaluation datasets ML expertise fine-tuning infrastructure model monitoring retraining pipelines privacy controls

Also, a domain-specific model may perform worse outside its domain.

A great medical embedding model may not be the best model for e-commerce search.

The Massive Text Embedding Benchmark paper makes an important point: embedding models should be evaluated across tasks, and no single embedding method dominates every task.

That is a very senior-engineering lesson:

Do not choose embeddings by hype. Choose them by evaluation.

Best use

Task-specific embeddings are best for:

high-accuracy enterprise search medical/legal/financial systems code search domain-specific RAG duplicate detection in specialized workflows support ticket routing fraud/security analysis recommendation systems with domain behavior

When should you use them?

Use domain-specific embeddings when general embeddings are not good enough and the business value justifies extra effort.

This usually happens when:

accuracy matters a lot domain vocabulary is specialized false positives are expensive you have enough data you can measure quality properly

10. Embeddings in RAG systems

In GenAI systems, embeddings are most commonly used for retrieval.

A simple RAG flow looks like this:

1. Split documents into chunks.
2. Convert each chunk into an embedding.
3. Store embeddings in a vector database.
4. User asks a question.
5. Convert the question into an embedding.
6. Find the nearest document chunks.
7. Pass those chunks to the LLM.
8. LLM generates an answer using retrieved context.

The embedding model does not generate the final answer.

It retrieves relevant context.

The LLM generates the final response.

This distinction matters.

Embedding model:

Finds relevant information.

LLM:

Reads, reasons, and writes an answer.

In a backend architecture, this may look like:

User Query ↓ Embedding Model ↓ Vector DB Search ↓ Top-K Relevant Chunks ↓ LLM Prompt ↓ Generated Answer

For production systems, I usually prefer hybrid retrieval:

BM25 + dense embeddings + reranker

Why?

Because dense embeddings are great at meaning.

BM25 is great at exact terms.

Rerankers are great at final relevance ordering.

That combination handles more real-world mess.

11. What embeddings cannot solve

Embeddings are powerful, but they are not a replacement for reasoning, databases, permissions, or good system design.

They cannot reliably solve:

1. Factual correctness

An embedding can retrieve a related document.

It does not guarantee that the document is correct, updated, or authoritative.

2. Complex reasoning

Embeddings are similarity tools.

They are not full reasoning engines.

They may know two pieces of text are related, but they do not prove an answer.

3. Exact numerical logic

Embeddings are weak for precise constraints like:

price < 1000 date after 2025–01–01 latency greater than 300 ms version exactly 1.2.7

Use databases, filters, and structured queries for that.

4. Permissions

A vector search system can retrieve sensitive chunks if you do not enforce access control.

Security must happen before or during retrieval.

Do not rely on the LLM to “be careful.”

That is how incidents get born.

5. Bad chunking

If your chunks are bad, your retrieval will be bad.

Embedding models cannot fully fix poorly split documents.

For example, if a chunk separates a heading from its explanation, the embedding may lose meaning.

6. Domain mismatch

General embeddings may fail on internal acronyms, logs, medical terms, legal language, or code-specific meaning.

7. Freshness

Embeddings are not automatically updated when documents change.

If content changes, you need a re-indexing strategy.

12. Practical decision guide

Here is how I would choose the right representation as an engineer.

13. How to evaluate embedding models

Do not pick an embedding model only because it has a fancy leaderboard score.

Evaluate it on your own data.

A simple evaluation set can look like this:

Query: “How do I reset MFA?” Expected document: “Multi-factor authentication reset process”

Query: “Payment webhook failing” Expected document: “Webhook retry and failure handling”

Query: “Rotate database credentials” Expected document: “Credential rotation runbook”

Then measure:

Recall@K Precision@K Mean Reciprocal Rank nDCG human relevance score answer quality in RAG latency cost failure cases

For RAG, retrieval quality often matters more than the LLM model.

Bad retrieval means bad context.

Bad context means bad answer.

Classic garbage in, garbage out — but now with vector databases and a nicer UI.

14. A simple mental model

Here is the clean version:

Bag of Words: “Which words exist?”

TF-IDF / BM25: “Which words are important?”

Word2Vec / GloVe: “What words have similar meanings?”

FastText: “What words have similar meanings, including rare/subword patterns?”

BERT: “What does this word mean in this sentence?”

Sentence embeddings: “What does this whole sentence or paragraph mean?”

OpenAI/general embedding APIs: “Give me strong production-ready text vectors without training my own model.”

Domain-specific embeddings: “Give me vectors optimized for my industry, data, and task.”

15. Final thoughts

Embedding models are one of the most useful ideas in modern AI engineering.

They convert messy human language into mathematical space.

That unlocks:

semantic search RAG recommendations deduplication clustering classification document discovery support automation code search

But the best engineers do not treat embeddings as magic.

They ask practical questions:

Do I need exact matching or semantic matching? Are my queries short or long? Is my domain generic or specialized? Do I need sentence vectors or token vectors? Can I use an API, or must I self-host? How will I evaluate retrieval quality? Do I need hybrid search? How will I handle access control? How will I re-index changed documents?

My practical recommendation:

For most modern GenAI applications, start with this:

BM25 + sentence/document embeddings + reranker

Use a general-purpose embedding API or a strong open-source sentence embedding model first.

Then move to domain-specific embeddings only when evaluation proves you need them.

That is the difference between building a demo and building a reliable system.

References

· Scikit-learn documentation: TfidfTransformer — TF-IDF and inverse document frequency.

· Elastic: Practical BM25 — Part 2 — BM25 scoring and ranking variables.

· Mikolov et al.: Efficient Estimation of Word Representations in Vector Space — The Word2Vec paper.

· Stanford NLP: GloVe project — Global vectors for word representation.

· Bojanowski et al.: Enriching Word Vectors with Subword Information — The FastText subword embedding paper.

· Devlin et al.: BERT — Pre-training of Deep Bidirectional Transformers for Language Understanding — The BERT paper.

· Reimers and Gurevych: Sentence-BERT — Sentence embeddings using siamese/triplet BERT networks.

· OpenAI documentation: Vector embeddings — OpenAI embedding models and use cases.

· Muennighoff et al.: MTEB — Massive Text Embedding Benchmark — Benchmarking embedding models across tasks.