**What Role Does the Embedding Model Actually Play in a RAG System?**

A few months ago, I built a RAG demo to help quickly retrieve recurring questions from our team’s on-call rotations. I went through three iterations of optimization: starting with pure vector retrieval, then upgrading to hybrid retrieval (which gave a noticeable boost), and finally replacing the embedding model from text-embedding-ada-002 to text-embedding-3-small.

The third experiment surprised me. According to public benchmarks, 3-small significantly outperforms ada-002 in discrimination ability. Yet in my RAG system, this advantage didn’t translate into any measurable improvement. This pushed me toward a more fundamental question: what role does the embedding model actually play in a RAG system?

The Setup

The dataset was small — just over 1,000 plain text documents, each ranging from a few hundred bytes to a few kilobytes. To make optimization measurable, I had an LLM generate an evaluation set from the original corpus, and ran the same evaluation after every change to produce a comparison report. (It looked rigorous on the surface — we’ll come back to this.)

The system consists of two pipelines. The ingestion pipeline converts documents into retrievable vectors and stores them in the database. The query pipeline takes user queries and returns generated answers.

Ingestion pipeline: import docs → receive & deduplicate → enqueue → async consume → rate limit → chunk → embed → store metadata + vectors

Query pipeline: query → embed → retrieve → top-K chunks → (chunks + query) → LLM → generated answer (with source citations)

v1: Vector retrieval only. Results were poor. I tried different chunk sizes (even no chunking at all), but nothing moved the needle.

v2: Introduced Elasticsearch with document metadata indexing, switching to hybrid retrieval. Results improved significantly, though several evaluation metrics remained low.

v3: Replaced the embedding model. Since I was on an OpenAI account, the model recommended to me was ada-002; I switched it to 3-small. To my surprise, nothing changed.

To understand why benchmark results didn’t translate to my system, I needed to dig into how embeddings actually work.

What Is an Embedding, Really?

Embedding isn’t a single technique — it’s a family of techniques for converting discrete data into dense vectors in a continuous vector space. The diagram below shows the basic taxonomy.

Embeddings come in two main flavors: word embeddings and sentence embeddings. Word embeddings can be aggregated into sentence embeddings via various pooling strategies.

Following an AI recommendation, I dug into a foundational paper on this topic: “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”. It traces the evolution from BERT to SBERT and explains why a dedicated sentence embedding architecture is needed.

BERT vs. SBERT: A Tail of Two Architectures

BERT uses a cross-encoder architecture: two sentences are concatenated with a [SEP] token and fed into the model together. This benefits from cross-sentence attention, capturing richer interactions between the two inputs. The downside: it can only be computed online — there’s no way to precompute representations. For large-scale retrieval — say, finding the most relevant document for a query from millions of documents — this means millions of model calls, which is rarely acceptable in production.

SBERT uses a Siamese (bi-encoder) structure. Built on top of BERT, it feeds two (or more) sentences through the same encoder — same model, same weights — independently, producing word embeddings for each. A pooling strategy then converts these into sentence-level embeddings, which can be fine-tuned with various objective functions.

The Siamese architecture has two costs: no cross-sentence attention, and the output is compressed to a fixed-length vector. This sacrifices some accuracy, but in return enables precomputation, giving order-of-magnitude efficiency gains over cross-encoders at retrieval time.

This is a classic engineering trade-off: use a fast-but-less-accurate method to narrow the search space, then apply a slower-but-more-accurate method for fine-grained ranking — striking a balance between latency and precision.

Three Objective Functions, Three Use Cases

The paper presents three objective functions, each suited to different data conditions:

(1) Classification objective — cross-entropy loss

Formula: o=softmax(Wt(u,v,|u−v|))

Best for: large-scale datasets with sentence pairs labeled by discrete categories (e.g., contradiction / entailment / neutral). Coarse-grained, low annotation cost.

(2) Regression objective — squared-error loss

Formula: cosine similarity of two sentence embeddings

Best for: smaller datasets with continuous similarity scores (e.g., 0–5 or 0–10 ratings). Fine-grained, higher annotation cost.

(3) Triplet objective — triplet loss

Formula: max(||sa−sp||−||sa−sn||+ε,0)

Best for: datasets with relative preferences but no absolute labels. For example: a user searches for query A, clicks on B but not C — so (A, B, C) becomes an (anchor, positive, negative) triplet.

CLS Pooliing vs. Mean Pooling

BERT outputs word embeddings along with a [CLS] token embedding, which is used for sentence level tasks in the original BERT. However, in the sentence embedding domain, many studies have shown that [CLS] pooling underperforms mean pooling. My own experiments confirmedthis: the absolute cosine similarity between similar sentences rose, but discrimination declined.

Two likely reasons:

1. The [CLS] token was designed for NSP, not for semantic representation. BERT’s pre-training includes a Next Sentence Prediction objective, where [CLS] is trained to determine whether two sentences appear consecutively in a corpus — not to capture the sentence’s semantic meaning.

2. [CLS] pooling relies on a single token, while mean pooling aggregates information across all tokens. This makes [CLS] pooling more brittle and information-poor by comparison.

What The Data Actually Shows

The following table compares model performance across three types of sentence pairs: similar, unrelated, and contradictory.

A few takeaways from the data:

1. Sentences with contradictory meanings still score high in similarity — sometimes comparable to genuinely similar pairs. This is because most embedding models are trained on topical relatedness, not on the kind of semantic equivalence we’d intuitively expect. Two sentences that disagree on a topic still share that topic, so the model rates them as related. This is a critical thing to remember in production.
2. Vanilla BERT with [CLS] pooling has worse discrimination than mean pooling, for the reasons discussed above.
3. SBERT shows significantly better discrimination than BERT+pooling. The gap between similar pairs and unrelated pairs is much wider — even though absolute similarity scores drop. For embeddings, discrimination matters more than absolute scores.
4. OpenAI’s 3-small / 3-large outperform ada-002 in discrimination. Encouraging on benchmarks — but, as we’ll see, this didn’t show up in my project.

Finally, let’s look at the evolution timeline of these models — this gives a sense of how to approach model selection in practice.

Back to the original question

Why didn’t switching embedding models help?

In hindsight, the answer is clear. My dataset has only ~1,000 documents, with topics tightly clustered. At this scale, pure vector retrieval already recalls most relevant documents, and hybrid retrieval’s keyword fallback catches the rest. By the time we look at top-K, the relevant documents are almost always there.

Embedding model differences mostly affect edge cases: when two documents are topically close but semantically distinct, a better embedding model orders them more accurately. But in my dataset, such edge cases are rare — so the differences between embedding models get smoothed out by the rest of the pipeline.

This finding aligns neatly with the two-stage architecture described in the SBERT paper: bi-encoders for coarse recall, cross-encoders for fine-grained ranking.

A Second, More Unconfortable Lesson

A second lesson, more uncomfortable than the first: my evaluation methodology may have been part of the problem. Using an LLM to generate evaluation queries from the same corpus risks creating questions whose answers are too easy to find — the LLM tends to paraphrase the source rather than test edge cases. A more rigorous setup would mix LLM-generated queries with real on-call questions, and explicitly include hard negatives. Without this, my evaluation might have been measuring the wrong thing — which is another way the embedding swap could have been “hidden” from me.

The lesson for me: the next thing to try isn’t another embedding model — it’s a reranker.