Your AI Didn’t Hallucinate. It Just Looked in the Wrong Place First.

The unglamorous machinery that decides whether your RAG app is brilliant or useless and why almost everyone blames the wrong component.

You build a RAG app. You wire up a powerful LLM, point it at your company’s documents, and ask it a question you know the answer to. It’s right there in the docs.

The model responds with a confident, fluent, beautifully worded answer. And it’s wrong.

So you do what everyone does. You blame the model. Maybe it’s not smart enough. Maybe you need a bigger one, a better prompt, a fine-tune. You spend a week tuning the generation step.

But here’s the uncomfortable truth. In a huge number of these cases, the model never had a chance. The right document never made it in front of it. The failure happened in the few hundred milliseconds before the LLM wrote a single word, in a step most people barely think about.

That step is retrieval ranking. And once you understand it, a startling amount of “the AI is unreliable” turns out to be “the retrieval was bad.”

Let me show you how it actually works, and why it quietly decides the fate of every RAG system you’ll ever build.

First: what an LLM actually does when it “looks something up”

A raw LLM only knows what was baked into its weights during training. It doesn’t know your internal docs, last week’s tickets, or today’s news. So we use a pattern called Retrieval-Augmented Generation (RAG): before the model answers, we fetch relevant documents and stuff them into its context window as reference material. The model then answers from that material.

Which means the whole system rests on a single fragile assumption:

The documents we hand the model actually contain the answer.

If they do, even a modest model can give a great grounded answer. If they don’t, the smartest model on earth will do the thing LLMs do best, which is produce something fluent and plausible. We call that a hallucination. But often it’s not the model malfunctioning. It’s the model faithfully working with the wrong source material it was given.

So the real question becomes: how do we pick the right documents out of thousands or millions? That’s retrieval ranking, and it has a hard problem at its core.

The fundamental tension: fast OR smart, pick one

Say your knowledge base has ten million chunks of text. A user asks a question and expects an answer in about a second.

You have two competing desires:

1. Be thorough. Carefully compare the question against every chunk to judge relevance precisely.
2. Be fast. Get those chunks to the LLM before the user gives up.

The catch is that the most accurate way to judge relevance is also the most expensive. Running your smartest relevance model against all ten million chunks for every single query would give gorgeous results, and take minutes per question. Unusable.

Nearly every serious system resolves this the same way. It splits the work into two stages.

Retrieval ranking in one picture. From millions of chunks down to the handful your LLM actually reads.

The two-stage architecture: retrieve, then rerank

This is the single most important idea in the whole field:

Stage 1 (Retrieval): A fast, cheap method casts a wide net and pulls a few hundred plausible candidates out of the millions.

Stage 2 (Reranking): A slower, smarter method carefully reorders just those few hundred, so the truly best ones land on top and into the LLM’s context.

Think of it like hiring. You don’t run deep interviews with all 10,000 applicants. A cheap filter gets you to 100 reasonable candidates, then you spend your expensive attention only on that shortlist.

Stage 1 optimizes for recall, meaning don’t miss the good stuff, even if you grab some junk along the way. Stage 2 optimizes for precision, meaning given this shortlist, get the order exactly right.

This matters enormously for RAG, because an LLM’s context window is small and expensive. You can only fit a handful of chunks in it. So it’s not enough to retrieve the right chunk somewhere in the top 500. It has to be ranked into the top 3 or 5 that actually make it into the prompt. Reranking is what gets the answer from “technically retrieved” to “actually seen by the model.”

The full pipeline. The user’s question flows through a fast retriever, then a precise reranker, and only the top few chunks reach the LLM.

Let’s look at each stage.

Stage 1: Retrieval, casting the wide net

There are two big families of retrieval, plus a hybrid that’s become the default in modern RAG.

Lexical retrieval (matching words)

The classic approach matches the actual words in the query against the words in documents. The workhorse is BM25, and despite being decades old, it’s still shockingly hard to beat.

BM25 is smarter than a naive word count. It builds in two intuitions. First, rare words matter more: in “quantum decoherence,” the word “quantum” carries more signal than “the.” Second, repetition has diminishing returns: a chunk mentioning “quantum” twenty times isn’t ten times more relevant than one mentioning it twice.

Lexical search is fast, interpretable, and excellent when the user’s exact words (names, error codes, product IDs) appear in the document. Its weakness is that it doesn’t understand meaning. Ask about “car” and it won’t connect you to a chunk that only says “automobile.” Different words, missed match.

Dense retrieval (matching meaning)

This is where embeddings come in, the same family of technology that sits underneath LLMs. An encoder converts both the query and every document chunk into a vector, which is a long list of numbers capturing meaning rather than words. Texts about similar concepts land close together in this space, and unrelated texts land far apart.

To retrieve, you embed the query and find the nearest document vectors. Now “car” and “automobile” sit right next to each other, because they mean the same thing, so the exact words no longer have to match.

Dense retrieval turns text into coordinates in “meaning space,” then finds the nearest neighbours, which is why synonyms and paraphrases still match.

This is called a bi-encoder, because query and documents are encoded separately. That independence is the trick. You embed all your chunks ahead of time into a vector database, and at query time you only embed the one incoming question and do a fast nearest-neighbour lookup. The catch is that because query and document never “see” each other during encoding, the matching is a little blunt. The model squashes a whole chunk into one vector and hopes it kept the right things.

Hybrid retrieval

Lexical and dense fail in different ways. Lexical misses synonyms, and dense sometimes fumbles exact keywords, names, or codes. So modern RAG systems increasingly run both and fuse the results, getting BM25’s keyword precision plus the semantic reach of embeddings.

Whatever method you use, Stage 1 outputs a shortlist of maybe 100 to 500 candidates that are probably relevant. Now we make it precise, because “probably” isn’t good enough to put in front of your LLM.

Stage 2: Reranking, the careful second look (the part people skip)

Here’s where the biggest quality jump in most RAG systems hides, and it turns on one architectural difference.

A bi-encoder encoded query and document separately. A cross-encoder does the opposite. It feeds the query and a document into the model together, and lets every word in the query attend directly to every word in the document.

The key difference. A bi-encoder processes query and document separately (fast, for retrieval), while a cross-encoder processes them together (accurate, for reranking).

This is dramatically more accurate. Instead of comparing two pre-computed summaries, the model reads the question and the chunk side by side and asks: given this exact question, how relevant is this exact chunk? It catches subtle relevance that a blunt vector comparison smears over.

So why not use a cross-encoder for everything? Cost. Because query and document must be processed together, nothing can be pre-computed. Every query and document pair is a fresh full model pass. Across ten million chunks per query, that’s hopeless. Across the few hundred candidates from Stage 1, it’s totally feasible. That is the magic of the two-stage design. The expensive, accurate model only ever looks at a tiny, pre-filtered shortlist.

The cross-encoder scores each candidate, you sort by that score, and the top few go into your LLM’s context. In practice, adding a reranker is one of the highest-leverage, lowest-effort upgrades you can make to a RAG pipeline. It’s often the difference between the right chunk sitting at rank 30 (invisible to the model) and rank 2 (right in the prompt).

How does the system “know” what’s relevant?

Mostly, it’s learned, not hand-coded, in a subfield called Learning to Rank. Rerankers are trained on examples of queries paired with documents labelled by relevance, from human raters and, at scale, from real user behaviour. Every click, every skipped result, every lingered-on link is a quiet signal that says this was relevant, that wasn’t. Over billions of interactions, the system learns what separates a good result from a bad one.

How do you measure if your ranking is any good?

You can’t fix what you can’t measure, and for RAG this is where most teams are flying blind. Three metrics are worth knowing.

Recall@k asks: of all the relevant chunks that exist, how many landed in the top k? This is your retrieval stage’s report card. Did the answer even make the shortlist?

Recall@k = (relevant documents found in top k) / (total relevant documents that exist

MRR (Mean Reciprocal Rank) asks: how high up is the first correct chunk? It’s ideal for question-answering, where you mostly need one right source near the top.

NDCG (Normalized Discounted Cumulative Gain) is the heavyweight. It rewards putting highly relevant results near the top and discounts good results that appear further down.

For RAG specifically, these aren’t academic. If your Recall@k is low, no amount of prompt engineering will save you, because the answer literally isn’t in the room. Measuring retrieval quality separately from generation quality is the single most useful debugging habit you can build.

Bringing it home: the failure was never the model

Let’s return to that confident, wrong answer from the start.

When the right chunk doesn’t make it into the top results, your LLM doesn’t say “I don’t know.” It does what it was built to do, which is generate a fluent, plausible continuation from whatever weak context it was given. The ranking failure becomes a confident lie, and you spend your week tuning the model, the one component that was working fine.

This is why retrieval ranking, a decades-old idea built for search engines, has quietly become one of the load-bearing pillars of modern AI. The chatbot gets the applause. The ranker does the work. And the quality of every grounded answer your system produces is capped by the quality of what your retrieval ranking puts in front of the model.

So the next time an AI gives you a confident, wrong answer, ask the better question. Not “why is this model so dumb?” but “what did it actually get to read first?”

The one-paragraph version

RAG only works if the right documents reach the model. Retrieval ranking is what decides that, and it solves the speed versus accuracy tradeoff by splitting the job in two. A fast retriever (lexical, dense, or hybrid) grabs a few hundred candidates optimizing for recall, then a slow, accurate cross-encoder reranker reorders that shortlist optimizing for precision, so only the best chunks enter the LLM’s context. Skip the reranker and the right answer often sits just out of reach at rank 30. Measure it with Recall@k and NDCG, not vibes. Because when an AI hallucinates, it usually didn’t fail to think. It failed to look in the right place first.

Building or debugging a RAG pipeline? I’d love to hear where retrieval has burned you (or saved you) in the responses. In my experience, the gap between a demo that works and a system that’s genuinely reliable is almost always hiding in the ranking.

And feel free to connect with me on LinkedIn.