Retrieve Broadly, Rerank Precisely: The Missing Stage in Your RAG Pipeline

Every RAG tutorial covers retrieval. Every RAG tutorial covers generation. Almost none of them cover what happens in between.

That middle layer — reranking — is where I saw the single biggest quality jump in our production pipeline. Faithfulness went from ~0.78 to 0.85–0.90. Not from a better prompt. Not from a bigger model. From re-scoring 20 documents before feeding 5 to the LLM.

Here’s everything I know about reranking — how it works under the hood, why it’s different from your embedding model, when it’s worth the latency cost, and how to implement it without overengineering.

The Problem Reranking Solves

Your retrieval pipeline — whether it’s pure vector search or hybrid — returns a ranked list of documents. Let’s say the top 20. That ranking is approximate.

Why approximate? Because your embedding model (the bi-encoder) made a shortcut. It encoded the query and every document independently, then compared their vectors using cosine similarity. It never looked at the query and document together. It’s comparing photographs of two people instead of putting them in the same room.

This shortcut is necessary — you can’t jointly process the query against every document in a 50,000-doc corpus. The latency would be minutes, not milliseconds. But the shortcut has a cost: the ranking is good enough to get the right documents into the top 20, but not precise enough to guarantee the best ones are in the top 5.

That top 5 is what your LLM sees. And the LLM trusts what you give it. Feed it the wrong 5 documents, and it’ll confidently generate the wrong answer from the wrong context.

Reranking fixes the ranking. It takes those top 20 approximate results and re-scores them with a model that does look at the query and document together.

Bi-Encoder vs. CrossEncoder — The Core Difference

This is the most important distinction in retrieval, and most tutorials gloss over it.

Bi-Encoder (Your Embedding Model)

A bi-encoder processes the query and document through two independent forward passes:

Query  → Encoder → Query Vector  (768/1536-dim)
                                    ↓
                              Cosine Similarity → Score
                                    ↑
Doc    → Encoder → Doc Vector    (768/1536-dim)

How it works:

1. The query gets encoded into a vector — once, at query time.
2. Every document was encoded into a vector at index time — this is precomputed and stored.
3. At query time, you just compute cosine similarity between the query vector and every stored doc vector.

Why it’s fast: Document vectors are precomputed. At query time, you only encode the query once, then do vector math (which is blazing fast with HNSW/IVF). Searching 500,000 documents takes milliseconds.

Why it’s shallow: The query and document never “see” each other. The encoder has to compress all possible meanings of a document into a single fixed-size vector, without knowing what question will be asked about it. This is like writing a summary of a book without knowing what the reader is looking for.

What it misses:

• Fine-grained relevance between specific query terms and specific document passages
• Negation handling (“which documents do NOT mention penalties”)
• Complex multi-hop queries where relevance depends on how query parts interact with document parts

CrossEncoder (The Reranker)

A CrossEncoder processes the query and document as a single concatenated input:

[Query + Document] → Transformer → Single Relevance Score (0 to 1)

How it works:

1. Concatenate the query and document into one text: [CLS] query text [SEP] document text [SEP]
2. Feed the entire concatenated text through a transformer model
3. The model outputs a single relevance score

Why it’s accurate: The transformer’s self-attention mechanism lets every token in the query attend to every token in the document — and vice versa. The model sees the query and document in conversation. It understands not just “are these about the same topic?” but “does this document actually answer this specific question?”

Why it’s slow: Nothing is precomputed. For every query, you need a separate forward pass for every candidate document. Reranking 20 documents = 20 forward passes. That’s why you can’t use a CrossEncoder for initial retrieval — running it against 50,000 documents would take seconds to minutes.

The tradeoff is clear: Bi-encoders are fast but approximate. CrossEncoders are slow but precise. The smart architecture uses both — bi-encoder to get the top 20, CrossEncoder to re-score and pick the best 5.

How CrossEncoder Attention Changes Everything

Let me show you why this matters with a concrete example from our production system.

Query: “What is the penalty for late filing of Form 1040?”

Document A: “Form 1040 must be filed by April 15. Taxpayers who file after the deadline may face a failure-to-file penalty of 5% of unpaid taxes per month, up to 25%.”

Document B: “Form 1040 is the standard individual income tax return used by U.S. taxpayers. It covers wages, salaries, tips, and other forms of income.”

A bi-encoder encodes both documents independently. Both are about Form 1040, both are about taxes. Their vectors will be reasonably close to the query vector. Document B might even rank higher if the corpus has more general Form 1040 content that biases the embedding space.

A CrossEncoder reads the query and each document together. When processing Document A, the attention mechanism connects “penalty” in the query to “failure-to-file penalty” in the document. When processing Document B, it finds no connection for “penalty” or “late filing” — just general Form 1040 information. Document A gets a high relevance score. Document B gets a low one.

This is the difference between “topically related” and “actually answers the question.”

The Production Pipeline — Where Reranking Fits

User Query
    │
    ▼
┌─────────────────────────────────────┐
│  Stage 1: Retrieval (Bi-Encoder)    │
│  Hybrid Search (HNSW + BM25 + RRF) │
│  Latency: ~100–200ms               │
│  Output: Top 20 candidates          │
└─────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────┐
│  Stage 2: Reranking (CrossEncoder)  │
│  Joint query-doc scoring            │
│  Latency: ~200–300ms               │
│  Output: Top 5, re-ordered          │
└─────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────┐
│  Stage 3: Generation (LLM)         │
│  GPT-4 with top-5 context          │
│  Latency: ~2–3s                    │
│  Output: Grounded response          │
└─────────────────────────────────────┘

Latency budget breakdown:

• Retrieval: 100–200ms (fast — precomputed vectors + HNSW)
• Reranking: 200–300ms (moderate — 20 forward passes on a small model)
• Generation: 2–3 seconds (slow — GPT-4 autoregressive generation)
• Total: 3–5 seconds

Notice that GPT-4 generation is 60–70% of the total latency. The reranker adds 200–300ms to a 3–5 second pipeline — a 5–10% increase for a massive quality improvement. The ROI is absurd.

Choosing a Reranker Model

Not all CrossEncoders are equal. Here’s what’s available:

Model Provider Speed Quality Best For cross-encoder/ms-marco-MiniLM-L-6-v2 SBERT Very fast Good Low-latency production, prototyping cross-encoder/ms-marco-MiniLM-L-12-v2 SBERT Fast Better Balanced production use BAAI/bge-reranker-v2-m3 BAAI Moderate Excellent Multilingual, high accuracy BAAI/bge-reranker-large BAAI Slower Excellent English, maximum accuracy Cohere Rerank v3 Cohere API-dependent Excellent Managed API, no GPU needed Jina Reranker v2 Jina AI Fast Very good Long documents (8K context)

Our choice: We used cross-encoder/ms-marco-MiniLM-L-6-v2 in production. Why? It's 6 layers (tiny), runs in ~15ms per document on CPU, and was accurate enough for our use case. Reranking 20 documents took ~300ms total without a GPU.

When to upgrade: If you’re seeing faithfulness below 0.85 despite good retrieval, try bge-reranker-large or Cohere Rerank. The accuracy gap between MiniLM-L-6 and bge-reranker-large is real — roughly 2–4% on NDCG@10 benchmarks. Whether that matters depends on your domain.

Implementation — It’s Simpler Than You Think

Option 1: Direct CrossEncoder (Self-Hosted)

from sentence_transformers import CrossEncoder

# Load once at startup
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, documents: list[str], top_k: int = 5):
    # Create query-document pairs
    pairs = [[query, doc] for doc in documents]

    # Score all pairs
    scores = reranker.predict(pairs)

    # Sort by score, return top_k
    scored_docs = sorted(
        zip(documents, scores), 
        key=lambda x: x[1], 
        reverse=True
    )
    return scored_docs[:top_k]

# Usage
query = "What is the penalty for late filing?"
top_20_docs = hybrid_search(query, k=20)  # From your retrieval
top_5_docs = rerank(query, top_20_docs, top_k=5)

Option 2: Cohere Rerank API (Managed)

import cohere

co = cohere.Client("your-api-key")

def rerank_cohere(query: str, documents: list[str], top_k: int = 5):
    response = co.rerank(
        model="rerank-english-v3.0",
        query=query,
        documents=documents,
        top_n=top_k
    )
    return [
        (documents[r.index], r.relevance_score) 
        for r in response.results
    ]

Option 3: LangChain Integration

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

# Setup
cross_encoder = HuggingFaceCrossEncoder(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
)
compressor = CrossEncoderReranker(
    model=cross_encoder, 
    top_n=5
)

# Wrap your existing retriever
reranking_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=your_hybrid_retriever  # Your existing retriever
)

# Use it — reranking happens automatically
docs = reranking_retriever.invoke("What is the penalty for late filing?")

The LangChain approach is the cleanest for production — it wraps reranking into your existing retriever chain with zero changes to the rest of your pipeline.

Tuning the Reranker — What I Learned

How Many Candidates to Retrieve?

The number you pass to the reranker matters more than you’d think.

• Too few (top 5 → rerank → top 5): Pointless. The reranker can only re-order what you give it. If the best document is at position 8, you’ll never see it.
• Too many (top 100 → rerank → top 5): Diminishing returns plus unnecessary latency. Each additional document costs ~15ms of reranking time.
• Sweet spot (top 15–25 → rerank → top 5): In our testing, top 20 captured the relevant documents 95%+ of the time while keeping reranking under 300ms.

How Many to Send to the LLM?

• Top 3: Fast, but you might miss supporting context.
• Top 5: Our sweet spot. Enough context without overwhelming the model.
• Top 10: Only if your documents are very short (< 200 tokens each). More context = more tokens = higher latency and cost.

When NOT to Use a Reranker

• Corpus under 1,000 documents: Your bi-encoder retrieval is probably precise enough already. The reranker won’t add much.
• Latency budget under 2 seconds: If every millisecond counts (real-time autocomplete, instant search), the 200–300ms overhead might not be affordable.
• Homogeneous short queries: If 95% of queries are simple keyword lookups (“price of product X”), BM25 alone might be sufficient.

The Impact — Our Numbers

Metric Without Reranker With CrossEncoder Reranker Faithfulness ~0.78 0.85–0.90 Context Precision ~0.72 0.82+ Answer Relevancy ~0.80 0.86+ Latency (added) — +200–300ms User complaints (wrong answers) Weekly Rare

The faithfulness jump alone justified the reranker. But the reduction in user complaints about wrong answers was what made stakeholders stop questioning the investment.

Reranking vs. Other Quality Improvements

Where does reranking sit relative to other things you could do?

Improvement Effort Quality Impact When to Do It Better chunking strategy Medium Medium First — garbage chunks = garbage retrieval Hybrid search (add BM25) Low High Second — handles keyword queries CrossEncoder reranking Low High Third — fixes ranking precision Better embedding model Medium Medium Fourth — if retrieval recall is still low Prompt engineering Low Low–Medium Anytime — but won’t fix bad retrieval Fine-tuning the LLM High Variable Last resort — expensive and fragile

Reranking sits in the rare “low effort, high impact” quadrant. It’s a few lines of code, a small open-source model, and 200ms of latency. Compare that to fine-tuning an LLM — weeks of work, thousands of dollars in compute, and results that might not generalize.

Advanced: Two-Stage Reranking

For very large corpora (1M+ documents), some teams use two reranking stages:

Retrieval (top 100) → Light reranker (top 20) → Heavy reranker (top 5) → LLM

• Light reranker: A small, fast CrossEncoder (MiniLM-L-6) that quickly filters 100 → 20
• Heavy reranker: A larger, more accurate model (bge-reranker-large or Cohere) that precisely ranks 20 → 5

We didn’t need this at 50,000 documents. But if your corpus is in the millions and retrieval recall drops, this two-stage approach recovers quality without blowing up latency.

The Bottom Line

Reranking is the highest-ROI upgrade you can add to a RAG pipeline after hybrid search. It’s low-effort, low-latency, and directly improves the metrics that matter — faithfulness, context precision, and answer relevancy.

The pattern is simple: retrieve broadly (top 20 via hybrid search), rerank precisely (CrossEncoder → top 5), generate from the best context. Three stages, each doing what it’s best at.

If you’re doing retrieval → generation without reranking in between, you’re leaving quality on the table. Add a CrossEncoder. It’ll take an afternoon to implement and permanently improve every answer your system generates.