Reranking in RAG: Cross-Encoders, Cohere Rerank & FlashRank.

Part 3 of my RAG Engineering Series, if you haven’t read Part 1 (PageIndex) and Part 2 (Hybrid Search + RRF), they’ll give you useful context, but you can follow along here independently.

The Bug That Wasn’t a Bug

Sometime ago while building my RAG pipeline, I had a problem I couldn’t explain. My retrieval was working. I could look at the top-5 chunks returned for any given query and manually verify that at least one of them contained exactly the right information. The answer was in there. And yet the LLM kept producing mediocre, hedging, sometimes flat-out wrong responses.

I spent a alot of time suspecting my chunking strategy. Then I blamed my embedding model. Then I rewrote my prompts three different ways. None of it moved the needle meaningfully.

Then I added one line of logging: the position of the “gold” chunk in the context window passed to the LLM.

Position 4. Position 5. Sometimes position 3. Occasionally position 1, but not often enough.

That was the whole problem. The right answer was there, it just wasn’t first. The LLM was reading a context window where the most relevant chunk was buried under three or four less relevant chunks that had happened to score higher in vector similarity. The context window was full, the gold chunk was near the bottom, and the model was giving it appropriate-but-insufficient weight.

Every engineer who has built a non-trivial RAG pipeline has hit this. Retrieval recall isn’t your problem. Retrieval precision at position 1 is your problem.

That’s what reranking fixes.

Why Vector Search Rankings Are Imprecise

Remember that embedding models are not trained to rank. They’re trained for semantic similarity at scale, to pull broadly relevant documents out of a corpus of millions. ANN (Approximate Nearest Neighbor) search is optimized for recall, not precision. The difference matters.

Think of it like casting a net in the ocean. A wide net with the right mesh size will reliably catch the fish you want, but the order in which fish end up in the net is essentially random. You’ve captured a great candidate set. The ranking within that set tells you almost nothing about which fish is best.

Your top-20 vector search results are probably all semantically relevant. Their order, though, is noise. The cosine distance between query embedding and document embedding is a blunt instrument for fine-grained relevance scoring. The model that produced those embeddings encoded each one independently, it had no idea, while encoding your document, what query it would eventually need to match against.

This is the fundamental mismatch reranking solves. And it leads directly to the two-stage retrieval paradigm that I now consider the correct mental model for production RAG:

1. Stage 1 (Retrieve broadly): Cast a wide net. Use vector search, hybrid search, whatever gives you the best recall. Retrieve N=50 or N=100 candidates.
2. Stage 2 (Rank precisely): Hand-pick the best fish from the net. Use a more powerful, query-aware model to score each candidate against the specific query and reorder them.

The two stages have different jobs and should use different tools. Conflating them, expecting your bi-encoder to do precise relevance ranking is where most pipelines go wrong.

What Is Reranking?

Reranking takes the candidate set from Stage 1 and re-scores each document against the query using a model that can actually see both together.

This is the key shift: your retrieval model embeds the query and each document independently, then computes similarity between the resulting vectors. The reranker embeds the query and document jointly, it sees them at the same time, can compute cross-attention between query tokens and document tokens, and produces a relevance score that’s far more calibrated.

The output is a reordered list. You take your top-50 from hybrid search, pass them through the reranker with the original query, get back 50 relevance scores, sort descending, and keep the top-5 for your LLM context. Those top-5 are now ordered by actual relevance, not by vector similarity approximation.

This also interacts beautifully with the PageIndex approach from Part 2. When your candidates are page-level units, i.e., coherent & self-contained pages rather than arbitrary sentence-level chunks, the reranker has richer, more contextually complete text to score. Reranking page-level candidates is particularly effective because each candidate is a meaningful unit, not a fragment that might lack context on its own.

Bi-Encoder vs Cross-Encoder: The Core Distinction

Bi-encoders: the architecture behind most embedding models. encode the query and document separately through the same (or similar) encoder, producing independent vectors. Similarity is computed as cosine distance or dot product between those vectors. This is what makes them fast and scalable: you can pre-compute document embeddings offline, and at query time you only need to encode the query and do a fast vector lookup. You can search millions of documents in milliseconds.

But there is a tradeoff: because the query and document are encoded independently, the model has no ability to compare specific query terms against specific document terms. It produces a single embedding that represents “the gist” of each text, and gist-to-gist comparison is inherently imprecise.

Cross-encoders: It flip this entirely. Instead of encoding query and document separately, you concatenate them([CLS] query [SEP] document [SEP] ) and pass the whole thing through the encoder as a single input. Every transformer attention head can now attend between query tokens and document tokens. The model can notice that the query asks about "side effects of metformin in diabetic patients over 65" and that the document contains exactly that phrase in a clinically relevant context. That kind of fine-grained term interaction is impossible in a bi-encoder.

The output of a cross-encoder is a single relevance score (typically a sigmoid over the final [CLS] token representation). That score is dramatically more accurate than cosine similarity for precise relevance ranking.

So why don’t we just use cross-encoders for everything? Cost. Cross-encoder inference is O(N) in the number of documents. every query-document pair requires a full forward pass. For a corpus of 10 million documents, running a cross-encoder against every one at query time would take minutes, not milliseconds. That’s why we use bi-encoders for Stage 1 (fast recall over the full corpus) and cross-encoders for Stage 2 (precise scoring of a small candidate set). The two stages complement each other.

Once I understood this architecture distinction, the whole two-stage paradigm clicked. You’re not choosing between fast and accurate, you’re using fast where you need scale and accurate where you need precision.

The Landscape of Reranking Options

There are three practical options, and the right choice depends on your latency budget, infrastructure, and quality requirements.

1. Open Source Cross-Encoders (sentence-transformers)

The most accessible starting point. The sentence-transformers library ships with cross-encoder support, and there are several excellent pre-trained models available:

• **cross-encoder/ms-marco-MiniLM-L-6-v2**: My default recommendation. Fast, small, trained on MS MARCO passage ranking. Great balance of speed and quality.
• **BAAI/bge-reranker-v2-m3**: Stronger quality, multilingual, but heavier. Worth it if you need non-English support.
• **cross-encoder/ms-marco-electra-base**: Higher quality than MiniLM but 3–4x slower.

On CPU, expect 100–300ms for scoring 50 candidates with the MiniLM model, depending on document length. On a single GPU, you can get this under 50ms easily. For many use cases, especially async pipelines, CPU inference with batching is totally acceptable.

The appeal here is obvious: no API dependency, no per-call cost, full control, works offline. If you’re cost-sensitive or running in an air-gapped environment, start here.

2. Cohere Rerank API

Cohere’s managed reranking API is the simplest integration. one API call, their model does the scoring, you get back ranked results. The current rerank-v3 model is genuinely strong, particularly on business and technical text.

The integration is clean, you send a query and a list of document strings, you get back an ordered list with relevance scores. No model hosting, no GPU required on your side.

The gotchas I’ve hit are- document length limits mean you sometimes need to truncate candidates before sending them, which can subtly hurt quality if the relevant passage is in the middle of a long document. API latency adds 100–400ms to your pipeline p50, and this can spike under load. At scale (millions of queries/day), the per-call cost adds up faster than you’d expect. Run the math before committing.

When to use it? if you want high quality without the operational overhead of running your own model, you’re in early stages and don’t want to manage model infrastructure, or you’re in a domain where Cohere’s model performs particularly well on your evaluation set.

3. FlashRank

FlashRank is a lightweight Python library specifically designed for low-latency CPU reranking. The models are small, quantized, and optimized for speed over raw quality but “lower quality” is relative, and for many production use cases, FlashRank quality is more than sufficient.

FlashRank shines when, for example, sub-20ms reranking of 50 candidates on CPU. If your latency budget is tight and you’re running on CPU-only infrastructure (common in cost-conscious production environments), FlashRank is often the only practical option. The quality gap vs. a full cross-encoder is real but smaller than you’d think for most general-domain corpora.

One thing to be aware of is that the quality difference between FlashRank model options (ms-marco-TinyBERT-L-2-v2 vs ms-marco-MiniLM-L-12-v2) is significant. Don't just grab the default without benchmarking on your data.

How Reranking Fits Into the RAG Pipeline

Let me be precise about the pattern, because the details matter.

The retrieve → rerank → truncate → generate pattern looks like this:

1. Retrieve broadly: Run hybrid search (dense + sparse + RRF, as covered in Part 2) to get N candidates. N should be large enough that the correct answer is almost certainly in the set. I typically use N=50. Going above 100 rarely helps and increases reranker latency.
2. Rerank precisely: Pass all N candidates and the original query to your reranker. Score every candidate. Sort by score descending.
3. Truncate to K: Keep only the top-K reranked results for LLM context assembly. K is typically 3–7, depending on your context window and document length.
4. Generate: Assemble context from top-K reranked chunks and call the LLM.

The N >> K principle is crucial. If N and K are close together (say, N=10, K=5), reranking adds latency but provides limited benefit. you’re not giving the reranker much room to work with. The power of reranking comes from having a large, noisy candidate set and dramatically compressing it to a small, precise set. I use N=50 → K=5 as my default ratio (10:1). You’re essentially saying “cast a wide net to guarantee recall, then ruthlessly curate for precision.”

The Full Pipeline

Below is the complete RAG pipeline with reranking, showing both the hybrid retrieval stage from Part 2 and the new reranking stage:

Stage 1 (blue) is your retrieval machinery from Part 2. Stage 2 (orange) is the new reranking layer. Keeping these mentally distinct helps when debugging quality issues: if your NDCG@5 is low, you need to figure out whether the problem is at the retrieval stage (gold answer not in top-50) or the reranking stage (gold answer in top-50 but reranker scores it poorly).

Request Lifecycle

The latency annotation is important context. Reranking adds real cost to your pipeline. On CPU with a MiniLM cross-encoder and N=50 candidates, expect 100–250ms. With FlashRank, closer to 15–30ms. With Cohere’s API, 150–400ms plus network. You need to decide whether your application’s latency budget can absorb this. For async or batch use cases, it almost always can. For real-time applications with sub-200ms SLAs, you may need to go FlashRank or skip reranking on short queries where retrieval precision is already high.

Putting It All Together: Implementation

1. Open Source Cross-Encoder

from sentence_transformers import CrossEncoder
import numpy as np

class CrossEncoderReranker:
    def __init__(self, model_name: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"):
        self.model = CrossEncoder(model_name, max_length=512)

    def rerank(
        self,
        query: str,
        candidates: list[dict],
        top_k: int = 5,
        text_field: str = "text"
    ) -> list[dict]:
        """
        Rerank candidates using cross-encoder.
        Each candidate dict must have a text_field key.
        Returns top_k candidates sorted by relevance score.
        """
        if not candidates:
            return []

        # Build query-document pairs for batch inference
        pairs = [(query, c[text_field]) for c in candidates]

        # Score all pairs, batch inference keeps GPU utilization high
        # On CPU with MiniLM-L-6, expect ~150-250ms for 50 pairs
        scores = self.model.predict(pairs, batch_size=32, show_progress_bar=False)

        # Attach scores and sort
        for candidate, score in zip(candidates, scores):
            candidate["rerank_score"] = float(score)

        reranked = sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)
        return reranked[:top_k]

2. Cohere Rerank Integration

import cohere
from typing import Optional

class CohereReranker:
    def __init__(self, api_key: str, model: str = "rerank-v3-5"):
        self.client = cohere.Client(api_key)
        self.model = model

    def rerank(
        self,
        query: str,
        candidates: list[dict],
        top_k: int = 5,
        text_field: str = "text",
        fallback_reranker: Optional[object] = None
    ) -> list[dict]:
        """
        Rerank using Cohere's managed API.
        Falls back to local reranker if API call fails.
        """
        if not candidates:
            return []

        # Cohere has a 10k token limit per document,truncate if needed
        # Silently truncating can hurt quality; log when it happens
        docs = []
        for c in candidates:
            text = c[text_field]
            if len(text) > 4000:  # conservative char limit
                text = text[:4000]
                print(f"Warning: truncated document to 4000 chars for Cohere rerank")
            docs.append(text)

        try:
            response = self.client.rerank(
                query=query,
                documents=docs,
                model=self.model,
                top_n=top_k
            )

            # Response contains results in ranked order with indices into original list
            reranked = []
            for result in response.results:
                candidate = candidates[result.index].copy()
                candidate["rerank_score"] = result.relevance_score
                reranked.append(candidate)

            return reranked

        except cohere.CohereAPIError as e:
            print(f"Cohere rerank failed: {e}. Falling back.")
            if fallback_reranker:
                return fallback_reranker.rerank(query, candidates, top_k, text_field)
            # If no fallback, return first top_k unranked,better than crashing
            return candidates[:top_k]

3. FlashRank Integration

from flashrank import Ranker, RerankRequest

class FlashRankReranker:
    def __init__(self, model_name: str = "ms-marco-MiniLM-L-12-v2"):
        # FlashRank downloads and caches models on first use
        # ms-marco-TinyBERT-L-2-v2: ~10ms for 50 docs,fastest, quality drops
        # ms-marco-MiniLM-L-12-v2: ~25ms for 50 docs,good quality/speed balance
        self.ranker = Ranker(model_name=model_name, cache_dir="/tmp/flashrank_cache")

    def rerank(
        self,
        query: str,
        candidates: list[dict],
        top_k: int = 5,
        text_field: str = "text"
    ) -> list[dict]:
        """
        Rerank using FlashRank,CPU-optimized, extremely fast.
        Typical latency: 15-30ms for N=50 on CPU (vs 150-250ms for full cross-encoder)
        Quality: ~85-90% of full cross-encoder quality on most domains
        """
        if not candidates:
            return []

        passages = [{"id": i, "text": c[text_field]} for i, c in enumerate(candidates)]
        request = RerankRequest(query=query, passages=passages)

        results = self.ranker.rerank(request)  # ~20ms on CPU for 50 docs

        # Map scores back to original candidates
        reranked = []
        for result in results[:top_k]:
            candidate = candidates[result["id"]].copy()
            candidate["rerank_score"] = result["score"]
            reranked.append(candidate)

        return reranked

4. Full Reranking Pipeline Class

from enum import Enum
from typing import Literal

class RerankerBackend(Enum):
    CROSS_ENCODER = "cross_encoder"
    COHERE = "cohere"
    FLASHRANK = "flashrank"

class RAGPipelineWithReranking:
    """
    Full RAG pipeline: hybrid_search → rerank → assemble_context → generate.
    Integrates with the hybrid search pipeline from Part 2.
    Supports hot-swappable reranker backends via config.
    """

    def __init__(
        self,
        hybrid_searcher,           # HybridSearcher from Part 2
        backend: RerankerBackend = RerankerBackend.FLASHRANK,
        retrieval_n: int = 50,     # Candidates to retrieve (wide net)
        rerank_k: int = 5,         # Candidates to keep for LLM (precise)
        cohere_api_key: str = None,
        cross_encoder_model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2",
        flashrank_model: str = "ms-marco-MiniLM-L-12-v2"
    ):
        self.searcher = hybrid_searcher
        self.retrieval_n = retrieval_n
        self.rerank_k = rerank_k

        # Initialize reranker based on backend config
        if backend == RerankerBackend.CROSS_ENCODER:
            self.reranker = CrossEncoderReranker(model_name=cross_encoder_model)
        elif backend == RerankerBackend.COHERE:
            fallback = FlashRankReranker(model_name=flashrank_model)
            self.reranker = CohereReranker(
                api_key=cohere_api_key,
                fallback_reranker=fallback
            )
        elif backend == RerankerBackend.FLASHRANK:
            self.reranker = FlashRankReranker(model_name=flashrank_model)
        else:
            raise ValueError(f"Unknown backend: {backend}")

    def retrieve_and_rerank(self, query: str) -> list[dict]:
        """Stage 1 + Stage 2: retrieve broadly, rank precisely."""
        # Stage 1: Hybrid search (dense + sparse + RRF), broad retrieval
        candidates = self.searcher.search(query, top_k=self.retrieval_n)

        # Stage 2: Rerank, precise relevance scoring
        # IMPORTANT: Always pass the ORIGINAL query, not a rewritten one
        reranked = self.reranker.rerank(
            query=query,
            candidates=candidates,
            top_k=self.rerank_k
        )

        return reranked

    def assemble_context(self, reranked_candidates: list[dict]) -> str:
        """Assemble LLM context from top-K reranked chunks."""
        parts = []
        for i, candidate in enumerate(reranked_candidates, 1):
            parts.append(f"[Source {i}]\n{candidate['text']}\n")
        return "\n".join(parts)

    def generate(self, query: str, llm_client) -> str:
        """Full pipeline: retrieve → rerank → assemble → generate."""
        reranked = self.retrieve_and_rerank(query)
        context = self.assemble_context(reranked)

        prompt = f"""Answer the following question based on the provided sources.

{context}

Question: {query}
Answer:"""

        return llm_client.complete(prompt)

Reranking Without a Reranker: LLM-Based Reranking

Worth knowing about, not always worth using.

RankGPT is an approach where you use an LLM itself to rerank candidates via a “listwise ranking” prompt. you give the model a list of passages and ask it to output them in order of relevance. The quality ceiling here is theoretically higher than any cross-encoder, because you’re using a massive generalist model with strong reasoning capabilities.

In practice, this is expensive and slow. Reranking 20 candidates with GPT-4 costs roughly $0.01–0.03 per query and adds 2–5 seconds of latency. For most production use cases, that’s not viable.

It makes sense when extremely high-stakes queries (medical, legal, financial) where wrong answers have severe consequences; very small candidate sets (10 or fewer); offline batch processing where latency doesn’t matter; or research/evaluation contexts where you’re establishing a quality ceiling to compare against.

It’s a useful mental model and occasionally the right tool. Don’t make it your default.

Measuring Reranking Impact

Don’t add reranking without measuring whether it’s actually helping. I’ve seen pipelines where reranking improved NDCG@5 by 18 percentage points and others where it added 200ms of latency for a 2-point gain. The delta depends on how noisy your Stage 1 retrieval is.

The metrics I use:

• NDCG@5 (Normalized Discounted Cumulative Gain): The gold standard for ranking quality. Measures whether the most relevant results appear at the top of your final K results. If your gold chunk jumps from position 4 to position 1, NDCG@5 captures that improvement.
• Precision@3: What fraction of your top-3 results are relevant? More interpretable than NDCG for quick sanity checks.
• MRR (Mean Reciprocal Rank): Average of 1/position of the first relevant result. Simple, intuitive, and very sensitive to whether you get the right answer at position 1 vs position 3.

Building an offline eval set: For every domain you care about, create 50–100 (query, expected_document_id) pairs. This doesn’t need to be exhaustive. even 50 queries gives you statistically meaningful signal on NDCG improvement. I do this by pulling real user queries from logs and manually annotating which page/chunk contains the correct answer.

What good looks like: In my pipeline, adding reranking on top of hybrid search (from Part 2) improved NDCG@5 by ~12–18 percentage points, depending on query type. Longer, more complex queries benefit the most. Short, keyword-style queries see less improvement because vector search is already fairly precise on them.

When Reranking Is NOT Worth It

I want to be honest about the trade-offs here, because I’ve seen people cargo-cult reranking into pipelines that didn’t need it. When to skip reranking -

• Small corpus, high retrieval precision: If you have 5,000 documents and your retrieval is already returning the right answer at position 1 most of the time, reranking adds latency with no quality benefit.
• Tight latency budgets and no room for even FlashRank: A 20ms addition matters if you’re building real-time autocomplete. Know your p95 budget.
• Already-excellent candidate quality: If you’ve done aggressive query expansion, have a highly tuned hybrid search, and your domain vocabulary is narrow and consistent, Stage 1 may already be doing most of the work.
• Reranker trained on a different domain than your corpus: A cross-encoder trained on MS MARCO passage ranking works great for web-document-style retrieval. It may perform poorly on medical literature, legal documents, or code. Measure on your data, not on published benchmarks.

The diminishing returns curve is real: reranking helps most when Stage 1 retrieval is noisy. If you’ve already done the hybrid search work from Part 2, you may find the incremental gain from reranking is smaller than it would have been on a pure vector search baseline. That’s fine, measure and decide accordingly.

Mistakes that can be avoided!

Real mistakes I made or have watched other engineers make:

Reranking too few candidates. If you retrieve top-5 and then rerank to top-3, you’re barely shuffling the deck. The gold answer needs to be in your retrieval set for reranking to help. Use N=30–50 at minimum.

Cross-encoder input length limits silently truncating documents. Most cross-encoders have a 512 token limit. If your page-level chunks are 800 tokens, they’re being silently truncated. The model sees the first 512 tokens and scores based on that. If the relevant passage is in the second half, your reranker never sees it. Fix: truncate documents before reranking and log when this happens.

Cohere API latency spiking under load. Cohere’s p50 latency is great. Their p99 is less predictable under bursty traffic. If you have latency SLAs, set up a local fallback and circuit break to it when Cohere response time exceeds threshold.

Reranker and retriever trained on different domains. Score miscalibration is subtle and hard to catch. The reranker might assign low scores to highly relevant documents because the writing style is different from its training data. Always evaluate on your specific domain.

Passing a rewritten query to the reranker instead of the original. If you do query rewriting or HyDE (Hypothetical Document Embeddings) in your retrieval stage, make sure you’re passing the original user query to the reranker, not the rewritten version. The reranker scores relevance to what the user actually asked, not your internal reformulation. This one bit me more than once.

FlashRank model quality varying significantly. The gap between FlashRank’s TinyBERT and MiniLM-L-12 options is substantial. Benchmark both on your eval set before deciding. TinyBERT is 2x faster but noticeably weaker on complex queries.

How to make decision here?

Use this table as a first quick looker -

Which Reranker Should I Use?

You can use this flow diagram to make descision -

So what matters in the end?

Reranking was the improvement that made my RAG pipeline feel genuinely reliable rather than probabilistically useful. Before reranking, I was playing a probability game, hoping the right chunk would happen to score high enough in vector similarity to land in position 1 or 2 of my context window. After reranking, I had a principled mechanism for ensuring the most relevant content led the context.

The two-stage paradigml, i.e., retrieve broadly, rank precisely is the right mental model for production RAG. Stage 1 gives you recall. Stage 2 gives you precision. Neither alone is sufficient.

If you’re starting today, add FlashRank first. It’s the lowest-friction path to measurable improvement. Run your eval set, measure NDCG@5 before and after. If you see a significant gain (which you probably will if your retrieval is at all noisy), you’ll know reranking is worth investing in further. From there, decide whether you need Cohere’s quality ceiling or whether FlashRank’s speed is more valuable to you.

The next post in this series covers RAG evaluation frameworks. how to build systematic evals, measure retrieval quality at scale, and know when your pipeline is actually ready for production. Because the honest truth is you can’t optimize what you can’t measure, and most teams skip evaluation just like they skip reranking. We’ll fix that next.

Check out Part 1 (PageIndex) and Part 2 (Hybrid Search + RRF) if you haven’t already. Thank youu.