Fixing the Last Mile of RAG with Cohere Rerank v4.0 on Microsoft Foundry

Your RAG pipeline retrieves ten chunks. Seven are relevant. Three are noise. The language model reads all ten, weighs them equally, and occasionally hallucinates from the noisy three instead of answering from the solid seven.

This is the last-mile retrieval problem. Vector similarity gets you in the right neighborhood, but it does not guarantee the right documents come first. The solution is reranking: a second-pass model that reads both the query and each retrieved chunk and scores how well each chunk actually answers the question; not just how similar the embeddings are.

Cohere Rerank v4.0 is available in the Foundry model catalog as a serverless endpoint, uses cross-encoding AI for semantic relevance scoring, and drops into your existing RAG pipeline with minimal code changes. In this article, we will build a RAG pipeline with reranking and measure the quality improvement.

Why Reranking Works: Embedding Similarity vs Semantic Relevance

Embedding similarity measures cosine distance between vector representations. It is fast and scales well, but it has a fundamental limitation: the embedding model was trained to capture general semantic meaning, not to judge whether a specific passage answers a specific question.

Reranking uses a cross-encoder; a model that reads the query and each document together (not separately) and scores relevance on a 0–1 scale. Cross-encoders are slower and cannot scale to thousands of documents, but they are far more accurate at judging relevance for a specific query. The pattern is:

• First pass: Retrieve top-50 candidates using vector similarity.
• Second pass: Rerank the top-50 with Cohere Rerank, select top-5.
• Generation: Pass the reranked top-5 to the LLM (higher quality context, lower token cost)

Step 1: Deploy Cohere Rerank v4.0 in Foundry

In the Foundry portal, navigate to the Model Catalog and search for ‘Cohere Rerank’. Deploy Rerank v4.0 Fast or Pro to a serverless endpoint. Copy the endpoint URL and key; you will use them directly in the Cohere client.

pip install cohere azure-search-documents azure-ai-projects azure-identity

Step 2: Build the Retrieval Layer with Azure AI Search

Set up a standard hybrid search retrieval function (vector + keyword).

This is your first pass, retrieve broadly:

from azure.search.documents import SearchClient
Set up a standard hybrid search retrieval function (vector + keyword). This is your first pass — retrieve broadly:
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
from azure.core.credentials import AzureKeyCredential
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(
    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential()
)
oai = project.get_openai_client(api_version='2024-10-21')

search_client = SearchClient(
    endpoint=os.environ["SEARCH_ENDPOINT"],
    index_name="docs-index",
    credential=AzureKeyCredential(os.environ["SEARCH_KEY"])
)

def retrieve_candidates(query: str, top_k: int = 25):
    embedding = oai.embeddings.create(
        input=query, model="text-embedding-3-large"
    ).data[0].embedding

    results = search_client.search(
        search_text=query,
        vector_queries=[VectorizedQuery(
            vector=embedding, k_nearest_neighbors=top_k,
            fields="content_vector"
        )],
        top=top_k,
        select=["id", "content", "source"]
    )
    return [{'id': r['id'], 'text': r['content'], 'source': r.get('source', '')} for r in results]

Step 3: Rerank with Cohere Rerank v4.0

Pass the retrieved candidates through Cohere Rerank.

The model reads query + document pairs and returns relevance scores:

import cohere

co = cohere.Client(
    api_key=os.environ["COHERE_API_KEY"],
    base_url=os.environ["COHERE_RERANK_ENDPOINT"]  # Your Foundry serverless endpoint
)

def rerank(query: str, candidates: list, top_n: int = 5):
    response = co.rerank(
        model="rerank-v4.0",
        query=query,
        documents=[c["text"] for c in candidates],
        top_n=top_n,
        return_documents=True
    )
    reranked = []
    for r in response.results:
        original = candidates[r.index]
        reranked.append({
            'text': original['text'],
            'source': original['source'],
            'relevance_score': r.relevance_score,
            'original_rank': r.index
        })
    return reranked

Step 4: Wire Retrieval + Reranking + Generation

def answer_with_reranking(query: str):
    # First pass: retrieve 25 candidates
    candidates = retrieve_candidates(query, top_k=25)

    # Second pass: rerank, keep top 5
    top_chunks = rerank(query, candidates, top_n=5)

    # Log reranking effect for observability
    for i, chunk in enumerate(top_chunks):
        print(f'Rank {i+1} (was #{chunk["original_rank"]+1}) | Score: {chunk["relevance_score"]:.3f}')

    # Generate grounded answer
    context = "\n\n---\n\n".join(c["text"] for c in top_chunks)
    response = oai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"Answer only using the context below.\n\n{context}"},
            {"role": "user", "content": query}
        ]
    )
    return response.choices[0].message.content

Measuring the Quality Improvement

The real impact of reranking shows up in your Groundedness evaluation scores. After adding Cohere Rerank to a typical RAG pipeline, expect:

• Groundedness score: +0.3 to +0.6 points on a 5-point scale (based on typical enterprise RAG benchmarks)
• Token cost reduction: 50–70% fewer tokens passed to the LLM (5 chunks instead of 25)
• Answer precision: Fewer hallucinations from off-topic chunks contaminating context

The latency cost is typically 150–300ms for the reranking call. For most production applications, this trade-off is very favorable — better quality answers with lower LLM cost, at the price of a sub-second extra step.

Run a Foundry evaluation before and after adding reranking. Use the same evaluation dataset and track Groundedness, Relevance, and Coherence. The numbers will tell you exactly how much improvement you get for your specific domain; do not rely on generic benchmarks for your production decision.

Thanks for reading my article and I hope you can take something away.

💯 Don’t forget to follow me on medium for more

💯 Don’t forget to follow me on **LinkedIn **for more

💯 leave some feedback

FURTHER Reading​

• https://docs.cohere.com/docs/rerank

#MVP Communities — Microsoft

That’s all !