Recall, MRR, NDCG — Stop Watching Your Loss Curve

Your training loss is going down. Recall@1 is silently regressing. Here’s how to actually measure retrieval quality — with worked examples you can compute by hand.

*Series: Fine-Tuning Embedding Models for Domain-Specific Retrieval Part 3 of 4*

Photo by Luke Chesser on Unsplash

There’s a particular kind of pain in machine learning where everything looks like it’s working — the loss drops, the gradients are healthy, the model seems to be converging — and then you evaluate the result and it’s worse than what you started with.

This happens all the time in embedding model training. The reasons are mostly technical (temperature too low and the model collapses to trivial solutions, hard negatives that are actually false negatives, overfitting on synthetic queries that don’t match real ones), but the root cause is the same: you were watching the wrong thing.

Training loss is a training diagnostic. Retrieval metrics are the objective. They’re loosely correlated at best. If you optimize for loss, you might get great loss curves and a terrible retriever.

This part teaches you the metrics that matter — Recall@k, MRR, NDCG — with worked examples you can compute on a napkin. Once you’ve got the intuition, you’ll never look at a loss curve the same way.

The Setup

Every retrieval metric starts from the same thing: a ranking. You have:

• A corpus of N chunks (passages your retriever can return).
• A test set of Q (query, correct_chunk) pairs.
• An embedding model that, given a query, produces a ranked list of all N chunks by relevance.

For each query q_i, the rank r_i is the position of the correct chunk in that ranked list (1-indexed: rank 1 = top of the list).

Here’s the code that gets you ranks:

import numpy as np
def ranks_from_scores(scores: np.ndarray, positive_idx: np.ndarray) -> np.ndarray:
    """
    scores       [Q, N]  - similarity of each query vs. each chunk
    positive_idx [Q]     - index of the correct chunk per query
    returns      [Q]     - 1-based rank of the positive for each query
    """
    pos_scores = scores[np.arange(scores.shape[0]), positive_idx]
    # rank = 1 + (number of docs scored strictly higher)
    ranks = (scores > pos_scores[:, None]).sum(axis=1) + 1
    return ranks

For the rest of this article, let’s use a running example with 10 test queries:

ranks = np.array([1, 3, 1, 7, 2, 1, 15, 4, 1, 2])
#                 ↑  ↑  ↑  ↑  ↑  ↑  ↑   ↑  ↑  ↑
#                Q0  Q1 Q2 Q3 Q4 Q5  Q6  Q7 Q8 Q9

Q0’s correct chunk was returned at rank 1 (perfect). Q6’s correct chunk came back at rank 15 (bad). Now we’ll compute every metric from this single array.

1. Recall@k — The Most Intuitive Metric

Question: What fraction of queries have the correct chunk in the top-k results?

Recall@k = (number of queries where rank ≤ k) / Q

def recall_at_k(ranks: np.ndarray, k: int) -> float:
    return float((ranks <= k).mean())

For our 10 queries:

ranks = [1, 3, 1, 7, 2, 1, 15, 4, 1, 2]

Recall@1:   ranks ≤ 1  → [T, F, T, F, F, T, F, F, T, F] → 4/10 = 0.40
Recall@5:   ranks ≤ 5  → [T, T, T, F, T, T, F, T, T, T] → 8/10 = 0.80
Recall@10:  ranks ≤ 10 → [T, T, T, T, T, T, F, T, T, T] → 9/10 = 0.90
Recall@50:  ranks ≤ 50 → [T, T, T, T, T, T, T, T, T, T] → 10/10 = 1.00

How to read these numbers:

• Recall@1 = 0.40: 40% of queries are answered perfectly — the correct chunk is the very first result.
• Recall@5 = 0.80: If you give the top-5 chunks to a reranker or an LLM, 80% of queries will have the answer somewhere in that context.
• Recall@10 = 0.90: Looking at the top-10, 90% of queries find their answer.
• Recall@50 = 1.00: Nothing is permanently missed — every answer is somewhere in the top 50.

Which k should you use?

It depends on your downstream system:

• Recall@1 — “Is my retriever good enough to use without a reranker?”
• Recall@5 or @10 — “I’m feeding retrieved passages into a reranker or RAG pipeline; how often is the answer present at all?”
• Recall@50 or @100 — “Coverage check — is anything being permanently missed?”

Match the metric to the system you’re building.

2. Mean Reciprocal Rank (MRR) — Rewarding High Ranks

Recall@k has one weakness: it treats a rank-1 result and a rank-k result identically (both are “in the top k”). But there’s a big difference between finding the answer first and finding it ninth.

MRR fixes this by averaging the reciprocal rank — 1 / rank — across queries.

MRR@k = (1/Q) · Σᵢ (1/rᵢ) if rᵢ ≤ k, else 0

def mrr_at_k(ranks: np.ndarray, k: int) -> float:
    in_topk = ranks <= k
    rr = np.where(in_topk, 1.0 / ranks, 0.0)
    return float(rr.mean())

For our example, computing MRR@10:

Query Rank Reciprocal Rank Q0 1 1/1 = 1.000 Q1 3 1/3 = 0.333 Q2 1 1/1 = 1.000 Q3 7 1/7 = 0.143 Q4 2 1/2 = 0.500 Q5 1 1/1 = 1.000 Q6 15 0 (rank > 10) Q7 4 1/4 = 0.250 Q8 1 1/1 = 1.000 Q9 2 1/2 = 0.500

MRR@10 = (1.000 + 0.333 + 1.000 + 0.143 + 0.500
        + 1.000 + 0 + 0.250 + 1.000 + 0.500) / 10
       = 5.726 / 10
       = 0.573

What MRR tells you

• Range: 0 to 1. Higher is better.
• Diminishing returns at lower ranks: moving a result from rank 5 to rank 4 gains you 1/4 - 1/5 = 0.05. Moving from rank 2 to rank 1 gains you 1/1 - 1/2 = 0.50 — 10× as much.
• Heavily weighted toward rank 1. If your downstream system uses only the top result, MRR is the right metric.

💡 MRR vs Mean Average Precision (MAP): When each query has exactly one correct answer (the common case), MRR equals MAP. They diverge only when a query can have multiple correct answers.

3. Normalized Discounted Cumulative Gain (NDCG)

NDCG is the standard leaderboard metric for ranked retrieval. It’s slightly more complex but captures something the others miss: how much each rank position is “worth.”

The intuition: getting the answer at rank 1 is worth a lot, rank 2 is worth less, rank 3 even less, and so on. The drop-off follows a logarithmic discount — large drops at first, gentler later.

Building up NDCG piece by piece

Gain — the relevance score at each position. In binary relevance (the common case), it’s 1 for the correct chunk, 0 otherwise.

Discount — a position-based weight. Position p has discount 1 / log₂(p+1).

Rank log₂(rank+1) Discount 1 log₂(2) = 1.000 1.000 2 log₂(3) = 1.585 0.631 3 log₂(4) = 2.000 0.500 4 log₂(5) = 2.322 0.431 5 log₂(6) = 2.585 0.387 10 log₂(11) = 3.459 0.289

DCG (Discounted Cumulative Gain): sum of gain × discount over the top-k results.

DCG@k = Σₚ₌₁ᵏ  gain(p) / log₂(p+1)

IDCG (Ideal DCG): the best possible DCG, i.e., what you’d get if the correct chunk were always at rank 1. For binary, single-positive retrieval: IDCG@k = 1.0.

NDCG@k = DCG@k / IDCG@k — a number in [0, 1] you can compare across queries.

def ndcg_at_k(ranks: np.ndarray, k: int) -> float:
    in_topk = ranks <= k
    dcg = np.where(in_topk, 1.0 / np.log2(ranks + 1.0), 0.0)
    return float(dcg.mean())

Worked example for NDCG@10

Query Rank log₂(rank+1) NDCG contribution Q0 1 1.000 1.000 Q1 3 2.000 0.500 Q2 1 1.000 1.000 Q3 7 3.000 0.333 Q4 2 1.585 0.631 Q5 1 1.000 1.000 Q6 15 (>k) 0.000 Q7 4 2.322 0.431 Q8 1 1.000 1.000 Q9 2 1.585 0.631

NDCG@10 = (1.000 + 0.500 + 1.000 + 0.333 + 0.631
        + 1.000 + 0.000 + 0.431 + 1.000 + 0.631) / 10
        = 6.526 / 10
        = 0.653

💡 Why NDCG is the leaderboard metric. NDCG smoothly combines “did we find the answer?” with “how high did it rank?” Unlike Recall@k, it doesn’t treat all top-k positions equally. Unlike MRR, it works with multi-graded relevance (relevant / partially relevant / not relevant) and can handle multiple positives per query. That generality is why papers and benchmarks report it.

All three are looking at the same ranks array; they just weight it differently. Pick one as your primary metric based on what your downstream system uses, and track all three to spot inconsistencies.

5. In-Batch MRR During Training: A Diagnostic, Not a Metric

While training, you’ll see a metric often called “in-batch MRR” or “in-batch accuracy.” It looks like an evaluation metric, but it’s not.

def in_batch_metrics(scores, n_negs):
    """scores: [B, B*(1+K)]  — query-vs-doc scores within the batch"""
    targets = torch.arange(scores.size(0)) * (1 + n_negs)
    ranks = (scores > scores.gather(1, targets[:, None])).sum(dim=1) + 1
    return {
        "acc": (ranks == 1).float().mean(),
        "mrr": (1.0 / ranks.float()).mean(),
    }

This ranks each query’s positive only against documents in the same training batch (typically a few dozen). The “MRR” you see going up during training is over this tiny in-batch pool, not the full corpus.

A typical, healthy training trajectory looks like:

Step 1:    loss=3.8   batch_acc=0.05   batch_mrr=0.20   (random init)
Step 50:   loss=2.1   batch_acc=0.35   batch_mrr=0.55   (learning basics)
Step 200:  loss=1.4   batch_acc=0.65   batch_mrr=0.75   (converging)
Step 500:  loss=1.1   batch_acc=0.78   batch_mrr=0.85   (good)

Use in-batch MRR as a sanity check. If it plateaus below ~0.5, something is wrong — your hard negatives might be false negatives, or your temperature is off.

But don’t checkpoint on it. A model that’s 0.95 in-batch MRR can still be worse than the base model at full-corpus retrieval. Run real evaluation, not the in-batch shortcut.

6. BEIR — Checking for Catastrophic Forgetting

After fine-tuning on your domain, your model is great at your domain. But did it forget how to do general retrieval?

BEIR (Benchmarking IR) is a collection of 18 retrieval benchmarks covering different domains: MSMARCO (web search), NQ (open-domain QA), HotpotQA (multi-hop), FiQA (financial), SCIDOCS (scientific papers), ArguAna (arguments), and more.

Running your fine-tuned model on a few BEIR slices is the standard way to check for catastrophic forgetting — has domain fine-tuning broken the model’s general retrieval ability?

from beir import util
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval import models

url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scidocs.zip"
data_path = util.download_and_unzip(url, "datasets")
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")
model = models.SentenceBERT("path/to/your/fine-tuned-model")
retriever = EvaluateRetrieval(model, score_function="dot")
results = retriever.retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
print(ndcg)  # {"NDCG@10": 0.xx, ...}

⚠️ The 2-point rule. If BEIR NDCG@10 drops by more than ~2 points absolute after fine-tuning, you’ve over-specialized. Solutions: lower the learning rate, train fewer epochs, or mix some general-domain pairs (e.g., MSMARCO) into your training data.

7. MTEB — The Comprehensive Embedding Benchmark

MTEB (Massive Text Embedding Benchmark) goes beyond retrieval. It covers 56 tasks across 8 categories: classification, clustering, pair classification, reranking, retrieval, semantic textual similarity (STS), summarization, and bitext mining.

The MTEB leaderboard is the standard way embedding models are compared publicly. For our purposes, you mostly care about the Retrieval subset.

from mteb import MTEB
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("path/to/your/fine-tuned-model")
evaluation = MTEB(tasks=["MSMARCO", "NQ", "SciFact"])
evaluation.run(model, output_folder="mteb_results")

Key MTEB retrieval tasks worth running after a domain fine-tune:

• SciFact — scientific claim retrieval from biomedical literature
• SCIDOCS — scientific paper retrieval
• NQ — open-domain QA
• FiQA — financial QA

These give you confidence that your fine-tuned model is still generally competent, not just specialized.

8. The Right Evaluation Protocol

When comparing a base model to your fine-tuned model, this is the protocol:

1. Load the full chunk corpus (every chunk, including ones from training documents).
2. Embed every chunk with the model being evaluated.
3. Load test queries (only from held-out documents — see Part 2 on splits).
4. Embed each query (don’t forget the instruction prefix!).
5. Compute the [Q, N] similarity matrix.
6. Get the rank of each query’s correct chunk.
7. Compute metrics.

corpus_embs = model.encode(all_chunk_texts)                  # [N, d]
query_embs  = model.encode([format_query(q) for q in qs])    # [Q, d]
scores = query_embs @ corpus_embs.T                          # [Q, N]
ranks = ranks_from_scores(scores, positive_indices)          # [Q]
metrics = {
    "recall@1":  recall_at_k(ranks, 1),
    "recall@5":  recall_at_k(ranks, 5),
    "recall@10": recall_at_k(ranks, 10),
    "ndcg@10":   ndcg_at_k(ranks, 10),
    "mrr@10":    mrr_at_k(ranks, 10),
}

The corpus is everything — your retriever competes against every document in the index at deployment time. The queries are held-out — the model has never seen the documents they correspond to.

9. A Realistic Numerical Example

Here’s what a realistic comparison between a base embedding model and a domain-fine-tuned version might look like:

Metric Base Model + Fine-Tune Δ Recall@1 0.478 0.584 +10.6% Recall@5 0.857 0.914 +5.7% NDCG@10 0.710 0.778 +6.8% MRR@10 0.632 0.718 +8.6%

A few things to notice:

• The Recall@1 gain (+10.6%) is the biggest. That’s exactly where domain fine-tuning helps most — pulling the right answer from rank 3 or rank 5 up to rank 1.
• The Recall@5 gain (+5.7%) is smaller. The base model was already pretty good at having the answer somewhere in the top 5; fine-tuning mostly improves precision at the very top.
• NDCG@10 captures the overall picture — both “did we find it?” and “did we rank it high?” — and shows a healthy +6.8% gain.

These numbers tell a coherent story: the fine-tuned model is sharper at the top of the ranking, which is exactly what you want for production RAG systems that feed the top-1 or top-3 results to an LLM.

10. Common Metric Traps

Trap 1: Evaluating on the training distribution. If your test queries are generated by the same LLM with the same prompt as your training queries, you’re measuring how well the model learned your synthetic distribution, not how well it serves real users. Always include some held-out real queries if you have them.

Trap 2: Recall@50 = 1.0 means nothing useful when N is small. If your corpus is 1,000 chunks and you’re reporting Recall@50, you’re looking at 5% of the corpus. Of course the answer is in there. Recall@50 is meaningful on a 1M-chunk corpus; on a 1K-chunk corpus, it’s vacuous.

Trap 3: Optimizing the wrong metric. If your downstream system retrieves the top-1 result and feeds it to an LLM, optimize Recall@1 or MRR. If it retrieves top-10 and reranks, optimize Recall@10. The metric must match the system.

Trap 4: Ignoring latency. NDCG@10 = 0.82 is worthless if your model takes 500ms to embed a query. Track embedding throughput (queries/second) alongside quality metrics. A 0.79 model that embeds in 50ms might be the better production choice.

Trap 5: Believing a single number. Statistical variance on small test sets is huge. A “+5%” gain on 100 test queries might disappear on a different sample. Compute bootstrap confidence intervals, or at minimum, evaluate on a few different held-out splits.

🔑 Key Takeaways

• Recall@k: intuitive, top-k hit rate. Use @1 for top-result systems; @5/@10 for reranker pipelines; @50/@100 for coverage.
• MRR: rewards getting the answer high in the ranking. Equivalent to MAP with single-positive queries.
• NDCG: the standard leaderboard metric. Logarithmically discounts lower ranks. Handles multi-graded relevance.
• Always evaluate on the full corpus, not just the test split. The model faces every document at deployment.
• Use BEIR/MTEB to check for catastrophic forgetting after domain fine-tuning. Drop more than 2 NDCG points and you’ve over-specialized.
• In-batch MRR is a training diagnostic, not an evaluation metric. Never checkpoint on it.

Next up — Part 4: The Full Pipeline. LoRA fine-tuning an 8B-parameter embedding model on your own documents, with hard-negative mining, synthetic query generation, and every gotcha that will silently wreck your training.