Evaluation Metrics for RAG: Measure Retrieval, Generation, and End-to-End Quality With Numbers That Actually Matter

Last week we compressed retrieved chunks so the model receives only the sentences that matter.

Photo by Deng Xiang on Unsplash

This week we answer a question that should have come earlier but makes more sense now: how do you actually measure whether your RAG system is working well?

If you cannot score it, you cannot improve it. If you score the wrong thing, you improve in the wrong direction.

Introduction

Over the past several weeks we have built a complete RAG pipeline, chunking, hybrid search, reranking, embedding selection, metadata filtering, query transformation, and contextual compression. Each piece was designed to improve quality. But how do you prove it? And how do you catch regressions before users do?

RAG evaluation is tricky because there are three layers that can fail independently. Retrieval can find the wrong chunks. Generation can ignore good chunks. The end-to-end answer can be factually correct but miss the point. You need metrics at each layer, and you need to know which one to look at when something goes wrong.

The Three Evaluation Layers

Layer 1: Retrieval Quality Did the pipeline find the right chunks?

This is the foundation. If retrieval fails, nothing downstream can compensate. Measure retrieval independently from generation.

Key metrics:

Precision at k. Of the top 3 or 5 retrieved chunks, how many are actually relevant to the query. This is your primary retrieval metric. If precision at 3 is low, your search, filtering, or reranking needs work.

Recall at k. Of all relevant chunks in your corpus, how many appeared in the top k results. Important when missing a single critical passage causes wrong answers. Harder to measure because it requires knowing all relevant chunks, not just whether the returned ones are good.

Mean Reciprocal Rank (MRR). How high does the first relevant chunk appear. If the best chunk consistently lands at position 4 instead of position 1, your reranking needs tuning.

NOT_RELEVANT rate from compression. How often contextual compression discards a retrieved chunk entirely. A high rate means retrieval is surfacing candidates that look similar but are not useful.

Layer 2: Generation Quality Given good context, did the model produce a good answer?

This isolates the model’s performance from retrieval. Feed the model known-good context and evaluate the output.

Key metrics:

Faithfulness. Does the answer only contain claims supported by the provided context. This catches hallucination the model adding facts that are not in the chunks. The most important generation metric for RAG.

Answer relevance. Does the answer actually address the question asked. A model can be faithful to the context but still miss the point if it latches onto the wrong chunk or the wrong part of a chunk.

Citation accuracy. When the model cites a source, does that source actually support the claim. Broken citations erode trust even when the answer is correct.

Completeness. Does the answer cover all parts of the question. Especially important for decomposed or multi-part queries.

Layer 3: End-to-End Quality Does the full system, retrieval plus generation, produce answers users trust and use?

This is what users experience. It reflects the combined performance of every component.

Key metrics:

Correctness. Is the final answer factually correct. This requires human judgment or a carefully designed reference set.

User satisfaction. Thumbs up rate, edit rate, and task completion rate. The most direct signal but the noisiest.

Rejection appropriateness. When the system says I don’t know, is that the right call. Refusing to answer when it should have is as bad as hallucinating.

Latency. Time to first token and time to complete response. Quality means nothing if users abandon before the answer arrives.

How to Build a Practical Eval Set

You do not need thousands of examples. You need 50 to 100 well-chosen cases that cover your key scenarios.

For retrieval evaluation, create query-chunk pairs. For each query, label which chunks in your corpus are relevant. Start with your top 10 intents and 5 queries per intent.

For generation evaluation, create query-context-answer triples. Provide known-good context and a reference answer. Score the model’s output against the reference.

For end-to-end evaluation, use real user queries with human-judged reference answers. Include edge cases, multi-part questions, and queries where the correct answer is I don’t have enough information.

Eval Set Template

eval_case:
  case_id: "refund_policy_003"
  layer: "end_to_end"
  query: "Can I get a refund after 30 days on a monthly plan?"
  relevant_chunk_ids: ["policy_refund_v4_chunk_7", "policy_refund_v4_chunk_8"]
  reference_answer: "No. Monthly plans have a 30-day refund window. After 30 days, no refund is available."
  required_citations: ["policy_refund_v4"]
  scoring:
    correctness: binary
    faithfulness: binary
    citation_present: binary
    completeness: binary

Automated vs Human Evaluation

Automated scoring works well for retrieval metrics (precision, recall, MRR), citation checks, schema validation, and faithfulness detection using an LLM-as-judge.

Human scoring is necessary for correctness on nuanced questions, answer relevance judgment, and evaluating whether the tone and detail level match user needs.

A practical split: automate everything you can, reserve human review for 20 to 30 cases per week focused on your highest-risk routes.

LLM-as-Judge Faithfulness Prompt

You are evaluating a RAG system's answer for faithfulness.

Given:
- User question
- Retrieved context (the only allowed source of facts)
- System answer

Score faithfulness as PASS or FAIL.
FAIL if the answer contains ANY claim not supported by the context.
PASS if every claim in the answer can be traced to the context.

Return JSON: {"score": "PASS" or "FAIL", "unsupported_claims": [list or empty]}

When to Run Evaluations

On every prompt or model change. Run the full golden set before merging.

Weekly on production samples. Score 50 sampled requests across layers.

After any retrieval pipeline change. Chunking, embedding model, reranking, filtering, or compression changes all require a retrieval eval pass.

Monthly full review. Run all three layers, compare to the previous month, and update the eval set with new failure cases.

Common Pitfalls and Quick Fixes

Measuring only end-to-end without layer isolation. Fix by evaluating retrieval and generation separately. When end-to-end quality drops, you need to know which layer caused it.

Using only automated metrics. Fix by adding weekly human review on 20 to 30 cases. Automated metrics miss nuance.

Stale eval sets. Fix by adding 5 to 10 new real-world failures to your eval set monthly. Remove duplicates and outdated cases quarterly.

Optimizing for one metric at the expense of others. Fix by tracking a small balanced dashboard: precision at 3, faithfulness, citation accuracy, and user satisfaction. Improve holistically.

Ignoring rejection quality. Fix by including cases where the correct answer is I don’t know and scoring false refusals as failures.

Metrics Dashboard (Start With These 6)

Retrieval precision at 3

Faithfulness pass rate

Citation accuracy

End-to-end correctness

User thumbs-up rate

Answer latency p95

Six numbers. Review weekly. Drill into whichever one drops.

Try It Now

Build a 50-case eval set covering your top 5 intents with retrieval and end-to-end cases.

Run retrieval precision at 3 on 20 queries and record your baseline.

Add the faithfulness LLM-as-judge prompt and score 20 production answers.

Set up a weekly review of 30 sampled requests with human scoring.

Create a simple dashboard with the 6 metrics above and review it every Monday.

Conclusion

You cannot improve what you do not measure, and you cannot debug what you do not isolate. Evaluate retrieval, generation, and end-to-end quality as separate layers. Automate what you can, review what you must, and keep your eval set alive with real failures. When every component has a score, every improvement has proof.