BM25 vs ELSER vs Jina: A Retrieval Benchmark Comparison

Evaluating a search system is rarely as simple as picking a model and comparing accuracy scores. Real-world search involves tradeoffs across relevance, latency, infrastructure cost, and operational complexity and those tradeoffs become especially visible when you compare fundamentally different retrieval approaches.

In this post, we benchmark three popular retrieval paradigms: traditional lexical search (BM25), sparse semantic retrieval (ELSER), and dense embeddings (Jina) using datasets from the BEIR benchmark suite. Our goal is not just to identify which approach performs best on standard information-retrieval metrics, but to understand why they behave differently, and what those differences mean in practice.

BEIR provides a convenient, standardized way to measure retrieval quality across diverse domains and query styles. But while benchmark scores like Recall@100 or NDCG@10 offer useful signals, they rarely tell the whole story. Production search systems must also account for index size, query latency, explainability, and ease of deployment factors that don’t show up directly in leaderboard rankings.

We’ll walk through how each approach performs under identical conditions, then step back to discuss a more important question: how should you evaluate your own search system? Along the way, we’ll show how to think about benchmarking as a diagnostic tool rather than a final verdict using BEIR to establish baselines, reveal behavioral differences between retrieval methods, and guide architectural decisions, while emphasizing the importance of validating results on your own data and real user queries.

By the end, you should have a clearer framework for running meaningful benchmarks and for translating those results into practical choices for building and improving production search.

You can explore the reproducible benchmarking notebook on GitHub: [BM25, ELSER and JINA.ipynb].

Understanding the benchmarking metrics: nDCG@10

You can’t improve what you don’t measure. Meaningful system improvement begins with establishing reliable metrics, without them, performance remains opaque and optimization becomes speculation instead of engineering. Normalized Discounted Cumulative Gain (NDCG) is a ranking quality metric commonly used in search engines, recommender systems, and ML ranking models to measure how good your ordered results are, especially when relevance is graded (not just relevant / not relevant).

Imagine you run a search, and each result has a relevance score (for example: 0 = irrelevant, 3 = highly relevant).

Two ideas drive NDCG:

1. Gain — more relevant items give more “credit”.
2. Discount — items lower in the list count less, because users rarely scroll far.

So:

• A highly relevant result at rank 1 is worth a lot.
• The same result at rank 10 is worth much less.

In simple terms: NDCG rewards putting highly relevant items near the top of the list.

Computing NDCG for Retrieval Evaluation

Step 1: DCG (Discounted Cumulative Gain)

For a ranked list of results:

Where:

• relᵢ = relevance of item at position i
• i = rank (starting at 1)
• denominator applies the discount

Higher ranks → smaller contribution.

Step 2: IDCG (Ideal DCG)

Now imagine the perfect ranking (sorted by relevance).

Compute DCG for that ideal list:

This represents the best possible score.

Step 3: Normalize → NDCG

Finally:

This gives a value between:

• 0 → terrible ranking
• 1 → perfect ranking

Tiny Example

Suppose relevance scores for top-5 results:

Your ranking: [3, 2, 0, 1, 3]

Ideal ranking: [3, 3, 2, 1, 0]

You compute:

• DCG from your list ≈ 5.85
• IDCG from ideal list ≈ 6.32
• NDCG = DCG / IDCG ≈ 0.92

Meaning: your ranking is ~92% as good as perfect.

NDCG measures how well a system ranks results by rewarding highly relevant items placed near the top, normalized against the best possible ranking.

NDCG@10 measures ranking quality over only the top 10 results.

The Dataset: BEIR Benchmark

For our evaluation, we used the BEIR Benchmark, a widely adopted benchmark suite designed to reflect real-world information retrieval scenarios across diverse domains such as scientific literature, question answering, fact checking, and web search.

BEIR provides three critical components required for rigorous ranking evaluation:

• A document corpus
• A set of natural-language queries
• Human-labeled relevance judgments (qrels)

This makes it ideal for measuring ranking quality with metrics like NDCG@10, since it supports multiple relevant documents per query and graded relevance scores.

By relying on BEIR, we ensure that our comparison between BM25, ELSER, and Jina embeddings is grounded in standardized, reproducible datasets rather than synthetic examples. More importantly, BEIR’s diversity allows us to observe how each retrieval approach performs across different search intents and content types from keyword-heavy queries to semantically complex questions.

Using BEIR also aligns our methodology with common academic and industry practices, making our results easier to interpret and compare with existing retrieval benchmarks.

Benchmarking Approach

To understand how traditional keyword search compares with modern semantic retrieval, we benchmarked BM25, ELSER, and Jina embeddings using a consistent, industry-standard evaluation setup.

All experiments were run against the same corpus from the BEIR Benchmark — a widely adopted collection of real-world search tasks with human relevance judgments.

We measured performance using NDCG@10 (Normalized Discounted Cumulative Gain at rank 10), which evaluates how well each system ranks highly relevant documents within the top ten results. NDCG is especially valuable because it supports graded relevance and strongly rewards placing the most useful documents near the top of the list — closely mirroring real user behavior.

To ensure a fair comparison:

• The same BEIR dataset was indexed three times:
• once using BM25 for lexical retrieval,
• once using ELSER for sparse semantic search,
• and once using dense embeddings from Jina AI.
• Each query was executed independently against all three systems.
• For every query, we collected the top-100 results and computed NDCG@10 using standard IR evaluation tooling.

This methodology isolates ranking quality while keeping everything else constant: same documents, same queries, same relevance labels.

The table below summarizes the NDCG@10 scores for each model across the evaluated BEIR datasets.

These results reveal a consistent pattern. Across six BEIR datasets, ELSER and Jina outperformed BM25, with semantic models leading on all benchmarks and each excelling on different tasks. Jina achieved the highest NDCG@10 on three datasets, while ELSER led on the remaining three, demonstrating that both dense and sparse semantic retrieval produce better-ranked results than traditional keyword search alone. Both ELSER and Jina consistently outperformed BM25, achieving an average NDCG@10 of 0.460 compared to BM25’s 0.412 — an improvement of approximately 12%.

The results highlight clear differences in retrieval behavior:

• BM25 provides strong exact-term matching and remains a solid lexical baseline but pure keyword matching struggles once queries become semantic, exploratory, or loosely phrased.
• ELSER wins on nfcorpus, arguana, and scifact, where sparse semantic expansion combined with Elasticsearch’s scoring model appears to work especially well.
• Jina dominates on fiqa and trec-covid, suggesting dense embeddings are particularly strong for question-style and concept-heavy queries.

In practice, both sparse and dense semantic retrieval deliver measurable gains over BM25 on BEIR tasks while BM25 remains a strong lexical baseline but falls behind when relevance depends on meaning rather than exact term overlap.

Conclusion

Our evaluation shows that semantic retrieval significantly improves ranking quality over traditional keyword search. This confirms that models designed to capture meaning consistently surface more relevant documents in the top results than lexical matching alone.

At the same time, no single semantic approach dominates across all tasks. Jina performs best on question-oriented and concept-heavy datasets, while ELSER leads where sparse semantic expansion integrates effectively with traditional scoring. BM25 remains valuable for exact-term precision but struggles as queries become more exploratory or semantically complex.

In this era, semantic search is no longer optional: it is essential for modern retrieval. However, the strongest systems are built by blending lexical and semantic approaches, guided by benchmarking and validated on real user data.