A Rigorous Benchmark with Multidimensional Evaluation for Deep Research Agents: From Answers to Reports

Establishing a new gold standard for evaluating the next generation of AI research agents

Summary

This paper presents Rigorous Bench, a meticulously constructed benchmark designed to evaluate Deep Research Agents (DRAs) — large language model (LLM)-based systems capable of autonomous information retrieval, multi-stage reasoning, and structured long-form report generation. Traditional benchmarks fall short in assessing such systems, focusing mainly on short, single-turn answers. Rigorous Bench fills this gap by introducing a multidimensional evaluation framework that quantifies semantic quality, topical focus, and retrieval trustworthiness, providing a transparent and reproducible way to evaluate DRAs’ performance.

Comprising 214 expert-curated queries across 10 domains (from technology to history and health), each entry includes rubrics, authoritative sources, and thematic keywords, ensuring nuanced and comprehensive assessment. Experiments across 13 advanced models — including Qwen Deep Research, Sonar, GPT-5, and Gemini-2.5-Pro — demonstrate that DRAs outperform traditional tool-augmented models, though key challenges in reasoning stability and coherence remain.

💡 Intuition

As AI evolves from static LLMs to interactive research agents, evaluation must evolve too. Existing metrics (like BLEU or ROUGE) capture surface similarity but fail to measure reasoning depth or citation reliability. Rigorous Bench reimagines benchmarking as a multi-dimensional evaluation problem, assessing how well an agent retrieves, reasons, and synthesizes — not just how well it predicts text.

🎯 Problem

Current benchmarks are too shallow for the deep reasoning and retrieval capabilities of DRAs. They primarily:

• Focus on short-form QA rather than long, structured reports.
• Lack criteria for citation accuracy, source credibility, and semantic focus.
• Depend on LLM-based similarity scoring, which introduces subjectivity and instability.

Without a rigorous framework, it is difficult to measure how well modern agents perform complex, multi-step research tasks involving synthesis from diverse information sources.

🛠️ Solution

The authors propose Rigorous Bench and its accompanying multidimensional evaluation framework, both designed to reflect real-world research processes.

1. Rigorous Bench Dataset

• 214 expert-authored tasks across ten domains (e.g., Law & Politics, Environment, Health, Business).
• Each query includes:
• Query-Specific Rubrics (QSRs) — expert-designed factual and logical criteria.
• General-Report Rubrics (GRRs) — standardized measures of structure, clarity, originality, and citation quality.
• Trustworthy Source Links (TSLs) — verified authoritative references.
• Focus-Anchor and Focus-Deviation Keywords (FAKs/FDKs) — used to measure topical consistency.

2. Evaluation Framework

A quantitative and transparent scoring system combining three core metrics:

• Semantic Quality (QSR + GRR): Evaluates factual accuracy and report structure using weighted normalization.
• Topical Focus (Semantic Drift): Penalizes missing key topics (FAKs) or introducing irrelevant content (FDKs).
• Retrieval Trustworthiness (TBoost): Rewards reports that cite verifiable, authoritative sources.

These are combined in the Integrated Score formula:

IntegratedScore=Quality×(1−SemanticDrift)×TrustworthyBoost×100\text{IntegratedScore} = \text{Quality} \times (1 — \text{SemanticDrift}) \times \text{TrustworthyBoost} \times 100IntegratedScore=Quality×(1−SemanticDrift)×TrustworthyBoost×100

3. Empirical Results

• Evaluated 13 leading models, including DRAs and web-search-augmented models.
• Qwen-Deep-Research ranked first in overall integrated score, followed by Sonar and o3-Deep-Research.
• Kimi-K2 (1T MoE agent) achieved the highest raw quality score.
• GPT-5 achieved the highest citation reliability, reflecting strong external grounding.
• DRAs used far more tokens but demonstrated superior report structure and coherence.

🚧 Limitations and Future Opportunities

Despite its rigor, the paper acknowledges several open challenges:

• Efficiency–Quality Trade-off: DRAs often consume excessive tokens for high-quality reasoning; adaptive search depth control is needed.
• Decomposition–Coherence Trade-off: Breaking queries into sub-tasks can lead to fragmented reasoning or off-topic retrieval.
• Evaluation Scope: While Rigorous Bench is comprehensive for text-based tasks, future benchmarks should extend to multimodal, collaborative, and interactive scenarios.

Future work could integrate:

• Dynamic reward shaping for more nuanced evaluation.
• Cross-modal inputs (vision, tables, code).
• Multi-agent collaboration to simulate real-world research teams.

🧭 Takeaway

This paper sets a new standard for evaluating AI research agents. By redefining benchmarking from “answer checking” to multi-dimensional report assessment, Rigorous Bench establishes the first rigorous, scalable, and interpretable framework for measuring reasoning depth, factual reliability, and report quality in DRAs.