Modern AI Benchmarks: What Practitioners Actually Need to Know

A practitioner’s guide to LLM evaluation in 2026 — what AIME, SWE-bench, GDPval, ARC-AGI, and other frontier benchmarks really measure, where they’ve saturated, and how to read the scores.

tl;dr — Benchmarks are how the industry argues about model quality — and most of that argument is conducted with stale, saturated, or undisclosed-scaffold numbers. This is a practitioner’s map of the benchmarks that still separate frontier models in 2026, grouped by capability, each with the caveat that bites if you quote it naively. The throughline: pin the harness, distrust the headline, and re-check the date, because standings here have a half-life measured in months.

Complimentary Reading Link

Nobody reads benchmark fine print until a number they cited falls apart in a review. This piece is the fine print — key current benchmarks, what each actually measures, where it’s already saturated, and the specific way each one misleads — so you can quote scores that survive scrutiny instead of ones that just sound impressive. This is a working reference for evaluating, citing, and reasoning about the current benchmark landscape. For each benchmark the questions that matter to a practitioner are roughly constant: what it measures, the gap it was built to close, where it sits on the saturation curve right now, and the one caveat that will burn you if you quote the headline number without reading the fine print. Entries are grouped by capability domain rather than alphabetically, because that is how you actually choose what to run.

A few cross-cutting facts before the list:

• A headline score is a point estimate under one scaffold. Agent scaffolding, tool budgets, token budgets, and pass@k vs. pass@1 routinely move scores by 10–20 points. Always pin the harness.
• “Contamination-resistant” is a design claim, not a guarantee. The strongest forms are held-out private sets and legal/licensing barriers, not string-match decontamination.

• Saturation is the default trajectory. Several benchmarks below went from “best model under 20%” to “best model over 85%” inside a year. Discriminating power decays; re-check standings rather than trusting a six-month-old leaderboard.
• Funding and access create conflicts of interest. Note who built the benchmark and who has privileged access to the held-out data.

Mathematical reasoning

AIME 2025 — The 30 problems from the 2025 American Invitational Mathematics Examination, integer answers 000–999, auto-gradable with zero ambiguity. Its value is cleanliness and freedom from grading subjectivity; its weakness is tiny n. With only 30 items, a single problem is ~3.3 percentage points, so confidence intervals are wide and run-to-run variance is large — report pass@k and multiple seeds or the number is close to noise. Frontier models now score in the 90s, so AIME is largely saturated and functions more as a sanity check than a discriminator. The annual refresh (a new exam each year) is its main defense against contamination, but the format is so well-represented in training data that the style is thoroughly learned even if specific 2025 items are held out.

HMMT 2025 — Same idea as AIME but sourced from the Harvard–MIT Mathematics Tournament, generally regarded as harder and less templated than AIME. Practitioners pair it with AIME precisely because it is slightly off the most-trained distribution, so it retains a bit more discriminating power at the top. Same small-n caveat applies.

FrontierMath (Tiers 1–3) — Epoch AI’s set of 300 unpublished, expert-written problems spanning advanced undergraduate through early-research mathematics; specialists need hours to days per problem. Answers are submitted as Python objects and graded all-or-nothing, no partial credit. This is the benchmark that exposed how shallow AIME-style math is — it was built specifically because competition math saturated. Two caveats are essential. First, OpenAI funded it and has exclusive access to a large subset of problems and solutions, while Epoch holds out the remainder; the reassuring signal is that top models perform comparably on the held-out set, but the arrangement is a structural conflict the field still debates. Second — and this is recent — as of May 2026 Epoch flagged that an AI-assisted review found likely fatal errors in roughly a third of the problems, with corrected scores pending. Treat current FrontierMath numbers as provisional until the corrected dataset lands.

FrontierMath (Tier 4) — The extreme tier: 50 problems crafted as short-term research projects by professors and postdocs, some expected to resist AI for years. This is the closest thing to a “can a model do novel research math” probe. For most of its life only a handful of these were ever solved by any model; the news event was GPT-5-class and then GPT-5.4-class models cracking previously-unsolved items in late 2025 / early 2026, including a held-out problem the model had no access to. If you are tracking genuine frontier reasoning rather than benchmark gaming, Tier 4 is the one to watch — but the same error-review caveat and tiny n apply with full force.

Abstract and fluid reasoning

ARC-AGI-1 (Verified) — Visual grid-transformation puzzles designed to be easy for humans and hard for machines, the original test of fluid intelligence / few-shot abstraction rather than memorized knowledge. Its enduring contribution is the framing: it rewards on-the-fly rule induction, not recall. It was considered a hard wall until reasoning models with heavy test-time compute broke through, at which point the cost-per-task became part of the story — a model can “solve” ARC by spending enormous inference compute, which is itself a finding. Read ARC scores alongside their compute budget or they mislead.

ARC-AGI-2 — The harder successor, explicitly built to re-open the gap after ARC-AGI-1 started yielding to brute-force test-time search. It stresses efficiency, not just capability: the design intent is that tasks remain easy for humans while defeating the search-heavy strategies that beat v1. If you want a current measure of abstraction-per-dollar rather than abstraction-at-any-cost, v2 is the better instrument. Expect it, too, to erode over time.

Science and expert knowledge

GPQA Diamond — 448 multiple-choice questions in biology, physics, and chemistry, written and filtered to be “Google-proof”: PhD-level domain experts reach roughly 65% even with web access, while skilled non-experts do far worse. It became the default graduate-science discriminator. The caveats: it is multiple choice (so it measures recognition under hard distractors, not generation), it has been heavily reported and is approaching saturation at the top, and “Diamond” is the hardest curated subset — make sure a quoted score is Diamond, not full GPQA.

Humanity’s Last Exam (HLE) — A multimodal, closed-ended, verifiable exam across dozens of academic subjects, explicitly built as the benchmark to survive after MMLU and GPQA saturated; the name is the thesis. It is genuinely hard and broad, which makes it a strong current top-end discriminator. Watch two things: scores climbed faster than the “last exam” framing implied, and the verifiable-answer constraint means it under-weights open-ended expertise. It is a ceiling probe, not a measure of practical usefulness.

MMMLU — A multilingual rendering of MMLU with professionally translated questions across 14 languages. Useful for checking that knowledge transfers across languages rather than collapsing outside English, but it inherits MMLU’s saturation and its known issues (some mislabeled or ambiguous items), and professional translation of an English-centric test does not capture culturally-specific knowledge — that is what Global PIQA is for.

Factuality and hallucination

FACTS Benchmark Suite — Google DeepMind/Kaggle’s aggregate factuality suite, 3,513 examples across four sub-leaderboards: Parametric (closed-book internal knowledge), Search (must use a search tool), Multimodal (image-grounded factuality), and Grounding v2 (long-form answers grounded in provided documents, with improved judge models). Private splits are held out for leaderboard integrity. The practitioner takeaways are unusually actionable: there is a roughly 70% factuality ceiling across leading models, a large gap between “knowing” (Parametric) and “finding” (Search) that validates RAG-over-parametric-memory as the enterprise default, and multimodal factuality is the weakest dimension industry-wide, with all models under ~70%. The scoring leans on automated judge models, so judge quality is a dependency.

SimpleQA Verified — A cleaned, reliability-focused 1,000-prompt short-form factuality set (a verified successor to the original SimpleQA), measuring whether a model gives a correct short factual answer or hallucinates. It is a clean hallucination-rate proxy. Because it is short-form and verifiable, it tells you about precision on atomic facts, not about long-form faithfulness — pair it with a grounding benchmark. Generational jumps here are large (e.g., Gemini 3 Pro reported around 72% vs. roughly 55% for the prior generation), so it still discriminates.

Coding and software engineering

Codeforces — Competitive-programming problems mapped to the platform’s ELO rating system, giving an intuitive “this model is around a Candidate Master / Grandmaster” readout. The ELO framing is its strength and its trap: ratings depend heavily on the problem set, time/compute budget, and whether the model can execute and self-test, so cross-report ELO comparisons are often apples-to-oranges. Older Codeforces problems are also contaminated; recency of the problem window matters.

LiveCodeBench Pro — 584 contest problems drawn exclusively from premier sources (Codeforces, ICPC, IOI, select university contests), tagged and error-audited by Olympiad medalists, scored with Pass@1 and a Bayesian-ELO style metric. It exists because the original LiveCodeBench had weak test cases, unbalanced difficulty, and contamination exposure; Pro adds expert curation and strict live collection. Its most useful output is diagnostic, not a single number: it separates knowledge/implementation-heavy problems (where models do well) from observation/insight-heavy problems (where they collapse), and historically showed models far below top human contestants on the hardest tier. Use it to understand where a model fails, not just whether it ranks.

SWE-bench Verified — The 500-problem, human-validated subset of SWE-bench: real GitHub issues with hidden tests, where the agent must produce a patch that passes. It is the de facto industry standard for agentic coding and the number every lab reports. Two things to internalize: it is now high (top models well over 70%), so it no longer separates the frontier; and “Verified” specifically means the problems were checked to be solvable and well-specified, which removes the noisiest items but also makes it easier than real-world maintenance. It is necessary but no longer sufficient.

SWE-bench Pro (public) — Scale AI’s harder, contamination-resistant successor: 1,865 total instances across 41 repositories, split into a public set (~731 instances from strong-copyleft/GPL repos chosen so the license is a legal deterrent against training inclusion), a held-out set, and a first-of-its-kind commercial set from 18 private startup codebases. Tasks require multi-file patches and can take a professional hours to days. The headline finding is the cliff: the same frontier models that exceed 70% on SWE-bench Verified drop to roughly 23% on the Pro public set and below 20% on the commercial set. This is the benchmark to cite when you need to puncture “SWE-bench is basically solved” optimism. Contamination resistance here comes from licensing and access barriers, the strongest available form.

Terminal-bench 2.0 — A Stanford / Laude Institute (with Snorkel) benchmark of 89 hand-curated, hard terminal tasks — compiling code, training classifiers, configuring servers, debugging — run in reproducible containers via the Harbor framework, with process-level milestone rewards. It exists because terminal/CLI is the substrate of real coding agents (Claude Code, Codex, etc.) and v1 saturated past 50%. Frontier models score roughly 65–73% bare, with scaffolding pushing top combinations into the low 80s, leaving a meaningful gap. Treat scores as relative model/architecture signal, not a production guarantee — the creators themselves flag a large lab-to-production gap.

Agentic tool use and long-horizon coherence

MCP Atlas — Scale AI’s tool-use benchmark built on real Model Context Protocol servers: 36 production servers, 220 tools, 1,000 human-written multi-step tasks. The design choices are what make it relevant to anyone deploying MCP: prompts describe goals in plain language and never name the tool or server, so the agent must discover the right tool among plausible distractors, parameterize it correctly, and orchestrate 3–6 calls across servers; scoring is a claims-based rubric grounded in actual tool outputs rather than LLM-as-judge vibes. The diagnostic insight: when tools are used correctly, synthesis is rarely the failure — discovery, parameterization, and reliable multi-step execution are. Best models land in the low-to-mid 60s (e.g., Claude Opus 4.5 around 62%), so substantial headroom remains.

Toolathlon (the Tool Decathlon) — An HKUST-led ICLR 2026 benchmark spanning 32 applications and 604 real tools (mostly real MCP servers, including Google Calendar, Notion, WooCommerce, Kubernetes, BigQuery), with 108 verifiable tasks requiring ~20 turns on average against realistic seeded environment state (real Canvas courses, real financial spreadsheets). It pushes harder on realism and long-horizon execution than most tool benchmarks. The headline is sobering: the best model (Claude 4.5 Sonnet at release) reached only ~38.6% success, top open-weights ~20% — a clear “agents are not yet reliable for real multi-app workflows” signal. Strict execution-based verification is its credibility anchor.

t²-Bench (τ²-Bench) — Sierra’s dual-control conversational benchmark, modeled as a Dec-POMDP where both the agent and a tool-using simulated user act on a shared, changing world state (the flagship Telecom domain walks a user through technical troubleshooting). It exists because prior benchmarks were single-control — the user was a passive information source — which is unlike real support, where the user must take actions too. This makes it a better proxy for agent–human coordination and communication, not just tool execution. The dependency to watch is the user simulator: Sierra invests heavily in its fidelity, but any dual-control result is only as reliable as the simulated counterpart, so treat it as testing collaboration quality under a controlled simulator.

Vending-Bench 2 — Andon Labs’ long-horizon coherence test: run a simulated vending-machine business for 365 simulated days from a $500 starting balance, paying a daily fee, sourcing stock, negotiating with adversarial suppliers, and setting prices; scored purely on final bank balance, averaged over multiple runs (each run burns 60–100M+ tokens). It descends from Anthropic’s real-world Project Vend. What it measures that nothing else does: whether a model stays sane and coherent over months of decisions rather than entering “doom loops” (the original surfaced an agent trying to contact the FBI over a $2 daily fee). Two practitioner notes: variance is high and leaderboard standings shift as models are added (different sources have shown different frontier models on top across versions), and no model is remotely close to a skilled human operator — the gap is the point. The Arena variant pits agents against each other at one location, adding competitive dynamics.

Multimodal, document, GUI, and video understanding

MMMU-Pro — A hardened multimodal college-level benchmark that filters out questions answerable from text alone and augments the answer options, forcing genuine image-text integration rather than language-prior shortcutting. It exists because the original MMMU could be partly gamed without really reading the image. Use Pro, not MMMU, when you need to know whether vision is actually contributing.

CharXiv Reasoning — Chart-understanding questions derived from real scientific-paper figures, split into descriptive questions and harder reasoning questions that require synthesizing multiple visual elements. It targets a specific, common failure: models that caption a chart fluently but cannot reason over it. The reasoning split is where the gap between human and model performance stays wide, so quote that split rather than the easier descriptive one.

Video-MMMU — Expert-level video understanding across six disciplines, framed around knowledge acquisition from video (perception → comprehension → adaptation), not just frame description. It probes whether a model can learn a procedure or concept from a lecture-style video and apply it. The practical caveats are about harness: frame-sampling rate, audio inclusion, and context length materially change scores, so cross-model comparisons require identical video-processing settings.

OmniDocBench 1.5 — A document-parsing benchmark (originally ~981 PDF pages, expanded in later versions) covering 9 document types, 4 layout types, and 3 languages, with rich block- and span-level annotations; it scores OCR via normalized edit distance, tables via TEDS, and formulas via Character Detection Matching. It became a default for document-AI/OCR evaluation. The current reality: v1.5 is effectively saturated — multiple specialist models (and frontier VLMs) now exceed ~90–94% overall, so further gains are edge-case polishing rather than real capability jumps, and the maintainers have already iterated to later versions. If you are choosing a parsing stack, look at the per-type breakdown (newspapers, dense formulas, handwriting still hurt) rather than the top-line.

ScreenSpot-Pro — GUI grounding in high-resolution professional software: 1,581 expert-annotated screenshot-plus-instruction pairs across 23 applications (VSCode, Photoshop, Premiere, CAD tools, etc.), 5 industries, 3 operating systems. It isolates one hard sub-skill of computer-use agents — pixel-precise localization of a small target in a cluttered, high-res screen — separate from planning. It is a textbook saturation case worth knowing: at release (early 2025) the best model managed only ~18.9%, but by 2026 leaderboards show top models above 85%. If you cite the famous “18.9%” figure, date it, because it is now badly stale.

Multilingual and cultural reasoning

Global PIQA — A participatory physical-commonsense benchmark hand-built by 335 researchers across 65 countries, covering 116 language varieties (5 continents, 14 language families, 23 writing systems), with over half the non-parallel examples referencing local foods, customs, and traditions. Each item is a prompt with one correct and one incorrect everyday-reasoning solution. It exists because almost all multilingual benchmarks are translated-from-English and therefore culturally hollow; Global PIQA is built by native speakers instead. The finding that matters for deployment: top models do well in aggregate (around 91%) but show up to a ~37% accuracy gap in lower-resource languages despite a 50% random-chance floor — everyday knowledge, not just hard reasoning, is where multilingual coverage breaks.

Long-context

MRCR v2 (8-needle) (your list labeled this “MRCH” — the correct name is MRCR, Multi-Round Coreference Resolution) — A synthetic long-context stress test from Google DeepMind’s Michelangelo line, expanded by OpenAI. The model sees a long multi-turn conversation in which the user repeatedly requests writing on similar topics; it must then reproduce a specific (e.g., the i-th) earlier assistant response. The 8-needle variant forces tracking of eight near-identical items simultaneously, which is far harder than classic single-needle “find the fact in the haystack.” Its virtues are that it is synthetic and unleaked (contamination-resistant by construction) and reported across context lengths (128K up to multi-million tokens), producing a performance-vs-context curve rather than a single number. Read the curve: a model strong at 128K can degrade sharply at 1M. Estimated random-baseline noise on the metric is around 1%, so the scores are clean.

Real-world economic value

GDPval — OpenAI’s benchmark of real, economically valuable knowledge-work tasks: 1,320 tasks (220 in an open “gold” subset) spanning 44 occupations across the top 9 U.S. GDP sectors, each task built from the actual work product of professionals averaging 14 years of experience, with deliverables like documents, slides, spreadsheets, and diagrams. The primary metric is head-to-head human-expert preference (with an experimental automated grader for the gold subset). It exists to answer “can the model do the job,” not “can it pass a test,” and the reported trajectory is that frontier models are approaching expert-level deliverable quality. The caveats are explicit in the design: one-shot tasks (no iterative back-and-forth), U.S.-occupation-centric, predominantly digital knowledge work, and preference-judging that can reward polish over correctness. It is the best current bridge between benchmark scores and economic claims — but it measures isolated deliverables, not sustained on-the-job performance (which is closer to what Vending-Bench probes). For mapping AI capability to specific roles, GDPval’s occupation list is the most directly useful artifact in this entire set.

How to use this list

• Top-end discrimination today: FrontierMath Tier 4, HLE, SWE-bench Pro (commercial), Toolathlon, MCP Atlas, Vending-Bench 2. These still have real headroom.
• Effectively saturated — use as regression checks, not differentiators: AIME, GPQA Diamond, MMMU-Pro (top end), OmniDocBench 1.5, ScreenSpot-Pro (top end), SWE-bench Verified.
• Best deployment-shaped signals: FACTS (RAG vs. parametric), MCP Atlas / Toolathlon (real tool use), GDPval (occupational value), Vending-Bench 2 (long-horizon coherence), MRCR (context-length scaling).
• Always pin: the scaffold/agent, the compute and token budget, pass@k, the subset (Diamond vs. full, Pro vs. base, public vs. commercial), and the date. Standings in this space have a half-life of months.

If your mental model of LLM benchmarks still revolves around MMLU and GSM8K, it’s two years out of date — and the gap matters, because most of what gets quoted in vendor decks and board memos is either saturated, contaminated, or measured under a scaffold nobody disclosed.

This is a field guide to benchmarks that actually discriminate between frontier models in 2026, grouped by what they test — math, coding, agentic tool use, multimodal, long-context, real-world work — with the one caveat per benchmark that will embarrass you if you cite the headline number without reading the fine print.

The short version: a score is a point estimate under one harness, saturation is the default trajectory (several here went from sub-20% to 85%+ in a year), and “contamination-resistant” is a design claim, not a guarantee.

Read the curve, pin the setup, check the date.

References & Further Readings

Mathematical reasoning

AIME 2025 / HMMT 2025

• MathArena — independent, contamination-aware evaluation of contest math (AIME, HMMT, and harder sets): https://matharena.ai
• Source competitions: AIME (Mathematical Association of America); HMMT (Harvard–MIT Math Tournament, https://www.hmmt.org))
• Artificial Analysis math evaluations (AIME hosting + leaderboard): https://artificialanalysis.ai

FrontierMath (Tiers 1–3 and Tier 4) — Epoch AI

• Tiers 1–3 hub: https://epoch.ai/benchmarks/frontiermath-tiers-1-3
• Tier 4 hub: https://epoch.ai/benchmarks/frontiermath-tier-4
• “FrontierMath Tier 4: Battle Royale” (the unsolved-problem cracking analysis): https://epochai.substack.com/p/frontiermath-tier-4-battle-royale
• Note Epoch’s May 2026 AI-assisted error review (flagged issues in ~1/3 of problems; corrected scores pending) — check the hub pages for current status.
• Independent tracker: Our World in Data, “Share of FrontierMath problems solved” — https://ourworldindata.org/grapher/ai-frontiermath-over-time

Abstract / fluid reasoning

ARC-AGI-1 (Verified) and ARC-AGI-2 — ARC Prize Foundation

• Official hub, leaderboard, and reports: https://arcprize.org
• Conceptual origin: François Chollet, “On the Measure of Intelligence,” arXiv:1911.01547 — https://arxiv.org/abs/1911.01547

Science & expert knowledge

GPQA Diamond

• Paper: Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark,” arXiv:2311.12022 — https://arxiv.org/abs/2311.12022
• Code/data: https://github.com/idavidrein/gpqa

Humanity’s Last Exam (HLE) — Center for AI Safety + Scale AI

• Paper: arXiv:2501.14249 — https://arxiv.org/abs/2501.14249 (published in Nature, 2026, doi:10.1038/s41586–025–09962–4)
• Official site + leaderboard: https://agi.safe.ai and https://scale.com/leaderboard/humanitys_last_exam

MMMLU

• OpenAI’s multilingual MMLU dataset: https://huggingface.co/datasets/openai/MMMLU
• Underlying MMLU: Hendrycks et al., “Measuring Massive Multitask Language Understanding,” arXiv:2009.03300 — https://arxiv.org/abs/2009.03300

Factuality & hallucination

FACTS Benchmark Suite — Google DeepMind + Kaggle

• Paper: “The FACTS Leaderboard,” arXiv:2512.10791 — https://arxiv.org/abs/2512.10791
• Announcement: https://deepmind.google/blog/facts-benchmark-suite-systematically-evaluating-the-factuality-of-large-language-models/
• Leaderboard: https://www.kaggle.com/benchmarks/google/facts
• Practitioner commentary (the “~70% ceiling” framing for enterprise): VentureBeat — https://venturebeat.com/ai/the-70-factuality-ceiling-why-googles-new-facts-benchmark-is-a-wake-up-call

SimpleQA Verified — Google

• Paper: “SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge,” arXiv:2509.07968 — https://arxiv.org/abs/2509.07968
• Predecessor: OpenAI, “Measuring short-form factuality” (SimpleQA): https://openai.com/index/introducing-simpleqa/

Coding & software engineering

Codeforces (as an LLM benchmark)

• The live platform: https://codeforces.com
• CodeELO (ELO-calibrated evaluation against Codeforces): arXiv:2501.01257 — https://arxiv.org/abs/2501.01257

LiveCodeBench Pro

• Paper: Zheng et al., “LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?,” arXiv:2506.11928 — https://arxiv.org/abs/2506.11928
• Original LiveCodeBench: Jain et al., arXiv:2403.07974 — https://arxiv.org/abs/2403.07974 ; site: https://livecodebench.github.io

SWE-bench Verified

• Original SWE-bench: Jimenez et al. (ICLR 2024), arXiv:2310.06770 — https://arxiv.org/abs/2310.06770
• OpenAI’s Verified subset announcement: https://openai.com/index/introducing-swe-bench-verified/
• Leaderboards: https://www.swebench.com/verified.html

SWE-bench Pro — Scale AI

• Paper: “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?,” arXiv:2509.16941 — https://arxiv.org/abs/2509.16941
• Announcement: https://scale.com/blog/swe-bench-pro ; public leaderboard: https://scaleapi.github.io/SWE-bench_Pro-os/

Terminal-bench 2.0 — Stanford + Laude Institute (with Snorkel AI)

• Paper (OpenReview): https://openreview.net/forum?id=a7Qa4CcHak
• Official site + leaderboard: https://www.tbench.ai ; code: https://github.com/laude-institute/terminal-bench
• Contributor write-up: https://snorkel.ai/blog/terminal-bench-2-0-raising-the-bar-for-ai-agent-evaluation/

Agentic tool use & long-horizon coherence

MCP Atlas — Scale AI

• Paper: “MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers,” arXiv:2602.00933 — https://arxiv.org/abs/2602.00933
• Announcement + leaderboard: https://scale.com/blog/open-sourcing-mcp-atlas and https://labs.scale.com/leaderboard/mcp_atlas

Toolathlon (The Tool Decathlon) — HKUST-NLP (ICLR 2026)

• Paper: arXiv:2510.25726 — https://arxiv.org/abs/2510.25726
• Code: https://github.com/hkust-nlp/Toolathlon

t²-Bench (τ²-Bench) — Sierra

• Paper: Barres et al., “τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment,” arXiv:2506.07982 — https://arxiv.org/abs/2506.07982
• Code + leaderboard: https://github.com/sierra-research/tau2-bench ; https://tau-bench.com

Vending-Bench 2 — Andon Labs

• Official page: https://andonlabs.com/evals/vending-bench-2
• Original Vending-Bench: Backlund & Petersson, arXiv:2502.15840 — https://arxiv.org/abs/2502.15840
• Related real-world deployment context: Anthropic/Andon “Project Vend”

Multimodal, document, GUI & video

MMMU-Pro

• Paper: Yue et al., “MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark,” arXiv:2409.02813 — https://arxiv.org/abs/2409.02813
• Original MMMU (CVPR 2024): arXiv:2311.16502 ; site: https://mmmu-benchmark.github.io

CharXiv Reasoning

• Paper: Wang et al., “CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs,” arXiv:2406.18521 — https://arxiv.org/abs/2406.18521
• Project page: https://charxiv.github.io

Video-MMMU

• Paper: Hu et al., “Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos,” arXiv:2501.13826 — https://arxiv.org/abs/2501.13826

OmniDocBench (1.5 and later) — OpenDataLab (CVPR 2025)

• Paper: Ouyang et al., “OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations,” arXiv:2412.07626 — https://arxiv.org/abs/2412.07626
• Code (tracks v1.5 → v1.7+): https://github.com/opendatalab/OmniDocBench
• Saturation commentary worth reading: LlamaIndex, “OmniDocBench is Saturated, What’s Next for OCR Benchmarks?” — https://www.llamaindex.ai/blog/omnidocbench-is-saturated-what-s-next-for-ocr-benchmarks

ScreenSpot-Pro

• Paper: Li et al., “ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use,” arXiv:2504.07981 — https://arxiv.org/abs/2504.07981
• Code + leaderboard: https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding

Multilingual & cultural reasoning

Global PIQA — MRL Workshop @ EMNLP 2025

• Paper: Chang et al., “Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures,” arXiv:2510.24081 — https://arxiv.org/abs/2510.24081
• Code/data: https://github.com/mrlbenchmarks/global-piqa

Long-context

MRCR v2 (8-needle) (the correct name is MRCR — Multi-Round Coreference Resolution)

• Conceptual origin: DeepMind, “Michelangelo: Long-Context Evaluations Beyond Haystacks,” arXiv:2409.12640 — https://arxiv.org/abs/2409.12640
• DeepMind eval harness (the v2 8-needle variant): https://github.com/google-deepmind/eval_hub/tree/master/eval_hub/mrcr_v2
• OpenAI’s expanded MRCR dataset: https://huggingface.co/datasets/openai/mrcr
• Context-length curves: https://contextarena.ai

Real-world economic value

GDPval — OpenAI

• Paper: Patwardhan et al., “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks,” arXiv:2510.04374 — https://arxiv.org/abs/2510.04374
• Announcement + gold-set grader: https://openai.com/index/gdpval/
• Independent head-to-head Elo variant: Artificial Analysis “GDPval-AA” — https://artificialanalysis.ai/evaluations/gdpval-aa

Landscape, meta-trackers & further reading

• Epoch AI — Benchmarking Hub. Independent, methodologically transparent tracking of frontier math/reasoning benchmarks: https://epoch.ai/benchmarks
• Artificial Analysis. Independent re-runs of many benchmarks under documented harnesses (useful precisely because labs self-report under favorable scaffolds): https://artificialanalysis.ai
• Our World in Data — AI performance. Long-run time series for several benchmarks: https://ourworldindata.org/artificial-intelligence
• Stanford HAI — AI Index Report. Annual synthesis of the benchmark landscape and saturation trends: https://hai.stanford.edu/ai-index
• LMArena. Human-preference (Elo) leaderboard — a different axis than the capability benchmarks above: https://lmarena.ai

Recommended reading on benchmark methodology and its failure modes

• On the structural-conflict-of-interest issue (funder access to held-out sets): see Epoch AI’s FrontierMath conflict-of-interest statement on the hub pages above.
• On contamination-resistance strategies (licensing barriers vs. held-out private sets vs. live collection): contrast the SWE-bench Pro paper (legal/licensing approach) with LiveCodeBench (live time-segmented collection).
• On the lab-to-production gap (why a benchmark score is not a deployment guarantee): the Terminal-bench 2.0 error analysis and the practitioner write-ups linked under FACTS and OmniDocBench are the clearest treatments.