I read 4 research papers on why metrics quietly stop working -from classic image classification to LLM-as-judge

In 2019, a team of researchers set out to do something that sounds almost too careful to be interesting: they rebuilt the ImageNet test set from scratch, copying the original data-collection process as faithfully as they could. Same source, same labeling pipeline, same selection criteria. No shortcuts.

Then they ran the best image classifiers of the era against it.

Accuracy dropped. Not by a rounding error-by 11 to 14 percentage points on ImageNet, and as much as 15 points on CIFAR-10. Nothing about the models had changed. Nothing about the “true” images-in-the-world had changed either, as far as anyone could tell. The only thing that changed was which specific photographs ended up in the test set.

The obvious explanation is that the field had been quietly cheating -peeking at the old test set too many times across years of published papers, unconsciously tuning models to its specific quirks. That’s a real phenomenon, and the researchers checked for it directly. It didn’t explain the gap. The drop survived the control.

That result should unsettle anyone who trusts a benchmark number. Not because the number was calculated wrong-the arithmetic was fine -but because the number quietly stopped meaning what everyone assumed it meant, and nobody would have known without going back and rebuilding the test.

That’s the subject of this piece. Not “accuracy isn’t everything”-you already know that. This is about something sharper: three separate, well-documented ways a metric can keep reporting a clean number while the thing it’s supposed to measure has already slipped out from underneath it. One is nobody’s fault. One is the field’s fault. One is the measuring instrument’s fault. All three produce the same symptom -a dashboard that still looks fine.

Part 1: When the world moves and nobody touches anything

The ImageNet story above comes from Recht, Roelofs, Schmidt, and Shankar’s 2019 paper, Do ImageNet Classifiers Generalize to ImageNet? It’s worth sitting with the methodology for a moment, because the methodology is the actual finding -the headline number is almost beside the point.

The researchers weren’t sloppy. They deliberately replicated the original ImageNet construction pipeline: same image sourcing strategy (Flickr search plus human annotation), same category definitions, same quality filtering. If a new test set built this way still produces a different accuracy number, the standard “you must have done something wrong” explanations start running out of road.

The leading candidate explanation, before this paper, was adaptive overfitting: the idea that years of researchers repeatedly evaluating on the same fixed test set -even without directly training on it -lets information leak through, because everyone unconsciously favors architectures and hyperparameters that happen to do well on that specific set. It’s a real statistical phenomenon, related to the multiple-comparisons problem in classical statistics. It’s also the comfortable explanation, because it implies the fix is simple: just don’t reuse test sets so much.

The authors tested this hypothesis and found that the accuracy drop remained unexplained by adaptive overfitting. Their more troubling conclusion was that the new test set had ended up subtly harder than the original, despite an almost identical construction process -small, hard-to-detect differences in image difficulty, framing, or ambiguity, compounding into a real performance gap.

Translate that into plain terms: a test set can be built with total integrity, evaluate a model that’s been trained with total integrity, and still stop faithfully representing what it’s supposed to represent -simply because the underlying data distribution is not as stable as everyone assumed. Nobody gamed anything. Nobody made a mistake. The world just wasn’t as fixed a target as the eval process pretended it was.

For anyone building systems that operate on real-world inputs -new camera sensors, new document formats, new lighting conditions, evolving fraud tactics -this isn’t a historical curiosity about a 2019 vision paper. It’s a standing warning: your eval set was only ever a snapshot, and snapshots age.

Part 2: When people move, on purpose

If distribution shift is the world quietly invalidating a metric, the next failure mode is people doing it, deliberately, often without technically breaking any rules.

The economist Charles Goodhart originally observed this in the context of UK monetary policy, in a form later summarized as: once a measure becomes a target, it stops being a good measure. In 2018, Manheim and Garrabrant formalized this into a taxonomy -Categorizing Variants of Goodhart’s Law -breaking the single folk-observation into four distinct statistical mechanisms:

A taxonomy is useful vocabulary, but vocabulary alone doesn’t tell you this is actually happening in AI research right now. For that, a 2025 paper called The Leaderboard Illusion is far more useful-and far more uncomfortable, because it’s about a benchmark many researchers currently treat as close to a gold standard: Chatbot Arena.

The paper argues that undisclosed private testing practices let a handful of providers test multiple variants of an unreleased model before choosing which one to submit publicly. Its most concrete data point: the paper reports that Meta tested 27 private LLM variants between January and March 2025, in the lead-up to the Llama 4 release, and states that only one public score was ultimately submitted- one that landed near the top of the leaderboard. According to the paper, the other variants’ scores were never disclosed.

It’s worth noticing that this single practice is actually two Goodhart mechanisms stacked on top of each other, not one clean case study. Publishing only the best of many private attempts is a straightforward extremal effect- picking the maximum of many draws inflates the apparent score above the “true” quality of any single model, the same statistical phenomenon behind p-hacking. But doing this privately, without disclosure to those relying on the leaderboard, is what the paper frames as adversarial -an agent aware of how the measurement works allegedly shaping its behavior to exploit it, at the expense of the metric’s validity for everyone relying on it. Manheim and Garrabrant’s taxonomy presents these as clean, separable categories. Real failures, it turns out, are often hybrids.

It’s also worth being fair to the other side of this: the platform behind Chatbot Arena publicly disputed several of the paper’s specific claims, including the reported share of open-weight models represented on the leaderboard. Some of this remains a live, contested disagreement rather than a fully settled scandal. That’s a more honest thing to sit with than a tidy morality tale- and arguably a more useful one, because it’s a reminder that even the critique of a benchmark is itself a claim that deserves scrutiny, not blind acceptance.

A natural objection here is: doesn’t periodic re-evaluation just fix this? It’s worth answering directly, because it’s the obvious next question. Refreshing a benchmark helps with Part 1’s failure mode- a new, updated test set can catch a world that’s moved. It does far less for this one. If the refreshed benchmark is built and gamed through the same process- private testing, selective disclosure -a new snapshot inherits the same vulnerability the old one had. Refreshing the data doesn’t fix a broken incentive structure sitting upstream of the data.

So far, both failure modes came from outside the metric itself -the world moved, or people moved, but the tool doing the measuring was sound in principle. What happens when the tool itself was never fully trustworthy to begin with?

Part 3: When the judge grades itself

Classification metrics like accuracy, precision, and recall all assume something modern generative AI mostly doesn’t have: a clean, objective ground-truth label. “Is this email spam” has a right answer. “Is this chatbot response helpful, honest, and well-written” mostly doesn’t -there’s no confusion matrix for prose quality.

The field’s practical answer has been LLM-as-a-judge: use a strong model to grade another model’s output, at a scale no team of human raters could match. Zheng et al.’s 2023 paper, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, is the paper that put this practice on a real empirical footing-and the headline result is genuinely reassuring at first glance: GPT-4 acting as a judge agreed with human preference judgments more than 80% of the time. That’s a strong number. For a lot of practical purposes, it’s good enough to build on.

But the same paper is also the one that names the specific ways this instrument can be quietly unreliable, even while producing that impressive agreement rate on average -and the concrete numbers are more striking than the general concept.

Verbosity bias. The authors designed a direct test: take 23 model answers that already contained a numbered list, ask GPT-4 to rephrase that list without adding any new information, and prepend the rephrased version to the original -so the “new” answer is longer and more repetitive, but contains exactly zero new substance. Then see whether each judge model prefers the padded version.

Claude-v1 and GPT-3.5 were fooled 91.3% of the time -judging pure padding as a better answer in nine cases out of ten. GPT-4 was fooled 8.7% of the time. That’s not a subtle statistical footnote; it’s a judge instrument that, for two of the three models tested, could be reliably manipulated by simply making an answer longer.

Position bias. A cleaner, more basic test: does a judge give the same verdict when you swap which answer appears first?

Under the paper’s default prompt, Claude-v1 gave a consistent verdict only 23.8% of the time -meaning more than three-quarters of the time, simply reordering two unchanged answers flipped its judgment. GPT-3.5 landed at 46.2%. GPT-4, the strongest of the three, still only reached 65.0% -meaning even the best-performing judge in this test disagreed with itself over a third of the time on nothing but answer order.

The paper names a third failure mode too: self-enhancement bias, where a judge tends to rate outputs from its own model family more favorably -the authors observed GPT-4 favoring itself with a 10% higher win rate, and Claude-v1 favoring itself with a 25% higher win rate, though they’re upfront that the effect is harder to isolate statistically than the two attacks above, given the limited data available.

None of this is the world drifting, and none of it is anyone deliberately gaming a leaderboard. The measuring instrument itself has systematic blind spots baked into how it evaluates-quietly present even on the roughly 80% of cases where it happens to land on the same answer a human would, and invisible unless someone specifically goes looking for the other 20%.

Here’s an honest gap, worth naming rather than papering over: this piece doesn’t get into why self-enhancement bias happens at a mechanistic level-whether it’s about a model favoring lower-perplexity, more “familiar-sounding” text, or something else entirely. That’s an active, unsettled research thread, and it’s outside what I chased down for this piece. It’s a genuinely open question, not a solved one-which feels like the right place to end up, given everything above it.

The pattern underneath all three

Line these up and something clicks that isn’t obvious from any single paper alone: precision, recall, a Chatbot Arena Elo score, and an LLM-judge rating look nothing alike, mechanically. One’s built from a confusion matrix. One’s built from human preference votes. One’s built from another model’s opinion. And yet all three fail the exact same way-a proxy that was a faithful stand-in for something real, at the moment someone built it, silently stops being faithful, through a different door each time. The world moves out from under a static test set. People, incentivized by the metric itself, move the metric out from under the world. Or the instrument doing the measuring was never fully aligned with the thing it claims to grade.

That’s not three unrelated cautionary tales. It’s the same disease, from classical computer vision in 2019 to generative AI evaluation in 2025 -which suggests it’s not a quirk of any one metric or any one era of machine learning. It’s closer to a property of measurement itself, whenever the thing being measured is complex enough to have a life independent of the number summarizing it.

Where that leaves you

There’s no clean five-step fix here, and I’d trust this piece less if it pretended otherwise. But the three failures do share one requirement for anyone trying to catch them: a metric needs to be periodically re-validated against the actual target it’s a proxy for -not just checked for internal consistency, not just monitored for whether the number changed. Distribution shift is at least partially addressable this way. Self-inflicted drift needs something harder: transparency into how a benchmark result was produced, not just what the result was. Instrument drift is the least solved of the three -the tool doing the validating is itself the thing in question, which is a strange, recursive problem the field is still working through.

If there’s one thing worth carrying out of this: the next time a metric looks clean, the right question isn’t “is this number accurate.” It’s “what would have to be true for this number to have quietly stopped meaning what I think it means” -and whether anyone has actually checked.

References

• Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). Do ImageNet Classifiers Generalize to ImageNet? — arXiv:1902.10811
• Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — arXiv:2306.05685
• Manheim, D. & Garrabrant, S. (2018). Categorizing Variants of Goodhart’s Law — arXiv:1803.04585
• Singh, S. et al. (2025). The Leaderboard Illusion — arXiv:2504.20879