The Oracle Problem

Here’s what’s happening these days: we have an AI that predicts experimental outcomes with 90% accuracy but cannot explain its reasoning. It tells us Experiment A will succeed, and Experiment B will fail. Which do we run? The rational choice destroys science.

A recent *Nature Human Behaviour* study by Xiaoliang Luo, Bradley Love, and colleagues at UCL delivered a striking result: large language models can predict neuroscience experimental outcomes more accurately than human experts. The AI, trained on published literature, outperformed neuroscientists at forecasting whether proposed studies would succeed or fail. A summary of the findings is also available here.

The immediate reaction: impressive. The deeper implication: troubling.

Here’s the oracle problem. You now have an AI system that predicts experimental outcomes with high accuracy but cannot explain its reasoning. It tells you Experiment A will likely succeed, and Experiment B will likely fail. What do you do?

The AI isn’t predicting the future. It’s predicting what aligns with past publication patterns.

The rational choice: Run Experiment A. It has higher predicted success, better publication odds, and more funding justification.

The problem: Science doesn’t advance through predicted successes. It advances through surprises. Results that violate existing patterns and force theoretical revision. And science progresses not just through prediction success, but through an understanding of why things happen the way they do!

If labs systematically favor experiments that their AI predicts will succeed, we’ve built a machine that optimizes for confirmatory research. The entire funding and publication system becomes biased toward work that fits existing literature patterns. Novel findings — the ones that don’t match the training data — become systematically disadvantaged.

The AI isn’t predicting the future. It’s predicting what aligns with past publication patterns. Those patterns are already biased: toward positive results, Western populations, well-funded topics, and established paradigms. An oracle trained on biased literature doesn’t transcend those biases — it codifies them.

Now scale this. Imagine grant reviewers start using AI prediction tools to assess “likelihood of success.” Imagine ethics committees using them to evaluate “scientific merit.” Imagine junior researchers optimizing their proposals for what the AI predicts will work.

The most valuable experiments are often the least predictable from existing literature.

You’ve created a system that perpetuates the exact structure of knowledge it was trained on. Research that challenges existing frameworks gets flagged as “unlikely to succeed” because it doesn’t match historical patterns. The paradigm-shifting experiments — the ones that look implausible until they work — never get funded.

This is the oracle problem at scale: prediction accuracy creates institutional pressure to avoid the very experiments that would generate genuinely new knowledge.

Consider the historical counterfactual. Would AI trained on pre-1980 psychology literature have predicted that Kahneman and Tversky’s heuristics and biases work would succeed? Almost certainly not—it violated the dominant rational-choice framework. Would AI trained on neuroscience before the mid-2000s have predicted that Lisa Feldman Barrett’s constructed emotion theory would hold up? Doubtful — it contradicts basic emotion theories.

The experiments that reshape fields are precisely the ones that don’t fit existing patterns. An oracle trained on those patterns would flag them as risky bets.

The Luo study is methodologically sound. The AI genuinely predicts better than humans. But prediction accuracy and scientific value are not the same thing. In fact, they may be inversely related. The most valuable experiments are often the least predictable from existing literature.

Science needs two things: confirmation, explanation, and surprise. AI optimizes for the first and systematically discourages the other two. That’s not a bug in the AI — it’s a feature of how prediction interacts with institutional incentives.

The path forward requires acknowledging what the oracle is actually doing. It’s not forecasting objective truth — it’s pattern-matching against a biased historical record. Those predictions are useful for some purposes (identifying likely replications, flagging methodological issues) but destructive for others (deciding what’s “worth” investigating).

If we let prediction accuracy drive research priorities, we’ll get a neuroscience that is internally consistent, highly confirmatory, and theoretically stagnant. A field that can predict outcomes but cannot generate insight.

An oracle that tells you what will happen based on what has happened is not a guide to discovery. It’s a recipe for intellectual lock-in.

The question facing neuroscience isn’t whether AI can predict experiments better than humans. It clearly can. The question is whether we’re disciplined enough not to let that capability destroy the incentive structure that enables genuine discovery.

I’m not optimistic.