AI Intelligence Explosion: Why the Singularity Is Wrong?

Google researchers published in Science: AI models

spontaneously develop internal debate. The Singularity is wrong about intelligence. What this means for agentic AI and alignment?

You have heard the story.

A machine gets smart. Smart enough to improve itself. That improvement makes it smarter. That smarter version improves itself faster. The cycle accelerates until the machine becomes incomprehensibly intelligent and at that point, humans are either obsolete or irrelevant or worse.

One mind. One point. One outcome.

You have heard this so many times it has become background noise. It sits in the corner of every AI conversation, neither fully believed nor fully dismissed. Most people building with AI day to day have filed it under “probably not my problem right now.”

In March 2026, a paper published in Science, not an AI blog, not a startup press release, Science, the most cited scientific journal in the world, made an argument this story deserves. The argument is that the whole framing is wrong. Not wrong about whether powerful AI is coming. Wrong about what intelligence itself actually is. And the evidence for that argument is already sitting inside the models you use today.

Part 1 — What Happens Inside a Reasoning Model When It Gets Hard Problems Right

You probably know what a reasoning model is at this point. When you ask Claude or DeepSeek to work through a problem step by step, you are watching extended thinking, the model reasoning before answering rather than generating a reply immediately.

The intuition most people hold is sensible: the model thinks longer, and thinking longer makes it smarter. Like a human who takes more time with a difficult problem.

Researchers at Google studied this assumption. What they found was unexpected enough that they published it in Science.

What they discovered: When frontier reasoning models like DeepSeek-R1 improve on hard tasks, they are not simply thinking longer. They are running something that looks much more like a conversation with themselves. Internal voices that disagree, challenge each other, test each other’s conclusions, and then reconcile. The researchers named it a society of thought.

This does not sound remarkable until you ask the obvious follow-up.

Were these models trained to do this?

They were not. None of them were designed to produce internal debates. The training reward was simple: get the right answer. When that reward was applied consistently, the models spontaneously began generating multi-perspective, conversational reasoning structures. No instructions. No explicit design. The debate emerged because debate is apparently what good reasoning requires — and the models discovered that entirely on their own.

Society of Thought:

A reasoning model working on a hard problem does not produce one stream of thought moving toward a conclusion. It generates multiple perspectives internally — one that proposes, one that challenges, one that verifies, one that reconciles.

These internal voices argue with each other until they converge on an answer. The model was never asked to do this. It started doing it because it made the answers better.

That is the finding. Now here's why it matters beyond the model itself.

Part 2 — Intelligence Has Always Been This Way

The paper’s deeper argument is about the history of intelligence, not just the current behaviour of AI models.

The researchers — James Evans from the University of Chicago and the Santa Fe Institute, Benjamin Bratton from the Berggruen Institute, and Blaise Agüera y Arcas from Google — are arguing that every major intelligence explosion in human history has been social. Not the individual.

Think about the transitions they point to.

Primate brains did not grow in isolation. They scaled alongside the social groups they had to navigate. The bigger the group, the more complex the relationships, the more cognitive capacity was needed to track them. Intelligence was not a property of individual minds. It was a property of collective life.

Language changed the structure of human coordination. It’s not just communication between individuals, it’s the creation of something that persisted beyond any single person’s memory. Shared meaning that accumulated over generations. That accumulation is intelligence of a kind no individual human possesses alone.

Writing externalised that accumulation further. A book knows things its author has forgotten. A library knows things no librarian has read.

Institutions made it durable and scalable. A company, a government, a university is an intelligence structure. It knows things no single person inside it knows. It acts on goals no individual inside it chose alone.

Each transition followed the same pattern: individual intelligence hitting a ceiling, a new social structure forming that allowed collective intelligence to operate at a higher scale.

The core argument: The Singularity story imagines intelligence as a single quantity, concentrated in a single system, growing exponentially until it becomes incomprehensibly large. The evolutionary record suggests the opposite. Intelligence has never worked that way. Every time intelligence has exploded — in primate evolution, in human language, in the rise of institutions — it exploded socially. The next explosion will too.

The researchers argue that AI is not a break from this pattern. This is the next step in it.

Part 3 — The Centaur: What Human-AI Collaboration Actually Looks Like?

So what does this mean in practice?

If intelligence is fundamentally social, the most powerful AI systems will not be the ones with the biggest individual models. They will be the ones with the most effective coordination structures — between models, and between models and humans.

You may have seen the word “centaur” appear in AI discussions recently. The paper uses it deliberately.

What is a centaur? Not the mythological creature. A chess concept. After Garry Kasparov lost to Deep Blue in 1997, he proposed a format called advanced chess: a human and a computer working as a team, combining the human’s strategic intuition with the computer’s tactical calculation.

The result? The centaur team outperformed both the best human players and the best computer players working alone. Neither the human nor the machine was sufficient. The combination exceeded both.

A centaur is not a human using a tool. It is a new kind of actor — one whose agency genuinely exceeds what either the human or the machine could produce independently.

The researchers argue that human-AI centaur configurations are not a transitional phase on the way to pure machine intelligence. They are a destination. A stable and productive form of intelligence that emerges when humans and AI systems are designed to genuinely complement each other.

If that is right, then the Singularity story was not just wrong about form. It was wrong about what to build toward.

Part 4 — The Alignment Problem Nobody Is Talking About

There is a practical shift in this paper that has received almost no attention outside academic circles.

The dominant approach to making AI systems safer and more reliable is called RLHF.

RLHF in plain English: Reinforcement Learning from Human Feedback. A human evaluates the model’s outputs — rating some good, some bad. The model learns to produce outputs that humans rate highly. Over enough iterations, the model converges toward what its evaluators prefer. This approach has produced real improvements and underlies nearly every major AI system deployed today

This approach works. It also has a structural limit.

RLHF is a dyadic process. One model. One human evaluator. One feedback signal. The alignment happens between those two parties in isolation.

The problem is that the world is not dyadic.

A model deployed in an organization interacts with dozens of different humans with different preferences, different priorities, and occasionally conflicting goals. It takes actions that affect people who never rated its outputs. It operates in a context that no single evaluator can fully anticipate.

What the researchers call for instead?

Institutional alignment — the design of protocols, norms, and structures that govern how AI agents interact with each other and with humans at scale. Not aligning one model to one human. Designing the social infrastructure within which many models and many humans operate together.

This is an argument for thinking about AI governance the way we think about organisational design. It’s not “what does this model want” but “what rules and incentives and checks govern the collective behaviour of this system.”

Nobody in the current public conversation about AI is building this layer. The companies building AI are focused on model capability. The companies building safety tools are focused on individual model alignment.

The institutional layer — the equivalent of constitutional design, of the rules that govern markets, of the norms that allow organisations to function, is largely unaddressed.

That is either an enormous problem or an enormous opportunity, depending on your disposition.

Part 5 — What Am I Honest About?

Here is what I want to say clearly before you take any of this too far.

This paper is a perspective piece published in Science. It makes a compelling argument grounded in evolutionary biology, cognitive science, and AI research. The society of thought finding — the spontaneous emergence of internal debate in reasoning models, is based on specific, cited, verifiable empirical work.

The broader argument about intelligence, social structures, and the shape of what comes next is a thesis. It is well-reasoned. It is also a prediction about how technology will develop, and predictions about technology have a poor track record regardless of the quality of the reasoning behind them.

A separate study surveyed twenty-five leading AI researchers — people at frontier AI companies and leading universities — about their beliefs on recursive AI improvement and intelligence explosions. They were deeply divided. Smart, informed, thoughtful people looking at the same evidence and reaching substantially different conclusions.

What that means: the society of thought finding is real and worth understanding. The city metaphor for intelligence, plural, distributed, social, is the most useful framework

I have found for thinking about where AI architecture is heading. The institutional alignment argument is the most important underaddressed problem in the AI safety conversation right now.

The specific timeline, the specific form of what comes next? Hold that loosely.

The One Idea Worth Taking With You

The models you use already contain something more than you can see.

The reasoning that happens inside them when they work on hard problems is not a single voice moving step by step toward a conclusion. It is something more like deliberation. Multiple perspectives, emerging under pressure, converging toward accuracy through internal disagreement.

It is not a metaphor for intelligence. According to three centuries of cognitive science and according to what the models themselves have spontaneously rediscovered, it is what intelligence actually is.

Building toward more of that, more genuine multi-perspective reasoning, more functional human-AI collaboration, more institutional structures governing how these systems interact, is the direction the research points. Not as an abstract possibility. As a continuation of the most consistent pattern in the history of intelligence itself.

The explosion is not coming. It is already in progress. It just doesn’t look like anyone’s science fiction.

Primary source: Evans, J., Bratton, B. & Agüera y Arcas, B. (2026). Agentic AI and the next intelligence explosion. Science, 391, eaeg1895. arXiv:2603.20639. Additional source: AI Researchers’ Perspectives on Automating AI R&D and Intelligence Explosions. arXiv:2603.03338v2, March 2026.