The AGI Distraction: Why the AI Industry Is Selling You the Wrong Fear

We are terrified of the car that might one day fly. Meanwhile, people are getting run over in the street.

What we fear vs. What is actually happening. Source: Claude Sonnet 4.6

There is a peculiar ritual that plays out in AI discourse with increasing frequency. A prominent researcher, a well-funded CEO, or a credentialed futurist takes the stage and warns us — gravely, earnestly, with the full weight of their authority — that Artificial General Intelligence is coming. That superintelligence is coming. That we must prepare. That the risks are existential.

The audience listens. Journalists take notes. Regulators schedule workshops.

And somewhere, right now, an AI system is fabricating a medical reference that a clinician is about to act on. Somewhere, a hiring algorithm is systematically excluding candidates it was never designed to evaluate. Somewhere, a legal AI is constructing a citation that does not exist — fluently, confidently, without any structural mechanism to stop it.

The house is on fire. We are debating whether the smoke alarm might, one day, become sentient.

The Problem With AGI as a Concept

Let me be precise about what I am accusing.

I am not accusing anyone of bad faith. I am accusing an entire field of intellectual imprecision so severe that it has become functionally indistinguishable from bad faith.

The term Artificial General Intelligence is used constantly and defined almost never. When pressed, practitioners offer proxies: a system that can do anything a human can do. A system that can improve itself recursively. A system that surpasses human performance across all domains. These definitions are not equivalent. They do not converge on the same set of systems. They do not suggest the same timelines, the same risks, or the same design requirements.

What is the unit of measurement for AGI? What experiment would falsify the claim that a given system has achieved it? Under what conditions would we say: no, this system is not AGI, and here is the structural reason why?

These questions are not asked. In their place, we have benchmark scores.

This is the first category error — and it matters, because how you measure something determines what you build.

The Benchmark Is Not the Territory

A benchmark is a proxy. This statement should be uncontroversial, but its implications are routinely ignored.

MMLU measures performance across 57 academic subjects. HumanEval measures the ability to write functional code. SWE-bench tests resolution of real software engineering issues. These are non-trivial tasks. A system that performs well on them is doing something genuinely impressive.

But benchmark performance tells you what a system produces under controlled conditions. It does not tell you how it produced it, whether the process is reproducible, whether the same reasoning path will occur tomorrow, or whether a human reviewer had any meaningful opportunity to intervene when the process went wrong.

Consider the analogy that keeps coming back to me: blood pressure. Blood pressure is a real and important indicator of cardiovascular health. Physicians measure it. Researchers study it. Its correlation with adverse outcomes is well-documented.

But blood pressure is not health. A person whose blood pressure is optimized through pharmaceutical intervention while their underlying arterial disease progresses is not a healthy person. They are a person with good numbers and a structural problem that the numbers are not measuring.

Benchmarks are blood pressure. Intelligence is the full cardiovascular system. The AI industry has been optimizing the measurement and calling it the thing being measured.

This is not merely a philosophical complaint. It has engineering consequences. When you define progress as benchmark improvement, you build systems that improve benchmarks. Those systems may or may not improve the structural properties that make AI safe to deploy — because safety, accountability, and justifiability are not on the benchmark. They are on a different axis entirely. An axis that most of the field has declined to measure.

The Scaling Hypothesis and Its Hidden Assumption

Beneath the AGI narrative lies an assumption that is rarely stated because, if stated plainly, it would demand immediate scrutiny: that capability and trustworthiness move together. That a more powerful model is a safer model. That if we scale long enough, the accountability problem will solve itself.

My experiments say otherwise.

In structural accountability tests run across multiple state-of-the-art models — measuring not accuracy, but traceability, reproducibility, and the capacity to distinguish grounded evidence from generated inference — the most capable models were not the most accountable ones. Capability and accountability are orthogonal dimensions. Scaling one does not scale the other.

This is not a technical footnote. It is a structural argument about what AI development has been optimizing for, and what it has been silently neglecting.

The car industry understood an analogous point decades ago. Maximum speed and crash safety are engineered independently. No one argues that if the engine is powerful enough, the brakes will eventually appear. Brakes are designed separately, from the beginning, as a structural requirement — not as an afterthought, not as a feature to be added once the performance targets are met.

The AI industry has not yet made this distinction cleanly. The result is systems with extraordinarily powerful engines and no guarantee that the brakes were designed at all.

“Hallucination Is Inevitable” Is Not an Absolution

Here I want to say something that I mean very precisely, because it is easy to misread.

I have argued in my own research — and I maintain — that hallucination is not a bug. It is a structural consequence of how language models work. A finite model trained on a distribution over possible texts will produce outputs that deviate from ground truth with nonzero probability. This is a mathematical fact, not a product defect. Eliminating hallucination entirely would require eliminating the generalization capacity that makes language models useful.

But I want to be equally precise about what this argument does not entail.

A car manufacturer cannot attach a sticker reading This vehicle may cause accidents and thereby discharge its obligation to build a safe vehicle. The inevitability of risk does not eliminate the obligation to minimize it structurally — through seatbelts, airbags, collision detection systems, and decades of engineering effort aimed specifically at reducing the gap between unavoidable risk and actual harm.

The same principle applies here. If hallucination is inevitable, then the obligation of every model provider and every deployer is to build the structural mechanisms that prevent hallucinated outputs from becoming uncontested inputs to the next decision in the chain. Fixed evidence boundaries. Separated functions. Contemporaneous records. Human oversight positioned where it can make a genuine epistemic difference — not as a rubber stamp after the decision has already been taken.

To acknowledge hallucination’s inevitability while providing no structural response to it is not honesty. It is the use of a true statement to avoid a real obligation.

And this is where the question of responsibility becomes unavoidable.

The Accountability Gap We Have Chosen Not to Close

We are extraordinarily sensitive to the harms caused by automobiles. We require manufacturers to meet crash safety standards before vehicles reach the market. We require drivers to be licensed. We require insurance. We maintain accident investigation bodies. We update safety regulations when new failure modes emerge. We hold manufacturers liable when design defects cause foreseeable harm.

The systems we have built around automotive risk are not perfect. But they represent a serious, sustained, institutionally backed effort to match accountability to consequence.

Now consider what we have built around AI systems that are actively displacing workers at scale, intervening in medical and legal decision-making, and shaping the information environment that democratic publics use to form political judgments.

A few disclosure requirements. Some voluntary commitments. A great deal of talk about future risk.

The asymmetry is not accidental. Automotive liability was not a gift from manufacturers. It was extracted, over decades, through litigation, regulation, and the sustained political pressure of constituencies who had suffered harm. The AI industry has not yet faced that reckoning — in part because the harms are diffuse, in part because the beneficiaries of the current arrangement are very well-resourced, and in part because the most prominent voices in AI discourse have been more interested in the drama of existential risk than in the unglamorous work of structural accountability.

Existential risk is a compelling narrative. It positions the speaker as a guardian of humanity. It justifies extraordinary resources and extraordinary latitude. It is also, conveniently, a risk that cannot be evaluated, tested, or falsified today — which means the people raising it cannot be held accountable for being wrong.

The risks that can be evaluated — that are being evaluated, right now, in courtrooms and clinics and hiring offices — receive comparatively little attention from the same voices.

This is not a coincidence. It is a choice. And it is a choice that all of us — developers, deployers, researchers, users, regulators, and observers — have collectively permitted.

What Fear Should Actually Look Like

I am not arguing that long-term risks deserve no attention. I am arguing that the current distribution of attention is badly miscalibrated.

The fear should not be AGI. Not because AGI is impossible, but because the concept is too undefined to be actionable. You cannot regulate what you cannot measure. You cannot build what you cannot specify. You cannot prevent what you cannot describe.

The fear should not be superintelligence. Not because recursive self-improvement is a trivial concern, but because we have not yet built systems that can reliably perform a two-key sort under identical conditions on consecutive runs. The distance between here and superintelligence is not one that warrants more attention than the distance between here and structural accountability.

The fear that is warranted — the fear that is actionable, measurable, and structurally addressable — is this:

We are deploying systems that make consequential judgments about human lives, at scale, without the structural mechanisms that we have required of every other decision-making institution that has operated at comparable stakes.

Courts must write reasoned opinions. Physicians must document differential diagnoses. Pilots must operate within recorded checklists. Auditors must maintain evidence trails. These requirements were not invented to slow things down. They were invented because experience showed, repeatedly, that capable human beings making high-stakes decisions without structural accountability produced errors that could not be corrected and harms that could not be attributed.

We built institutions to compensate for what human cognition cannot reliably do on its own. We are now deploying AI systems — whose failure modes differ from human failure modes in important ways, and are in some respects more severe — without the equivalent institutional compensation.

The intelligence that cannot be held to account is not a future problem. It is a present one.

This Is Our Responsibility

I want to end without the easy comfort of assigning blame to a single actor.

Yes, the AI industry has marketed capability as safety. Yes, the research community has built evaluation frameworks that measure what is easy to measure, and treated the absence of accountability metrics as evidence that accountability is not a measurable property. Yes, regulators have moved slowly.

But the rest of us have also chosen to be impressed by the benchmarks, to accept the disclaimers, to marvel at the fluency, and to treat the anxiety about AGI as evidence of seriousness rather than as a displacement of responsibility from present to future.

We are all participants in this. The public fascination with superintelligence — with the drama, the stakes, the cinematic scale of existential risk — has consumed the attention that should have gone to the quieter, harder, less glamorous question of whether the systems we are using today can show us why they made the decision they just made, and what we can do when they were wrong.

The accountable machine is not a utopian vision. It is not a demand for perfection. It is a demand for the minimum condition that responsible judgment has always required: that the basis of the decision can be shown, the error can be found, and someone can be held to account.

We built that demand into courts. Into hospitals. Into aviation. Into financial auditing.

We have not built it into AI. We have accepted its absence.

That acceptance — not the technology, not the models, not the researchers — is the problem we are in the best position to change.

If this argument resonated

The problems raised in this article — the benchmark illusion, the category error, the accountability gap, the misplaced fear — are developed in full in my book, The Accountable Machine: Why AI Must Justify Every Decision It Makes.

It traces the path from philosophical discomfort to working architecture: what justified judgment actually requires, why scaling cannot produce it, and what it looks like when accountability is designed in from the start rather than disclaimed away.

**The Accountable Machine — available on Amazon**

Myung Ho Kim is the author of The Accountable Machine: Why AI Must Justify Every Decision It Makes and the architect of the Structured Cognitive Loop (SCL). He is Lead Professor of the AI Microdegree Program at JEI University, South Korea. He writes at medium.com/@enkiluv.