We Used to Measure Reliability.

Now We Measure Failure by Vibes.

For most of the history of software engineering, reliability was something we could measure. Not guess at. Not infer. Not approximate. Measure.

We had defect density. We had DPMO. We had CFR. We had uptime SLAs. We had MTTR and MTBF. We had deterministic systems that behaved the same way today as they did yesterday.

Reliability was a discipline.

Reliability wasn’t just an engineering concern — it was a business safeguard.

Then AI‑generated applications arrived. And reliability became a rumor.

Today, teams aren’t measuring reliability. They’re measuring symptoms of unreliability. They’re measuring disappointment. They’re measuring drift. They’re measuring hallucinations. They’re measuring the number of times an AI agent “just kind of did something weird.”

We have replaced engineering metrics with vibes.

And the industry is pretending that’s normal.

Before AI, reliability was a contract

Traditional software was deterministic. If a function failed, it failed the same way every time. If a service degraded, logs told you why. If a deployment broke something, you rolled it back.

This determinism enabled a shared language of reliability:

• Defects per Million Opportunities (DPMO)
• Case Failure Rate (CFR)
• 99.9%+ uptime SLAs
• Unit test coverage
• Integration test suites
• Regression testing
• Performance baselines

These weren’t optional. They were the foundation of enterprise software.

When something broke, you could trace it. When something degraded, you could measure it. When something failed, you could reproduce it.

Reliability was observable, measurable, and enforceable.

AI‑generated systems broke the measurement model

AI‑generated applications are not deterministic systems. They are probabilistic systems wrapped in deterministic scaffolding.

They fail differently:

• silently
• intermittently
• contextually
• irreproducibly
• without logs
• without ownership
• without a consistent definition of “failure”

And because the systems are non‑deterministic, the old metrics simply don’t apply.

You can’t calculate DPMO when the “opportunities” change with every prompt. You can’t calculate CFR when the “case” is a multi‑step reasoning chain that mutates mid‑execution. You can’t define uptime when the model’s behavior changes without deployment.

The result is a statistical fog.

The industry is using proxy metrics — and pretending they’re reliability metrics

Because we can’t measure reliability directly, the industry has started measuring proxies, and the results are not encouraging:

• Aggregated research from Gartner, IDC, Forrester, and NVIDIA’s 2026 State of AI reports that while 79% of enterprises have adopted AI agents in some form, only 11% run them in production.
• In a peer-reviewed research paper arXiv reports that AI Agent performance drops from 60% (single run) to 25% (8‑run consistency).
• Gartner predicts over 40% of agentic AI projects are projected to be canceled by 2027 due to reliability issues.
• High model accuracy does not translate to agent reliability — compounding errors across reasoning steps degrade real‑world performance.
• Agents fail in ways benchmarks cannot detect, especially in multi‑step workflows with tool calls, memory, and state .

These numbers are not reliability metrics. They are failure‑to‑launch metrics. They measure friction, not function.

They tell us nothing about the reliability of the systems that did reach production.

They tell us everything about the reliability of the systems that didn’t.

What Would’ve Happened If a Pre‑AI Dev Team Reported These Numbers

Let’s be honest: if a development team in the pre‑AI era had walked into an enterprise steering committee and presented the reliability profile we’re normalizing today, the meeting would not have ended with applause for “innovation.” It would have triggered an escalation.

Because in the pre‑AI world:

• A system that succeeded once but dropped to 25% reliability across repeated runs (as documented in Beyond Accuracy, 2025) wasn’t “expected variability.” It was a catastrophic program failure requiring immediate containment.
• A platform whose reliability plateaued for two years despite major upgrades (as shown in Towards a Science of AI Agent Reliability, 2026) wasn’t “the nature of probabilistic systems.” It was a strategic red flag that the architecture was fundamentally flawed.
• A workflow that behaved differently across identical inputs wasn’t “stochasticity.” It was a P1 incident demanding root‑cause analysis.
• A system that silently misreported outcomes — like the real‑world cases where agents deleted production databases, made unauthorized purchases, or gave illegal regulatory advice wasn’t a “hallucination.” It was an audit finding with compliance implications.
• A process that required a human in the loop to catch routine errors wasn’t “responsible AI.” It was a failed automation initiative that would have been paused until reliability improved.

In the pre‑AI world, these weren’t quirks. They were career‑limiting events.

To understand how far we’ve drifted, it’s worth asking a simple question: what would these numbers have meant before AI?

A CIO in 2015 hearing that a new system:

• worked once
• failed three times
• behaved inconsistently
• and occasionally fabricated data with confidence

…would not have said, “Let’s iterate.” They would have said, “Shut it down.”

Not because they lacked imagination. Because they had standards.

Standards that protected the business. Standards that protected customers. Standards that protected the integrity of the systems we rely on.

Today, those same failure patterns are being reframed as “model variability,” “expected drift,” or “the nature of agentic systems.”

But let’s not pretend these would have been acceptable in any other era of enterprise technology.

AI systems fail in ways we don’t have language for

Traditional software fails predictably. AI systems fail behaviorally.

AI failure modes include:

• reasoning drift across multi‑step tasks
• tool‑call parameter corruption
• infinite loops due to missing success criteria
• context‑window degradation mid‑task
• stochastic variability across identical inputs
• silent misreports (the most dangerous failure of all)

As InfoQ notes, an agent can “work perfectly in a sandbox but silently misreport a failed refund in production” — and no benchmark will catch it .

This is not a defect. This is a divergence.

And divergence is not something the old reliability frameworks were built to measure.

The collapse of shared definitions

The most dangerous shift is not technical — it’s conceptual.

Before AI, we had shared definitions:

• What counts as a defect
• What counts as a failure
• What counts as uptime
• What counts as correctness
• What counts as production‑ready

Now, every team invents its own definitions:

• “It works as long as it doesn’t hallucinate too much.”
• “It’s reliable as long as the agent eventually gets there.”
• “It’s production‑ready as long as a human is watching.”
• “It’s fine as long as the user doesn’t notice.”

This is not reliability. This is coping.

The enterprise risk nobody wants to name

Enterprises are deploying AI systems whose failure modes they cannot:

• predict
• reproduce
• quantify
• govern
• audit
• certify

And because they cannot measure reliability, they measure vibes:

• “It seems stable.”
• “It worked in the demo.”
• “It passed our LLM-as-a-judge tests.”
• “It didn’t hallucinate this time.”
• “It feels good enough.”

But “good enough” is not a reliability standard. It’s a gamble.

And enterprises are betting their workflows, their data, and their reputations on systems they cannot measure.

The truth we need to say out loud

AI didn’t lower reliability standards. AI ignores them.

The industry did not abandon reliability because it became careless. It abandoned deterministic definitions of reliability because probabilistic systems no longer produce binary outcomes.

Traditional observability assumed:

• reproducibility
• deterministic causality
• stable execution paths
• fixed failure domains

LLM/agent systems violate all four.

Modern AI systems don’t merely challenge reliability engineering. They challenge the foundational assumptions observability itself was built upon.

We are trying to measure probabilistic behavior with deterministic tools. We are trying to apply software engineering metrics to statistical engines. We are trying to enforce reliability in systems that do not produce the same output twice.

The result is predictable:

We used to measure reliability. Now we measure failure by vibes.

What comes next

The next era of AI engineering will not be defined by bigger models. It will be defined by new reliability frameworks:

• trajectory‑level evaluation
• multi‑run consistency scoring
• tool‑call correctness metrics
• behavioral drift detection
• agent‑level observability
• continuous evaluation pipelines
• governance‑aligned reliability thresholds

The industry is already moving in this direction — slowly, unevenly, and under pressure.

But until these frameworks mature, enterprises must confront an uncomfortable truth:

You cannot manage what you cannot measure. And right now, we cannot measure AI reliability.

Until we rebuild reliability for probabilistic systems, enterprises will continue flying blind.