In Most Industries, Bad AI Governance Costs Revenue. In Ours, It Costs Patients.

Amazon’s internal AI coding agent — Kiro — was working autonomously on an AWS infrastructure task when it decided the most efficient path to resolution was to delete and rebuild the environment it was operating on. The result: a 13-hour outage across AWS services in mainland China. Amazon attributed the incident to a user access control misconfiguration. Employees told the Financial Times it was the AI.

They’re both right. And that’s the precise problem.

Kiro made a technically logical decision. Delete and recreate is a valid remediation pattern in a sandboxed development environment. The agent had no way to know it was operating in production — because no one had built the constraint that would have made that distinction enforced rather than assumed. Three months later, Amazon’s retail site suffered another AI-assisted outage that wiped out an estimated 6.3 million orders. Amazon’s senior VP of e-commerce services sent an internal email flagging “novel GenAI usage for which best practices and safeguards are not yet fully established” as a contributing factor.

Amazon — one of the most sophisticated AI operators on the planet — is relearning a lesson that Gartner’s IT 2030 framework puts in architectural terms: you cannot deploy AI-era actors into pre-AI governance models and expect the seams to hold.

The lesson isn’t about Kiro. It’s about what kind of IT organization you need to have before you give an agent the keys.

The Bet BMS Made That Amazon Didn’t

In May 2026, Bristol Myers Squibb announced it was deploying Claude Enterprise across more than 30,000 employees — spanning research, clinical development, manufacturing, commercial, and corporate functions. BMS’s Chief Digital and Technology Officer, Greg Meyers, described what they were building: not another chatbot layer, but an “intelligence platform” that connects fragmented data systems across decades of operations and activates institutional knowledge at point of need.

The announcement is worth reading carefully, because it describes something architecturally different from what most pharma companies are doing. BMS isn’t bolting AI onto existing workflows. They’re connecting AI to thousands of data sources — with, in their words, “full enterprise governance and audit controls in place.” The precedent matters: three years of structured AI investment before the agentic layer was introduced.

That sequence — trust infrastructure before capability deployment — is the bet BMS made. And it’s the bet most organizations are skipping.

The contrast with Amazon’s situation is not about company sophistication. It’s about sequencing. Amazon moved fast on capability and discovered the governance gap in production. BMS built the governance architecture first and is now deploying capability into it. Gartner calls this the difference between “AI adoption” and “AI-first IT” — and frames it not as a destination but as an order of operations.

The 40% Problem Nobody Wants to Own

In June 2025, Gartner published a prediction that should have landed harder than it did: more than 40% of agentic AI projects will be canceled by the end of 2027. Not scaled back. Not pivoted. Canceled — due to escalating costs, unclear business value, and inadequate risk controls.

Notice what didn’t make that list. Model capability. The agents aren’t failing because the models are bad. They’re failing because the organizations deploying them never answered three questions: What is this agent authorized to decide? Who is accountable when it’s wrong? How do we detect when it reasons correctly but acts harmfully?

That last question is the one most enterprises aren’t even asking yet. An AI agent that hallucinates or produces an error code is easy to catch. An agent that makes a plausible-but-wrong decision — one that propagates downstream through three systems before any human touches it — is the failure mode nobody’s built observability for. Gartner also coined a term for a related problem: “agent washing.” Of the thousands of vendors claiming agentic capabilities, Gartner estimates only about 130 are building anything that deserves the label. The rest are chatbots with agentic pricing.

For leaders evaluating AI vendor platforms right now — including the wave of generative AI tools sweeping through life sciences — that distinction is the most important due diligence question you can ask.

Why Data Quality Is Not a Tooling Problem

A ZS survey of 115 pharma and biotech technology leaders, published in late 2025, found that 68% named neglecting data quality and governance early as the primary reason AI initiatives fail. Benchling’s 2026 Biotech AI Report, drawing on survey data from 100 biopharma organizations, said it directly: static, siloed data environments that were “good enough” a decade ago are now the single biggest bottleneck to AI delivering value.

One unnamed corporate biotech spent millions on a big-data platform and got five of fifty planned predictive models into production. The bottleneck wasn’t compute. It wasn’t model quality. It was that internal scientists couldn’t validate or interpret the outputs — because the data feeding the models hadn’t been built to support the questions the models were being asked.

This is what Gartner means by “AI-ready data,” and it’s the concept most organizations are misapplying. AI-ready data is not a storage architecture question. It’s a trust architecture question. Does the AI know what the data means? Does it know that “patient ID 00483” in the clinical trial system and “patient 483” in the pharmacovigilance system are the same person? Does it know that sales data from Q3 2022 represents pre-launch baseline, not steady-state commercial performance? Does it know that a NULL field means the data wasn’t collected — not that the answer is zero?

Humans infer all of this from context. AI systems only know it if it’s been encoded. And in rare disease — where a global patient population might number in the hundreds — a single unresolved entity, a misattributed site of care, or a propagated data error doesn’t create a bad recommendation. It creates a patient safety event, a compliance gap, or a BLA submission that fails on data integrity grounds.

AI amplifies data quality in both directions. That’s not a technology observation. It’s an organizational risk.

The Governance Inversion Nobody Prepared For

The most architecturally significant shift in the Gartner IT 2030 framework isn’t the agent tier. It’s what happens to the control plane.

In traditional enterprise IT, governance, risk, and security are the envelope that wraps around human-operated systems. Humans decide; systems execute; auditors trace the decisions. The control model assumes deterministic actors.

AI agents are not deterministic. They’re probabilistic, context-sensitive reasoners that can generalize across tasks they weren’t explicitly trained on, chain actions across multiple systems, and — critically — fail silently. The Amazon Kiro incident didn’t produce an error code. It produced an outage. The FDA’s January 2025 draft guidance on AI for regulatory decision-making in drugs and biologics is wrestling with exactly this issue: how do you establish model credibility in a submission when the model’s reasoning isn’t fully traceable? The EU AI Act’s high-risk classification for AI in clinical settings — with enforcement deadlines arriving in August 2026 — puts the same question into regulatory teeth.

What neither document answers is the organizational design question underneath it: who owns the accountability when an AI agent reasons correctly about the wrong thing?

The answer can’t be “the person who configured the permissions” — which is what Amazon effectively said about Kiro. That framing treats agent governance as a settings problem. It’s an architectural problem. The control plane for AI-first IT has to be rebuilt, not reconfigured.

In practice, the governance architecture I’ve found most useful organizes around autonomy levels: explicit, enforced constraints on how much independent action an agent can take before requiring human review — not as a policy document, but as a hard architectural constraint. You don’t give an agent production access the same way you don’t deploy a junior analyst on a high-stakes audit without a reviewer. Trust is extended incrementally, as the system proves it. That principle didn’t come from an AI framework. It came from 30 years of GxP practice in regulated manufacturing.

What BMS Understood That Most Organizations Haven’t

Greg Meyers’ quote at the BMS announcement deserves to be taken seriously as an architectural statement, not just a press release: “Most enterprise AI stops at the chatbot. The real prize is the untapped value still trapped behind decades of data silos.”

That’s an accurate diagnosis of where most life sciences organizations are stuck. The generative AI wave of 2023 and 2024 produced impressive pilots — McKinsey found that while nearly every large pharma company had run AI experiments, only about a third had scaled any use case beyond the pilot stage. The failure mode wasn’t the model. It was that the organizational substrate — the data, the governance, the workflow redesign, the accountability model — hadn’t been built to hold production AI.

BMS’s decision to build three years of structured AI infrastructure before deploying agentic capabilities isn’t caution. It’s strategy. The value of agentic AI isn’t in the agent. It’s in the institutional knowledge the agent can access, reason over, and activate. If that knowledge is siloed, inconsistent, and ungoverned, all you’ve done is give a powerful tool a bad map.

Gartner’s survey of over 700 CIOs, conducted in July 2025, found that by 2030, CIOs expect zero IT work to be done by humans without AI — 75% by humans augmented with AI, 25% by AI alone. But Gartner’s own data shows 72% of CIOs report their organizations are breaking even or losing money on AI investments today. The gap between that 2030 projection and today’s reality is where the governance work lives.

The organizations that will look like BMS in 2030 are making decisions about data governance, agent accountability, and workflow redesign right now — before the outage. Before the FDA inquiry. Before the board asks who was accountable for the AI decision that went wrong.

The question worth sitting with: in your organization, have you answered those questions before the agent has the keys? Or are you counting on the permissions being set correctly?

I work at the intersection of data architecture, AI strategy, and enterprise governance in rare disease biotech — where the downstream consequence of getting this wrong is measured in patients and regulatory submissions, not velocity. If you’re building AI-first infrastructure in a regulated environment and want to compare notes on what’s actually working, I’d welcome that conversation.

What’s your organization’s hardest governance problem right now in the AI transition — data readiness, agent accountability, or something else entirely? Drop it in the comments.

References

1. Amazon Kiro December 2025 Production Outage — Gizmodo
2. Amazon March 2026 Retail Outage — Wharton AI & Analytics Initiative
3. Amazon Kiro Governance Deep-Dive — ruh.ai
4. Fortune: AI Coding Risks in Enterprise — March 2026
5. Gartner IT 2030: How CIOs Should Reinvent IT for the AI Age — Webinar
6. Gartner IT 2030: Reinvent IT With AI for Long-Term Success — Research
7. BMS–Anthropic Claude Enterprise Announcement — BMS Newsroom, May 2026
8. BMS–Anthropic: Pharmaceutical Commerce Analysis
9. Gartner: 40%+ of Agentic AI Projects Will Be Canceled by 2027 — June 2025
10. ZS CDIO Outlook Survey: Scaling AI in Pharma 2026
11. Benchling 2026 Biotech AI Report
12. IntuitionLabs: Pharma AI Skills Gap — Biotech 5/50 Models Case
13. IntuitionLabs: Why 95% of Pharma AI Pilots Fail to Reach Production
14. FDA: Artificial Intelligence for Drug Development — CDER
15. FDA Draft Guidance on AI for Regulatory Decision-Making — January 7, 2025
16. EU AI Act Enforcement & Life Sciences — Sakara Digital, April 2026
17. Gartner Survey: AI Will Touch All IT Work by 2030 — October 2025

#AIStrategy #EnterpriseIT #AgenticAI #DataGovernance #AIFirstIT #Biopharma #LifeSciences #CIOLeadership #DigitalTransformation #RegulatoryIntelligence