Governed on Paper, Ungoverned in the Loop

How AI governance became an assurance artifact while physical risk moved into the real-time control loop.

The control room does not look like a place where a catastrophe would begin, and that unremarkable quality is the first thing worth attending to. The room is quiet in the way that only heavily instrumented rooms are quiet, its calm underwritten by hundreds of processes that have been delegated, over years, to systems that no longer ask permission before they act.

A single operator sits before a wall of screens on which an optimization model, trained to hold a physical system inside its safe and efficient envelope, is making decisions at a cadence no human hand could match. It adjusts a setpoint here, rebalances a load there, throttling, rerouting, and smoothing, doing all of it faster than the operator can read the log that records it. On one of the screens, a small green indicator reports that human oversight is active. The operator, who has watched this system perform flawlessly for months, and who has learned to trust the green indicator more than his own attention, is in the narrowest and most literal sense in the loop. In every sense that would matter during an incident, he is not.

This is the situation that AI governance in critical infrastructure is supposed to address, and it is worth stating plainly, before the frameworks and acronyms arrive to complicate it, that the dominant AI governance model does not yet reach the moment described above.

It reaches the moments on either side of it.

It reaches the procurement decision, the risk assessment, the conformity declaration, the annual audit, and the post-incident review. It arrives before the operator sits down and it arrives after he stands up, but it does not arrive during the seconds in which the system acts and the physics commits. Those seconds are where the abstract risk becomes a physical event.

The Two Layers of Governance

Governance operates through two distinct but interdependent layers, and the entire difficulty of this subject lives in the space between them.

The first layer is governance through evidence. In this layer, governing a system means establishing and documenting the decisions, responsibilities, and assurances that demonstrate the system was built and deployed responsibly, assessed against a recognized standard, and accompanied by the artifacts that a regulator, auditor, or insurer would expect to find, and that a court might later examine. Model cards, data sheets, risk registers, conformity assessments, impact analyses, and attestations of human oversight all belong to this layer. They represent real work and often excellent work, answering a central question: can the organization show that it took AI risk seriously? What they do not do, and were never designed to do, is act.

The second layer is governance through control. In this sense, governing a system requires translating policy, authority, and risk tolerance into live mechanisms placed in the operating path. Those mechanisms constrain what the system is permitted to do while it is doing it. They intervene within the response time the physical process requires, bringing the operation to a safe state before it exceeds its tolerable envelope. A relay that trips a breaker, a safety instrumented function that closes a valve, a supervisory limit that a controller physically cannot exceed, and a fail-safe that defaults to a defined safe condition when confidence collapses all belong to this layer. They do not merely record a governance decision. They make it executable.

Control without evidence cannot demonstrate why limits were selected, who authorized them, or whether they remain adequate. Evidence without control cannot enforce them when the system acts. Producing governance evidence does not mean the organization has also made its limits executable.

That assumption becomes most dangerous precisely where the two look most alike: in the attestation of human oversight. This phrase appears across many contemporary AI governance and regulatory instruments, reads on paper as an active control, and can turn out, when the system moves faster than a human can follow, to be only evidence. The green indicator is evidence of assigned oversight. It is not evidence that effective intervention remains possible.

The Oversight Latency Gap

I want to give the mechanism a name, because naming it makes it possible to see it recurring across sectors that otherwise appear to have nothing in common. I will call it the oversight latency gap, by which I mean the widening distance between the speed and authority with which an AI system acts inside a physical process and the time, information, comprehension, and authority available to the human operators and institutional mechanisms responsible for intervening.

At the operational level, the gap is not just a metaphor. It can be measured in the interval between system action and effective intervention, expressed in milliseconds where the system is a controller and in months where the relevant mechanism is institutional governance. The failure mode this essay describes is what happens when a decision capable of producing physical consequences falls inside that interval, where the AI can act before those charged with oversight can detect, understand, authorize, and execute an effective intervention.

The gap has always existed wherever automation touched physical systems, and the disciplines that manage physical risk have understood it for decades. That is why a chemical plant does not rely on an operator noticing a runaway reaction and typing a command, but on an instrumented function designed to act within a specified response time regardless of whether anyone is watching. What is new is not the gap but the character of the thing now placed inside it.

A conventional controller is built around logic that can be specified, inspected, tested, and bounded, even though no complex engineered system is free from unanticipated interactions or design error. A learned model presents a different assurance problem. Its behavior emerges partly from training data and optimization rather than solely from logic written in advance, making its operating envelope harder to specify exhaustively and its unsafe responses harder to anticipate before deployment. Its most dangerous errors can be the confident ones, outputs delivered without a reliable operational signal of the uncertainty beneath them.

We have taken a class of component whose behavior cannot be anticipated exhaustively and inserted it into systems where an unsafe action can become physically irreversible before anyone understands what happened, then relied on assurance processes calibrated to a slower and more legible world to govern the result.

The Governance Gap Cascade

The failure does not happen all at once, and it does not happen because anyone behaved recklessly. It happens because a sequence of individually reasonable moves composes, over time, into a system that is governed on paper and ungoverned in the loop.

I call this sequence the governance gap cascade. Like every cascade worth the name, its stages are causally linked, each one producing the conditions for the next, so that an organization can pass through all five while believing, at every step, that it is doing responsible work.

Stage One: Assurance Substitution

Under the combined pressure of a competitive imperative to deploy and a regulatory imperative to comply, the organization procures and fields its AI capability against a documentation standard. The artifacts required by that standard become, by a subtle and rarely noticed substitution, the working definition of what it means for the system to be governed. The risk assessment is completed, the conformity declaration is signed, the human-oversight box is checked, and the organization now possesses, in a literal and defensible sense, the evidence expected of a governed system. The dominant incentive was not to prove that the system could be constrained at runtime. It was to satisfy the assessment, and the organization treated the evidence requested by that assessment as the boundary of the work.

Stage Two: Autonomy Creep

A system fielded as an assistant, bounded and advisory and closely watched, expands its effective operational authority through increments that do not trigger a meaningful reassessment. Each increment is justified by the system’s excellent performance, reauthorization is expensive, and the increment feels minor. Recommendation hardens into action. Optimization within a supervised range becomes control across an unsupervised one. The human meant to approve each decision comes to approve batches, then to approve exceptions, then merely to be notified.

The taxonomy that the autonomy community uses for this progression, moving from human-in-the-loop to human-on-the-loop to human-out-of-the-loop, describes a drift that almost never announces itself.

SAE International, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (J3016), together with Sheridan, T.B., Humans and Automation.

No single step feels like a decision to remove human control. The system’s real authority quietly outgrows the envelope its assurance case was written for, while the evidence remains anchored to the system that was originally approved. The failure is not merely that the software changed. It is that no governance or change-control process treated an increase in operational authority as a material change to the safety case.

Stage Three: Oversight Collapse

This is the stage at which the latency gap stops being latent. The moment arrives when the system encounters a situation its training did not cover, acts on a confident error, or processes a corrupted input. It does so at the speed at which it does everything, and the human whose oversight was attested cannot intervene inside the window that matters.

Sometimes the window is too short in an absolute sense, a matter of a second or a fraction of one. Sometimes the window is long enough in principle, but the operator’s attention has decayed across months of flawless performance into automation complacency, the predictable degradation of vigilance and active monitoring that can follow long periods of reliable automated performance.

Parasuraman, R., and Riley, V. (1997), “Humans and Automation: Use, Misuse, Disuse, Abuse,” Human Factors.

Sometimes the interface hands control back to the human at the last moment with a time margin that would suffice for a machine but fails for a person. In every version, the structural result is the same. Oversight that appeared viable on paper proves ineffective in operation, and the explanation arrives after the physical process has crossed the point at which intervention could still have prevented the outcome.

Stage Four: Physical Consequence

This stage distinguishes safety-critical physical systems from domains in which an AI error does not directly alter the physical world.

A setpoint moves and a chemical concentration climbs. A false condition triggers protective isolation and a section of the electrical grid goes dark. A valve opens and pressure rises beyond tolerance. A dose is delivered, a turbine exceeds its operating envelope, or a physical asset enters a state from which recovery is no longer immediate. The digital error crosses directly into the physical layer, where its consequences are measured in outages, contamination, damaged capital assets, and harm to human life. Recovery may still be possible. Reversal is not, because the physics has already happened.

Stage Five: The Accountability Vacuum

In this final stage, the organization discovers what kind of governance it actually had. The incident is over and the questions begin, focusing on the very issues that effective runtime control might have resolved and documentary evidence alone cannot. Who decided? On what basis? Could anyone have stopped it, and if so, who possessed the authority and how much time did they have?

The audit trail, built to demonstrate compliance, reconstructs that the administrative process was followed. It may still be unable to reconstruct why the system acted as it did, because the available telemetry can show what the system produced without providing a sufficiently legible causal account of the decision. Accountability diffuses across a chain where every link points to another: the operator to the model, the vendor to the configuration, the configuration to the regulatory framework, and the executive to the assurance file proving the organization met every requirement. Regulators and insurers examining that file may conclude that the organization possessed a functioning assurance process without possessing an effective runtime control system. The trust accumulated over years collapses in an afternoon, leaving the organization less capable of deploying AI responsibly than it was before. The organization discovers that one layer of governance remained intact while the layer capable of constraining runtime behavior was missing.

What the Physical World Does to a Mistake

The cascade can be read in incidents that already exist, provided one reads them for the right thing. That thing is not the specific technology that failed, but the relationship between the speed of an automated action and the reach of human intervention. One must also be careful, as the discipline of this subject demands, to classify each incident precisely rather than press it into a shape it does not have.

Consider the water treatment plant in Oldsmar, Florida, on a Friday morning in February of 2021. An intruder gained remote access to the supervisory control workstation and changed the setpoint for sodium hydroxide, the compound that regulates acidity and is, in concentration, a poison, from 100 parts per million to 11,100 parts per million. This was more than a hundredfold increase to a level that would have made the water dangerous.

Florida Department of Health, Incident at Oldsmar Water Treatment Plant (2021); U.S. Cybersecurity and Infrastructure Security Agency (CISA), Oldsmar Water Treatment Facility Cyber Intrusion (2021).

This was not an AI incident, and I will not pretend that it was. It was a cyber intrusion into an industrial control system. It belongs in this essay for a different and more uncomfortable reason: what prevented the altered setting from becoming an operational consequence.

A human being happened to be looking at the screen, noticed the cursor moving on its own, and reversed the setting before it could affect the treatment process. In contemporary governance terms, he was the human in the loop.

The lesson of Oldsmar is not simply that human oversight works. It is that the intervention window remained long enough for a person to detect and reverse the action, with automated alarms still standing behind that person as a second line.

Florida Department of Health (2021); CISA (2021).

At Oldsmar, human oversight worked at the speed of a moving cursor. We have quietly assumed the same arrangement will remain effective as system tempo moves toward machine speed.

Consider then the case that does not flatter us: the developmental automated driving system operated by Uber’s Advanced Technologies Group that struck and killed a pedestrian in Tempe, Arizona, in March of 2018. This incident was investigated in unusual depth by the National Transportation Safety Board and is therefore unusually legible.

Here the system was a genuine machine-learning perception and control stack. The human in the loop was a trained safety operator seated behind the wheel with the specific job of intervening, yet the intervention did not come in time. The board found that the system detected the pedestrian approximately 5.6 seconds before impact but repeatedly changed its classification of her and failed to predict her path accurately. The vehicle’s factory automatic emergency braking had been disabled during automated operation. The system was designed to suppress its own braking for a full second while it alerted the operator and handed control back, a design choice intended to suppress potentially unnecessary severe maneuvers, but one that also consumed a critical second of the remaining intervention margin. The operator, whose attention had drifted from the road, began to react less than a second before the collision.

National Transportation Safety Board, Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian, HAR-19/03 (2019).

The board identified automation complacency as a contributing factor and located the deeper organizational failure in a safety culture that had not built adequate countermeasures against predictable human-performance limitations. Tempe is the oversight latency gap in its purest and most tragic form: a human nominally in control who, at the system’s operating tempo and within the intervention window it left her, could not act effectively in time.

Consider finally an analogous case from financial markets: the collapse of Knight Capital on August 1, 2012. This was not an AI failure, and it was not a physical infrastructure incident, but it demonstrates the latency problem with a clarity the AI cases have not yet had occasion to match.

A flawed software deployment reactivated dormant code in the firm’s automated trading system. Over roughly 45 minutes, that system sent millions of unintended orders into the market and accumulated billions of dollars in unauthorized positions. It did so while the firm remained connected to live markets and while its own engineers, watching the losses mount, could not diagnose and halt the behavior before the losses compounded.

When it was over, the firm had lost more than $460 million, several times its annual earnings.

U.S. Securities and Exchange Commission, Knight Capital Americas LLC Agrees to Pay $12 Million for Market Access Violations, Release №2013–222 (2013).

Knight demonstrates that the oversight latency gap does not require a learned model to threaten the survival of an institution. It requires only an automated system fast enough that human comprehension lags its action, connected to a domain in which decisions become costly or irreversible before comprehension arrives.

What AI adds is not the gap, which was already there, but a component whose behavior is harder to anticipate and whose errors are harder to see coming, operating inside the same unforgiving interval.

The Human Alibi

There is a person inside this failure mode whose position deserves more moral attention than it usually receives. It is the operator, control-room engineer, or plant safety officer whose name and signature convert the organization’s claim of human oversight into an assigned human responsibility.

The problem is not the assignment of human responsibility itself. It is the assignment of responsibility without the corresponding information, authority, and intervention window required to exercise it.

This person may be asked to carry accountability for a system that, at the tempo it operates, he cannot actually control, and to sign, periodically, a document affirming that he can. When the operating arrangement has outrun the operator’s practical authority, he may know, because he sits with the system every day, that the green indicator describes an oversight capacity he does not truly possess.

When the system’s decisive action occurs faster than he can intervene, his practical role can collapse into being the name on the record afterward.

The operational structure has quietly converted him from a controller into an alibi.

He is the human the organization points to when the regulator asks who was overseeing the system. He is the human the vendor points to when the model’s behavior is questioned. In the arrangement as it actually functions rather than as it is described, he becomes a place for responsibility to come to rest before it reaches the leaders who designed the incentives, while the model itself remains incapable of bearing accountability.

The condition this produces has a name in the literature, and it is not burnout, though burnout often accompanies it. Jonathan Shay, writing from his work with combat veterans, described moral injury in terms of a betrayal of what is right by a legitimate authority in a high-stakes situation.

Jonathan Shay, Achilles in Vietnam: Combat Trauma and the Undoing of Character (1994).

Organizational scholars have since carried the concept into the workplace, where it explains a kind of damage that exhaustion alone does not.

Dean, W., Talbot, S., and Dean, A., Reframing Clinician Distress: Moral Injury Not Burnout, Federal Practitioner (2019).

The operator asked to become the human alibi may be exposed to the conditions associated with this form of injury. The betrayal of what is right lies in being made responsible for what one cannot control. The authority implicated in the betrayal is the leadership structure that designed and ratified the operating model. The stakes are the highest there are, because the system he has been made responsible for may govern water, power, transportation, treatment, or another physical process on which human life depends.

Not every mismatch produces moral injury. The risk emerges when the operator recognizes the mismatch, is required to affirm or enact it, and later confronts consequences for which the organization assigned responsibility without control.

What breaks in a person so situated is not primarily his energy. It is his trust, his self-respect, and his capacity to believe that his signature means what it says. Organizations that institutionalize this arrangement risk accumulating a debt of moral injury among precisely the experienced operators they can least afford to lose.

The Regulatory Map and Its Blank Spaces

It would be a relief to report that the regulatory apparatus has closed this gap. The more honest report is that regulation, standards bodies, and sector authorities have begun, with genuine seriousness, to map it, but the map is not yet the territory.

The European Union’s AI Act reaches critical infrastructure through Annex III, which classifies certain AI systems intended for use as safety components in specified critical-infrastructure functions as high-risk. Those systems are subject to obligations involving risk management, data governance, logging, transparency, human oversight, accuracy, robustness, and cybersecurity.

Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III.

Following the political agreement on the AI Omnibus, the obligations for high-risk systems in areas including critical infrastructure are now scheduled to apply from December 2, 2027. For AI systems integrated into regulated products such as industrial machinery, the corresponding rules are scheduled to apply from August 2, 2028.

European Commission, AI Act Implementation Timeline (2026 update).

This is real and consequential law. It also leaves an operationally important classification boundary. The obligations attach to defined high-risk use cases, including certain systems used as safety components in critical infrastructure. An optimization system that contributes materially to safe operation may not always be classified in the same way as a formally designated safety component. The Commission’s 2026 draft classification guidance is intended to clarify such questions, but the boundary remains operationally important until that guidance is finalized and tested in enforcement.

European Commission, Draft Guidelines on the Classification of High-Risk AI Systems (Targeted Consultation, 2026).

In the United States, the National Institute of Standards and Technology has given the field one of its most influential organizing structures in the AI Risk Management Framework. Its four functions of govern, map, measure, and manage have become widely adopted terms in enterprise AI risk.

National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100–1 (2023).

The framework does not itself prescribe a maturity model by which an organization can grade the quality or depth of its implementation. That is not a failure of the framework, but it does not provide organizations with a common scale, within the framework itself, for distinguishing nominal adoption from operational maturity.

Recognizing that critical infrastructure needs more than the general framework, NIST released in April of 2026 a concept note for a dedicated profile on trustworthy AI in critical infrastructure. Aimed squarely at operators in energy, water, and transportation running AI across information technology, operational technology, and industrial control systems, it is intended to help those operators define their trustworthiness requirements and press them upstream to the vendors who build the models.

National Institute of Standards and Technology, Concept Note: AI RMF Profile for Trustworthy AI in Critical Infrastructure (April 2026).

It is directed at the problem this essay identifies, but, as of this writing, remains a concept note rather than a completed profile, while the deployments it aims to guide are already underway.

NIST’s related work on the cybersecurity of AI, the Cyber AI Profile issued as a preliminary draft in December of 2025 and built onto the Cybersecurity Framework, addresses the adjacent and vital problem of securing AI systems and using AI in defense. However, it is scoped to cybersecurity rather than to the real-time safety of AI in control loops, and it too is a draft.

National Institute of Standards and Technology, Cyber AI Profile, Preliminary Public Draft (December 2025).

The pattern across all of this is not negligence. It is lag. AI-specific regulation, standards, and guidance are developing in the correct direction and with genuine seriousness, but much of that work remains behind the deployments it aims to govern. It moves at the deliberate pace of standards development into an environment moving at the pace of commercial deployment. This is not the same interval as runtime intervention latency, but it has the same structure: deployment authority advances faster than the mechanisms intended to govern it.

What Actually Works

The temptation, having described a gap this comprehensively, is to conclude that the problem is intractable. The evidence points the other way, because one of the disciplines that AI governance in critical infrastructure most urgently needs to absorb already exists, has developed over decades, and provides a proven body of methods for reducing physical risk. Its central lesson is precisely the distinction this essay began with: governance that acts is different in kind from governance that documents.

The functional safety tradition codified in IEC 61508 and its sector adaptations provides the clearest established model for engineering automated systems toward defined and independently assessed risk-reduction targets.

International Electrotechnical Commission, IEC 61508: Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, Parts 1–7.

Several features of its method are directly instructive. It does not ask whether a system is safe in the abstract. It defines a specific safety function, assigns it a safety integrity level according to how much risk reduction the situation demands, and then requires that the function meet a quantified target for its probability of dangerous failure, supported across the safety lifecycle by verification, validation, and appropriately independent assessment.

IEC 61508–1 and IEC 61508–2.

It builds safety instrumented functions capable of acting independently of the primary control process, closing a valve or moving the process toward a defined safe state within a specified response time, whether or not a human is watching and whether or not the primary control system requests the action. It evaluates dangerous failure behavior, diagnostic coverage, independence, and whether the safety function can meet its required performance target.

This is a mature engineering expression of governance through control. Its relevance to AI is not that a learned model can be inserted unchanged into a conventional functional-safety argument. The more useful principle is architectural: physical safety need not depend on proving that the model will always behave correctly. The model can be permitted to optimize inside what I will call a deterministic enclosure, in which independently assessed safety functions constrain the physical envelope regardless of what the model requests. The probabilistic component is denied unilateral authority to commit an unsafe action, while the broader assurance case still addresses the model, its inputs, its integration, and the controls surrounding it. Assurance of the safe physical envelope rests on the deterministic enclosure rather than on confidence in the model alone. Deterministic does not mean infallible. It means that the protective limits, permitted actions, and defined safe-state responses are explicitly specified and independently testable rather than learned implicitly from data.

By enclosure I mean an architectural boundary, enforced independently of the learned model, that constrains the physical consequences the model can produce. Jose Spena (2026).

EASA’s developing AI assurance approach offers an operating-model expression of the same principle. Its concept papers organize applications by the degree of authority the AI holds, from systems that assist a human, to systems that cooperate under human oversight, to systems that operate with the human remote or absent. The trajectory supports a clear operating-model inference: as machine authority rises and immediate human authority falls, runtime monitoring, operational constraints, and automatic safeguards must increase to compensate.

The aviation assurance approach treats effective human oversight as a property that must be designed into the system through the interface, timing, alerting, and retained ability to override, rather than a capacity that can be assumed to exist merely because a human is present. It pairs this with a concept it calls learning assurance, an attempt to bring the training of the model itself inside the assurance argument rather than treating the model as an oracle whose outputs are trusted because the process around them was documented.

European Union Aviation Safety Agency, Concept Papers on AI Trustworthiness and Learning Assurance (2023–2025 series).

The field’s own practitioners are converging on the same principle from the direction of operations. The State of AI in OT Cybersecurity 2026, based on a survey of 302 operational-technology and industrial-control-system cybersecurity leaders conducted between April and July 2026, found that 87.7% of respondents were already using, evaluating, piloting, or planning AI for OT cybersecurity. Yet only 11.9% had formally mapped and reviewed which AI-driven decisions could directly affect physical processes, safety systems, or operational continuity.

Takepoint Research, The State of AI in OT Cybersecurity 2026 (July 2026).

The surrounding findings make the distance more consequential. Although 63.9% recognized physical-consequence risks as a top-tier or emerging concern, only 15.6% were actively enforcing an OT-specific AI governance policy. Another 34.8% still had such policies in draft, while 30.8% depended entirely on general enterprise IT policies that were not designed around kinetic process hazards. Nearly nine in ten respondents reported movement toward AI in OT cybersecurity, but fewer than one in eight had formally mapped where AI-driven decisions could produce physical consequences.

Jonathon Gordon, Directing Analyst at Takepoint Research, expressed the operating-model problem directly: organizations are interested in deploying AI but continue to struggle with implementing it safely, effectively, and with operational control.

Takepoint Research (2026).

He argued that the organizations likely to progress furthest will be those that expand AI use without granting it more authority than their controls, evidence, and operating models can support.

That is the argument of this essay rendered as operational evidence. The decisive question is not how much AI an organization has deployed, but whether the authority granted to that AI remains inside the control capacity the organization can actually exercise.

Closing that gap requires more than a better document. It requires an operating model in which the authority granted to an AI system is bounded by controls capable of acting within the required intervention window. The safe envelope must be enforced through mechanisms that do not depend on a human being remaining continuously attentive or reacting faster than human cognition allows. Oversight must be engineered rather than merely attested, and every increase in autonomy must be matched by a corresponding increase in executable control.

Guidance by Role

The people who read this will occupy different seats, and the gap looks different, demanding different action from each of them.

For the Chief Information Security Officer: The discipline is to stop treating AI in operational environments as a mere extension of enterprise security. Recognize that the consequence layer is physical. The model’s integrity, its inputs, and its authority to act are now safety concerns as well as cybersecurity concerns. The security program must actively partner with whoever owns functional safety rather than duplicating or ignoring them. The right question is not only whether the model and its environment are secure, but whether a compromised or mistaken model can request a physical action that no independent control would stop.

For the Chief Information Officer and Chief Technology Officer: The discipline is to make autonomy creep visible and attach an explicit control and assurance cost to it. Require that any expansion of a deployed system’s operational authority pass through the same scrutiny as its initial deployment. The assurance case written for an advisor does not cover a controller. The drift from one to the other is the exact mechanism by which governed systems become ungoverned without anyone deciding that they should. Require control engineering, functional safety, cybersecurity, and enterprise architecture to review any change that expands the system’s ability to alter a physical process.

For the Chief Risk Officer and Operational Risk Function: The discipline is to insist that the risk register explicitly distinguish between systems whose worst outcome is a bad recommendation and systems whose worst outcome is a physical event. Refuse to let the two be managed with the same instruments. Risk tolerances that may be reasonable for the first can be unacceptable for the second, and an aggregated AI risk score that blends them conceals the exact exposure that matters.

For the Compliance and Governance Officer: The discipline here is the hardest, because it requires resisting the substitution named at the outset. Treat the completed conformity file as the beginning of governance rather than its conclusion. Ask, of every attestation of human oversight, the practical operational question: could the human named in this document detect the problem, understand it, decide, and intervene with the information, authority, and time the system actually provides? Record the honest answer, even when it is inconvenient.

For Operational Technology and Plant Leadership: The discipline is to defend the authority of the independent safety layer against pressure to let optimizing models reach past it. Protect operators from becoming alibis by ensuring that whatever they are asked to attest is something they can actually execute in real time. An oversight role that cannot be performed is not a safeguard. It is a trap laid for the person holding it.

For the Board of Directors: The discipline, finally, is to ask a single question and keep asking it until the answer is coherent: does the organization’s ability to control its AI systems in operation keep pace with the authority it has granted them? Understand that an answer expressed entirely in the language of frameworks and completed assessments demonstrates governance through evidence. It does not necessarily demonstrate governance through control.

Conclusion

Seen from the altitude at which operating models become visible, the failure is singular and structural.

An organization adopts AI into a physical system under pressures that reward visible deployment and documented compliance. It governs that adoption with an apparatus built to produce evidence. The apparatus does its job, the evidence accumulates, and everyone involved behaves responsibly by the standard they were given.

Meanwhile, the system’s authority migrates by reasonable increments toward the point at which it acts faster than anyone can follow. One day, it encounters a situation it was not built for and commits, at machine speed, an action that no restoration of the prior digital state can undo. The documented governance never became executable governance. The accountability that should follow cannot find a place to land, and the trust that took years to build is gone by evening.

The current operating environment has made this dynamic increasingly visible. The development of high-risk AI requirements, the increasing scrutiny of nominal human oversight, and the demonstrated mismatch between machine-speed action and effective human intervention all point toward a single conclusion: static documentation cannot substitute for runtime control. The specific failure may not have been foreseeable, but the operating condition that allowed it to become physical was. The operating model shipped with that condition already inside it, embedded in the gap between the governance the organization could show and the control it could exercise.

The remedy is not less ambition, and it is not more paperwork. It is an operating model that closes the interval. It grants authority only as fast as it can build executable control to match. It enforces the safe envelope through mechanisms that do not depend on human speed or sustained attention. Oversight is engineered into the operating path rather than merely attested on paper. It does not wait for an incident, an audit finding, or an enforceable mandate to build the control architecture that physical safety already requires.

The operator is still in the control room. The system is still humming through its thousands of decisions. The small green indicator still reports that human oversight is active.

Whether that indicator is telling the truth is not a question the indicator can answer. It is a question about the operating model behind it.

Note: This essay reflects the regulatory landscape, standards, and publicly available guidance as of late July 2026. Regulatory instruments, draft standards, and sector guidance in this area are evolving quickly, and readers making decisions should consult the primary sources cited in their current form.

AI #AIGovernance #CriticalInfrastructure #OperationalTechnology #ICS #Cybersecurity #FunctionalSafety #RiskManagement #GRC #NIST #EUAIAct #OTSecurity #AISafety #Compliance #Leadership