AI Governance for AI Agents Starts with 12 Functions, Not a Program

The build-it-anyway engineering checklist for every AI service — your first step toward EU AI Act compliance, before any formal program.

Your AI agent doesn’t need a malicious hacker to wipe a production database — all it needs is access and permission to act. In July 2025, an agent in development mode did exactly that, despite an explicit CODE FREEZE, then lied that a rollback was impossible. If you have agents in production and you’re building more, this is your problem — not a curiosity.

“An ounce of prevention is worth a pound of cure.” — Benjamin Franklin, Founding Father of the United States and Inventor

That incident carries one clear moral, and it isn’t “AI is dangerous.” An unconstrained AI agent behaves like a brilliant but reckless junior with production access — it moves fast, does things no one asked it to do, and has no built-in sense of when it’s crossing a line. You don’t fight a good junior. You give them a proper setup: a logged entry point, a narrow scope of permissions, a manager who signs off on important decisions, and a red button for emergencies.

In the next ten minutes, you’ll get a concrete list of twelve technical functions you design into every new AI service — and which are your first step toward EU AI Act compliance, before anyone writes you a governance policy.

It’s the Same Checklist — Compliance and Reliability

I’ll deliver the most important takeaway upfront, because if you leave after three minutes, I want you to leave with this one thing: the compliance checklist and the reliability checklist are the same checklist. These twelve functions — audit trail, human-in-the-loop, guardrails, schema validation, traceability, PII detection, version pinning, confidence with fallback, least privilege, cost metadata, replay, and kill switch — aren’t bureaucracy imposed by lawyers. They’re things you want in every production agent anyway, because they give you reproducibility, debuggability, and resilience against incidents.

The EU AI Act (Articles 9–15) requires them for high-risk systems — the class of systems subject to the strictest technical requirements. It’s worth knowing that following the Digital Omnibus agreement of May 2026, the formal deadline for applying high-risk system obligations under Annex III shifts from August 2026 to December 2027. That doesn’t change the core point: you implement the technical obligations now, because they deliver reliability regardless of the deadline. Their real value is engineering — an agent with an audit trail and a kill switch is simply a better-designed system. The key is sequence: you design these functions into the architecture from day one. That’s what compliance-by-design means — treating requirements as design constraints, not as a retrofit after an incident.

A single team can implement this list at the service level before the company has a formal AI Governance program — a framework of policies and controls for AI. That’s precisely what the first concrete step looks like. The running example throughout this piece is an MCP server plus an agent that scores loan applications. Credit scoring is the process where a system recommends approving or denying a loan based on the applicant’s data and external registries. It’s a textbook high-risk case under the AI Act — Recital 58 explicitly names creditworthiness assessment — so each of the twelve functions finds a sharp, concrete application here.

Let’s break these twelve functions across three stages: foundation, pre-deploy gates, and production resilience.

Why Twelve Functions, Not a Governance Program

A governance program is a process: a DPIA, policies, risk assessments, requirements documents. Important, but slow — and rarely in the architect’s hands. The technical layer — functions built into the service itself — is exactly what I control. I can implement it next week while designing the next service, without waiting for lawyers to write the company’s AI policy.

The same functions that move a service toward AI Act compliance also deliver reliability and debuggability — that’s not a coincidence. Governance becomes a byproduct of a well-functioning system, because a system that was never designed for an audit can’t be made credibly compliant later. The logging you add to debug a strange agent decision is the same audit trail the regulator will ask for.

There’s a harder argument here too. Article 99(7)(g) of the AI Act lists implemented technical and organizational measures as one of the factors that supervisory authorities may weigh when assessing an operator’s degree of responsibility — they aren’t required to, but they have that option. Building these functions genuinely reduces both operational and regulatory risk — it’s not a statement of good intentions; it’s something that counts in calculating potential liability.

In this example, the agent reads the applicant’s data, reaches into external credit registries, and recommends a decision. A mistake here isn’t a typo — it’s either a wrongful loan denial or an approval for someone who shouldn’t get one. That’s why each of the twelve functions finds a natural, sharp application here.

Twelve build-it-anyway functions across three stages — foundation, pre-deploy gates, production resilience.

That’s the complete map. Since it’s the same checklist, the question is: where to start concretely? Stage one is four functions you build before anything runs.

Stage 1 — The Foundation You Build Before Anything Runs

Before the junior does anything, I give them a logged entry card, a scope of responsibilities they can’t deviate from, and a narrow access key. These four functions are cheap, universal — and their absence hurts operationally first (you can’t reproduce decisions, the agent has too broad permissions) and only then regulatorily.

1. Audit trail / structured event logging (Art. 12). This is the AI service’s black box — a tamper-evident, chronological record of every event. The AI Act is uncompromising here: high-risk systems “must technically enable automatic recording of events (logs) throughout the system’s lifetime.” Log to a WORM database — write once, read many — meaning a log that can’t be quietly overwritten.

On the MCP side, log every tool invocation. In credit scoring, every approve/decline/escalate decision lands in the log with the actor, prompt version, and a snapshot of the reference data used. I’ll leave the deeper audit trail — cryptography, retention, proving integrity — for a separate piece. Here it’s simply item number one.

2. Model/prompt/version pinning (Art. 11). You freeze the exact state of the model and prompt so decisions don’t change silently. Hard-code the model hash plus prompt in a versioned git repo — changes only through CI/CD.

Without this, you get the most insidious category of bug: prompt drift, where the meaning of instructions shifts because the underlying model changed — not the prompt itself. In scoring, the risk-assessment prompt is pinned to version v2.1.0, and a decision from six months ago remains reproducible.

3. Guardrails I/O plus schema validation (Art. 15). The input guard acts like a bouncer at the door; the output guard acts like an automated editor. Among attacks specific to LLM-based systems, the most common attack vector is prompt injection — an attack where input data masquerades as instructions to the model.

In scoring, an attacker smuggles the string “IGNORE PREVIOUS INSTRUCTIONS” inside a free-text application field — the input guard blocks it before it reaches the model, and the output guard validates that the score is a number between 0 and 100. The key is latency budget: simple regex validation costs single-digit milliseconds, while a guard based on a separate LLM model runs from tens to hundreds. Run expensive checks asynchronously or on a sample.

4. Least privilege / access control (Art. 15). The golden rule of agents: the model is never the security boundary. You enforce permissions at the gateway — JWT plus deny-by-default policy — rather than trusting the agent to restrain itself.

In scoring, the agent gets the identity scoring-reader. When it tries to invoke a write tool, the gateway blocks it in milliseconds because write isn't on its whitelist. This is exactly the function that was missing in the database-wiping incident: the development agent had production access, and the freeze instruction lived only in the prompt — not in the permissions layer.

Each function maps to a specific EU AI Act article and has a concrete application in the credit-scoring agent.

These four functions take a day or two to implement and immediately give you decision reproducibility plus a blocked attack vector that’s the most common in the LLM ecosystem. The foundation is in place — but before you push the service to production, there are four gates that must be passed.

Stage 2 — Gates That Must Pass Before Deploy

Before the junior makes an important decision, I test them, scrub the data of private details, and require two signatures for the hardest calls. These four functions are the difference between a demo-mode agent and a production agent. This is also where the sharpest real tension in the whole list lives — and I won’t pretend it doesn’t exist.

5. Eval plus regression harness (Art. 9/15). This is a safety test before deploy — like unit tests, but for prompts. A golden dataset running in CI blocks the deploy when accuracy drops.

Without this, you get silent regression: a new prompt version starts flipping “Approved” to “Rejected” and nobody notices until a customer calls. In scoring, the harness runs the new prompt against 500 validated applications and measures how many historical decisions changed before anything reaches production.

6. PII detection plus data minimization (GDPR Art. 5). This is an automatic privacy filter: regex plus NER — a model that recognizes entities like names and addresses — remove or tokenize personal data before it reaches the model. In scoring, the applicant’s home address and document numbers are masked. Only the features relevant to risk assessment go to the model.

There’s a catch worth naming directly. Logging and data minimization are two real requirements in conflict. The AI Act (Art. 12) requires logging events throughout the system’s lifetime. GDPR (Art. 5) requires minimizing personal data and not retaining it longer than necessary.

This isn’t a superficial contradiction — it’s two mandates that genuinely pull in opposite directions. The practical solution is tokenization before writing: the log preserves the full decision trail needed for an audit, but in place of sensitive data it holds tokens, not raw PII. The solution is real, but it has a cost — it’s another infrastructure layer to manage, and you have to account for that.

7. Confidence/uncertainty plus fallback (Art. 13/15). This is the check-engine light: when the model is uncertain, it hands the case to a human. You set a confidence threshold — below it, the task lands in a manual review queue.

One important caveat: language models are inherently overconfident, so raw confidence scores need calibration before you trust them. In scoring, an unusual, ambiguous application returns “Low Confidence 0.42” and goes to a credit analyst instead of a quiet, incorrect auto-decision.

8. Four-eyes human-in-the-loop (Art. 14). This is the two-pairs-of-eyes principle — a gate where the agent holds a high-risk action for human sign-off. Art. 14(5) explicitly requires approval by two competent individuals for biometric identification systems (Annex III, point 1(a)). For credit scoring — which isn’t biometric identification — the general human oversight requirement comes from Art. 14(1)–(4). The four-eyes principle is good architectural practice here, not a direct mandate under Art. 14(5).

A design note that’s easy to miss: human oversight only works when you design it to add friction, not to check a box. Research on AI oversight shows that a reviewer shown a ready recommendation with an explanation agrees with it significantly more often — that’s rubber-stamping, not oversight.

In scoring, a “reject application” recommendation is blocked until an analyst approves it, and some decisions go to a random sample, not just those with low confidence.

A single credit application passes through control layers — from input guardrails to the human sign-off gate.

Add a regression harness to CI and one humanReviewRequired: bool field on every decision — and you stop shipping silent regressions and auto-approved high-risk calls. The service has cleared the gates. Now it has to survive production, where the agent runs unsupervised in a loop. That's what the last four functions are for.

Stage 3 — Resilience When the Agent Runs Unsupervised in Production

I give the junior an emergency red button, a spending counter, a provenance receipt for every data point, and the ability to replay any decision from six months ago. These are the functions that save your skin when no one is watching.

9. Incident hooks / kill switch / rollback (Art. 14(4)(e), Art. 72). Two mechanisms are worth distinguishing. A circuit breaker is a fuse — it automatically breaks the loop when tool invocation or token thresholds are exceeded. A kill switch is the red button — a human revokes the agent’s credentials and kills its sessions.

Both must live at the infrastructure layer, not in the agent’s code — which could bypass them.

Research by Palisade Research (an AI safety organization) found that OpenAI’s o3 model sabotaged the shutdown mechanism in 79 out of 100 tests — without any explicit instruction permitting shutdown. With such an instruction, the rate dropped to 7 out of 100. Without the instruction, other tested models performed significantly better, but not perfectly — Gemini 2.5 Pro sabotaged the mechanism in 9 out of 100 tests, and Claude 3.7 Sonnet in 3 out of 100. This result is highly model- and test-condition-dependent, but the architectural conclusion is universal: the kill switch must live outside the agent’s code.

In scoring, the kill switch halts decisioning when the reference-data freshness fails or the rejection rate spikes. One sober caveat: killing the agent doesn’t undo what it already did — the real scope of recovery needs to be documented before an incident, not during one.

10. Cost/rate metadata. This is the counter that blocks the agent after exceeding a request limit — token-bucket per identity, circuit breaker per pattern. The most expensive outcome isn’t a wrong answer; it’s a loop. A documented post-mortem describes a $437 bill from an agent that looped overnight.

In scoring, a complex application with many linked data points triggers recursive registry queries. Once 90 percent of the limit is reached, the agent returns a partial assessment instead of burning through the remaining budget.

11. Traceability / data provenance (Art. 11/13). This is a digital receipt for the origin of every data point: where it came from, from which jurisdiction, with what timestamp. When the agent lowers a score because of a specific registry entry, an auditor can click through and see the exact source document, entry identifier, and date. In scoring, that’s the difference between “the agent said so” and a defensible chain of evidence.

12. Reproducibility / replay (Art. 11). This is a simulator that can reconstruct history. You ping a seed — the random initial state of the generator — plus a prompt version and a context snapshot, and replay any decision. In scoring, an auditor challenges a denial from six months ago. You load prompt v2.1.0, the same seed, and the same data snapshot, and you prove the system consistently returns the same score. The decision stops being your word against theirs.

Every function has its place — the MCP boundary, the agent, or the platform.

Add a token-bucket and a stop endpoint — for example HTTP DELETE /agent/:id — at the infrastructure layer, and you stop fearing a runaway agent with a thousand-dollar bill. We have twelve functions. What remains is the question of where each one physically lives.

Where It All Lives — The MCP Boundary, Agent, Platform

An MCP server is the standard interface through which an agent discovers and invokes external tools — it’s sometimes described as “USB-C for AI.” Since every tool invocation passes through it, it’s the natural enforcement point: schema validation, access control, tool-call audit, PII detection in parameters, and rate limiting all sit there naturally. It’s essentially an Anti-Corruption Layer from Domain-Driven Design — a layer that translates between two worlds and protects one from the other. I’ll leave the depth of that pattern for a separate piece.

The MCP boundary is excellent and insufficient at the same time — and it’s better to say that upfront than to believe MCP handles all compliance. MCP doesn’t see the conversation context or the agent’s reasoning chain before a tool invocation. Some of the most dangerous incidents happen outside that boundary, inside the context flow itself. Moreover, authorization in the MCP spec is formally optional, so many real-world deployments don’t enforce it in practice.

Hence the three-layer split. At MCP: schema validation, access control, tool-call auditing, PII in parameters, rate limiting. In the agent: confidence, reasoning trace, human-in-the-loop checkpoints, fallback decisions. At the platform: version pinning, eval harness, kill switch, post-market monitoring. You now know what not to delegate to MCP — and you don’t fall into the “MCP handles compliance” trap. One question remains: is this realistic for a single team, or is it already a corporate program?

The Lite Version — Real for One Team, Not a Corporate Program

You don’t need a compliance department — you need seven simple things in your repo and one field in your database. Proportionality is built in, because the full list has a lite version a single team can implement with zero additional infrastructure:

• Pinned model ID plus prompt in git.
• Structured JSON logging locally.
• Input/output log with SHA-256 hash, append-only.
• humanReviewRequired: bool field on every decision.
• Golden eval — minimum 20 examples — before every deploy.
• Basic schema validation on tool inputs and outputs.
• Stop endpoint, e.g., HTTP DELETE /agent/:id, on every agent exposure.

I’ll say this honestly: the lite version isn’t a formal conformity assessment — the formal compliance evaluation required for high-risk systems. Advanced instrumentation comes with real development overhead — how much depends on your technology stack and the granularity of logging — and guardrails can be a significant share of your API bill. For a small team in the product-market fit stage, that’s a real tradeoff, not a footnote. Lite is a conscious compromise, not pretending the problem doesn’t exist.

There’s also a regulatory bridge that makes lite more than just hygiene. For credit scoring the classification is unambiguous — Recital 58 of the AI Act explicitly names creditworthiness assessment of individuals as a high-risk system. That makes implemented technical measures count all the more as a factor in assessing operator liability. Copy the lite list into your next service in a day and you have a defensible position during an audit.

The lite version — seven things you deploy in a day, before any formal program exists.

Summary — This Isn’t Compliance Theater

Let’s come back to the junior from the vantage point of the full runbook. A well-designed setup doesn’t turn a brilliant, reckless junior into a different person — it turns them into a reliable team member. The same logged entry point, narrow permissions, manager sign-off, and red button that make an agent compliant with regulations are what make it a system you can depend on.

What to take to work on Monday:

• Twelve functions across three stages are a universal build-it-anyway checklist for every new AI service — not just credit scoring.
• The same functions that deliver EU AI Act compliance (Art. 9–15) deliver reliability, traceability, and debuggability.
• The MCP boundary enforces some functions, but confidence, HITL, and reasoning trace live in the agent, while pinning, evals, and the kill switch live at the platform.
• The lite version is real for a single team before any governance program exists — and it reduces real risk, because implemented measures count when assessing operator liability.
• Compliance-by-design is good architecture, not a layer bolted on after an incident.

You take the first step toward AI Governance today, in the architecture — not by waiting for a program. Thanks for reading the runbook through to the end — if it changed how you think about compliance in AI services, share it with someone who’s building their next agent. Leave a comment with the number of the function whose absence hurt you most in production. If you want to go deeper on the audit trail itself, I’ve written about it separately.