Autonomy Is Easy to Sell. Control Is Hard to Design.

AI agents are usually sold with one promise: they can keep going after the human stops typing.

That is the attractive part.

It is also the dangerous part.

A chatbot waits. An agent watches, decides, calls tools, crosses application boundaries, writes files, sends messages, opens tickets, edits code, schedules meetings, and sometimes proposes actions before the user has fully formed the request.

The shift sounds technical, but the real question is not technical at all.

The question is control.

By late June 2026, that question no longer felt like a distant governance topic. The Five Eyes intelligence alliance warned that advanced AI models could reshape offensive and defensive cyber capabilities within months, not years.

Enterprise AI vendors are now talking about approval gates, audit logs, sandboxing, role-based access control, and agent identity not as optional compliance extras, but as basic deployment requirements.

Meanwhile, developers, maintainers, and platform owners are discovering a more ordinary truth: an agent that acts quickly can also move responsibility away from the person who should have owned the decision.

That is why the next AI-agent race may not be about who has the most autonomous system.

It may be about who can make the control surface boring enough to trust.

The wrong question is: how much can the agent do?

Everyone wants agents that can act without waiting for a human at every step. That is the pitch, the fear, and the engineering challenge packed into one sentence.

But asking “how much can the agent do?” is already the wrong starting point.

A more useful question is: what can the user still understand, stop, inspect, and reverse after the agent starts acting?

This is where AI agents become a product design problem rather than a model capability problem.

The old software boundary was relatively clear. A user clicked a button. An app performed an action. A log might exist somewhere. The user might not understand all the backend details, but the action still had a visible relationship to the interface.

Agents blur that relationship.

A proactive agent does not merely answer. It selects moments. It infers conditions. It may anticipate needs before the user has named them, or decide that a change is worth making before the user has said so explicitly.

If the system hides that process, the user is no longer driving the loop. The user is reacting to an interface that has already oriented itself.

That may be useful.

It may even be delightful.

It is also the moment where autonomy becomes a design liability.

The OODA loop gets weird when everyone has agents

Military language is useful here, even outside the military. The OODA loop — observe, orient, decide, act — assumes that faster observation and faster decision create advantage.

Agentic AI disturbs that assumption.

Defense One’s reporting on agentic AI and cyber operations captures the tension. If defenders can observe, test, draft, patch, and respond at machine speed, that matters. But attackers can also probe, adapt, generate variants, chain tools, and exploit weak systems faster.

The advantage does not automatically go to the side with more information. It goes to the side with better control over action.

This applies outside cybersecurity too.

In software maintenance, an agent can prepare a pull request. In customer support, it can draft a response. In sales operations, it can enrich a lead. In personal productivity, it can monitor a calendar, summarize email, and propose the next action.

These are not science fiction use cases anymore. They are becoming normal.

The advocate’s case is easy to understand. Human workflows are full of small stalls. People forget to follow up. They get tired. They skip boring maintenance work. They lose context between tools. Agents are attractive because they reduce friction.

I feel that pull myself. While experimenting with local LLM projects, code agents, scripts, and app boundaries, I have seen the moment when a tool stops feeling like a chatbot and starts feeling like a small operating layer. It can continue after I stop typing. It can hold a task across several steps. It can make the computer feel less passive.

That is precisely why the control question gets sharper.

A passive assistant can be annoying. An active assistant can be wrong at scale.

Maintainers do not reject AI. They reject missing ownership.

The open-source example shows the human version of the same problem.

June Kim’s “Speedrunning Open Source” experiment sent hundreds of pull requests into open-source repositories. The interesting part was not simply that AI was involved.

Maintainers were not conducting a philosophical trial about machine authorship. They were asking ordinary maintainer questions:

• What changed?
• Why did it change?
• Does the patch respect the project?
• Is there a real person here who will answer when it breaks?

That distinction matters.

A request made by someone with a purpose reads differently from a patch dropped by a machine-shaped fog. The problem is not that a model helped write code. The problem is that nobody seems to have thought.

This is the skeptic’s strongest point. Agentic AI can scale laziness as easily as competence. It can generate tickets, pull requests, emails, reports, security scans, and meeting notes that look like productivity while pushing review cost onto other people.

A company may celebrate output while the maintenance burden quietly moves downstream.

That is not automation. That is externalized cleanup.

Control has at least four surfaces

If “control” remains an abstract governance word, it will not help product teams. It needs to become a design object.

Recent work on runtime governance, AI identity, and intent-governed tool authorization points in the same direction: policies have to follow the path of action, not just the model call.

For AI agents, I think the control surface has at least four parts.

1. Trigger surface

What causes the agent to start acting?

A manual button is one kind of trigger. A scheduled run is another. A change in a database, a new email, a calendar conflict, a security alert, or an inferred user need are all different trigger types.

The more invisible the trigger, the more dangerous the agent becomes.

A user should be able to answer a simple question: why did this start now?

If the answer is hidden inside a model’s inference, the product has already created a trust problem.

2. Permission surface

What can the agent read, change, send, delete, buy, publish, or expose?

This is where product design must separate autonomy from authority.

An agent may be allowed to analyze more than it can execute. It may be allowed to draft but not send. It may be allowed to recommend a database change but not apply it.

It may be allowed to create a pull request but not merge it. It may be allowed to summarize customer data but not export it.

The smallest useful permission set is not a security afterthought. It is part of the product.

The best agent products will not simply ask, “Can this integration call this tool?” They will ask, “Does this action still match the user’s actual intent in this session?”

3. Evidence surface

Can a tired human understand why the agent did what it did?

This is different from having logs. Many systems have logs that only a compliance team, backend engineer, or incident responder can read. That is not enough.

Agent logs need to be human-legible at the point of responsibility.

What did the agent observe? What rule or instruction did it rely on? What alternatives did it reject? What did it change? What did it leave untouched?

The goal is not to expose every token or internal reasoning trace. The goal is to preserve enough evidence for accountability.

A useful agent should leave behind a trail that says: this is what I saw, this is what I did, and this is why the action was within bounds.

4. Recovery surface

What happens when the agent is wrong?

This may be the most neglected surface.

If an agent writes a draft, recovery is easy. Delete the draft. If it edits a file, maybe version control is enough.

If it sends a customer email, changes a production setting, deletes records, makes a purchase, or triggers a workflow across multiple systems, recovery becomes much harder.

A serious agent product needs rollback, interruption, escalation, and ownership.

• Who can stop it?
• Who gets notified?
• What can be reversed?
• What must require approval because reversal is expensive or impossible?

This is where “human in the loop” often becomes too vague. The question is not whether a human appears somewhere. The question is whether the human appears at the right point of consequence.

Apple has the hardware version of the same problem

The App Store story adds another layer.

iOS was built around sandboxing. Apps have limited permissions, visible boundaries, and an approval model that tries to protect users from unexpected behavior.

An AI agent wants almost the opposite. It wants to cross contexts, combine actions, remember goals, operate over time, and touch whatever is needed to finish the task.

That does not mean Apple should ban agents. It means agent behavior has to become legible without destroying the usefulness that made agents attractive.

I have felt this wall while building small apps and trying to reach hardware or system behavior from one app. You hit the sandbox. Then you work around it with Shortcuts, scripts, local bridges, or awkward handoffs.

That awkwardness is not just inconvenience. It is the operating system expressing a philosophy: apps should not casually become everything.

AI agents challenge that philosophy because they are most useful when they are not trapped inside one box.

So the product question becomes colder and more precise:

What cross-app action should be a privilege rather than a personality feature?

The future agent will need an owner

Anthropic’s 2028 scenarios frame the question at the level of global AI leadership: who writes the rules for powerful AI systems?

That question matters. But there is also a smaller version inside every product team, IT department, and workflow.

Who owns the agent?

Not in the branding sense. In the accountability sense.

If an agent touches production code, is the owner the developer, the team lead, the vendor, or the company that approved the workflow?

If it sends a customer message, is the owner the support rep, the manager, the product team, or the agent platform?

If it reads private data across tools, who decided that the access was necessary?

These are not philosophical edge cases. They are deployment questions.

The operator’s answer is not “ban agents.” It is also not “let them do everything and patch the rules later.”

The middle path is more boring and more useful: limited pilots, narrow permissions, visible triggers, readable evidence, reversible steps, and a named human or team attached to consequential decisions.

That sounds less exciting than full autonomy.

Good.

Boring is underrated infrastructure.

The real product question

The next generation of AI agents will not be judged only by how intelligent they seem in a demo. Demos reward surprise. Production rewards restraint.

A great agent product will know when to act, but also when to ask. It will be able to move quickly, but not silently. It will reduce human fatigue without turning humans into spectators. It will make delegation feel powerful without making accountability disappear.

The market will keep selling autonomy because autonomy is easy to explain.

Control is harder. It is less cinematic. It requires permission design, evidence design, recovery design, and organizational design.

It requires saying no to some actions the model could technically perform. It requires making some workflows slower on purpose.

But that may be the difference between an agent that impresses people and an agent that organizations can actually trust.

Maybe autonomy becomes useful only when control becomes boring.

Sources

• Pentagon leaders love agentic AI. But it’s giving cyber criminals nation-state-like powers
• Speedrunning Open Source
• Anthropic’s Cat Wu says that in the future, AI will anticipate your needs before you know what they are
• Apple plans to allow AI agent apps in the App Store
• 2028: Two scenarios for global AI leadership
• Five Eyes intelligence alliance warns that new AI models pose urgent cyber risk
• Runtime Governance for AI Agents: Policies on Paths
• AI Identity: Standards, Gaps, and Research Directions for AI Agents
• Intent-Governed Tool Authorization for AI Agents