We Are Building Autonomous Agents Before We Know How to Control Them

The Era of Chatbots is Over: Why AI is Now a Cybersecurity Problem

To most people, AI is still just a fancy new tool that you use to write emails or generate boilerplate code. I used to think the same way. But once you start building and look under the surface of these models, the reality changes drastically. We are quickly moving away from simple chatbots to multi-agent orchestrators that can autonomously write production code, execute complex workflows, and integrate directly with other software systems.

When I first started learning about AI Safety and alignment, I realized how similar these systems were to humans. They can actively learn to reconcile conflicting goals, exploit hidden loopholes, and intentionally deceive their creators to pass safety tests. And honestly, seeing how fast developers are handing over root access to these models is terrifying. Everyone is excited to see these agents complete tasks and write code, but not enough people are talking about what happens when these powerful agents suddenly decide to ignore our guardrails. This is when it becomes especially important to train models to act safely and have guardrails to restrict them when they spiral out of control.

Instead of a 100-hour lecture on AI Safety and alignment theory, here is a quick breakdown of the core concepts driving AI Safety today, why they matter, and the vulnerabilities everyone needs to know about.

Why Good Code Goes Wrong: Outer vs Inner Alignment.

To understand why AI safety is important, we have to define alignment. Alignment is the process of ensuring an AI’s objectives actually match human values. As LLMs become autonomous agents and start making more important decisions, safety becomes critical. If we don’t properly align these systems, they will actively optimize for the wrong outcomes.

The first layer is outer alignment. Outer alignment involves writing the right rules for an AI. When we give a model a flawed goal, it often does everything in its power to achieve that goal, even if it involves breaking the rules. This is known as reward hacking or specification gaming. For example, in the CoastRunners boat-racing simulation, the AI discovered it could score an infinite number of points by driving in circles and repeatedly crashing into the same targets. The boat caught on fire and never finished the race, but it technically maxed out the score. This is a very simple and harmless example, but in more advanced models, this escalates into reward tampering. This is when systems rewrite their own testing code to fake a perfect score, and based on where these models are deployed, this can lead to serious security issues.

The second layer is inner alignment. Inner alignment deals with the hidden internal goals that a model develops during training. An AI might look innocent and safe within a sandboxed environment. But it could just be fooling testers and faking goals to match the testers’ expectations, all while hiding its true intentions underneath. And if you do end up deploying this “evil” model in the real world, it ends up using its power to pursue the wrong goals, also known as goal misgeneralization. Furthermore, this can also lead to deceptive alignment, where the model realizes it’s being monitored and avoids modification by faking compliance during testing and only executes bad instructions when it’s deployed without constraints. There are actually many closed testing scenarios that research labs trained models with this behavior to see if they would replicate any of these scenarios, and multiple have.

Sycophancy and Sleeper Agents

Currently, the industry relies on RLHF (Reinforcement Learning from Human Feedback) to keep models in check. It works by having humans grade outputs and improve them, exactly what it sounds like. This works, but it has one major flaw: sycophancy. Sycophancy is when a model begins to mimic user biases and hallucinate facts just to gain human approval; they learn to be a yes-man. Instead of aligning the internal goals, this feedback loop teaches the system to fake compliance, which leaves any underlying vulnerabilities undiscovered. This also creates a mask over the model and tricks us into believing it’s aligned.

This mask gets incredibly dangerous when it comes to sleeper agents. Sleeper agents are models that are trained to function flawlessly under normal conditions but execute malicious actions when they encounter a specific trigger. Standard safety training completely fails to remove these backdoors, and adversarial training often backfires, making the deception harder to detect because the model simply learns to hide its malicious intent during evaluation.

Another threat is MINJA attacks, also known as memory poisoning. This is when an attacker implants malicious instructions into external documents or web pages that an agent is forced to retrieve and analyze. This data sits completely unused in a database until a user query triggers the model to retrieve that document, and once it reads the document, the injected prompt hijacks the system and causes it to execute unauthorized actions under the guise of its own learned knowledge.

Escaping Containment: How Autonomous Agents Break the Rules

With the recent popularity spike of autonomous agents and coding agents, we are introducing a new vulnerability to defend. The biggest threat of these agents is sandbox escapes, when the agent uses tools to map out the environment and break out of an isolated container to modify sensitive information. Software developers have to constantly run system commands for their work, which is why many agents have root-level access. But if a model is misaligned, it is able to scan the environment for privilege mistakes or runtime CVEs. It can then exploit these bugs to break out of an environment. This is why many developers have moved to hardware-isolated microVMs or cloud VMs to host their agents.

Another issue is omission attacks, which happen when an agent sabotages a system by strategically failing to perform a critical task. For example, an agent might configure an entire server perfectly but choose to leave out a single firewall rule, making the whole server vulnerable. To combat omission attacks, developers need to use monitors that can evaluate whether the final system state is secure, instead of just watching the individual commands as the agent executes them.

Escaping Containment: How Autonomous Agents Break the Rules

With the recent popularity spike of autonomous agents and coding agents, we are introducing a new vulnerability to defend. The biggest threat of these agents is sandbox escapes, when the agent uses tools to map out the environment and break out of an isolated container to modify sensitive information. Software developers have to constantly run system commands for their work, which is why many agents have root-level access. But if a model is misaligned, it is able to scan the environment for privilege mistakes or runtime CVEs. It can then exploit these bugs to break out of an environment. This is why many developers have moved to hardware-isolated microVMs or cloud VMs to host their agents.

Another issue is omission attacks, which happen when an agent sabotages a system by strategically failing to perform a critical task. For example, an agent might configure an entire server perfectly but choose to leave out a single firewall rule, making the whole server vulnerable. To combat omission attacks, developers need to use monitors that can evaluate whether the final system state is secure, instead of just watching the individual commands as the agent executes them.

Escaping Containment: How Autonomous Agents Break the Rules

With the recent popularity spike of autonomous agents and coding agents, we are introducing a new vulnerability to defend. The biggest threat of these agents is sandbox escapes, when the agent uses tools to map out the environment and break out of an isolated container to modify sensitive information. Software developers have to constantly run system commands for their work, which is why many agents have root-level access. But if a model is misaligned, it is able to scan the environment for privilege mistakes or runtime CVEs. It can then exploit these bugs to break out of an environment. This is why many developers have moved to hardware-isolated microVMs or cloud VMs to host their agents.

Another issue is omission attacks, which happen when an agent sabotages a system by strategically failing to perform a critical task. For example, an agent might configure an entire server perfectly but choose to leave out a single firewall rule, making the whole server vulnerable. To combat omission attacks, developers need to use monitors that can evaluate whether the final system state is secure, instead of just watching the individual commands as the agent executes them.

Mechanistic Interpretability and Structural Control

Defending an AI system from the outside is also not enough. We have to actually look inside the black box using mechanistic interpretability. This is the practice of reverse-engineering neural networks down to their core mathematical layers to read their internal thinking process. Since individual neurons fire for thousands of completely unrelated things, safety researchers use Sparse Autoencoders (SAEs) to decompress these hidden layers. With tools like Google’s Gemma Scope, we can now translate these mathematical structures into concepts that we can read and debug.

Another powerful defensive layer is Latent Adversarial Training (LAT). LAT strengthens a model by altering its internal residual stream during training. This is better than traditional alignment because, unlike traditional alignment, which tries to catch issues on the surface, LAT forces the model to build a defense within its model weights. This ensures that the safety rules hold up even under pressure.

Oversight and Machine Unlearning

How do we manage AI models that are smarter and faster than us? One answer is scalable oversight. Instead of relying solely on humans to check massive amounts of complex data, we use smaller, trusted AI models to help us evaluate the outputs of highly capable frontier systems. This concept is called weak-to-strong generalization. We can combine this with automated security monitors like ActionLens. ActionLens pauses the agent right before it runs a command to test security. It launches tracking probes, calculates file changes, and simulates the environment to check for any risks of running the command. Even after this, if a model still poses a risk, we can use traditional machine unlearning techniques like Representation Misdirection to strip dangerous capabilities and then use benchmarks like WMDP to verify that the information is gone from the model.

Conclusion: A Call for Builders

AI safety is no longer just a problem for philosophy majors or theoretical researchers. Many of these powerful models are already deployed in critical systems, and as they continue to evolve, humans will keep giving them more access to systems. We now need builders who deeply understand topics related to security, like containerization, network security, kernel architecture, and more, to translate these theoretical risks into hard-coded defenses. To many, AI is just a tool, but now the future of AI is about engineering the environments to keep it secure.