How to Build Self-Optimizing Multi-Agent Systems for Production

A guide to tailoring topologies, feedback loops, and quality gates for self-optimizing workflows

Single-prompt LLM agents often fail in production due to infinite loops, duplicate tool calls, or hallucinations. For complex, multi-step tasks, single prompts do not scale.

To build reliable systems, production architectures often transition from single prompts to Multi-Agent Systems (MAS). Dividing a task among specialized agents distributes the computational load, manages context windows, and handles multi-turn workflows.

There is no universal multi-agent architecture. An effective team must be designed specifically for the target use case. A team structured to write software will not work for academic research or scientific exploration. To build a reliable system, you must tailor the collaboration topology, roles, quality gates, hardening strategies, and optimization loops directly to the constraints of the target workflow.

Many proposed multi-agent architectures are either too complex (involving dozens of agents with fragile routing) or too simple, relying on basic two-agent loops (Generator-Critic) that easily get stuck in local minima. A simple generator-critic loop often suffers from oscillation, where the generator repeatedly writes flawed fixes because it only reacts to the immediate error without understanding its historical pattern.

The key to production-grade quality is the Three-Agent Optimization Loop, modeled after Automatic Prompt Optimization (APO). By introducing a third, meta-cognitive agent, the Optimizer, we decouple evaluation from instruction refinement. The Optimizer monitors the history of executions, analyzes failure patterns, and dynamically refines the instructions fed to the worker. This creates a self-correcting system that systematically converges on high-quality outputs.

This post explains the engineering process of designing, building, and optimizing specialized multi-agent teams for production tasks.

1. The Three-Agent Optimization Topology

To build a self-optimizing agent team, we structure the collaboration space around three distinct, specialized roles:

The first role is the Worker, which executes the concrete task (such as writing code, drafting text, or generating hypotheses) based on an input prompt and a set of instructions. The second role is the Evaluator, which measures the quality of the Worker’s output against objective gates. The Evaluator does not attempt to fix the output; it only runs tests, verifies sources, or scores rubrics, capturing raw error logs, stack traces, and quality metrics. The third role is the Optimizer, a meta-cognitive layer that maintains a memory of the execution history. If the Evaluator flags a failure, the Optimizer analyzes the current draft, the evaluation errors, and the history of past attempts. It identifies systemic patterns of error and writes a refined set of instructions or a prompt wrapper for the Worker’s next attempt, preventing oscillation and driving convergence.

2. Case Studies: Tailoring Optimization Loops for Specific Tasks

To create an effective agent team, we must customize this three-agent topology, its quality gates, and its optimization loops for the unique constraints of the target workflow.

Case Study 1: The Code Optimizer (Software Engineering)

This system implements software features from text descriptions into tested, type-safe code using a three-agent debugging and optimization loop.

In this system, the Coder writes the initial code by modifying repository files using Model Context Protocol (MCP) tools, and the Tester runs it in an isolated Docker sandbox using pytest to capture stack traces and test failures. If a test fails, the Optimizer Agent reviews the Coder’s draft, the test failures, and the history of previous debugging attempts. Instead of writing code directly, the Optimizer identifies the root cause of the bug (such as a recurring edge-case index error) and generates a structured debugging plan with refined coding constraints for the Coder’s next attempt.

Before any code is accepted, it must pass two strict quality gates. First, static analysis via pytype verifies syntax and type correctness. Second, a dynamic test suite via pytest must achieve a 100% pass rate. To control execution costs, we enforce an iteration budget capping the loop at 5 attempts, halting the run for human review if exceeded. Finally, to prevent the Coder from modifying test assertions to falsely pass the run, we enforce process isolation: the test files are kept read-only completely outside the writeable sandbox.

Case Study 2: The Self-Correcting Curation Team (Research & Writing)

This system, modeled after Stanford’s STORM framework, conducts deep research, verifies sources, and compiles a cited document using a three-agent writing, auditing, and editing loop.

The Writer performs a retrieval-augmented search and writes a draft section. Next, the Auditor audits the draft by running semantic similarity checks against the raw source documents, scoring every sentence for factual grounding and citation accuracy. If the Auditor flags ungrounded claims, the Editor analyzes the draft, the grounding scores, and the history of previous critiques. The Editor identifies content gaps (such as a bias toward a single perspective or missing technical evidence) and generates a refined writing directive and style guide to instruct the Writer on how to restructure the content.

The final document is rejected if any sentence contains a claim that cannot be traced back to a verified source URL. To prevent database lock contention during parallel writes, we enforce state partitioning, keeping intermediate drafting debates private between the Writer and the Editor. To optimize the search budget, the system runs similarity checks on all outbound queries, merging duplicate tasks before execution.

Case Study 3: The Autonomous Scientist (Scientific Discovery)

This system, modeled after Sakana AI’s The AI Scientist, automates the scientific method by generating machine learning hypotheses, running training runs, analyzing results, and writing a LaTeX paper using a three-agent exploration and optimization loop.

The Researcher proposes a machine learning hypothesis and writes the training script. Next, the Reviewer executes the script in a GPU sandbox and writes a detailed peer review using conference rubrics, scoring the run on novelty and correctness. If the paper scores below a 6/10, the Advisor reviews the hypothesis, the experimental results, the reviewer’s critiques, and the history of previously tested hypotheses. The Advisor identifies systemic failures (such as gradient explosion or learning rate issues) and generates optimized prompt constraints and search spaces to guide the Researcher’s next hypothesis.

Before conducting new experiments, the system runs a baseline verification to confirm the environment is correct and prevent false claims of improvement. The final paper must be compiled in LaTeX and achieve an automated review score of >= 6/10. To prevent the Researcher from modifying the evaluation script, we enforce read-only permissions on all evaluation files, keeping them completely outside the writeable sandbox.

3. Design Principles for Self-Correcting Loops

Implementing a three-agent optimization loop in production requires adhering to three core software engineering principles to ensure the loop converges:

A. Isolate the Meta-Cognitive Layer

Do not combine the Coder/Writer role with the Optimizer role in a single agent prompt. LLMs struggle to simultaneously execute a task and objectively analyze their own history of failures. Decoupling the Worker (execution) from the Optimizer (meta-analysis) allows you to use a smaller, faster model for execution, and a highly capable, reasoning-focused model for optimization.

B. Structure the Trajectory Memory

The Optimizer cannot refine instructions if the execution history is a chaotic log of raw text. You must maintain a structured Trajectory Memory. For every iteration, log the instruction set or prompt fed to the Worker, the exact output generated by the Worker, and the structured evaluation metrics and error logs returned by the Evaluator to ensure consistency. This structured log allows the Optimizer to perform in-context regression analysis, identifying what changes in the instructions led to improvements or regressions.

C. Enforce Strict Convergence Boundaries

Because optimization loops dynamically rewrite prompts, they are susceptible to drift and infinite loops. We define these limits using two strict boundaries. The success boundary terminates the loop immediately when the output passes all verification gates. The failure boundary terminates the loop and alerts an engineer if the iteration budget is exhausted, or if the Optimizer detects a loop (such as when the last three instruction refinements yield identical evaluation scores, indicating the system has hit a local minimum).

Conclusion

An effective multi-agent team is not a complex, sprawling network, nor is it a fragile two-agent loop. For many production use cases, a three-agent self-optimizing loop that pairs a Worker and an Evaluator with an Optimizer offers a practical path to reliability.

By separating execution, evaluation, and optimization, logging structured trajectories, and enforcing strict convergence boundaries, you can build autonomous systems that systematically learn from their own failures and deliver consistent, self-improving quality in production.