Beyond Prompt Optimization: Optimizing the Whole Agent Harness

Most agent optimizers tune the prompt. Yet in modern agentic systems, orchestration, tool usage, runtime behavior, and execution policies often contribute as much to performance as the prompt itself. Methods such as OPRO, MIPROv2, TextGrad, and GEPA primarily optimize prompt-level behavior, while leaving most harness-level decisions fixed and hand tuned by humans. Instead, we optimize the entire harness using Meta-Harness.

Optimizing the Harness

Meta-Harness treats optimization as a search over harness code, prompts, and system structure rather than treating the prompt as the sole optimization surface. An LLM proposer reads prior traces, proposes a candidate change, and evaluates it on a small search set. A candidate is accepted only if it is Pareto-improving across our tracked objectives, and only if the trace shows the intended mechanism actually triggered and explained the observed result. Strong candidates are then promoted to a held-out validation set.

Optimization loop context and the harness surfaces the proposer could modify

Two insights drive the approach. The first insight builds on prior works. Methods such as GEPA showed that optimization improves when full execution traces are available, rather than collapsing outcomes into a single scalar reward. They also demonstrated the value of maintaining a Pareto frontier instead of optimizing a single metric. The second, and more important, insight is to optimize the entire harness. A coding-agent (proposer) edits the harness directly. Each candidate run logs code, scores, and traces to the filesystem, and later proposers use that history as input.

We extended Meta-Harness in two ways:

• Multi-objective optimization. In addition to pass rate, we optimize quality, token usage, LLM calls, tool calls, and wall-clock time.
• Multi-agent harness surface. We let the meta-harness modify planner/executor boundaries and the dispatcher, so it can optimize a multi-agent system.

Search Results

The figure below shows the optimization trajectory on the search split.

The left panel plots each candidate by pass rate against the dollar cost to evaluate it, with the Pareto frontier traced through the best trade-offs. The cheapest configuration on the frontier ran at $1.90 per candidate at a 70% pass rate; the best-quality one reached 100% at $5.26, where one “candidate” is a full harness evaluated across the whole search set.

The right panel shows progress over time: the best-so-far pass rate hit optimal within the first handful of candidates, after which the optimizer spent the remaining runs driving down the cost. The plot shows only two axes, pass rate and cost, for readability, but acceptance uses the full multi-objective criterion, which includes six objectives; pass rate, quality, token usage, LLM calls, tool calls, and wall-clock time. Only candidates that improve this broader frontier are retained.

The Biggest Wins Came from Changing the Harness

The four highest-impact, most trace-verifiable mechanisms:

• Planner/executor boundaries improved reliability. Separating plan structure from execution reduced “false-complete” patterns. Planner output stayed as task structure, while deliverables were produced only in executor steps.
• Dispatcher policies reduced redundant work. Small dispatcher-rule edits changed what executors did first and cut duplicated work. Executors were instructed to check for reusable capabilities before writing new logic.
• Runtime and setup choices changed efficiency. Infrastructure edits, not prompt edits, moved latency and cost. For example, changing package-install and setup paths removed repeated environment-setup overhead in long chains.
• Tool-output filtering kept context focused. Guardrails on very large tool responses prevented context flooding. Long command output, for instance, was truncated to a head-plus-tail view so the next reasoning step stayed on track.

In each case we could see the mechanism fire in the traces. Not just a better score, but a verifiable reason for it.

The Optimizer Doubled as a Debugger

Some of the most important wins were not optimizations at all. They were bug fixes in the harness itself. Along the way, the meta-harness found and fixed bugs such as:

• Dependency-propagation bug. Downstream steps ran after an upstream setup failure. An install command failed, but a later step still read artifacts that should only exist after successful setup.
• Evaluator-state race. Checkpoint logic ran before the final completion state had propagated. A run was archived as complete while late task updates were still in flight, producing inconsistent final status.

These came from the same loop: propose, evaluate, inspect traces, repair, repeat. A search designed to find better harnesses also surfaced the latent defects in the one we already had.

Comparison with Expert-Tuned Baselines

The combination of the highest-impact mechanisms did not merely improve performance on the search split; it generalized to a larger held-out evaluation set.

Held-out generalization vs. an expert-tuned prompt baseline.

Takeaways

• Give the optimizer the harness code, not just the prompts. The edits that mattered most were structural and operational, and unreachable from the prompt.
• Keep the search set small, but stress it. Include long-chain tasks so brittle setup and dependency bugs fail loudly during search instead of quietly in production.
• Trust traces, not scores. Treat every improvement as provisional until the trace shows the intended mechanism actually fired.

The Bigger Picture

This pattern has appeared repeatedly throughout machine learning. Once a method is given a sufficiently rich search space and the compute to explore, it surpasses the humans who would otherwise tune the system by hand. Features were hand-engineered before they were learned. Prompts were hand-written before they were optimized. Each time, the handcrafted version was a placeholder, useful until the search caught up. Sutton called this the bitter lesson: general methods that scale with compute beat hand-encoded human knowledge.

The harness is next: the code, orchestration, and runtime choices around the model that we still tune by hand. Define the objective, give the optimization loop access to the harness and its traces, and let it explore. In doing so, it can discover better configurations than we would have designed manually and expose failures we did not know existed.

References

• Large Language Models as Optimizers
• DSPy MIPROv2
• TextGrad
• GEPA
• Meta-Harness