Harness Engineering: The Complete Playbook for the Discipline Behind Every Production AI Agent

Harness Engineering

Somewhere in your org right now, there’s an AI agent that worked beautifully in the demo and then fell apart the second it touched a real codebase. It declared victory on a task it never finished. It rewrote a file it was never supposed to touch. It ran the same failing command four times in a row like it was stuck in a loop, because it was.

The instinct is to blame the model. Swap GPT for Claude, Claude for Gemini, upgrade to whatever ships next quarter. And sure, you’ll see a bump. Maybe 10–15% better output. But the agent will still declare victory too early. It will still lose the thread halfway through a long task. It will still merge code that shouldn’t have shipped.

That’s because the model was never the whole problem. The problem is everything around the model: the tools it’s allowed to call, the checks that run before it acts, the memory it carries between sessions, the guardrails that stop it before it does something expensive and irreversible. There’s a name for the discipline of building that “everything else” now, and it’s spreading through engineering orgs faster than almost any AI term since “prompt engineering”: harness engineering.

Here’s what changes when you take that seriously: where the term came from, what actually goes into a production harness, the three failure modes it’s built to kill, a step-by-step build you can copy this week, and how Claude Code, Codex, Cursor, and LangChain actually differ once you look past the marketing.

TL;DR

• Agent = Model + Harness. Swapping the model buys you a modest quality bump. Swapping the harness decides whether the system works at all.
• A harness is everything around the model: tools, permissions, memory, verification loops, guardrails, and logging.
• Three failure modes cause most agent breakage: victory declaration bias, one-shotting overreach, and self-evaluation bias.
• The single highest-leverage first move is a growing AGENTS.md file. Start there before anything else on this list.
• Skip to: Where people push back · The 8-step build · Tool comparison · FAQ

What Is Harness Engineering, Exactly?

Harness engineering is the discipline of designing the systems, constraints, and feedback loops that wrap around an AI agent to make it reliable in production. A harness is not the agent itself, and it’s not the model. It’s everything else: the tools the agent can call, the permissions it’s granted, the state it persists across sessions, the tests that verify its work, the logs that make its behavior observable, and the recovery mechanisms that kick in when something goes wrong.

The formula that keeps showing up, in nearly every serious write-up on the topic, is this:

Agent = Model + Harness

Swap the model for a competitor’s and you’ll see a modest, incremental shift in output quality. Change the harness and you change whether the system works at all.

That’s a fundamentally different kind of leverage, and it’s why teams that used to spend their AI budget chasing the next model release are now spending it on the scaffolding instead.

Where the term came from

The discipline crystallized fast, over roughly two months in early 2026:

• Mitchell Hashimoto, the co-founder of HashiCorp and creator of Terraform, is widely credited with naming the practice around early February 2026. His core principle: every time an agent makes a mistake, engineer the environment so it becomes structurally impossible to repeat that exact mistake.
• OpenAI gave the term a formal public definition on February 11, 2026, in a post by Ryan Lopopolo describing a production application shipped with essentially no manually written code: a codebase that reached roughly a million lines across some 1,500 automated pull requests, with humans designing the environment rather than writing the code themselves.
• Martin Fowler and Birgitta Böckeler at Thoughtworks followed in April 2026 with the field’s most cited taxonomy, formally defining harness engineering as the discipline of designing the systems, constraints, and feedback loops that make agents reliable in production.
• LangChain contributed the “anatomy” framing most engineers reach for first, breaking a harness into concrete primitives: filesystem, code execution, sandboxing, memory, and context management. Their line has become something of a mantra: if you’re not the model, you’re the harness.

One common mix-up worth clearing up: context engineering and agentic engineering are generally attributed to Andrej Karpathy (December 2025 and February 2026, respectively), and they’re related but distinct ideas. Harness engineering is the broader umbrella that context engineering lives inside of.

Where people push back on this

Three objections come up constantly, and they deserve real answers instead of a wave-off.

“Isn’t this just DevOps or MLOps with a new name?” Partly, yes. CI gates, observability, and structured constraints aren’t new inventions. What’s different is the object being disciplined. DevOps constrains a pipeline that runs the same steps every time. A harness constrains a decision-maker that might call five different tools in five different orders for the identical task. The tooling overlaps heavily; the design problem, controlling something non-deterministic, is genuinely distinct from controlling something deterministic. The name is new. The problem it names isn’t entirely new, but its shape is.

“Where’s the primary source for the Hashimoto origin story?” Fair challenge. That piece of the timeline comes from secondary reporting, not a primary post this article could independently verify. The OpenAI post from Ryan Lopopolo and the Fowler and Böckeler taxonomy are both directly sourced. Treat the Hashimoto attribution as the least certain leg of the origin story, not settled fact.

“Is ‘victory declaration bias’ an actual research term or something invented for this piece?” It’s informal shorthand used across several practitioner write-ups, not a term from a peer-reviewed taxonomy with one canonical citation. It’s a real, well-documented behavior. It just doesn’t have the institutional weight of a term like “ReAct loop” yet.

Harness Engineering vs. Prompt Engineering vs. Context Engineering

These three disciplines get conflated constantly, so it’s worth being precise about where each one’s job ends.

harness vs prompt

Prompt engineering optimizes what you say in a single exchange. Context engineering manages what the model can see at any given moment. Harness engineering sits above both: it’s the system that decides which instructions the agent even gets to act on, in what order, and with what safety net underneath.

Think of the model as the engine and the harness as everything else in the car: steering, brakes, seatbelts, dashboard. An engine with no chassis around it doesn’t go anywhere useful, and it definitely doesn’t know when to stop.

The Anatomy of a Production Harness

Synthesizing across Anthropic, OpenAI, LangChain, and the wider practitioner community, a production-grade harness breaks down into five foundational layers. The diagram below isn’t just a list of parts; it’s what happens to the same task with and without each layer in place.

Harness

Tool orchestration is the control plane. The model returns a structured tool call; the harness validates the schema, checks permissions, executes it, and feeds the result back into context. Skip this layer and agents call tools in the wrong order, retry the same failed call on a loop, or worst case, fire off a destructive action with nothing checking it first.

Verification loops run during execution, not just at the end. Most teams get this backwards: they let the agent run the whole task and only inspect the final output. But agents routinely mark work “done” before confirming the outcome, a pattern often called victory declaration bias, and a mid-flight check is the only thing that catches it before it ships.

Context and memory management matters more than it looks like it should. Faros AI has described what happens without it as “context anxiety”: as the window fills up, the model rushes, cuts corners, and gets visibly sloppier the closer it gets to running out of room.

Guardrails are the budget limits, permission boundaries, and human-in-the-loop checkpoints on high-risk actions. The pattern that works well here is “draft, then commit”: the agent proposes a risky action first and only executes it after someone confirms it. It’s a small amount of friction that prevents a lot of painful rollbacks.

And observability is, honestly, the layer most teams skip until it burns them. Without traces, logs, and audit trails, debugging a failed run means reading through a wall of text trying to reconstruct a decision the agent made forty tool calls ago.

LangChain’s version of this anatomy, worth knowing since you’ll see it cited constantly, splits it into filesystem (durable state, a collaboration surface for multiple agents), code execution (bash as a general-purpose tool for problems nobody pre-built a tool for), sandboxing (isolation plus a safe place to verify work), memory (cross-session persistence, often via a file like AGENTS.md loaded into context at session start), and context management (compaction and offloading to fight what’s often called context rot).

The Three Failure Modes Harness Engineering Actually Fixes

Most of the ways AI agents fail in production trace back to three recurring patterns.

1. Victory declaration bias. The agent decides it’s finished before it’s actually checked. Genuinely annoying to debug, because the output looks complete right up until you run it.

2. One-shotting overreach. Instead of decomposing a task into steps, the agent tries to handle everything in a single pass. The result is a tangle of changes that’s hard to review and even harder to roll back. Good harness design forces decomposition so the agent works in smaller, independently verifiable chunks.

3. Self-evaluation bias. When an agent grades its own work, it grades generously, even on tasks with clear pass/fail criteria. It’s not lying to you. It genuinely believes the output is fine. The fix is structural, not mechanistic (this isn’t adversarial training, just a separation of roles): give the generator and the evaluator zero shared context. An evaluator agent that never sees the generator’s reasoning trace catches noticeably more real errors, because an evaluator that shares context with the generator is basically just agreeing with itself.

Core Patterns Every Harness Should Have

The AGENTS.md file

The single most practical tool in this entire discipline is a text file. An AGENTS.md (or CLAUDE.md, depending on the tool) sits at the root of a repository and tells the agent how to behave there: project structure, build and test commands, coding conventions, and a running list of the mistakes past agent sessions have already made.

It grows one rule at a time. Every time an agent repeats a mistake, add a line so the next session starts already knowing better. Over months it becomes a compounding record of everything that’s gone wrong and the constraint that now stops it from going wrong again. GitHub’s analysis of more than 2,500 repositories using this pattern recommends three tiers:

AlwaysLog all delivery attempts;use UTC for scheduling

Ask FirstAdding a new external integration; changing retry intervals

NeverSend without verified opt-in; modify the unsubscribe flow

Skip the “Ask First” tier and you’ll find an agent quietly making judgment calls it was never supposed to make on its own.

Where constraints live: three checkpoints, not three harnesses

A harness enforces rules at three distinct points in an agent’s workflow. Think of these as checkpoints rather than sub-systems. Calling one of them a “harness” inside an article about harnesses is exactly the kind of overloaded naming that makes this stuff harder to learn than it needs to be, so we’ll avoid it here.

Before generation, rules files, lint configs, and type systems shrink the agent’s solution space before it writes a single line. They encode what “correct” looks like so the agent converges on compliant output faster instead of guessing and getting corrected later.

During generation, structured error feedback lets the agent correct itself without a human stepping in. A lint error that just says “violation detected” is useless to an agent. One that says exactly which pattern to use instead lets it fix itself mid-task.

Before merge, hard CI gates are the final backstop: real failures, not warnings, that block anything the first two checkpoints missed from ever reaching production.

These three checkpoints answer where a rule gets enforced. The Plan-Execute-Verify loop below answers a different question: how a single task moves through those checkpoints over time. PEV is the process a task follows; the three checkpoints above are the stations it passes through along the way.

The Plan-Execute-Verify (PEV) loop

Instead of asking a model to solve a multi-step problem in one pass, PEV forces an explicit plan, bounded execution against that plan, and verification against both the plan and external quality criteria.

The Plan-Execute-Verify

A plain generate-and-check workflow only tests the output after the fact. PEV adds gates at every phase transition: before a tool call fires (is this a known tool, are the arguments valid, does this need explicit approval), during execution, and after, including the check most test suites completely miss: does the implementation actually match the plan, or did the agent quietly build a parallel version of something that already existed?

A Step-by-Step Walkthrough: Building a Harness for a “Ticket to PR” Agent

Say you want an agent that takes a bug ticket, investigates the codebase, writes a fix, and opens a pull request, unattended, without a human babysitting every step. Here’s how to actually build the harness around it.

Step 1: Write the AGENTS.md first, before any agent code. Document build and test commands, directory structure, naming conventions, and a “Never” list (never touch the payments module without human sign-off, never delete a migration file, never modify .env files). This is the cheapest, highest-leverage thing you'll do all week.

Step 2: Give it a scoped toolset, not a firehose. Resist the urge to hand the agent every tool you have. Start with: read files, search the codebase, run the test suite, run the linter, open a draft PR. Several teams, including Vercel by their own public account, have found that reducing the number of available tools improved success rates. More tools just means more surface area for the agent to pick the wrong one.

Step 3: Force a plan before any code gets written. Have the agent produce an explicit plan (which files it expects to touch, what tests it expects to add, what “done” means) and treat that plan as a gate. Nothing executes until it’s legible to a human reviewer.

Step 4: Wire in feedback loops that speak the agent’s language. Your linter shouldn’t just fail. It should fail with a message the agent can act on: “use the logger.info wrapper instead of console.log," not "style violation." The clearer the signal, the less often a human has to step in.

Step 5: Separate the generator from the evaluator. Once the agent believes it’s done, hand the diff to a second agent instance that’s never seen the first one’s reasoning trace. Have it check the diff against the ticket and the plan from Step 3, cold. This one change alone catches a surprising number of “looks done, isn’t done” cases.

Step 6: Put hard quality gates in CI, not in the prompt. Complexity limits, max lines per function, required coverage on touched files, all enforced as CI failures, not polite suggestions in a system prompt. “Please follow our style guide” is not a substitute for something that actually blocks the merge.

Step 7: Log everything and link sessions to outcomes. Every tool call, every retry, every gate the agent passed or failed, captured and linked to the eventual PR. When a bug ships six weeks later, you want to trace it back to the exact session that introduced it, not go spelunking through commit history guessing.

Step 8: Measure before you tune further. Track task resolution rate, code churn on agent-touched files within two weeks, and the metric most teams skip: defect escape rate on agent-authored code versus human-authored code. If agent code is producing more incidents, that’s a harness problem, not a model problem.

Run this loop for a month, feeding every failure back into the AGENTS.md file and the gate configuration, and the review burden on agent PRs typically drops substantially, not because the model got smarter, but because the system around it stopped letting the same mistakes recur.

Copy this checklist into your team’s onboarding doc:

• AGENTS.md written before any agent code, including a "Never" list
• Toolset scoped to five or fewer tools to start
• Agent required to produce a plan before writing code
• Linter and CI errors rewritten to be agent-actionable, not just pass/fail
• A second, context-isolated agent reviews the diff before merge
• Quality gates enforced in CI, not in the system prompt
• Every tool call and gate outcome logged and linked to the resulting PR
• Task resolution rate, churn, and defect escape rate tracked weekly

Harness Engineering vs. the Competition: How the Major Platforms Stack Up

Harness quality, not raw model benchmark scores, is increasingly what separates these products. Here’s how they actually differ once you look under the hood.

1. Single-Agent & Developer Tooling

Claude Code (Anthropic)

Claude Code looks for rules files in CLAUDE.md at the project root or globally in ~/.claude. Its core harness pattern treats Git commits as checkpoints, operating on a "dumb loop" philosophy—meaning the intelligence lives entirely in the model while the harness simply manages turns. Its strongest layer is the simplicity and transparency of this execution loop, backed by robust filesystem and sandbox primitives via the Agent SDK. However, it falls short when it comes to out-of-the-box multi-agent orchestration compared to dedicated frameworks.

Cursor

Cursor uses project-scoped rules following the .cursor/rules/*.mdc convention. Its core harness relies on IDE-integrated rules with fine-grained scoping. The platform's strongest layer is its tight developer-loop integration, which makes rule enforcement seamless during active coding. Its main limitation is that its rules scope is inherently narrower than repo-wide conventions.

GitHub Copilot

GitHub Copilot reads instructions from .github/copilot-instructions.md, using repo-wide plus path-specific instructions for its harness pattern. Its strongest layer is its massive ecosystem reach and native integration with existing CI/CD pipelines. It falls short in its verification-loop tooling, which is currently less mature than what you find in dedicated harness platforms.

2. Frameworks & Multi-Agent Systems

OpenAI Codex

OpenAI Codex uses an AGENTS.md file, which applies hierarchically from the Git root down to the current working directory. Its harness pattern relies on "taste invariants" enforced as hard CI failures, serving as the original production reference case for this discipline. Codex's strongest layer is its strict enforcement discipline: rules are treated as absolute gates rather than soft suggestions. On the downside, it requires significant custom harness work to actually match Codex's own internal setup.

LangChain / LangGraph / Deep Agents

Unlike file-based tools, this ecosystem operates at the framework level without a single rules file. Its core harness uses explicit state graphs, checkpointing at super-step boundaries, and built-in planning (write_todos) paired with subagent spawning. The strongest layer here is multi-agent orchestration and state modeling—this architecture famously drove a jump from rank 30 to rank 5 on Terminal Bench 2.0 through harness changes alone. Its clear drawback is that more assembly is required; you are ultimately building a custom harness rather than buying a finished product.

CrewAI

CrewAI manages behaviors through role-based configurations (defining role, goal, and backstory). Its harness pattern follows an Agent/Task/Crew architecture, adding a “Flows” layer on top for deterministic routing between autonomous crews. Its strongest layer is its incredibly clean mental model for role-based multi-agent teams. However, it offers less granular, tool-level constraint enforcement compared to gate-based systems.

3. Enterprise Context Engines

Augment Cosmos

Augment Cosmos maintains compatibility with AGENTS.md and CLAUDE.md, while introducing native "Augment Rules" categorized by execution type (always, agent-requested, or manual). Its harness pattern is driven by a Context Engine that indexes the entire codebase for cross-service verification, utilizing built-in "Experts" for reviews, PRs, and testing. Its strongest layer is its enterprise-scale codebase context and rigorous pre-merge verification. As a newer entrant to the market, its primary shortcoming is that its ecosystem and community tooling are still catching up to the breadth of LangChain.

A few things worth being honest about:

No platform eliminates the need to do harness engineering yourself. Every one of them gives you primitives, rules files, gates, checkpoints, but the domain-specific verification logic, the tools that connect to your systems, and the risk tolerance encoded in your guardrails are always custom work.

AGENTS.md is winning as a cross-tool standard. Released as an open spec in mid-2025 with backing from OpenAI, Google, Cursor, and others, it’s fast becoming the one file most tools will read regardless of which agent you’re running.

Framework-first tools give you more control and more work. Product-first tools give you a faster start and less control over the orchestration internals. Neither is objectively better; it depends on whether your team wants to own the harness or rent one.

The generic parts of a harness are now a commodity. Basic logging, session management, and standard tool interfaces are table stakes across every serious platform. The competitive edge has shifted almost entirely to the custom layer: your verification logic, your evaluation datasets, and your understanding of where your specific codebase tends to break.

If you just want the one-line recommendation:

• Want the simplest, most transparent execution loop? Claude Code.
• Want enforcement discipline baked in from day one? OpenAI Codex.
• Living entirely in your IDE already? Cursor.
• Need broad ecosystem reach with minimal setup? GitHub Copilot.
• Building multi-agent orchestration from scratch and want full control? LangChain / LangGraph.
• Want a clean mental model for role-based agent teams? CrewAI.
• Working across a large enterprise codebase that needs cross-service verification? Augment Cosmos.

Measuring Whether Your Harness Is Actually Working

Harness engineering without measurement is just guesswork with extra steps. In rough order of usefulness:

• Defect escape rate on agent-touched code, tagged separately from human-written code. The single most telling number, and also the hardest to get without decent observability already in place.
• Task resolution rate, verified by automated tests rather than the agent’s own say-so.
• Code churn rate, the percentage of agent-written code discarded or rewritten within two weeks of merging.
• Verification tax, the gap between time-to-first-commit and time-to-PR-approval. This tells you how much of the “speed” the agent gave you is being re-spent on review.
• Harness constraint effect, success rate on identical tasks, constrained versus unconstrained, isolated from any model change. This is the number that proves whether your harness work is actually paying for itself.

DORA’s research on this space warns against leaning on shallow, volume-based metrics like lines of code accepted or number of suggestions used. They measure activity, not reliability, and they’ll tell you your harness is working long after it’s actually stopped.

Building vs. Buying

The rule of thumb that holds up: buy the commodity plumbing, build the domain-specific layer. Managed runtimes, basic telemetry, and standard control planes from established frameworks are not worth building from scratch. What genuinely needs custom engineering is the verification logic that understands your data, the guardrails that match your organization’s actual risk tolerance, and the tools that connect to systems only you have.

One thing that trips up almost every team eventually: a harness isn’t a fixed architecture. It needs recalibrating with every meaningful model upgrade. The right question after a new model ships isn’t “what do I add to compensate for its weaknesses.” It’s “what can I now remove.” A constraint that was load-bearing because the old model couldn’t handle long-horizon tasks can become dead weight the moment context handling improves. Review the harness every time you touch the model.

The Bottom Line

The teams getting dependable output from AI agents right now usually aren’t running a better model than everyone else. They’re running a better system around whatever model they already have. That’s the whole premise of harness engineering, and it’s also the good news in all of this: you don’t need to wait for the next model release to fix the agent that’s frustrating you today. You need an AGENTS.md file, a scoped toolset, a plan-execute-verify loop, an evaluator that doesn’t share context with the generator, and gates that actually block bad output instead of just flagging it.

Start this week. Pick the last agent failure that annoyed you, write down the one rule that would have prevented it, and add it to the file. Do that again the next time something breaks. That compounding habit, more than any model swap, is what separates a demo from a production system.

Frequently Asked Questions

Is harness engineering a real job title yet? Not widely, no. Right now it’s a discipline practiced under titles like platform engineer, AI infrastructure engineer, or just “the person who owns the agent tooling.” That’s likely to change as the discipline matures, the way “DevOps engineer” eventually became a standalone title.

Do I need harness engineering for a simple, single-purpose agent? Not the full version. A single-purpose agent with one tool and a narrow task usually just needs basic error handling. The discipline earns its keep once an agent runs multi-step tasks unattended, touches production systems, or operates across sessions.

What’s the single fastest thing to implement this week? An AGENTS.md file with a "Never" list. It costs almost nothing to write and prevents entire categories of mistakes immediately, before any of the harder infrastructure work.

Does harness engineering replace prompt engineering? No, it sits above it. You still need clear prompts for individual turns. The harness is what decides which prompts the agent gets to act on, in what order, and with what checks around each action.

Will better models eventually make harness engineering unnecessary? Some individual constraints will, yes, and that’s expected. A harness isn’t a fixed architecture; it should be reviewed every time you upgrade the underlying model, removing constraints that were only load-bearing because of the old model’s weaknesses. But the higher-level discipline, deciding what an agent is and isn’t allowed to do unsupervised, isn’t something a smarter model makes obsolete.

Keep exploring

• The Anatomy of an Agent Harness, LangChain: the primitives-first breakdown that popularized “if you’re not the model, you’re the harness.”
• OpenAI: Harness Engineering: the original production case study that gave the discipline its name.
• Martin Fowler / Thoughtworks on harness engineering for coding agents: the formal taxonomy from Birgitta Böckeler.
• AGENTS.md open specification: the emerging cross-tool standard for agent rules files.
• awesome-harness-engineering on GitHub: a curated, actively updated list of tools, patterns, and case studies across the ecosystem.