Building a Production Agent Harness: Turning Claude Code Into a Multi-Agent Engineering Pipeline

A coding agent on its own is a brain in a jar. It can think, generate code, call a function — but it cannot answer your Slack DM at 3am, retry a failing CI job, fix the merge conflict that just appeared on its open MR, or remember that a reviewer’s question yesterday is still unanswered.

The thing that makes the brain useful is the harness: the runtime scaffold around the LLM that gives it senses, hands, and memory. Event ingestion, agent orchestration, persistent state, self-healing loops, observability, and a control surface for the human operator. This article is about building one — not as a research demo, but as a production system that watches a Slack channel, opens MRs against five internal repos, addresses reviewer comments overnight, and quietly self-heals when a CI run goes red.

We built ours as an internal system — a multi-agent pipeline built on top of Claude Code that has now been running continuously for several months, ingesting from Slack and dispatching real engineering work. This is the top-down view: what it does, how the components fit together, and the production scars that shape the design.

In this article:

• Why a Harness, Not Just an Agent
• What Our Harness Does
• Three Gaps a Harness Must Close
• Layer 1 — Event Ingestion
• Layer 2 — Agent Orchestration ↳ The Autonomous Agent Loop in Detail ↳ The Self-Improvement Layer
• Layer 3 — Persistent State
• Layer 4 — Self-Healing Loops
• Layer 5 — Observability
• Layer 6 — Human-in-the-Loop Control
• Beyond the Six Layers
• End-to-End Walkthrough
• Comparison to Other Harnesses

Why a Harness, Not Just an Agent

When ChatGPT plugins arrived in 2023, many teams tried the obvious thing: drop a chat UI in front of an LLM with a few function-call tools and call it “the engineering agent.” The pattern almost works for demos and never works in production. Three failure modes appear within the first week:

1. It loses context the moment the operator closes the tab. Memory needs to outlive a single chat session. A real engineering task spans days: file a JIRA, branch, draft, review, address review comments, CI passes, merge. No single LLM call holds that state.
2. It can’t react when something changes externally. A reviewer posts a comment at 4pm. The CI fails on the third commit. A teammate replies in the thread. The agent must wake up on those events, not poll forever or wait for the user to re-prompt.
3. It cannot recover from its own failures. The agent pushes a commit; CI breaks; the operator has to come back tomorrow and re-explain the failure. Or the agent’s Okta (an identity management platform) session expires mid-task and the whole pipeline dies silently. Both happen daily. The cost: engineers getting paged on evenings and weekends for work the agent was supposed to have handled.

Solving these is the harness layer’s job. The harness is what differentiates a claude.ai chat tab from systems like Claude Code itself, Cursor’s background agents, Cognition’s Devin, or what we built. The brain (the model) is essentially interchangeable across these — what changes is the harness around it.

The brain talks to the world only through the harness. The harness is doing nearly all the engineering work that makes an LLM behave like a system.

What Our Harness Does

We started with a concrete problem: a Slack channel — call it #support — accumulates ~30 tickets per week, each of which needs an investigation followed (usually) by a code change in one of five repos. Each ticket spans days. Reviewers leave comments on MRs. CI flakes. Sometimes an investigation turns out to be a duplicate of last month’s. The team carrying this load was getting paged at evenings and weekends, and the work was not improving — the same shapes of tickets returned every cycle.

The harness has three responsibilities, in order of intervention depth:

1. Investigate — when a new ticket appears in #support, follow the thread, gather context across repos, post a structured analysis back in the same thread.
2. Fix — when the operator approves, open an MR (Merge Request) with the proposed change, address reviewer comments, watch CI, fix CI when it breaks, get to merge.
3. Self-improve — every closed case feeds back: if a recurring pattern appears (e.g. five tickets in the same area touch the same config), generate gaps + propose changes to the harness itself.

The third responsibility is the one that distinguishes this from a fancy ticketing bot. The harness’s substrate is its own source code, and the LLM has write access to it via PRs that go through human review.

Six logical layers, each addressed below.

Three Gaps a Harness Must Close

The harness is built around one insight: a stateless, reactive LLM is not a production system. The six layers exist to close three gaps that a bare LLM cannot close itself.

The first gap is reactivity: the LLM cannot wake up on its own. Layer 1 (Event Ingestion) closes it — Slack mentions, GitLab CI results, and PagerDuty pages all funnel into a unified dispatch queue. The LLM never polls; it gets called.

The second gap is persistence: the LLM forgets between sessions, loses context between machines, and cannot distinguish between “I addressed that” and “that commit actually landed on origin.” Layer 3 (Persistent State) closes it — in-memory for per-process dedup, local JSON for operational maps, and git-synced workspaces for durable case state. Cross-machine session continuity (covered later) is an extension of the same idea: state lives in git, not in a process.

The third gap is quality: without structure, an LLM reasoning loop writes confident conclusions from weak evidence, declares investigations complete without exhausting accessible sources, and generates MR descriptions that smell AI-written in the first sentence. Layer 2 (Agent Orchestration) and the investigation loop close this gap — structured output contracts, quality gates on every iteration, adversarial review after the loop, and self-healing loops that close the outer cycle of code-change → CI → merge.

What ties the layers together is the compounding property: every closed case makes the next one faster. An approved investigation becomes a knowledge base entry. An approved MR becomes a pattern the gap-analyzer can cite. A finalized case opens an improvement MR against the harness itself. The system narrows the class of tickets it handles poorly, one cycle at a time.

Layer 1 — Event Ingestion

The harness needs to wake up on three external signal sources:

• Slack messages — channel mentions, thread replies, self-DM admin commands.
• GitLab activity — MR review comments, CI pipeline results, pipeline failures.
• PagerDuty alerts — when an oncall page references a ticket the harness has seen, surface the context.

Each has different latency and reliability properties, so we use a different ingress for each.

Two practical things matter here that the textbook description does not show.

Eventual consistency. Slack’s conversations.history endpoint can lag by 1–30 seconds. If our poller saw an empty result and naïvely advanced its cursor, any message whose ts lands in that window would be permanently skipped. We instead freeze the cursor on empty polls and retry from the same oldest timestamp until at least one message comes back. Duplicate processing across retries is filtered out by a per-message-ts dedup set.

**Socket Mode + poller belt-and-suspenders.** Socket Mode gives us sub-second latency for thread replies. But WebSocket connections drop. The poller is the safety net — it catches anything Socket Mode missed using a shared socket-dedup file so the same message isn’t dispatched twice. Without this, messages arriving during connection drops would vanish silently — the operator has no signal they were missed.

The mr-monitor cron loop is different in character. GitLab review comments don’t push events into our pipeline; we have to poll. Its hardest problem turned out to be scaling polling itself: as the harness picks up more long-lived MRs, its 10-minute polling loop accumulates dead-thread cursors that each cost a Slack API call per cycle. Untreated, those generated sustained 429 rate-limit storms that starved real polling. We added auto-eviction of cursors when the upstream API returns thread_not_found — the cursor disappears from the next cycle’s loop. (Pre-fix: 12 retries per cycle for hours; post-fix: zero.)

Layer 2 — Agent Orchestration

This is where the LLM actually runs. The harness spawns workers as transient systemd units, each carrying a JSON payload that names a workflow + a case_dir + a thread context.

Each pipeline is a long-running invocation of Claude Code (the brain) with a curated prompt and a writeable case_dir. The case_dir is the agent’s scratch space: it contains the case’s TASK.md, accumulated followup_transcript_*.md files, generated gap_report*.md files, and intermediate JSON artifacts.

The choreography matters. We don’t have one agent that does everything — we have several specialized ones, each chained at a defined hand-off point:

• **oncall_run runs the initial investigation — up to 20 reasoning iterations**, quality-gated — with an explicit self-critique step at the end. Output: an ANALYSIS.md (what’s wrong) and TASK.md (what to do).
• **case_followup* is the long-lived chat agent. The operator can DM-reply on a case thread for days; each reply re-invokes case_followup with the full conversation history. Crucially, case_followup also invokes gap_analyzer as a sub-agent every N replies (self-improvement loop covered below) — that’s the loop that produces `gap_report.md` artifacts pointing at the harness’s own code.
• **finalize_case is the consolidator. When a case goes idle for four hours OR an operator runs admin: finalize <case>, it collects every gap_report, filters out the false positives, and opens one MR** with the surviving gap fixes. The MR targets the harness’s own codebase, not the original ticket’s repo. This is the self-improvement loop in concrete form.
• **dev-agent** is the doer. It runs in a per-case git worktree, makes the code change, pushes, watches CI, fixes CI if it breaks, addresses reviewer comments. Three task modes: address_review, fix_ci, and task (the original implementation).

Two things about this graph are non-obvious.

First: everything happens in worktrees, not the main repo checkout. Concurrent cases can be in flight at the same time without stepping on each other. The poller / dev-agent dispatcher passes the worktree path in the payload; the LLM never sees the main checkout.

Second: case_dir is the single source of truth. State files in ~/.harness/ (we’ll see them next) map a Slack thread → a case_dir. The agent always operates relative to case_dir. This makes the rest of the system much simpler: rename the case_dir and as long as the map is updated, every workflow follows.

The orchestration graph above shows what runs. The section below explains how it runs — the mechanics that turn a Slack command into a completed investigation.

The Autonomous Agent Loop in Detail

The single sentence answer for “what does this thing actually do” is: the operator types one Slack command, and a closed loop of iterated reasoning, self-critique, and tool use runs without further intervention until either the work is done or the system honestly admits it cannot proceed. That sentence hides almost every interesting decision. This section unpacks it.

Four kinds of agent loop run inside the harness, each tuned to a different problem shape:

These are the shapes. Below is the substance — each mechanism in the order it runs. The diagram above maps to all of them; the nodes will make sense after reading through. (Gate codes: G1–G4, N1–N3, N5, A1–A3 · D1–D15)

Before the Loop: The Pre-Commitment Lock

Before the first Claude call fires, two safeguards run. The first is structural.

Before the loop starts: the pre-commitment lock. The first thing the on-call investigation pipeline does — before Claude runs a single investigation iteration — is ask Claude to commit to a hypothesis_slate: at least six candidate explanations for the problem, plus four to five evaluation criteria for choosing among them. These are frozen in kickoff_precommitment.json and treated as immutable for the rest of the investigation.

This is the most important methodological safeguard in the system, and the least obvious. Without it, an LLM reasoning loop converges toward whatever hypothesis its early evidence most supports, then retrospectively frames all subsequent evidence to confirm it. The pre-commitment lock doesn’t prevent convergence — it forces the investigation to begin with a full hypothesis space and document the path from that space to a conclusion. The adversarial reviewer (Phase B) checks whether the conclusion is the one the initial slate would have predicted given the evidence, not just whether the conclusion is internally consistent.

Also before the investigation loop begins: the empirical anchor (Phase C). On the first iteration only, the harness runs a set of OS-level commands without LLM involvement — journalctl error tails, disk and memory snapshots, log file discovery. The output is injected as raw evidence into the first investigation prompt. This gives Claude a factual system state snapshot before it has formed any hypothesis. It's non-blocking (errors are silently swallowed) and takes less than a second; its value is that Claude's first reasoning step starts from ground truth rather than from the problem description alone.

Structured Output: The Completion Report Schema

Every iteration ends with a parseable JSON contract — the completion_report — that the harness uses to decide what happens next. The shape is fixed:

{
  "status": "IN_PROGRESS" | "BLOCKED" | "COMPLETE",
  "confidence": 78,
  "open_questions": ["..."],
  "unchecked_sources": [{"name": "...", "access_status": "..."}],
  "contradiction_register": ["..."],
  "assumption_register": ["..."],
  "adjacent_problems": [{"summary": "...", "status": "...", "blocker": "..."}],
  "draft_response": "the user-facing Slack message"
}

Free-form prose is allowed before and after this block, but the parsed report is what the gates evaluate and what state files persist. This is the single biggest leverage point we found for keeping LLM output usable in a pipeline: trade some prose latitude for a parseable contract. Without the schema, every downstream step would need to re-extract facts from prose. With it, the gates run as deterministic Python.

Quality Gates: The Structure That Makes Confidence Mean Something

The confidence number would be marketing noise if Claude could just write confidence: 95 without backing it up. The harness enforces a structural contract via a set of quality gates that run on every iteration’s output, blocking advancement unless evidence and confidence agree.

Three families of gates run after every Claude turn:

• G-gates (logical consistency) enforce that reasoning doesn’t contradict itself across rounds. The clearest example: G1 fires if open_questions grew this round but confidence rose — denominator expansion invalidates any upward move. The family also checks that tension language in prose registers a contradiction, and that neighboring problems are tracked with valid status objects, not placeholder strings. Four gates total.
• N-gates (structural completeness) check that investigation artifacts are fully formed, not stubs. N1: every executable fix action must pair with a verify_action carrying description, expected outcome, and timing. Proposing a fix without specifying how you’d verify it is treated as incomplete work. Three gates total.
• A-gates (assertion ceilings) apply mechanical limits Claude cannot override through prose. A1 computes a hard confidence ceiling: 1.0 − (open_questions × 0.08) − (unchecked_sources × 0.05) — no matter what Claude writes, the gate caps it arithmetically. Other A-gates block COMPLETE status when adjacent problems remain open, require a non-empty assumption register, and mandate that at least one proposed action maps directly to the root cause. Seven gates total.

Two additional spot-checks: EQ1 requires root cause nodes to cite hard (non-INFERRED) evidence or accept a 60% cap. P7 reduces the ceiling by 7% for each adversarial dimension marked FAIL, floor 40%.

The gate framework contains 19 functions. G-gates fire on what Claude writes, N-gates on what Claude omits, A-gates on what Claude claims.

Gate violations become **pending_guard_notes* that are prepended to the next* iteration’s prompt under ## SYSTEM QUALITY GUARD NOTES (from previous iteration). The instruction is explicit: address all notes before proceeding. This is the forcing mechanism — gates don’t block Claude; they make his previous violations the first thing he reads in the next round. The model’s politeness is irrelevant; the structural enforcement is what moves the investigation.

There is also a context-pinning step (A7) that runs at the end of every prompt builder, independent of violations: unresolved contradictions and unverified assumptions from the current report are reattached to the bottom of the next prompt verbatim. This prevents the well-documented “attention fade” pattern where a model acknowledges a complication in round 3, then silently drops it by round 7.

Confidence as a Concrete Number

The most important design decision is that the agent self-assigns a confidence: <int> in every iteration’s structured output, on a 0–100 scale. That number isn’t ceremonial — it gates real behaviour:

• **< 70**: cannot exit the investigation loop. The pipeline forces up to three additional rounds, injecting “you said you were done but confidence is below the bar” as a guard note.
• **≥ 70 AND status: COMPLETE AND no open_questions: gate opens to the adversarial review** phase (red-team / blue-team).
• **≥ 95: gate opens for the agent to auto-execute non-destructive actions** (Jira comment, status transition, ticket assignment). Below 95, those actions stay as draft suggestions for the operator.
• Drop > 10% across two consecutive rounds: the loop exits with reason degrading — the agent is going backwards, stop and tell the operator.

The full confidence trajectory of every case is written into ANALYSIS.md as a sparkline: 45% → 62% → 78% → 80%. That sparkline is the artifact the operator looks at to triage whether to trust the result. It’s also what the harness’s own self-improvement loop reads when deciding which past cases are worth re-running with a different prompt.

Three Force-Continue Mechanisms

The diagram above shows three branches where the exit check fires ready_for_adversarial or blocked but the loop does not exit. Each corresponds to a force-continue condition — cases where the system overrides the exit signal and injects a guard note instead:

• CONFIDENCE_FORCE: status=COMPLETE + open_questions=[] but confidence < 70 — the model declared done while quietly flagging uncertainty through the number. Forces up to 3 more iterations (these rounds are counted within the 20-iteration budget, not added on top of it).
• CAUSAL_CHAIN_FORCE: status=COMPLETE + confidence ≥ 70 but causal_chain_complete != true — a symptom-level finding without a traced root cause. Forces up to 3 more iterations (these rounds are counted within the 20-iteration budget, not added on top of it).
• G_BLOCKER_FORCE: status=BLOCKED but accessible technical sources were skipped or marked unavailable via inference rather than an actual error. The investigation cannot exit as blocked without genuinely exhausting what it could reach. Up to 3 continuations (counted within the 20-iteration budget, not added on top of it).

Together these three prevent the most common premature exits: confident-sounding uncertainty, symptom-level completion, and hallucinated unavailability.

Adversarial Review: The Red Team Inside

Once the on-call investigation pipeline declares itself done with confidence ≥ 70, control passes to a separate Claude invocation playing the role of an adversarial reviewer. The decision tree for what happens inside that review:

The reviewer doesn’t get to see the same prompt as the investigator — it gets the analysis text plus an explicit checklist of 15 evaluation dimensions (D1 through D15: assumption validity, evidence chain, causal reasoning, alternative hypotheses, etc.). It produces a structured verdict:

Three exit conditions out of adversarial review:

1. Critical issues found → critical notes get injected into the next investigation iteration as pending_guard_notes, and the loop restarts. The investigator now has explicit "the reviewer thinks you missed X" feedback.
2. Only minor issues, first pass → loop back one more iteration with the minor critique.
3. Two consecutive minor-only passes OR a clean pass → exit complete. The investigation is published.

The choice to make adversarial review a separate Claude invocation with a different prompt is deliberate. The same model playing a different role catches a meaningful fraction of self-flattery that single-prompt iteration misses. We’ve seen confidence claims revised down from 90 to 65 between an investigation and its adversarial review on the same case, the reviewer pointing at exactly the assumption the investigator had glossed.

There is a second, structurally different review running alongside this one: the red team review (G5). Where the adversarial reviewer reads the full investigation narrative, the red team reviewer sees only two things: the original problem statement and the final conclusion. No intermediate reasoning, no evidence trail, no prior rounds. It produces an independent hypothesis and flags any gaps between what the investigation found and what a cold reader would have predicted.

The purpose is cognitive isolation, not coverage. The adversarial review checks whether the investigation is structurally complete (D1-D15). The red team checks whether the conclusion would have been reachable without the investigation’s framing — whether the reasoning is genuinely grounded in the problem, or whether the investigation anchored on an early finding and the conclusion followed from the anchor rather than from the evidence. Both run as separate Claude calls; both must pass before the case is marked COMPLETE.

Five Honest Exit Reasons

The investigation loop can end for five reasons, and which one fires is logged and shown back to the operator:

exit=complete       confidence=89%  iterations=12  elapsed=13min
exit=blocked        confidence=62%  iterations=4   elapsed=4min    (cannot proceed without info)
exit=stalled        confidence=58%  iterations=7   elapsed=22min   (two rounds, zero new facts)
exit=degrading      confidence=42%  iterations=5   elapsed=8min    (dropped >15% twice)
exit=timeout        confidence=73%  iterations=18  elapsed=50min

Of these, blocked is the most operator-actionable: the agent has registered explicit blockers in unchecked_sources or adjacent_problems, and human judgment is needed to unblock. The other four are technical failure modes the harness handles by surfacing them, not by retrying blindly.Tool Access: What Claude Can Reach From Inside

Tool Access: What Claude Can Reach From Inside

Tool boundaries matter here because confidence gates depend partly on what Claude can actually reach — a gate that says “check unchecked_sources” is only meaningful if the tools to check them are available. Each pipeline launches a Claude Code subprocess with --permission-mode bypassPermissions. There is no inline tool allowlist in the prompt; the boundary is set by what’s actually wired up in the MCP (Model Context Protocol) catalog. The catalog covers five external systems:

A few constraints are worth calling out, because they shape behaviour more than the model card does:

• **Bash is explicitly disallowed in the investigation pipeline.** The investigation loop runs Claude without Bash access (--disallowedTools Bash) to prevent git race conditions — two Claude processes modifying the same working tree concurrently would corrupt state. Bash is available in the dev-agent pipeline, which runs in an isolated worktree.
• Slack write tools are blocked. send_message_to_self and edit_message_to_self are disallowed in every pipeline invocation. Claude cannot post directly to Slack; all outbound communication goes through the draft_response field in the structured completion report, which the harness then posts after validating the exit conditions.
• **bypassPermissions ≠ unrestricted in practice.** The framing prompt for each pipeline encodes hard rules: ANALYSIS.md is editable but message.json is immutable; writes outside case_dir are auto-reverted via git diff after the run; PROD DB writes must wait for explicit operator confirmation per command instance. Safety is at the framing layer + a post-run boundary check, not at the tool-call layer.
• Cascading fallbacks are encoded in prompts, not hardcoded. When search_datadog_services returns permission-denied, the prompt instructs the agent to pivot to mcp__glean_default__search with Jira keywords as proxy. The fallback chain is part of how the SOP (Standard Operating Procedure) gets enforced, not part of the runtime.
• Read-only tools run in parallel; write tools run serially. The streaming-mode subprocess parses tool_use events as they happen and dispatches them concurrently up to 10 at a time for read operations (Datadog queries, Glean searches, file reads). Anything that writes — Bash with side effects, edits, MR comments, DB writes — runs one at a time. This lets a single investigation iteration do six parallel queries against logs while still maintaining write ordering.

The Same Pattern, Four Sizes

Every pipeline in the harness uses some variant of the same skeleton:

The on-call investigation pipeline runs up to 20 iterations of this. The task analyze phase runs up to 8. The implement phase runs once but loops the build step three times. The dev-agent runs once per spawn with no inner loop, but mr-monitor effectively externalizes the iteration by spawning a fresh dev-agent every time CI fails — up to three consecutive times before escalating to a human.

This consistency is intentional. The four loops are different because they solve different problems, but they share the same vocabulary: confidence is a number, exit is a named reason, self-critique is a separate Claude call, escalation is a Slack DM into the originating thread. An operator learns the shape once and recognizes it everywhere.

The Self-Improvement Layer

This closes the loop introduced earlier: the harness modifies its own substrate, reviewed by a human. Two loops do this, operating at different timescales.

Loop 1 — Per-case structural repair (gap → PR). After each investigation closes, gap_analyzer identifies where the harness underperformed: data sources skipped, wrong tools chosen, pipeline bugs, missing tests, format failures in MR descriptions. Gaps are filtered by category and confidence, then passed to finalize_case. Four hours after the case goes idle, finalize_case spawns patch_suggester in a dedicated worktree. The result is one PR against the harness’s own codebase — editing code files, prompt templates, or SOP sections. The PR requires a human merge. It does not require a human to write it.

Loop 2 — Cross-case behavioral reinforcement (failure mode → SOP suggestion). Each case’s auto_retrospective step catalogs which failure modes fired — categorized by type: data-gathering failures (did the investigation skip accessible sources?), quality gate violations (did confidence claims contradict the evidence?), and analysis completeness failures (were root causes traced all the way through?). These accumulate in the case registry per project. A weekly cron (knowledge-updater) counts frequencies across the last ten cases and generates SOP suggestions when any mode crosses a threshold: 3 out of 10 triggers a suggestion, 6 out of 10 an alert. The suggestions are specific: “‘skipped direct DB query’ triggered 3 times this month — SOP updated: always run the direct table query before inferring schema.” A human reviews and applies them. This loop is deliberately not auto-applied — automated decisions that compound mistakes are a known failure class in self-modifying systems.

Why this makes the harness personalized. The SOP is the harness’s model of how your team investigates. It starts generic. As cases accumulate and failure modes repeat, the kb-updater builds a picture of this project’s specific failure patterns. After 20–30 cases, a project’s SOP looks nothing like a generic template — it reflects the specific data sources that get skipped, the tools that get misapplied, the investigation framings that consistently miss the mark in this codebase. Each project has its own SOP. Team A’s failures teach Team A’s SOP; they don’t contaminate Team B’s.

This is what makes the learning durable. A context window survives one session. A merged code change is versioned, reviewed, tested, and permanent. A SOP update is operator-reviewed and carried into every future investigation. The substrate improves; the next case runs on a better harness than the previous one.

Layer 3 — Persistent State

Agent orchestration is what runs per-event. Persistent state is what survives when the process restarts, the machine reboots, or the operator switches from laptop to CVM. The harness’s state lives in three layers, with sharply different characteristics:

In-memory caches are tiny and rebuildable. They live inside the poller / keepalive process and are reconstructed on restart from local state.

Local state (~/.harness/*.json) is the operational map. The key files:

• task-thread-map.json — Slack thread_ts → JIRA id + absolute task_dir path.
• oncall_thread_map.json — same shape but for on-call investigation cases.
• monitored-mrs.json — list of MRs the harness opened + their CI status.
• patch-pr-state.json — per-case finalize status (merged, no_gaps, failed, …).
• oncall-state.json — kill switch flag (more in Layer 6).
• socket-dedup — recent message ts's processed by Socket Mode.This layer is local to the machine and not shared. Restart-safe but if the machine dies, this is lost.

Git-synced state is the durable layer. Case workspaces (case-workspaces/<ticket-id>/) commit and push to a private git repo on every interesting state change. A second machine cloning the same repo can resume the same conversation context. This is how we get cross-machine session continuity — the operator can chat with the harness from their laptop or from a remote dev VM without losing context.

We added a third role for git-synced state: memory consolidation. Long conversations accumulate dozens of followup_transcript_*.md files. After enough activity, a sub-agent reads them all, rewrites TASK.md to include the distilled state, and the workflow stops feeding individual transcripts to subsequent turns. Without this, prompts grow past the model’s context window. With this, there is a real risk we’d over-compress and lose information — and we did, once, dropping a 14KB document to 587B. That incident is responsible for three layers of protection now: a “UPDATE not REWRITE” prompt template, an output-size guard that rejects writes under 50% of input size, and a per-case lockfile to prevent concurrent consolidation runs racing.

The lookup graph from a single Slack thread to the case’s TASK.md is small but worth showing:

Two state files, same value shape: an absolute path to the case workspace. Every workflow eventually resolves to a case_dir, then operates relative to it. Renaming the workspace root — which we did once mid-project — is a single migration script that rewrites the absolute paths in both maps in lockstep with the directory move.

Durable state is the foundation. What happens when operations on top of that state fail? The harness doesn’t surface failures to the operator — it recovers from them.

Layer 4 — Self-Healing Loops

This is where the harness stops being a notifier and starts being autonomous. Three loops, each closed without operator intervention.

The first two are fully autonomous. The third is intentionally not — OKTA refresh requires a hardware-bound MFA prompt, so the agent stops itself and asks for human action with a 30-second runbook in the DM. This is human-in-the-loop by design, not by capability gap.

A subtle property of these loops: retry budgets and self-recognition. CI auto-fix has a per-MR cap (3 consecutive failures → escalate to operator). Reviewer auto-reply uses author-aware dedup — if the bot’s last note in a discussion is more recent than the reviewer’s, the loop doesn’t re-fire (the bot already replied). The OKTA helper has a 15-minute approval window. Without these the loops occasionally enter spirals — early in the project, an MR with a flaky integration test went through 80 no-op “fix CI” commits in a single afternoon before we caught it.

Three-Layer Self-Heal: When the Deterministic Rule Isn’t Enough

The CI auto-fix and reviewer-reply loops above are end-to-end recovery flows. Underneath those flows, every primitive operation that can fail (push, rebase, MR comment, branch reset) sits on a separate, smaller 3-layer architecture for per-operation self-recovery. This is a different axis from the 3-layer safety model in Layer 6 — that one is about restraint (what the agent is allowed to do), this one is about recovery (what happens when an allowed operation fails). Easy to confuse; worth keeping straight.

L1 — Deterministic guard. Sub-second, rule-based, always-on. For push, that’s _pre_push_rebase(repo_path, branch): every time the harness is about to push, it first git fetch origin and git rebase origin/<branch>. If the rebase is clean, push proceeds; if there’s a conflict or the fetch fails, L1 aborts cleanly, enumerates the conflict files, and hands off to L2. L1 cannot do anything creative — that’s the point. The rule is small enough to reason about and bounded enough to run on every operation without a token cost.

L2 — Agentic self-heal loop. When L1 can’t fix the situation, the harness spawns a constrained Claude session (_self_heal_push_loop) with the full failure context: git state, conflicting files, the original task that motivated the push, and a tight set of red lines (never force-push to a protected branch, never drop unpushed commits, never blind-pick a conflict side). Three constraints make this safe:

1. Bounded retry. ≤3 attempts per incident, with a 4-minute timeout per Claude turn. After every Claude action the harness re-runs the real git push — Claude’s belief about whether the conflict is resolved is never trusted; git push exit 0 is ground truth. Claude can claim “fixed it” all it wants, but if the next push returns non-zero, the loop counts that as a failed attempt.
2. Confidence gate ≥ 70. Borrowed verbatim from the on-call investigation pipeline’s oncall_run config: Claude self-rates 0–100 on each turn, and below 70 it must bail rather than act. Below-floor self-reports do not get to touch the working tree.
3. Truthfulness invariant preserved. L2 success means a real commit landed on origin. The downstream reply gate (the “Done — updated” reply to the reviewer) is still hard-wired to git push exit 0, not to anything Claude says about its own success. L2 cannot bypass the reply gate; it can only feed it a real push.

L3 — Operator escalation. L2 bailed (low confidence) or exhausted attempts → the harness DMs the operator with the full L2 trail: every attempt, every action Claude took, every confidence rating, the final git status. The operator picks up from a fully-instrumented state, not from “something went wrong.” Time scale: hours, not seconds.

Two principles worth surfacing:

• L2 is per-incident, not always-on. L1 runs on every push because it’s cheap. L2 burns tokens, so it only fires when L1 has already failed — the cost is gated by the rare case, not the common one.
• Ground truth on the wire. Whatever Claude believes about its work, validation re-runs the real operation. For push, that’s git push exit code. For test fixes, that’s running the actual test. For MR replies, that’s the reply gate checking the push exit. The agent’s self-report is never the ground truth — the harness re-checks against the wire.

This pattern generalizes beyond push. Any harness operation that has a clear deterministic recovery (L1), a constrained agentic recovery (L2), and a clean escalation surface (L3) can be wrapped the same way. The shape is what makes the difference between an autonomous loop that recovers from its known failures and one that needs an operator the moment the world shifts under it.

Layer 5 — Observability

The harness writes to four observability surfaces, each tuned to a different consumer:

• Live Slack updates — when an agent is running, a placeholder message in the case thread cycles through status messages (Reading TASK.md → Searching codebase → Drafting patch) and resolves to Done or Failed: <reason>. This is the streaming reporter; it edits a single Slack message so threads don’t get flooded.
• Structured logs — ~/.harness/logs/agent.log and ~/.harness/logs/error.log. Every spawn, every chain dispatch, every retry attempt gets a structured event. This is what we grep when diagnosing.
• Slack DMs for terminal events — MR opened, MR merged, OKTA expired, CI escalation. These are first-class operator notifications, threaded under the case if applicable.
• System log — ~/.harness/logs/system.log for cross-cutting events: deploy completions, watchdog actions, sync status.

We made an explicit choice not to build a dashboard. Slack and grep are the consoles. A dashboard adds a third surface to keep current and is the first thing engineers stop reading when it gets stale. The streaming-Slack-message pattern is enough operational signal that we have never wanted a UI on top.

MCP Health Monitoring

The harness depends on five external MCPs (Slack, Datadog, Jira/Confluence, Glean, GitLab). Each has its own OAuth token, its own refresh cadence, and its own opaque failure modes. Untreated, MCP token expiry shows up as a Claude session that mysteriously returns “I don’t have access to that tool” with no operator-visible error.

A small service — mcp-watchdog — closes that gap. Every 10 minutes it polls each MCP’s health endpoint, attempts a no-op tool call, and if it fails:

1. Attempts a silent token refresh.
2. If refresh fails, posts a Slack DM to the operator: “Atlassian MCP needs reauth — click here.”
3. After two consecutive disconnects (the DISCONNECT_CONFIRM_THRESHOLD), escalates with a :warning: reaction on the most recent ticket.

Without this loop, an expired token surfaces as “the agent is stupid today” — a class of failure that’s incredibly painful to diagnose because nothing in agent.log says “token expired”; it says “tool returned empty result.” Making auth health a first-class observability surface is a step we should have built earlier.

Layer 6 — Human-in-the-Loop Control

The most-used surface, and the one we most underbuilt initially. All operator interactions go through a single channel — the self-DM — organized into six command families:

• Task — admin: task <jira> · --auto for unattended execution. Triggers the full analyze → implement → MR pipeline.
• On-call — admin: oncall-run <url> starts an investigation from a Slack thread. oncall-toggle enables or disables the auto-monitor.
• MR control — admin: mr rescan re-polls a specific MR. mr pause/resume gates the auto-reply loop on that MR.
• Pause — admin: oncall pause [duration] silences automatic dispatch. oncall resume/status lifts it or shows current state.
• Finalize — admin: finalize <case> consolidates gap reports and opens the improvement PR against the harness.
• Registry — admin: register-thread manually associates a Slack thread to a case. close-case marks it done.

The pattern that took the longest to get right is the pause kill switch. The first version paused everything — including admin commands themselves. The operator who paused the system would then be unable to resume it. The second version paused too narrowly and the operator’s intent (“stop the noise”) was not honored. The current version is calibrated to a precise distinction: pause silences automatic dispatch driven by other people’s activity, but never silences the operator’s own admin commands or in-thread interactions. The dispatch sites in the poller divide cleanly into two groups: those that represent automatic activity driven by other people (gated when paused), and those that represent the operator’s own intent (always-on).

This calibration was learned the hard way: shortly after deploying the first pause, a reviewer-comment auto-reply silently fired on a different case and we missed it because the operator’s pause appeared to be in effect. The current design treats pause as directed silence — quiet the system while preserving operator agency.

A picture clarifies the split:

The rule: operator’s own actions never silenced; other people’s activity gated when paused. That’s all admin: oncall pause controls.

Three Layers of Safety

The pause kill switch is the most visible safety surface, but not the only one. Two further layers enforce safety without any runtime configuration — one in code, one in deployment convention:

The Slack channel allowlist matters more than it sounds. Early in the project we had a near-miss where a misconfigured Slack token had write access to internal team channels — meaning the agent could have posted directly to teammates, bypassing the operator entirely, with no way for the team to distinguish it from a human message. After that, every outbound chat.postMessage validates the destination channel against an allowlist before sending. The only allowed destination is the operator's self-DM (the channel ID of the operator's own DM with the bot — a single hard-coded value). If the agent ever tries to message a teammate directly, it must surface the content back to the operator who can choose to forward. That's an architectural choice we keep coming back to: the agent communicates outward only through the operator's mailbox.

Beyond the Six Layers: The Pieces That Make It Production

The six layers above are the load-bearing structure. Several mechanisms sit outside the layer taxonomy but are equally critical — these are the parts that, in our experience, are the difference between a harness that demos well and one that actually runs uninterrupted.

Multi-Project Routing via a Single YAML

The harness serves two projects today, each with distinct operational rules, and adding a third requires zero code changes — just an entry in agent-config/projects.yaml

projects:
  team-a:
    sop: "team-a/SOP.md"
    kb: "team-a/KNOWLEDGE-BASE.md"
    dfr_threads_dir: "team-a/threads"
    task_dir: "task-workspaces"
    slack_channels: ["support-channel-a", "support-channel-b"]
    jira_prefixes: ["AAA", "BBB", "CCC"]
  team-b:
    sop: "team-b/SOP.md"
    kb: "team-b/knowledge-base/kb.md"
    dfr_threads_dir: "team-b/threads"
    task_dir: "task-workspaces"
    slack_channels: []
    jira_prefixes: ["XX", "YY"]

A resolve_project() function routes every incoming signal by precedence: Jira prefix > Slack channel > default. Each project gets its own SOP file, KB file, and directory tree. The investigation prompt assembly automatically picks up the right SOP and KB based on which project the signal resolves to. This is the trick that lets a single harness instance serve two distinct teams with two distinct sets of operational rules.

The pattern generalizes. Any org with multiple product teams running on the same harness can use the same approach: declarative config, no per-project code branches. It also forces a clean discipline — anything specific to one project must live in its SOP file, not in code, because code is shared across projects.

Cross-Machine Session Continuity

The operator works from two machines: a laptop and a long-lived CVM in the data centre. The same Claude session needs to be resumable from either. That’s hard, because Claude Code persists session state to local disk by default — restart from another machine and the session is invisible.

The harness solves this with a git-backed session store:

The mechanism in three parts:

1. Sessions are git-tracked. Each session’s .jsonl + .meta file is committed and pushed to a private repo on every interesting state change. The repo is the source of truth.
2. A 5-second background fetcher runs on each machine, keeping the local copy current. This pre-pulls context so a Claude session resume doesn’t pay network latency on every keystroke.
3. Session UUID lookup by CWD. When Claude starts in a directory, the session-start.sh hook scans the local .meta files for one whose CWD matches and resumes the corresponding UUID. The same session UUID will resume on whichever machine the operator is on.

The sync hook (user-prompt-submit.sh, which fires after every Claude prompt) does three operations in a strict order, serialized via flock:

1. Fetch + reset. git fetch origin main && git reset --hard — now the local active.jsonl contains the other machine’s transcript.
2. Extract delta before overwriting. Read the watermark (how many lines we’ve already injected from the other machine), extract only the new lines since then, save to a temp file. This step must happen before step 3 — an earlier bug had them swapped, so the inject always read the machine’s own transcript back to itself.
3. Push own transcript. Copy the local Claude session transcript to active.jsonl, commit, push. Then update the watermark to the local linecount.

After the flock releases, the delta from step 2 is injected into Claude as a system message: “The other machine added these N messages since we last synced.” The next prompt sees both machines’ context.

A .linecount file tracks the true total line count separately from the file size, because wc -l gives wrong answers when the transcript has been truncated for push (we truncate to 1,000 lines before pushing to avoid the 25 MiB git limit). Without .linecount, the watermark would reset on every truncation, causing the inject to re-deliver thousands of already-seen lines.

The flock-serialized git operation prevents index.lock corruption from concurrent pushes (we hit this once with a stuck session start and a concurrent meta-enricher run). A throttle (configurable, default 30 seconds) skips the full sync on rapid keystrokes, avoiding per-character git latency.

The “git-as-state-store” pattern is the broader idea worth lifting from here. We treat git not just as a code repo but as a **CRDT-like durable log of agent state**, with the operator’s two machines as eventually-consistent readers.

One mechanism is closely related to cross-machine continuity: memory consolidation. The memory_consolidate step (rewriting transcripts into a clean TASK.md/ANALYSIS.md) doesn’t fire on a fixed timer. A separate service — **consolidate_trigger** — evaluates each turn against four signals: a commit SHA or decision verb in the transcript, a just-closed review thread, a case workspace exceeding 5KB of accumulated transcript, or at least one hour since the last consolidation. Any of these fires consolidation. The principle is that information stops accumulating after a decision is made, not after a fixed amount of time — the four signals together approximate “is the conversation at a natural rest state right now?” without requiring the operator to mark it.

Worktrees Are Not an Implementation Detail

Earlier we said “everything happens in worktrees.” That’s worth expanding because the architectural choice it represents is genuinely load-bearing.

The original design had the agent emit a diff, and the harness called git apply in the main repo checkout. This had two failure modes that bit us repeatedly:

1. Brittle diff headers. Claude would emit a diff with line numbers that were correct at generation time but wrong after a concurrent commit landed. git apply would reject the whole patch.
2. Main tree gets stuck. If git apply --check failed midway, the main repo would be left in a half-applied state. The next case to run on the same repo would inherit the dirty tree.

The fix is the worktree_manager service. Before any agent runs that needs to modify code, the manager creates a sandboxed worktree at ~/.worktrees/case-<case_id>-<stamp>/ checked out from a clean branch. Claude operates inside that worktree using its native Edit/Write tools — no git apply, no diff text round-trip. The agent works the same way a human engineer does: open a fresh branch in a fresh checkout, edit, commit, push.

The worktree is torn down when the MR is merged or closed. If the agent fails partway, the worktree is the only thing affected — the main checkout and other concurrent cases are untouched.

This is what makes safe concurrent agents possible at all on a single machine. Without per-case worktrees, two cases on the same repo would race on filesystem state. With them, they’re isolated by construction.

Dogfooding: The Harness Fixes Its Own Pull Requests

The harness’s source lives in a private GitHub repo. When the gap-report → finalize → MR loop produces an improvement MR against the harness itself, that MR has its own CI: pytest, linters, Python syntax check. If CI fails on a harness-improvement MR, a separate service — pr-ci-fixer — picks it up the same way mr-monitor picks up failures on customer MRs.

The result: the harness self-heals not just the code it writes for tickets, but also the code it writes about itself. The MR that adds a new retry budget can fail CI because the budget breaks an existing test; pr-ci-fixer notices, spawns a Claude session in a worktree, patches, re-pushes, and the human reviewer sees a green MR they can just merge. We’ve watched this loop close on real harness changes more than once.

Dogfooding the loop on the harness’s own source is the load-bearing test of “is this actually production.” If the agent can keep its own CI green on its own pull requests, it’s probably ready to be trusted on customer ones too.

Test Architecture: Five Layers for Five Failure Classes

The harness has 4,656 test functions across 126 files. That number is misleading without a framework for understanding what each test catches, because not all tests are equal.

The L0–L4 framework assigns every test to one of five layers based on what it isolates:

• L0 (pure boundary): No mocks, no filesystem, no IO. Tests pure logic — gate functions, JSON extraction, confidence ceiling math. A test that passes at L0 is guaranteed correct regardless of deployment environment.
• L1 (file isolation): Real filesystem in a tmpdir, real threading, no mock for pathlib. Catches concurrent write races and state file atomicity bugs that L0 misses by design.
• L2 (service isolation): Real subprocess execution with mocked external APIs (Slack, GitLab, Claude). Catches integration bugs between service components — dispatch routing, payload schemas, state file transitions — that L1 misses because L1 only isolates filesystem behavior.
• L3 (sandbox): Real git repo (via subprocess), fake Slack and Claude responses. Tests full workflow sequences — task registration, implement-phase kickoff, pending confirmation — against a real git object model.
• L4 (integration matrix): Six sub-categories covering actor contracts (L4a), admin × role matrix (L4b), cross-run state consistency (L4c), inter-service chains (L4d), external actor roles (L4e), methodology gaps (L4f), and temporal sequencing (L4h/L4i).
• L4g (production replay): Thirteen tests that reproduce specific production incidents as regression tests. These are written after a bug is found in production — the test encodes the exact actor, context, and state that triggered the bug, and is kept permanently to prevent regression.

The L4g tests are the most expensive to write and the most valuable to keep. They encode institutional memory about failure modes that are otherwise invisible until they recur.

The Anti-AI-Flavor Style Guide

Every MR the harness opens passes through one more filter before being submitted: a style SOP that exists specifically to make the output not look AI-generated. The rules are blunt:

• Zero AI attribution. No Co-Authored-By: Claude, no “Generated with Claude Code”, no [bot] suffix in commit messages. The MR is the team’s, not the model’s.
• No fill-in-the-blank headings. “What this MR does / why we need it” templates are banned. The MR description is written as continuous prose with a concrete Changes: bullet list, the way an engineer would write it manually.
• No bold inline tag structure. **Root cause:** followed by a paragraph is banned. So is **Fix:**, **Risk:**. Tag-style bolding is the single most reliable AI-flavor tell.
• Proof of work must be command-reproducible. Not “verified locally” — pytest tests/test_foo.py::test_bar -v with the actual command and the relevant 3 lines of output.

The SOP is injected into the patch-generation prompt at the point where the MR description is composed. It exists separate from the prompt template because it’s operator-curated: the SOP can be edited at any time and the next MR picks up the new rules, no daemon restart, no code change. The Gap Loop can even propose edits to the SOP itself — meta-rules about how the agent should write MRs end up as MRs against the meta-rule file.

The reason this matters more than it sounds: nothing kills trust in an automated MR pipeline faster than the team realizing every MR description “smells AI.” Once the smell is there, the merge bar shifts up, the agent’s actual signal gets buried, and operators start wanting to rewrite descriptions manually — which defeats the point. The style SOP is what keeps the MR descriptions blending in with human-authored ones, which is what keeps the loop functional.

End-to-End Walkthrough: One Ticket, One MR

Here’s what happens when a real ticket arrives. Times are approximate.

End-to-end, this loop typically completes in 30–90 minutes of wall-clock time for a small-to-medium ticket. The operator’s total active engagement is usually under 5 minutes: confirming the analysis, optionally guiding ambiguous decisions, optionally clicking merge.

The investigations themselves accumulate. After 30 cases the harness has 30 gap_report*.md files, each pointing at something the LLM thought could be improved. A bi-weekly admin: gap-patterns run clusters these into recurring failure modes, the next finalize converts the top ones into MRs against the harness repo, and the harness improves itself. This is the self-improvement loop made concrete.

Comparison to Other Harnesses

Several other systems share the harness shape. Where ours overlaps and where it diverges:

Where ours is unusually deep:

• Pre-commitment hypothesis lock — Claude commits to ≥6 candidate hypotheses before running a single investigation iteration, preventing post-hoc rationalization. No commercial harness we’ve found ships an equivalent.
• 19-function quality gate framework with mechanical confidence ceilings — confidence is computed as a hard arithmetic cap (1.0 − open_q×0.08 − unchecked×0.05), not just a self-reported number. Three gate families (logical consistency, structural completeness, assertion ceilings) plus per-dimension adversarial penalties. No commercial harness we’ve found ships this.
• Self-improvement that modifies the harness’s own codebase — gap_reports → filtered PR against the harness’s own code/prompts/SOP. The closest academic equivalent (SICA, arXiv 2504) achieves benchmark gains but is offline research; no production system ships autonomous harness code modification.
• Project-scoped SOP personalization — each project’s SOP evolves based on its own failure mode frequencies. After 20–30 cases, the SOP reflects domain-specific investigation failures. No other production harness implements this.
• Multi-agent orchestration with explicit hand-offs — oncall_run → case_followup → gap_analyzer → finalize_case → dev-agent, each with a defined hand-off schema and isolated worktrees, not one agent doing everything.
• Cross-machine state sync via git-backed watermark injection — the same Claude session resumes on laptop or remote VM with full context, without manual sync.
• Built-in CI auto-fix + reviewer auto-reply loops — only Devin (commercial) ships comparable built-in self-healing; every other framework requires custom wiring.

Where ours is deliberately thin:

• No formal evaluation harness. We don’t score the LLM on a benchmark. Production feedback is the eval.
• No dashboard. Slack + logs.
• No cost meter. Token usage is tracked loosely but not surfaced as a dashboard. Cost is bounded by the human-in-loop confirmation steps.

Outlook

The harness is still evolving — coverage gaps in autonomous loops, cost metering, a formal eval harness — none of them blockers. The system already does what it was built for: convert Slack signals into merged code, address reviewer feedback overnight, self-heal CI, feed its own improvements back into the next version of itself.

The deeper lesson is the meta-loop. Every scar in the harness was caught because a real ticket exercised it. The build loop and the operate loop are the same loop, just at different timescales. Production signal flows back to the harness’s own source; the agent proposes a fix; a human reviews and merges; the next class of bug is one closer to being trimmed.

The system is not done. A production harness that thinks it’s done is one that’s stopped accumulating scars — which is another way of saying it’s no longer running on real load.

Build the loop first. Make it autonomous second. The order matters because an autonomous system without a feedback path into its own substrate is just a faster way to ship the same set of mistakes, at scale.