Inside the harness: How Grafana Assistant investigates incidents

Learn how we built a deep investigation harness for root-cause analysis in the Grafana Cloud stack.

LLMs are good at producing a confident root-cause answer in an incident. They’re not as good at noticing when that answer is wrong.

Within a few turns an LLM will name a culprit. Sometimes it’s right, but often it latches onto the first plausible signal and stops looking. Every subsequent turn defends the conclusion instead of testing it. Internally, we’ve started jokingly calling this phenomenon the “smoking-gun” problem, where the model confidently tricks itself and negatively affects the direction of root-cause analysis (RCA).

This is one of the obstacles we’ve had to overcome as we build Grafana Assistant Investigations to be a pragmatic, evidence-backed, and efficient tool for digging deep into an incident.

Assistant Investigations is an agent built around a simple harness for complex SRE investigation workflows on top of your existing observability setup. You ask it about an incident or a symptom; it forms hypotheses, queries your observability stack, captures evidence, and writes a report. The model does the reasoning while the harness owns the loop: what gets injected into context, when state mutates, when the model is allowed to conclude, and what happens when the context window fills up.

In this blog, I’ll cover how we thought about the design for building the harness and the process we used to evaluate it.

An overview of Assistant Investigations in Assistant workspace

How Assistant Investigations adapts

When an incident comes in, the system doesn’t just answer it. First, it pulls assumptions out of the symptoms and creates hypotheses. It then uses tool calling to query the observability stack (Mimir, Loki, Tempo, and Pyroscope) and other context exploration tools to test those hypotheses. Finally, it, reads what comes back and adjusts as needed.

The shape of an investigation is adaptive. Most of the engineering isn’t in any single prompt; it’s in the levers that shape how the model moves through that loop.

The loop is simple yet allows enough autonomy for the model to dig deep into RCA:

Diagram showing Assistant Investigation flow

The model iterates between querying for evidence until each hypothesis lands somewhere definite. It then runs through multiple quality gates to ensure it isn’t eagerly latching onto the smoking gun.

In designing the harness, we’ve found that these four principles help provide the right balance between effectiveness and not being overly restrictive:

• Planning: Assumptions become hypotheses; hypotheses point at the evidence the model needs to find. The model adapts as evidence comes in, but each hypothesis has to resolve before it can wrap up. The plan itself is editable through a dedicated tool, so the working state holds together across long runs.
• Human-in-the-loop: Investigations don’t have to run to completion before someone can act on one. A user can interrupt, redirect, or pause mid-run, and the model picks up from where the steering came in. The same channel surfaces the model’s reasoning in real-time, so working with an investigation feels closer to collaboration than to wait-and-review.
• Context engineering: Three mechanisms do most of the work. Compaction allows the investigation to grow past the context-window, a small set of heuristics nudge the model back on track when it drifts, and a file-based memory offloads context that shouldn’t be compacted. All three share the same principle: simpler prompting techniques give the model better reasoning and autonomy than trying to micromanage its process.
• Evidence attribution: Tools produce evidence, evidence stitches into causal chains. Causal chains are what let the model build confidence in a direction, or pivot when something doesn’t fit. Observability tools are always cited by the model, because evidence is an audit record, not a paraphrase the model might revise later.

Models have improved at instruction-following by an order of magnitude over the past year, but we continue to find that the simpler the prompt and techniques we used for context engineering, the better the performance would get. I’ll discuss this further later in this post.

None of these principles are particularly clever, but we’ve found that, collectively, they’re effective in helping us design a deep investigation harness to keep the model on track and truthful.

Hypothesis-driven

Though not designed specifically for observability, analysis of competing hypotheses (ACH) is an established analytic methodology built to counter the same smoking-gun problem. Rather than setting out to find evidence to support the most likely hypothesis, this approach looks to minimize bias by considering all hypotheses equally.

We’ve borrowed that idea in how we designed the model to create plans. Most agent loops look like a long chain of tool calls; in Assistant Investigations, hypotheses are first-class states.

Each hypothesis has to be resolved through a conclusion gate before the model can write a report. In this context, resolved means one of four outcomes: root cause, symptom, disproven, or blocked. If the model tries to wrap up early, the harness pushes back.

Alongside the conclusion gate, the harness enforces a set of conditional gates before an investigation can finalize:

• Earlier findings are non-disposable. A reviewer scans the full conversation for findings that contradict the leading hypothesis. If contradicting evidence exists, the conclusion has to either refute it with something stronger or fold it in.
• “X is broken” is not a root cause but a continuation point. The harness asks for the trigger (deploy, config change, credential expiry, capacity exhaustion, dependency outage) or an explicit unresolved boundary.
• One incident can have multiple causes. Rank by evidence strength; don’t force a single winner.

The enforcement is structural rather than instructional. The plan, the hypothesis statuses, the conclusion gate, and the reviewer pass are not optional — they’re code paths it has to traverse.

A reminder can soften under context pressure, but a code path cannot. The reviewer is the cleanest example. It runs as a separate inference step with its own prompt and its own gate, matching a pattern called Reflexion, which states that a structurally distinct critique step outperforms asking the model to be careful within a single pass.

A small amount of structure goes a long way. The harness doesn’t overly constrain the model’s reasoning — only the criteria of what counts as a finished investigation. The model picks the queries, reads the results, and writes the conclusions; the harness refuses to accept conclusions that haven’t passed these gates. The structure looks rigid, but in practice it pushes the model to dig deeper into root-cause analysis rather than stopping at the first plausible answer.

An investigation pivoting on new evidence

Surviving compaction

Real incidents may span multiple time windows, dozens of services, and various datasources. A model’s context window can quickly fill up during an investigation.

Compaction is the harness’ response when the conversation grows past a token threshold. Older messages are summarized while a structured ledger of what happened is preserved verbatim.

The harness compacts at a threshold that is measured in tokens. When compaction fires, three things happen that keep continuity:

1. A persisted system log is re-injected. Evidence captures, epoch markers, and significant events survive the compaction. The model still sees “epoch 5 captured Prometheus data at 14:30” after the raw tool exchange is summarized away.
2. The most recent entries are kept verbatim. Step-to-step continuity matters more than aggressive compression on the freshest material.
3. Working memory lives in files, not the context window. Hypotheses, evidence, and notes are file-backed, per chat. They survive compaction and are durable resources the harness can load. The context window is a working surface; the filestore is the source of truth.

The net effect: an investigation can run long enough to actually solve a hard incident without the model losing track of what it already proved. This allows for investigations to grow far beyond the limit of the length for the context-window.

Note: This approach is quite similar to the principle for memory consolidation in episodic memory: the time-tagged record of events survives while raw sensory detail fades.

Evaluation, benchmarking and optimization

A harness has a lot of tuning surfaces: prompts, steering rules, compaction strategy, tool definitions. You cannot tune them by vibes. We focused on three areas: evaluation, benchmarking and optimization.

Evaluation

Evaluation is how we measure quality. We built an offline evaluation framework alongside the harness. Each evaluation case pairs an investigation prompt with an on-call team’s post-mortem from an internal incident that has already been resolved.

We’ve found that real incidents are better evaluation cases than trying to create synthetic cases. We then replay the incident in a controlled environment and verify the agent’s conclusions against the post-mortem. Grading is fact-backed: criteria can attach live queryable facts (Prometheus queries, Loki lookups, Grafana API calls) so the judge is grounded in a controlled-environment.

We grade outcomes, not paths, since multiple paths can be correct. Total token usage and end-to-end time are recorded with every run, because an agent that arrives at the right answer too slowly or too expensively isn’t useful at incident time.

Offline eval is half the picture; we also run online eval against live traffic through our own AI observability tooling, which catches regressions the offline suite can’t anticipate.

Benchmarking

Benchmarking is how we do comparisons, and the cleanest way to compare is to make solutions compete. Bot-programming arenas like CodinGame work this way: you don’t grade a bot in isolation, you drop it into a league against every other approach and let the ranking fall out.

We built an internal arena on the same idea, running the same incidents across successive versions of the harness, our gcx CLI driving a general-purpose agent, and a generic agent wired to the Grafana MCP server, all in our production environment under real conditions. Each approach reaches for evidence differently, and watching them compete on the same incident shows which tools are pulling their weight and which prompts actually steer the model. This allows us to identify gaps, improve and iterate.

The first comparison was the current harness against the initial version it replaced. We saw root-cause correctness more than double, the upstream cause picked up several times more often, and investigations finished in roughly half the time. The biggest single gain was in upstream-cause identification, where the previous version rarely traced back past the immediate failure and the current one usually does.

Most of those gains didn’t come from one big change. Another prototype relied heavily on files for the model’s reasoning, which read clean in evals but was too slow to be practical in an incident. Multi-agent setups and other architectural complications mostly lost to the simpler version with the same constraints.

The pattern repeated often enough that we treat it as the default: if a more complex design doesn’t show a measurable win, the simple one ships. Balancing rigid harness mechanics against speed and accuracy has been the real challenge.

Note: All numbers come from internal benchmarks, with investigations replayed against post-mortems from real incidents and graded with an LLM-as-judge against the on-call team’s findings.

Optimization

Optimization is how we improve. We utilized an internal approach with our evaluation dataset to do both prompt and harness optimization, creating a feedback loop to optimize both. This allowed us to make changes in isolation, run a training set, run a holdout if the training set improves, then promote or backtrack. A couple useful principles we learned throughout the process:

• Structural enforcement beats instructions. Harness gates outperform “you MUST” in prompts.
• Training + holdout. Otherwise you overfit your eval suite without noticing.

One thing to note: We learned that a tight research loop often leads to overfitting, which results in a harness that scores beautifully on the cases you measured and fails on the next incident. The accuracy gap is rarely the prompt; it’s missing context. We’d rather under-constrain the harness and invest in the surfaces that let users extend it (skills, rules, MCP servers) than chase a percentage point with rules that won’t generalize.

What’s next

Assistant Investigations gives engineers a fast way to investigate incidents inside the Grafana Cloud stack: drop in, follow the evidence, reach a conclusion they can act on. LLMs love a smoking gun, though, and a fast conclusion that isn’t grounded in evidence just sends engineers back to verify it by hand.

There’s a lot of interest in the community in making agentic development feel cohesive across the development lifecycle — AI tooling that works better together as part of one connected workflow. Assistant Investigations is one part of that, and the interesting work ahead is in how those pieces come together for the next phase of agentic observability. This will include pulling more context directly into the investigation loop — code being an area we’re researching next.