Building the Reflector: How Self-Correcting Agents Actually Compute What Went Wrong

The gap function primitive, five shipped implementations, and why naive critique loops often degrade performance

Everyone wants their agent to “reflect.” The demo clip that goes viral is always the one where the agent notices its own mistake and fixes it. Read a dozen paper abstracts and it sounds easy. Bolt a critic onto your main agent, loop until the critic is satisfied, ship the system with “self-correcting” in the changelog.

Most of those implementations are broken in interesting ways. Some actively make the agent worse. The difference between a Reflector that helps and a Reflector that hurts comes down to one thing: the quality of the signal it uses to decide what went wrong.

What You’ll Learn in This Article:

• The gap function primitive: The irreducible operation every Reflector computes, expressed as a typed function from spec/trace/evidence to a structured error signal
• Five shipped implementations: How SELF-REFINE, CRITIC, Reflexion/LATS, MAGICORE, and OSC each build a different kind of Reflector, with the tradeoffs that separate them
• The detect-then-patch split: Why real production Reflectors decompose into “where is the error” plus “what change fixes it,” and why naive critique loops conflate the two
• The guardrails that actually matter: Gating, localization, and early stopping, with numbers from CRITIC (14.3% wrong corrections), Reflexion (16.3% false positives on MBPP), and MAGICORE (1.5% drop without a learned critic)

What the Reflector Actually Computes

In metacognitive agent systems, the Reflector is the component that turns observation into an actionable error signal. Across the research literature and open-source implementations, the operation it performs reduces to one thing. Call it the gap function.

gap = g(expectation/spec, observed trace, evidence/tools, memory)
patch = h(gap, policy, constraints)

Both functions matter, and most implementations fuse them together at their peril. The g function takes four inputs (what the task required, what actually happened, what external evidence says, what memory remembers) and computes an error signal. The h function takes that signal plus the agent's policy and constraints and computes a concrete repair.

The gap is not necessarily a scalar. In the strongest implementations it’s a structured object: a critique, plus evidence that supports the critique, plus a suggested repair. That structure is what lets downstream steps act on it intelligently instead of just “try again, but worse.”

This primitive shows up explicitly in at least five research families, each with its own answer to what goes inside the g function.

How Does an Agent Decide Something Went Wrong?

Agents rarely “decide” failure from introspection alone. What actually works is externalized signals that make the gap measurable. Across shipped systems there are six recurring ones.

Environment reward or success flag. Reflexion and LATS both assume tasks where an environment can label success or failure at the end of a trajectory. That binary signal is the starting point. Reflection only gets generated when the task reports failure.

Executable checks. Reflexion’s programming setup generates unit tests, runs them, and uses pass/fail behavior to trigger refinement. This is the cleanest possible gap signal in a sense: if the test fails, something is wrong, and the failure message often tells you what. But the same paper also quantifies the risk of fabricated tests, which I’ll come back to.

Tool-backed verification. CRITIC’s whole design is built around tools producing the wrongness signal. The agent emits a claim, the tool (web search, code execution, toxicity scoring) returns evidence, and the comparison between claim and evidence becomes the critique.

Learned critic scores. MAGICORE uses a Process Reward Model (PRM) that assigns a score to each step of a reasoning chain. Low scores localize where the error is, not just that there is one. That localization matters more than people expect.

LLM-as-judge rubrics. SELF-REFINE uses the same model (or a different one) prompted with a rubric to score its own output along multiple dimensions. This is the cheapest option and also the one most prone to drift.

Learned gap modules. OSC (EMNLP 2025) goes furthest, introducing a learnable gap analyzer that quantifies divergence between an agent’s internal state and its model of what the collaborator or environment expects. The gap is not a prompt, it’s a trained function.

Notice the progression: the further down the list you go, the more decoupled the signal becomes from the agent’s own narration. That’s the whole point.

Five Families of Reflector Implementations

The five research families worth knowing are not variations on a theme. They answer the “what’s in g" question differently.

SELF-REFINE. The simplest option. Generate output, prompt the same model for natural-language feedback on the output, prompt it again to produce a refined version. Alternate FEEDBACK and REFINE steps until a stopping heuristic fires. The gap is textual. The gating is usually “always iterate up to k times.” This is the default in a lot of prototypes, and it’s where most trouble starts.

CRITIC. Instead of asking the model to critique itself, ask a tool. The agent produces an initial output, then issues verification actions (search queries, code execution, toxicity scoring), and the tool’s response becomes the evidence that drives a correction. The gap is a structured object with four fields: a claim, a verification action, the evidence returned, and the critique that binds evidence to a suggested fix. CRITIC reports +7.7 F1 on QA, +7.0% on math reasoning, and a 79.2% reduction in toxicity probability.

Reflexion and LATS. These store reflection as memory. After a failed trajectory, the agent writes a first-person self-reflection and appends it to an experience buffer. The next trial conditions on the buffer. Reflexion frames this as “verbal reinforcement learning.” LATS combines it with Monte Carlo Tree Search so the reflection becomes a “semantic gradient” that steers subsequent rollouts through the decision tree.

MAGICORE. The interesting move here is that the Reflector localizes errors before rewriting them. A learned PRM scores each step of a reasoning chain. The agent only refines the steps that scored low. With GPT-3.5-Turbo, MAGICORE (iter=3) scores 80.9 average vs 77.3 for best-of-k=120. With Llama3–8B it’s 75.6 vs 71.0. The ablation that matters: replacing the PRM with the LLM itself for scoring drops average performance by 1.5%.

OSC. Formalizes the gap directly. It introduces a learnable gap analyzer G_{i,j} and a learnable function f_gap that quantifies divergence between an agent’s internal state and its model of another agent’s state. The output is a scalar that directly gates communication policy. This is the most principled version of the primitive in the literature so far.

The Detect-Then-Patch Split

If you zoom out on the five families, every serious Reflector decomposes into two separable functions.

Detect and localize. “Where is the error?” Failing unit tests in Reflexion. Low PRM step scores in MAGICORE. Inconsistency with retrieved evidence in CRITIC. Low terminal value in LATS. Each of these answers a different kind of “where.”

Generate and plan a patch. “What change would fix it?” NL self-reflection that becomes a hint for next trial (Reflexion, LATS). Draft revision conditioned on a critique (SELF-REFINE). Answer regeneration conditioned on tool critiques (CRITIC). Rewriting only the low-scoring segments (MAGICORE).

The framework-level pattern that LangChain ecosystems have productized is basically “two nodes: main agent then critic/judge, loop if critique exists.” The langgraph-reflection package encodes this explicitly. A main agent runs, a judge node runs a structured check, and the graph loops until the judge returns nothing to critique.

Here is the judge in examples/coding.py:

# Repo: langchain-ai/langgraph-reflection
def try_running(state: dict) -> dict | None:
    model = init_chat_model(model="o3-mini")
    extraction = model.bind_tools([ExtractPythonCode, NoCode])
    er = extraction.invoke([{"role": "system", "content": SYSTEM_PROMPT}] + state["messages"])
    if len(er.tool_calls) == 0 or er.tool_calls[0]["name"] != "ExtractPythonCode":
        return None
    code = er.tool_calls[0]["args"]["python_code"]
    evaluator = create_pyright_evaluator()
    result = evaluator(outputs=code)
    # Gate: only reflect when the check fails
    if not result["score"]:
        return {"messages": [{"role": "user", "content": f"Pyright found: {result['comment']}\nTry to fix it..."}]}
    return None

Two things about this snippet. First, the judge returns None when there's nothing to critique, which is how the graph knows to stop. Second, the gate is a real tool (Pyright), not an LLM grading an LLM. That's what makes it a CRITIC-style Reflector in framework clothing.

Reflexion’s Lesson: Grounding Changes Everything

Reflexion demonstrates large gains when failure is externally legible. On HumanEval Python it reports pass@1 of 91.0 compared to GPT-4’s 80.1 in the same table. On Rust translation and LeetcodeHardGym it posts strong numbers too. Adding the self-reflection step on HotPotQA yields roughly an 8% absolute boost over the episodic-memory-only variant.

But the same paper contains a warning that matters more than the headline numbers. When Reflexion has to generate its own unit tests in settings without pre-existing evaluators, the tests can be flaky. The paper quantifies a false positive rate of 16.3% on MBPP Python compared to 1.4% on HumanEval Python. A false positive here means the test passes but the solution is wrong. The agent believes it is correct and stops.

That 16.3% number is a canonical failure of the gap detector. The signal that drives reflection is miscalibrated, so either reflection fails to trigger when it should or the agent stops early on a wrong answer. And the problem gets worse the more the agent has to fabricate its own evaluator.

The lesson is not “Reflexion is bad.” The lesson is “Reflexion is strong when failure is externally legible and brittle when it isn’t.” Every Reflector inherits that property. Build on a ground-truth signal and you get real improvement. Build on the model’s own opinion of its output and you get a coin flip dressed up in cognitive-science vocabulary.

From Prompted Critique to Learned Critic Signals

A consistent theme across 2024 and 2025 is that prompted self-critique is not a reliable error signal, especially on longer or higher-precision tasks. Two pieces of evidence are worth internalizing.

The NeurIPS 2024 RISE paper (“Teaching Language Model Agents How to Self-Improve”) reports that SELF-REFINE “largely degrades performance” across GSM8K and MATH in their comparisons. Their trained approach beats it by a significant margin turn over turn. This is not a subtle result. It’s “the naive loop actively makes things worse on math benchmarks.”

MAGICORE shows the same thing at finer resolution. In their expanded table, SELF-REFINE at iteration 1 slightly underperforms zero-shot chain-of-thought on Llama3–8B-Instruct (57.1 to 56.3 average). MAGICORE’s PRM-driven targeted refinement substantially improves on the same baseline. The ablation statement that matters most: replacing the PRM with the LLM itself for refinement drops average performance by 1.5%.

That 1.5% number is the whole lesson. The loop structure is almost the same. The only difference is whether the critic signal comes from a learned module or from the same LLM that made the mistake. And the learned module wins.

If you’re designing a Reflector today and you’re budget-constrained, spend your budget on the critic signal, not the loop. A cheap loop with a strong signal beats an expensive loop with a weak one.

Gap Representations and Failure Modes

Across every implementation the gap ends up in one of three shapes.

Scalar gap. A single number. Binary success/fail from Reflexion. Toxicity probability from CRITIC (they use a threshold of 10% to stop). PRM step scores from MAGICORE. Scalars are great for gating decisions (continue? stop? escalate?) and nearly useless for generating a specific repair. You can’t rewrite step 3 because “the score was 0.4.”

Textual gap. A natural-language critique. Reflexion’s first-person self-reflection hints. SELF-REFINE’s multi-aspect rubric. These are great for driving a repair because the language is expressive enough to describe what specifically to change. They’re terrible for gating decisions because calibration is near-random.

Structured gap object. A typed record with named fields. CRITIC’s evidence-plus-critique structure is an early example. LangGraph reflection’s “return None if no critique, else return critique message” contract is another. The natural design recommendation that falls out of these sources is a gap object with fields like error_type, location, evidence, severity, confidence, suggested_fix, and stop. This mirrors how CRITIC and MAGICORE separate localization from correction.

These representations interact with four failure modes that show up empirically across the literature.

Oscillation and runaway loops. Marginal gains diminish fast. CRITIC explicitly notes diminishing benefit after a few iterations and caps its loops tight.

Over-correction and false positives. CRITIC’s own error analysis table shows a 14.3% wrong correction rate in one study. Reflexion’s 16.3% MBPP false positive rate I mentioned earlier is a similar failure shape.

“Everything looks good” syndrome. Prompted critics produce generic feedback that misses the actual error. LATS’s own commentary notes this as a reason to move toward tool-backed or learned-critic approaches.

Reflection as wasted compute. LATS reports HotPotQA token counts of around 173,290 for one LATS configuration versus 210,215 for ToT (ReAct). CRITIC notes roughly 2x latency overhead per correction round. MAGICORE’s plots highlight how naive self-consistency scaling saturates while burning tokens.

The three guardrails that recur as best practice across every paper: gate reflection on a strong signal, localize before editing, and stop early and explicitly. Every serious Reflector I’ve read about does all three.

A Practical Blueprint

Here’s the synthesis. If you’re building a Reflector today, these are the five questions to answer honestly before you write any code.

1. What’s your ground-truth signal? A failing test. A tool execution result. A trained step-level score. A retrieved document that contradicts the claim. Not “the LLM’s opinion.” If the only answer you have is “the model will grade itself,” you’re building SELF-REFINE on math and you already know how that ends.

2. Can you localize before you edit? MAGICORE’s whole contribution is that you can. A step-level score tells you which span to rewrite, not just that the answer is wrong. Localization cuts down the search space for the repair and avoids collateral damage to correct steps.

3. What’s your gap representation? Scalar if you’re gating. Textual if you’re repairing. Structured object (with error_type, location, evidence, severity, confidence, suggested_fix, stop) if you want a pipeline that can scale. Most production teams end up at structured objects whether they plan to or not.

4. How is reflection gated? Reflexively looping is a bug. Real systems gate on a strong signal: failed test, low PRM, tool-evidence mismatch, low terminal value. If there’s no signal, there’s no reflection. That rule alone eliminates most of the regression cases in the literature.

5. How does it stop? Max iterations plus a “no critique” contract plus a stability check. Framework demos like langgraph-reflection encode this at the graph level with a judge that returns None when there's nothing to say. Steal that contract.

Get those five right and the Reflector starts to earn the name. Get any one of them wrong and you’re shipping a “self-correcting agent” that’s worse than the baseline on math benchmarks by exactly the amount a trained PRM would have caught.

The good news is that the primitive is stable now. Every serious implementation since 2023 is a variation on gap = g(spec, trace, evidence, memory) with a different answer for what goes inside g. Pick your signal, build your gap object, constrain your loop. That’s what “metacognitive agent” means when the hype clears.