cascadeflow Made My Agent Ignore Its Own Confidence Score

I built an agent that fixes broken CI pipelines automatically. The plan was simple: detect a failure, recall whether we’d seen it before, let the model draft a fix, open a PR. Standard stuff.

Then I got to the routing logic, and I had to make a decision I didn’t expect: sometimes the agent has to ignore how confident it is.

What the system actually does

The project is called Continuum. It watches a repository’s CI pipeline. When something fails, it doesn’t just throw the error at an LLM and hope. It runs a fixed loop:

1. Detect — pull the failure signature, the branch, the commit, the implicated files.
2. Recall — query Hindsight for anything that looks like this failure. Has this exact shape of bug happened before? Was the fix that worked last time verified, or did it get refuted?
3. Route — decide which model tier gets to touch this incident, using cascadeflow.
4. Investigate — the chosen model proposes a concrete patch: files, line changes, a diagnosis.
5. Propose — open a real branch, commit the change, open a real pull request. A human still has to merge it. The agent doesn’t get write access to main.
6. Verify — poll the repo’s own GitHub Actions run. If it’s green, the fix gets written back into Hindsight as verified memory. If it's red, the fix gets marked refuted, and the incident escalates to a stronger model for a second attempt.

That last branch point — verified vs. refuted — turned out to be the whole story of this project.

The part I didn’t expect to build: memory that can say no

Most memory demos show a system getting more confident over time. Mine had to handle the opposite case too: what happens when the system remembers a fix that turned out to be wrong?

Here’s the actual confidence calculation from the routing service:

if (bestMemoryMatch) {
  const isVerified = bestMemoryMatch.state === 'verified';
  const isHypothesis = bestMemoryMatch.state === 'hypothesis';
  const isRefuted = bestMemoryMatch.state === 'refuted';

  if (isVerified && bestMemoryMatch.similarity >= 0.8) {
    confidence += 0.45;
  } else if (isVerified) {
    confidence += 0.3;
  } else if (isHypothesis) {
    confidence += 0.15;
  } else if (isRefuted) {
    confidence -= 0.25; // avoiding a known bad approach
  }
}

A refuted memory doesn't get deleted. It stays in Hindsight as a negative signal. If the current incident looks similar to a fix that already failed CI once, the confidence score gets actively penalized, not just left at neutral. The agent isn't just recalling what worked — it's recalling what it already tried and shouldn't try again.

This is the piece that made memory feel less like a cache and more like judgment.

Where cascadeflow stops trusting the confidence score entirely

Here’s the part that surprised me while I was writing the routing service. I’d built this whole confidence-weighted system — memory match strength, log complexity, file count, all feeding into a 0–1 score that decides cheap tier vs. capable tier. Then I hit a case where none of that should matter: what if the failing code touches auth/ or .github/workflows/?

A 95% confidence score on an authentication change is not the same risk as a 95% confidence score on a typo fix. So the router checks file paths before it ever computes a confidence score at all:

const matchedPattern = implicatedFiles.find(file =>
  highRiskPatterns.some(pattern => file.toLowerCase().includes(pattern.toLowerCase()))
);

if (matchedPattern) {
  const decision = {
    tier: 'capable' as const,
    confidence: 1.0,
    explanation: `Escalated to Capable Tier due to high-risk file overlap. Pattern matched: "${matchedPattern}". Safety override takes precedence over cost optimization.`,
    escalated: true,
  };
  return decision;
}

If the file path matches a high-risk pattern, the incident goes straight to the capable model tier — full stop, no memory lookup, no cost optimization. cascadeflow is supposed to save money by routing known problems to a cheap model. Here I was writing a rule that explicitly throws that optimization away for an entire category of files. That felt wrong for about five minutes, until I realized the point of a routing layer isn’t “always pick the cheapest option” — it’s “pick the right option,” and sometimes right means ignoring your own scoring function on purpose.

What happens after a fix fails once

The other piece I wasn’t sure would actually work end-to-end: retrying with escalation instead of just retrying with the same model.

retryCount++;
console.log(`[VERIFICATION] Fix failed verification. Retry count: ${retryCount}/${maxRetries}`);

if (retryCount >= maxRetries) {
  incident = await db.updateIncident(incidentId, { state: 'escalated' });
}

When CI rejects a proposed fix, the incident doesn’t just die or repeat blindly. It goes back through investigation with a stronger model tier and the knowledge that the previous attempt was wrong. In practice, that looks like: first attempt proposes a patch, CI fails it, the failing patch gets stored as refuted, second attempt gets a higher-tier model and the refuted context, second attempt passes CI, the fix gets stored as verified. The loop doesn't trust itself on the first try — it has to earn trust from a system outside itself (the repo's own test suite) before anything becomes permanent.

Why the PR step matters more than it looks

I want to be direct about one thing, because it’s the question every engineer asks first: no, the agent does not merge its own code. It opens a pull request. A human approves it. The only thing that happens automatically is drafting the fix and verifying it against CI before asking for that approval. The verification step existing at all is what makes the verified memory label mean something — without it, "verified" would just be a name for "the AI said so," which isn't verification, it's a guess with better branding.

Lessons that generalized past this one project

• A memory system needs a way to be wrong. If your architecture only has “match found” and “no match,” you’re one bad fix away from an agent that confidently repeats its own mistakes. refuted state was maybe 20 lines of extra logic and it changed how the whole system behaved.
• Confidence scores should have a ceiling that isn’t math. Some decisions shouldn’t be probabilistic. File-path-based safety overrides that bypass the scoring function entirely turned out to be more reliable than trying to make the score itself account for “this is auth code.”
• Routing decisions are worth persisting, not just acting on. Every routing decision gets written to the database with its full explanation string — tier, confidence, and the specific memory match that drove it. That audit trail turned out to be more useful for debugging my own code than for any dashboard.
• Retry-with-escalation beats retry-with-repetition. Giving a failed attempt to the same model again rarely helps; giving it to a stronger model, with explicit knowledge that the last approach was refuted, is a meaningfully different problem than the first attempt saw.
• “It works” isn’t the same claim as “it’s verified.” Building the distinction directly into the memory state machine, rather than treating it as a UI label, is what made the rest of the system trustworthy to reason about.

The Hindsight part of this project is the part people notice first — persistent memory always demos well. But the part I’d keep if I rebuilt this from scratch is the cascadeflow routing logic that knows when not to trust its own confidence number. Memory tells you what happened before. Routing has to decide how much that’s allowed to matter right now — and sometimes the right answer is: not at all.