Building a Debate Engine for Classifier Edge Cases

A technical deep-dive into confidence routing, multi-persona debate modes, and pluggable judge strategies in llm-jury

Most writing about classifier performance stays focused on the model itself: better training data, better features, better architectures, better prompting. That is obviously important. But in production, a different question shows up very quickly: what do you do with the cases where the model is not actually sure?

That question was the starting point for llm-jury.

The SDK takes a simple idea seriously. If a classifier is confident, let it decide and move on. If it is not, do not force a brittle answer. Escalate the case to a more expensive but more thoughtful path: a structured LLM debate between multiple personas, followed by a judging step that turns the transcript into a final verdict.

The pattern is inspired by the CEJ paper, When in Doubt, Deliberate: Confidence-Based Routing to Expert Debate for Sexism Detection, which describes a two-stage setup: a primary classifier for high-confidence cases and a persona-based reasoning module for uncertain ones. But what interested me was not just the result in that one paper. It was the architecture hiding underneath it. Once you separate the mechanism from the domain, you get something that can work as reusable middleware.

This article is the architecture tour. I want to walk through how the routing works, what the four debate modes actually change, why the judge strategy pattern matters, and how the LLMClient protocol makes the rest of the system swappable without turning the codebase into a mess.

The core pipeline

At a high level, every classification in llm-jury follows the same shape.

The input goes to a primary classifier first. That classifier returns a label and a confidence score. If the result clears the confidence threshold, the system returns immediately. If it does not, the case is escalated into a debate engine. The debate produces a transcript, the judge turns that transcript into a verdict, and the whole thing comes back as a serialisable result with reasoning and cost metadata attached.

What I like about this structure is that it stays simple even as the internals get more sophisticated. There is still one entry point. There are still only a few meaningful exit paths. The easy cases stay cheap. The hard cases get more reasoning. And if costs run away, the system can fall back gracefully instead of pretending nothing happened.

That fallback behaviour matters more than it sounds. When you wrap a production classifier in LLM calls, you are choosing to depend on components that are slower, more expensive, and less predictable than the model you started with. The architecture has to acknowledge that from day one.

Why confidence routing matters more than people think

The confidence threshold is probably the single most important number in the whole system.

Set it too high and you end up escalating everything. That gives you a lot of debate, but it also gives you more cost, more latency, and less separation between the easy cases and the hard ones. Set it too low and you miss the exact edge cases the debate engine was built to handle.

The reason I like threshold-based routing as a design pattern is that it forces you to treat uncertainty as something operational, not just statistical. A low-confidence prediction is not just an interesting number in a report. It is a signal that the system should behave differently.

That idea is very close to what the CEJ paper formalises. The paper’s two-stage setup uses dynamic routing so that high-confidence examples are classified directly, while uncertain ones are escalated to the expert debate module.

In llm-jury, the default escalation rule is deliberately simple: if confidence < threshold, escalate. But I did not want the whole framework locked into one routing rule, so there is also an escalation_override hook. That lets you inject whatever logic your domain needs: margin-based routing for multi-class classification, rule-based escalation for sensitive categories, or even cohort-based experiments.

That flexibility matters because confidence alone is not always enough. Sometimes what matters is not low confidence in absolute terms, but a narrow gap between the top two labels. Sometimes the model is “confident” in a way that still feels untrustworthy in practice. The override hook exists for that reason.

The four debate modes

One thing I did not want was a system that only knew how to deliberate in one way.

The research paper uses a single structured reasoning flow, which makes sense in a paper. But once you start thinking about production use, you quickly realise there is more than one useful communication topology. Some teams care more about latency. Some care more about explainability. Some want a stress test. Some want a synthesis.

That is why llm-jury has four debate modes rather than one.

Independent mode

Independent mode is the simplest and cheapest. Every persona looks at the input separately, without seeing what the others said.

Under the hood, this is the cleanest mode. Persona calls can run in parallel, which means total wall-clock time is bounded by the slowest call rather than the sum of all calls. If you care about cost and speed, this is the obvious baseline.

It is also a good reminder that “multi-persona” does not always have to mean “full debate.” Sometimes all you really need is several independent views and a cheap way to aggregate them.

Sequential mode

Sequential mode changes one thing: each persona sees the reasoning that came before it.

This produces a very different dynamic. Instead of multiple views arriving independently, you get a chain of reasoning. Later personas can reinforce, refine, or challenge what earlier ones said.

I find this mode useful when the persona order actually means something. If you have a natural flow like “fact-finder, then analyst, then decision-maker,” sequential reasoning can feel more coherent than parallel opinions.

The trade-off is obvious, though: you lose parallelism. Sequential mode is inherently serial, so it is slower by design.

Deliberation mode

Deliberation mode is the default because it is the closest thing to the CEJ pattern that inspired the project.

Here the personas first respond independently, then they get a second chance after seeing the first round, and they are explicitly asked to engage with one another’s reasoning before a summarisation stage and final judgment.

This is where the architecture becomes most interesting to me.

A lot of systems stop at “get three opinions and vote.” Deliberation mode is deliberately more expensive than that because it is trying to produce something richer than just plurality. The key move is the engagement requirement: personas are pushed to respond to each other, not just restate themselves. That is very much in the spirit of the CEJ paper, which routes uncertain cases into a multi-persona reasoning process consolidated by a judge model.

If you care about auditability, this is usually the best mode. It gives you the fullest transcript and the clearest explanation of how the system arrived where it did.

Adversarial mode

Adversarial mode is less about consensus and more about pressure testing.

Here the personas are assigned opposing stances so that some are pushed toward a severe reading and others toward a more defensive one. The point is not harmony. The point is to surface the strongest argument against the current trajectory before the system commits to a verdict.

This mode is especially useful in legal, risk, and compliance-style scenarios, where being able to say “we considered the strongest counter-position” matters as much as the final answer itself.

Why the judge strategy pattern matters

Once the debate is complete, the system still has to decide how to turn that transcript into a single verdict.

I did not want that logic hardcoded into one final step, because different environments need different trade-offs. So the judge is implemented as a strategy.

At the interface level, the contract is intentionally tiny:

class JudgeStrategy(ABC):
    @abstractmethod
    async def judge(self, transcript: DebateTranscript, labels: list[str]) -> Verdict:
        raise NotImplementedError

That simple interface buys a lot.

You can use a majority-vote judge when you want speed and low cost. You can use a weighted judge when persona confidences matter. You can use an LLM judge when you want synthesis and richer reasoning. And if you have historical reliability data for your personas, you can use Bayesian aggregation instead.

The important thing is not just that there are several judges. It is that the rest of the debate engine does not need to care which one you chose. That separation keeps the architecture extensible without making the main pipeline harder to understand.

In practice, I think of the judge layer as the place where the system answers a very human question: how much interpretation do we want after the debate itself?

Sometimes none. Sometimes a lot.

The LLMClient protocol is the real glue

If there is one abstraction doing more hidden work than most people realise, it is the LLMClient protocol.

The whole system’s interaction with LLMs is mediated through one structural interface:

class LLMClient(Protocol):
    async def complete(
        self,
        model: str,
        system_prompt: str,
        prompt: str,
        temperature: float | None = 0.0,
    ) -> dict[str, Any]: ...

I like this pattern because it is boring in the best possible way.

There is no elaborate inheritance tree here. No framework lock-in. No “you must subclass our magical base client.” If an object exposes the right complete method and returns the expected shape, the rest of the system can use it.

That means the same debate engine can run through LiteLLM, through a direct OpenAI or Anthropic wrapper, through an internal enterprise gateway, or through a local model behind some thin async shim. The architecture stays the same because the protocol boundary is so small.

This is also where the project becomes much more production-friendly than a paper implementation. Research code can afford to assume one provider. Middleware should not.

Cost control is not a nice-to-have

Any architecture like this lives or dies on cost discipline.

A debate engine sounds attractive right up until someone realises it is quietly multiplying inference spend. So cost control in llm-jury happens in layers rather than in one heroic final safeguard.

The first and most important layer is routing. Most inputs should never hit the debate path at all. That is where the biggest savings come from.

The second layer is early termination. If deliberation has clearly converged, there is no point paying for extra rounds just because the configuration technically allows them.

The third layer is the hard budget cap. If the debate exceeds max_debate_cost_usd, the system can return the primary result with explicit fallback metadata instead of pretending the expensive path completed normally.

The fourth layer is observability. Every persona response tracks token use and cost. Every transcript aggregates that information. Every verdict carries enough metadata for someone to inspect what happened later.

I think this is one of the biggest differences between an interesting demo and a deployable system. In production, cost has to be part of the architecture, not a dashboard someone checks afterwards.

Wrapping the classifier layer cleanly

Another design choice I cared about from the start was not forcing users into one kind of classifier.

Most engineers already have a model. They do not want a framework that begins by asking them to rebuild their whole stack.

So the classifier contract is minimal. If something can classify text asynchronously and return a label plus confidence, it can fit. That leads naturally to adapters: an LLM-backed classifier, a scikit-learn wrapper, a Hugging Face wrapper, and a generic function adapter for everything else.

That last one is especially important. In my experience, production systems always have a weird edge case. Some internal API. Some older service. Some heuristic that people still rely on. A framework becomes much more usable the moment it can absorb those without drama.

What I was optimising for

Stepping back, the architecture is really built around one principle: graceful composition.

Every major part of the system can be swapped. Every expensive path has a fallback. Every debate mode changes the communication pattern without changing the outer pipeline. Every judge changes the aggregation logic without changing the debate engine. Every LLM provider sits behind the same boundary.

That is what lets the system stay flexible without turning into a bag of special cases.

Just as importantly, the result is still inspectable. The final verdict is not just a label. It carries the primary classifier’s confidence, the escalation decision, the debate transcript, the judge reasoning, cost metadata, and timing information. In regulated or high-stakes settings, that difference matters. A raw label is just a decision. A structured trail is something you can actually defend.

Closing thought

The part of classifier design that gets the most attention is usually the model. That makes sense. But once you start shipping systems into real environments, another question becomes unavoidable: what do you do with uncertainty?

For me, llm-jury is one answer to that question.

Do not pretend the model is equally trustworthy on every input. Do not pay for full reasoning on everything. Route the easy cases through the fast path. Escalate the messy ones. Keep the components swappable. Keep the costs visible. Keep the fallbacks honest.

That is the architecture in one sentence.

llm-jury is open-source and available on **GitHub and [PyPI](https://pypi.org/project/llm-jury/)**.

The research pattern behind it comes from When in Doubt, Deliberate: Confidence-Based Routing to Expert Debate for Sexism Detection, by Anwar Alajmi and Gabriele Pergola.