From Research Paper to Production: How an Academic Framework Became Open-Source Middleware

Mapping the CEJ paper’s ideas to llm-jury’s architecture: what we kept, what we extended, and what we deliberately left out

In December 2025, Anwar Alajmi and Gabriele Pergola published When in Doubt, Deliberate: Confidence-Based Routing to Expert Debate for Sexism Detection. The paper introduces a framework called Collaborative Expert Judgment, or CEJ, where a classifier handles the obvious cases and escalates the uncertain ones to a structured debate between multiple LLM personas.

The paper is grounded in one domain: sexism detection. llm-jury takes the same underlying mechanism and turns it into something more general — a domain-agnostic, pip-installable SDK that can sit on top of different classifiers in different domains.

This piece is really about translation. Not translation in the language sense, but in the engineering sense: how a research idea becomes production middleware. What carried over cleanly, what had to be generalised, and what we chose not to build.

If you are coming at this from the research side, it is a look at what happens when a paper leaves the benchmark world and meets production constraints. If you are coming at it from the engineering side, it is a look at the research DNA behind the SDK.

What the paper actually proved

The CEJ paper makes a few important claims, and I think it is worth being precise about them.

Confidence-based routing is what makes the whole thing practical

The core architectural move is surprisingly simple: not every input gets the expensive path. A specialist classifier handles the clear-cut cases, and only the low-confidence cases are escalated to the debate pipeline. In the paper’s setup, that specialist model used targeted training choices — including class-balanced focal loss, class-aware batching, and threshold calibration — to deal with label imbalance and noisy supervision.

That matters because the value here is not just “more reasoning.” It is selective reasoning. The paper’s abstract explicitly frames the system as a two-stage design: a fast classifier for high-confidence cases, and a CEJ module for ambiguous ones.

The gains show up where the base classifier is weakest

The current arXiv abstract reports +4.48% on EDOS Task A, +1.30% on EDOS Task B, and +2.72% on EXIST 2025 Task 1.1.

What interested me more than the headline numbers was the logic behind them. The paper is not trying to improve the easy cases. Those are already being handled by the primary classifier. The point is to improve the borderline, ambiguous cases where confidence drops and mistakes become more costly.

That is what makes the framework useful beyond the benchmark. It does not treat all predictions as equally hard. It puts extra reasoning exactly where the base system is least trustworthy.

The debate structure matters

The paper does not stop at “ask several personas and count the votes.” Its CEJ setup is explicitly structured around multiple personas and a judge model that consolidates their reasoning. The abstract describes the CEJ module as a mechanism that “prompts multiple personas and consolidates their reasoning through a judge model.”

That detail matters. Once you move beyond single-shot classification, the question is not just whether multiple perspectives help. It is whether the interaction between those perspectives produces something better than parallel independent answers. The paper’s answer is yes, and that shaped the way I approached the SDK.

How that maps to llm-jury

The paper gives you a concrete implementation for one domain. llm-jury takes the same ideas and breaks them into reusable components.

The easiest way to think about it is concept by concept.

The paper’s primary classifier becomes a general classifier interface. Instead of assuming one specific fine-tuned model, llm-jury accepts anything that can return a label and a confidence score. That might be a scikit-learn model, a Hugging Face pipeline, an LLM-based classifier, or just a custom function.

The paper’s threshold routing becomes a configurable calibration and override mechanism. The default remains simple — confidence below threshold means escalate — but the SDK leaves room for more custom routing logic.

The paper’s fixed personas become a registry system. Instead of hardcoding experts for one task, the SDK provides persona sets for several domains and lets users define their own.

The paper’s single CEJ pipeline becomes one debate mode among several. Deliberation mode is the closest match to the original paper, but it sits inside a broader framework that can support other interaction styles.

And the paper’s single judge model becomes a strategy interface. In some settings, a majority vote is enough. In others, you want weighted voting or a full LLM judge.

If you instantiate Jury with the defaults, the result is intentionally close to the original CEJ flow: independent initial opinions, structured debate, neutral summarisation, then a final verdict.

That was deliberate. I did not want the generalisation to erase the original mechanism.

What we extended beyond the paper

A research paper has different priorities from a production SDK. The paper needs to show that a mechanism works. A production library needs to make that mechanism usable under different constraints.

That is where most of the extensions came from.

We added multiple debate modes

The paper uses one structure, which is what llm-jury calls deliberation mode. That remains the default, because it is the closest thing to the original CEJ setup.

But in practice, different teams want different trade-offs.

Some want a cheaper path where personas answer independently and there is no interaction. Some want a sequential flow, where each persona sees the previous reasoning and builds on it. Some want an adversarial setup, where different personas are explicitly pushed into opposing interpretations so the strongest counter-arguments surface.

Those modes are not in the paper because they are not necessary for the paper’s claim. But they matter once you move into production settings with different latency, cost, and auditability requirements.

We added multiple judge strategies

The paper uses a single judge model at the end of the process. That makes sense in a research setup. In production, though, that final LLM call is not always the right trade-off.

Sometimes a majority vote is enough. Sometimes weighted voting makes more sense. Sometimes you want something more probabilistic, like Bayesian aggregation with persona reliability priors.

That flexibility matters because the right judgment mechanism depends on the domain and the cost profile. There is no good reason to hardcode one option if the architecture can support several.

We made the classifier pluggable

This was probably the biggest practical shift.

The paper starts from a specific fine-tuned model. That is natural in research. But most engineers already have a classifier. They are not looking to recreate somebody else’s training pipeline from scratch. They want to know whether they can add this mechanism to the system they already run.

So the SDK abstracts the classifier behind an interface. If you already have something that produces a label and a confidence score, you can wrap it. That means the debate layer becomes something you can add without retraining your whole stack.

We added cost controls

Papers usually care about benchmark performance first. Production systems have to care about budgets.

That is why the SDK includes things like per-verdict cost caps, token and cost tracking, early termination when the debate has clearly converged, and stats around escalation rate and savings.

None of that changes the core CEJ idea. But it absolutely changes whether a team can run it safely in a real environment.

We made personas domain-agnostic

The paper’s personas are specific to sexism detection, which makes perfect sense for the problem it is solving. But once the goal becomes reuse, that coupling has to go.

So the SDK ships with persona registries for several domains and a way to define your own. The important part is not the exact labels on the personas. It is the pattern: bring multiple interpretive lenses to bear on an uncertain case, then force them to engage.

What we deliberately did not build

I think this part matters just as much.

There is always a temptation, when turning a research pattern into a product, to keep extending until you have accidentally built something completely different. I wanted to avoid that.

So some things were left out on purpose.

We did not implement the paper’s training pipeline

The CEJ paper includes specific training choices for the base classifier: class-balanced focal loss, class-aware batching, and post-hoc threshold calibration. llm-jury does not implement that training stack.

That is not because it is unimportant. It is because it belongs to a different layer of the system.

llm-jury is middleware. It wraps an existing classifier. It is not intended to be a training framework. If somebody wants to reproduce the paper’s full training setup, that is a separate concern from the debate layer itself.

We did not make margin-based routing the default

The paper discusses routing uncertain cases dynamically, and one sensible extension in multi-class settings is to look not just at top confidence but at the margin between the top two predictions. That is a good idea.

But I chose not to make that the default behavior in the SDK.

The reason was simplicity. A plain confidence threshold is easier to understand, easier to explain, and easier to adopt. At the same time, the SDK exposes an override hook so users who want margin-based logic can add it themselves.

That felt like the right balance: keep the default simple, keep the escape hatch open.

We did not ship the paper’s benchmark setup

The paper evaluates on EDOS and EXIST. That is exactly what it should do. But those benchmarks are tied to one domain and one research question.

The SDK does not bundle those evaluation pipelines because most users are not working on sexism detection. What they need is not a copy of the paper’s benchmark harness. They need a way to calibrate the mechanism on their own data.

That is why the calibrator matters more in the SDK than benchmark replication.

We did not overcomplicate the personas

It is easy to imagine richer features here: persona self-reflection, session memory, explicit metacognition, dynamic role reassignment. I considered some of that and left it out.

Not because it is impossible, but because it adds complexity faster than it adds proven value. Without good evidence that those layers materially improve the debate, I would rather keep the mechanism legible.

In general, I wanted llm-jury to stay close to the core idea rather than drift too quickly into speculative additions.

Why open-sourcing this pattern matters

There is a familiar gap between research and deployment.

A paper can show that a mechanism works on a benchmark. That is already a real contribution. But production systems need more than a mechanism. They need interfaces. They need fallbacks. They need cost controls. They need auditability.

What the CEJ paper does well is isolate a mechanism that is genuinely useful: confidence-based routing plus multi-persona reasoning for uncertain cases. What llm-jury tries to do is make that mechanism composable.

The Classifier abstraction decouples it from any single model. The judge strategies decouple it from any single voting method. The persona registry decouples it from any single domain. The calibrator decouples it from any single dataset.

None of those abstractions are novel by themselves. They are standard engineering moves. The interesting part is that the CEJ pattern decomposes cleanly into them without losing the heart of what made the original research useful.

Closing thoughts

The CEJ paper makes a real contribution. It shows that confidence-based routing to structured expert-style debate can improve classification on ambiguous inputs, and that it can do so without sending every case through an expensive reasoning pipeline.

What we built with llm-jury is not an attempt to replace the paper, or to claim some grand improvement over it. It is a translation layer between the research insight and the realities of deployment.

The SDK’s claim is simpler than that. It says: if this pattern is useful, it should not stay trapped inside one paper, one dataset, or one training setup.

The best research ideas are often the ones that disappear into infrastructure.

If you are running a classifier in production and your errors cluster in the ambiguity zone — and they usually do — this pattern is worth testing. The difference now is that you do not need to rebuild the whole research stack to try it.

llm-jury is open-source and available on **GitHub and [PyPI](https://pypi.org/project/llm-jury/)**.

The paper: Alajmi, A. & Pergola, G. (2025). When in Doubt, Deliberate: Confidence-Based Routing to Expert Debate for Sexism Detection. arXiv:2512.23732.