What If We Ran AI Queries the Way We Run Clinical Trials?

A case for independent replication as the missing trust layer in natural language data systems

When a pharmaceutical company wants to prove a drug works, it doesn’t run one instance of a pivotal trial and call it done. Historically, regulators required two independent pivotal trials — each conducted across different sites, with different investigators, under controlled conditions — before approving a drug. (The FDA moved away from this as a default requirement in early 2026, but the underlying statistical logic was neat.) If both trials arrived at the same conclusion, confidence in that conclusion went up dramatically. If they diverged, that divergence was itself a signal worth investigating. This is not caution for caution’s sake. It is an epistemological principle: independent replication of the same test, arriving at the same result, is one of the strongest signals of truth we have.

The analogy is not perfect — and it is worth saying so upfront. Clinical trials replicate to control for random error and biological variability across genuinely independent experiments. The architecture proposed here runs computations against the same underlying data, generated by models that may share training lineage. The failures are not independent in the strict statistical sense. A closer technical parallel could be method comparison studies in laboratory medicine, where two analytical instruments run the same sample and disagreement flags an assay problem (something I remember from my biotechnology labs). The clinical trial framing is just being borrowed for its intuitive clarity, not its methodological precision. The core claim is simpler and more defensible: heterogeneous computational paths have non-overlapping failure modes, and agreement across them is a stronger signal than agreement within a single path.

With that caveat on the table, the intuition holds.

Now consider how most organizations currently query their data using AI.

A user types a question in natural language. A large language model based system converts it to SQL. SQL executes against a database. The result comes back. The user looks at the number and decides.

This is not for lack of trying to make that chain reliable. The field has produced genuinely sophisticated mitigations — schema linking with RAG pipelines that enrich prompts with metadata before generation, boundary-aware abstention that refuses to answer rather than guess, execution feedback loops that catch runtime errors and regenerate, fine-tuning on domain-specific query history, and human-in-the-loop review gates before results reach decision-makers. These have improved accuracy meaningfully. Organizations deploying them are not being naive.

But they share a structural limitation. Self-reflection, LLM-as-judge evaluation, and execution feedback all route the verification signal back through the same model that generated the original query. A judge that shares training lineage with the generator shares its blind spots. Research has found a consistent ceiling here: LLMs struggle to self-assess their own SQL outputs reliably, and when prompted to correct a query that is already correct, they tend to modify it anyway — introducing new errors where none existed (I call it over-enthusiastic corrections). The failure modes are correlated because the system is the same, looking at itself.

The gap is not in any individual technique. It is structural. There is no independent check — independent in the strict sense of a different execution engine, a different query language, a different computational path to the same underlying data.

Independent replication as an architectural principle

Here is the core proposal: instead of trusting one agent, run two or more in parallel, independently, and compare what they return.

Not the same model asked twice. Not the same query reworded. Two genuinely different computational paths to the same answer — different query languages, different execution engines, different underlying representations of the data.

Agent A receives the user’s question and converts it to SQL, which executes against a relational database. Agent B receives the same question and converts it to PySpark, which processes the same underlying data through a distributed compute layer. Agent C converts the same question to R — using dplyr or data.table — which processes the data through an in-memory analytical engine with different aggregation semantics, different floating point handling, and different null behavior than either SQL or PySpark.

The choice of engines is deliberate. SQL and R do not share execution lineage. A relational query planner and an in-memory R dataframe engine will not make the same computational mistakes on the same query. DAX against a semantic model was an early candidate — and a strong one, given how differently VertiPaq aggregates from a relational engine — but the setup complexity made R the more practical starting point. The goal is not more agents. It is agents whose failure modes do not overlap. Each agent may also use a different underlying language model — further reducing the chance that a shared model bias propagates identically across all three computational paths.

These pathways are not statistically independent in the strict sense, but are designed to reduce correlated failure modes across heterogeneous execution paths.

Each agent works independently. None sees the other’s outputs until the comparison node receives all results.

If the agents agree that agreement is meaningful. Not because any single agent is infallible, but because the probability of multiple independent systems making the same mistake in the same direction is substantially lower than the probability of one system making it. This is the replication principle applied to data infrastructure.

If the agents disagree, that disagreement is equally meaningful — and arguably more valuable. A discrepancy does not just flag an error. It surfaces a question: why did two correct-looking computations return different answers? The answer to that question might be a model hallucination, a data model inconsistency, a semantic ambiguity in the original question, or a genuine difference in how two systems interpret a business concept. Each of these has a different resolution path.

What makes this different from existing approaches

The natural language to SQL space has been heavily researched. There are benchmarks, leaderboards, and papers measuring model accuracy on increasingly complex queries. There is also work on detecting hallucinations by asking the same model the same question multiple times and checking for consistency.

This proposal is different in a specific way: it requires heterogeneous independence, not just repetitive independence.

Asking the same model the same question ten times does not create independent trials. The model has the same weights, the same training data, the same biases. It will fail in correlated ways. The failures cluster. This is what existing hallucination detection approaches do — they sample the same model multiple times and check for consistency. A model that is confidently wrong will be consistently wrong.

What creates genuine independence is traversing a different computational path — a different query language compiled to a different execution engine against a different internal representation of the data. SQL and R are not the same thing. A relational query planner and an in-memory dataframe engine do not share failure modes. When both arrive at the same answer, something more robust has happened.

The closest technical precedent I found online is differential testing — the technique used in compiler engineering to find bugs by running the same source code through GCC and Clang and comparing outputs. Discrepancies don’t tell you which compiler is wrong, but they tell you something is wrong. The same logic applies here: cross-engine disagreement is a signal that something in the computation deserves scrutiny, even without knowing which agent made the error.

The architecture in plain terms

High Level Architecture

The system has seven major components:

Query intake. The user submits a question in natural language.

Query review node. Before any query is generated, the system evaluates whether the question is answerable as stated. Is it specific enough? Is it ambiguous in ways that would produce different valid interpretations? If so, the system asks the user to clarify before proceeding. This step prevents downstream divergence caused by underspecified input rather than agent error. Also, it gatekeeps unwanted questions away.

Structured rewrite. The clarified question is rewritten into a canonical, unambiguous form that will be sent identically to all agents.

Manifest dispatch. A structured document — a manifest — is sent to every agent simultaneously. This manifest (a JSON file) contains not just the query, but the table definitions, field descriptions, data types, constraints, referential integrity rules, domain context, and a glossary that maps business language to schema concepts. The manifest is the contract between the orchestrator and the agents. It ensures every agent is working from the same understanding of what the data means, not just what the tables contain.

Parallel agent execution. Each agent independently generates its query in its native language and executes it against its engine. No agent sees another’s output.

Comparator node. The outputs are collected and compared. If all agents agree, the result is returned with a high confidence indicator. If a majority agree, the majority result is returned with a lower confidence indicator and a note that divergence was detected. If there is substantial disagreement across agents, the system escalates — flagging the result as unreliable and surfacing the raw outputs for human review.

Response to user. The user receives not just a number, but a confidence signal attached to it. High confidence means independent systems converged. Medium confidence means most did. Low confidence means a human should look before acting.

The confidence number

One of the more interesting engineering questions in this architecture is how to assign a number to agreement.

A simple approach is agreement rate: three of three agents agree, confidence is high; two of three agree, confidence is medium. But this treats all agents as equally independent, which they are not. SQL and PySpark share more computational lineage than SQL and a JavaScript dataframe engine. A more sophisticated scoring system might weight agreement by the conceptual distance between the agents’ execution paths — the further apart the paths, the more a convergent result means.

Another dimension is the granularity of agreement. Agents might return the same top-line number but differ at the row level. Or they might agree on ninety-eight percent of rows and diverge on two percent. That two percent is probably where the interesting question lives.

Neither of these is a solved problem. That is partly what makes this worth writing about.

What the manifest solves

The glossary section of the manifest deserves particular attention, because it addresses the most common source of silent query errors: the gap between how business users describe things and how data teams have modelled them.

Business language is rarely a clean match to schema column names. “Revenue” might mean gross or net depending on who is asking. “Last quarter” might mean fiscal or calendar. “Top” might mean a ranked ten or a ranked five. A glossary that resolves these terms before any agent generates a query does not make the model smarter — it narrows the space of valid interpretations so that independent agents are more likely to converge on the correct one rather than each picking a different plausible one.

The manifest is not just metadata. It is a semantic contract that makes independent replication meaningful rather than coincidental.

Consider a user who asks: “show me top products by revenue last quarter.” The manifest glossary tells every agent that “revenue” maps to net_sales_amount not gross_revenue, that “last quarter” means the completed fiscal quarter using fiscal_period_end_date not a calendar date, and that “top” means a default display limit of ten unless specified. Without the manifest, Agent A might use gross revenue and Agent B net sales — they will disagree, but the disagreement tells you nothing useful about query correctness. It just reflects an unresolved semantic ambiguity. With the manifest, both agents work from the same definitions. Now disagreement means something went wrong in the computation, not in the interpretation.

What divergence tells you

The escalation path — when agents substantially disagree — is in some ways the most valuable output of the system.

When two independently computed answers to the same question differ, one of several things is true. The model generating one of the queries made an error the other did not. The two execution engines interpret some aspect of the query differently, which might reveal an ambiguity in the data model. The question itself was interpretable in more than one legitimate way, and both agents found a valid interpretation. Or the underlying data has quality issues — nulls, duplicates, type inconsistencies — that surface differently depending on how each engine handles them. That too is a signal worth having.

None of these is a failure in the sense of something going wrong unexpectedly. All of them are information. A system that never escalates is not a reliable system — it is a system that has given up trying to know when it might be wrong.

Beyond text-to-SQL

The replication principle proposed here is not specific to natural language querying. Any agentic system where the correctness of an output cannot be easily verified by the user is a candidate for this architecture. Two agents generating the same function in different programming languages, their outputs validated by shared unit tests. Two agents summarising the same document independently, their outputs scored for semantic agreement. Two agents extracting structured data from the same unstructured source, their outputs diffed field by field. The pattern is the same in every case: define a shared input contract, dispatch to heterogeneous independent agents, compare outputs, score confidence by agreement. The specific agents, languages, and engines are implementation details. The principle is not.

Why now

The reason this architecture is timely is not that natural language querying is new. It is that organizations are beginning to deploy it in contexts where the cost of a wrong answer is real.

This approach is likely overkill for routine analytics, but becomes relevant where decision risk is high.

A wrong number in a dashboard is embarrassing. A wrong number informing a resource allocation decision, a safety review, or a regulatory submission is something else. The gap between “AI can generate queries” and “AI-generated queries can be trusted in production” is not primarily a model capability gap. It is an architecture gap.

Independent replication does not close that gap entirely. No system does. But it introduces the kind of epistemic discipline that high-stakes fields have applied to experimental findings for decades — the principle that a result you cannot replicate independently is a result you should hold more lightly.

That principle translates. The engineering to implement it is not prohibitive. The question is whether the organisations deploying these systems will require it before something goes wrong, or after.

There is a final irony worth noting. In early 2026, the FDA moved away from requiring two independent pivotal trials as the default standard for drug approval — precisely because advances in trial design, biostatistics, and evidence generation have matured to the point where a single well-designed study can be sufficiently persuasive. The underlying logic of the two-trial paradigm, however, remains intact: replication was required because methodology was not yet trustworthy enough to stake a regulatory decision on a single result. That condition — methodology not yet trustworthy enough — describes exactly where LLM-generated data queries stand today. Perhaps the two-trial logic still has work to do — just not where we first expected it.

This article describes a proposed architecture and does not reflect the views or systems of any employer.