Clinical AI Is Tested Constantly…But Is It Actually Validated?

Clinical AI is being tested more than ever. The problem is that we are often testing the wrong things.

Introduction

A recent Nature Medicine paper produced a striking result: general-purpose frontier language models outperformed specialist clinical AI tools across medical knowledge questions, clinician-alignment tasks and queries submitted by doctors in a real clinical environment. [1]

On 500 USMLE-style MedQA questions, Gemini 3.1 Pro achieved 97.4% accuracy. GPT-5.2 scored 94.2%, Claude Opus 4.6 scored 90.2%, OpenEvidence scored 89.6% and UpToDate Expert AI scored 88.4%.

The obvious question was: why did the general-purpose models outperform systems designed specifically for medicine?

Was it the size and diversity of their training data? Better reasoning? More extensive alignment? Faster development cycles? Greater exposure to unusual edge cases? Or did retrieval-augmented generation occasionally distract the specialist systems by supplying irrelevant or poorly integrated information?

These are important questions.

However, a discussion I had after someone had shared the paper, raised a more fundamental one:

Do clinical AI systems undergo enough validation before we begin treating their benchmark performance as evidence of clinical competence?

My answer is that clinical AI is undergoing an enormous amount of testing, but comparatively little testing that establishes real clinical utility.

A wrong answer does not necessarily or automatically equate to a harmed patient.

One response to the paper was that an accurary score of 97.4% still represents a 2.6% failure rate and that such a rate would be unacceptable in medicine.

That interpretation is understandable, but it confuses benchmark error with clinical harm.

On a set of 500 multiple-choice questions, 97.4% accuracy means that the model selected the keyed answer on 487 questions and did not select it on 13. It does not mean that the model would harm 2.6% of patients.

The benchmark did not ask the model to manage 500 patients. It tested whether the model could answer written questions designed to assess medical knowledge and pattern recognition of international medical graduates wanting to practice in the USA.

The consequences of an error depend on the intended use.

A wrong answer in an educational quiz is not equivalent to an incorrect drug dose, a missed pulmonary embolism or a failure to recognise sepsis. Conversely, a model might select the correct answer while providing an unsafe explanation, expressing inappropriate certainty or failing to identify information that is missing from the question.

This is why “accuracy” alone is an inadequate description of clinical safety.

The relevant questions are not simply how often was the model wrong?

They are:

• What kind of errors did it make?
• Under what conditions did they occur?
• Were they detectable?
• Could they be corrected?
• Did the system fail safely?
• What would happen if a clinician or patient acted on the output?

Medicine does not require every individual component to be infallible. It requires the complete system to manage foreseeable risk.

This was more than an examination benchmark…but it was not a clinical trial

It is also important not to undersell the Nature Medicine study.

The researchers did not rely solely on 500 MedQA questions. They also evaluated 500 HealthBench prompts and created a real clinical queries benchmark from 100 de-identified questions submitted by physicians using an approved institutional GPT system at NYU Langone Health.

Twelve clinicians, blinded to model identity, evaluated the responses for clinical correctness, completeness, safety and clarity. This produced 1,800 model–question annotations. [1]

That is a considerably stronger design than simply asking several chatbots to sit a medical examination.

However, “real clinical queries” should not be confused with prospective evaluation in clinical care. The questions came from one American health system. The study assessed written responses rather than patient outcomes. It did not evaluate whether using the models changed diagnoses, treatment decisions, consultation times, resource use or rates of patient harm.

The authors also acknowledged that they did not assess citation quality or response latency. The proprietary clinical tools had to be tested through their browser interfaces, while the frontier models were accessed through APIs. HealthBench was partly graded by frontier LLMs, including models that were themselves being evaluated. [1]

The real-query component therefore provides valuable evidence of clinician-rated performance. It does not establish that Gemini, GPT or Claude is safe to deploy as an autonomous clinical decision-maker.

Those are different claims requiring different evidence.

The missing step is often external validation!

My greatest concern is not that the frontier models made 13 errors on MedQA. It is that we still know comparatively little about where their performance will generalise.

The real-world queries in the study were drawn from one institution and evaluated by US clinicians. That leaves several questions unanswered.

Would the ordering remain the same in the NHS? In a rural hospital? In Papua New Guinea, Singapore, Nigeria, India or Hungary? Would it remain the same for paediatric patients, pregnant women, people with multiple co-morbidities or patients communicating in a second language? Would the models perform equally well when information is incomplete, contradictory, misspelt or presented in an unfamiliar clinical format?

Proprietary developers may never disclose every item in their training data. But external validity cannot be established merely by inspecting a training dataset anyway.

It must be demonstrated by testing the system across independent populations, institutions, specialties, languages, prevalence levels and clinical workflows.

A model that performs well at a major academic centre may fail when disease prevalence changes, local guidelines differ, there are semantic differences in understanding of terminology, documentation is less complete or the patient population is under-represented.

This is where much clinical AI research remains weakest.

A 2026 systematic review identified 4,609 peer-reviewed studies of LLMs in clinical medicine. Yet most evaluated simulated scenarios or examination-style tasks. Only 1,048 used real-world patient data, and just 19 were prospective randomised trials. At least a quarter of the studies had sample sizes below 30. [2]

We do not have a shortage of medical AI papers. We have a shortage of sufficiently realistic, prospective and patient-centred evaluations.

Input validation matters, but it must match the system

The conversation also raised the importance of validating inputs before a model generates an output. That principle is correct, but input validation means different things for different AI systems.

An arrhythmia classifier analysing an ICU waveform must recognise artefact, disconnected electrodes and corrupted signals. A radiology model must detect incompatible image formats, poor positioning and scans outside its validated acquisition parameters.

For an LLM, input validation may mean recognising missing clinical information, contradictions, uncertain terminology, inappropriate requests, unreliable retrieved documents or instructions that attempt to manipulate the system.

A clean-text medical benchmark does not test these problems because it was not designed to do so. That does not invalidate the benchmark. It limits the claim we can make from it.

Every evaluation should be judged against its stated purpose. The mistake occurs when evidence of medical question-answering ability is subsequently promoted as evidence of bedside safety.

Harm is more subtle than hallucination

Clinical AI evaluation also focuses too heavily on fabricated facts.

Hallucination is important, but it is only one failure mode.

A model may provide factually correct information yet omit the most urgent investigation. It may recommend a reasonable treatment but fail to account for renal impairment. It may generate a technically accurate differential diagnosis while placing a life-threatening condition too low on the list.

The NOHARM study illustrates this distinction. It evaluated 31 LLMs using 100 real primary-care-to-specialist consultation cases. Potential severe harm reached 22.2% for some models, and 76.6% of the harmful errors involved omissions rather than explicitly incorrect recommendations. Safety scores correlated only moderately with conventional medical-knowledge benchmarks. [3]

Importantly, the study did not find that every model was less safe than doctors. Its best-performing models outperformed generalist physicians on its safety measure. [3]

The lesson is therefore not that humans are safe and AI is dangerous.

It is that: knowledge, correctness and clinical safety are related but separate constructs and each must be measured directly.

An independent checker cannot magically create zero risk!

It is tempting to propose an independent auditing system that checks every model response and forces the error rate to zero.

Independent verification can certainly reduce risk. Retrieval from trusted sources, deterministic rule checks, dose-range validation, contradiction detection and mandatory clinician review may all form useful safety barriers.

But the verifier is itself part of the clinical system.

Who validates the validator? What happens when the generator and checker share the same blind spot? Can the system identify an omission when neither component knows what is missing? Does the verification layer remain reliable after the underlying model is updated?

“Zero hallucination” or “zero failure” may be an appealing engineering objective. It is not a credible general guarantee across open-ended clinical medicine.

Good clinical safety engineering does not depend on declaring risk abolished. It identifies hazards, estimates their consequences, introduces controls, specifies escalation routes and continuously monitors what happens after deployment.

International Good Machine Learning Practice principles consequently address the entire product lifecycle rather than a single pre-release accuracy test. [4] In England, DCB0129 places clinical-risk-management responsibilities on health-software manufacturers, while DCB0160 applies corresponding responsibilities to organisations deploying and using those systems. [5]

Validation must follow the intended use

The amount of validation required should depend on what the system is permitted to do.

A tool that reformats a clinician-approved note does not require the same evidence as an autonomous triage system. A literature-search assistant does not present the same risk as an agent that orders investigations or changes medication.

Before testing begins, developers should specify:

Who will use the system? For which patients? Using which inputs? To produce what output? In which clinical setting? Who remains responsible for the decision? What happens when the system is uncertain or unavailable?

Without a sufficiently narrow intended purpose, even an impressive validation result becomes difficult to interpret.

This principle is reflected in UK medical-device guidance, where qualification and classification of software depend heavily on its intended medical purpose. [6] In the European Union, AI-based software intended for medical purposes may also fall within high-risk requirements involving risk management, data quality, user information and human oversight. [7]

This does not mean that ChatGPT or Gemini is universally “banned” from clinical environments. Nor does the presence of a medical brand make a specialist AI system clinically validated.

Regulatory status, institutional approval and evidence of clinical benefit are related, but they are not interchangeable.

We should stop asking whether the model passed

The Nature Medicine paper provides compelling evidence that domain-specific branding does not guarantee superior clinical performance. For the tasks studied, frontier general-purpose models were better.

That finding should influence how healthcare organisations evaluate procurement claims.

But it should not become evidence that a general-purpose chatbot is ready to practise medicine.

The next phase of clinical AI evaluation must move beyond asking whether a model can answer questions. We must examine whether the complete sociotechnical system performs safely in the hands of its intended users, with imperfect inputs, across diverse populations and under real clinical pressures.

That means testing technical performance, external validity, human–AI interaction, workflow integration, failure severity, subgroup performance and post-deployment drift.

The correct conclusion is not that a 2.6% error rate is automatically unacceptable…It is that 97.4% accuracy answers only one small part of the validation question.

Until we test the remainder, the score tells us that the model is an exceptional examination candidate.

It does not yet tell us that it is a safe clinical colleague.

References

1. Vishwanath K, Alyakin A, Ghosh M, et al. General-purpose large language models outperform specialized clinical AI tools on medical benchmarks. Nature Medicine. 2026;32:2405–2409.
2. Chen SF, et al. LLM-assisted systematic review of large language models in clinical medicine. Nature Medicine. 2026.
3. Wu D, et al. First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultations. 2025.
4. International Medical Device Regulators Forum. Good Machine Learning Practice for Medical Device Development: Guiding Principles.
5. NHS England. Digital clinical safety assurance: DCB0129 and DCB0160.
6. Medicines and Healthcare products Regulatory Agency. Software and artificial intelligence as a medical device.
7. European Commission. Artificial intelligence in healthcare and requirements for high-risk medical AI systems.