Medicine’s Hidden Advantage in AI Edge Case Testing

by Anthony MacKenzie-Gureje

One of the most interesting aspects of working as both a doctor and an AI Trainer/AI prompt engineer AI is realising that medicine may already possess something that many AI teams are desperately trying to build: a culture of documenting edge cases.

As AI systems become increasingly capable, their limitations are often found not in common scenarios but at the margins. The straightforward cases are rarely the problem. The challenge lies in the unusual presentation, the rare exception, the atypical combination of factors, or the situation that falls just outside the distribution of examples the model has previously encountered.

In AI evaluation, we often refer to these as “edge cases”, but in medicine, we call them case reports.

For centuries, clinicians have documented unusual presentations of disease, unexpected complications, novel associations, rare syndromes, diagnostic pitfalls, and treatment outcomes that diverged from expectations. Entire journals are dedicated to publishing these observations. Case reports and case series occupy the lower tiers of the traditional evidence hierarchy, yet they remain valued because they teach us something important: reality is often messier than the textbook!

The patient with myocardial infarction presenting solely with jaw pain.

The child whose rare metabolic disorder masquerades as a common infection.

The adverse drug reaction that occurs once in tens of thousands of prescriptions.

These are not statistical noise. They are reminders that medicine is practised on individuals rather than averages within a defined population. This mindset aligns remarkably well with modern AI evaluation.

Many organisations approach AI testing by measuring aggregate performance. How often is the answer correct? What is the accuracy? How does the model perform on benchmark datasets?

These metrics are useful, but they can create a false sense of confidence. An AI system that performs exceptionally well on common cases may still fail catastrophically when faced with uncommon but clinically important scenarios, or clinical scenarios where the physiological parameters are at the extremes of what are expected.

The most valuable evaluation often occurs at the edges. What happens when symptoms are incomplete? What happens when information is contradictory? What happens when two rare conditions coexist? What happens when a presentation superficially resembles one diagnosis but is actually another? What happens when a physician must reach an acute diagnosis under a time constraint? What happens when a physician is forced to manage an acutely unwell patient in an austere environment?

Medicine has spent decades cataloguing precisely these situations and this is where I think healthcare has a unique advantage over many other disciplines adopting AI.

In numerous fields, edge cases are treated as anomalies or unfortunate deviations from an ideal process. Organisations focus on the typical workflow and optimise for the expected outcome. The exceptions may be discussed informally but are rarely captured systematically.

Medicine has always taken a different path. As doctors, we celebrate unusual cases because they are educational. We publish them. We debate them. We learn from them. We preserve them in the literature for future clinicians, who may encounter something similar. There is even a dedicated EQUATOR network framework (the CARE guidelines) for the standard reporting of case reports. (1)

As a result, healthcare possesses an extensive, distributed repository of real-world edge cases spanning decades and, in some specialties, centuries. Of course, the best edge-case testing still comes from lived clinical experience. The cases that stay with us after a difficult night shift, a difficult operation or procedure, a missed diagnosis, or an unexpected outcome often provide the richest material for evaluating AI systems. They carry contextual nuances that may never make it into a publication.

Yet the medical literature offers something invaluable: a scalable starting point. Every unusual case report can be viewed through a new lens, not only as a clinical learning resource but also as a potential AI evaluation scenario.

Even when accounting for publication bias if a case was surprising enough to warrant publication, there is a reasonable chance it could challenge an AI system as well.

As healthcare organisations increasingly deploy AI tools, I suspect we will see a growing convergence between clinical education and AI evaluation. The same cases that teach junior doctors diagnostic reasoning may become some of the most effective tools for stress-testing clinical AI systems.

Perhaps one of medicine’s most under-appreciated contributions to AI is not a new algorithm or dataset. It is a longstanding recognition that the exceptions matter. Because in medicine, the edge cases are sometimes where the most important lessons are to be found.

References

(1) Riley DS, Barber MS, Kienle GS, Aronson JK, von Schoen-Angerer T, Tugwell P, Kiene H, Helfand M, Altman DG, Sox H, Werthmann PG, Moher D, Rison RA, Shamseer L, Koch CA, Sun GH, Hanaway P, Sudak NL, Kaszkin-Bettag M, Carpenter JE, Gagnier JJ. CARE guidelines for case reports: explanation and elaboration document. J Clin Epidemiol. 2017 Sep;89:218–235. doi: 10.1016/j.jclinepi.2017.04.026. Epub 2017 May 18. PMID: 28529185.