AI didn’t invent this problem. Healthcare UX has been solving it for decades.

People are making high-stakes decisions with AI. Healthcare UX has been designing for failure, calibrated trust, and human accountability all along.

After nearly 10 years designing healthcare products, and more recently AI products for clinicians, I’ve realised healthcare teaches designers a few instincts that are suddenly useful everywhere.

A while ago, I added a confirmation dialog to a flow of patient deletion. The user was about to delete a patient, and before the deletion went through, a dialog appeared asking them to confirm.

A colleague questioned it, as the dialog added friction: why an extra click between the user and what they wanted to do? Why interrupt the flow?

They weren’t wrong. Most of us are trained to remove friction wherever we can. A confirmation dialog is usually the kind of thing that gets challenged in a design review. But after years designing healthcare products, I’d learned to ask a different question.

What happens the one time someone deletes the wrong patient?

Not because they’re careless. Because they’re tired. They’re moving quickly between consultations. They clicked the wrong row.

Suddenly, that extra click doesn’t feel like friction anymore. It feels like the interface acknowledging that humans make mistakes.

Looking back, we were both applying good design instincts. We were just optimising for different things: my colleague was optimising for the path where everything goes right, I was optimising for the path where something eventually goes wrong.

I’ve been thinking about that difference a lot lately because AI has made it relevant far beyond healthcare.

Looking back, I think healthcare was training me for AI long before I realized it.

It taught me to design for failure instead of perfection, to think about trust instead of adoption, and to be intentional about where the system’s responsibility ends and the human’s begins.

Those same instincts now shape almost every AI product I’m working on.

Designing for failure, not just success

It’s natural to focus on the happy path first. The user does the expected thing, gets the expected result, and moves on.

Healthcare doesn’t let you stay there for long.

One thing I learned very quickly is that someone will eventually use your product in exactly the way you didn’t expect.

Not because users are unskilled. Because they’re interrupted. Because they’re under pressure. Because they’re switching between patients while thinking about three other things.

You stop asking only: “Does this work?” and start asking:“What happens when it doesn’t?”

That second question changes the design. It’s the difference between deleting immediately and asking for confirmation. It’s the difference between an alarm that relies only on colour and one that also uses shape and text, because not everyone perceives colour the same way.

These aren’t paranoid additions. They’re the design taking responsibility for the moments that matter most, which are almost never the happy path.

AI suddenly makes this mindset relevant for almost every product team.

By definition, AI systems can produce outputs that are wrong. Sometimes subtly. Sometimes confidently. Sometimes in ways users won’t immediately notice.

If we only design for the moments when the model is correct, we’re ignoring the moments that matter most.

Calibrated trust

One phrase I keep coming back to is calibrated trust.

The goal isn’t to make people trust AI. The goal is to help them trust it the right amount.

Too little trust, and they won’t use it. Too much, and they’ll rely on it precisely when they shouldn’t.

I saw this very clearly during usability testing with clinicians. They were comfortable using AI to summarise patient histories, surface relevant guidelines and prepare for consultations. No hesitation. Then the system generated a suggestion. Not a dangerous one. Not even a wrong one. Just a sentence that shifted from presenting information to interpreting it.

One clinician immediately said:

“The system should not give suggestions.”

That wasn’t distrust. It was calibrated trust. They trusted the AI. Just not with that.

The system had quietly crossed a boundary they understood instinctively. Designing for calibrated trust means making those boundaries visible. Show where information came from. Help users verify it. Be clear about what the AI knows and what it doesn’t.

Most importantly, don’t encourage users to outsource judgement when judgement is exactly what they’re there to provide.

A surprising amount of good AI design is about restraint. About deciding what the system shouldn’t do.

Role clarity before feature clarity

That leads to the question I think many AI teams skip.

We spend enormous energy deciding what the system can do.

We spend much less time deciding who the system should be.

Is it an assistant? An evidence provider? A second pair of eyes? A collaborator? Or is it quietly becoming the decision-maker?

Before deciding what the AI should do, I think we need to decide what role we want it to play.

Healthcare makes this easier because accountability is already clear. The clinician makes the decision. The clinician is responsible for the outcome. The AI supports that process.

Once you know the role, many design decisions become surprisingly obvious: how the AI speaks, how confident it should sound, whether it makes recommendations or presents options, what information it shows, when it deliberately holds back.

Those aren’t only product decisions. They’re interaction design decisions.

Why this matters now

For a long time, designing for trust, risk and human accountability felt like a specialty.

It was something you did because regulation required it.

AI has changed that.

Products are now helping people make decisions about their health, finances, careers and legal situations. The context changes. The underlying design problem doesn’t.

It’s still a human making a decision with the support of a system they’re ultimately accountable for.

Healthcare UX has been working on that problem for decades. Not because we predicted AI. Because we couldn’t afford not to.

Design for the moments when the system gets it wrong.

Design for trust that is earned instead of assumed.

Design with a clear boundary between what the AI owns and what the human owns.

AI didn’t invent these questions. It simply made them everyone’s problem.

I’m a Senior AI Product Designer with 9+ years of experience, currently building AI systems for healthcare. I write about design governance, risk, and accountability in high-stakes products. Follow along, there’s more coming.