Your AI Model Can Be Right and Still Fail an Audit — A Lesson in AI Explainability

What a difficult examination taught us about the difference between explaining AI to a data scientist and explaining it to a regulator

It was a Thursday afternoon when the call came in from one of our clients — a mid-size property and casualty insurer we’d been working with for about a year.

Their compliance lead was on the line, voice tight: “The state insurance commissioner’s office wants to schedule an examination. They’ve flagged our AI-assisted claims system. They want to understand how it makes denial decisions.”

Their lead ML engineer jumped in: “We have full explainability. We’re using SHAP values.”

There was a pause. Then the compliance lead: “What’s a SHAP value?”

That pause — and everything that followed over the next eighteen months — changed how we think about AI explainability entirely.

They Had Explanations. Just Not for Anyone Who Mattered.

Our client — let’s call their lead ML engineer Arjun — had spent the better part of 2022 building what his team was genuinely proud of: a gradient-boosted classifier — a traditional machine learning model — that assisted claims adjusters in processing residential water damage claims.

Not an LLM but a well-tuned, structured ML model trained on historical claims data, with tabular features like claim age, damage type, documentation completeness, and prior claim history. The model was good. Their accuracy metrics were strong. And when regulators or auditors asked how the model worked, they had a beautiful dashboard ready to go.

Feature importance scores. SHAP waterfall charts. Local explanations for individual predictions. The works.

The data science team understood it immediately. The head of actuarial science nodded along. Everyone with a statistics background got it.

But the insurance commissioner’s office didn’t send a data scientist. They sent a lawyer and a senior policy examiner.

Two people whose entire job is to protect consumers from unfair claims handling. Two people who, when they saw a blue bar labeled “days_since_last_claim: −0.23,” looked at Arjun like he had just handed them a document written in Mandarin.

Even when explanation techniques like SHAP or LIME are applied, the outputs may be difficult to interpret or may not correspond to meaningful features from a domain expert’s perspective. Arjun had read that line in a paper once and nodded abstractly. Now he was living it, in a conference room, with the state’s insurance commissioner’s office on the other side of the table.

They spent two hours in that room. By the end, Arjun had answered exactly zero of their actual questions. Not because the model was wrong. Because the explanations were in the wrong language for the wrong audience.

The Audience Problem Nobody Talks About

Here’s what the ML community gets backwards about explainability.

SHAP and LIME were designed for data scientists — for the engineers who need to debug models, catch data leakage, and validate that the right features are driving predictions. For a traditional ML model like Arjun’s, they are the correct tool. The SHAP values were technically accurate. The feature attributions were real. The explanations faithfully represented what the gradient-boosted model had actually done.

That was never the problem. The problem was the audience.

But the people who most need AI systems to be explainable — customers whose claims were denied, regulators enforcing fair treatment, frontline adjusters defending a decision to a policyholder — aren’t data scientists. They’re asking a fundamentally different question.

A data scientist asks: Which features drove this prediction, and by how much?

A regulator asks: Why was this person’s claim denied, and is that reason legal and fair?

A customer asks: What did I do wrong, and what can I do differently?

SHAP answers the first question elegantly. It answers the second and third questions not at all.

After the examination, the client had three months to remediate — and they brought us in to help redesign the architecture. The core question we debated together was whether to keep the traditional ML model and find a better way to present its outputs, or to migrate to a fundamentally different approach. We ultimately recommended the latter: replacing the gradient-boosted classifier with an LLM-powered decision assistant built around chain-of-thought prompting. Not because the old model was inaccurate, but because no amount of presentation polish was going to turn a SHAP waterfall chart into something a regulator could meaningfully audit. That’s when chain-of-thought reasoning reframed the entire problem.

What Chain-of-Thought Actually Does

The idea is deceptively simple. Instead of training a model and then separately trying to explain what it did after the fact, you instruct the model to reason out loud before it reaches a conclusion.

A standard prompt to an LLM might be: “Based on this claim file, should this water damage claim be approved or denied?”

A chain-of-thought prompt says: “Based on this claim file, reason step by step through the relevant policy clauses, the evidence submitted, and any gaps in documentation. Show your reasoning before reaching your conclusion.”

Chain-of-thought prompting significantly enhances the reasoning capabilities of large language models by encouraging them to articulate their thought processes, leading to improved accuracy and interpretability.

What comes out is not a feature importance chart. It’s a paragraph that reads something like:

“Policy Section 4.2 covers sudden and accidental water damage. The claimant submitted photos dated March 14th showing water staining along the basement wall. The adjuster’s report notes the staining pattern is consistent with gradual seepage over an extended period rather than a sudden event. Section 4.7 of the policy explicitly excludes damage resulting from gradual seepage. On this basis, the claim does not meet the coverage criteria under Section 4.2 and falls under the Section 4.7 exclusion. Recommendation: Deny.”

A regulator reading that doesn’t need a statistics degree. A customer receiving that explanation understands exactly what happened and exactly what their policy says. A claims adjuster can hand that reasoning to a policyholder and have a real conversation. The explanation and the conclusion come from the same reasoning process, written in plain language, referencing actual policy text — not a bar chart that requires a statistics degree to decode.

The Part That Surprised Everyone: It Also Gets More Things Right

We expected chain-of-thought to improve the communicability of AI decisions.

What neither we nor the client anticipated was that it would also improve the quality of those decisions.

When Arjun first heard this claim, he was skeptical. Making the model explain itself improves accuracy? That sounds like a nice story.

But the research backs it up. By decomposing complex problems into smaller, more digestible parts, LLMs can process each segment individually, leading to more precise and reliable answers. This granular approach reduces the likelihood of errors that might occur when attempting to solve multifaceted problems in a single step.

The intuition, once you think about it, is obvious. A model that must write out its reasoning step by step is forced to confront each step individually. It can’t jump to a conclusion. It can’t skip the part where it checks whether the exclusion clause actually applies. The requirement to show work creates the same discipline in an AI that it creates in a student taking an exam.

The reasoning chain preceding the answer illustrates the model’s thought process, enabling the audience to understand how the answer is derived. But more than that — writing out the reasoning chain forces the model to actually have a reasoning chain, rather than pattern-matching directly from input to output.

In the client’s own parallel evaluation, run across 500 historical claims where the ground-truth correct decision was known, the results were striking. The legacy model (traditional ML with SHAP explanations) had an error rate of 11% on edge cases — claims that involved multiple policy clauses, ambiguous documentation, or overlapping exclusions. The chain-of-thought system’s error rate on those same cases dropped to 4%.

The explanation and the quality improvement came from the same mechanism. We didn’t have to choose between being understandable and being accurate.

What They Built — and What Happened in the Next Examination

Eighteen months after that first painful regulatory examination, the commissioner’s office came back for a follow-up review. Different examiner, same mandate: satisfy us that your AI system makes fair, explainable decisions.

Arjun sat down in the same conference room. This time, instead of pulling up a SHAP dashboard, he pulled up the reasoning trace from three representative claim decisions — one approval, one denial, one referral to a senior adjuster.

The examiner read through the denial reasoning. She underlined two sentences with her pen. She looked up and said: “So the model is saying the damage pattern is inconsistent with sudden and accidental. Who makes that determination in the physical inspection?”

That was a real question. A meaningful one. Arjun walked her through the adjuster workflow, the inspection criteria, the documentation standards. She had understood the AI’s reasoning well enough to probe the weak point in it — not the AI’s reasoning, but the underlying process it was reasoning about.

At the end of the examination, she wrote in her notes: “The insurer’s AI-assisted claims system produces decision rationales in plain language that reference specific policy provisions. The reasoning is auditable by non-technical staff.”

They passed. When Arjun called to tell us, he said: “The examiner asked better questions in that room than our internal audit team ever did.”

The Honest Caveats

Chain-of-thought reasoning is not a magic bullet, and doing a disservice to clients means pretending otherwise.

The reasoning trace the model produces is a plausible reconstruction of how the conclusion was reached. Prior work has found that the presence of correct factual information in the intermediate reasoning steps was not always a reliable indicator of correct model output — meaning a model can occasionally produce a coherent-sounding rationale that doesn’t fully reflect the underlying computation. The reasoning trace is auditable and catches most errors. It is not a perfect window into the model’s internals.

This means human review still matters. For high-stakes decisions — claim denials, coverage disputes, anything that ends up in a complaints process — the chain-of-thought trace is the starting point for human adjudication, not the end of it.

The regulator can read the reasoning. The adjuster can verify it against the policy. The system is auditable in a way that a SHAP chart never was for a non-technical audience.

There is also a scale and latency consideration. Generating a reasoning trace takes more tokens and more time than a direct classification. For decisions that truly are low-stakes and high-volume, the overhead may not be justified. For decisions that affect real people’s financial lives — insurance, lending, healthcare, employment — the overhead is not a cost. It’s the minimum standard.

What This Engagement Taught Us

If we could go back to that Thursday afternoon before Arjun’s compliance call, we’d say one thing to his team:

Your explainability tools are excellent for your data scientists. They are useless for everyone else who matters.

The people whose lives are affected by AI decisions — and the regulators who protect them — don’t need feature attribution scores. They need a clear account of what the AI considered, what it concluded, and why. They need the kind of explanation you could give a reasonable person over a cup of coffee.

Chain-of-thought reasoning doesn’t bolt that explanation on afterward. It builds it into the decision itself. The AI that must reason out loud is a more accountable AI — not because a layer of post-hoc justification has been added, but because articulating the reasoning is part of how the conclusion is reached.

That’s not merely a presentation improvement. That’s a different philosophy of what it means for an AI system to be trustworthy.

If you’re building AI systems in regulated industries and wrestling with the explainability gap, we’d be glad to compare notes. This is a solvable problem — but only once you stop trying to solve it for data scientists.