Explainable AI: Insights from market and academic research

*Written by Roosevelt Faulkner and Phil Doyle, PhD

Edited by Carole Baden*

A 3D illustration of interconnected digital objects arranged in a flowing double-helix pattern, representing the integration of diverse data sources and systems in a hybrid cloud environment.

Trust is not a feature, it’s an outcome

As AI systems become more capable and more autonomous, enterprise teams face a deceptively simple question: how do you help people trust AI appropriately , neither blindly following it nor unnecessarily ignoring it? This review synthesizes market and academic research to answer that question for product designers, developers, and business leaders.

The central finding is that explainability is not a feature you add to a model; good explainability is a design outcome. An explanation only works if the right person receives the right information at the right moment. What a developer needs to debug a model, what an auditor needs to satisfy a regulator, and what an SRE needs to when investigating an issue are three entirely different things[8][18]. Explanation is not one-size-fit-all, it has to fit the individual person.

This review covers five practical areas:

• What explainability is actually for
• What kind of explanation is best for different types of users
• What technical tools are available
• How to design trustworthy interfaces
• What changes when AI systems become agentic, executing workflows autonomously rather than just responding to queries.

This last area, the agentic shift is where the stakes are highest and current practice is least mature.

The goal throughout is calibrated trust; users who rely on AI have confidence in it when it’s reliable, question it when it’s not, and have the visibility they need to tell the difference [19][13].

The goal is well-calibrated trust. This is the sweet spot. The user has an accurate understanding of the AI’s capabilities — its strengths and, crucially, its weaknesses. They know when to rely on it and when to be skeptical. Victor Yocco, PhD, UX Researcher, ServiceNow

Explainability isn’t about showing your work. It’s about helping people act.

The instinct is to treat explainability as a transparency feature — show users the reasoning, job done. But effective explainability is more than making information visible. It’s about making the right information meaningful to the right person at the right moment, and that’s a harder design problem than most teams expect [7][9].

Explainability isn’t a property of the model; it’s something that happens at the moment a specific person interacts with a specific output [8][9][15]. The same model decision might need a three-sentence plain-language summary for a customer, a detailed audit trail for a regulator, and a feature attribution breakdown for a developer. None of those is more “correct” than the others — they serve different purposes, for different people, with different stakes [6][16][18]. A good explainability strategy accounts for all of them and recognizes that over-trust and under-trust are equally costly failure modes [8][9][15].

The same explanation doesn’t work for everyone.

Developers need explanations that are technically faithful, meaning the explanation accurately reflects what the model actually did, not a plausible-sounding approximation. They use explanations to debug, catch bias, and verify that model behavior matches intent. Inaccurate but readable explanations are worse than useless here [8].

Auditors and compliance teams need traceability, documented evidence that a decision can be reconstructed and justified. This means version histories, data provenance, and mathematically rigorous attribution. Speed doesn’t matter; defensibility does [3].

End users such as SREs and Cloud engineers need actionability. Less concerned with how the model works, they want to know where the information came from, how confident the system is, and whether they can trust this particular output enough to act on it.

Use LIME for users. Use SHAP for audits.

Most enterprise AI relies on post-hoc methods — tools that explain a model’s decisions after the fact without changing the model itself [12]. The two dominant options are LIME and SHAP, and the choice between them is essentially a tradeoff between speed and rigor. Use LIME for real-time user interfaces. Use SHAP for audits, compliance, and model monitoring. SHAP is slower but consistent and defensible [2]. LIME is faster but less reliable [8][18][20]. Most enterprise teams need both.

One emerging approach that seems promising is using LLMs to translate SHAP outputs into plain-language narratives, to bridge the gap between what data scientists produce and what business users can act on [1][18]. The risk here is that fluent narratives can sound confident even when imprecise. The narrative needs to be faithful to the actual model logic, not just plausible.

Most users either skip explanations or misread them. Both are design problems, not user problems.

Two failure modes dominate in practice: users skip explanations entirely due to overload or habit, or they misapply them due to cognitive bias or time pressure [11]. The most effective design response to both is the same: progressive disclosure. Lead with a short, plain summary enough for the user to act confidently in routine cases. Make detail available on demand. Never front-load the full reasoning; it crowds out the signal and most users won’t engage with it until they need to [2][8].

A three-tier model works well in practice [10][11][17]:

• Tier 1: Always visible: A one or two sentence summary of what the AI did and why.
• Tier 2: On engagement: The main factors driving the decision, revealed when a user clicks in.
• Tier 3: On demand: Full attribution, chain-of-thought, and validation details for users who need to investigate or audit.

Strategic friction is underused. In high-stakes decisions, deliberately slowing users down by requiring a confirmation step, or surfacing a key piece of reasoning before they can proceed meaningfully, improves engagement with explanations [11][13][18]. The goal isn’t to annoy users; it’s to interrupt the habit of clicking through without thinking.

The north star metric is whether users can accurately predict how the system will behave in situations they haven’t seen before, a measure researchers call mental model soundness. If they can, the interface is working. If they’re regularly surprised by outputs, the explanation layer isn’t doing its job [6][8].

Establishing accountability with agentic systems is a delicate balance

Agentic AI doesn’t just raise the stakes. It changes the rules.

Everything we’ve described thus far assumes a relatively static interaction: the user asks, the model responds, an explanation accompanies the response. Agentic AI fundamentally changes that process. When systems are executing multi-step workflows autonomously (e.g. taking actions, calling tools, making sequential decisions) a single point-in-time explanation isn’t enough [8][14][20].

Governance has to move from pre-release review to real-time monitoring [15]. Errors compound across steps, and by the time a human reviews an output, several downstream actions may have already been taken [13]. The system architecture needs to support intervention at any point in a workflow, and agents need to generate traceable justification for each significant choice during planning and execution, not just provide a summary at the end [4].

Users become orchestrators, not operators [5]. In multi-agent systems where clusters of specialized agents collaborate on complex tasks, users aren’t monitoring individual decisions. They’re setting goals, reviewing outcomes, and intervening when something looks wrong [17][20]. Interface design needs to support high-level visibility into what agent clusters are doing, define escalation paths when something deviates, and provide organizational metaphors (e.g. think “the Finance Team” rather than “agents 4, 7, and 12”) that help users reason about system behavior without tracking every sub-process [4][13][15].

The accountability question gets complicated. Responsibility for agent behavior is distributed across the model developer, the system deployer, and the user. This distribution is appropriate, but it creates a real risk of what researchers call a “moral crumple zone”, which is the human nominally in the loop gets blamed for failures they had no practical ability to prevent. Good audit trails and action logs are the main defense. Users need enough visibility to one or the other so they can intervene effectively or demonstrate credibly that they couldn’t have [4][15][16].

Build on it, don’t bolt it on

Explainability research is maturing quickly, but practice is lagging behind. Most enterprise teams are still treating it as a compliance requirement rather than a design discipline. That gap is where trust gets lost. The teams that close it will be the ones who recognize that trust is a user outcome: something you measure, iterate on, and improve over time, not something you achieve by attaching a SHAP chart to a decision. As AI systems take on more autonomous roles, the window for human intervention gets smaller and the cost of misplaced trust gets higher. Building explainability into system architecture from the start, not as an afterthought, is how that cost gets managed.

References

1. Bilal, Ebert and Lin (2025) LLMs for Explainable AI: A Comprehensive Survey
2. Chromik and Butz (2021) Human-XAI Interaction: A Review and Design Principles for Explanation User Interfaces
3. Delev (2025) AI Transparency Requirements: Compliance and Implementation
4. Harkar (2025) IBMs Agentic AI Governance Playbook
5. Krishnan (2025) How IBM makes AI based on trust, fairness and explainability
6. Kulesza, Stumpf, Burnett and Kwan (2012) Tell Me More? The Effects of Mental Model Soundness on Personalizing an Intelligent Agent
7. Kulesza, Burnett, Weng-Keen Wong and Stumpf (2015) Principles of Explanatory Debugging to Personalize Interactive Machine Learning
8. Liao and Varshney (2022) Human-Centered Explainable AI (XAI): From Algorithms to User Experiences, IBM Research
9. Lim, Dey and Avrahami (2009) Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems
10. Microsoft (2026) Progressive Disclosure agent Skills
11. Naiseh (2021) C-XAI: Design Method for Explainable AI Interfaces to Enhance Trust Calibration
12. Rambod (2025) The Epistemology of Transparency: A Comprehensive Analysis of Explainable Artificial Intelligence Frameworks, Mathematical Foundations, and Regulatory Architectures
13. Schömbs, Zhang, Goncalves and Johal (2026) From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systems
14. Shankar (2022) Explainable AI (xAI) in Critical Decision Systems Algorithm, Frameworks, and Case Studies
15. Shavit et al. (2023) Practices for Governing Agentic AI Systems
16. The European Commission (2020) ETHICS GUIDELINES FOR TRUSTWORTHY AI
17. Thoutam (2025) SHAP vs LIME: Which XAI Tool Is Right for Your Use Case?
18. Upol Ehsan and Mark O. Riedl (2020) Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach
19. Victor Yocco (2025) The Psychology of Trust In AI: A Guide to Measuring and Designing for User Confidence
20. Zebin, Sudjianto, Li and Zhang (2024) Inherently Interpretable Tree Ensemble Learning

About the authors

Roosevelt Faulkner is a User Research Lead at IBM, based in Austin, Texas. The above article is personal and does not necessarily represent IBM’s positions, strategies, or opinions.

Phil Doyle, PhD, is a User Research Lead at IBM, based in Dublin, Ireland The above article is personal and does not necessarily represent IBM’s positions, strategies, or opinions.

This work was created with an even blend of human and AI contributions. AI was used to make stylistic edits, such as changes to structure, wording, and clarity. AI was used to make content edits, such as changes to scope, information, and ideas. AI was prompted for its contributions, or AI assistance was enabled. AI-generated content was reviewed and approved. The following model(s) or application(s) were used: Microsoft Copilot Chat, Google NotebookLLM, and Claude (Sonnet 4.6).