Your Model Made a Decision. Can You Explain It?

A practical guide to SHAP and LIME — two tools that turn black-box predictions into something a human can actually understand

For anyone who has been working in data and AI I believe you have heared the question that keeps coming up — in audits, in board meetings, in conversations with regulators — is not “how accurate is your model?”

It is: “why did it make that decision?”

For a loan rejection. For a hiring filter. For a medical diagnosis recommendation. Accuracy is a score. Explainability is accountability.

With the EU AI Act now in force and high-risk AI systems required to provide meaningful explanations under Article 86, this is no longer a nice-to-have. It is a legal obligation.

So I put together two notebooks — one for SHAP and one for LIME — published on GitHub, and I want to walk you through what I learned building them, what each tool actually does, and critically, when you should reach for one over the other.

The problem with black boxes

A Logistic Regression trained in 1990 was explainable by design — you could look at the coefficients. Modern models are not. A Gradient Boosting ensemble with 500 trees, or a neural network with millions of parameters, makes predictions through a process that is deeply non-linear and practically impossible to interpret by reading the model internals.

This is where SHAP and LIME come in. Both are post-hoc explainability methods — they do not change the model, they interpret it from the outside. Think of them as translators between the model’s mathematical world and the human world of reasons and decisions.

SHAP — The rigorous one

SHAP stands for SHapley Additive exPlanations. The name comes from cooperative game theory — specifically from Lloyd Shapley’s 1951 work on how to fairly distribute the payout of a cooperative game among its players.

The analogy to machine learning is precise: your features are the players, the prediction is the payout, and SHAP calculates each feature’s fair contribution to that prediction by considering every possible combination of features. It is mathematically guaranteed to be the only attribution method that satisfies four specific fairness axioms simultaneously — consistency, dummy, efficiency, and symmetry.

In practical terms, what SHAP gives you is this: for any single prediction, it tells you exactly how much each feature pushed the outcome up or down from the model’s average prediction. And it does this in a way that is provably correct, not approximated.

I used the Adult Income dataset in the notebook — predicting whether a person earns over $50K based on demographics and employment data. This is a deliberately uncomfortable dataset because it involves sensitive attributes: sex, race, marital status. Exactly the kind of model where explainability and fairness intersect.

The waterfall plot — start here with stakeholders

The waterfall plot decomposes a single prediction step by step, showing how the model arrived at its conclusion for one specific person. It is where I usually start when presenting to a governance board or compliance team, because the logic is immediately readable even to non-technical audiences.

The waterfall plot for one instance from the Adult Income dataset. E[f(X)] = 0.202 is the model’s average prediction across all training data — roughly one in five people are predicted to earn above $50K. f(x) = 0.005 is what the model predicted for this specific person: almost zero probability of high income. Each bar is one feature’s contribution. Blue bars push the prediction down, red bars push it up. They sum exactly to the difference between the two end values — no rounding, no approximation. For this person, Marital Status alone accounts for a drop of 0.07, and Relationship status drops it another 0.04. By the time all features are applied, the prediction has fallen from 0.202 to just 0.005. This is the kind of evidence that belongs in a governance audit.

What makes the waterfall plot powerful in a governance context is that it gives you a precise, auditable answer to the question a regulator or affected individual might ask: “Why was this person’s application treated differently?” You can point to each bar and give a reason. That traceability is what Article 86 of the EU AI Act requires.

The bar plot — which features matter globally

Before going deep on individual predictions, it is worth understanding which features the model relies on most across the entire dataset. The bar plot shows the mean absolute SHAP value for each feature — a measure of average influence that is mathematically grounded rather than heuristic.

Mean absolute SHAP values across 200 test instances — the model’s global feature importance. Marital Status and Education level dominate. Unlike tree-based feature importance metrics, SHAP importance is consistent: if a feature is genuinely more influential than another in this model, its SHAP value will always be larger. This makes it defensible in a governance context in a way that traditional importance scores are not.

The bar plot is the natural starting point for a governance conversation about the model overall. It answers: what is this model actually doing, at the population level? If Marital Status is the top feature and your use case is employment decisions, that is a question worth asking out loud in a review meeting.

The beeswarm plot — the most informative governance visualization

If I had to pick one SHAP visualization to show a governance board, it would be this one. The beeswarm plot puts every instance and every feature on the same canvas simultaneously.

The beeswarm plot across 200 test instances. Each dot is one person. The horizontal position is the SHAP value — how much that feature shifted the prediction. The color is the feature value itself: red means a high value for that feature, blue means low. Features are ordered by global importance from top to bottom. Notice the Sex row fifth from the top: the blue cluster on the negative side represents female passengers, and it sits consistently to the left of zero — meaning the model is using gender to reduce income predictions. Whether that reflects genuine economic reality or constitutes discrimination is not a question SHAP can answer. But it surfaces the pattern clearly enough for the right people to ask it.

The beeswarm plot also reveals something that the bar plot cannot: how consistent a feature’s effect is. Capital Gain, for instance, shows a wide horizontal spread — when it is high it has a massive positive impact, but for most people it is zero and its effect is negligible. Marital Status shows tight clusters, meaning its effect is consistent and predictable. These are very different risk profiles from a governance perspective.

The decision plot — tracing the path to a prediction

The decision plot traces the exact path from the model’s average prediction to the final prediction for a specific instance, adding one feature’s contribution at a time. When you overlay multiple instances — high-income and low-income predictions together — you see precisely where the model separates the two groups.

Decision plot comparing 8 high-income predictions (orange lines) vs 8 low-income predictions (blue lines). All lines start together on the left at the baseline (0.202) and separate as features are applied one by one. The point where the two groups diverge most sharply — where orange lines swing right and blue lines swing left — is the point where the model is making its most consequential distinctions. This is the visualization I would show an auditor who asks: “What is this model actually doing to separate approved applicants from rejected ones?”

The decision plot is particularly useful when you need to explain a model’s behavior to someone who is not a data scientist. The visual metaphor of paths diverging is intuitive. You do not need to explain SHAP values — you can simply say: “These two groups start at the same place. Here is where the model starts treating them differently, and here is why.”

Fairness screening — where SHAP becomes a governance tool

This is the step that takes SHAP from a data science tool to a compliance instrument. If a model is systematically using sensitive attributes like Sex or Race to influence predictions — even as secondary factors, even subtly — SHAP will show it.

SHAP value distributions for Sex and Race across 200 test instances. For Sex, the distribution is not centered at zero — there is a consistent negative skew, meaning the model uses gender to reduce income predictions for a subset of individuals. The mean SHAP value for a sensitive attribute tells you the direction of that influence. The spread of the distribution tells you how widespread it is. Under EU AI Act Article 10, documenting exactly this kind of analysis is part of what proper data governance looks like for high-risk AI systems.

It is important to be precise about what this analysis shows and does not show. A non-zero SHAP value for Sex does not automatically mean the model is discriminating — the economic data it was trained on reflects real-world inequality, and the model has learned those patterns. What SHAP does is make the pattern visible and measurable. The decision about whether it is acceptable sits with the governance board, not the data scientist. SHAP just makes sure that decision cannot be avoided.

LIME — The intuitive one

LIME stands for Local Interpretable Model-agnostic Explanations. It was introduced in a 2016 paper by Ribeiro, Singh, and Guestrin, and the core idea is elegant in its simplicity.

LIME does not try to explain the entire model. It explains a single prediction by asking: what simple linear model, trained only on the neighborhood around this data point, approximates the black-box model’s behavior locally?

It works by generating many slightly perturbed versions of the instance you want to explain, running them all through the black-box model, and fitting a linear regression on the outputs. The coefficients of that linear model are the LIME weights: how much each feature contributed to this specific prediction, in this specific context.

“LIME works with any model — including vendor models and APIs where you have no access to the internals. That makes it practically important in enterprise AI governance contexts.”

I used the Titanic survival dataset, which makes the explanations immediately human-readable. A passenger’s sex, ticket class, and fare price are things anyone can reason about — which makes LIME explanations easy to present to non-technical stakeholders.

Explaining a single prediction

The simplest LIME output is a horizontal bar chart for one instance. Green bars push toward the predicted class, red bars push away from it. The length of each bar represents the magnitude of that feature’s contribution to this specific prediction.

LIME explanation for one Titanic passenger. Sex_le (female = 0) and Pclass_le (first class = 1) dominate — both pushing strongly toward survival. The model has learned that being female and travelling first class were the strongest predictors of survival. For this passenger, these two features together account for the vast majority of the prediction. In a governance context, this is the kind of output you would generate on demand when an affected individual asks why the model made a particular decision about them — directly addressing the right to explanation under Article 86

The key thing to notice is that LIME gives you a local explanation — it is telling you what mattered for this passenger, not for passengers in general. A different passenger might have a completely different set of contributing features even if the final prediction is the same. This is both LIME’s strength and its limitation.

The comparison that reveals what the model considers decisive

Running LIME on a passenger who survived and one who did not, then plotting them side by side, is one of the most instructive analyses in the notebook. It reveals the features the model considers decisive — the ones that appear with large weights in opposite directions.

Left: A passenger who survived (96% model confidence). Right: A passenger who did not (only 7% predicted probability of survival — the model was almost certain of their fate). Sex_le dominates in both explanations but in opposite directions. For the survivor (female, first class), Sex_le pushes strongly positive. For the non-survivor (male, second class), Sex_le pushes strongly negative and is the single largest contributor to the low survival prediction. The model’s decision for these two passengers turns almost entirely on gender and ticket class. For Titanic survival, this reflects history. For any modern high-stakes classification system, this pattern would be an immediate governance concern requiring documented review.

This comparison is also useful for detecting model instability. If two passengers with similar profiles receive very different LIME explanations — different features driving their predictions — that suggests the model’s decision boundary in that region may be unstable or hard to justify. That is worth flagging regardless of whether the predictions themselves are accurate.

Aggregate importance — the global picture from local explanations

Most LIME tutorials explain one instance and stop there. Running LIME across fifty instances and averaging the absolute weights gives something more useful: a proxy for global feature importance, inferred entirely from local explanations.

Mean absolute LIME weights across 50 test instances. Sex_le dominates by a factor of nearly 3x over the next most important feature (Pclass_le). This aggregate view — computed by averaging the absolute LIME weights across many individual explanations — gives a global picture of model behavior without requiring access to model internals. It is less mathematically rigorous than SHAP’s global importance, but it works on any black-box model, including vendor APIs. From a governance perspective, the gap between Sex_le and everything else is a finding that cannot be ignored.

The aggregate importance plot is particularly valuable in contexts where you are auditing a third-party model. You may not know what algorithm was used, you may not have access to the model file, you may only be able to call a prediction API. LIME can work in all of these situations, and the aggregate analysis still gives you a meaningful picture of what the model is relying on.

SHAP vs LIME — when to use which

This is the question I get asked most often, so I will be direct about it.

Use SHAP when you need to understand the model itself. When you are doing a governance audit, when you need global feature importance that is mathematically justified, when you want to study feature interactions, when you need to present findings to a regulator or an AI ethics board. SHAP is slower to compute but its outputs carry theoretical guarantees that LIME’s do not.

Use LIME when you need to explain a specific decision to a specific person. When a customer wants to know why their loan was rejected. When a candidate wants to understand why they were filtered out. When you have a vendor model you cannot access internally. LIME is faster for single predictions, works with any model type including image and text classifiers, and produces explanations that translate easily into plain language.

There is also a fundamental difference in what they guarantee. SHAP guarantees that the feature attributions are theoretically correct — consistent, fair, and exhaustive. LIME makes no such guarantee. It is an approximation, and the quality of that approximation depends on the complexity of the model’s decision boundary near the instance being explained. Running LIME twice on the same instance can give slightly different results. That is not a flaw — it is a known property of local approximation — but it matters for governance documentation where reproducibility is expected.

In practice, for serious AI governance work, I use both. SHAP for the systematic model-level analysis. LIME for the individual-level explanations that need to be communicated to people who are actually affected by the model’s decisions.

The EU AI Act angle

Article 13 requires that high-risk AI systems be transparent — providers must give deployers enough information to understand the system’s behavior. Article 86 gives individuals the right to an explanation when an AI system makes a significant decision about them. Neither article specifies how to implement these requirements technically.

That gap is where tools like SHAP and LIME live. What I have tried to do in both notebooks is not just demonstrate the technical mechanics but frame each analysis as a governance artifact — something that could be attached to an audit, submitted to a regulator, or shared with an AI ethics committee. Every notebook ends with a compliance checklist mapping the analysis to specific EU AI Act articles.

Explainability tools are not just interesting data science. They are part of the infrastructure of accountable AI. The sooner development teams treat them that way, the less scrambling there will be when regulators start asking questions.

The notebooks

Both are on GitHub, fully commented, and ready to run in VS Code, Jupyter, or Google Colab without any local setup beyond installing the packages.

The SHAP notebook covers: waterfall plot, force plot, bar plot, beeswarm plot, heatmap, scatter and dependence plots with interaction coloring, single and multi-instance decision plots, fairness screening for Sex and Race, and a governance compliance summary.

The LIME notebook covers: exploratory data analysis, single instance explanation, side-by-side comparison of opposite outcomes, aggregate feature importance across fifty instances, SP-LIME for global model coverage, and a governance summary including a direct SHAP vs LIME comparison.

**GitHub Repository**

https://nbviewer.org/github/ash-zoheri/ashzoheri/blob/main/lime_example_enhanced.ipynb

https://nbviewer.org/github/ash-zoheri/ashzoheri/blob/main/SHAP_example_enhanced.ipynb