The Right to Explanation is Enforceable: What GDPR Article 22 and EU AI Act Demand from Your AI Models.

There is a question that keeps surfacing in AI governance conversations, and it is more legal than it is technical.

When a credit model declines an applicant, what exactly does the institution owe that person?

Under GDPR Article 22, the answer is: meaningful information about the logic involved. Under EU AI Act Article 13, the answer extends further, high-risk AI systems must be designed so that deployers can interpret outputs and use them appropriately. Under Article 26 deployers themselves carry the obligation of human oversight and operational accountability.

The gap between those regulatory requirements and what most teams actually build is where SHAP and LIME live. They are not nice-to-haves for data science teams. They are the mechanism by which explainability obligations are implemented.

This piece is about how to read them as a governance professional.

The Setup: What You Are Actually Governing

Take a credit loan model. It operates across three feature clusters:

1. Financial behavior: checking account status, credit history, savings status. This tells you how the applicant has handled money in the past

2. Loan specifics: duration, amount, installment rate. This tells you the structure of the obligation being evaluated.

3. Personal context: age, employment status, number of dependents. This provides background risk context.

The target variable is binary. (0) means good credit. (1) means default.

Your job as an AI governance officer is not to build the model. It is to interrogate whether its patterns are explainable, fair, and lawful. To do that, you need to understand what the model is actually doing, and that requires understanding the algorithm first.

XGBoost: What You Need to Know Without Being a Data Scientist

The model in this context is XGBoost, a gradient-boosted tree algorithm. It builds hundreds of small decision trees sequentially, each one learning from the errors of the previous.

Think of it this way: you have 200 junior analysts, each trained on slightly different case files. No single analyst is perfectly reliable. But if 140 of them independently flag an applicant as likely to default, that consensus carries real weight.

XGBoost does not output a hard yes or no. It outputs a “probability score” — specifically, the predict_proba function returns two numbers for each applicant: the probability of good credit and the probability of default. A threshold, often 0.5, converts that score into a binary decision.

This matters for GRC for a reason that is not technical. Changing the threshold changes who gets declined. Now, this is a business decision that carries compliance implications.

In instances where investigations are expensive and time-consuming, compliance teams may raise the threshold to reduce false positives. If regulatory penalties for missing a high-risk applicant are severe, they may lower it to reduce false negatives. The trade-off between those two error types is ultimately governed by business objectives and the cost structure of being wrong, and it should be explicitly documented as such.

SHAP: The Compliance Practitioner’s Framework

SHAP (SHapley Additive exPlanations) answers a specific question: how much did each feature contribute to this prediction?

It draws from cooperative game theory, attributing each feature’s contribution fairly across all possible combinations of features. For governance purposes, what matters is not the mathematics. What matters is the three levels of analysis SHAP gives you and what each one is for.

Global View: The Beeswarm (Summary Plot)

The beeswarm plot shows feature importance across the entire dataset. Each dot represents one applicant. The x-axis shows the SHAP value — the direction and magnitude of each feature’s contribution. The colour shows the feature value (high or low).

Fig 1: Beeswarm plot visualizing each feature contribution to an applicant loan decision

This is your systemic pattern detection tool. When you look at a beeswarm and see that checking account status consistently pushes predictions upward for applicants with low balances, you are looking at a pattern that needs to be interrogated for fairness and potential proxy discrimination.

For regulators and senior leadership who find the beeswarm too dense, the bar plot gives a simpler global view, just the mean absolute SHAP values, ranked. Less nuanced, but more communicable. So it’s important that you know your audience.

Fig 2: A bar plot visualizing the contributing features of an applicant.

Local View: The Waterfall Plot

The waterfall plot operates on a single applicant. This is where Article 22 compliance becomes concrete.

Fig 3: A waterfall plot visualizing the contributing feature of an applicant loan decision.

A waterfall might read:

• Base model prediction: 0.15 (the model’s average starting expectation)

• Checking account status (poor): +0.28 (pushed the score up toward default)

• Loan duration (short): -0.09 (pulled it slightly back toward good credit)

• Final prediction: 0.72 — default likely

That is a human-readable explanation of an automated decision. It is not a summary of what the model generally does. It is a specific account of what happened to this applicant, in this instance, on this day.

Under GDPR Article 22, when an applicant exercises their right to explanation, the waterfall plot is your delivery mechanism. The obligation is not satisfied by pointing to documentation about how the model works in general. It is satisfied by providing information about the logic that governed their decision specifically. SHAP waterfall plots are built for exactly that.

LIME: The Auditor’s Tool for Black Boxes

LIME (Local Interpretable Model-agnostic Explanations) takes a fundamentally different approach. Rather than computing exact Shapley values using the model’s internals, LIME perturbs the input data slightly, observes how the predictions change, and fits a simple linear model to those local observations.

The analogy that holds up well: you cannot see inside the box, but you can poke it. Push slightly on one side and measure the response. Push from another angle. After enough structured pokes, you build a rough map of how the model responds to that specific prediction.

Fig 4: A LIME plot visualizing the contributing features for an applicant denied loan.

LIME is particularly useful in four governance scenarios:

1. When you do not have access to the model’s internals. If you are auditing a third-party vendor’s model and they will not give you access to model weights or architecture, LIME lets you interrogate behaviour from the outside. You become the regulator poking the box.

2. When the model is not tree-based. SHAP has native, computationally efficient implementations for tree models. For neural networks or other complex architectures, LIME’s model-agnostic approach is often more practical.

3. When you need a readable narrative quickly. LIME’s output is a simple weighted list, this feature pushed toward default, this one pulled away. It is immediately legible to non-technical stakeholders.

4. When you are deliberately probing model stability. This is counterintuitive, but LIME’s primary limitation becomes a governance tool when used intentionally.

Fig 5: Differences between SHAP and LIME visualization.

LIME’s Critical Limitation: Stochasticity

LIME uses random sampling. Run it twice on the same applicant and you get slightly different feature weights. This is not a bug you can configure away — it is structural to how LIME builds its local approximation.

The compliance implication is serious: if your explanation delivery mechanism uses LIME, you could give the same applicant different explanations on different days.

This stochasticity concern lands directly under NIST AI RMF GOVERN 1.2, which requires that the characteristics of trustworthy AI, including transparency and explainability to be integrated into organizational policies and practices. If your explanation methodology produces variable outputs, that variability must be documented, bounded, and governed. It cannot simply exist as an undisclosed technical characteristic.

It also surfaces under EU AI Act Article 13, which requires that high-risk systems be designed so that outputs can be interpreted appropriately. An explanation mechanism that produces inconsistent results for the same input does not meet that standard without additional controls.

And under Article 26, the deployer’s obligation to maintain human oversight and monitor system operation implies that the tools used to support that oversight must themselves be reliable. LIME, used without controls, does not satisfy that requirement unconditionally.

The practical guidance: where LIME is used for explanation delivery, institutions should run multiple iterations, average the results, and document the variance. The explanation provided should be the stabilized output, not a single LIME run. Where explanations will be delivered to applicants or regulators, SHAP’s deterministic outputs are generally the more defensible choice.

What This Means for the AI Governance Function

Explainability is not a data science deliverable that governance teams review after the fact. It is a compliance requirement that governance teams must specify before deployment.

This means:

Explainability methodology must be documented in the AI system’s risk assessment, not just the fact that explainability tools are used, but which tool, for what purpose, in what format, and to whom the outputs are disclosed.

Threshold decisions must be formally owned. The decision to set a classification threshold at 0.5, 0.4, or 0.6 is a governance decision with disparate impact implications. It should not be a default left in a configuration file.

LIME variance must be disclosed if LIME is the explanation mechanism. Regulators assessing GDPR Article 22 compliance or EU AI Act Article 13 conformity will ask how consistent the explanations are. If the answer is “it depends on the random seed,” that is not a compliant answer.

Waterfall plots must be stored, not just generated on demand. The right to explanation under Article 22 can be exercised at any time after a decision is made. If you only generate explanations on demand without storing them, you risk generating a different explanation post-hoc than the one the model would have produced at the time of decision. For tree-based models with SHAP, this risk is lower because SHAP values are deterministic. But the explanation must be reproducible.

SHAP and LIME are not reporting tools. They go way beyond that.

The AI governance officer who understands waterfall plots can defend a credit decision to a regulator. The one who understands LIME’s stochasticity can write a vendor assessment clause that actually protects the institution. The one who understands threshold mechanics can push back on a business decision that looks like a configuration choice but functions as a lending policy.

Explainability frameworks are only as useful as the governance function that reads them correctly. The frameworks themselves are ready. The question is whether the governance function is.