Module 4 — Explainability: LIME, SHAP and their limits

An applicant rejected by the credit model wants to know why. A regulator wants to verify that the model is not using a forbidden feature. A data scientist wants to check that the model is behaving reasonably outside the training distribution. All three needs are called "explainability", and they are not the same problem. This module distinguishes local from global explanations, applies SHAP and LIME to the credit scorer, and marks the line between what these tools tell us and what they do not.

Global versus local explanations

A global explanation describes the model's overall behavior. Which features drive predictions in general? How does the average prediction change when income goes from 20 000 to 60 000? These are answered by feature importance rankings, partial dependence plots, and accumulated local effects. They are useful for auditors and for debugging.

A local explanation targets one prediction: this specific applicant was rejected, which features contributed most, and by how much? A rejected applicant does not care about the model's average behavior, they want to know what happened in their file. Local explanations are the ones that end up in the notification letter and, in some jurisdictions, are legally mandated.

Both are computed after training, on a model that already exists. That is why the family is called post-hoc explainability, in contrast with interpretable by design models (short decision trees, linear models with a handful of features) that need no separate explanation because their prediction and its reason are the same object.

SHAP and Shapley values

SHAP (SHapley Additive exPlanations) borrows an idea from cooperative game theory. The prediction for one applicant is treated as a "payoff", and each feature is a "player" who might or might not participate. Shapley values assign to each feature the average marginal contribution it makes to the prediction over all possible orderings of the players. The result is the unique attribution scheme that satisfies four axioms — efficiency, symmetry, dummy, additivity — that a fair sharing rule should have.

Two properties make SHAP dominant in practice. First, contributions are additive: the applicant's rejection probability equals the base rate plus the sum of every feature's contribution, so the attribution is auditable. Second, TreeExplainer computes the exact values for tree-based models (gradient boosting, random forest) in polynomial time. For other models, KernelExplainer provides an approximation.

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Global: mean absolute SHAP per feature
shap.summary_plot(shap_values, X_test, plot_type="bar")

# Local: waterfall for applicant number 42
shap.waterfall_plot(shap.Explanation(
    values=shap_values[42],
    base_values=explainer.expected_value,
    data=X_test.iloc[42],
    feature_names=X_test.columns,
))

On the credit scorer, the global bar chart puts existing debt, income and payment history at the top — reassuring. The waterfall on applicant 42 shows that ZIP code contributed +0.14 to the rejection probability, the third-largest positive contributor. That is exactly the proxy variable of module 2 showing up in a single decision, in a form the compliance team can read.

LIME and its instability

LIME (Local Interpretable Model-agnostic Explanations) takes a different route. It perturbs the input around the applicant, gets predictions for each perturbation, and fits a simple linear model to those perturbations. The linear model's coefficients are the explanation.

LIME is intuitive but exposes a real weakness: the perturbation scheme is a hyperparameter. Change the perturbation distribution, the number of samples, or the kernel width, and the explanation shifts — sometimes drastically. Two LIME runs on the same prediction can produce different top features. That instability makes LIME unreliable as a legal document. It remains useful as a debugging aid, where the qualitative signal is enough.

The lesson is not "LIME is bad" but "an explanation method has its own uncertainty, and that uncertainty must be reported alongside the explanation". SHAP has its own edge cases — feature correlation confuses the additivity story, and interventional versus observational SHAP can disagree — but its instability is far smaller.

The line between explanation and justification

A model's explanation says what the model did. It does not say what the model should have done. If SHAP reveals that the credit model rejected an applicant because of a ZIP-code effect that mirrors a historical redlining pattern, the explanation is faithful and the decision is unjust. Fixing the injustice requires changing the model or the data, not writing a better explanation.

The failure mode is common. A team runs SHAP, sees a plausible-looking chart, and concludes the model is fair. But SHAP shows how the model weighs its inputs; it does not evaluate whether the model should be using those inputs at all. Explainability is a necessary tool of the audit, not its verdict. The audit's verdict comes from combining explanations with the fairness metrics of module 3 and the data-source analysis of module 2.

When to prefer an interpretable model

For many tabular problems, a decision tree of depth four, a rule list, or a generalized additive model reach accuracy within a couple of points of gradient boosting. When they do, they should be preferred: the "explanation" is the model itself, no separate library, no instability, no axiom to trust. Rudin (2019) argues forcefully that in high-stakes settings — criminal justice, health, credit — the burden of proof should be on the team that chooses an opaque model over an interpretable one. That argument is worth quoting in the model card whenever a black-box model is retained.

Summary

• Global explanations describe the model's overall behavior; local explanations target one prediction — both are needed and address different audiences.
• SHAP attributes the prediction to features using Shapley values from cooperative game theory; TreeExplainer gives exact, fast results on tree models.
• LIME fits a linear surrogate on perturbations; its instability across runs makes it unfit as a legal document but useful for debugging.
• An explanation is not a justification: a faithful account of an unjust model does not make it just — that requires changing the model or the data.

Next module: keeping personal data out of the model's memory with anonymization, k-anonymity and differential privacy.