Lesson 3 — Explainability and its limits
Two questions get merged, and separating them clarifies a great deal:
Why did the model produce this output for this case? — a technical question, partially answerable. Can the person affected understand and contest the decision? — a governance question, and the one that matters.
An answer to the first does not automatically supply the second.
What the methods actually give you
Feature attribution assigns each input a contribution to a specific prediction. Two families dominate: one perturbs inputs and fits a simple local model to the results, another draws on cooperative game theory to allocate credit among features. Both output something like:
Loan declined. Income contributed −0.3, existing debt −0.4, employment length +0.1, postcode −0.2.
Genuinely useful for debugging, for spotting a model relying on something it should not, and for satisfying a duty to give reasons.
Counterfactual explanations state what would have needed to differ:
Approved if income were €4,000 higher, or if existing debt were €10,000 lower.
Frequently more useful to an affected person, because it is actionable — with the caveat that some counterfactuals are not achievable, and "be five years younger" is not advice.
Saliency maps highlight the regions of an image that most influenced a prediction. Excellent at catching a model reading the background, the timestamp or a ruler placed in the frame.
Example-based explanation shows similar training cases that led to this outcome, which non-technical audiences often find more convincing than numbers.
Five limits worth stating plainly
1. They are approximations of behaviour, not descriptions of reasoning. A network with billions of parameters does not reason in terms of feature contributions. An attribution is a simplified account that approximates the model's behaviour near this input. Useful, and not a window into what happened.
2. Methods disagree. Run two attribution methods on the same prediction and you can get different rankings. There is no ground truth for "the correct explanation", which means an explanation cannot be validated in the way a prediction can.
3. They can be manipulated. Research has demonstrated models constructed to behave in a discriminatory way while producing innocuous-looking explanations. An explanation is therefore not proof of anything about the model's behaviour, which matters when it is used as evidence of compliance.
4. Correlation is not causation. An attribution says a feature was associated with the prediction, not that changing it in the world would change the outcome. Acting on that distinction incorrectly leads people to optimise proxies.
5. Explanation is not justification. "Declined because your postcode contributed negatively" explains the mechanism and does not justify it. Arguably it reveals a problem. A satisfying explanation of an unacceptable decision is still an unacceptable decision.
Interpretable by construction, which is usually better
The alternative to explaining a complex model is using one whose behaviour is legible from the start.
| Interpretable model | What you can read directly |
|---|---|
| Linear or logistic regression | Each feature's weight and direction |
| Decision tree, kept shallow | The exact rule path taken |
| Rule lists | An explicit if-then sequence |
| Generalised additive models | Each feature's effect as a curve you can inspect |
The received wisdom is that this costs too much accuracy. On tabular data with well-constructed features, the gap is frequently small — sometimes negligible — and the argument that you must accept opacity to get performance has been challenged persuasively for exactly the high-stakes decisions where it is invoked most.
A defensible position: for a decision that materially affects a person, start with an interpretable model and require a compelling accuracy argument before adopting an opaque one. The burden of proof runs the other way from how it is usually applied.
What an affected person actually needs
Explainability discussions often centre on what engineers find interesting rather than on what a person needs in order to contest a decision. The latter is:
That a decision was made about them, and that a system was involved. Notification comes before explanation, and is frequently missing.
The main factors, in language they use. Not attribution values. "Your existing debt relative to income was the largest factor."
What would change the outcome. Actionable and truthful, including when the honest answer is "nothing you can change soon".
How to challenge it. A route to a human with authority to reconsider, within a stated timeframe.
Whether the same thing happens to people like them. Aggregate outcome information, which almost nobody provides and which is what makes systemic discrimination visible to those affected.
The last two do more for accountability than any attribution method, and neither is a technical problem.
A note on language models
Asking a language model to explain its own answer produces a plausible narrative that is generated by the same process as the answer and is not a report of internal computation. It may be accurate. It may be a confident rationalisation of an arbitrary output, and there is no way to tell from the text.
Treat such self-explanations as unverified claims. Where an explanation carries weight, ground it in something checkable — a retrieved passage, a tool output, a rule — rather than the model's account of itself.
Explainability is a technical capability. Contestability is the goal. A system nobody can challenge is unaccountable regardless of how many attribution plots it produces, and a system with a clear appeal route to an empowered human is accountable even when the model is opaque.
In three sentences
Attribution methods, counterfactuals, saliency maps and example-based explanations are genuinely useful for debugging and for meeting a duty to give reasons, and they are approximations of a model's behaviour rather than descriptions of its reasoning — they disagree with each other, can be deliberately manipulated, confuse association with causation, and explaining an unacceptable decision does not justify it. For decisions that materially affect people, a model that is interpretable by construction is usually the better choice, since the accuracy gap on well-featured tabular data is frequently small and the burden of proof should fall on adopting opacity rather than on avoiding it. What an affected person actually needs is notification, the main factors in plain language, what would change the outcome, a route to a human with authority to reconsider, and aggregate information about people like them — and contestability, not explainability, is what makes a system accountable.