Four questions wearing one name
A project starts with someone saying the model has to be explainable, everyone agreeing, and nobody asking to whom or for what. Six weeks later there is an attribution library wired into the serving path, a dashboard of feature importances, and a customer-facing team that still cannot answer the only question they are ever asked, which is what the applicant should do differently. The work was competent. It answered a question nobody had.
Four distinct questions travel under the word explainability, and they have different audiences, different techniques and different costs. Why this outcome for this case — a per-decision attribution, asked by the person affected and the person defending the decision. What does the model use in general — a global description, asked by a validator or a reviewer. Is it relying on something it should not — a fairness and soundness question, asked by risk. What would have to change for a different outcome — recourse, asked by everyone who has to talk to a customer, and the only one of the four that is directly actionable.
Decide which of the four you owe, to whom, before choosing a method. Most disappointment in this area comes from delivering a good answer to question two when the organization needed question four.
You are probably here because
- Someone asked why a specific decision was made and the answer took two engineers a day
- The reasons given to a customer changed after a retrain and nobody could say why
- Two explanation methods disagree about the same case and there is no rule for which wins
- A reviewer wants a list of what the model relies on and the honest answer is a thousand-node ensemble
The first is a plumbing problem, the second is the stability requirement nobody wrote down, the third is a policy gap, and the fourth is often a signal that the model is more complex than the decision justifies.
Global explanation: what the model uses in general
Global methods describe average behaviour, and the useful ones are cheap. Permutation importance measures the loss increase when a feature is shuffled, which is intuitive and carries one caveat worth taking seriously: under correlated features it splits credit unpredictably between them, so two features that carry the same information can both look unimportant. In financial data, where half the features are transformations of the same three underlying quantities, this is the normal case rather than the exception.
Accumulated local effects plots show how the prediction moves with a feature while respecting the joint distribution, which makes them the right default over partial dependence plots — the latter average over feature combinations that never occur, and produce a curve for a region of the input space that has no examples in it.
Surrogate models — fitting a decision tree to mimic the complex model — are popular and should be used with a stated fidelity number and nothing else. A surrogate with an R-squared of 0.6 against the model it is explaining is not a description of the model; it is a different model with a pleasant appearance. Publish the agreement rate beside every surrogate or do not publish it.
Local attribution: why this case
For an individual decision, Shapley-value attributions have the properties you want in an argument: contributions sum to the difference between the prediction and a baseline, and the method has a defensible axiomatic story. For tree ensembles there is an exact algorithm fast enough for production. For other model families it is approximate and considerably more expensive, and the approximation error is rarely reported.
Two cautions matter more than the technique choice. First, the baseline is a choice with consequences. Attributions are always relative to something — a dataset average, a reference population, an all-missing input — and the chosen reference changes the explanation. Pick it deliberately, write it down, and use the same one everywhere, because two teams using different backgrounds will produce different explanations of the same decision and both will be correct.
Second, attribution is not causation. A Shapley value says how much a feature moved this prediction given how the model was fitted. It does not say that changing the feature would change the outcome, because the model has learned correlations. Presenting attributions to a customer as advice — "your outcome would improve if this number were higher" — is a claim the method does not support, and it is the most common misuse we see.
| Question | Audience | Fits | Where it goes wrong |
|---|---|---|---|
| What does it use in general? | Validator, reviewer | Permutation importance, ALE plots, a fidelity-scored surrogate | Correlated features split credit; surrogate reported without fidelity |
| Why this case? | Case handler, the affected person | Shapley attributions with a fixed, documented baseline | Read as causal; baseline undocumented and inconsistent between teams |
| Is it using something it should not? | Risk, oversight | Proxy analysis, subgroup performance, constraint testing | Testing only the excluded field and not its correlates |
| What would change the outcome? | Customer, front line | Counterfactuals restricted to actionable features | Counterfactuals that change age or postcode, which nobody can act on |
Recourse is what people actually want
Ask someone who received an adverse decision what they want to know and it is never a ranked list of contributions. It is what to do about it. A counterfactual explanation answers exactly that: the smallest change to the inputs that flips the outcome. It is more useful, more testable and easier to communicate than any attribution.
Making it work requires constraints the naive formulation lacks. Only actionable features may move — you can change a balance, not a date of birth. Directions must be plausible: some features only go up. Correlated features must move together, or the counterfactual describes a person who cannot exist. And an unstable counterfactual, one that jumps to a different recommendation on a trivially different input, is worse than useless because it destroys trust the first time two similar cases get contradictory advice.
Test recourse the same way you would test any promise: sample counterfactuals, apply the suggested change to the input, re-score, and confirm the outcome actually flips. It is a cheap test and it fails more often than teams expect, usually because the counterfactual search worked on a preprocessed representation and the mapping back to real-world quantities was approximate.
The requirement nobody specifies: stability
In many settings a declined applicant must be told the principal reasons for the decision, and those reasons are typically drawn from a fixed set of codes. This is the most concrete explainability requirement in commercial practice and it exposes a property that no explainability library optimizes for.
The reasons have to be stable. Two applicants with near-identical files should get the same reasons. The same applicant reapplying next month with unchanged circumstances should get the same reasons. A retrain should not silently change the reason distribution across the whole book. None of that follows from a method with high local fidelity — attributions on a boosted ensemble can reorder for a small input change, because the model itself is locally jagged, and a perfectly faithful explanation of a jagged model is a jagged explanation.
So treat stability as a first-class, measured requirement. Perturb inputs slightly and measure how often the top reasons change. Track the distribution of issued reason codes over time and alert when it shifts, which doubles as one of the better model-drift monitors available. And accept the trade honestly: a slightly less faithful explanation that is stable is usually the better product, because an explanation that changes without cause is one that cannot be defended in any conversation.
What an explanation has to be, in order of how often it is missing
How often each property is absent when we are first shown an explainability implementation. Judgment from repeated reviews. Fidelity is the one every library optimizes and the one least often missing.
Sometimes the answer is to constrain the model
There is a route that gets skipped because it feels like a retreat, and it is frequently the best engineering available: make the model easier to explain rather than explaining a hard one.
Monotonic constraints force a feature's effect to run in one direction. Most gradient boosting libraries support them and they cost surprisingly little accuracy on tabular financial data. The gain is large: a monotonic model produces explanations that never contradict domain sense, and it removes an entire class of embarrassment where an explanation says a favourable input hurt the applicant because the model found a local quirk.
Feature grouping and count limits. A model with twenty engineered features that a domain expert recognizes is dramatically easier to defend than one with four hundred automatically generated features, and on tabular data the accuracy difference is often within noise. Measure it before assuming otherwise.
Inherently interpretable models. Scorecards and small generalized additive models remain competitive on tabular problems more often than the field's enthusiasm suggests. The right move is to measure the gap on your data — fit both, compare on the metric that matters, and price the difference. If the complex model buys half a point of a metric and costs a permanent explainability programme, that is a trade worth naming out loud rather than assuming.
The explanation must come from the production path
The most damaging defect in this area is quiet: explanations computed in a notebook, on a different preprocessing pipeline from the one serving traffic. Everything looks fine, the numbers are plausible, and the explanation describes a model that is not the one making decisions. Compute explanations from the same artifact and the same preprocessing as the prediction, in the same call, and store them with the decision. If the latency budget cannot take it, store the inputs and the model version and compute asynchronously — but never recompute later from a rebuilt pipeline and present the result as what happened.
Send us one declined case and the explanation you produce for it.
Email a de-identified case, the explanation your system generates, and who receives it, to contact@precisionfederal.com. You get back a written read on which of the four questions it actually answers, whether it will hold up under a stability test, and what we would change. Two business days. No charge and no meeting.
contact@precisionfederal.comReproducing an explanation months later
A decision gets disputed long after it was made. Reproducing the explanation then requires five things pinned and stored: the model artifact and its version, the exact input feature values as they were at decision time, the preprocessing code version, the explainer library version, and the background or reference dataset the attribution was computed against. Miss any one and you can produce a number, but not the number.
The reference dataset is the one that gets lost. It is usually a sample taken at training time, held in memory, never versioned, and gone after the next deployment. Version it like a model artifact and store its identifier with every explanation.
The cheaper and more reliable alternative is to store the explanation itself with the decision. Storage is inexpensive; reconstruction is expensive and error-prone. Write the top contributing factors, the reason codes, the model version and the explainer version into the decision record at the time it is made, and the dispute conversation becomes a lookup instead of a reconstruction project.
Explaining a generative system
Attribution methods do not transfer usefully to language models, and the honest position is to say so rather than to ship a token-heatmap that looks like an explanation. Attention weights are not explanations — that has been shown repeatedly, and attention patterns can be altered substantially without changing the output. A model's own stated reasoning is a plausible narrative generated after the fact, not a trace of the computation, and treating it as one is the single most common error in this area right now.
What does work for generative systems is provenance. Where the output is grounded in source material, cite the specific passage, link it, and let a reader verify. That is a stronger form of accountability than any attribution over a decision model, because it does not ask anyone to trust the mechanism — it shows the evidence and lets the reader check. For anything that must be defended, design the system so that every assertion carries a locatable source, and treat an unsourced assertion as a defect rather than an output.
The mistakes we are called in to fix
- Attribution values shown to customers or front-line staff, who cannot act on a signed contribution
- Explanations computed on a different preprocessing path from the one that served the decision
- An undocumented, inconsistent baseline, so two teams explain the same case differently and both are right
- No stability measurement, and reason codes that reshuffle after every retrain
- Counterfactuals over non-actionable features, advising a customer to be younger
- A surrogate model presented without its fidelity number
- The reference dataset never versioned, making old explanations unreproducible
- A model chosen for a fraction of a point whose explainability cost was never priced
A five-week shape for an explainability layer
Typical build sequence
Step six is the test that matters and the one that gets skipped. Take twenty real cases with their explanations to the people who handle disputes, and ask them to defend each decision using only what the system provided. The failures are immediate and specific: a factor nobody can define, a reason that contradicts what the customer was told at application, an explanation that is technically correct and answers a question the customer did not ask. None of those show up in a fidelity metric.
Before you rely on it
- Each audience is named, with the question they are owed written down
- The accuracy gap against a simpler alternative has been measured, not assumed
- One documented baseline, used everywhere
- Explanations computed on the production artifact and preprocessing path
- Explanation, reason codes and all versions stored with the decision
- Stability measured under perturbation and tracked across retrains
- Reason-code distribution monitored over time, with an alert threshold
- Counterfactuals restricted to actionable features and verified to flip the outcome
- Any surrogate published with its fidelity number
- Real explanations tested on the people who must deliver them
Bottom line
Explainability work goes wrong less often through weak technique than through answering the wrong question. Decide who is owed what before choosing a method. Prefer recourse over attribution wherever a person is affected, because recourse is what they are asking for. Treat stability as a requirement equal to fidelity, since an explanation that moves without cause cannot be defended by anyone. Consider constraining the model instead of explaining it, and price that trade rather than assuming it. And compute the explanation where the decision is made, storing it alongside, because an explanation reconstructed later from a rebuilt pipeline is a plausible story about a model that may no longer exist.
Frequently asked questions
They are a strong answer to one of the four questions — why this case — and they are usually the wrong artifact to hand to the affected person, who wants to know what would change the outcome. They are also not causal: an attribution describes what moved the prediction given how the model was fitted, not what would happen if the input changed. Use them for case handlers and validators, use counterfactuals for customers, and document the baseline you compute against.
Because they are faithful to a model that changed. Boosted ensembles are locally jagged, so small differences in the fitted surface reorder attributions even where the decision is unchanged. This is why stability has to be an explicit requirement: perturb inputs and measure how often the top factors move, track the reason-code distribution across releases, and accept slightly lower fidelity in exchange for explanations that hold still. An unstable explanation cannot be defended in any conversation.
Often, and the decision should be a measurement rather than a preference. Fit a scorecard or a small additive model alongside the complex one and compare on the metric that matters. On tabular problems the gap is frequently small, and where it is small the interpretable model wins once you price the ongoing cost of an explainability layer, its failure modes and the disputes it will not settle. Where the gap is genuinely large, that is worth knowing too, and it is a defensible reason to take on the complexity.
Not with attribution methods in any way that survives scrutiny. Attention weights are not explanations, and a model's own account of its reasoning is a plausible narrative produced after the fact rather than a trace of the computation. What works instead is provenance: ground the output in source material and cite the specific passage so a reader can verify. That is stronger than any attribution because it shows evidence rather than asking for trust in the mechanism.
The model artifact and version, the exact input values at decision time, the preprocessing code version, the explainer library version, and the reference dataset the attribution was computed against — the last of which is the one teams lose, because it is usually an unversioned in-memory sample. The simpler and more reliable answer is to store the explanation itself with the decision record at the time it is made. Storage is cheap; reconstruction is expensive and rarely exact.