A prediction is a leftover
Start with the thing we will split, not with a formula. A house-price model looks at three features and returns a dollar amount. For the average house — every feature at its mean, which we have already centered to zero — the model says $280k. For this house it says $364k.
The difference, $84k, is the leftover. Every attribution method on this page is an argument about how to divide that leftover among Size, Location, and Age. Nothing else is being explained: not “importance in the dataset,” not “what would happen if we renovated,” just why this model’s number moved off the baseline for this house.
Size, Location, and Age are already in standard deviations from the training mean, so 0 means “ordinary.” The product terms are interactions: Size is worth more in a good Location, and Age hurts more in a good Location. Those products are why “drop one feature and look at the change” will not give a unique answer.
drag Size × Location down to 0. The leftover still splits, but every join-order later in the page will suddenly agree. Interaction is the reason Shapley has work to do.
A coalition is a team of features that get to be real
To talk about “what Size contributed,” we have to say what the model does when Size is missing. The convention used throughout this page — the same one behind interventional SHAP — is blunt and mechanical:
- Features in the team (the coalition) keep this house’s values.
- Everyone else is replaced by
0, the average house.
Write v(S) for the model’s price when coalition S is present. The empty team v(∅) is the baseline. The full team v({Size, Location, Age}) is the actual prediction. Every other team is a hypothetical house that is this house in some respects and average in the rest.
The 3D lattice on the right is the power set. Each vertex is one coalition. An edge means “add exactly one feature.” Click a vertex, or use the chips.
turn Size on alone, then add Location, then reset and add Location first. The second step is not the same size. That difference is a marginal contribution, and it depends on who was already in the room.
v(S ∪ {i}) − v(S) — what feature i added to one particular team S. Size joining the empty team adds its main effect only. Size joining a team that already has Location also picks up the interaction. Both numbers are true. Neither is the credit for Size.
Order matters, so we walk every order
Lloyd Shapley’s 1953 picture: line the features up in some order, start from the empty team, and let them join one at a time. Each feature is paid whatever it added at the moment it walked in. That walk is a path along the cube, empty corner to full corner, three edges long.
There are 3! = 6 such paths. They do not pay the same, because of the interaction terms. The table fills as you walk; the last row is the average — and that average is the thing we came for.
Pick an order and step. Each column is what that feature was paid on that walk. The average of a column is the Shapley value.
The Shapley value is the average arrival gift
For each feature, collect the six (in general, n!) arrival gifts and average them. That average is the feature’s Shapley value for this house, written φᵢ. Equivalently — and this is the form you will see in papers — average the marginal contribution over every team that does not yet contain the feature, with a weight that makes each arrival position equally likely:
The weight |S|! (n−|S|−1)! / n! is the chance that, in a random shuffle of all features, the members of S are exactly the people who arrived before i. It is not a machine-learning trick. It is “average over orders,” rewritten as a sum over teams.
Four fairness rules pick Shapley uniquely
Plenty of ways exist to split a leftover. Shapley’s theorem says that if you want four properties at once, you are forced to use this average. On this page they are not abstractions — each one is a knob or a readout you can check.
- Efficiency. The shares sum to
v(N) − v(∅). Already on the bars above. - Dummy. A feature that never changes the price, no matter which team it joins, gets
φ = 0. Toggle the dummy below. - Symmetry. Two features that always add the same amount when joining the same teams get the same share. Set the two main effects equal and zero the interactions; the bars match.
- Additivity. If you explain two models and add the explanations, you have explained the summed model. (This is why TreeSHAP can explain a forest by explaining each tree.)
The dummy is present in the instance — it has a real value — and the estimator still pays it nothing, because every marginal contribution is zero. “Present in the data” is not “responsible for the leftover.”
Exact Shapley enumerates 2n teams
Three features: 8 coalitions, 6 orders. We listed them. Ten features: 1,024 coalitions, 3.6 million orders. Twenty features: about a million coalitions and 2.4 × 1018 orders. A modern tabular model with 50 columns is not a thing you brute-force at prediction time.
That is the entire reason this tutorial is about estimation. The Shapley value is defined as an average over a set we cannot list. An estimator is a procedure that approximates that average by looking at only some of the teams, or by exploiting extra structure (a tree, a linear model) so that the average can be computed without visiting the set.
The orange curve is coalitions — what KernelSHAP samples from. The teal curve is orders — what permutation sampling samples from. Both are unusable raw; the art is sampling them in a way that the average still converges.
Permutation sampling: draw walks, average gifts
The most literal estimator: pick a random order of the features, walk it, record the three arrival gifts, repeat, and average. Each walk is an unbiased draw from the Shapley definition, so the running mean converges to the true φ. The cube on the right shows the walks as they happen. The chart below shows the running mean against the exact values (ghost bars), which we still have because n = 3.
This is the Monte Carlo method of Štrumbelj & Kononenko. It is unbiased, embarrassingly parallel, and slow to tighten: the variance of a single walk is the variance of those interaction-driven arrival gifts. Antithetic pairs — always following a walk with its reverse — cancel some of that variance, because a feature that arrived first last time arrives last this time.
draw 1 a few times with interaction high — the bars jump. Then zero the interaction slider in section 01 and draw again. With no interactions every walk pays the same, so one sample is already exact. Estimation is hard exactly when features cooperate.
KernelSHAP: Shapley as a weighted regression
Lundberg & Lee (2017) noticed that the Shapley values are also the coefficients of a very particular linear explanation. Pretend the price of a coalition is a sum of per-feature credits:
That formula is the SHAP waterfall in algebra: start at the baseline, add a credit for each feature that is present. It cannot match v(S) on every team unless the model is already additive. So we fit it. The loss is weighted least squares, and the weights — the Shapley kernel — are chosen so that the best-fit φ is exactly the Shapley value when every coalition is included:
The kernel is U-shaped: almost-empty and almost-full teams get huge weight; middling teams get little. Empty and full are pinned by the efficiency constraint rather than given infinite weight in the sum. When n is large we sample coalitions with probability proportional to π(S) and fit on the sample. That sample-and-fit step is KernelSHAP.
With all 6 interior coalitions the fit recovers exact Shapley (up to solver noise). Drop the sample size and refit: the bars wander. Unlike permutation sampling, a small KernelSHAP sample is generally biased — it is a regression on an incomplete design — but the kernel’s emphasis on the extremes often gives lower error for the same evaluation budget on larger n.
Local credit, then a global picture
Everything so far is local: one house, one leftover, three shares. That is what you want when a person asks “why did this appraisal come out high?” A global summary is a statistic of many local explanations. The usual one is mean absolute Shapley value — how large a feature’s credit tends to be, ignoring sign. A beeswarm keeps the signs and the house-to-house scatter, so you can see that Age usually subtracts, Size usually adds, and Location’s credit depends on Size because of the interaction.
Color is the feature’s own value on that house. Size’s cloud sits to the right and goes from dark (small house, small or negative credit) to light (large house, large credit). That slope is the local explanation agreeing with the model’s main effect, house after house.
predict 2n times. A tree already partitions the input space. TreeSHAP walks the tree with a dynamic program and accumulates each feature’s weighted marginals in polynomial time in the number of leaves and depth. It computes the same Shapley values this page defines, not a different notion of importance — it just uses the tree as an index over coalitions. Path-dependent TreeSHAP is a different, faster variant that does not match interventional Shapley; the distinction is one of the catches below.
Fair for the model is not the same as true of the world
Shapley values are a theorem about cooperative games. Mapping a model onto a game — features as players, v(S) as payout — smuggles in choices that the theorem cannot audit.
What “missing” means
This page replaces absent features with the mean. That is an interventional (or baseline) value function: it breaks whatever dependence Size and Location had in the training data. You will invent houses that never occur — a mansion in a terrible location, a shack on the park. Observational SHAP instead asks the data “among houses with this Size, what Location is typical?” and conditions. The two procedures answer different questions and can rank features differently when inputs are correlated. Neither is causal. Causal attribution needs a causal model, not a cooperative game on input coordinates.
Correlated features split credit, even if one is a proxy
If Location is a noisy copy of Size, Shapley will still pay both, because both change v(S) when they arrive. That is correct for the game and misleading if you planned to drop “redundant” sensors. Dummy only fires when the coefficient is literally zero, not when the feature is a correlate.
Estimation error is real
On a 50-feature model, published SHAP values are estimates. Permutation sampling is unbiased but noisy. KernelSHAP with a small coalition sample is typically biased. Always treat close bars as a tie. The efficiency identity (Σ φᵢ = leftover) still holds for KernelSHAP by construction of the constraint; it is not evidence that the split among features has converged.
The leftover is the model’s leftover
If the model is wrong, you will fairly attribute a wrong number. Shapley will not tell you that Age is a stand-in for an unmeasured renovation flag. It tells you how this function used these inputs.
If you remember four sentences
- A Shapley value is the average amount a feature adds to the model’s output when it joins a team, averaging over every team (equivalently: every join-order).
- The shares add up to prediction minus baseline. That leftover is the only pie being cut.
- We estimate because 2n is impossible. Permutation sampling averages random walks; KernelSHAP fits a weighted linear explanation on sampled teams; TreeSHAP computes the same object by walking a tree.
- The mapping from “feature absent” to a number is a choice (mean baseline vs. conditional), and a fair split of a model is not a causal claim about the world.
Further reading: Shapley, “A value for n-person games” (1953); Štrumbelj & Kononenko, “Explaining prediction models and individual predictions with feature contributions” (2014); Lundberg & Lee, “A unified approach to interpreting model predictions” (NeurIPS 2017); Janzing, Minorics & Blöbaum on interventional vs. observational attributions; Molnar, Interpretable Machine Learning, the SHAP chapter. The Key Driver Analysis lesson in this same Learn folder uses Shapley to split R² across survey drivers — a sibling of this construction, with a different payout v(S).