Day 65: SHAP Explainability: Fundamentals · Applied Decision Intelligence & Data Architecture

Written lesson

Time	Block
0:00 – 0:20	65.1 Why explain forecasts, and why feature importance isn't enough
0:20 – 0:45	65.2 Shapley values: a fair way to share a prediction
0:45 – 0:53	Break
0:53 – 1:20	65.3 TreeSHAP for XGBoost: global explanations
1:20 – 1:40	65.4 Explaining a single forecast: waterfall plots
1:40 – 1:48	Break
1:48 – 2:08	65.5 Out-of-sample importance: permutation importance
2:08 – 2:30	65.6 What SHAP does not tell you · mini-project

65.1 Why Explain Forecasts? (20 min)

A Bank of Canada economist presented with "the model forecasts 3.1% inflation in 12 months" will immediately ask why. Explanations serve several purposes:

Purpose	Question
Trust	does the model rely on sensible signals, or on artefacts?
Communication	can the forecast be narrated ("persistence is fading, momentum is rising")?
Debugging	is a feature leaking the future, or broken by a data error?
Monitoring	have the model's drivers changed (Day 66)?

The limits of built-in "feature importance"

XGBoost's feature_importances_ (Day 62) has problems:

it measures how much a feature was used in training (e.g. total gain), not how much it matters for forecasts;
it is global only: it can't explain this forecast;
it's biased towards features with many possible split points, and it splits credit arbitrarily among correlated features (our yoy lags are highly correlated);
it has no sign: does a feature push the forecast up or down?

SHAP values fix most of these issues, with a solid theoretical foundation.

The idea from cooperative game theory

Lloyd Shapley (Nobel Prize in Economics, 2012) asked: several players cooperate to produce a payoff; how should it be divided fairly? His answer: give each player their average marginal contribution across all orders in which the players could have joined.

SHAP (Lundberg & Lee, 2017) applies this to predictions:

the players are the features;
the payoff is the difference between this prediction and the average prediction (the base value).

A tiny example by hand

Suppose a model forecasts inflation from two features, current inflation A and momentum B. Average prediction (base value): 2.0%. For one month:

Features known	Prediction
none (base value)	2.0
A only	3.0
B only	2.4
A and B	3.6

The two possible joining orders:

A first, then B: A adds 3.0 − 2.0 = 1.0; B adds 3.6 − 3.0 = 0.6.
B first, then A: B adds 2.4 − 2.0 = 0.4; A adds 3.6 − 2.4 = 1.2.

Average over orders: φ_A = (1.0 + 1.2)/2 = 1.1 and φ_B = (0.6 + 0.4)/2 = 0.5.

Check: base value + φ_A + φ_B = 2.0 + 1.1 + 0.5 = 3.6, exactly the prediction.

Properties that make SHAP attractive

Property	Meaning
Additivity (efficiency)	base value + sum of SHAP values = the prediction, exactly
Symmetry	two features contributing identically get equal credit
Dummy	a feature the model never uses gets zero
Sign	positive values push the forecast up; negative values push it down

Additivity is what makes SHAP perfect for economists: it's a decomposition, just like the CPI contributions on Day 19 (components summing to headline) or growth accounting.

With many features there are too many orderings to enumerate. TreeSHAP (Lundberg et al., 2020) computes exact Shapley values for tree ensembles efficiently by exploiting the tree structure. That's why SHAP pairs so well with XGBoost.

65.3 TreeSHAP for XGBoost: Global Explanations (28 min)

Create forecasting/explain.py:

"""SHAP helpers for tree models (XGBoost)."""
import numpy as np
import pandas as pd
import xgboost as xgb


def xgb_contributions(model, X: pd.DataFrame) -> pd.DataFrame:
    """Exact TreeSHAP contributions computed by XGBoost itself.

    Returns one column per feature plus 'base_value'; each row sums to the prediction.
    """
    booster = model.get_booster()
    contrib = booster.predict(xgb.DMatrix(X), pred_contribs=True)
    out = pd.DataFrame(contrib, index=X.index, columns=list(X.columns) + ["base_value"])
    return out


def check_additivity(model, X: pd.DataFrame, contrib: pd.DataFrame, tol: float = 1e-3):
    recon = contrib.sum(axis=1).to_numpy()
    pred = model.predict(X)
    gap = np.abs(recon - pred).max()
    assert gap < tol, f"SHAP additivity violated: max gap {gap}"
    return gap

XGBoost can compute SHAP contributions natively (pred_contribs=True). It's robust across versions and needs no extra library. The shap package adds rich plotting on top.

Fit a model to explain

import matplotlib.pyplot as plt
import shap

from forecasting.data import load_cpi
from forecasting.features import univariate_cpi_features
from forecasting.models import xgb_factory

df = load_cpi()
X = univariate_cpi_features(df)
h = 12
target = df["cpi_yoy"].shift(-h)
train = X.notna().all(axis=1) & target.notna()

model = xgb_factory().fit(X[train], target[train])
X_explain = X[X.notna().all(axis=1)].loc["2012-01":]
contrib = xgb_contributions(model, X_explain)
print("additivity gap:", check_additivity(model, X_explain, contrib))

(This model is trained on all available rows, which is fine for explaining the model. For forecasts we always use the rolling harness. Day 66 explains each real-time forecast.)

Global picture: which features drive forecasts, and in which direction?

explanation = shap.Explanation(values=contrib.drop(columns="base_value").to_numpy(),
                               base_values=contrib["base_value"].to_numpy(),
                               data=X_explain.to_numpy(),
                               feature_names=list(X_explain.columns))

shap.plots.bar(explanation, max_display=12, show=False)
plt.tight_layout(); plt.savefig("data/forecasts/shap_bar_h12.png", dpi=120); plt.close()

shap.plots.beeswarm(explanation, max_display=12, show=False)
plt.tight_layout(); plt.savefig("data/forecasts/shap_beeswarm_h12.png", dpi=120); plt.close()

How to read them:

Bar plot: mean |SHAP| per feature, i.e. average impact on forecasts in percentage points of inflation. Unlike gain importance, this is in the units of the forecast.
Beeswarm: each dot is one month. Horizontal position = SHAP value (push up or down); colour = the feature's value (red high, blue low). A feature whose red dots sit on the right pushes forecasts up when it is high.

We build the shap.Explanation from XGBoost's own contributions. If shap.TreeExplainer(model) works with your versions, it gives the same numbers. Using pred_contribs avoids occasional version mismatches between shap and xgboost.

Try it 65.3

From the beeswarm for h = 12: does high current inflation (yoy_lag0) push the 12-month-ahead forecast up as much as you'd expect from a random walk? What does that suggest about how the model treats persistence at long horizons?

Discussion

Typically yoy_lag0 and the moving averages dominate and push forecasts up when inflation is high, but less than one-for-one: high inflation today raises the 12-month forecast by less than its own distance from the mean. That's mean reversion, which an inflation-targeting regime makes plausible. The model has learned that deviations from ~2% tend to shrink over a year. It's an economically sensible pattern, and SHAP made it visible.

65.4 Explaining a Single Forecast: Waterfall Plots (20 min)

Global plots describe the model. Local explanations describe one forecast, which is what an economist asks about in a briefing.

def waterfall(origin: str, path: str):
    i = list(X_explain.index).index(pd.Timestamp(origin))
    shap.plots.waterfall(explanation[i], max_display=10, show=False)
    plt.title(f"Why the model forecast {model.predict(X_explain.iloc[[i]])[0]:.2f}% "
              f"for {pd.Timestamp(origin) + pd.DateOffset(months=h):%b %Y}")
    plt.tight_layout(); plt.savefig(path, dpi=120); plt.close()

waterfall("2021-06-01", "data/forecasts/waterfall_2021_06.png")   # as inflation began to surge
waterfall(str(X_explain.index.max().date()), "data/forecasts/waterfall_latest.png")

A waterfall starts at the base value (the average forecast), adds each feature's contribution as a bar (red up, blue down), and ends at this forecast. It's a growth-accounting chart for a single prediction.

Write a two-sentence narrative for each waterfall, the way you'd brief a policymaker:

"The model expects inflation to ease to 2.4% by June 2027. Current inflation (+0.6pp) still pushes the forecast above average, but slowing 3-month momentum (−0.3pp) points to further easing."

65.5 Out-of-Sample Importance: Permutation Importance (20 min)

SHAP explains how the model uses features. Permutation importance asks a different, complementary question: how much worse do forecasts become if a feature's information is destroyed? It's measured out of sample, on data the model didn't train on.

The method: shuffle one feature's column in the test data (breaking its link with the target), re-predict, and measure the increase in error.

from sklearn.inspection import permutation_importance

cut = "2015-12"
tr = train & (X.index <= cut)
te = X.notna().all(axis=1) & target.notna() & (X.index > cut)
m = xgb_factory().fit(X[tr], target[tr])

perm = permutation_importance(m, X[te], target[te], scoring="neg_root_mean_squared_error",
                              n_repeats=30, random_state=0)
imp = (pd.Series(perm.importances_mean, index=X.columns)
         .sort_values(ascending=False))
print(imp.head(10).round(3))           # increase in RMSE (pp) when the feature is shuffled

Method	Answers	Data used
SHAP	how does the model use each feature, per forecast?	any (usually training or recent data)
Permutation importance	how much does each feature help accuracy?	held-out data

A feature with large SHAP values but near-zero permutation importance is used heavily but doesn't improve out-of-sample accuracy. That's often the case for one of several correlated features: shuffling one lag hurts little because its neighbours carry the same information.

65.6 What SHAP Does Not Tell You · Mini-Project (23 min)

SHAP is	SHAP is not
an exact decomposition of the model's prediction	a statement about the economy
descriptive of learned associations	causal: "momentum causes inflation" doesn't follow
sensitive to how correlated features share credit	a unique ranking when features are near-duplicates
relative to a base value (the training average)	an absolute measure of a feature's "effect"

If an economist asks "what would happen to inflation if the Bank raised rates?", SHAP cannot answer. That's a causal question, the subject of Module 8 (DoubleML).

Mini-project: an explanation memo

In notebooks/day65.ipynb, produce a one-page explanation memo for the latest h = 12 forecast:

The forecast and its 80% interval (Day 63 method).
The waterfall plot, and your two-sentence narrative.
The global bar plot, with one sentence on what drives the model in general.
Permutation importance, and one sentence on which features actually help accuracy.
One explicit caveat about what the explanation does not show.

Day 65 checklist

I can explain why gain-based feature importance is insufficient
I can compute Shapley values by hand for a two-feature example, and state their properties
I compute exact TreeSHAP contributions with XGBoost and verify additivity
I produce and read bar, beeswarm and waterfall plots, and narrate a forecast
I contrast SHAP with out-of-sample permutation importance
I can state clearly what SHAP explanations do not mean

git add forecasting notebooks && git commit -m "Day 65: Shapley values, TreeSHAP, waterfalls, permutation importance" && git push

SHAP Explainability: Fundamentals

What you'll learn today

Today's plan