| Time | Block |
|---|---|
| 0:00 – 0:25 | 62.1 Decision trees: splitting the data into regimes |
| 0:25 – 0:45 | 62.2 Overfitting and the depth trade-off |
| 0:45 – 0:53 | Break |
| 0:53 – 1:15 | 62.3 Ensembles: bagging and random forests |
| 1:15 – 1:40 | 62.4 Gradient boosting from scratch, then XGBoost |
| 1:40 – 1:48 | Break |
| 1:48 – 2:08 | 62.5 Trees can't extrapolate: what that means for macro data |
| 2:08 – 2:30 | 62.6 Mini-project: trees in the horse race |
62.1 Decision Trees: Splitting the Data into Regimes (25 min)
A regression tree predicts by asking a sequence of yes/no questions about the features, and predicting the average target of the training rows that end up in the same final group (leaf).
yoy_lag0 ≤ 3.1 ?
/ \
yes no
mom_ma3 ≤ 0.25 ? yoy_lag0 ≤ 5.4 ?
/ \ / \
leaf: 1.6 leaf: 2.4 leaf: 4.2 leaf: 6.3
Each split is chosen to make the leaves as homogeneous as possible (minimising squared error). The result is a piecewise-constant function: the feature space is divided into rectangles, each with one prediction.
Economists can read this as a model of regimes: "when inflation is low and momentum weak, expect ~1.6% next year; when inflation is high and accelerating, expect ~6%". Trees capture non-linearities and interactions automatically, without you specifying them.
from sklearn.tree import DecisionTreeRegressor, export_text
from forecasting.data import load_cpi
from forecasting.features import univariate_cpi_features
df = load_cpi()
X = univariate_cpi_features(df)
h = 12
data = X.join(df["cpi_yoy"].shift(-h).rename("target")).dropna()
train = data.loc[:"2019-12"]
tree = DecisionTreeRegressor(max_depth=2, random_state=0)
tree.fit(train.drop(columns="target"), train["target"])
print(export_text(tree, feature_names=list(X.columns)))
Read the printed rules. Which features did the tree choose? Do the thresholds make economic sense?
(Training on rows up to 2019-12 with h = 12 uses targets up to 2020-12. That's fine for illustration; for evaluation we always use the harness.)
62.2 Overfitting and the Depth Trade-Off (20 min)
A tree can keep splitting until every leaf holds one training row: a perfect in-sample fit, and terrible forecasts.
import numpy as np
from sklearn.metrics import mean_squared_error
test = data.loc["2021-01":]
for depth in [1, 2, 3, 5, 8, None]:
m = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(
train.drop(columns="target"), train["target"])
rmse_in = np.sqrt(mean_squared_error(train["target"], m.predict(train.drop(columns="target"))))
rmse_out = np.sqrt(mean_squared_error(test["target"], m.predict(test.drop(columns="target"))))
print(f"depth={str(depth):>4} in-sample RMSE={rmse_in:5.2f} out-of-sample RMSE={rmse_out:5.2f}")
You'll see the classic pattern: in-sample error falls towards zero as depth grows; out-of-sample error first falls, then rises. That's bias–variance (61.1), visible in two columns.
The controls that limit a tree's flexibility:
| Parameter | Effect |
|---|---|
max_depth | maximum number of successive questions |
min_samples_leaf | a leaf must contain at least this many months (e.g. 12) |
max_features | consider only a random subset of features at each split (used by forests) |
The 2021–2023 test period contains the post-pandemic inflation surge, which lies outside most of the training data's range. Keep this in mind for 62.5.
62.3 Ensembles: Bagging and Random Forests (23 min)
Single trees are unstable: a small change in the data can change the first split and hence everything below it (high variance). The fix is to average many trees.
| Method | How | Reduces |
|---|---|---|
| Bagging | train each tree on a bootstrap resample of the rows; average predictions | variance |
| Random forest (Breiman, 2001) | bagging + each split considers only a random subset of features | variance, further, by de-correlating trees |
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(n_estimators=500, min_samples_leaf=6, max_features=0.5,
random_state=0, n_jobs=-1)
rf.fit(train.drop(columns="target"), train["target"])
print("RF out-of-sample RMSE:",
np.sqrt(mean_squared_error(test["target"], rf.predict(test.drop(columns="target")))))
Random forests are robust with little tuning, which is why they've performed well in some inflation-forecasting studies (e.g. Medeiros, Vasconcelos, Veiga and Zilberman, 2021, Journal of Business & Economic Statistics, for US inflation in a data-rich setting). Whether they help for Canadian inflation with our features is an empirical question, and your harness answers it.
Bootstrap and time series: resampling rows ignores time ordering, but here it's used only to build diverse trees within the training window. Evaluation stays strictly rolling-origin, so no future information leaks.
62.4 Gradient Boosting from Scratch, then XGBoost (25 min)
Random forests build trees independently and average them. Boosting builds trees sequentially: each new tree is fitted to the errors (residuals) of the ensemble so far, and added with a small weight (the learning rate). The idea is due to Friedman (2001).
Boosting in ten lines
from sklearn.tree import DecisionTreeRegressor
Xtr, ytr = train.drop(columns="target"), train["target"]
Xte, yte = test.drop(columns="target"), test["target"]
learning_rate, n_rounds = 0.1, 200
pred_tr = np.full(len(ytr), ytr.mean()) # start from the mean
pred_te = np.full(len(yte), ytr.mean())
for m in range(n_rounds):
residual = ytr - pred_tr # what the ensemble still gets wrong
stump = DecisionTreeRegressor(max_depth=2).fit(Xtr, residual)
pred_tr += learning_rate * stump.predict(Xtr)
pred_te += learning_rate * stump.predict(Xte)
if m in (0, 9, 49, 199):
print(f"round {m+1:>3}: in-sample RMSE {np.sqrt(np.mean((ytr - pred_tr)**2)):.3f} "
f"out-of-sample RMSE {np.sqrt(np.mean((yte - pred_te)**2)):.3f}")
That's the whole idea: start simple, then repeatedly correct the remaining errors with small trees, taking small steps. The learning rate makes each correction cautious, so many small trees generalise better than a few large ones.
XGBoost
XGBoost (Chen & Guestrin, 2016) is a fast, regularised implementation of gradient boosting, and one of the most widely used ML tools for tabular data.
from xgboost import XGBRegressor
xgb = XGBRegressor(
n_estimators=500, # maximum number of trees
learning_rate=0.03, # shrinkage per tree
max_depth=3, # small trees
min_child_weight=5, # minimum (weighted) rows per leaf
subsample=0.8, # row subsampling per tree (adds randomness, reduces variance)
colsample_bytree=0.8, # feature subsampling per tree
reg_lambda=1.0, # L2 penalty on leaf values
random_state=0,
)
xgb.fit(Xtr, ytr)
print("XGB out-of-sample RMSE:", np.sqrt(np.mean((yte - xgb.predict(Xte)) ** 2)))
Early stopping: let validation data choose the number of trees
Too many trees overfit. Early stopping monitors error on a validation set and stops when it stops improving. For time series, the validation set must be the most recent part of the training window, never a random sample:
split = int(len(Xtr) * 0.8) # last 20% of the training window
xgb_es = XGBRegressor(n_estimators=2000, learning_rate=0.03, max_depth=3,
subsample=0.8, colsample_bytree=0.8, random_state=0,
early_stopping_rounds=50)
xgb_es.fit(Xtr.iloc[:split], ytr.iloc[:split],
eval_set=[(Xtr.iloc[split:], ytr.iloc[split:])], verbose=False)
print("best number of trees:", xgb_es.best_iteration + 1)
Key XGBoost parameters
| Parameter | Typical range | Lower → | Higher → |
|---|---|---|---|
learning_rate | 0.01–0.1 | more trees needed, often better generalisation | faster, riskier |
max_depth | 2–6 | simpler interactions | more complex interactions, more variance |
n_estimators | 100–2000 (with early stopping) | underfit | overfit (without early stopping) |
min_child_weight | 1–20 | more flexible leaves | smoother predictions |
subsample, colsample_bytree | 0.5–1.0 | more randomness, less variance | less randomness |
reg_lambda, reg_alpha | 0–10 | weaker regularisation | stronger regularisation |
Systematic tuning of these, without leakage, is Day 64.
62.5 Trees Can't Extrapolate: What That Means for Macro Data (20 min)
Because tree predictions are averages of training targets, a tree model can never predict a value outside the range it has seen. For trending or level data this is a serious limitation.
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
lvl = df[["cpi"]].copy()
lvl["t"] = np.arange(len(lvl)) # a time trend as the only feature
tr, te = lvl.loc[:"2019-12"], lvl.loc["2020-01":]
lin = LinearRegression().fit(tr[["t"]], tr["cpi"])
xgb_lvl = XGBRegressor(n_estimators=300, max_depth=3, learning_rate=0.05).fit(tr[["t"]], tr["cpi"])
plt.figure(figsize=(10, 4))
plt.plot(lvl.index, lvl["cpi"], label="CPI level (actual)")
plt.plot(te.index, lin.predict(te[["t"]]), label="linear trend")
plt.plot(te.index, xgb_lvl.predict(te[["t"]]), label="XGBoost")
plt.axvline(pd.Timestamp("2020-01-01"), linestyle="--", color="grey")
plt.legend(); plt.title("Trees can't extrapolate a trend"); plt.tight_layout()
plt.savefig("data/forecasts/extrapolation.png", dpi=120)
The XGBoost forecast of the CPI level goes flat at the last training value, while the price level keeps rising.
Consequences for macro forecasting
| Do | Don't |
|---|---|
| model stationary transformations: growth rates, yoy inflation, changes, spreads | feed levels of trending series (CPI index, GDP in dollars, population) |
| express features relative to history (e.g. deviation from a moving average) | rely on trees to project trends |
| remember that unprecedented values (2021–22 inflation) are partly outside the training range | expect trees to forecast beyond historical extremes |
That's why our target is yoy inflation, not the CPI index, and why the Day 68 feature set uses rates, changes and spreads from the marts. Each of these is a transformation that dbt already computes (Days 19–20, 39).
62.6 Mini-Project: Trees in the Horse Race (23 min)
Add tree models to Day 61's comparison, using the same harness, horizons and test period. Add to forecasting/models.py:
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor
from xgboost import XGBRegressor
def tree_factory():
return DecisionTreeRegressor(max_depth=3, min_samples_leaf=12, random_state=0)
def rf_factory():
return RandomForestRegressor(n_estimators=300, min_samples_leaf=6, max_features=0.5,
random_state=0, n_jobs=-1)
def xgb_factory():
return XGBRegressor(n_estimators=300, learning_rate=0.03, max_depth=3, min_child_weight=5,
subsample=0.8, colsample_bytree=0.8, reg_lambda=1.0,
random_state=0, n_jobs=2)
candidates.update({"tree": tree_factory, "random_forest": rf_factory, "xgboost": xgb_factory})
results = pd.concat([rolling_origin(X, y, f, H, "2012-01-01", name, refit_every=3)
for name, f in candidates.items()], ignore_index=True)
results.to_parquet("data/forecasts/day62_results.parquet")
print(score(results).round(3).to_string(index=False))
refit_every=3 retrains every three months instead of every month, to keep the runtime reasonable. Note it in your results: refit frequency is part of the method.
Questions to answer in notebooks/day62.ipynb
- At which horizons, if any, do tree models beat the naive benchmark? Ridge?
- Plot forecasts against actuals for h = 12, 2019–2025. How did each model handle the 2021–22 surge, and does 62.5 explain what you see?
- Plot
xgb.feature_importances_from a model fitted on the full training window. Which features matter? (Importances are a rough guide; Days 64–67 use SHAP for a principled explanation.)
Day 62 checklist
- I can explain how regression trees split, predict, and capture regimes and interactions
- I can demonstrate overfitting with tree depth, and name the parameters that control it
- I understand bagging, random forests and boosting (and implemented boosting from scratch)
- I can fit XGBoost with sensible parameters and time-ordered early stopping
- I know trees can't extrapolate, and I model stationary transformations
- Trees, forests and XGBoost are in the horse race under identical, leak-free evaluation
git add forecasting notebooks && git commit -m "Day 62: trees, forests, boosting, XGBoost in the backtest" && git push