| Time | Block |
|---|---|
| 0:00 – 0:23 | 61.1 Prediction vs inference: ŷ problems and β̂ problems |
| 0:23 – 0:40 | 61.2 Setting up the modelling environment and data |
| 0:40 – 0:48 | Break |
| 0:48 – 1:10 | 61.3 Framing a forecast: horizon, information set, target |
| 1:10 – 1:35 | 61.4 Evaluation without leakage: rolling-origin backtests |
| 1:35 – 1:43 | Break |
| 1:43 – 2:05 | 61.5 Benchmarks and metrics |
| 2:05 – 2:30 | 61.6 Mini-project: the evaluation harness + first linear models |
61.1 Prediction vs Inference (23 min)
Econometrics training focuses on β̂: estimating a parameter (an elasticity, a treatment effect) and its uncertainty. Machine learning focuses on ŷ: predicting an outcome as accurately as possible out of sample. Mullainathan and Spiess (2017, Journal of Economic Perspectives) frame this as the difference between "β̂ problems" and "ŷ problems".
| Inference (β̂) | Prediction (ŷ) | |
|---|---|---|
| question | what is the effect of X on Y? | what will Y be? |
| success | unbiased, efficient β̂; valid standard errors | small out-of-sample error |
| model choice | theory, identification | whatever predicts best, honestly evaluated |
| coefficients | the object of interest | often uninterpretable, and that's acceptable |
| main danger | omitted-variable bias, endogeneity | overfitting, leakage |
| this course | Module 8 (DoubleML) | Module 7 |
Forecasting CPI inflation is a ŷ problem: the Bank of Canada, markets and businesses want to know where inflation is going. Explaining why matters too, and Days 64–67 add explainability (SHAP). But the yardstick is forecast accuracy.
Bias, variance and overfitting
A model's expected out-of-sample squared error decomposes into:
error = bias² + variance + irreducible noise
- Bias: the model is too simple to capture the pattern (e.g. a constant forecast).
- Variance: the model is so flexible that it fits noise in the training sample, and its forecasts change wildly with small changes in data.
- Noise: the part of inflation no model can predict (shocks).
Flexible ML models reduce bias but risk variance. Every technique this week (regularisation, tree depth limits, learning rates, early stopping, cross-validation) is a way to trade bias against variance using out-of-sample evidence.
Economist's intuition: adding regressors to an OLS model always increases in-sample R². It doesn't always improve forecasts, for the same reason. In-sample fit is not the target.
61.2 Setting Up the Modelling Environment and Data (18 min)
A separate environment, as before (Day 42)
Modelling libraries are large and evolve quickly; keep them isolated from the pipeline:
cd /workspaces/macro-data-course
python -m venv .venv-ml
source .venv-ml/bin/activate
pip install -U pip
pip install pandas pyarrow numpy scikit-learn xgboost statsmodels matplotlib ipykernel tabulate
mkdir -p forecasting && pip freeze > forecasting/requirements-ml.txt
python -c "import xgboost, sklearn, statsmodels; print(xgboost.__version__, sklearn.__version__, statsmodels.__version__)"
Add .venv-ml/ and data/forecasts/ to .gitignore. In VS Code, notebooks in notebooks/ can select .venv-ml as their kernel (Day 4). Explore in notebooks, keep reusable code in forecasting/.
A mart for modelling
Models should read marts, never raw tables: they get tested, documented data ("dashboards present; dbt computes" applies to models too). We need a long CPI history, which macro_monthly (from 2006) doesn't have. Add a mart in macro_dbt/models/marts/ml/ml_cpi_monthly.sql:
-- Grain: one row per month, Canada. CPI inputs for forecasting (Module 7).
-- CPI (18-10-0004-01) is not seasonally adjusted and is rarely revised.
select
month,
max(cpi) filter (where component = 'All-items') as cpi,
max(yoy) filter (where component = 'All-items') as cpi_yoy,
max(mom) filter (where component = 'All-items') as cpi_mom,
max(yoy) filter (where component = 'All-items excluding food and energy') as core_yoy
from {{ ref('int_cpi__growth') }}
where geo_code = 'CA'
group by month
with tests (unique on month, sequential_values monthly, Day 36) in _ml__models.yml. Build it in the pipeline environment while no other run is writing:
source .venv/bin/activate
cd macro_dbt && dbt build --target prod --select ml_cpi_monthly && cd ..
ls -l data/marts/ml_cpi_monthly.parquet # exported by the Day 40 post-hook
source .venv-ml/bin/activate
(Also add ml_cpi_monthly to an exposure, e.g. forecasting_models, so the lineage graph shows that models depend on it, as on Day 40.)
Loading data: forecasting/data.py
"""Load modelling inputs from the exported dbt marts."""
from pathlib import Path
import pandas as pd
REPO = Path(__file__).resolve().parents[1]
MARTS = REPO / "data" / "marts"
def load_cpi(start: str = "1992-01-01") -> pd.DataFrame:
"""Monthly CPI inputs from ml_cpi_monthly, on a regular month-start index."""
df = pd.read_parquet(MARTS / "ml_cpi_monthly.parquet")
df["month"] = pd.to_datetime(df["month"])
df = df.set_index("month").sort_index().asfreq("MS")
df = df.loc[start:]
if df["cpi_yoy"].isna().any():
missing = df.index[df["cpi_yoy"].isna()]
raise ValueError(f"cpi_yoy has gaps, e.g. {list(missing[:3])}")
return df
Why start in 1992? The Bank of Canada and the federal government adopted inflation targets in 1991, and inflation behaved very differently before (the 1970s–80s). Mixing regimes can teach a model patterns that no longer apply. This is an economic judgement, and you'll test its effect in an exercise.
asfreq("MS") puts the data on a regular monthly index, so a missing month would appear as NaN and the check fails loudly, instead of a lag silently pointing to the wrong month (Day 5's lesson).
61.3 Framing a Forecast (23 min)
Every forecast needs three precise definitions.
| Element | Definition | Our CPI example |
|---|---|---|
| Forecast origin t | the last period whose data you know when forecasting | the latest published CPI month |
| Horizon h | how far ahead | 1, 3, 6, 12 months |
| Target | what exactly you forecast | year-over-year CPI inflation in month t + h |
| Information set | everything knowable at t | CPI up to t (and, from Day 68, other series published by then) |
Direct vs recursive multi-step forecasts
| Strategy | How | Pros | Cons |
|---|---|---|---|
| Direct | a separate model per horizon: predict yₜ₊ₕ from features at t | no error accumulation; any features allowed | one model per horizon |
| Recursive | one 1-step model, iterated: feed predictions back as lags | one model | errors compound; needs forecasts of every feature |
We use direct forecasting by default (Day 63 compares both).
The supervised dataset
For horizon h, each row pairs features known at month t with the target at t + h:
feature date t features (known at t) target (y at t+h)
2019-01 y_t, y_{t-1}, …, mom_t, … y at 2019-01 + h
2019-02 … y at 2019-02 + h
In pandas, the target is the series shifted backwards by h:
target = y.shift(-h) # row t holds y_{t+h}
This is the same shift you used on Day 5. Note the minus sign: we're pulling the future value back to the origin row.
Publication lags. CPI for month t is published during month t+1. When we say "origin t", we mean the moment just after CPI for t is published. For univariate CPI forecasting that's all we need. On Day 68, when we add other series, we'll have to check that each one is actually published by then (Day 17's ASOF logic).
61.4 Evaluation Without Leakage: Rolling-Origin Backtests (25 min)
Why ordinary cross-validation is wrong here
Standard ML cross-validation shuffles rows into folds. For time series that means training on 2024 to predict 2015: using the future to predict the past. Errors look small, and they're fake.
The rolling-origin (pseudo out-of-sample) backtest
Simulate what a forecaster would have done in real time:
origin 2015-01: train on all rows whose target date ≤ 2015-01 → forecast y(2015-01 + h)
origin 2015-02: retrain (expanding window) → forecast y(2015-02 + h)
…
origin 2025-xx: …
Collect all forecasts, compare them with what actually happened, and compute errors. This is the standard way macroeconomists evaluate forecasts (pseudo out-of-sample because we use final data, not real-time vintages; more on Day 68).
The subtle leakage trap: training rows near the origin
At origin t, a training row with feature date s has target date s + h. That target is only known if s + h ≤ t. Rows with s between t − h + 1 and t have targets in the future relative to the origin, so they must be excluded from training, even though their features are known.
t = origin, h = 3
feature dates: … t-4 t-3 t-2 t-1 t
target dates: … t-1 t t+1 t+2 t+3
usable? … ✓ ✓ ✗ ✗ ✗(predict this one)
Forgetting this is the most common forecasting bug, and it makes multi-horizon results look far too good.
Other leakage sources to watch
| Leak | Example | Fix |
|---|---|---|
| centred rolling windows | y.rolling(3, center=True) uses t+1 | only trailing windows |
| full-sample scaling | standardising with the mean of all data | fit scalers on training rows only (pipelines, 61.6) |
| target in features | cpi_yoy at t+h sneaks in via a join | build features only from data dated ≤ t |
| revised data | using today's GDP vintage as if known in 2015 | real-time vintages (Days 27, 37; used on Day 68) |
| tuning on the test period | choosing parameters by test error | tune on validation folds before the test period (Days 64–67) |
61.5 Benchmarks and Metrics (23 min)
Always beat a simple benchmark first
A sophisticated model is only interesting if it beats simple ones out of sample.
| Benchmark | Forecast of yoy inflation at t + h | Rationale |
|---|---|---|
| Naive / random walk | yoy at t | "inflation stays where it is", famously hard to beat at short horizons (Atkeson & Ohanian, 2001) |
| Historical mean | average yoy in the training window | "inflation returns to its average" |
| Target | 2.0% | "inflation returns to the Bank's target" (a natural benchmark under inflation targeting) |
| AR(p) | autoregression on past yoy | the classic statistical benchmark (Day 63) |
Metrics
| Metric | Formula | Notes |
|---|---|---|
| MAE | mean |error| | in percentage points; robust to occasional large errors |
| RMSE | √mean(error²) | penalises big misses more |
| Relative RMSE | RMSE(model) ÷ RMSE(naive) | < 1 means the model beats the naive benchmark; the headline number in forecasting papers |
| MAPE | mean |error / actual| | avoid for inflation: it explodes when inflation is near zero (e.g. 2009, 2015, 2020) |
Report errors per horizon. A model can win at 1 month and lose at 12.
61.6 Mini-Project: The Evaluation Harness + First Linear Models (25 min)
forecasting/features.py
"""Leakage-safe features: everything in row t uses data dated ≤ t only."""
import pandas as pd
def lag_features(y: pd.Series, n_lags: int, prefix: str) -> pd.DataFrame:
"""Columns prefix_lag0 … prefix_lag{n-1}: lag0 is the value at t itself (known at t)."""
return pd.concat({f"{prefix}_lag{k}": y.shift(k) for k in range(n_lags)}, axis=1)
def univariate_cpi_features(df: pd.DataFrame, n_lags: int = 12) -> pd.DataFrame:
y, mom = df["cpi_yoy"], df["cpi_mom"]
X = lag_features(y, n_lags, "yoy")
X["yoy_ma3"] = y.rolling(3).mean() # trailing windows only
X["yoy_ma12"] = y.rolling(12).mean()
X["mom_lag0"] = mom
X["mom_ma3"] = mom.rolling(3).mean()
X["month_of_year"] = X.index.month
return X
forecasting/evaluate.py
"""Rolling-origin (pseudo out-of-sample) evaluation and forecast metrics."""
import numpy as np
import pandas as pd
def rolling_origin(X: pd.DataFrame, y: pd.Series, model_factory, horizons,
test_start: str, model_name: str, refit_every: int = 1,
min_train: int = 60) -> pd.DataFrame:
"""Direct multi-horizon backtest with an expanding window.
At origin t, training rows are those whose TARGET date (feature date + h) is ≤ t.
"""
rows = []
feature_ok = X.notna().all(axis=1)
origins = X.index[X.index >= pd.Timestamp(test_start)]
last_obs = y.index.max()
for h in horizons:
target = y.shift(-h)
target_dates = X.index.shift(h, freq="MS")
model = None
for i, t in enumerate(origins):
t_target = t + pd.DateOffset(months=h)
if t_target > last_obs:
break # outcome not yet observed
train = feature_ok & target.notna() & (target_dates <= t)
if train.sum() < min_train or not feature_ok.loc[t]:
continue
if model is None or i % refit_every == 0:
model = model_factory()
model.fit(X[train], target[train])
pred = float(model.predict(X.loc[[t]])[0])
rows.append({"model": model_name, "h": h, "origin": t, "target_date": t_target,
"forecast": pred, "actual": float(y.loc[t_target])})
return pd.DataFrame(rows)
def score(results: pd.DataFrame, benchmark: str = "naive") -> pd.DataFrame:
"""MAE, RMSE and RMSE relative to the benchmark, per model and horizon,
computed on the origins common to all models (a fair comparison)."""
common = results.groupby(["h", "origin"])["model"].transform("nunique") == \
results["model"].nunique()
r = results[common].assign(err=lambda d: d["forecast"] - d["actual"])
out = r.groupby(["model", "h"])["err"].agg(
MAE=lambda e: e.abs().mean(),
RMSE=lambda e: np.sqrt((e ** 2).mean()),
n="size").reset_index()
bench = out[out["model"] == benchmark].set_index("h")["RMSE"]
out["rel_RMSE"] = out["RMSE"] / out["h"].map(bench)
return out.sort_values(["h", "rel_RMSE"])
X.index.shift(h, freq="MS") moves every month-start date forward by h months: the target date of each row.
forecasting/models.py: benchmarks as scikit-learn-style objects
"""Benchmarks with fit/predict, so they plug into the same harness as ML models."""
import numpy as np
class Naive:
"""Random walk: forecast = the latest observed value (feature yoy_lag0)."""
def fit(self, X, y):
return self
def predict(self, X):
return X["yoy_lag0"].to_numpy()
class HistoricalMean:
def fit(self, X, y):
self.mean_ = float(np.mean(y))
return self
def predict(self, X):
return np.full(len(X), self.mean_)
class Constant:
def __init__(self, value=2.0):
self.value = value
def fit(self, X, y):
return self
def predict(self, X):
return np.full(len(X), self.value)
Run the first horse race
In notebooks/day61.ipynb (kernel .venv-ml), or a script:
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from forecasting.data import load_cpi
from forecasting.evaluate import rolling_origin, score
from forecasting.features import univariate_cpi_features
from forecasting.models import Constant, HistoricalMean, Naive
df = load_cpi()
X, y = univariate_cpi_features(df), df["cpi_yoy"]
H = [1, 3, 6, 12]
candidates = {
"naive": Naive,
"hist_mean": HistoricalMean,
"target_2pct": Constant,
"ols_lags": LinearRegression,
"ridge_lags": lambda: make_pipeline(StandardScaler(),
RidgeCV(alphas=[0.1, 1, 10, 100, 1000])),
}
results = pd.concat([rolling_origin(X, y, f, H, "2012-01-01", name)
for name, f in candidates.items()], ignore_index=True)
print(score(results).round(3).to_string(index=False))
Things to notice:
make_pipeline(StandardScaler(), RidgeCV(...))fits the scaler inside each training window. Scaling with full-sample statistics would leak (61.4). Pipelines make the leak-free version the easy version.RidgeCVchooses its penalty by internal cross-validation. That's a small leak in principle, since its folds aren't time-ordered; Days 64–67 do tuning properly.- Read the
rel_RMSEcolumn. Does anything beat the naive forecast at h = 1? At h = 12? Does "return to target" do well at long horizons? Write down your observations. Day 63 tests whether the differences are statistically meaningful.
Try it 61.6
Re-run with load_cpi(start="1960-01-01"). Which models change most, and why might pre-1991 data help or hurt forecasts of post-2012 inflation?
Discussion
The historical mean typically gets worse: it's pulled up by the high inflation of the 1970s–80s, which hasn't recurred under inflation targeting. Lag-based linear models may change less, since they mostly use recent values, but their coefficients now blend two regimes with different persistence. More data is only better if it comes from the same data-generating process. Regime changes are why macro forecasters think hard about sample start dates.
Day 61 checklist
- I can explain ŷ vs β̂ problems and the bias–variance trade-off
-
.venv-mlis set up; models read the testedml_cpi_monthlymart viaload_cpi - I can define origin, horizon, target and information set, and build a direct supervised dataset
- My rolling-origin harness excludes training rows whose targets lie after the origin
- I compare every model with naive, mean and target benchmarks using MAE, RMSE and relative RMSE
git add forecasting macro_dbt notebooks .gitignore
git commit -m "Day 61: ML environment, CPI mart, rolling-origin harness, benchmarks"
git push