Written lesson
| Time | Block |
|---|---|
| 0:00 – 0:15 | 70.1 From notebook results to a forecasting service |
| 0:15 – 0:45 | 70.2 run_forecasts.py: reproducible forecasts with metadata |
| 0:45 – 0:53 | Break |
| 0:53 – 1:15 | 70.3 Forecast marts in dbt: history, latest, and live accuracy |
| 1:15 – 1:35 | 70.4 Quality gates and monitoring for forecasts |
| 1:35 – 1:43 | Break |
| 1:43 – 2:05 | 70.5 Orchestration: the forecast subflow |
| 2:05 – 2:20 | 70.6 The model card and the forecast dashboard |
| 2:20 – 2:30 | 70.7 Module 7 recap and the road to causal inference |
70.1 From Notebook Results to a Forecasting Service (15 min)
Backtests answer "how good would this have been?". Production answers "what do we forecast today, and how is it doing?". The principles you applied to data apply to forecasts:
| Data platform principle | For forecasts |
|---|---|
| raw data is kept, versioned and traceable (Modules 3, 5) | every published forecast is kept, with the data, code and model that produced it |
| marts are tested before use (Module 4) | forecasts pass quality checks before reaching dashboards |
| quality is monitored over time (Day 47) | live accuracy is tracked as outcomes arrive: a true out-of-sample test that no backtest can match |
| documented contracts (Days 40–41) | a model card states intended use and limits |
| orchestration with retries and alerts (Module 6) | forecasting is a task in the pipeline, triggered by new data |
Retraining policy for this course:
- Every run: refit the chosen models on all data available (cheap for our sizes) with fixed hyperparameters from
MODEL_DECISIONS.md. - Quarterly: rerun the leaderboards (Days 63, 67–69) and review the model decisions. Changes are made deliberately, through a reviewed update to
MODEL_DECISIONS.mdand the specs, never automatically.
70.2 run_forecasts.py: Reproducible Forecasts with Metadata (30 min)
Create forecasting/run_forecasts.py (runs in .venv-ml; exit codes follow the platform convention: 0 ok · 1 failed · 2 warnings):
"""Produce today's CPI and GDP forecasts, intervals and explanations, with full metadata.
Usage: python -m forecasting.run_forecasts
Writes: data/forecasts/published/run_<ts>.parquet (forecasts)
data/forecasts/published/expl_<ts>.parquet (SHAP family contributions)
Exit: 0 ok · 1 failure · 2 warnings (e.g. inputs outside the training range)
"""
import hashlib
import sys
import uuid
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
import pandas as pd
import xgboost
from forecasting.availability import build_features, load_monthly_panel, load_spec
from forecasting.data import MARTS, load_cpi, load_gdp
from forecasting.explain import xgb_contributions
from forecasting.features import univariate_cpi_features
from forecasting.models import xgb_factory
from forecasting.monitoring import out_of_range_report
from forecasting.nowcast import nowcast_dataset
OUT = Path("data/forecasts/published")
CPI_H = [1, 3, 6, 12]
def fingerprint(paths) -> str:
"""Hash of the input files' contents: identifies exactly which data produced a forecast."""
h = hashlib.sha256()
for p in sorted(paths):
h.update(Path(p).read_bytes())
return h.hexdigest()[:16]
def interval_from_backtest(errors: pd.Series, point: float) -> tuple[float, float]:
lo, hi = np.quantile(errors.dropna(), [0.1, 0.9])
return point + lo, point + hi
def cpi_forecasts(run_meta: dict, backtest: pd.DataFrame) -> tuple[list, list, list]:
cpi, panel = load_cpi(), load_monthly_panel()
spec = load_spec("forecasting/feature_spec_cpi.yml")
X_mac, fam = build_features(panel, spec)
X = univariate_cpi_features(cpi).join(X_mac, how="left").loc["2006-01":]
y = cpi["cpi_yoy"]
origin = y.index.max()
rows, expl, warnings = [], [], []
for h in CPI_H:
target = y.shift(-h)
known = X.index.shift(h, freq="MS") <= origin
train = X.notna().any(axis=1) & target.notna() & known
model = xgb_factory().fit(X[train], target[train])
x0 = X.loc[[origin]]
point = float(model.predict(x0)[0])
bt = backtest.query("model == 'xgb_multivariate' and h == @h")
lo, hi = interval_from_backtest(bt["forecast"] - bt["actual"], point)
rows.append({**run_meta, "target": "cpi_yoy", "model": "xgb_multivariate", "horizon": h,
"origin": origin, "target_period": origin + pd.DateOffset(months=h),
"forecast": point, "q10": lo, "q90": hi})
rows.append({**run_meta, "target": "cpi_yoy", "model": "naive", "horizon": h,
"origin": origin, "target_period": origin + pd.DateOffset(months=h),
"forecast": float(y.loc[origin]), "q10": np.nan, "q90": np.nan})
c = xgb_contributions(model, x0).iloc[0]
for feat, val in c.items():
expl.append({**run_meta, "target": "cpi_yoy", "horizon": h, "feature": feat,
"family": fam.get(feat, "base_value" if feat == "base_value" else "own_history"),
"contribution": float(val)})
warnings += out_of_range_report(X[train], x0, label=f"cpi h={h}")
return rows, expl, warnings
def gdp_nowcast(run_meta: dict) -> tuple[list, list]:
panel, gdp = load_monthly_panel(), load_gdp()["real_qoq_ann"]
gspec = load_spec("forecasting/feature_spec_gdp.yml")
rows, warnings = [], []
for hq, label in [(0, "nowcast"), (1, "next_quarter")]:
D = nowcast_dataset(panel, gdp, gspec, horizon_q=hq)
feats = [c for c in D.columns if c not in ("target", "target_quarter", "target_known")]
origin = D.index.max()
train = D["target"].notna() & (D["target_known"] <= origin)
model = xgb_factory().fit(D.loc[train, feats], D.loc[train, "target"])
point = float(model.predict(D.loc[[origin], feats])[0])
rows.append({**run_meta, "target": "real_gdp_qoq_ann", "model": f"xgb_{label}",
"horizon": hq, "origin": origin,
"target_period": D.loc[origin, "target_quarter"],
"forecast": point, "q10": np.nan, "q90": np.nan})
warnings += out_of_range_report(D.loc[train, feats], D.loc[[origin], feats],
label=f"gdp {label}")
return rows, warnings
def main() -> int:
OUT.mkdir(parents=True, exist_ok=True)
now = datetime.now(timezone.utc)
inputs = [MARTS / f for f in ("macro_monthly.parquet", "ml_cpi_monthly.parquet",
"ml_gdp_quarterly.parquet")]
run_meta = {
"run_id": uuid.uuid4().hex[:12],
"as_of": now,
"data_fingerprint": fingerprint(inputs),
"spec_fingerprint": fingerprint([Path("forecasting/feature_spec_cpi.yml"),
Path("forecasting/feature_spec_gdp.yml")]),
"xgboost_version": xgboost.__version__,
}
try:
backtest = pd.read_parquet("data/forecasts/cpi_multivariate.parquet")
cpi_rows, expl, w1 = cpi_forecasts(run_meta, backtest)
gdp_rows, w2 = gdp_nowcast(run_meta)
except Exception as e:
print(f"✗ forecasting failed: {type(e).__name__}: {e}")
return 1
stamp = now.strftime("%Y%m%dT%H%M%S")
pd.DataFrame(cpi_rows + gdp_rows).to_parquet(OUT / f"run_{stamp}.parquet", index=False)
pd.DataFrame(expl).to_parquet(OUT / f"expl_{stamp}.parquet", index=False)
print(f"✓ published run {run_meta['run_id']} ({len(cpi_rows) + len(gdp_rows)} forecasts)")
warnings = w1 + w2
for w in warnings:
print(f"⚠ {w}")
return 2 if warnings else 0
if __name__ == "__main__":
sys.exit(main())
Every published row carries its provenance: run_id, as_of, a fingerprint of the input data, a fingerprint of the feature specs, and the library version. Six months from now you'll be able to answer "which data and which features produced the forecast we briefed in March?". That's the forecasting counterpart of Day 37's snapshots and Day 47's validation history.
The naive forecast is published alongside the model's, so live accuracy (70.3) always has its benchmark.
70.3 Forecast Marts in dbt: History, Latest, and Live Accuracy (23 min)
Published forecasts are data, so model them in dbt (Day 47's external-source pattern).
macro_dbt/models/staging/forecasts/_forecasts__sources.yml
version: 2
sources:
- name: forecasts
description: "Forecasts published by forecasting/run_forecasts.py (Module 7)."
meta:
external_location: "../data/forecasts/published/run_*.parquet"
tables:
- name: runs
stg_forecasts__runs.sql casts types: cast(origin as date), cast(target_period as date), cast(as_of as timestamp).
marts/forecasts/forecast_latest.sql
-- Grain: one row per target × model × horizon, from the most recent run.
with ranked as (
select *,
row_number() over (partition by target, model, horizon order by as_of desc) as rn
from {{ ref('stg_forecasts__runs') }}
)
select * exclude (rn) from ranked where rn = 1
marts/forecasts/forecast_accuracy.sql: the payoff of keeping every forecast:
-- Grain: one row per published forecast whose outcome is now known.
-- This is TRUE out-of-sample evaluation: forecasts were made before outcomes existed.
with f as (
select * from {{ ref('stg_forecasts__runs') }}
),
cpi_actual as (
select month as target_period, cpi_yoy as actual
from {{ ref('ml_cpi_monthly') }}
),
gdp_actual as (
select quarter as target_period, real_qoq_ann as actual
from {{ ref('ml_gdp_quarterly') }}
),
joined as (
select f.*, a.actual
from f
join cpi_actual as a using (target_period)
where f.target = 'cpi_yoy'
union all by name
select f.*, a.actual
from f
join gdp_actual as a using (target_period)
where f.target = 'real_gdp_qoq_ann'
)
select
*,
forecast - actual as error,
abs(forecast - actual) as abs_error,
case when q10 is not null
then actual between q10 and q90 end as inside_80
from joined
where actual is not null
Add a small summary mart, forecast_track_record: MAE and RMSE per target, model and horizon over the last 24 months of live forecasts, with the model's RMSE relative to the naive forecast published in the same runs.
Why this matters more than any backtest: live accuracy uses forecasts that were genuinely made before the outcomes. No leakage is possible, no tuning on the test period, and no pretraining contamination for Chronos. Over time,
forecast_track_recordbecomes the most credible evidence about your models.
70.4 Quality Gates and Monitoring for Forecasts (20 min)
Inputs outside the training range
Trees can't extrapolate (Day 62.5). When today's inputs lie outside the range seen in training, the forecast deserves a warning. Create forecasting/monitoring.py:
"""Monitoring helpers: flag inputs outside the training distribution."""
import pandas as pd
def out_of_range_report(X_train: pd.DataFrame, x_now: pd.DataFrame, label: str,
z_limit: float = 4.0) -> list[str]:
"""Warn for features outside the training min–max, or more than z_limit std from the mean."""
warnings = []
for col in x_now.columns:
v = x_now[col].iloc[0]
s = X_train[col].dropna()
if pd.isna(v) or s.empty:
continue
if v < s.min() or v > s.max():
warnings.append(f"{label}: {col}={v:.2f} outside training range "
f"[{s.min():.2f}, {s.max():.2f}]")
elif s.std() > 0 and abs(v - s.mean()) / s.std() > z_limit:
warnings.append(f"{label}: {col}={v:.2f} is {abs(v - s.mean()) / s.std():.1f} std "
f"from its training mean")
return warnings
run_forecasts.py returns exit code 2 when such warnings occur, so the forecast is published with a flag (Day 53's CompletedWithWarnings) and the dashboard can say so.
GX checks on published forecasts
Register marts.forecast_latest as a GX table asset (Day 43, setup_gx.py) and add a spec, quality/specs/forecast_latest.yml (Day 44 format):
description: "Latest published forecasts: completeness and coherence before dashboards use them."
asset: forecast_latest
batch_definition: full
owner: macro-team
critical:
- type: ExpectColumnValuesToNotBeNull
kwargs: {column: forecast}
meta: {plan_id: F01, dimension: completeness}
- type: ExpectColumnPairValuesAToBeGreaterThanB
kwargs: {column_A: q90, column_B: forecast, or_equal: true, ignore_row_if: either_value_is_missing}
meta: {plan_id: F02, dimension: consistency, rationale: "Upper bound ≥ point forecast."}
- type: ExpectColumnPairValuesAToBeGreaterThanB
kwargs: {column_A: forecast, column_B: q10, or_equal: true, ignore_row_if: either_value_is_missing}
meta: {plan_id: F03, dimension: consistency, rationale: "Point forecast ≥ lower bound."}
- type: ExpectTableRowCountToBeBetween
kwargs: {min_value: 10}
meta: {plan_id: F04, dimension: completeness,
rationale: "4 CPI horizons × 2 models + 2 GDP targets."}
warning:
- type: ExpectColumnValuesToBeBetween
kwargs: {column: forecast, min_value: -5, max_value: 15, mostly: 1.0}
meta: {plan_id: F05, dimension: plausibility,
rationale: "CPI forecasts outside −5…15% (or GDP outside it) need review before publication."}
Rebuild suites (python quality/build_suites.py). The forecast checks join the warehouse critical/warning checkpoints, so run_quality.sh now protects forecasts too.
70.5 Orchestration: The Forecast Subflow (23 min)
Add the interpreter for the modelling environment to orchestration/settings.py:
ML_PY = str(REPO / ".venv-ml" / "bin" / "python")
and a step policy (Day 54):
"forecast_models": dict(retries=1, delays=[300], timeout=1800,
transient=(1,), warning=(2,)),
(A Chronos model download or a temporary file-system issue is worth one retry. Exit code 2, inputs out of range, is a warning.)
Create orchestration/flows/forecast.py:
"""Forecast subflow: produce forecasts, model them in dbt, validate, publish."""
from prefect import flow
from orchestration.settings import ML_PY
from orchestration.tasks.commands import step
from orchestration.tasks.dbt import dbt_invoke
@flow(name="forecast")
def forecast_flow(target: str = "prod", drill: dict | None = None) -> dict:
states = {"forecast_models": step("forecast_models",
[ML_PY, "-m", "forecasting.run_forecasts"], drill=drill)}
if states["forecast_models"].is_completed():
states["dbt_forecasts"] = dbt_invoke.with_options(name="dbt:forecasts")(
["build", "--target", target, "--select", "source:forecasts+"], return_state=True)
return states
Where it runs:
daily_refresh(Day 60): afterquality_flowand beforepublish_flow, so dashboards include the newest forecasts. Ifforecast_modelsfails, the policy publishes dashboards without updated forecasts (yesterday's remain, with theiras_ofdates visible) and ends the run CompletedWithWarnings.statcan_release(Day 59): after a CPI, LFS or GDP release is published, callforecast_flow(), so new information reaches the forecast within minutes.
Forecasting reads only Parquet marts, so it needs no DuckDB lock. The dbt step that models forecasts is a writer, and dbt_invoke already takes the writer slot (Day 59). Update STATE_POLICY.md with the new rows.
70.6 The Model Card and the Forecast Dashboard (15 min)
forecasting/MODEL_CARD.md
A model card (Mitchell et al., 2019) documents a model the way a data contract documents a dataset. Write one with these sections:
# Model card — Canadian CPI inflation and real GDP growth forecasts
## Intended use
Statistical forecasts of CPI yoy inflation (1–12 months) and real GDP growth (nowcast, next quarter)
for internal briefings and dashboards. NOT for: causal policy evaluation; official projections.
## Models
- CPI: XGBoost (direct, one model per horizon), multivariate features (feature_spec_cpi.yml);
benchmark: naive (published alongside). Hyperparameters: MODEL_DECISIONS.md.
- GDP: XGBoost nowcast/next-quarter on ragged-edge monthly indicators (feature_spec_gdp.yml).
## Data
Marts: ml_cpi_monthly, macro_monthly, ml_gdp_quarterly (dbt, tested; GX-validated).
Publication lags encoded per feature. GDP target = latest vintage.
## Evaluation
Pseudo-real-time rolling-origin backtests from 2014 (CPI) / 2012 (GDP); Diebold–Mariano vs benchmarks;
interval coverage; results in data/forecasts/*leaderboard*. Live track record: marts.forecast_track_record.
## Limitations and caveats
- Tree models cannot extrapolate beyond the training range (flagged at run time).
- 2020–21 outliers dominate GDP errors; results reported with and without.
- Backtests use revised data for LFS, OECD CLI and GDP (pseudo-real-time).
- SHAP explanations describe the model's associations, not causal effects.
- Chronos (comparator) may have seen public macro series in pretraining.
## Monitoring
Out-of-range input warnings (exit code 2); GX checks F01–F05; live accuracy vs naive; explanation drift.
## Ownership and review
Owner: macro-team. Quarterly re-evaluation; changes via reviewed MODEL_DECISIONS.md updates.
The forecast dashboard
Extend Day 66's forecast_explained page in dashboards/build_dashboards.py to read the dbt-exported marts (forecast_latest, forecast_track_record, and the latest expl_*.parquet):
- CPI fan chart: history, model forecast with the 80% band, and the naive forecast for contrast.
- GDP nowcast for the current quarter, with the latest published quarter beside it.
- Drivers: SHAP family contributions for the 12-month CPI forecast.
- Track record: live RMSE vs naive over time, from
forecast_track_record. - The header shows the forecast
as_of, any out-of-range warning, and the model card's one-line caveat.
Add an exposure forecast_dashboard depending on forecast_latest and forecast_track_record (Day 40), and add the latest 12-month CPI forecast and GDP nowcast as KPIs in macro_kpis (Day 39).
70.7 Module 7 Recap and the Road to Causal Inference (10 min)
What you built in Module 7 (Days 61–70)
| Days | Capability |
|---|---|
| 61 | ŷ vs β̂; a leak-free rolling-origin harness; benchmarks and metrics |
| 62 | trees, forests, boosting from scratch, XGBoost; no extrapolation |
| 63 | AR benchmarks; direct vs recursive; Diebold–Mariano; prediction intervals |
| 64 | leak-free tuning with time-series CV; nested evaluation |
| 65–66 | SHAP: theory, global and local explanations, decompositions over time, explanation drift |
| 67 | zero-shot Chronos; probabilistic scoring; combinations; contamination checks |
| 68 | multivariate CPI forecasts with publication lags; ablation by information family |
| 69 | GDP nowcasting with a ragged edge; robust losses; honest COVID reporting |
| 70 | production forecasts with provenance, live accuracy, quality gates, orchestration, a model card |
How this completes the capstone's "Model & Forecast" row
| Capstone deliverable | Where |
|---|---|
| next-quarter GDP forecasts | Day 69 nowcast and next-quarter models, published daily (Day 70) |
| CPI forecasts | Days 68–70, with explanations and track record |
| integrated with the Prefect-orchestrated pipeline | Day 70's forecast subflow, triggered by schedules and StatCan releases |
| interactive visualisation layer | the forecast dashboard page and cockpit KPIs |
Looking ahead: Module 8 — Causal Inference with DoubleML (Days 71–80)
Forecasting asks what will happen. Policy asks what would happen if…. Throughout this module we warned that SHAP values and feature importance are not causal. Module 8 tackles causality directly:
- Days 71–73: causal inference theory, confounding, and directed acyclic graphs (DAGs) for macro questions.
- Days 74–77: nuisance parameters, cross-fitting and the partially linear model: using flexible ML (including the XGBoost skills you've just built) to control for confounders, while estimating a causal parameter with valid standard errors.
- Days 78–80: estimating the causal impact of Bank of Canada overnight rate hikes on housing starts, controlling for business-cycle indicators. You'll add a housing-starts series through the platform you built (a new dlt source or StatCan table via WAP, a staging model, tests and a mart), the same path every series in this course has followed.
Day 70 checklist
-
run_forecasts.pypublishes forecasts with run IDs, data and spec fingerprints, versions and benchmark forecasts - dbt models keep every forecast, the latest forecasts, and live accuracy against realised outcomes
- Inputs outside the training range raise warnings; GX validates forecast coherence before publication
- The forecast subflow runs in the daily refresh and after relevant StatCan releases, with policy rows documented
- A model card states intended use, data, evaluation, limitations and monitoring
- The dashboard shows forecasts, intervals, drivers, track record and caveats
git add .
git commit -m "Day 70: production forecasts, live accuracy, quality gates, model card — Module 7 complete"
git push