| Time | Block |
|---|---|
| 0:00 – 0:20 | 67.1 Time-series foundation models |
| 0:20 – 0:43 | 67.2 Installing Chronos and a first zero-shot forecast |
| 0:43 – 0:50 | Break |
| 0:50 – 1:20 | 67.3 Chronos in the rolling-origin harness |
| 1:20 – 1:40 | 67.4 Probabilistic evaluation: pinball loss and coverage |
| 1:40 – 1:48 | Break |
| 1:48 – 2:08 | 67.5 Design choices: context length, levels vs rates, model size |
| 2:08 – 2:30 | 67.6 Forecast combinations and the Day 67 leaderboard |
67.1 Time-Series Foundation Models (20 min)
Every model so far was trained on our data. A foundation model is pretrained on a very large, diverse collection of time series, and can then forecast a new series zero-shot: given only its recent history (the context), with no training on it.
Chronos (Ansari et al., 2024, "Chronos: Learning the Language of Time Series") is a family of open-source models from Amazon. Its key idea is to treat forecasting like language modelling:
- scale each series (so CPI in percent and GDP in dollars look alike);
- quantise values into a vocabulary of tokens;
- use a language-model architecture to predict the next tokens, sampling many futures, which yields probabilistic forecasts (quantiles).
Later variants (e.g. Chronos-Bolt) are much faster and produce quantiles directly, which suits a CPU-only Codespace.
| Models trained on our data (XGBoost, AR) | Zero-shot foundation model (Chronos) | |
|---|---|---|
| training | on our series, at every origin | none (pretrained once by the provider) |
| inputs | engineered features | raw history of the series (context) |
| strengths | uses our features and economic knowledge | no feature engineering; probabilistic by default; works with short histories |
| weaknesses | needs enough history; tuning | can't use our covariates (in the basic setup); a large model to download; possible training-data overlap |
Contamination caveat. Foundation models are pretrained on public data. If similar macro series were in the pretraining corpus, a backtest over historical periods could be optimistic, because the model may have "seen" the future. The cleanest evaluation uses origins after the model's release. Keep this in mind when interpreting results (67.6).
67.2 Installing Chronos and a First Zero-Shot Forecast (23 min)
Install a CPU-only build of PyTorch first (much smaller than the default GPU build), then the Chronos package:
source .venv-ml/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install chronos-forecasting
pip freeze > forecasting/requirements-ml.txt
python -c "import chronos, torch; print('chronos ok, torch', torch.__version__)"
The model weights are downloaded from the Hugging Face Hub the first time you load a model (a few hundred MB for small variants). They're cached in ~/.cache/huggingface/.
A first forecast
import numpy as np
import torch
from chronos import BaseChronosPipeline
from forecasting.data import load_cpi
pipe = BaseChronosPipeline.from_pretrained(
"amazon/chronos-bolt-small", # fast, CPU-friendly; try "-tiny" or "-base" later
device_map="cpu",
torch_dtype=torch.float32,
)
y = load_cpi()["cpi_yoy"]
context = torch.tensor(y.to_numpy(), dtype=torch.float32)
quantiles, mean = pipe.predict_quantiles(context=context, prediction_length=12,
quantile_levels=[0.1, 0.5, 0.9])
q = quantiles[0].numpy() # shape: (12 horizons, 3 quantiles)
future = pd.date_range(y.index.max() + pd.DateOffset(months=1), periods=12, freq="MS")
print(pd.DataFrame(q, index=future, columns=["q10", "median", "q90"]).round(2))
Plot the history, the median path and the 10–90% band. Does the forecast revert towards ~2%? How wide is the band at 12 months compared with your Day 63 XGBoost intervals?
Pin the model version for reproducibility, as you pinned package versions (Day 3). Hugging Face models can be updated; passing a specific
revision="<commit hash>"tofrom_pretrainedfreezes it. Record the model name and revision in every report.
67.3 Chronos in the Rolling-Origin Harness (30 min)
To compare fairly, Chronos must forecast from the same origins, using only data up to each origin. Because it needs no training, we can batch all origins in one call.
Create forecasting/chronos_models.py:
"""Zero-shot Chronos forecasts in the rolling-origin framework."""
import pandas as pd
import torch
from chronos import BaseChronosPipeline
QUANTILES = [0.1, 0.5, 0.9]
def load_pipeline(name: str = "amazon/chronos-bolt-small", revision: str | None = None):
kwargs = {"revision": revision} if revision else {}
return BaseChronosPipeline.from_pretrained(name, device_map="cpu",
torch_dtype=torch.float32, **kwargs)
def rolling_origin_chronos(y: pd.Series, horizons, test_start: str, pipe,
model_name: str = "chronos", max_context: int | None = None,
batch_size: int = 32) -> pd.DataFrame:
"""For each origin t, forecast from y[:t] only (optionally the last `max_context` points)."""
origins = [t for t in y.index[y.index >= pd.Timestamp(test_start)]]
H = max(horizons)
rows = []
for start in range(0, len(origins), batch_size):
batch = origins[start:start + batch_size]
contexts = []
for t in batch:
hist = y.loc[:t].to_numpy()
if max_context:
hist = hist[-max_context:]
contexts.append(torch.tensor(hist, dtype=torch.float32))
q, _ = pipe.predict_quantiles(context=contexts, prediction_length=H,
quantile_levels=QUANTILES)
for j, t in enumerate(batch):
for h in horizons:
t_target = t + pd.DateOffset(months=h)
if t_target > y.index.max():
continue
q10, q50, q90 = (float(v) for v in q[j, h - 1].tolist())
rows.append({"model": model_name, "h": h, "origin": t, "target_date": t_target,
"forecast": q50, "q10": q10, "q90": q90,
"actual": float(y.loc[t_target])})
return pd.DataFrame(rows)
Passing a list of tensors of different lengths is fine: the pipeline handles the padding. Each origin sees only its own past, y.loc[:t], so there's no leakage by construction.
from forecasting.chronos_models import load_pipeline, rolling_origin_chronos
pipe = load_pipeline("amazon/chronos-bolt-small")
chronos = rolling_origin_chronos(y, [1, 3, 6, 12], "2012-01-01", pipe, "chronos_bolt_small")
chronos.to_parquet("data/forecasts/chronos_bolt_small.parquet", index=False)
prev = pd.read_parquet("data/forecasts/univariate_forecasts.parquet")
board = score(pd.concat([prev, chronos[prev.columns]]))
print(board.query("model in ['naive', 'ar_aic', 'xgboost', 'chronos_bolt_small']").round(3))
for h in [1, 3, 6, 12]:
print(diebold_mariano(pd.concat([prev, chronos[prev.columns]]), "chronos_bolt_small", "naive", h))
Record the runtime: Chronos-Bolt on CPU handles a few hundred contexts in seconds to minutes, with no training at any origin. That's a remarkable contrast with nested XGBoost tuning (Day 64).
67.4 Probabilistic Evaluation: Pinball Loss and Coverage (20 min)
Point metrics (RMSE) ignore uncertainty. Chronos produces quantiles, so evaluate them as quantiles.
Pinball (quantile) loss
For quantile level α, forecast q and outcome y:
L_α(y, q) = α · (y − q) if y ≥ q, else (1 − α) · (q − y)
It's the loss that quantile regression minimises (Day 63.5). Lower is better. Averaging across α = 0.1, 0.5 and 0.9 summarises the whole forecast distribution.
Create forecasting/scoring.py:
"""Probabilistic forecast scores and forecast combinations."""
import numpy as np
import pandas as pd
def pinball(y, q, alpha):
diff = y - q
return np.where(diff >= 0, alpha * diff, (alpha - 1) * diff)
def quantile_scores(res: pd.DataFrame) -> pd.DataFrame:
"""Expects columns q10, forecast (median), q90, actual. Per model and horizon."""
r = res.assign(
pinball_avg=(pinball(res["actual"], res["q10"], 0.1)
+ pinball(res["actual"], res["forecast"], 0.5)
+ pinball(res["actual"], res["q90"], 0.9)) / 3,
inside=(res["actual"] >= res["q10"]) & (res["actual"] <= res["q90"]),
width=res["q90"] - res["q10"],
)
return (r.groupby(["model", "h"])
.agg(pinball_avg=("pinball_avg", "mean"),
coverage_80=("inside", "mean"),
avg_width=("width", "mean"))
.reset_index())
Compare Chronos with your Day 63 XGBoost quantile models (add their q10/q90 columns to their results, using the same column names):
print(quantile_scores(chronos).round(3))
| Score | Good value |
|---|---|
| average pinball loss | lower than competitors |
| 80% coverage | close to 0.80 |
| average width | as narrow as possible given good coverage |
Check coverage by year again (Day 63.5). Do Chronos's intervals widen when inflation becomes volatile? Foundation models scale their uncertainty to the recent volatility in the context, which can help them react faster to regime changes than models calibrated on calm training periods.
67.5 Design Choices: Context Length, Levels vs Rates, Model Size (20 min)
Zero-shot doesn't mean choice-free. Run three small experiments, each through rolling_origin_chronos and score/quantile_scores.
Experiment 1 — How much history should Chronos see?
for ctx in [36, 120, None]: # 3 years, 10 years, all since 1992
r = rolling_origin_chronos(y, [1, 12], "2012-01-01", pipe, f"chronos_ctx{ctx or 'all'}",
max_context=ctx)
print(ctx, score(pd.concat([r[prev.columns], prev.query("model == 'naive'")]))
.query("model != 'naive'")[["h", "rel_RMSE"]].to_dict("records"))
Short contexts let the model focus on the current regime; long contexts show it the full range of behaviour, including mean reversion towards target.
Experiment 2 — Forecast the level, then compute inflation
Instead of forecasting yoy inflation directly, give Chronos the CPI index (with its seasonal pattern) and convert its forecast into yoy:
yoy(t+h) = 100 × (CPI(t+h) / CPI(t+h−12) − 1), where CPI(t+h−12) is already known for h ≤ 12.
cpi = load_cpi()["cpi"]
lvl = rolling_origin_chronos(cpi, [1, 3, 6, 12], "2012-01-01", pipe, "chronos_level")
lvl["forecast"] = 100 * (lvl["forecast"] / [cpi.loc[d - pd.DateOffset(months=12)]
for d in lvl["target_date"]] - 1)
lvl["actual"] = [float(y.loc[d]) for d in lvl["target_date"]]
(Convert q10 and q90 the same way if you want interval scores.) Which works better: forecasting the rate, or forecasting the level and deriving the rate? There's no universal answer, but you can now test it in an afternoon.
Experiment 3 — Model size
Try amazon/chronos-bolt-tiny and amazon/chronos-bolt-base. Record accuracy and runtime. In production, a slightly less accurate model that runs in seconds may be preferable to a larger one that takes minutes, depending on the deadline (Day 54's retry-budget thinking).
Beyond zero-shot. Chronos models can be fine-tuned on your own data, and newer variants can take covariates (other series as inputs). Both are worth exploring after this module. Check the package documentation for your installed version, since these capabilities evolve quickly.
67.6 Forecast Combinations and the Day 67 Leaderboard (23 min)
Combining forecasts
A long literature, going back to Bates and Granger (1969), finds that averaging forecasts from different models often beats the individual models, especially when they make different kinds of errors. Simple equal weights are notoriously hard to beat with "optimised" weights (sometimes called the forecast combination puzzle).
Our models are genuinely different: AR (linear persistence), XGBoost (non-linear, feature-based), Chronos (pretrained, context-based). Add to forecasting/scoring.py:
def combine(results: pd.DataFrame, members: list[str], name: str = "combo_equal") -> pd.DataFrame:
"""Equal-weight average of point forecasts on origins where all members are available."""
r = results[results["model"].isin(members)]
complete = r.groupby(["h", "origin"])["model"].transform("nunique") == len(members)
combo = (r[complete].groupby(["h", "origin", "target_date"], as_index=False)
.agg(forecast=("forecast", "mean"), actual=("actual", "first")))
return combo.assign(model=name)
The Day 67 leaderboard
Write forecasting/leaderboard_day67.py that assembles naive, AR(AIC), XGBoost (default), XGBoost (tuned, Day 64), Chronos-Bolt (best context from 67.5) and the equal-weight combination of AR + XGBoost + Chronos. It should report, per horizon:
- RMSE, relative RMSE, and the Diebold–Mariano test vs naive;
- pinball loss and 80% coverage for models with quantiles;
- the same table restricted to origins after the Chronos model's release date, as a contamination check (67.1).
Save it to data/forecasts/leaderboard_day67.parquet with a Markdown report, including package versions and the Chronos model name and revision.
How to read the result, and what to carry into Days 68–70
Common patterns, which you should verify on your own data:
- At short horizons, naive and AR remain very hard to beat; differences are often insignificant.
- At longer horizons, models with mean reversion (AR, XGBoost trained on the inflation-targeting era, and often Chronos with a long context) tend to beat naive.
- The combination is frequently among the best and rarely the worst, which makes it a robust default.
- On the post-release window the sample is short, so conclusions are tentative. Say so.
Decision for Days 68–70: carry forward (1) AR(AIC) as the benchmark, (2) XGBoost, which will gain multivariate features from the marts, (3) Chronos as a zero-shot comparator, and (4) the equal-weight combination. Record this decision, with your evidence, in forecasting/MODEL_DECISIONS.md.
Recap: Days 64–67
| Topic | You can now… |
|---|---|
| tuning | tune with time-series CV and a horizon gap; use randomized search and Optuna; evaluate tuning as a nested procedure |
| explainability | compute exact TreeSHAP contributions, verify additivity, and read global, local and dependence plots |
| explanations over time | explain each real-time forecast, decompose forecasts by feature family, monitor explanation drift |
| communication | publish an explained forecast as a dashboard page with honest caveats |
| foundation models | run Chronos zero-shot in the same rolling-origin harness, batched over origins |
| probabilistic evaluation | score quantile forecasts with pinball loss and coverage |
| robustness | combine forecasts and check for training-data contamination |
Looking ahead: Days 68–70
The curriculum's goal for this module: forecasting Canadian CPI and real GDP growth using the historical features engineered in earlier modules. You'll build a multivariate feature mart from macro_monthly and macro_quarterly (rates, spreads, FX, oil, labour market, core measures), respect publication lags and data vintages so the backtest reflects what was really known, extend everything to quarterly GDP, and turn the forecasts, explanations and dashboard into tasks in the Prefect pipeline.
Day 67 checklist
- I can explain what a time-series foundation model is, and the contamination caveat
- Chronos runs on CPU with a pinned model version, producing quantile forecasts
- Chronos is evaluated on exactly the same origins and horizons as the other models, without leakage
- I score probabilistic forecasts with pinball loss and coverage
- I tested context length, level-vs-rate targets and model size
- A combined forecast and a full leaderboard (with a post-release check) inform documented model decisions
git add forecasting notebooks && git commit -m "Day 67: Chronos zero-shot, probabilistic scores, combinations, leaderboard" && git push