| Time | Block |
|---|---|
| 0:00 – 0:20 | 64.1 Parameters, hyperparameters and the tuning trap |
| 0:20 – 0:45 | 64.2 Time-series cross-validation with a gap |
| 0:45 – 0:53 | Break |
| 0:53 – 1:15 | 64.3 Randomized search |
| 1:15 – 1:40 | 64.4 Bayesian optimisation with Optuna |
| 1:40 – 1:48 | Break |
| 1:48 – 2:08 | 64.5 Nested evaluation: tuning inside the rolling origin |
| 2:08 – 2:30 | 64.6 Mini-project: does tuning pay? |
64.1 Parameters, Hyperparameters and the Tuning Trap (20 min)
| Parameters | Hyperparameters | |
|---|---|---|
| what | learned from data during fitting | chosen before fitting |
| examples | tree splits and leaf values; regression coefficients | max_depth, learning_rate, n_estimators, min_child_weight |
| how chosen | the fitting algorithm | you, using validation evidence |
Tuning means trying many hyperparameter settings and keeping the best. The trap: if "best" is judged on the test period, the test period stops being a test. Its errors become optimistic, for the same reason that adding regressors raises in-sample R². With enough tries, something always looks good by chance.
Three roles for data
|──────── training ────────|── validation ──|─────────── test (rolling origins) ───────────|
fit parameters choose hyper- judge the final method, ONCE
parameters
For time series, the three blocks must be in chronological order, and in a rolling-origin backtest each origin has its own training and validation inside its own past (64.5).
Rule: the test period may be used to evaluate a method (including how it tunes itself), never to choose settings. If you look at test results and then change the search space, you've tuned on the test, so say so honestly or set aside a fresh test period.
64.2 Time-Series Cross-Validation with a Gap (25 min)
scikit-learn's TimeSeriesSplit creates expanding-window folds in chronological order:
fold 1: train [══════] validate [──]
fold 2: train [════════] validate [──]
fold 3: train [══════════] validate [──]
Why a gap is essential for horizon h
Each row's target is h months ahead (Day 61). A training row at the end of a fold has a target date inside the validation period, which a real forecaster standing at the start of validation wouldn't yet know. That's the same trap as Day 61.4, inside cross-validation. The fix is a gap of h rows between training and validation:
from sklearn.model_selection import TimeSeriesSplit
h = 12
cv = TimeSeriesSplit(n_splits=4, test_size=24, gap=h)
from forecasting.data import load_cpi
from forecasting.features import univariate_cpi_features
df = load_cpi()
X = univariate_cpi_features(df)
target = df["cpi_yoy"].shift(-h).rename("target")
data = X.join(target).dropna().loc[:"2011-12"] # tuning happens BEFORE the test period
Xtr, ytr = data.drop(columns="target"), data["target"]
for k, (tr, va) in enumerate(cv.split(Xtr)):
print(f"fold {k}: train {Xtr.index[tr[0]]:%Y-%m}→{Xtr.index[tr[-1]]:%Y-%m} "
f"validate {Xtr.index[va[0]]:%Y-%m}→{Xtr.index[va[-1]]:%Y-%m}")
Check the printout: between the last training month and the first validation month there are exactly h skipped rows. The last training row's target date then falls just before validation starts. No future information leaks.
TimeSeriesSplit argument | Meaning |
|---|---|
n_splits | number of folds |
test_size | rows per validation fold (24 = two years) |
gap | rows skipped between train and validation; set to the forecast horizon h |
64.3 Randomized Search (23 min)
Grid search tries every combination: with 6 hyperparameters and 4 values each, that's 4,096 settings × folds. Randomized search samples a fixed number of settings from distributions, and usually finds equally good settings much faster (most hyperparameters matter little; random sampling explores the important ones more efficiently).
Create forecasting/tuning.py:
"""Hyperparameter tuning for direct-horizon models, without leakage."""
import numpy as np
from scipy.stats import loguniform, randint, uniform
from sklearn.model_selection import RandomizedSearchCV, TimeSeriesSplit
from xgboost import XGBRegressor
XGB_SPACE = {
"n_estimators": randint(100, 800),
"learning_rate": loguniform(0.01, 0.2), # log scale: 0.01 and 0.1 equally likely to be explored
"max_depth": randint(2, 6),
"min_child_weight": randint(1, 20),
"subsample": uniform(0.5, 0.5), # uniform on [0.5, 1.0]
"colsample_bytree": uniform(0.5, 0.5),
"reg_lambda": loguniform(0.1, 20),
}
def ts_cv(h: int, n_splits: int = 4, val_size: int = 24) -> TimeSeriesSplit:
return TimeSeriesSplit(n_splits=n_splits, test_size=val_size, gap=h)
def random_search_xgb(X, y, h: int, n_iter: int = 30, seed: int = 0) -> RandomizedSearchCV:
search = RandomizedSearchCV(
XGBRegressor(random_state=seed, n_jobs=2),
XGB_SPACE, n_iter=n_iter, cv=ts_cv(h),
scoring="neg_root_mean_squared_error", random_state=seed, n_jobs=1, refit=True)
return search.fit(X, y)
search = random_search_xgb(Xtr, ytr, h=12, n_iter=40)
print("best CV RMSE:", -search.best_score_)
print("best params:", search.best_params_)
import pandas as pd
cvres = pd.DataFrame(search.cv_results_).sort_values("rank_test_score")
print(cvres[["param_max_depth", "param_learning_rate", "param_n_estimators",
"mean_test_score", "std_test_score"]].head(10))
Look at std_test_score: if the top settings' scores differ by less than their fold-to-fold standard deviation, they're statistically indistinguishable. Prefer the simpler one (shallower trees, fewer estimators).
loguniform(0.01, 0.2) samples on a logarithmic scale. For parameters spanning orders of magnitude (learning rates, penalties), that's the natural choice.
64.4 Bayesian Optimisation with Optuna (25 min)
Randomized search ignores what it has learned. Bayesian optimisation uses past trials to propose more promising settings. Optuna is a popular library implementing this (with its default TPE sampler).
Add to forecasting/tuning.py:
import optuna
def optuna_xgb(X, y, h: int, n_trials: int = 50, seed: int = 0):
cv = ts_cv(h)
def objective(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 100, 800),
"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.2, log=True),
"max_depth": trial.suggest_int("max_depth", 2, 5),
"min_child_weight": trial.suggest_int("min_child_weight", 1, 20),
"subsample": trial.suggest_float("subsample", 0.5, 1.0),
"colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
"reg_lambda": trial.suggest_float("reg_lambda", 0.1, 20, log=True),
}
scores = []
for tr, va in cv.split(X):
m = XGBRegressor(**params, random_state=seed, n_jobs=2).fit(X.iloc[tr], y.iloc[tr])
scores.append(np.sqrt(np.mean((y.iloc[va] - m.predict(X.iloc[va])) ** 2)))
return float(np.mean(scores))
optuna.logging.set_verbosity(optuna.logging.WARNING)
study = optuna.create_study(direction="minimize",
sampler=optuna.samplers.TPESampler(seed=seed))
study.optimize(objective, n_trials=n_trials)
return study
study = optuna_xgb(Xtr, ytr, h=12, n_trials=60)
print("best CV RMSE:", study.best_value)
print("best params:", study.best_params)
print("which hyperparameters mattered:",
optuna.importance.get_param_importances(study))
The parameter importances answer a practical question: which knobs are worth tuning? Often learning_rate and max_depth dominate, and the rest barely matter. Narrowing the search to the important ones makes later tuning faster and more stable.
| Search method | Strength | Weakness |
|---|---|---|
| grid | exhaustive, simple | explodes combinatorially |
| randomized | cheap, surprisingly effective | doesn't learn from results |
| Bayesian (Optuna) | sample-efficient; reports parameter importance | more machinery; still noisy with small samples |
Macro reality check: with ~240 monthly training rows and validation folds of 24 months, CV estimates are noisy. Tuning can easily chase noise. That's why the next block evaluates the whole tuning procedure out of sample.
64.5 Nested Evaluation: Tuning Inside the Rolling Origin (20 min)
To know whether tuning helps forecasts, evaluate the procedure "tune on the past, then forecast" in the rolling-origin harness. At each refit, the model tunes itself using only data available at that origin. That's nested evaluation: inner loop (tuning CV) inside outer loop (rolling origins).
Add a self-tuning estimator to forecasting/tuning.py:
class TunedXGB:
"""An XGBoost that tunes its own hyperparameters on its training data (time-series CV)."""
def __init__(self, h: int, n_iter: int = 20, seed: int = 0):
self.h, self.n_iter, self.seed = h, n_iter, seed
def fit(self, X, y):
search = random_search_xgb(X, y, self.h, self.n_iter, self.seed)
self.best_params_ = search.best_params_
self.model_ = search.best_estimator_
return self
def predict(self, X):
return self.model_.predict(X)
Because the gap depends on h, build one factory per horizon and call the harness horizon by horizon:
from forecasting.evaluate import rolling_origin
from forecasting.tuning import TunedXGB
tuned = pd.concat([
rolling_origin(X, df["cpi_yoy"], (lambda h=h: TunedXGB(h, n_iter=20)), [h],
"2012-01-01", "xgb_tuned", refit_every=12)
for h in [1, 3, 6, 12]], ignore_index=True)
tuned.to_parquet("data/forecasts/tuning_nested.parquet")
refit_every=12re-tunes once a year of origins, a realistic cadence and affordable compute: about 14 tunings × 20 settings × 4 folds per horizon.lambda h=h: ...freezes the loop variable (the same gotcha as Day 22).- Expect this to take several minutes. Run it in a terminal (
python -c "...") or let a notebook cell run while you read 64.6.
64.6 Mini-Project: Does Tuning Pay? (23 min)
- Load the Day 63 forecasts (
data/forecasts/univariate_forecasts.parquet) and addxgb_tuned. - Score with
score()and run Diebold–Mariano tests:xgb_tunedvsxgboost(default), andxgb_tunedvsnaive, at each horizon. - Inspect how the chosen hyperparameters changed over time. Store
best_params_per refit by adding a hook inTunedXGB.fitthat appends{origin_end: X.index.max(), **best_params_}to a list, then plotmax_depthandlearning_rateacross refits. - Write a short conclusion in
notebooks/day64.ipynb.
import pandas as pd
from forecasting.evaluate import diebold_mariano, score
prev = pd.read_parquet("data/forecasts/univariate_forecasts.parquet")
allres = pd.concat([prev, pd.read_parquet("data/forecasts/tuning_nested.parquet")])
print(score(allres).query("model in ['naive', 'ar_aic', 'xgboost', 'xgb_tuned']").round(3))
for h in [1, 3, 6, 12]:
print(diebold_mariano(allres, "xgb_tuned", "xgboost", h))
What students typically find, and why it matters
Gains from tuning are often small and not statistically significant for a univariate macro series of this length. Default settings that are already conservative (shallow trees, low learning rate, subsampling) capture most of the benefit. The chosen hyperparameters often jump around between refits, a sign that the CV objective is flat and noisy.
The professional conclusion isn't "tuning is useless". It's: use sensible conservative defaults; tune only the parameters that matter; and always evaluate the tuning procedure out of sample before claiming gains. With richer feature sets (Day 68), tuning tends to matter more.
Day 64 checklist
- I can distinguish parameters from hyperparameters, and explain why tuning on the test period invalidates it
- My time-series CV uses chronological folds with a gap equal to the horizon
- I can run randomized search and Optuna, and read CV variability and parameter importances
- I evaluated "tune-then-forecast" as a procedure, nested inside the rolling origin
- I reported whether tuning helped, with Diebold–Mariano evidence
git add forecasting notebooks && git commit -m "Day 64: leak-free tuning, randomized search, Optuna, nested evaluation" && git push