Interpretation, Sensitivity and Heterogeneity

79.1 Reading the Results as an Economist (20 min)

Interpret along four dimensions, in this order:

1. Sign: do increases in the overnight rate reduce housing starts? Economic theory (higher borrowing costs for builders and buyers) predicts a negative effect.
2. Timing: at which horizon is the effect largest? Monetary transmission to housing is generally described as operating with lags of several quarters: mortgage rates adjust, demand and pre-sales respond, and project starts follow. Does your profile peak at 12–24 months rather than 3?
3. Magnitude: express it per 25bp and in units of SAAR starts at today's level (78.4). Is it economically large? Compare it with the typical month-to-month volatility of starts.
4. Precision: do the intervals exclude zero? At which horizons? With ~20 years and a few tightening cycles, wide intervals are an expected, honest outcome. An imprecise estimate is a valid result.

Language discipline. Write "under the identifying assumptions of the design, a 25bp increase is estimated to reduce housing starts by about X% after 12 months (95% CI: …)". Don't write "rate hikes reduce housing starts by X%" as if it were a law of nature. The causal claim is conditional on assumptions that you will now stress-test.

79.2 Sensitivity to Unobserved Confounding (30 min)

Day 73's DAG flagged unobserved confounders: sentiment, supply constraints, population growth. The placebo test can reveal some confounding, but not prove its absence. Sensitivity analysis asks the complementary question:

How strong would an unobserved confounder have to be to change our conclusion?

DoubleML implements the omitted-variable-bias framework for causal ML of Chernozhukov, Cinelli, Newey, Sharma and Syrgkanis ("Long Story Short: Omitted Variable Bias in Causal Machine Learning"). It's parameterised by:

It reports bounds on θ for given strengths, and robustness values: the minimal strength (with cf_y = cf_d) at which the bound reaches zero.

Create causal/sensitivity.py:

"""DoubleML sensitivity analysis for the housing design (blocked splits)."""
import doubleml as dml
import numpy as np
from sklearn.base import clone

from causal.dml import blocked_splits


def fit_doubleml_plr(df, x_cols, learner, h, n_folds=5, n_rep=3):
    data = dml.DoubleMLData(df.reset_index(drop=True), y_col="y", d_cols="d", x_cols=x_cols)
    plr = dml.DoubleMLPLR(data, ml_l=clone(learner), ml_m=clone(learner),
                          score="partialling out")
    n = len(df)
    step = max(1, n // (n_folds * n_rep))
    plr.set_sample_splitting(all_smpls=[blocked_splits(n, n_folds, buffer=h + 6, offset=r * step)
                                        for r in range(n_rep)])
    plr.fit()
    return plr


def sensitivity(plr, cf_y=0.03, cf_d=0.03, rho=1.0):
    plr.sensitivity_analysis(cf_y=cf_y, cf_d=cf_d, rho=rho)
    return plr.sensitivity_summary


def benchmark(plr, control_names):
    """How strong is an OBSERVED control? Calibrates 'plausible' unobserved strengths."""
    return {c: plr.sensitivity_benchmark(benchmarking_set=[c]) for c in control_names}

Run it at the horizon where the main effect is largest (e.g. h = 12):

from causal.sensitivity import benchmark, fit_doubleml_plr, sensitivity

x_cols = [c for c in df.columns if c not in ("d", "y")]
plr = fit_doubleml_plr(df, x_cols, rf, h=12)
print(sensitivity(plr, cf_y=0.03, cf_d=0.03))
fig = plr.sensitivity_plot()                      # a contour plot (plotly)
fig.write_html("data/causal/sensitivity_h12.html")

bench = benchmark(plr, ["cpi_yoy", "unemp", "housing_growth_l1_6"])
for name, b in bench.items():
    print(name); print(b)

Reading it

• The contour plot shows how the lower bound of θ moves as confounding strength grows. The zero contour marks where the conclusion would flip.
• Benchmarking answers "an unobserved confounder as strong as inflation (or past housing growth) would imply…". If the robustness value is larger than the strength of your most important observed confounders, the result would survive an omitted variable as strong as anything we did control for. That's a powerful, communicable statement.
• If a confounder weaker than observed ones could overturn the result, the conclusion is fragile, and the memo must say so.

Caveats: the sensitivity bounds rest on the same model; DoubleML's sensitivity intervals use its default (independence-based) variance, so treat the bounds on the point estimate as the main output, and keep the HAC intervals (Day 76) for inference. Check the sensitivity_* API in your installed DoubleML version: names and outputs have evolved.

79.3 Asymmetry: Are Hikes Different from Cuts? (25 min)

The curriculum asks about rate hikes. Our θ averages over increases and decreases. Do hikes and cuts have symmetric effects? Housing could respond more sharply to tightening (credit constraints bind) than to easing.

The partially linear model handles several treatments: split the rate change into its positive and negative parts,

D⁺ = max(D, 0), D⁻ = min(D, 0), Y = θ⁺·D⁺ + θ⁻·D⁻ + g(X) + ε

and partial X out of Y, D⁺ and D⁻ separately. Then regress the Y-residuals on both treatment residuals jointly (multivariate Frisch–Waugh–Lovell, Day 74.2).

Create causal/heterogeneity.py:

"""PLR with several treatments: asymmetry (hikes vs cuts) and regime interactions."""
import numpy as np
import pandas as pd
import statsmodels.api as sm
from sklearn.base import clone

from causal.dml import blocked_splits, hac_lags_for


def fit_plr_multi_ts(y, D: pd.DataFrame, X, learner, h, n_folds=5, buffer=None):
    """Y = Σ θ_k D_k + g(X) + ε, cross-fitted with blocked folds; HAC inference."""
    y, X = np.asarray(y, float), np.asarray(X, float)
    Dm = D.to_numpy(float)
    n = len(y)
    splits = blocked_splits(n, n_folds, buffer if buffer is not None else h + 6)
    y_res = np.full(n, np.nan)
    D_res = np.full_like(Dm, np.nan)
    for tr, te in splits:
        y_res[te] = y[te] - clone(learner).fit(X[tr], y[tr]).predict(X[te])
        for k in range(Dm.shape[1]):
            D_res[te, k] = Dm[te, k] - clone(learner).fit(X[tr], Dm[tr, k]).predict(X[te])
    fit = sm.OLS(y_res, D_res).fit(cov_type="HAC", cov_kwds={"maxlags": hac_lags_for(n, h)})
    return pd.DataFrame({"theta": fit.params, "se": fit.bse,
                         "ci_lo": fit.conf_int()[:, 0], "ci_hi": fit.conf_int()[:, 1]},
                        index=D.columns), fit

from causal.heterogeneity import fit_plr_multi_ts

D = pd.DataFrame({"hike": df["d"].clip(lower=0), "cut": df["d"].clip(upper=0)}, index=df.index)
asym, fit = fit_plr_multi_ts(df["y"], D, df.drop(columns=["d", "y"]), rf, h=12)
print(asym.round(2))
# Test θ⁺ = θ⁻ (symmetry), with HAC covariance
print(fit.t_test("x1 - x2 = 0"))

(x1 and x2 are statsmodels' default names for the two columns when passing NumPy arrays.)

Interpretation:

• θ⁺ (hike): effect per 1pp of increase, the curriculum's quantity of interest.
• θ⁻ (cut): effect per 1pp of the negative change. Because D⁻ ≤ 0, a negative θ⁻ means cuts raise starts.
• The symmetry test asks whether the data can distinguish the two. With few cycles of each kind, it often can't. Report that honestly rather than over-reading a point difference.

Overlap matters doubly here: each treatment's residual variation comes from its own episodes. Check how many months have D⁺ > 0 and D⁻ < 0, and where (78.5's diagnostic, for each part).

79.4 Heterogeneity: Does the Effect Depend on the Economic Regime? (20 min)

Does tightening bite harder in some conditions, e.g. when inflation is high, or when rates are already elevated? Use an interaction treatment: with a pre-treatment regime indicator R (part of X),

Y = θ_low·D + δ·(D × R) + g(X) + ε, θ_high = θ_low + δ

regime = (df["cpi_yoy"] > 3).astype(float)               # pre-treatment: lagged inflation above 3%
D2 = pd.DataFrame({"d": df["d"], "d_x_high_inflation": df["d"] * regime}, index=df.index)
het, fit2 = fit_plr_multi_ts(df["y"], D2, df.drop(columns=["d", "y"]), rf, h=12)
print(het.round(2))
theta_high = het.loc["d", "theta"] + het.loc["d_x_high_inflation", "theta"]
print(f"θ (low-inflation regime):  {het.loc['d', 'theta']:+.2f}")
print(f"θ (high-inflation regime): {theta_high:+.2f}")
print(fit2.t_test("x1 + x2 = 0"))                         # is the high-regime effect non-zero?

Choose one or two regime hypotheses in advance, from economic reasoning, and record them in DEVIATIONS.md (they weren't in the Day 73 spec). Testing ten regimes and reporting the one that "works" is the specification-searching problem that pre-registration exists to prevent.

Optional extension: provincial heterogeneity

If you load the full CMHC/StatCan table through WAP (Day 50) with provincial starts, you can ask whether effects differ across provinces (e.g. higher-priced housing markets). That's a panel design, which raises new questions (common national policy, cross-province dependence). DoubleML offers clustered-data tools for such settings, a natural project after the course.

79.5 Naive vs DML: Telling the Confounding Story with Data (18 min)

One chart communicates the whole module. For each horizon, plot three estimates with intervals:

1. naive (no controls);
2. linear controls (OLS nuisances);
3. DML (ML nuisances, blocked cross-fitting, HAC).

from sklearn.linear_model import LinearRegression

from causal.dml import fit_plr_ts, hac_lags_for

rows = []
for h in [3, 6, 12, 18, 24]:
    dfh = estimation_frame(panel, spec, panel["housing_starts"], h=h, log_outcome=True)
    Xh = dfh.drop(columns=["d", "y"])
    nv = sm.OLS(dfh["y"], sm.add_constant(dfh["d"])).fit(cov_type="HAC",
                                                          cov_kwds={"maxlags": hac_lags_for(len(dfh), h)})
    rows.append({"h": h, "method": "naive", "theta": nv.params["d"],
                 "lo": nv.conf_int().loc["d", 0], "hi": nv.conf_int().loc["d", 1]})
    for name, L in [("linear controls", LinearRegression()), ("DML (random forest)", rf)]:
        r = fit_plr_ts(dfh["y"], dfh["d"], Xh, L, L, h=h)
        rows.append({"h": h, "method": name, "theta": r["theta"], "lo": r["ci_lo"], "hi": r["ci_hi"]})
comp = pd.DataFrame(rows)

fig, ax = plt.subplots(figsize=(9, 4.5))
for i, (m, g) in enumerate(comp.groupby("method", sort=False)):
    x = g["h"] + (i - 1) * 0.6
    ax.errorbar(x, g["theta"], yerr=[g["theta"] - g["lo"], g["hi"] - g["theta"]],
                fmt="o-", capsize=3, label=m)
ax.axhline(0, color="grey")
ax.set_xlabel("months after the rate change")
ax.set_ylabel("% change in housing starts per 1pp")
ax.set_title("Effect of overnight-rate increases on housing starts: naive vs adjusted")
ax.legend(); plt.tight_layout(); plt.savefig("data/causal/naive_vs_dml.png", dpi=120)

If the naive line sits above the adjusted ones, the gap is the confounding bias from booms that drive both hikes and housing, the Day 71 simulation now visible in real Canadian data. If linear and DML differ, the gap reflects non-linear confounding that only flexible controls capture.

79.6 Writing the Policy Memo (23 min)

Write causal/MEMO.md, two pages at most, for a reader at the Bank of Canada or a housing ministry who isn't a methods specialist:

# The effect of policy-rate increases on Canadian housing starts

## Bottom line (3 sentences)
Under the identifying assumptions below, a 25bp increase in the overnight rate is estimated to change
housing starts by X% (95% CI: [a, b]) after 12 months, with the effect building over …
The estimate is [robust / fragile] to …; an unobserved factor would need to be [stronger than inflation] to …
A naive comparison [understates / reverses] the effect because …

## Question and design
- Question; unit (month); treatment; outcome; horizons
- Identification: selection on observables — controls for the Bank's information set before each decision
  (inflation, labour market, activity, global rates, past policy, past housing), with publication lags
- Estimator: double/debiased ML (partially linear model), blocked cross-fitting, HAC standard errors
- Pre-registered on <date>; deviations: see DEVIATIONS.md

## Results
- Dynamic effect chart (naive vs adjusted) and table (per 25bp, % and units SAAR)
- Hikes vs cuts; regime heterogeneity (pre-specified only)

## How much should we trust this?
- Robustness checks (placebo, negative control, alternative treatment, pandemic, learners, buffers)
- Sensitivity: robustness value vs benchmarks
- Where the identifying variation comes from (which tightening cycles)

## Limitations
- Unobserved confounders (sentiment, supply constraints, population) — bounded, not eliminated
- Anticipation via forward guidance; few tightening cycles; national aggregate only
- SAAR monthly starts are noisy; results are averages over the sample period

## Data and reproducibility
Platform lineage: StatCan/CMHC → dlt → dbt (macro_monthly) → GX checks; code: causal/; run tag: …

Peer review: swap memos with a classmate. The reviewer checks that every claim in the bottom line is supported by a table or figure, that limitations are stated plainly, and that no result appears that wasn't pre-registered or listed as a deviation.

Day 79 checklist

• I interpreted sign, timing, magnitude (per 25bp, in units) and precision, with careful causal language
• I quantified sensitivity to unobserved confounding and benchmarked it against observed controls
• I estimated hike vs cut effects and tested symmetry, with overlap caveats
• I tested one or two pre-specified regime interactions, recorded as deviations
• A naive-vs-adjusted chart shows the confounding story in real data
• causal/MEMO.md communicates results, uncertainty and limitations, and passed peer review

git add causal notebooks && git commit -m "Day 79: interpretation, sensitivity, asymmetry, heterogeneity, memo" && git push