Production: Contracts, Exposures, Docs and the Cut-Over

Time	Block
0:00 – 0:23	40.1 Model contracts: promises about a mart's shape
0:23 – 0:40	40.2 Exposures: putting the dashboards on the lineage graph
0:40 – 0:48	Break
0:48 – 1:05	40.3 Documentation that scales: doc blocks and an overview page
1:05 – 1:20	40.4 Exporting marts for the notebook
1:20 – 1:28	Break
1:28 – 2:08	40.5 The cut-over: dev → prod with a regression test
2:08 – 2:30	40.6 Retiring Module 2 · the final pipeline · Module 4 recap

---

40.1 Model Contracts (23 min)

Tests check data after a model is built. A model contract is a promise about a model's shape (column names, data types, not-null constraints) that dbt checks when building. If the SQL would produce a different shape, the build fails before consumers ever see it. It's Day 25's schema contract idea, applied to your own marts.

Add to models/marts/rates/_rates__models.yml:

  - name: fct_yield_curve
    config:
      contract:
        enforced: true
    columns:
      - name: curve_point_key
        data_type: varchar
        constraints:
          - type: not_null
      - name: obs_date
        data_type: date
        constraints:
          - type: not_null
      - name: country
        data_type: varchar
      - name: tenor_years
        data_type: integer
      - name: yield_pct
        data_type: double
      - name: series_id
        data_type: varchar

cd macro_dbt
dbt build --select fct_yield_curve

Now break the promise: rename yield_pct to yld in the SQL and rebuild. dbt refuses, listing the mismatch. Restore it.

When enforced, the contract must list every column. That's easy for narrow facts like fct_yield_curve and macro_kpis, and laborious for wide one-big-tables, where tests on key columns are often a better trade-off.

Mechanism	Checks	When
dlt schema contract (Day 25)	raw data entering the warehouse	ingestion
dbt model contract	a mart's columns and types	build time
dbt data tests (Day 36)	values: keys, ranges, relationships	after build
Great Expectations (Days 41–50)	statistical profiles of data	after build / on schedule

If your adapter version reports that contracts or a constraint type aren't supported, keep data_types for documentation, drop the unsupported constraints, and rely on not_null tests instead.

Add a contract to macro_kpis yourself (columns pillar, kpi, kpi_code, value, as_of, signal). Watch for as_of: it mixes month, quarter and day dates, so check the type DuckDB infers with dbt show.

---

40.2 Exposures: Putting the Dashboards on the Lineage Graph (18 min)

Your lineage graph ends at the marts. But the reason the marts exist is downstream: the notebook dashboards, forecasting and causal work. Exposures declare those consumers so they appear on the graph.

Create models/exposures.yml:

version: 2

exposures:
  - name: rates_fx_dashboard
    label: "Rates & FX dashboard (notebook Module 02)"
    type: notebook
    maturity: medium
    description: >
      Policy rate path, Canadian and US yield curves, 10y–2y spread, USD/CAD vs the
      BoC–Fed differential, real policy rate.
    depends_on:
      - ref('fct_yield_curve')
      - ref('fct_open_economy_daily')
      - ref('macro_monthly')
    owner:
      name: Macro Data Course team
      email: course-team@example.com

  - name: open_economy_dashboard
    label: "Open Economy dashboard (notebook Module 06)"
    type: notebook
    maturity: medium
    description: "USD/CAD vs WTI and rate spreads, rolling correlations and betas, FX volatility."
    depends_on:
      - ref('fct_open_economy_daily')
      - ref('macro_monthly')
    owner:
      name: Macro Data Course team

  - name: macro_cockpit
    label: "Macro cockpit KPI grid (notebook Module 07)"
    type: dashboard
    maturity: low
    depends_on:
      - ref('macro_kpis')
    owner:
      name: Macro Data Course team

Now you can ask impact questions before changing anything:

dbt ls --select +exposure:open_economy_dashboard          # everything the dashboard depends on
dbt ls --select stg_statcan__cpi+ --resource-type exposure  # which dashboards a CPI change affects
dbt build --select +exposure:macro_cockpit                 # rebuild just what the cockpit needs

Exposures appear as orange nodes at the right edge of the docs lineage graph.

---

40.3 Documentation That Scales (18 min)

Doc blocks: write a definition once, use it everywhere

The same column (usd_cad, ca_us_2y_spread_bp, …) appears in several models. Instead of copying descriptions, define them once in models/docs.md:

{% docs usd_cad %}
Canadian dollars per US dollar (the inverse of the BoC's FXCADUSD series).
**Higher = weaker CAD.** Daily values are BoC closing rates, forward-filled at most
`max_ffill_days` calendar days across holidays.
{% enddocs %}

{% docs ca_us_2y_spread_bp %}
Canada 2-year benchmark yield minus US 2-year Treasury constant-maturity yield, in basis
points. Positive = Canadian short-term yields above US ones.
{% enddocs %}

{% docs rolling_window_note %}
Rolling statistics use `rolling_window_days` business days (default 60) and are NULL
until the window is complete. Based on daily **changes**, not levels (see analysis
`levels_vs_changes`).
{% enddocs %}

Reference them in YAML:

      - name: usd_cad
        description: '{{ doc("usd_cad") }}'
      - name: corr_usdcad_2y_spread
        description: 'Rolling correlation of USD/CAD log returns with changes in the CA–US 2y spread. {{ doc("rolling_window_note") }}'

An overview page

The docs site's home page comes from a special doc block. Add to models/docs.md:

{% docs __overview__ %}
# Canadian Macro Data Mart

Built by the MSc Economics *Applied Decision Intelligence* course.

**Pipeline:** dlt (`ingest.py`) → `raw_dlt` → dbt staging → intermediate → marts.

**Start here:**
- `macro_monthly`: monthly one-big-table (rates, FX, CPI, real rates, open economy)
- `macro_quarterly`: quarterly GDP growth, deflator, recession flags
- `fct_open_economy_daily`: daily spreads, returns, volatility, rolling correlations
- `fct_yield_curve`: Canadian and US yield curves (long format)
- `macro_kpis`: latest KPI values with signals

Conventions: see `CONVENTIONS.md` in the repository.
{% enddocs %}

dbt docs generate && dbt docs serve --port 8080

---

40.4 Exporting Marts for the Notebook (15 min)

The dashboard notebook reads Parquet files (Day 20). Let dbt write them automatically after each mart is built in prod, using a post-hook: SQL that dbt runs right after a model.

Create macros/export_parquet.sql:

{#
  Post-hook: in prod, copy the just-built mart to <export_dir>/<model>.parquet.
  In other targets it does nothing (returns a harmless statement).
#}
{% macro export_parquet() %}
    {% if target.name == 'prod' and model.config.materialized in ['table', 'incremental'] %}
        copy {{ this }} to '{{ var("export_dir") }}/{{ this.name }}.parquet' (format parquet)
    {% else %}
        select 1
    {% endif %}
{% endmacro %}

In dbt_project.yml:

vars:
  export_dir: "../data/marts"

models:
  macro_dbt:
    marts:
      +materialized: table
      +schema: marts
      +post-hook:
        - "{{ export_parquet() }}"

• {{ this }} is the model's own table; {{ this.name }} its name.
• model.config.materialized lets the hook skip views and ephemeral models.
• DuckDB's COPY … TO needs the folder to exist: the pipeline script creates data/marts/ first (40.6).

---

40.5 The Cut-Over: dev → prod with a Regression Test (40 min)

Until now, production dashboards read Module 2's hand-built objects. Today dbt takes over the clean schema names (staging, intermediate, marts, seeds). We follow the Day 30 recipe: snapshot → change → compare.

Step 0 — Freeze

Make sure nobody else writes to macro.duckdb during the cut-over: close all DuckDB CLI sessions and notebooks.

Step 1 — Snapshot the current production marts

cd /workspaces/macro-data-course
mkdir -p data/migration
duckdb macro.duckdb -readonly -c "
COPY marts.macro_monthly   TO 'data/migration/m2_macro_monthly.parquet'   (FORMAT parquet);
COPY marts.macro_quarterly TO 'data/migration/m2_macro_quarterly.parquet' (FORMAT parquet);
COPY marts.policy_cycles   TO 'data/migration/m2_policy_cycles.parquet'   (FORMAT parquet);"

Step 2 — Check what prod would build, without building it

cd macro_dbt
dbt ls --target prod --resource-type model --output json --output-keys "name schema" | head -40

Every model should show a clean schema (staging, intermediate, marts), with no dev_ prefixes. That's the generate_schema_name macro from Day 32 doing its job.

Step 3 — Build prod

mkdir -p ../data/marts
dbt build --target prod

dbt replaces same-named Module 2 objects (for example staging.stg_statcan__cpi, marts.macro_monthly) and creates the new ones. The post-hook writes Parquet exports to data/marts/.

Step 4 — Regression test against the snapshot

Create analyses/cutover_regression.sql:

{% set checks = [
    {"name": "macro_monthly", "key": "month",
     "cols": ["overnight_rate", "fed_funds", "headline_yoy", "core_ex_fe_yoy", "real_rate",
              "spread_10_2_bp", "usd_cad"]},
    {"name": "macro_quarterly", "key": "quarter",
     "cols": ["real_qoq_ann", "real_yoy", "overnight_rate_avg", "headline_yoy_avg"]},
    {"name": "policy_cycles", "key": "cycle_id",
     "cols": ["start_date", "end_date", "n_moves", "total_bp", "rate_before", "rate_after"]},
] %}

{% for c in checks %}
select
    '{{ c.name }}'  as mart,
    (select count(*) from (
        select {{ c.key }}{% for col in c.cols %}, round(cast({{ col }} as double), 6){% endfor %}
        from 'data/migration/m2_{{ c.name }}.parquet'
        except
        select {{ c.key }}{% for col in c.cols %}, round(cast({{ col }} as double), 6){% endfor %}
        from {{ ref(c.name) }}
    )) as rows_changed_or_missing
{{ 'union all' if not loop.last }}
{% endfor %}

Run it from the repository root (the Parquet paths are relative to it):

dbt compile --target prod --select cutover_regression
cd .. && duckdb macro.duckdb -readonly -c "$(cat macro_dbt/target/compiled/macro_dbt/analyses/cutover_regression.sql)" && cd macro_dbt

(round(cast(… as double), 6) makes dates and numbers comparable and ignores floating-point noise. Adjust the column lists if any Module 2 names differ from yours.)

Interpreting the result (the Day 30 rules):

Pattern	Meaning	Action
0 everywhere	clean cut-over	proceed
differences only in the most recent months	new data since the snapshot	note and proceed
differences in old months	revisions (lookback), or a logic difference	inspect the rows; fix or document before proceeding
many rows missing	a filter or join changed (e.g. a leftover dev filter)	fix, rebuild prod, re-run

For any non-zero count, list the actual rows with the same except query (without count(*)) and explain each difference in data/migration/NOTES.md.

Step 5 — Remove objects dbt doesn't manage

Module 2 left objects that no dbt model replaced (e.g. staging.stg_macro__observations, old intermediate.* names). An operational macro can compare the database with dbt's graph of known nodes. Create macros/ops/drop_orphaned_relations.sql:

{% macro drop_orphaned_relations(schemas=['staging', 'intermediate', 'marts', 'seeds'],
                                 dry_run=true) %}
    {% if target.name != 'prod' %}
        {{ exceptions.raise_compiler_error("Run with --target prod") }}
    {% endif %}

    {# Every relation dbt manages in this target #}
    {% set managed = [] %}
    {% for node in graph.nodes.values()
           | selectattr("resource_type", "in", ["model", "seed", "snapshot"]) %}
        {% do managed.append((node.schema ~ "." ~ node.alias) | lower) %}
    {% endfor %}

    {% set query %}
        select table_schema, table_name, table_type
        from information_schema.tables
        where table_schema in ({% for s in schemas %}'{{ s }}'{{ ', ' if not loop.last }}{% endfor %})
        order by 1, 2
    {% endset %}

    {% for row in run_query(query) %}
        {% set fqn = (row[0] ~ "." ~ row[1]) | lower %}
        {% if fqn not in managed %}
            {% set kind = "view" if row[2] == "VIEW" else "table" %}
            {% if dry_run %}
                {{ log("would drop " ~ kind ~ " " ~ fqn, info=True) }}
            {% else %}
                {% do run_query("drop " ~ kind ~ " if exists " ~ fqn) %}
                {{ log("dropped " ~ kind ~ " " ~ fqn, info=True) }}
            {% endif %}
        {% endif %}
    {% endfor %}
{% endmacro %}

dbt run-operation drop_orphaned_relations --target prod                        # review the list
dbt run-operation drop_orphaned_relations --target prod --args '{dry_run: false}'

graph is dbt's in-memory map of the whole project, available to macros. Reading it lets operational macros reason about what the project defines, the same information the lineage graph shows.

Finally, drop the retired Module 2 raw tables (keep raw_dlt, of course):

-- in the DuckDB CLI, after confirming nothing reads them
drop table if exists raw.observations;
drop table if exists raw.statcan_cpi;
drop table if exists raw.statcan_cpi_full;
drop table if exists raw.statcan_gdp_full;
drop table if exists raw.dim_series;
drop table if exists raw.boc_decisions;      -- now a dbt seed
drop schema if exists scratch cascade;       -- Day 14 experiments

---

40.6 Retiring Module 2 · the Final Pipeline · Module 4 Recap (23 min)

Move legacy code aside

Keep it in the repository for reference, clearly marked as retired:

cd /workspaces/macro-data-course
mkdir -p legacy
git mv sql legacy/sql
git mv load_warehouse.py run_tests.py legacy/
git mv run_extract.py macro_extract legacy/ 2>/dev/null || true
echo "Retired at the Day 40 dbt cut-over. See macro_dbt/ and ingest.py." > legacy/README.md

The final pipeline.sh

#!/usr/bin/env bash
# pipeline.sh — ingest (dlt) → transform & test (dbt, prod) → export → docs
set -e
source .venv/bin/activate

python ingest.py || [ $? -eq 2 ]            # tolerate partial ingestion failures (Day 30)

mkdir -p data/marts
cd macro_dbt
dbt deps --quiet
dbt source freshness --target prod || echo "⚠ source freshness warnings — see above"
dbt build --target prod                      # seeds, models, snapshots, tests, exports
dbt docs generate --target prod
cd ..

echo "✓ Pipeline complete: marts in macro.duckdb and data/marts/*.parquet"

./pipeline.sh
ls -lh data/marts/

Point the notebook at the mart

In the dashboard notebook, the data-fetching cells (Cells 2–5) are replaced by:

import pandas as pd

monthly   = pd.read_parquet("data/marts/macro_monthly.parquet")
quarterly = pd.read_parquet("data/marts/macro_quarterly.parquet")
daily     = pd.read_parquet("data/marts/fct_open_economy_daily.parquet")
curve     = pd.read_parquet("data/marts/fct_yield_curve.parquet")
kpis      = pd.read_parquet("data/marts/macro_kpis.parquet")

A quick check that the open-economy data tells a coherent story, in the notebook's style:

import plotly.graph_objects as go
from plotly.subplots import make_subplots

d = daily[daily["date_day"] >= "2018-01-01"]
fig = make_subplots(specs=[[{"secondary_y": True}]])
fig.add_trace(go.Scatter(x=d["date_day"], y=d["usd_cad"], name="USD/CAD"), secondary_y=False)
fig.add_trace(go.Scatter(x=d["date_day"], y=d["ca_us_2y_spread_bp"],
                         name="CA–US 2y spread (bp)"), secondary_y=True)
fig.update_layout(title="USD/CAD vs the Canada–US 2-year spread")
fig.show()

Every number on every chart now comes from tested, documented, version-controlled models, and the notebook contains only presentation.

Module 4 recap (Days 31–40)

Days	You built
31–33	a dbt project over DuckDB: sources, ref(), materialisations, seeds, docs, Jinja
34	modular layers: one staging model per source table, concept-based intermediates, facts & dimensions
35	DRY macros: growth_rates, pct_change, monthly_value, overlaps_recession, run_query from seeds
36	tests: generic, dbt_utils, custom, singular, unit tests; source freshness
37	incremental fct_observations, CPI snapshot, generated staging, operational macros, conventions
38	mart design from consumer questions; yield-curve fact; aligned business-day market panel
39	open-economy daily fact (spreads, returns, vol, correlations, betas); monthly/quarterly OBTs; KPIs
40	contracts, exposures, doc blocks, Parquet exports, a regression-tested cut-over to prod

The architecture after Module 4

BoC · FRED · StatCan · (World Bank)
          │  dlt (ingest.py): incremental, contracts, resilient
          ▼
   raw_dlt.*  ──►  dbt: staging → intermediate → marts   (prod, tested, documented)
                              │
                              ├── macro.duckdb: marts.*
                              └── data/marts/*.parquet ──► notebook dashboards (exposures)

Looking ahead: Module 5 — Great Expectations (Days 41–50)

dbt tests check rules you wrote. Great Expectations adds expectation suites built from data profiles, human-readable Data Docs, and validation of incoming StatCan data, such as unemployment between 0 and 100% and dates continuous without gaps, at the boundary where data enters the warehouse. Your marts, contracts and exposures from today become the assets GE protects.

Day 40 checklist

• Narrow marts have enforced contracts; wide ones have tests on key columns
• Exposures put the notebook dashboards on the lineage graph, and I can select by exposure
• Shared column definitions live in doc blocks; the docs site has an overview page
• Marts are exported to Parquet by a post-hook in prod
• The cut-over was regression-tested against a snapshot, with differences explained in writing
• Orphaned objects and retired raw tables were removed with a guarded macro and reviewed SQL
• ./pipeline.sh runs ingestion, dbt in prod, exports and docs end to end

git add .
git commit -m "Day 40: contracts, exposures, docs, exports, prod cut-over — Module 4 complete"
git push