Dashboards, Production Readiness and the End of Part II

Time	Block
0:00 – 0:30	60.1 The last mile: the notebook's dashboards as code
0:30 – 0:50	60.2 daily_refresh v4: the complete pipeline
0:50 – 0:58	Break
0:58 – 1:20	60.3 Dev and prod environments: testing the whole platform safely
1:20 – 1:40	60.4 Backups and restore drills
1:40 – 1:48	Break
1:48 – 2:15	60.5 The production-readiness review
2:15 – 2:30	60.6 Part II recap and a look at Part III

---

60.1 The Last Mile: The Notebook's Dashboards as Code (30 min)

The course notebook draws its dashboards from data it fetches itself (Cells 2–5). Since Day 40, those numbers come from tested marts in data/marts/*.parquet. The final step is to make building the dashboards part of the pipeline: a script that reads the marts and writes standalone HTML files, exactly as the notebook's fig.write_html(...) calls did.

Create dashboards/build_dashboards.py (runs in .venv; install plotly there if needed):

"""Build the macro dashboards (HTML) from the dbt marts. Called by the pipeline."""
from datetime import datetime, timezone
from pathlib import Path

import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots

REPO = Path(__file__).resolve().parents[1]
MARTS = REPO / "data" / "marts"
SITE = REPO / "site"

# The notebook's navy palette
C = {"primary": "#4A90D9", "positive": "#3DA882", "warning": "#D4924A",
     "negative": "#C95555", "background": "#0D1B3E", "text": "#C8D8F0", "muted": "#A0B4CC"}
LAYOUT = dict(plot_bgcolor=C["background"], paper_bgcolor=C["background"],
              font=dict(color=C["text"], family="Calibri, Segoe UI, sans-serif"))


def banner(as_of: str, quality: pd.DataFrame | None) -> str:
    """Header shown on every dashboard: data date, build time, quality status."""
    built = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
    q = ""
    if quality is not None and not quality.empty:
        crit = quality[quality["checkpoint"] == "critical"]
        ok = not crit.empty and int(crit["failures"].iloc[0]) == 0
        q = ("✅ critical quality checks passing" if ok
             else "⚠️ critical quality checks FAILING — interpret with caution")
    return f"Data as of <b>{as_of}</b> · built {built} · {q}"


def rates_fx(monthly: pd.DataFrame, daily: pd.DataFrame, quality) -> Path:
    fig = make_subplots(rows=2, cols=1, vertical_spacing=0.12,
                        subplot_titles=["Policy rates and real rate (%)",
                                        "USD/CAD vs Canada–US 2-year spread"],
                        specs=[[{}], [{"secondary_y": True}]])
    m = monthly[monthly["month"] >= "2015-01-01"]
    for col, name, colour in [("overnight_rate", "BoC overnight", C["primary"]),
                              ("fed_funds", "Fed funds", C["warning"]),
                              ("real_rate", "Real policy rate", C["positive"])]:
        fig.add_trace(go.Scatter(x=m["month"], y=m[col], name=name,
                                 line=dict(color=colour, width=2)), row=1, col=1)
    d = daily[daily["date_day"] >= "2018-01-01"]
    fig.add_trace(go.Scatter(x=d["date_day"], y=d["usd_cad"], name="USD/CAD",
                             line=dict(color=C["primary"], width=1.5)), row=2, col=1)
    fig.add_trace(go.Scatter(x=d["date_day"], y=d["ca_us_2y_spread_bp"], name="CA–US 2y (bp)",
                             line=dict(color=C["negative"], width=1.5)),
                  row=2, col=1, secondary_y=True)
    as_of = str(m["month"].max())[:7]
    fig.update_layout(**LAYOUT, height=900,
                      title=dict(text=f"<b>Rates & FX</b><br><sup>{banner(as_of, quality)}</sup>"))
    out = SITE / "rates_fx.html"
    fig.write_html(out)
    return out


def labour(lfs: pd.DataFrame, quality) -> Path:
    latest = lfs["month"].max()
    snap = lfs[lfs["month"] == latest].sort_values("unemployment_rate")
    colours = [C["negative"] if g == "CA" else C["primary"] for g in snap["geo_code"]]
    fig = go.Figure(go.Bar(x=snap["unemployment_rate"], y=snap["geo_code"], orientation="h",
                           marker_color=colours, text=snap["unemployment_rate"].round(1),
                           textposition="outside"))
    fig.update_layout(**LAYOUT, height=600,
                      title=dict(text=f"<b>Unemployment rate by province (SA, 15+)</b>"
                                      f"<br><sup>{banner(str(latest)[:7], quality)}</sup>"),
                      xaxis_title="%")
    out = SITE / "labour.html"
    fig.write_html(out)
    return out


def index(pages: list[Path]) -> Path:
    links = "\n".join(f'<li><a href="{p.name}">{p.stem.replace("_", " ").title()}</a></li>'
                      for p in pages)
    html = (f"<html><body style='background:{C['background']};color:{C['text']};"
            f"font-family:Calibri,sans-serif'><h1>Canadian Macro Dashboards</h1>"
            f"<ul>{links}</ul></body></html>")
    out = SITE / "index.html"
    out.write_text(html)
    return out


def build() -> list[str]:
    SITE.mkdir(exist_ok=True)
    monthly = pd.read_parquet(MARTS / "macro_monthly.parquet")
    daily = pd.read_parquet(MARTS / "fct_open_economy_daily.parquet")
    lfs = pd.read_parquet(MARTS / "lfs_provincial_monthly.parquet")
    quality_path = MARTS / "quality_latest.parquet"
    quality = pd.read_parquet(quality_path) if quality_path.exists() else None
    pages = [rates_fx(monthly, daily, quality), labour(lfs, quality)]
    pages.append(index(pages))
    return [str(p) for p in pages]


if __name__ == "__main__":
    print("\n".join(build()))

python dashboards/build_dashboards.py
python -m http.server 8090 --directory site        # Codespaces forwards port 8090

Design points:

• Every dashboard says when the data is from, when it was built, and whether quality checks pass. A chart without its "as of" date and trust status is an invitation to misread it.
• The script only reads Parquet: no database connection, no lock needed.
• site/ is generated output: add it to .gitignore.

Exercise: port the remaining notebook modules

Port the notebook's other modules into build_dashboards.py, one function each, reading only marts. That covers Module 01 (macro overview), Module 03 (fiscal, from the fiscal seeds if you added them), Module 05 (business cycle) and Module 07 (cockpit, from macro_kpis). Where the notebook computed something itself, check whether a mart already provides it, and if not, add it to dbt rather than to the dashboard script. Dashboards present; dbt computes.

---

60.2 daily_refresh v4: The Complete Pipeline (20 min)

"""Daily macro refresh — v4 (Day 60): the complete orchestrated pipeline."""
from prefect import flow, get_run_logger, task

from dashboards.build_dashboards import build as build_dashboards
from orchestration.flows.statcan_release import statcan_release
# extract_load_flow, transform_flow, quality_flow, outcome(), hooks, history: Days 53–58


@task(retries=1, retry_delay_seconds=30)
def build_dashboards_task() -> list[str]:
    pages = build_dashboards()
    get_run_logger().info(f"dashboards built: {pages}")
    return pages


@flow(name="publish")
def publish_flow(skip_briefing: bool = False) -> dict:
    states = {"dashboards": build_dashboards_task(return_state=True)}
    if not skip_briefing:
        states["briefing"] = publish_briefing(return_state=True)           # Day 53 artifact
    return states


@flow(name="daily-refresh",
      on_failure=[alert_on_failure], on_crashed=[alert_on_failure],
      on_completion=[alert_on_warnings])
def daily_refresh(target: str = "prod", force_transform: bool = False,
                  sources: list[str] = ("boc", "fred", "statcan")):
    states = {}
    states.update(extract_load_flow(sources))                 # 1. dlt, per source
    states["statcan_release"] = statcan_release(target=target, return_state=True)  # 2. WAP (idempotent)
    states.update(transform_flow(target=target, force=force_transform))  # 3. targeted dbt
    if outcome(states.get("dbt_build")) != "failed":
        states.update(quality_flow())                          # 4. GX warehouse
        if outcome(states.get("gx_warehouse")) != "failed":
            states.update(publish_flow())                      # 5. dashboards + briefing
    ...                                                        # 6. policy → final state, history, artifacts

Points to note:

• WAP runs inside the daily refresh too, as a safety net (59.4). On most days it finds nothing and finishes in seconds.
• Dashboards are published only if critical quality checks pass. Otherwise yesterday's dashboards stay online, and their banner (built yesterday) still reflects yesterday's status. The policy from Day 53 extends naturally: no publication on critical failure.
• Update STATE_POLICY.md with the statcan_release and publish rows.

---

60.3 Dev and Prod Environments: Testing the Whole Platform Safely (23 min)

You've tested pieces in isolation (dev schemas in dbt, practice pipelines in dlt, throwaway GX assets). The whole platform deserves a dev environment: same code, different warehouse file, so you can run end-to-end experiments without risking production.

Add to orchestration/settings.py:

ENVIRONMENTS = {
    "prod": {"db": REPO / "macro.duckdb", "dbt_target": "prod"},
    "dev":  {"db": REPO / "dev_macro.duckdb", "dbt_target": "dev"},
}


def activate(environment: str):
    """Point every tool at the chosen environment (in-process tools and subprocesses)."""
    db = ENVIRONMENTS[environment]["db"]
    updates = {
        "MACRO_DUCKDB_PATH": str(db),                      # dbt profiles, WAP, fingerprints
        "MACRO_DUCKDB_URL": f"duckdb:///{db}",             # Great Expectations
        "DESTINATION__DUCKDB__CREDENTIALS": str(db),       # dlt (Day 23)
    }
    os.environ.update(updates)
    ENV.update(updates)
    return ENVIRONMENTS[environment]["dbt_target"]

Give daily_refresh a parameter environment: str = "prod", call target = activate(environment) first, and pass target downstream. Variables (fingerprints) are already keyed by target (58.6).

Creating the dev copy

# with no pipeline running (the copy must not happen during a write)
cp macro.duckdb dev_macro.duckdb

Then run the whole pipeline against dev:

from orchestration.flows.daily_refresh import daily_refresh
daily_refresh(environment="dev", force_transform=True)

GX assets point at ${MACRO_DUCKDB_URL} (Day 43), so they follow the environment automatically. The dbt dev target builds into dev_* schemas inside dev_macro.duckdb. Dev is fully separated from prod, at the file level and the schema level.

Add a dev-refresh deployment (no schedule) to prefect.yaml with parameters: {environment: dev}, so anyone can test a change end to end from the UI.

---

60.4 Backups and Restore Drills (20 min)

Most of the warehouse can be rebuilt from the APIs plus code. Some of it cannot:

Can be rebuilt	Cannot be rebuilt (must be backed up)
observations (re-extract from APIs)	dlt vintages of revised data (Day 27): past values the APIs no longer serve
staging, intermediate, marts (dbt)	dbt snapshots of CPI revisions (Day 37)
full StatCan tables (re-download)	quarantined releases (Day 50): evidence of what StatCan sent
	GX and run history (data/quality, data/orchestration)
	seeds and code: in Git, already backed up

A backup flow

Create orchestration/flows/backup.py:

"""Weekly backup of the warehouse and of the history files."""
import shutil
from datetime import datetime, timezone

import duckdb
from prefect import flow, get_run_logger, task

from orchestration.settings import ENV, REPO
from orchestration.tasks.locks import duckdb_access

BACKUP_DIR = REPO / "backups"
KEEP = 8


@task(timeout_seconds=3600)
def export_database() -> str:
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S")
    target = BACKUP_DIR / f"macro_{stamp}"
    with duckdb_access(write=False):
        with duckdb.connect(ENV["MACRO_DUCKDB_PATH"], read_only=True) as con:
            con.execute(f"EXPORT DATABASE '{target}' (FORMAT parquet)")
    for folder in ("data/quality", "data/orchestration"):
        src = REPO / folder
        if src.exists():
            shutil.copytree(src, target / "history" / src.name)
    return str(target)


@task
def prune_old_backups():
    backups = sorted(BACKUP_DIR.glob("macro_*"))
    for old in backups[:-KEEP]:
        shutil.rmtree(old)
        get_run_logger().info(f"pruned {old.name}")


@flow(name="backup")
def backup():
    BACKUP_DIR.mkdir(exist_ok=True)
    path = export_database()
    prune_old_backups()
    get_run_logger().info(f"backup written to {path}")

EXPORT DATABASE writes every schema and table as Parquet files plus the SQL needed to recreate them. Schedule it weekly in prefect.yaml (e.g. Sundays 06:00 ET, before the weekly rebuild).

A backup inside the same Codespace protects against mistakes (a bad DROP), not against losing the machine. In production, copy backups to separate storage (Day 56's DEPLOYMENT.md).

The restore drill

A backup you haven't restored is a hope, the same principle as every drill in this course:

duckdb restored.duckdb -c "IMPORT DATABASE 'backups/macro_<stamp>'"
duckdb restored.duckdb -readonly -c "
  select table_schema, count(*) as tables from information_schema.tables group by 1 order by 1"

Compare row counts of key tables between macro.duckdb and restored.duckdb (the Day 30 regression technique), time the restore, and record both in PRODUCTION_READINESS.md. Then point a dev run at the restored file (cp restored.duckdb dev_macro.duckdb) and confirm the pipeline runs end to end on it.

---

60.5 The Production-Readiness Review (28 min)

Professional teams don't declare a system "production-ready" by feel. They walk through a checklist with evidence. Create PRODUCTION_READINESS.md and score each item ✅ done (with evidence) · 🟡 partial · ❌ missing. Link evidence: files, run names, artifacts, drill logs.

# Production readiness review — Canadian macro data platform

## 1. Correctness
- [ ] Every mart has documented grain, tests on keys, and contracts where narrow (Days 36, 40)
- [ ] Boundary audits on all full StatCan tables; 0–100% and continuity contracts (Days 48–50)
- [ ] Warehouse GX checkpoints with critical/warning split (Days 44–47)
- [ ] Regression tests performed at every migration (Days 30, 40)
- [ ] Cross-source consistency checks (CPI vector vs table; LFS vector vs table)

## 2. Reliability
- [ ] Retries only for transient failures; documented exit codes (Day 54)
- [ ] Timeouts on every step; retry budget fits the deadline (Day 54)
- [ ] Readers–writer lock prevents DuckDB collisions across all runs (Day 59)
- [ ] Idempotent steps: re-running any flow is safe (markers after success)
- [ ] Failure drills: ingestion, quarantine, dbt, GX, overlapping runs (Days 53–59)

## 3. Observability
- [ ] Every run visible in Prefect with per-step state, logs and artifacts
- [ ] Run, quality and ingestion history queryable as data (Days 47, 57)
- [ ] Pipeline-health and data-quality KPIs in the cockpit (Days 50, 57)
- [ ] Watchdog for silent stoppage (Day 57)

## 4. Timeliness
- [ ] Schedules aligned with BoC and StatCan publication times, America/Toronto (Day 56)
- [ ] Release watcher reacts within ~10 minutes of StatCan releases (Day 59)
- [ ] Freshness tests per series (Days 36, 49)

## 5. Security
- [ ] No secrets in Git (`git log -p | grep -i key` reviewed); Secret blocks / Codespaces secrets
- [ ] Read-only connections for every reader
- [ ] Data Docs and backups treated as data (not committed)

## 6. Recoverability
- [ ] Weekly backups including vintages, snapshots, quarantine and history (Day 60)
- [ ] Restore drill performed and timed (Day 60)
- [ ] Full rebuild from APIs drill performed and timed (`--full-refresh`)

## 7. Operability & documentation
- [ ] RUNBOOK.md covers every alert type
- [ ] DATA_QUALITY_PLAN.md, STATE_POLICY.md, EXIT_CODES.md, DEPLOYMENT.md current
- [ ] A colleague can operate the platform from the docs alone (tested, Day 47 method)
- [ ] Dev environment for end-to-end testing (Day 60)

## Known gaps and risks
| Gap | Risk | Mitigation / plan |
|---|---|---|
| Runs only while the Codespace is active | missed schedules | DEPLOYMENT.md: always-on host or managed service |
| Watchdog runs on the same machine | can't detect machine loss | external monitoring in production |
| … | | |

The review exercise

Work in pairs: each student reviews their partner's platform against the checklist, asking for evidence for every ✅. Unsupported ticks become 🟡. Agree on the three most important gaps, and write a short remediation plan for each. This is exactly how real teams prepare a system for launch.

---

60.6 Part II Recap and a Look at Part III (15 min)

What you built in Parts I and II (Days 1–60)

Module	Days	Layer	Tools
1	1–10	foundations: Python, pandas, REST APIs	Python, pandas, requests
2	11–20	analytical SQL and a layered warehouse	DuckDB
3	21–30	declarative, incremental, resilient ingestion	dlt
4	31–40	modular, tested transformations and the macro mart	dbt
5	41–50	data contracts, validation, Write–Audit–Publish	Great Expectations
6	51–60	orchestration: retries, caching, schedules, events, operations	Prefect

The platform, in one sentence

Every weekday, within minutes of the Bank of Canada and Statistics Canada publishing, a self-monitoring pipeline extracts the new data, quarantines anything that breaks its contracts, rebuilds only the affected models, validates the results, and publishes dashboards that say how fresh and how trustworthy every number is.

How the pieces map to the curriculum's capstone table

Capstone component	Technology	Where you built it
Extract & load	Python, dlt, DuckDB	Modules 1–3; per-source tasks (Day 58)
Transform & test	dbt, Great Expectations	Modules 4–5; targeted builds and gating (Days 58–60)
Model & forecast	XGBoost, Chronos, DoubleML	Part III, built on these marts
Orchestration & visualisation	Prefect, Plotly	Module 6; dashboards (Day 60)

Looking ahead: Part III — Advanced Analytics (Days 61–80)

Everything you've built becomes the foundation for modelling:

• Module 7 — Forecasting with XGBoost & Chronos (Days 61–70): features engineered from macro_monthly and macro_quarterly; forecasting Canadian CPI and real GDP growth; SHAP explanations; zero-shot forecasting with Chronos. The vintages and snapshots you preserved (Days 27, 37, 60) make honest, real-time forecast evaluation possible: training only on data as it was known at each point in time.
• Module 8 — Causal inference with DoubleML (Days 71–80): estimating the effect of Bank of Canada rate hikes on housing starts, controlling for the business cycle. It uses the policy-cycle mart (Day 17), the decision log seed and the open-economy controls (Day 39).
• The final capstone: forecasts and causal estimates become new marts, produced by new Prefect tasks in the same pipeline, and shown on the same dashboards, validated by the same quality layer.

Day 60 checklist

• The notebook's dashboards are built by code from marts, with "as of" dates and quality banners
• daily_refresh v4 links dlt → WAP → dbt → GX → dashboards, publishing only when quality allows
• A dev environment runs the whole platform safely against a copy of the warehouse
• Weekly backups include everything that can't be rebuilt; a restore drill succeeded and was timed
• PRODUCTION_READINESS.md is completed with evidence, peer-reviewed, and lists remediation plans
• I can explain how Parts I–II support forecasting and causal analysis in Part III

git add . && git commit -m "Day 60: dashboards, daily_refresh v4, dev/prod, backups, readiness review — Part II complete"
git push