Module 5 · Days 41–50 · Part II

Quality Assurance with Great Expectations

Contract-driven data engineering and anomaly detection.

Start Day 41

0/10 days completed

0/10 videos published

What you'll learn

1
Days 41–43
Data quality principles, initializing contexts, and connecting to DuckDB sources.
By the end: Students can reason about data quality systematically (dimensions, contracts, severities, ownership). They write a **data quality plan** for the macro warehouse, set up a **Great Expectations (GX) project**, connect it to DuckDB tables at every layer (raw_dlt, staging, marts), and run first validations against real Canadian data, including a cross-source consistency check.
2
Days 44–47
Expectation suites, profiling datasets, and automated Data Docs.
By the end: The data quality plan from Day 41 is fully **executable**. Expectation suites are generated from reviewed YAML specifications ("suites as code"), and profiling of the actual data proposes thresholds that economists confirm. Checkpoints separate **critical** from **warning** checks, and every run produces a browsable **Data Docs** website plus a validation **history** you can query and trend in DuckDB.
3
Days 48–50
Validating incoming StatCan data: unemployment rates within 0–100%, continuous dates.
By the end: New StatCan releases (the full Labour Force Survey table, plus CPI and GDP) no longer flow straight into the warehouse. They're **written** to a holding area, **audited** by Great Expectations against a precise contract, and only then **published**, or **quarantined** with an alert. The contract encodes the curriculum's headline rules exactly: *unemployment rates remain between 0 and 100%* and *dates are continuous without gaps*. It adds accounting identities, cross-source checks and release-completeness checks. dbt builds provincial labour-market marts from published data only, and the warehouse records data quality as a KPI.

Tools

Great ExpectationsDuckDBData Docs

The ten days

DAY 41

Data Quality Principles

5 outcomes Soon

DAY 42

Installing Great Expectations and Initializing a Data Context

6 outcomes Soon

DAY 43

Connecting GX to DuckDB

Validation Definitions and Checkpoints

5 outcomes Soon

DAY 47

Automated Data Docs and Validation History

7 outcomes Soon

DAY 48

Write–Audit–Publish and the Boundary Contract

6 outcomes Soon

DAY 49

The Headline Contracts

6 outcomes Soon

DAY 50

Publish or Quarantine, and Closing the Loop

6 outcomes Soon

Before you start: module setup and project structure

Audience: MSc Economics students who completed Modules 1–4 (Days 1–40) Environment: GitHub Codespaces · macro.duckdb in production (dbt cut-over, Day 40) · a new, separate .venv-gx Time per day: ~2.5 hours (including breaks and exercises) Goal by end of Day 43: Students can reason about data quality systematically (dimensions, contracts, severities, ownership). They write a data quality plan for the macro warehouse, set up a Great Expectations (GX) project, connect it to DuckDB tables at every layer (raw_dlt, staging, marts), and run first validations against real Canadian data, including a cross-source consistency check.

Why another quality tool? We already have tests…

You've built quality checks at every stage:

Where	Tool	What it checks	Day
Ingestion	dlt schema contracts, Pydantic	shape and types of incoming records	25
Ingestion	resilience layer	HTTP failures, bad payloads, StatCan `FAILED` items	29
Warehouse	dbt tests	keys, ranges, relationships, gaps, freshness	36
Warehouse	dbt contracts	mart column names and types	40

These are rules you thought of, expressed in developer tools. Great Expectations adds:

Expectations as a shared, readable contract: "unemployment rate between 0 and 100%" as a declarative statement that economists, not just engineers, can read and review.
Rich validation results: not just pass/fail, but how many rows failed, which values, and what share.
Data Docs: a human-readable website of expectations and validation history (Days 44–47).
Validation at boundaries and on batches: checking each newly arrived month of StatCan data, independent of how it's transformed later (Days 48–50).
Distributional checks: means, quantiles and z-scores, not just row-by-row rules.

GX doesn't replace dbt tests. dbt tests protect the transformation logic; GX protects the data contract with the outside world and with consumers. Many teams use both, and you will too.

The plan for Module 5

Days	Focus
41–43	Principles, a quality plan, GX installation, data contexts, DuckDB connection, first validations
44–47	Expectation suites, profiling datasets, checkpoints, automated Data Docs
48–50	Validating incoming StatCan data (unemployment 0–100%, continuous dates), integration with the pipeline

Target structure (by end of Day 43)

macro-data-course/
├── .venv/                      ← dlt + dbt (Modules 3–4)
├── .venv-gx/                   ← NEW: Great Expectations environment
├── quality/                    ← NEW: the GX project
│   ├── DATA_QUALITY_PLAN.md
│   ├── requirements-gx.txt
│   ├── gx/                     ← created by GX (the Data Context)
│   │   ├── great_expectations.yml
│   │   ├── expectations/
│   │   ├── validation_definitions/
│   │   ├── checkpoints/
│   │   └── uncommitted/        ← local-only: config variables, Data Docs, results
│   ├── gx_helpers.py
│   ├── setup_gx.py
│   └── smoke_check.py