Module 3 · Days 21–30 · Part I

Data Ingestion with dlt

Declarative data pipelines and automated extraction.

Start Day 21

0/10 days completed

0/10 videos published

What you'll learn

1
Days 21–23
dlt pipelines, sources vs. resources, and secrets management.
By the end: Students rebuild their Day 10 macro_extract package with **dlt** (data load tool). They organise BoC, FRED and StatCan extraction into sources and resources, keep configuration and secrets out of their code, and load everything into macro.duckdb with one command.
2
Days 24–27
Schema evolution, flattening nested JSON, and incremental loading.
By the end: Students control how their schemas change and protect curated tables with schema contracts. They load deeply nested API responses (Valet, StatCan WDS metadata) and query them relationally. And they upgrade macro_ingest to fetch **only what's new**, absorb statistical **revisions**, and keep a history of **data vintages**.
3
Days 28–30
Resilient extractors for the BoC Valet API and StatCan endpoints, loading into DuckDB.
By the end: Students run a **production-grade ingestion layer**. It uses dlt's REST client, survives network failures, rate limits and bad payloads, isolates failing series, records errors and run summaries, alerts when something breaks, and feeds the whole macro.duckdb warehouse (staging → marts) from raw_dlt. The old Day 10 extraction path is retired.

Tools

dltREST APIsDuckDBPydantic

The ten days

DAY 21

Your First dlt Pipelines

Configuration, Secrets and the macro_ingest Package

Schema Contracts and Row Validation

6 outcomes Soon

DAY 26

Flattening Nested JSON API Responses

Resilience Engineering

8 outcomes Soon

DAY 30

macro_ingest v2 → The Warehouse

6 outcomes Soon

Before you start: module setup and project structure

Audience: MSc Economics students who completed Modules 1–2 (Days 1–20) Environment: GitHub Codespaces · course .venv · DuckDB CLI · macro.duckdb Time per day: ~2.5 hours (including breaks and exercises) Goal by end of Day 23: Students rebuild their Day 10 macro_extract package with dlt (data load tool). They organise BoC, FRED and StatCan extraction into sources and resources, keep configuration and secrets out of their code, and load everything into macro.duckdb with one command.

Why a new tool now?

On Days 8–10 you wrote extraction by hand, and it worked. Look at how much code went into things that have nothing to do with economics:

You wrote by hand (Days 8–10)	dlt provides
retrying HTTP session (`net.py`)	`dlt.sources.helpers.requests`, with retries built in
a common output schema (`schema.py`)	automatic schema inference, typing and evolution
writing Parquet, then loading DuckDB (Day 13)	loads directly into DuckDB (and 20+ other destinations)
a raw landing zone (`rawstore.py`)	load packages kept on disk for every run
a catalogue in a Python file	configuration files and environment variables
API keys in env vars, read manually	a full secrets system with clear lookup rules
`fetched_at` column, logging	`_dlt_load_id`, `_dlt_loads` table, run traces

dlt lets you keep your part (knowing which economic series to fetch and how to parse them) and hands the plumbing to a well-tested library.

You'll recognise almost everything here from Days 8–10. That's deliberate: dlt makes most sense once you've felt the pain it removes.

Vocabulary

dlt term	Meaning	Macro example
resource	a function that yields data for one table	BoC observations
source	a group of related resources sharing config and credentials	"Bank of Canada"
pipeline	moves data from sources to a destination	`macro_ingest`
destination	where data lands	DuckDB (`macro.duckdb`)
dataset	the schema inside the destination	`raw_dlt`

Every pipeline.run() goes through three stages: extract (call your resources) → normalize (infer schema, flatten, type) → load (write to the destination).

Target folder structure (by end of Day 23)

macro-data-course/
├── .dlt/
│   ├── config.toml          ← non-secret settings (committed)
│   └── secrets.toml         ← secrets (NEVER committed)
├── macro_ingest/            ← dlt version of macro_extract
│   ├── __init__.py
│   ├── boc.py
│   ├── fred.py
│   └── statcan.py
├── ingest.py                ← runs all sources into macro.duckdb
├── day21/  day22/  day23/   ← practice scripts