Day 47: Automated Data Docs and Validation History · Applied Decision Intelligence & Data Architecture

Written lesson

Time	Block
0:00 – 0:25	47.1 Building and browsing Data Docs
0:25 – 0:48	47.2 Making Data Docs readable for economists
0:48 – 0:55	Break
0:55 – 1:20	47.3 A queryable validation history
1:20 – 1:40	47.4 Trends: is data quality getting better or worse?
1:40 – 1:48	Break
1:48 – 2:03	47.5 Connecting Data Docs, dbt docs and the pipeline
2:03 – 2:30	47.6 Capstone: `run_quality.sh` and a guided review

47.1 Building and Browsing Data Docs (25 min)

Data Docs is a static website that GX generates from your suites and validation results. Your checkpoints already rebuild it via UpdateDataDocsAction. You can also build it directly:

from gx_helpers import get_context
context = get_context()
context.build_data_docs()
print(context.get_docs_sites_urls())

The site is written to quality/gx/uncommitted/data_docs/local_site/. Codespaces has no desktop browser, so serve the folder over a forwarded port:

python -m http.server 8081 --directory quality/gx/uncommitted/data_docs/local_site

Codespaces offers to open port 8081 in your browser. Stop the server with Ctrl+C.

A guided tour

Page	Shows	Use it to…
Home / index	every suite and every validation run, newest first, with pass/fail	see at a glance whether today's run is green
Expectation suite pages	each expectation in plain English, with parameters	review what we promise about each asset
Validation result pages	one run: each expectation's outcome, observed values, unexpected examples	investigate what failed and by how much

Open a failed run (from Try it 46.4, if you kept it) and follow a failure from the index to the exact unexpected values.

Data Docs contains data values (e.g. unexpected examples). That's why it lives under uncommitted/: treat it like the data itself. Share it deliberately, not by committing it to a public repository.

47.2 Making Data Docs Readable for Economists (23 min)

Data Docs renders expectations as sentences ("values must be between 0 and 100"). Three things make them genuinely useful to non-engineers.

1. Descriptive names. unemployment_rate__critical reads better than suite_3. The naming convention from Day 44 already pays off here.

2. Rich meta. Everything in an expectation's meta is displayed with it. The plan_id, dimension, owner and rationale fields from your specs appear right next to each check, so a reader sees why a rule exists and who to ask.

3. Notes in markdown. Data Docs has long supported a special notes entry in meta, rendered as formatted text. Add it for your most important expectations:

  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: value, min_value: 0, max_value: 100}
    meta:
      plan_id: Q01
      dimension: validity
      rationale: "A rate is a share of the labour force."
      notes:
        format: markdown
        content:
          - "**Source:** StatCan Labour Force Survey (vector v2062815), seasonally adjusted."
          - "**If this fails:** do not publish; check `raw_dlt.statcan_observations` for the vector."

(If your GX version doesn't render notes, the content still appears under meta in the rendered page.)

A suite-level description

Suites can carry meta too. Give rebuild_suite a description argument and pass spec.get("description", "") from main(), so each suite page opens with a sentence explaining its purpose:

def rebuild_suite(context, name, items, owner=None, description=""):
    try:
        context.suites.delete(name)
    except Exception:
        pass
    suite = context.suites.add(gx.ExpectationSuite(
        name=name, meta={"description": description, "owner": owner}))
    for item in items:
        suite.add_expectation(build_expectation(item, owner))
    suite.save()
    return suite

and add a description: line to each spec, e.g. "Monthly unemployment rate as published by StatCan (LFS, SA). Critical checks follow from definitions."

47.3 A Queryable Validation History (25 min)

Data Docs shows runs one by one. Economists and managers also ask trend questions: Which checks fail most often? Are warnings increasing since StatCan changed its release schedule? Did quality improve after the Day 30 migration? For that we need validation results as data.

Extend run_checkpoints.py to append one row per expectation per run to a Parquet file, one file per run:

from datetime import datetime, timezone
from pathlib import Path

import pandas as pd

HISTORY_DIR = Path(__file__).resolve().parent.parent / "data" / "quality" / "runs"


def history_rows(checkpoint_name, checkpoint_result, run_time):
    rows = []
    for vr in checkpoint_result.run_results.values():
        for r in vr.results:
            cfg = r.expectation_config
            meta = cfg.meta or {}
            res = r.result or {}
            observed = res.get("observed_value")
            rows.append({
                "run_time": run_time,
                "checkpoint": checkpoint_name,
                "suite": vr.suite_name,
                "asset": vr.suite_name.split("__")[0],
                "plan_id": meta.get("plan_id"),
                "dimension": meta.get("dimension"),
                "expectation_type": cfg.type,
                "column_name": cfg.kwargs.get("column"),
                "success": bool(r.success),
                "element_count": res.get("element_count"),
                "unexpected_count": res.get("unexpected_count"),
                "unexpected_percent": res.get("unexpected_percent"),
                "observed_value": None if observed is None else str(observed),
            })
    return rows


def save_history(rows, run_time):
    HISTORY_DIR.mkdir(parents=True, exist_ok=True)
    path = HISTORY_DIR / f"run_{run_time:%Y%m%dT%H%M%S}.parquet"
    pd.DataFrame(rows).to_parquet(path, index=False)
    return path

In main(), set run_time = datetime.now(timezone.utc) once, collect history_rows(...) for each checkpoint, and call save_history(...) before exiting, so failed runs are recorded too.

Design choices:

One file per run makes appending trivial and safe: no read-modify-write of a growing file.
Parquet keeps types (Day 13), and DuckDB reads a whole folder with a glob.
observed_value is stored as text because it can be a number, a list or a dict, depending on the expectation.
data/quality/ is under data/, which is already git-ignored (Day 4).

Run the checkpoints a few times (break and restore a spec in between) to create some history.

47.4 Trends: Is Data Quality Getting Better or Worse? (20 min)

Query the history directly with DuckDB (from either environment):

duckdb -c "
select checkpoint,
       date_trunc('day', run_time)                  as day,
       count(*)                                     as checks,
       count(*) filter (where not success)          as failures,
       round(100.0 * avg(success::int), 1)          as pct_passing
from 'data/quality/runs/*.parquet'
group by all
order by day desc, checkpoint"

Which checks fail most often?

select plan_id, asset, expectation_type, column_name,
       count(*)                             as runs,
       count(*) filter (where not success)  as failed_runs,
       max(run_time) filter (where not success) as last_failure
from 'data/quality/runs/*.parquet'
group by all
having count(*) filter (where not success) > 0
order by failed_runs desc;

Quality by dimension over time

select dimension,
       date_trunc('week', run_time)          as week,
       round(100.0 * avg(success::int), 1)   as pct_passing
from 'data/quality/runs/*.parquet'
group by all
order by week desc, dimension;

Into the warehouse with dbt (a preview of integration)

Because the history is Parquet, dbt-duckdb can read it as a source, putting quality results on the lineage graph and in the marts. In macro_dbt/models/staging/quality/_quality__sources.yml:

version: 2

sources:
  - name: quality
    description: "Great Expectations validation history (quality/run_checkpoints.py)."
    meta:
      external_location: "../data/quality/runs/*.parquet"
    tables:
      - name: gx_results

Then models/staging/quality/stg_quality__gx_results.sql can select * from {{ source('quality', 'gx_results') }} and a small mart can summarise pass rates per asset for the macro cockpit (Day 39's KPI table). We'll wire this in fully on Days 48–50.

Why track quality over time? A single run tells you whether today's data is usable. The trend tells you whether your sources and pipeline are healthy, which is exactly what you need when deciding whether to trust a new data provider or justify engineering time.

47.5 Connecting Data Docs, dbt Docs and the Pipeline (15 min)

Cross-link the two documentation sites

Readers of dbt docs (Day 40) should know quality results exist. Add to the __overview__ doc block in macro_dbt/models/docs.md:

**Data quality:** every mart and key raw table is validated by Great Expectations.
See the Data Docs site (served from `quality/gx/uncommitted/data_docs/local_site`)
and the quality plan in `quality/DATA_QUALITY_PLAN.md`.

And declare Data Docs as an exposure consuming the validated models, so the lineage graph shows that quality checks depend on them:

  - name: gx_data_docs
    label: "Great Expectations Data Docs"
    type: application
    maturity: medium
    description: "Validation of raw, staging and mart tables against the data quality plan."
    depends_on:
      - ref('stg_statcan__cpi')
      - ref('int_macro__observations')
      - ref('macro_monthly')
      - ref('fct_open_economy_daily')
    owner:
      name: Macro Data Course team

Where quality runs in the pipeline

ingest.py (dlt)  →  dbt build --target prod  →  run_checkpoints.py (GX)  →  publish / alert
                                                   │
                                                   ├─ exit 0: publish marts
                                                   ├─ exit 2: publish, flag for review
                                                   └─ exit 1: do NOT publish; alert

On Days 48–50 we'll refine this so that GX validates incoming StatCan data before dbt transforms it, and so that "publishing" really depends on the result.

47.6 Capstone: `run_quality.sh` and a Guided Review (28 min)

One command for the quality layer

Create quality/run_quality.sh:

#!/usr/bin/env bash
# run_quality.sh — (re)build GX objects from specs, run checkpoints, build Data Docs.
# Exit code is that of run_checkpoints.py: 0 ok · 1 critical · 2 warnings.
set -u
cd "$(dirname "$0")/.."                       # repository root
source .venv-gx/bin/activate
export MACRO_DUCKDB_URL="${MACRO_DUCKDB_URL:-duckdb:///$(pwd)/macro.duckdb}"

python quality/setup_gx.py      || exit 1
python quality/build_suites.py  || exit 1
python quality/run_checkpoints.py
status=$?

(cd quality && python -c "from gx_helpers import get_context; get_context().build_data_docs()")

echo "Data Docs: quality/gx/uncommitted/data_docs/local_site/index.html"
exit $status

chmod +x quality/run_quality.sh
./quality/run_quality.sh; echo "exit code: $?"

Notes:

set -u stops on undefined variables, but not on non-zero exit codes, because we want to build Data Docs even after a failed validation, then return the validation's exit code.
${MACRO_DUCKDB_URL:-...} uses the existing variable if set, and otherwise a default. That's shell syntax for "value or default".

Hook it into the main pipeline

Add to the end of pipeline.sh (Day 40), after the dbt build:

./quality/run_quality.sh
quality_status=$?
if [ $quality_status -eq 1 ]; then
    echo "⛔ Critical data quality failure — see Data Docs. Marts must not be used."
    exit 1
fi
[ $quality_status -eq 2 ] && echo "⚠ Data quality warnings — review Data Docs."
echo "✓ Pipeline and quality checks complete"

A guided review for a non-technical reader

Write quality/HOW_TO_READ_DATA_QUALITY.md (about one page) for an economist colleague who has never seen GX. It should explain:

how to open Data Docs, and what green and red mean;
the difference between critical and warning, with one example of each from our data;
how to find why a check exists (plan ID, rationale) and who owns it;
how to see whether quality is improving (the trend queries from 47.4);
what to do when something is red: the escalation path from the quality plan.

Then ask a classmate to follow the document without your help and note where they got stuck. Revise accordingly. Documentation that hasn't been tested on a real reader is like code that hasn't been run.

Recap: Days 44–47

Capability	Built
Suites organised by asset and severity, linked to the plan	naming convention, `meta`
Expectations as reviewed configuration	`specs/*.yml`, `build_suites.py`, `REVIEW_LOG.md`
Data-driven thresholds with human judgement	`profile_assets.py`, drafts, change profiling
Distributional checks for unit errors and stuck feeds	mean, stdev, quantiles, unique counts
Reusable validation runs with actions	validation definitions, `critical`/`warning` checkpoints
Pipeline-aware outcomes	`run_checkpoints.py` exit codes 0/1/2, one alert channel
Human-readable evidence	Data Docs with rationales and notes
Quality as data	Parquet history, DuckDB trend queries, dbt source preview

Looking ahead: Days 48–50

Add the full Labour Force Survey table as a new StatCan source and validate it at the boundary, before dbt runs.
Encode the curriculum's two headline contracts precisely: unemployment rates between 0 and 100% for every geography, sex and age group, and dates continuous without gaps, per series.
Use batch-level validation of each new release and decide what happens to a bad release: quarantine, reject or publish with a flag.
Wire GX into the pipeline with dbt reading the quality history, ready for Prefect orchestration in Module 6.

Day 47 checklist

I can build, serve and navigate Data Docs, and I treat them as containing data
Expectations and suites carry rationales and notes that non-engineers can follow
Every checkpoint run appends a queryable history, including failed runs
I can analyse quality trends by check, asset and dimension in DuckDB
Data Docs and dbt docs cross-reference each other; Data Docs is a dbt exposure
run_quality.sh runs the whole quality layer and gates the pipeline via exit codes
A colleague can read our data quality results using my guide

git add quality/ macro_dbt/ pipeline.sh
git commit -m "Day 47: Data Docs, validation history, run_quality.sh"
git push

Automated Data Docs and Validation History

What you'll learn today

Today's plan