Expectation Suites

44.1 Designing Suites (23 min)

An expectation suite is a named collection of expectations validated together against one batch. How you group expectations into suites decides how useful your results are.

Three design principles

1. One asset per suite. A suite describes one kind of data. Don't mix CPI and exchange-rate expectations; they fail for different reasons and have different owners.

2. Split by severity. A run should be able to answer "may we publish?" (critical) separately from "should someone take a look?" (warning). The simplest robust way is two suites per asset:

unemployment_rate__critical    ← definitions: 0–100%, not null, no duplicate months
unemployment_rate__warning     ← history-based: typical ranges, distributions

3. Every expectation traces back to the plan. Each expectation carries meta with its plan ID, dimension, owner and rationale, so a failure in Data Docs immediately tells a reader why the check exists.

Naming convention

This mirrors dbt's double-underscore naming (Day 34), so names carry their meaning.

What goes where?

44.2 Suites as Code: YAML Specifications (23 min)

On Day 42 you built suites in Python by calling add_expectation repeatedly, and noticed that re-running could create duplicates. The professional fix is to declare expectations in reviewable files and generate suites from them.

Why YAML specs?

• Reviewable: an economist can read and approve min_value: 0, max_value: 100 without reading Python.
• Diffable: git diff quality/specs/ shows exactly which thresholds changed and when.
• Deterministic: suites are regenerated from specs every time, so there's no drift and no duplicates.
• Linked to the plan: each item carries its plan ID.

It's the same idea as dbt's YAML tests (Day 36) and dlt's config.toml (Day 23): configuration as data, reviewed like code.

A spec file

Create quality/specs/unemployment_rate.yml:

# Expectations for the StatCan unemployment rate (vector v2062815, monthly, SA)
asset: unemployment_rate
batch_definition: full
owner: macro-team

critical:
  - type: ExpectColumnValuesToNotBeNull
    kwargs: {column: value}
    meta:
      plan_id: Q01a
      dimension: completeness
      rationale: "Every published month must have a value."

  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: value, min_value: 0, max_value: 100}
    meta:
      plan_id: Q01
      dimension: validity
      rationale: "A rate is a share of the labour force: it cannot be below 0% or above 100%."

  - type: ExpectColumnValuesToBeUnique
    kwargs: {column: obs_date}
    meta:
      plan_id: Q02a
      dimension: uniqueness
      rationale: "One observation per month."

warning:
  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: value, min_value: 2.5, max_value: 16, mostly: 0.999}
    meta:
      plan_id: Q11
      dimension: plausibility
      rationale: >
        Canada's unemployment rate has historically stayed within roughly this band
        (confirm with the Day 45 profile). Values outside deserve review, not rejection.

Structure of each item:

The warning range above is a placeholder. On Day 45 you'll replace it with values derived from the data profile.

44.3 A Suite Builder That Regenerates Deterministically (28 min)

Create quality/build_suites.py. On each run it:

1. reads every specs/*.yml file;
2. deletes any existing suite with the same name, then recreates it from the spec (no duplicates, no drift);
3. saves the suite as JSON in gx/expectations/.

(On Day 46 the same script also creates validation definitions and checkpoints.)

"""Build GX expectation suites from YAML specifications in quality/specs/.

Suites are regenerated from scratch on every run, so the specs are the single
source of truth. Usage:  python quality/build_suites.py
"""
import great_expectations as gx
import yaml

from gx_helpers import PROJECT_ROOT, get_context

SPEC_DIR = PROJECT_ROOT / "specs"
SEVERITIES = ("critical", "warning")


def load_specs():
    """Yield (spec_name, spec_dict) for every top-level YAML spec (not drafts)."""
    for path in sorted(SPEC_DIR.glob("*.yml")):
        with open(path) as f:
            spec = yaml.safe_load(f)
        if "asset" not in spec:
            raise ValueError(f"{path.name}: missing 'asset'")
        yield path.stem, spec


def build_expectation(item, owner=None):
    """Turn one spec item into a GX Expectation object."""
    try:
        cls = getattr(gx.expectations, item["type"])
    except AttributeError:
        raise ValueError(f"Unknown expectation type: {item['type']}")
    meta = dict(item.get("meta", {}))
    if owner:
        meta.setdefault("owner", owner)
    return cls(**item.get("kwargs", {}), meta=meta)


def rebuild_suite(context, name, items, owner=None):
    """Delete (if present) and recreate a suite from spec items."""
    try:
        context.suites.delete(name)
    except Exception:
        pass
    suite = context.suites.add(gx.ExpectationSuite(name=name))
    for item in items:
        suite.add_expectation(build_expectation(item, owner))
    suite.save()
    return suite


def main():
    context = get_context()
    built = []
    for spec_name, spec in load_specs():
        for severity in SEVERITIES:
            items = spec.get(severity) or []
            if items:
                name = f"{spec['asset']}__{severity}"
                rebuild_suite(context, name, items, spec.get("owner"))
                built.append((name, len(items)))
    for name, n in built:
        print(f"✓ {name:<55} {n} expectation(s)")


if __name__ == "__main__":
    main()

Key techniques:

• getattr(gx.expectations, item["type"]) looks up a class by its name as text. That's what lets a YAML file say ExpectColumnValuesToBeBetween and Python create the right object. A typo in the YAML raises a clear error.
• cls(**item.get("kwargs", {}), meta=meta): ** unpacks a dictionary into keyword arguments. {"column": "value", "min_value": 0} becomes column="value", min_value=0.
• Delete, then recreate guarantees the stored suite equals the spec.

python quality/build_suites.py
ls quality/gx/expectations/
cat quality/gx/expectations/unemployment_rate__critical.json | head -40

Once Day 46 adds validation definitions that reference suites, deleting a suite first requires deleting what references it. The Day 46 version of this script handles that order: checkpoints → validation definitions → suites, then rebuilds in reverse.

44.4 Writing Specs for the Core Assets (23 min)

Write specs for the assets registered on Day 43. Here are two complete examples, followed by guidance for the rest.

quality/specs/macro_monthly.yml: a mart consumers read

asset: macro_monthly
batch_definition: full
owner: macro-team

critical:
  - type: ExpectTableColumnsToMatchSet
    kwargs:
      column_set: [month, overnight_rate, fed_funds, yield_2yr, yield_10yr, usd_cad,
                   unemployment_rate, headline_yoy, core_ex_fe_yoy, real_rate]
      exact_match: false
    meta: {plan_id: Q20, dimension: validity,
           rationale: "Columns the dashboards depend on (exposures, Day 40) must exist."}

  - type: ExpectColumnValuesToBeUnique
    kwargs: {column: month}
    meta: {plan_id: Q07, dimension: uniqueness}

  - type: ExpectColumnValuesToNotBeNull
    kwargs: {column: month}
    meta: {plan_id: Q07, dimension: completeness}

  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: unemployment_rate, min_value: 0, max_value: 100}
    meta: {plan_id: Q01m, dimension: validity}

  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: overnight_rate, min_value: -1, max_value: 25}
    meta: {plan_id: Q21, dimension: validity}

  - type: ExpectTableRowCountToBeBetween
    kwargs: {min_value: 240}
    meta: {plan_id: Q22, dimension: completeness,
           rationale: "Panel starts 2006-01; at least 20 years of months must be present."}

warning:
  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: headline_yoy, min_value: -3, max_value: 10, mostly: 0.99}
    meta: {plan_id: Q23, dimension: plausibility}

  - type: ExpectColumnMeanToBeBetween
    kwargs: {column: yield_10yr, min_value: 1, max_value: 8}
    meta: {plan_id: Q24, dimension: plausibility,
           rationale: "Catches unit errors (0.0375 instead of 3.75), Day 41 Story 1."}

quality/specs/cpi_vector_vs_table_mismatches.yml: the consistency check

asset: cpi_vector_vs_table_mismatches
batch_definition: full
owner: macro-team

critical:
  - type: ExpectTableRowCountToEqual
    kwargs: {value: 0}
    meta:
      plan_id: Q04
      dimension: consistency
      rationale: >
        CPI All-items from the WDS vector and from the full table must agree for every
        month present in both. Rows in this asset are disagreements.

Guidance for the remaining specs

Try it 44.4

Write the critical part of quality/specs/stg_statcan__cpi.yml, using batch_definition: by_month (the latest month only).

asset: stg_statcan__cpi
batch_definition: by_month
owner: macro-team

critical:
  - type: ExpectColumnValuesToNotBeNull
    kwargs: {column: cpi}
    meta: {plan_id: Q09, dimension: completeness}

  - type: ExpectColumnValuesToBeBetween
    kwargs: {column: cpi, min_value: 0, strict_min: true}
    meta: {plan_id: Q09, dimension: validity, rationale: "A price index is positive."}

  - type: ExpectCompoundColumnsToBeUnique
    kwargs: {column_list: [month, geo, component, index_base]}
    meta: {plan_id: Q05, dimension: uniqueness}

  - type: ExpectColumnValuesToBeInSet
    kwargs: {column: geo_level, value_set: [national, province, city]}
    meta: {plan_id: Q25, dimension: validity}

44.5 Reading Suite Results Programmatically (20 min)

Validating a whole suite returns an ExpectationSuiteValidationResult. Knowing its structure lets you build reports, alerts and history.

Create quality/day44_inspect_results.py:

from gx_helpers import get_context

context = get_context()
ds = context.data_sources.get("macro_duckdb")
batch = ds.get_asset("macro_monthly").get_batch_definition("full").get_batch()

suite = context.suites.get("macro_monthly__critical")
res = batch.validate(suite, result_format="SUMMARY")

print("Suite success:", res.success)
print("Statistics:", res.statistics)          # evaluated, successful, unsuccessful, success_percent

for r in res.results:                          # one ExpectationValidationResult per expectation
    cfg = r.expectation_config
    meta = cfg.meta or {}
    detail = r.result.get("unexpected_count", r.result.get("observed_value"))
    print(f"{'✓' if r.success else '✗'} [{meta.get('plan_id', '-'):<5}] "
          f"{meta.get('dimension', ''):<13} {cfg.type:<40} "
          f"{cfg.kwargs.get('column', ''):<20} {detail}")

Grouping by quality dimension

Because every expectation carries meta.dimension, you can report quality the way the Day 41 plan thinks about it:

from collections import Counter

by_dim, failed_dim = Counter(), Counter()
for r in res.results:
    dim = (r.expectation_config.meta or {}).get("dimension", "unknown")
    by_dim[dim] += 1
    failed_dim[dim] += 0 if r.success else 1

for dim in sorted(by_dim):
    print(f"{dim:<14} {by_dim[dim] - failed_dim[dim]}/{by_dim[dim]} passing")

44.6 Mini-Project: Every Plan Check in a Spec (20 min)

1. Write specs for all assets in the table in 44.4, so that every GX-assigned check in DATA_QUALITY_PLAN.md appears in exactly one spec with its plan ID.
2. Run python quality/build_suites.py and confirm one suite per asset and severity.
3. Write quality/day44_plan_coverage.py, which reads the plan's check table and the specs and prints any GX-assigned plan ID that has no expectation. A quality plan that isn't implemented is just a wish.

"""List GX-assigned plan IDs that have no expectation in any spec."""
import re

import yaml

from gx_helpers import PROJECT_ROOT

plan = (PROJECT_ROOT / "DATA_QUALITY_PLAN.md").read_text()
# Rows look like: | Q01 | asset | dimension | ... | GX (Day 43) |
planned_gx = {m.group(1) for m in re.finditer(r"^\|\s*(Q\d+)\s*\|.*\|\s*GX", plan, re.M)}

implemented = set()
for path in (PROJECT_ROOT / "specs").glob("*.yml"):
    spec = yaml.safe_load(path.read_text())
    for sev in ("critical", "warning"):
        for item in spec.get(sev) or []:
            pid = (item.get("meta") or {}).get("plan_id", "")
            implemented.add(re.match(r"Q\d+", pid).group(0) if re.match(r"Q\d+", pid) else pid)

missing = sorted(planned_gx - implemented)
print("Planned for GX:", len(planned_gx), "| implemented:", len(planned_gx & implemented))
print("Missing:", missing or "none ✓")

Day 44 checklist

• My suites follow one-asset-per-suite and critical/warning separation, with consistent names
• Every expectation lives in a reviewed YAML spec with plan ID, dimension and rationale
• build_suites.py regenerates suites deterministically from specs
• I can read suite results programmatically and report by quality dimension
• A coverage script proves every GX-assigned plan check is implemented

git add quality/ && git commit -m "Day 44: suites as code from YAML specs" && git push