Installing Great Expectations and Initializing a Data Context

42.1 A Separate Environment, and Why (20 min)

dlt, dbt and GX are large libraries with many dependencies. Installing them all into one environment risks version conflicts: one tool needs version X of a shared library, another needs Y. The robust solution is the one you learned on Day 3: one virtual environment per tool family.

cd /workspaces/macro-data-course
python -m venv .venv-gx
source .venv-gx/bin/activate
pip install --upgrade pip
pip install great_expectations duckdb duckdb-engine pandas
pip freeze > quality/requirements-gx.txt
python -c "import great_expectations as gx; print(gx.__version__)"

• great_expectations: GX Core, the open-source library.
• duckdb-engine: lets GX (which talks to SQL databases through SQLAlchemy) connect to DuckDB.

Add .venv-gx/ to .gitignore.

⚠️ Version warning. GX changed substantially at version 1.0. Many tutorials online still show the older API (great_expectations init on the command line, "Data Assistants", context.sources.add_...). This course uses the 1.x Python API (gx.get_context(mode=...), context.data_sources.add_..., gx.expectations.Expect...). If you find code online, check which version it targets.

42.2 The GX Mental Model (20 min)

GX organises everything in a hierarchy. Learn these words and the rest of the module becomes straightforward.

Data Context  ── the GX project: configuration + stored objects
│
├── Data Source        how to connect         e.g. "macro_duckdb" (SQL connection)
│   └── Data Asset     what to read           e.g. table staging.stg_statcan__cpi
│       └── Batch Definition   which slice    e.g. the whole table, or one month
│           └── Batch          the actual data at validation time
│
├── Expectation        one testable statement  "values in column X are between 0 and 100"
├── Expectation Suite  a named group of expectations for one asset
│
├── Validation Definition  = Batch Definition + Expectation Suite
├── Checkpoint             runs validation definitions + actions (Days 44–47)
│   └── Actions            e.g. update Data Docs, send a notification
│
└── Data Docs          generated website of expectations and results (Days 44–47)

Mapping GX to what you know

42.3 A First Validation in an Ephemeral Context (25 min)

An ephemeral context lives only in memory: nothing is saved. It's perfect for learning.

Create quality/day42_first_steps.py:

import great_expectations as gx
import pandas as pd

# 1. An in-memory Data Context
context = gx.get_context(mode="ephemeral")

# 2. Some data: the notebook's provincial fiscal positions (Day 4)
df = pd.DataFrame({
    "province": ["BC", "AB", "SK", "MB", "ON", "QC", "NB", "NS", "PE", "NL"],
    "balance_pct_gdp": [-0.8, 0.2, 0.5, -0.9, -1.4, -1.2, -0.5, -0.3, -0.7, 0.1],
})

# 3. Data Source → Data Asset → Batch Definition → Batch
source = context.data_sources.add_pandas(name="pandas_lab")
asset = source.add_dataframe_asset(name="provinces")
batch_definition = asset.add_batch_definition_whole_dataframe(name="whole")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})

print(batch.head())

# 4. An expectation, validated against the batch
expectation = gx.expectations.ExpectColumnValuesToBeBetween(
    column="balance_pct_gdp", min_value=-5, max_value=5
)
result = batch.validate(expectation)
print("Success:", result.success)
print(result.result)

source .venv-gx/bin/activate
python quality/day42_first_steps.py

You just wrote your first executable data contract clause: provincial balances lie between −5% and +5% of GDP.

Make it fail on purpose

Change Ontario's value to -14.0 (a slipped decimal point: a unit error, like Story 1) and run again. Now read result.result:

{'element_count': 10, 'unexpected_count': 1, 'unexpected_percent': 10.0,
 'partial_unexpected_list': [-14.0], ...}

GX tells you how many rows failed, what share, and which values, far more than a pass/fail flag.

42.4 Anatomy of an Expectation and Its Result (25 min)

The parts of an expectation

gx.expectations.ExpectColumnValuesToBeBetween(
    column="value",          # what it applies to
    min_value=0,             # parameters of the rule
    max_value=100,
    mostly=0.99,             # optional: minimum share of rows that must pass
    meta={"dimension": "validity", "severity": "critical", "plan_id": "Q01"},
)

• Class names read as sentences: ExpectColumnValuesToBeBetween = "expect column values to be between…".
• mostly (Day 41.5) tolerates a share of failures.
• meta is free-form metadata. We'll use it to link each expectation to the quality plan (ID, dimension, severity). It appears in Data Docs.

Families of expectations

Result formats

result = batch.validate(expectation, result_format="COMPLETE")

Use SUMMARY by default. COMPLETE can be large on big tables.

Validating several expectations at once: a first suite

suite = gx.ExpectationSuite(name="provinces_suite")
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="province"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="province"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(
    column="province",
    value_set=["BC", "AB", "SK", "MB", "ON", "QC", "NB", "NS", "PE", "NL"]))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(
    column="balance_pct_gdp", min_value=-5, max_value=5))
suite.add_expectation(gx.expectations.ExpectTableRowCountToEqual(value=10))

suite = context.suites.add(suite)          # register with the context
results = batch.validate(suite)

print("Suite success:", results.success)
for r in results.results:
    print(f"{'✓' if r.success else '✗'} {r.expectation_config.type}")

Suites get the full treatment on Days 44–47.

Try it 42.4

Add an aggregate expectation to the suite that would catch a unit error across the whole column, even if every individual value stayed inside ±5. (Hint: what happens to the mean if all values are accidentally divided by 100?)

suite.add_expectation(gx.expectations.ExpectColumnStdevToBeBetween(
    column="balance_pct_gdp", min_value=0.2, max_value=3,
    meta={"dimension": "plausibility", "note": "catches /100 or ×100 unit errors"}))

42.5 A File-Based Data Context: the GX Project (23 min)

An ephemeral context forgets everything. For real work, GX stores its configuration and objects on disk, in a folder you commit to Git, just like the dbt project.

# quality/day42_init_project.py
import great_expectations as gx

context = gx.get_context(mode="file", project_root_dir="quality")
print(type(context).__name__)
print("Project root:", context.root_directory)

python quality/day42_init_project.py
find quality/gx -maxdepth 2 | sort

GX created quality/gx/:

GX writes its own .gitignore inside gx/ to exclude uncommitted/. Confirm with:

cat quality/gx/.gitignore
git status quality/

Running gx.get_context(mode="file", project_root_dir="quality") a second time loads the existing project instead of creating a new one, so every later script starts with this same line.

A helper module for the project

Create quality/gx_helpers.py. Every script in quality/ will import it:

"""Shared helpers for the macro GX project."""
from pathlib import Path

import great_expectations as gx

PROJECT_ROOT = Path(__file__).resolve().parent          # .../quality


def get_context():
    """Load (or create) the file-based Data Context in quality/gx."""
    return gx.get_context(mode="file", project_root_dir=str(PROJECT_ROOT))


def get_or_add_suite(context, name):
    """Return the named suite, creating it if it doesn't exist yet."""
    try:
        return context.suites.get(name)
    except Exception:
        return context.suites.add(gx.ExpectationSuite(name=name))

Path(__file__).resolve().parent is the folder containing this file, so the helper works no matter which directory you run scripts from.

42.6 Mini-Project: Validating the BoC Decision Log (23 min)

The decision log (macro_dbt/seeds/seed_boc_decisions.csv) is hand-maintained, the kind of data most prone to typos. Write quality/day42_decisions.py that:

1. Loads the file context with gx_helpers.get_context().
2. Reads the CSV with pandas (parse_dates=["decision_date"]).
3. Registers a pandas data source seeds_pandas, asset boc_decisions, and a whole-dataframe batch definition (reuse them if they already exist).
4. Builds and saves a suite boc_decisions_suite with at least six expectations:
   • decision_date not null and unique;
   • rate between 0 and 10;
   • change_bp in the set of multiples of 25 from −100 to +100;
   • at least 25 rows;
   • columns exactly decision_date, rate, change_bp, note;
   • (stretch) rate never negative and change_bp consistent with the previous rate. Try it and discover why that isn't a single-row expectation. (Answer: it needs row ordering, so it belongs in a SQL-backed check or in dbt, as on Day 36.)
5. Validates, prints each expectation's result, and exits with code 1 if the suite fails.

"""Validate the hand-maintained BoC decision log seed."""
import sys

import great_expectations as gx
import pandas as pd

from gx_helpers import PROJECT_ROOT, get_context, get_or_add_suite

CSV = PROJECT_ROOT.parent / "macro_dbt" / "seeds" / "seed_boc_decisions.csv"

context = get_context()
df = pd.read_csv(CSV, parse_dates=["decision_date"])

# Data source → asset → batch definition (get or add, so the script can re-run)
try:
    source = context.data_sources.get("seeds_pandas")
except Exception:
    source = context.data_sources.add_pandas(name="seeds_pandas")
try:
    asset = source.get_asset("boc_decisions")
except Exception:
    asset = source.add_dataframe_asset(name="boc_decisions")
try:
    batch_def = asset.get_batch_definition("whole")
except Exception:
    batch_def = asset.add_batch_definition_whole_dataframe(name="whole")

batch = batch_def.get_batch(batch_parameters={"dataframe": df})

suite = get_or_add_suite(context, "boc_decisions_suite")
E = gx.expectations
for exp in [
    E.ExpectColumnValuesToNotBeNull(column="decision_date"),
    E.ExpectColumnValuesToBeUnique(column="decision_date"),
    E.ExpectColumnValuesToBeBetween(column="rate", min_value=0, max_value=10),
    E.ExpectColumnValuesToBeInSet(column="change_bp",
                                  value_set=list(range(-100, 101, 25))),
    E.ExpectTableRowCountToBeBetween(min_value=25),
    E.ExpectTableColumnsToMatchSet(column_set=["decision_date", "rate", "change_bp", "note"],
                                   exact_match=True),
]:
    suite.add_expectation(exp)
suite.save()

results = batch.validate(suite)
for r in results.results:
    print(f"{'✓' if r.success else '✗'} {r.expectation_config.type} "
          f"{r.expectation_config.kwargs.get('column', '')}")
print("Suite success:", results.success)
sys.exit(0 if results.success else 1)

Day 42 checklist

• GX runs in its own environment, and I know why separate environments prevent conflicts
• I can explain Data Context, Data Source, Data Asset, Batch Definition, Batch, Expectation, Suite
• I can validate an expectation in an ephemeral context and read the detailed result
• I know the expectation families (table, column map, aggregate, multi-column) and result_format
• I created a file-based Data Context and know what to commit and what stays in uncommitted/
• I validated the BoC decision log with a saved suite and a meaningful exit code

git add quality/ .gitignore
git commit -m "Day 42: GX environment, context, first suites"
git push