How Web APIs Work: The Bank of Canada Valet API

8.1 Concepts (20 min)

The request–response cycle

An API works like the reference desk at a statistical agency's library:

1. You fill in a request slip (the URL): "Series V39079, from 2024 onwards, as JSON."
2. The librarian (the server) checks it and returns a response. It includes a status stamp (found / not found / come back later) and the data itself.

Your Python code is the client. Every call follows this cycle.

Anatomy of a URL

https://www.bankofcanada.ca/valet/observations/V39079/json?start_date=2024-01-01&end_date=2024-12-31
└─┬─┘   └────────┬────────┘ └────────────┬──────────────┘ └──────────────────┬─────────────────────┘
scheme         host                    path                        query string (parameters)

HTTP methods

Status codes: the stamp on the response

Rule of thumb: 4xx = your mistake (retrying won't help); 5xx and 429 = their problem (retrying often helps). This distinction drives the retry logic on Day 9.

Try it in the terminal first

curl is a command-line client. -s hides the progress bar and -i shows the response headers:

curl -s "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3"
curl -s -i "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3" | head -15
curl -s -i "https://www.bankofcanada.ca/valet/observations/NOT_A_SERIES/json" | head -1

The last command shows what an error looks like.

8.2 The requests Library in Depth (25 min)

Create day08/requests_basics.py.

Example 1 — Inspecting a response

import requests

url = "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=5"
r = requests.get(url, timeout=20)

print("Status:", r.status_code)                     # 200
print("OK?", r.ok)                                  # True for 2xx codes
print("Content type:", r.headers["Content-Type"])   # application/json...
print("Response time:", r.elapsed.total_seconds(), "s")
print("First 300 characters:\n", r.text[:300])

r.text is the raw text. r.json() converts JSON text into Python dictionaries and lists.

Example 2 — Let requests build the query string

Gluing strings together is error-prone. Pass parameters as a dictionary instead:

base = "https://www.bankofcanada.ca/valet/observations/V39079/json"
params = {"start_date": "2025-01-01", "end_date": "2025-06-30"}

r = requests.get(base, params=params, timeout=20)
print(r.url)          # see the URL requests built for you
data = r.json()
print(len(data["observations"]), "observations")

Example 3 — Always use a timeout

Without timeout, a hung server can freeze your script forever.

r = requests.get(base, params={"recent": 1}, timeout=20)   # seconds

Habit: every requests call gets a timeout. No exceptions.

Example 4 — Turning bad statuses into errors

bad = requests.get(
    "https://www.bankofcanada.ca/valet/observations/NOT_A_SERIES/json",
    timeout=20)
print(bad.status_code)

try:
    bad.raise_for_status()
except requests.HTTPError as e:
    print("Request failed:", e)

raise_for_status() does nothing on success and raises an error on 4xx/5xx. The course notebook calls it in boc_fetch and fred_fetch.

8.3 JSON: Reading Nested Data (25 min)

JSON (JavaScript Object Notation) is the most common format for API data. It maps directly onto Python:

Create day08/json_explore.py.

Example 1 — Pretty-printing

import json
import requests

r = requests.get(
    "https://www.bankofcanada.ca/valet/observations/V39079/json",
    params={"recent": 3}, timeout=20)
payload = r.json()

print(json.dumps(payload, indent=2))

Example 2 — Exploring an unknown structure, one level at a time

With a new API, work out the shape before writing any parsing code:

print(type(payload))                 # dict
print(list(payload.keys()))          # ['terms', 'seriesDetail', 'observations']

print(type(payload["observations"])) # list
first = payload["observations"][0]
print(first)                         # {'d': '...', 'V39079': {'v': '2.25'}}
print(first.keys())

print(payload["seriesDetail"])       # metadata: label, description, dimension

Example 3 — Navigating nested values

obs = payload["observations"][-1]
date = obs["d"]
value = obs["V39079"]["v"]           # dict inside a dict
label = payload["seriesDetail"]["V39079"]["label"]

print(f"{label}: {value}% on {date}")
print(type(value))                   # str — numbers arrive as text (Day 1!)

Example 4 — Safe navigation with .get

Real responses sometimes lack a key. Chained .get calls with defaults never crash:

weird = {"d": "2026-01-06"}                        # no series key at all
print(weird.get("V39079", {}).get("v", ""))        # "" instead of a KeyError

This is the exact pattern in the notebook's boc_fetch: val = o.get(series, {}).

Example 5 — Saving and reloading raw JSON

import os

os.makedirs("data/raw/boc", exist_ok=True)
with open("data/raw/boc/V39079_sample.json", "w") as f:
    json.dump(payload, f, indent=2)

with open("data/raw/boc/V39079_sample.json") as f:
    reloaded = json.load(f)
print(reloaded == payload)      # True

Saving raw responses lets you re-parse later without calling the API again. We'll build on this on Day 9.

Try it 8.3

Fetch the last 5 observations of FXCADUSD. Print the series label, and each date with its value converted to float and formatted to 4 decimals.

code = "FXCADUSD"
r = requests.get(f"https://www.bankofcanada.ca/valet/observations/{code}/json",
                 params={"recent": 5}, timeout=20)
r.raise_for_status()
p = r.json()

print(p["seriesDetail"][code]["label"])
for obs in p["observations"]:
    print(obs["d"], f"{float(obs[code]['v']):.4f}")

8.4 Exploring the Valet API (30 min)

The Valet API has several useful endpoints beyond single-series observations. Create day08/valet_explore.py.

Useful query parameters: start_date, end_date (YYYY-MM-DD) and recent (last n observations).

Example 1 — Searching the series catalogue

How did the notebook authors know V39079 is the overnight rate? They searched the catalogue:

import requests

BASE = "https://www.bankofcanada.ca/valet"

r = requests.get(f"{BASE}/lists/series/json", timeout=60)
r.raise_for_status()
catalogue = r.json().get("series", {})
print(f"{len(catalogue):,} series available")

keyword = "overnight"
for code, info in catalogue.items():
    label = info.get("label", "")
    if keyword.lower() in label.lower():
        print(f"{code:<25} {label}")

Try other keywords: "CPI", "USD", "10 year".

Example 2 — Several series in one request

codes = ["BD.CDN.2YR.DQ.YLD", "BD.CDN.5YR.DQ.YLD", "BD.CDN.10YR.DQ.YLD"]
r = requests.get(f"{BASE}/observations/{','.join(codes)}/json",
                 params={"recent": 3}, timeout=30)
r.raise_for_status()
p = r.json()

print(list(p["seriesDetail"].keys()))
print(p["observations"][-1])     # one date, all three series

Example 3 — A whole group (as in the notebook's Module 08)

r = requests.get(f"{BASE}/observations/group/BOND_YIELDS_BENCHMARK/json",
                 params={"recent": 2}, timeout=30)
r.raise_for_status()
g = r.json()

print(g.get("groupDetail", {}).get("label"))
for code, info in g["seriesDetail"].items():
    print(f"{code:<25} {info['label']}")

Example 4 — Series metadata

r = requests.get(f"{BASE}/series/V39079/json", timeout=20)
print(json.dumps(r.json(), indent=2)[:800])

Documentation habit: when you add a series to a project, record its code, label and source URL. Six months later, V122531 means nothing to anyone.

8.5 From JSON to a Tidy DataFrame (20 min)

Valet responses are wide: each observation holds one date and several series. We want a long (tidy) table with one row per date and series. Long format is how databases and dbt (Days 31–40) prefer data.

Valet JSON (wide)                         Tidy (long)
{"d": "2026-01-05",                       date        series_id            value
 "BD.CDN.2YR.DQ.YLD": {"v": "2.27"},  →   2026-01-05  BD.CDN.2YR.DQ.YLD    2.27
 "BD.CDN.10YR.DQ.YLD": {"v": "3.75"}}     2026-01-05  BD.CDN.10YR.DQ.YLD   3.75

Create day08/valet_to_df.py:

import pandas as pd
import requests


def parse_valet(payload):
    """Convert a Valet observations payload into a long DataFrame.

    Columns: date (datetime), series_id (str), value (float).
    Unusable values ('', 'NA', 'Bank holiday', missing keys) are dropped.
    """
    series_ids = list(payload.get("seriesDetail", {}).keys())
    rows = []
    for obs in payload.get("observations", []):
        for sid in series_ids:
            cell = obs.get(sid)
            if isinstance(cell, dict):
                rows.append({"date": obs.get("d"), "series_id": sid,
                             "value": cell.get("v")})

    df = pd.DataFrame(rows, columns=["date", "series_id", "value"])
    df["date"] = pd.to_datetime(df["date"])
    df["value"] = pd.to_numeric(df["value"], errors="coerce")
    return df.dropna(subset=["value"]).reset_index(drop=True)


codes = ["V39079", "BD.CDN.2YR.DQ.YLD", "BD.CDN.10YR.DQ.YLD"]
r = requests.get(
    f"https://www.bankofcanada.ca/valet/observations/{','.join(codes)}/json",
    params={"start_date": "2025-01-01"}, timeout=30)
r.raise_for_status()

long_df = parse_valet(r.json())
print(long_df.head())
print(long_df.groupby("series_id")["value"].agg(["count", "last"]))

# Wide again whenever you need it (Day 6)
wide = long_df.pivot(index="date", columns="series_id", values="value")
print(wide.tail())

Notice the separation: fetching (network) and parsing (pure data transformation) are different steps. A parser that takes a plain dictionary can be tested without internet access. We rely on this on Day 10.

8.6 Mini-Project: A Reusable fetch_valet Function (15 min)

Create day08/boc_fetch.py with a function:

def fetch_valet(series_ids, start_date=None, end_date=None, recent=None):
    ...

Requirements:

1. Accept one code (a string) or several (a list).
2. Only include the parameters that were actually given.
3. Use a timeout and raise_for_status().
4. Return the long DataFrame from parse_valet.
5. Test it: overnight rate since 2022; the three yields for the last 10 days.

import requests
from valet_to_df import parse_valet   # reuse the parser from 8.5

BASE = "https://www.bankofcanada.ca/valet"


def fetch_valet(series_ids, start_date=None, end_date=None, recent=None):
    """Fetch one or more BoC Valet series as a long DataFrame."""
    if isinstance(series_ids, str):
        series_ids = [series_ids]

    params = {}
    if start_date:
        params["start_date"] = start_date
    if end_date:
        params["end_date"] = end_date
    if recent:
        params["recent"] = recent

    url = f"{BASE}/observations/{','.join(series_ids)}/json"
    r = requests.get(url, params=params, timeout=30)
    r.raise_for_status()
    return parse_valet(r.json())


if __name__ == "__main__":
    print(fetch_valet("V39079", start_date="2022-01-01").tail())
    yields = ["BD.CDN.2YR.DQ.YLD", "BD.CDN.5YR.DQ.YLD", "BD.CDN.10YR.DQ.YLD"]
    print(fetch_valet(yields, recent=10))

Compare with the notebook's boc_fetch: yours handles several series per call, returns long format, and separates parsing from fetching. What does the notebook's version do that yours doesn't? (It prints a summary line and names the value column. You'll add logging on Day 9.)

Day 8 checklist

• I can read a URL's parts and interpret status codes
• I use params= and timeout= on every request
• I can explore unfamiliar JSON step by step and navigate it safely with .get
• I can search the Valet catalogue and fetch several series or a whole group
• I keep fetching and parsing in separate functions

git add .
git commit -m "Day 8: HTTP, JSON, BoC Valet extraction"
git push