Scheduling and Deployments

Time	Block
0:00 – 0:18	56.1 From scripts to deployments
0:18 – 0:38	56.2 Quick scheduling with serve()
0:38 – 0:45	Break
0:45 – 1:08	56.3 When should the refresh run? Cron, time zones and release calendars
1:08 – 1:43	56.4 Work pools, workers and prefect.yaml
1:43 – 1:50	Break
1:50 – 2:10	56.5 Running deployments on demand
2:10 – 2:30	56.6 Keeping it alive: what "production" needs beyond a Codespace

---

56.1 From Scripts to Deployments (18 min)

So far you've started flows by running Python. A deployment tells Prefect how and when to run a flow without you:

Concept	Meaning	Our example
Deployment	a flow + its default parameters + schedules + where to run	daily-refresh/prod
Schedule	when to create runs automatically	weekdays 17:30 Toronto time
Work pool	a queue of scheduled runs waiting for somewhere to execute	local-process
Worker	a long-running process that picks runs from a pool and executes them	prefect worker start in your Codespace

Two ways to get there:

flow.serve()	prefect deploy + work pool + worker
one command; the serving process runs the flows	deployments defined in prefect.yaml; workers run them
perfect for learning and small setups	the standard for teams and real infrastructure
stops when the process stops	workers can run on different machines, in containers, in the cloud

We'll do both: serve() first to see schedules working, then the full setup.

---

56.2 Quick Scheduling with serve() (20 min)

Create orchestration/flows/serve_local.py:

"""Serve deployments from this process (learning setup)."""
from prefect import serve

from orchestration.flows.cpi_briefing import cpi_briefing
from orchestration.flows.daily_refresh import daily_refresh

if __name__ == "__main__":
    briefing = cpi_briefing.to_deployment(
        name="cpi-briefing-every-15-min",
        interval=900,                          # seconds: every 15 minutes (demo only)
        parameters={"latest_n": 25},
    )
    refresh = daily_refresh.to_deployment(
        name="daily-refresh-demo",
        cron="*/30 * * * *",                   # every 30 minutes (demo only)
    )
    serve(briefing, refresh)

python -m orchestration.flows.serve_local

The process keeps running. In the UI:

• Deployments lists both deployments;
• each deployment's page shows upcoming scheduled runs;
• use Run → Quick run or Custom run (with different parameters) to start one now.

Watch a scheduled cpi-briefing run appear, execute (mostly Cached, thanks to Day 55), and complete. Stop the process with Ctrl+C: scheduled runs stop being executed, which is exactly the limitation 56.4 solves.

---

56.3 When Should the Refresh Run? (23 min)

A schedule is an economic decision: run when new data is likely to be available, soon enough to be useful.

When our sources publish

Source	Typical timing (check each provider's release calendar)	Consequence
BoC exchange rates and bond yields (Valet)	once per business day, late afternoon Eastern time	run after ~17:00 ET
FRED daily series (e.g. Treasury yields, WTI)	updated the following business day	yesterday's values arrive with the evening run
StatCan releases (CPI, LFS, GDP)	08:30 Eastern time on release days	a morning run shortly after 08:30 ET catches them

That suggests two runs on weekdays:

1. 09:15 ET: catches StatCan's 08:30 releases (WAP audits them; dashboards update the same morning).
2. 17:30 ET: catches BoC's end-of-day rates.

Thanks to change detection (Day 30) and fingerprints (Day 55), a run that finds nothing new is cheap, so scheduling generously costs little.

Cron in five fields

┌──────── minute (0–59)
│ ┌────── hour (0–23)
│ │ ┌──── day of month (1–31)
│ │ │ ┌── month (1–12)
│ │ │ │ ┌ day of week (0–6, Sunday = 0; 1–5 = Monday–Friday)
│ │ │ │ │
15 9 * * 1-5      → 09:15 every weekday
30 17 * * 1-5     → 17:30 every weekday
0 7 * * 1         → 07:00 every Monday

Time zones: the daylight-saving trap

Cron schedules default to UTC. Toronto is UTC−5 in winter and UTC−4 in summer, so a UTC cron of 15 13 * * 1-5 means 08:15 ET in winter (too early for 08:30 releases!) and 09:15 ET in summer. Always give schedules an explicit time zone, America/Toronto, and let Prefect handle daylight saving. That's what prefect.yaml does below.

Holidays

Cron doesn't know Canadian statutory holidays. On holidays our runs find no new data and finish quickly, which is harmless. Skipping them explicitly (e.g. with a calendar check at the start of the flow) is possible, but rarely worth the complexity. Note it as a conscious decision in the runbook.

---

56.4 Work Pools, Workers and prefect.yaml (35 min)

Step 1 — A work pool

A process work pool runs flows as local subprocesses, which suits a Codespace:

prefect work-pool create local-process --type process
prefect work-pool ls

Step 2 — prefect.yaml: deployments as code

Create prefect.yaml at the repository root:

# prefect.yaml — deployments for the macro data platform
name: macro-data-course
prefect-version: 3.0.0

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /workspaces/macro-data-course

deployments:
  - name: prod-morning
    description: "Weekday refresh after StatCan's 08:30 ET releases."
    entrypoint: orchestration/flows/daily_refresh.py:daily_refresh
    work_pool:
      name: local-process
      job_variables:
        env:
          PYTHONPATH: /workspaces/macro-data-course
    parameters: {}
    tags: [macro, prod]
    concurrency_limit: 1
    schedules:
      - cron: "15 9 * * 1-5"
        timezone: "America/Toronto"

  - name: prod-evening
    description: "Weekday refresh after the Bank of Canada's end-of-day rates."
    entrypoint: orchestration/flows/daily_refresh.py:daily_refresh
    work_pool:
      name: local-process
      job_variables:
        env:
          PYTHONPATH: /workspaces/macro-data-course
    parameters: {}
    tags: [macro, prod]
    concurrency_limit: 1
    schedules:
      - cron: "30 17 * * 1-5"
        timezone: "America/Toronto"

  - name: weekly-full-rebuild
    description: "Sunday rebuild ignoring fingerprints (catches anything skipped)."
    entrypoint: orchestration/flows/daily_refresh.py:daily_refresh
    work_pool:
      name: local-process
      job_variables:
        env:
          PYTHONPATH: /workspaces/macro-data-course
    parameters:
      force_transform: true
    tags: [macro, prod, weekly]
    concurrency_limit: 1
    schedules:
      - cron: "0 7 * * 0"
        timezone: "America/Toronto"

  - name: cpi-briefing
    description: "CPI briefing; cheap thanks to release-keyed caching."
    entrypoint: orchestration/flows/cpi_briefing.py:cpi_briefing
    work_pool:
      name: local-process
      job_variables:
        env:
          PYTHONPATH: /workspaces/macro-data-course
    parameters:
      latest_n: 25
    schedules:
      - cron: "45 8 * * 1-5"
        timezone: "America/Toronto"

Key points:

• entrypoint is path/to/file.py:flow_function.
• pull → set_working_directory runs flows from the repository root, so relative paths (ingest.py, macro_dbt/) work.
• PYTHONPATH lets the flow file import orchestration.*.
• concurrency_limit: 1 means at most one run of this deployment at a time. If the morning run is still going when the next is due, the new one waits instead of fighting over the DuckDB lock. (Runs of different deployments are coordinated on Day 57.)
• parameters override the flow's defaults for that deployment (the weekly rebuild forces dbt).

Step 3 — Deploy and start a worker

prefect deploy --all                      # registers all deployments from prefect.yaml
prefect deployment ls

In a third terminal (with .venv active, so the worker runs flows with the right libraries):

source .venv/bin/activate
prefect worker start --pool local-process

The worker polls the pool and executes runs when they're due. In the UI, open Work Pools → local-process to see the worker's status, and Deployments to see each deployment's upcoming runs, displayed in your time zone.

prefect.yaml is code: commit it. Schedules, parameters and concurrency rules are now reviewed and versioned like everything else, a pattern you've used for dbt (dbt_project.yml), dlt (config.toml) and GX (specs).

---

56.5 Running Deployments on Demand (20 min)

Schedules aren't the only way to trigger runs.

From the CLI

prefect deployment run 'daily-refresh/prod-evening'
prefect deployment run 'daily-refresh/prod-evening' --param force_transform=true
prefect deployment run 'cpi-briefing/cpi-briefing' --param latest_n=49

From the UI

Deployments → daily-refresh/prod-evening → Run → Custom run: the parameter form is generated from the flow's type hints (Day 51), so non-programmers can trigger a forced rebuild safely.

From Python (e.g. from another flow)

from prefect.deployments import run_deployment

run = run_deployment("daily-refresh/prod-evening",
                     parameters={"force_transform": True},
                     timeout=0)          # 0 = don't wait for it to finish

Pausing and resuming schedules

During maintenance (e.g. while restructuring dbt models), pause schedules without deleting anything:

prefect deployment schedule ls 'daily-refresh/prod-evening'
prefect deployment schedule pause 'daily-refresh/prod-evening' <schedule-id>
prefect deployment schedule resume 'daily-refresh/prod-evening' <schedule-id>

(or use the toggle on the deployment page in the UI).

Try it 56.5

Trigger prod-evening twice in quick succession from the CLI. What happens to the second run, and why? Where do you see it in the UI?

Answer

Because of concurrency_limit: 1, the second run waits (it's held back rather than executed) until the first finishes. It appears in the deployment's run list in a waiting state, then runs. Without the limit, both would start, and the second would likely fail on the DuckDB write lock, or worse, interleave writes.

---

56.6 Keeping It Alive: What "Production" Needs Beyond a Codespace (20 min)

Be honest about the environment: a Codespace stops after a period of inactivity, and when it stops, so do your Prefect server and worker. No schedule fires while it's asleep. For this course that's fine: schedules run while you work, and you've learned the real mechanics. For a real team:

Need	Options
always-on scheduler and UI	a small always-on VM or server running prefect server; or Prefect Cloud (managed server and UI)
always-on execution	a worker on that VM; or a managed/serverless work pool; or containers (Docker, Kubernetes pools)
the warehouse outside the Codespace	macro.duckdb on a persistent disk, or a hosted DuckDB-compatible service such as MotherDuck (a dlt/dbt destination change, Day 23)
secrets outside the Codespace	Prefect Secret blocks (Day 57) or the platform's secret manager

A lighter alternative for small projects is a scheduled CI job (e.g. GitHub Actions with a schedule: trigger) that checks out the repository, restores the warehouse from persistent storage, runs python -m orchestration.flows.daily_refresh, and saves the warehouse back. You lose the always-on UI but keep the code and flows unchanged.

The key point: your flows, deployments and policies don't change between these options. Only where the server, workers and warehouse live changes. That's the payoff of separating orchestration logic from infrastructure.

Record your chosen production setup (even if hypothetical) in orchestration/DEPLOYMENT.md, with what would run where and how secrets and the warehouse are stored.

Day 56 checklist

• I can explain deployments, schedules, work pools and workers, and when to use serve() vs prefect deploy
• Schedules follow the data: after StatCan's morning releases and the BoC's end-of-day rates, in America/Toronto time
• prefect.yaml defines prod, weekly-rebuild and briefing deployments with parameters and concurrency limits
• A worker executes scheduled runs from the local-process pool
• I can trigger, parameterise, pause and resume deployments from the CLI, the UI and Python
• I've documented what a real production setup needs beyond a Codespace

git add prefect.yaml orchestration && git commit -m "Day 56: deployments, schedules, work pool, worker" && git push