From 72000ebfa1bf260b09b21eee56a4f4116dab7b85 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 1 Sep 2026 09:28:44 +1000 Subject: [PATCH 1/3] P4 groundwork: manifest, four-stage builder and provenance/ for business_cycle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the Phase 6 backfill and settles #13, as the on-ramp to the P4 dynamic-snapshot pilot. - lectures/business_cycle_data.csv.yml: the last unmanifested dataset gets its manifest — class dynamic-snapshot, cadence annual, the first `pattern` column entry (YR\d{4}), consumers empty (intro's business_cycle still makes the live wbgapi call the builder reproduces; adoption is the P4 decision). integrity.upstream is `diverged` / `upstream-moved`: a full re-run against live WDI on 2026-09-01 revised 236 of 320 overlap cells (max 1.5 pp) and appended YR2024-YR2025. Register entry on #39 to follow. - builders/business_cycle.py: fetch -> pre-process -> validate -> write. validate() asserts the grid, the five economies, percent units, the one structural null and recency, and bounds the overlap window against the committed snapshot at 5 pp with a printed revision summary — equality is the wrong test for a revised aggregate. --out-dir gives a dry run. Not run into lectures/: the bytes are unchanged (a migration moves bytes). - provenance/: the two World Bank metadata dumps leave the published tree (#13, option 3). Not served, no manifests, regenerated every run; README says why they exist and what they are evidence for. - migration.yml record (landed, P4), manifest-schema.yml documents the `pattern` form, CATALOG.md regenerated, and the counts in AGENTS.md, builders/README.md, requirements.txt and four PLAN.md boxes updated. Gates: check_consumed_files 41 manifests / 0 errors; strict audit problem set unchanged against the same clones; builder dry run passes on live WDI. See #13, #14, #39. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 8 +- CATALOG.md | 3 +- PLAN.md | 8 +- builders/README.md | 14 +- builders/business_cycle.py | 180 ++++++++++++++++-- lectures/business_cycle_data.csv.yml | 175 +++++++++++++++++ manifest-schema.yml | 8 +- migration.yml | 17 ++ provenance/README.md | 26 +++ .../business_cycle_info.md | 0 .../business_cycle_metadata.md | 0 requirements.txt | 4 +- 12 files changed, 411 insertions(+), 32 deletions(-) create mode 100644 lectures/business_cycle_data.csv.yml create mode 100644 provenance/README.md rename {lectures => provenance}/business_cycle_info.md (100%) rename {lectures => provenance}/business_cycle_metadata.md (100%) diff --git a/AGENTS.md b/AGENTS.md index ae6e61a..360104b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -112,7 +112,7 @@ The limits: **50 MiB** warns on push, **100 MiB** (104,857,600 B) is a hard bloc **One builder per published dataset, in `builders/`, named for the dataset it produces:** `builders/.` builds `lectures/.`. The stem is the dataset's, not the lecture's — `builders/japan_earthquakes.py` writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` field predictable and lets CI assert it. -Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — `business_cycle.py` writes three. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. +Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — the SCF and Forbes builders each write two. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. A builder's **provenance byproducts** — upstream metadata dumps that are not datasets — go to `provenance/`, never `lectures/`: `business_cycle.py` writes one dataset there and two dumps here. `scripts/` is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart. @@ -154,12 +154,14 @@ The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/Quan ``` lectures/ # the published tree — flat, live on Pages; read via raw URLs # today, qeld.url() once the package ships (PLAN-QELD-PACKAGE.md) - # 21 files, 18 with manifests (business_cycle's three still - # need theirs — see #13). Manifests are sidecars: .yml + # 41 datasets, 41 manifests (complete since 2026-09-01). + # Manifests are sidecars: .yml builders/ # one builder per published dataset — NOT published # builders/.py builds lectures/. sources/ # inputs a builder cannot re-fetch — NOT published, per-path LFS # no manifests; sources/README.md is the audit trail +provenance/ # upstream metadata dumps a builder writes beside its data — + # NOT published, no manifests, regenerated every run (#13) scripts/ # repo tooling — NOT published, produces no dataset build_catalog.py # generates CATALOG.md from the manifests build_audit.py # the audit dashboard: scan lecture repos → audit.json → site/ diff --git a/CATALOG.md b/CATALOG.md index 2b43413..47f2c40 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -6,7 +6,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**40 datasets** · 40 read by lectures today · 113.1 MB total · 34 permitted / 6 restricted redistribution +**41 datasets** · 40 read by lectures today, 1 awaiting repoint · 113.2 MB total · 35 permitted / 6 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | @@ -18,6 +18,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)
French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)
[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)
⚠️ BROKEN reader (measured 2026-08-19): fetches this dataset through a stale `base_url` still pointing at lecture-python-intro's deleted `datasets/` copy (french_rev.md:70-75), which serves 404 | | [**bbh_macro_quarterly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_macro_quarterly.csv)
Bhandari-Borovička-Ho replication — quarterly US macro series for the belief-wedge VAR, 1955Q1-2019Q4 | constructed | [Replication package for "Survey data and subjective beliefs in business cycle models" (Bhandari, Borovička and Ho), file `data input/FRED/data_FRED.xlsx`](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 31.5 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) | | [**bbh_michigan_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_michigan_monthly.csv)
Michigan Surveys of Consumers monthly aggregates and the US unemployment rate, 1978-01 to 2020-03 (BBH replication extract) | constructed | [Bhandari, Borovička and Ho replication package (Zenodo), carrying University of Michigan Surveys of Consumers published aggregates and a US Bureau of Labor Statistics series retrieved via FRED](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ⚠️ restricted | ✅ verified | ✅ committed | 11.9 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) | +| [**business_cycle_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/business_cycle_data.csv)
World Bank GDP growth (annual %) — USA, ARG, GBR, GRC, JPN, 1960 to 2023 | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ⇄ diverged | ✅ committed | 5.8 KB | — | | [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)
French Revolution — monthly specie value of the assignat, 1791-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)
⚠️ Reads a local `datasets/` copy, not this file
[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)
⚠️ Course fork with a live Pages site; reads its own blob-identical `datasets/` copy (french_rev.md:715-716), not this file, and its `base_url` still points at lecture-python-intro
[python-lecture-sandpit.myst · french_rev.md](https://github.com/QuantEcon/python-lecture-sandpit.myst/blob/main/lectures/french_rev.md)
⚠️ Public sandpit holding `lectures/_static/` copies | | [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)
The Ends of Four Big Inflations — appendix tables, transcribed | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)
[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md)
[lecture-intro.zh-cn · inflation_history.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inflation_history.md)
[test-actions-lecture-intro · inflation_history.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inflation_history.md) | | [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)
World Population Review — Brazilian city populations, 2023 | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | diff --git a/PLAN.md b/PLAN.md index 441fe33..795e8c1 100644 --- a/PLAN.md +++ b/PLAN.md @@ -297,7 +297,7 @@ Full automation: - [x] Audit dashboard workflow ([#20](https://github.com/QuantEcon/data-lectures/issues/20), added 2026-07-17): `.github/workflows/audit-dashboard.yml` rebuilds the full-universe data audit + migration tracker from the 8 lecture repos' `main` (push to main / weekly / dispatch) and deploys it with the published tree to Pages. Strict mode fails the build on an unannotated data reference or a `migration.yml` status the scan contradicts - [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14) -- [ ] Retrofit `builders/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) +- [x] Retrofit `builders/business_cycle.py` to the four-stage builder contract — **done 2026-09-01**, with the two provenance dumps moved out of the published tree to `provenance/` ([#13](https://github.com/QuantEcon/data-lectures/issues/13)). Its `validate()` is the first to face a *revised* upstream: it bounds the overlap window (5 pp) and prints the revision summary rather than asserting equality, which is the review surface the refresh-as-PR workflow below will use. It previously had fetch/transform/write but no validate stage. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4) - [ ] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — relocates API fragility from 7 lecture repos' CI into one scheduled job here - [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list @@ -305,8 +305,8 @@ Full automation: ### Phase 6 — Metadata backfill for existing holdings -- [ ] Manifest per dataset for the files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below. **Largely done** — 43 non-`.yml` files, 40 manifests (2026-09-01); the only dataset still lacking one is `business_cycle_data.csv` (`business_cycle_info.md` and `business_cycle_metadata.md` are prose, not datasets, so the gap is 1 file and not 3) -- [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared +- [x] Manifest per dataset for the files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below. **Complete 2026-09-01** — 41 datasets, 41 manifests; the last was `business_cycle_data.csv`, and the two `business_cycle` `.md` dumps left `lectures/` for `provenance/` the same day (`business_cycle_info.md` and `business_cycle_metadata.md` are prose, not datasets, so the gap is 1 file and not 3) +- [x] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot — `class: dynamic-snapshot`, `cadence: annual`, declared 2026-09-01 - [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand **Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). Inherited data — anything the lecture repos already serve publicly — migrates with its licence recorded **as found**, including `redistribution: restricted` and `name: null` where that is the honest answer. Moving the same bytes to a canonical host with better provenance and an explicit licence field improves on the status quo, so the migration does not wait on review; what needs further thought is tracked in [#35](https://github.com/QuantEcon/data-lectures/issues/35) with alternatives, and resolved before this repo is ever promoted as a branded public open-data host. That promotion is the gate, not each file's move — and with the custom domain deferred indefinitely (2026-08-12, D11), no such promotion is scheduled: the #35 inventory stays open and the gate binds only if a public host is someday established after all. This generalises the exception AGENTS.md already carried for `countries.csv`, and applies to **inherited** data only — a genuinely new dataset still establishes its licence before it lands @@ -321,7 +321,7 @@ Verify that what this repo holds is actually the data it claims to be — agains - [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8. Recorded **in the repoint PR** as a one-time gate, reproducible later from the manifest's `sha256` — not a manifest field (P1 decision) - [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record `sha256`, `status`, what it was compared `against`, and the date in the manifest's `integrity.upstream` -- [ ] **Constructed / dynamic files**: re-run the committed builder (`builders/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot +- [x] **Constructed / dynamic files**: re-run the committed builder (`builders/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot — **done 2026-09-01, and they do not agree, by design**: the World Bank revised 236 of 320 overlap cells (max 1.5 pp) and appended two years. Recorded as `diverged` / `upstream-moved` in the manifest and in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39); the finding is what set the builder's 5 pp overlap bound - [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so (`status: unverifiable` with a one-line `note` — the honest known status, per P1) - [ ] **Unverifiable or failing files**: flag in the manifest and open an issue — do not promote a file to the canonical URL namespace with a known-bad or unknown integrity status diff --git a/builders/README.md b/builders/README.md index 9876e6d..874d0c6 100644 --- a/builders/README.md +++ b/builders/README.md @@ -12,8 +12,10 @@ writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` field predictable and lets CI assert it. **Where one builder produces a set of files**, name it for the set and let each -file's manifest point at the same builder path. `business_cycle.py` writes three -files; the SCF and Forbes builders each write two. The rule is the default, not +file's manifest point at the same builder path. The SCF and Forbes builders +each write two. Provenance byproducts — upstream metadata a builder dumps +beside its dataset — go to `provenance/`, not `lectures/`; `business_cycle.py` +writes one dataset and two such dumps. The rule is the default, not an invariant — what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. @@ -46,7 +48,7 @@ re-fetched** — see `AGENTS.md`. | `fred_data.py` | `fred_data.csv` | committed — fetches six FRED series live over a pinned 1953-04..2024-12 window (yields and the recession dummy are stable history, unlike the BBH national-accounts snapshot). Reproduces its output byte for byte (2026-08-18) | | `hansen_singleton_1982_data.py` | `hansen_singleton_1982_data.csv` | committed — fetches FRED and the Ken French factors live. Reproduces its output byte for byte (2026-08-13) | | `hansen_singleton_1983_data.py` | `hansen_singleton_1983_data.csv` | committed — the same construction plus a T-bill leg, so its output is a strict superset of the 1982 file's. Reproduces its output byte for byte (2026-08-13) | -| `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | run by hand, no validate stage yet (PLAN Phase 5); its three outputs are the repo's only unmanifested files | +| `business_cycle.py` | `business_cycle_data.csv` (plus two dumps to `provenance/`) | committed — the repo's one **dynamic snapshot** (`cadence: annual`), retrofitted to the four-stage contract 2026-09-01. Fetches live WDI; does NOT reproduce its bytes and is not meant to — the World Bank revises the series (63 of 64 year columns moved between the 2025-02 vintage and 2026-09-01). validate() bounds the overlap window at 5 pp and prints the revision summary, which is the review surface for a refresh PR | | `webscrape_forbes.ipynb` | `forbes-global2000.csv`, `forbes-billionaires.csv` | **committed-frozen** — an undocumented Forbes API, a spoofed user-agent and hardcoded GDPR consent cookies. Defects recorded in the two manifests rather than fixed | | `generating_mini.md` | `SCF_plus_mini.csv`, `SCF_plus_mini_no_weights.csv` | **committed-frozen** — its `to_csv` calls are commented out upstream and stay that way. As written it still fetches the `high_dim_data` URL; that URL is historical, and the input is now committed at `sources/SCF_plus.dta`. See `sources/README.md` | | `usa-gini-nwealth-tincome-lincome.ipynb` | `usa-gini-nwealth-tincome-lincome.csv` | **committed-frozen** — three independent reasons, any one sufficient: no validate stage; it raises under the pinned pandas 3 (`np.asarray` of a Series is read-only under copy-on-write, so `rd.shuffle` fails — the lecture got the `.copy()` fix in QuantEcon/lecture-python-intro#776, this notebook did not); and it is non-deterministic, so it cannot reproduce its own bytes. It is also the only builder here whose input is **another file in this repo** | @@ -78,9 +80,9 @@ fewer than the datasets because `generating_mini.md` and `webscrape_forbes.ipynb` each produce two. The remaining **10** have none: they carry `builder_status: unrecovered` in their manifests, which is the Phase 9 recovery backlog, kept visible rather than hidden by reclassifying the file as -`verbatim`. The table above lists a **17th** builder, `business_cycle.py`, which -no manifest references — its three outputs are the repo's only unmanifested -files. +`verbatim`. The table above also lists `business_cycle.py`, the one builder +for a `dynamic-snapshot` dataset rather than a `constructed` one; every file in +`lectures/` has had a manifest since 2026-09-01. Repo tooling — the audit dashboard and the catalog generator — lives in `scripts/` and is not a builder. diff --git a/builders/business_cycle.py b/builders/business_cycle.py index 6a976e4..72db4d0 100644 --- a/builders/business_cycle.py +++ b/builders/business_cycle.py @@ -1,27 +1,177 @@ +#!/usr/bin/env python3 +""" +Builder for lectures/business_cycle_data.csv. + +Fetches annual real GDP growth (World Bank WDI series NY.GDP.MKTP.KD.ZG) for +five economies -- USA, ARG, GBR, GRC and JPN -- exactly as the intro +`business_cycle` lecture fetches it live with wbgapi, and writes it as one +wide CSV: one row per economy, one `YR` column per year from 1960. + +This is the repo's one DYNAMIC SNAPSHOT (`class: dynamic-snapshot`, +`cadence: annual` in the manifest). Unlike the frozen extracts beside it, the +World Bank revises this series continuously -- national-accounts rebasing +moves historical growth rates by up to about 1.5 percentage points -- so a +refresh is NOT expected to reproduce the committed bytes, and validate() does +not ask it to. Measured 2026-09-01: 63 of the 64 overlapping year columns had +at least one revised cell (236 of 320 cells; median change 0.0, 99th +percentile 1.0, maximum 1.5), and two new columns (YR2024, YR2025) had +appeared. That delta is recorded in the manifest's integrity.upstream block +and in the register at QuantEcon/data-lectures#39. + +What validate() DOES assert is the contract a consumer can rely on: the +shape, the column grid, the fixed set of economies, percent units, the one +structural null (YR1960, growth being undefined in the series' first year), +recency, and -- against the previously committed snapshot -- that no revision +exceeds MAX_REVISION percentage points and no previously populated cell has +gone empty. It prints the overlap-window summary on every run; that summary +is the review surface for a refresh PR (PLAN Phase 5). + +Two provenance dumps are written beside the data, to provenance/ (NOT +lectures/ -- they are not datasets; QuantEcon/data-lectures#13): the series +metadata, which is where the CC BY-4.0 licence the manifest cites is stated, +and the `wb.series.info(q='GDP growth')` listing the lecture teaches. + +Stages: fetch -> pre-process -> validate -> write. Writes only on validation +pass, so a failed refresh leaves the last-good snapshot in place. + +Usage: + python builders/business_cycle.py # refresh in place + python builders/business_cycle.py --out-dir D # write everything to D + # (a dry run; validates + # against lectures/ still) + +Requires pandas and wbgapi (requirements.txt). +""" +import argparse +import datetime as dt import os +import re + +import pandas as pd import wbgapi as wb CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__)) REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR) PUBLISHED_DIR = os.path.join(REPO_ROOT, 'lectures') +PROVENANCE_DIR = os.path.join(REPO_ROOT, 'provenance') + +OUT_FILE = 'business_cycle_data.csv' +METADATA_FILE = 'business_cycle_metadata.md' +INFO_FILE = 'business_cycle_info.md' + +SERIES = 'NY.GDP.MKTP.KD.ZG' +# The five economies the lecture plots, in the order it names them. wbgapi +# returns rows in its own order (JPN, GRC, GBR, ARG, USA on every run so far); +# that order is kept as returned so a refresh diff is values-only. +ECONOMIES = ['USA', 'ARG', 'GBR', 'GRC', 'JPN'] +FIRST_YEAR = 1960 +YEAR_COL = re.compile(r'^YR(\d{4})$') + +# Overlap-window policy for a revised aggregate. The World Bank's routine +# revisions to this series have measured at most 1.5 pp (2026-09-01, 2025-02 +# vintage vs live); a change past this bound is a rebasing, a units switch or +# an upstream defect, and wants a human before it ships. +MAX_REVISION = 5.0 # percentage points, any single cell +# Percent units: a fetch that came back as a ratio (0.02 for 2%) or as an +# index level would pass every shape check and rescale the lecture's figure. +MIN_ABS_MAX, MAX_ABS = 1.0, 50.0 +# A refresh whose newest year is older than this many years behind today +# means the fetch returned a stale or truncated panel. +MAX_STALENESS_YEARS = 2 + + +def fetch(): + """Live WDI, exactly the lecture's own three calls.""" + frame = wb.data.DataFrame(SERIES, ECONOMIES, labels=True) + metadata = str(wb.series.metadata.get(SERIES)) + info = str(wb.series.info(q='GDP growth')) + return frame, metadata, info + + +def pre_process(frame): + # wbgapi's index is the ISO3 code, named `economy`; `Country` is the label + # column `labels=True` adds. Keep the layout the lecture (and the + # committed file) uses: Country first, then the year columns ascending. + frame = frame.copy() + frame.index.name = 'economy' + years = sorted(c for c in frame.columns if YEAR_COL.match(c)) + return frame[['Country'] + years] + + +def _years(frame): + return [int(YEAR_COL.match(c).group(1)) for c in frame.columns if YEAR_COL.match(c)] + + +def validate(frame, previous=None): + """Refuse to write anything that is not the shape we expect.""" + # Grid: Country, then YR..YR with no gap. + assert list(frame.columns[:1]) == ['Country'], list(frame.columns[:3]) + years = _years(frame) + assert len(years) == len(frame.columns) - 1, 'non-year column present' + assert years[0] == FIRST_YEAR, years[0] + assert years == list(range(FIRST_YEAR, years[-1] + 1)), 'gap in the year grid' + year_cols = [f'YR{y}' for y in years] + + # Economies: exactly the five, one row each. + assert frame.index.name == 'economy' + assert sorted(frame.index) == sorted(ECONOMIES), sorted(frame.index) + assert frame['Country'].notnull().all() + + # Dtypes and units. + values = frame[year_cols] + assert all(pd.api.types.is_float_dtype(values[c]) for c in year_cols), 'non-float year column' + assert values.abs().max().max() >= MIN_ABS_MAX, 'values look like ratios, not percent' + assert values.abs().max().max() <= MAX_ABS, 'growth rate out of band' + + # The one structural null: growth is undefined in the series' first year. + nulls = values.isnull().sum() + assert dict(nulls[nulls > 0]) == {f'YR{FIRST_YEAR}': len(ECONOMIES)}, dict(nulls[nulls > 0]) + + # Recency. + assert years[-1] >= dt.date.today().year - MAX_STALENESS_YEARS, f'newest year is {years[-1]}' + + # Overlap window against the last-good snapshot: revisions are expected, + # bounded, and reported; a lost observation or a rescale is not. + if previous is not None: + prev_years = [f'YR{y}' for y in _years(previous)] + assert set(prev_years) <= set(year_cols), 'a year column disappeared' + assert sorted(previous.index) == sorted(frame.index), 'the economy set changed' + old = previous.loc[frame.index, prev_years] + new = frame.loc[frame.index, prev_years] + assert not (old.notnull() & new.isnull()).any().any(), 'a populated cell went empty' + diff = (old - new).abs() + changed = int((diff > 1e-9).sum().sum()) + worst = float(diff.max().max()) + print(f'overlap window {prev_years[0]}..{prev_years[-1]}: ' + f'{changed} of {diff.size} cells revised, max |change| {worst:.3f} pp; ' + f'new columns: {sorted(set(year_cols) - set(prev_years)) or "none"}') + assert worst <= MAX_REVISION, f'revision of {worst:.3f} pp exceeds {MAX_REVISION}' + -def run(): - series_info = wb.series.info(q='GDP growth') - file_name = 'business_cycle_info.md' - with open(os.path.join(PUBLISHED_DIR, file_name), 'w') as f: - f.write(str(series_info)) +def run(out_dir=None): + data_dir = out_dir or PUBLISHED_DIR + prov_dir = out_dir or PROVENANCE_DIR + previous_path = os.path.join(PUBLISHED_DIR, OUT_FILE) + previous = (pd.read_csv(previous_path, index_col=0) + if os.path.exists(previous_path) else None) - gdp_growth_df = wb.data.DataFrame('NY.GDP.MKTP.KD.ZG', - ['USA', 'ARG', 'GBR', 'GRC', 'JPN'], - labels=True) - file_name = 'business_cycle_data.csv' - gdp_growth_df.to_csv(os.path.join(PUBLISHED_DIR, file_name)) + frame, metadata, info = fetch() + frame = pre_process(frame) + validate(frame, previous) - series_metadata = wb.series.metadata.get('NY.GDP.MKTP.KD.ZG') - file_name = 'business_cycle_metadata.md' - with open(os.path.join(PUBLISHED_DIR, file_name), 'w') as f: - f.write(str(series_metadata)) + os.makedirs(data_dir, exist_ok=True) + os.makedirs(prov_dir, exist_ok=True) + frame.to_csv(os.path.join(data_dir, OUT_FILE)) + with open(os.path.join(prov_dir, METADATA_FILE), 'w') as f: + f.write(metadata) + with open(os.path.join(prov_dir, INFO_FILE), 'w') as f: + f.write(info) + years = _years(frame) + print(f'wrote {OUT_FILE}: {frame.shape[0]} economies x {len(years)} years ' + f'({years[0]} .. {years[-1]}) -> {data_dir}') if __name__ == '__main__': - run() + ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0]) + ap.add_argument('--out-dir', help='write outputs here instead of lectures/ and provenance/') + run(ap.parse_args().out_dir) diff --git a/lectures/business_cycle_data.csv.yml b/lectures/business_cycle_data.csv.yml new file mode 100644 index 0000000..a60e00e --- /dev/null +++ b/lectures/business_cycle_data.csv.yml @@ -0,0 +1,175 @@ +# Manifest for business_cycle_data.csv — the repo's one DYNAMIC SNAPSHOT, and +# the prototype for the P4 pilot (PLAN Phase 8: a maintained snapshot twin for +# lectures that fetch live API data). The file has sat here since the Feb 2025 +# migration (QuantEcon/data b857c5c, then the 2026-07-16 flatten in #10) +# without a manifest; this one closes the Phase 6 backfill. +# +# Read this before refreshing: the World Bank REVISES this series. A re-run of +# the builder is not expected to reproduce these bytes and validate() does not +# ask it to — see integrity.upstream. A refresh is a deliberate in-place +# update of a tracking snapshot, reviewed through the builder's overlap-window +# summary (PLAN Phase 5, refresh-as-PR), not a correction and not a new +# vintage under a new name. + +filename: business_cycle_data.csv +title: World Bank GDP growth (annual %) — USA, ARG, GBR, GRC, JPN, 1960 to 2023 +description: > + Annual real GDP growth (percent, constant local currency) for the United + States, Argentina, the United Kingdom, Greece and Japan, extracted from World + Bank WDI series NY.GDP.MKTP.KD.ZG in the wide layout wbgapi emits: one row + per economy (ISO3 code as the index, country name as `Country`), one + `YR` column per year from 1960. It is the snapshot twin of the live + `wb.data.DataFrame` call the intro `business_cycle` lecture makes; the call + is reproduced verbatim by the builder. + +# Constructed, tracking a moving source: the World Bank republishes and revises +# this series continuously, and the file is meant to be refreshed in place on +# the cadence below rather than frozen. +class: dynamic-snapshot + +source: + name: World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files) + series: NY.GDP.MKTP.KD.ZG + url: https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG + doi: null + version: > + Unknown for the committed bytes — WDI carries a database-level + `lastupdated` stamp but the export recorded none (see `retrieved`). The + live database at the 2026-09-01 check reported lastupdated 2026-07-13 and + had moved on from these bytes (see integrity.upstream). + citation: > + World Bank, World Development Indicators, series NY.GDP.MKTP.KD.ZG (GDP + growth, annual %), from World Bank national accounts data and OECD + National Accounts data files. Retrieved via wbgapi. + note: > + Five economies and the wide `YR` layout are the lecture's choices, + not the source's — the builder passes exactly the lecture's arguments + (`wb.data.DataFrame('NY.GDP.MKTP.KD.ZG', ['USA','ARG','GBR','GRC','JPN'], + labels=True)`). Row order is whatever wbgapi returns (JPN, GRC, GBR, ARG, + USA on every run so far) and is kept as returned so a refresh diff is + values-only. + +license: + name: CC BY-4.0 + url: https://datacatalog.worldbank.org/public-licenses#cc-by + redistribution: permitted + verified: 2026-09-01 + note: > + Stated by the source itself in the series metadata (`License_Type: + CC BY-4.0`, `License_URL` as above), which the builder dumps to + provenance/business_cycle_metadata.md on every run; re-read from the live + metadata on the verification date. World Bank open data under CC BY-4.0 + is the model case for this repo's cache-with-attribution policy + (AGENTS.md, "Licensing and attribution"). + +retrieved: null # The bytes were produced by the committed + # builder, in QuantEcon/data commit b857c5c of + # 2025-02-16 — but that is when QuantEcon ran + # the script and committed the output, and + # AGENTS.md says not to reconstruct a + # retrieval date from git history. The vintage + # is pinned by content instead: YR2023 is the + # last column and is populated for all five + # economies, which places the export after the + # WDI release that first carried 2023 growth, + # and integrity.upstream records how far the + # live series has moved since. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# No migration check applies: no lecture has ever read this file (it was +# "never adopted" — PLAN "The audit (2026-07-15)"), so there is no prior copy +# to byte-match and no figure a change here could alter today. + +integrity: + sha256: 20e928d444b749ae52bc6c29283c316fe0c7e9cc4a215a0303bdfb365a7bc38d + upstream: + # `diverged`, not `failing`: the builder was re-run in full against live + # WDI and its output differs from these bytes in the way a revised + # aggregate is expected to. For a dynamic snapshot this is the normal + # state between refreshes, not a defect — the refresh is the resolution, + # and it is a deliberate PR, not something this manifest asks for. + status: diverged + date: 2026-09-01 + against: builders/business_cycle.py # live WDI, lastupdated 2026-07-13 + note: > + Re-run of the builder against live WDI on 2026-09-01: same five rows, + same layout, same single structural null, but 236 of the 320 + overlapping cells revised and two new year columns. Expected for a + tracking snapshot; see `delta`. + delta_kind: upstream-moved + delta: > + Live WDI (lastupdated 2026-07-13) vs the committed file, measured + 2026-09-01 with wbgapi 1.0.12 / pandas 2.3.3. Shape 5x67 against 5x65: + YR2024 and YR2025 added, populated for all five economies. In the + 1960-2023 overlap, 63 of 64 year columns carry at least one revised + cell — 236 of 320 cells in all, median |change| 0.0 pp, 90th percentile + 0.29, 99th percentile 1.00, maximum 1.50 pp. Nulls unchanged (YR1960, + all five). No column renamed, no economy added or dropped, row order + unchanged. + delta_evidence: > + `upstream-moved` is the only kind possible here: the committed file is + the verbatim output of `wb.data.DataFrame(...).to_csv()` with no + post-processing, and the builder that wrote it is the one committed + beside it, so a local edit has nowhere to have happened. The pattern of + the delta — small revisions spread across the whole history, largest in + the most recent years, plus two appended years — is the signature of + national-accounts revision and rebasing (WDI aggregates on constant + 2015 prices, and member countries revise back-series), not of any + change on this side. The revisions are why validate() bounds the + overlap window at 5 pp rather than asserting equality. + register: QuantEcon/data-lectures#39 + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-09-01 (pandas 2.3.3) and asserted by +# validate() on every builder run. This is the first manifest to use a +# `pattern` column entry (manifest-schema.yml; QuantEcon/data-lectures#14): the +# year columns are one column per year and their count grows on every refresh, +# so enumerating them would hard-code an end year into a file whose purpose is +# to move. + +schema: + format: csv + columns: + - {name: economy, dtype: str, description: "ISO3 code — wbgapi's index, written as the CSV's first column; the lecture reads it with index_col=0"} + - {name: Country, dtype: str, description: "country name, the label column wbgapi adds with labels=True"} + - {pattern: 'YR\d{4}', dtype: float64, description: "annual real GDP growth, percent, one column per year on an unbroken grid from YR1960 to the newest year the source carries (YR2023 in the committed bytes). Observed -10.89 to 13.20"} + row_count_floor: 5 # exact by construction: the builder asks for + # five economies and validate() asserts the + # set. Columns, not rows, are what grow. + date_range: {start: 1960, end: 2023} # `end` is the newest year column in + # the COMMITTED bytes; a refresh PR + # updates it, which is what makes the + # refresh diff self-describing. + + # Growth is undefined in the series' first year, so YR1960 is empty for all + # five economies and nothing else is. validate() asserts exactly this hole. + known_nulls: {YR1960: 5} + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# None. No lecture reads these bytes: intro's business_cycle.md (lines 83, 109 +# and 512 on 2026-09-01) and its lecture-wasm mirror make the live wbgapi call +# the builder reproduces, and the org-wide code search for this basename on +# 2026-09-01 returned only this repo. Whether those lectures adopt the snapshot +# — and whether the live call stays as the lesson, with the snapshot as the +# wasm twin (AGENTS.md, "Live APIs") — is the P4 decision (PLAN Phase 8), not +# something this manifest settles. When a consumer is added, `cadence` below +# becomes load-bearing. +consumers: [] + +builder: builders/business_cycle.py +builder_status: committed # four-stage since 2026-09-01 (PLAN Phase 5); + # `committed` here asserts the builder RUNS and + # validates, not that it reproduces the bytes — + # a dynamic snapshot is not meant to. + +# Dynamic snapshots only. The series is annual; the World Bank revises it +# several times a year but the lecture's figures change materially only when +# a new year lands (WDI typically carries year T from mid T+1). +cadence: annual diff --git a/manifest-schema.yml b/manifest-schema.yml index fd9529b..b772e7b 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -168,7 +168,13 @@ schema: columns: - {name: economy, dtype: string, description: ISO3 country code} - {name: Country, dtype: string, description: Country name} - - {name: YR1960, dtype: float, description: Annual GDP growth (%), one column per year} + # A column entry carries EITHER `name` (one exact column) OR `pattern` (a + # regex matching a run of columns), never both. `pattern` is for files + # whose column set grows — one column per year here — where enumerating + # them would hard-code an end year into a file whose purpose is to move. + # First used by business_cycle_data.csv.yml (2026-09-01); proposed in + # QuantEcon/data-lectures#14. + - {pattern: 'YR\d{4}', dtype: float64, description: Annual GDP growth (%), one column per year from YR1960} row_count_floor: 5 # PR validation fails below this. For a # frozen historical extract set it to the # exact row count by design (P1 lingcod = diff --git a/migration.yml b/migration.yml index 7ceaac4..0aae525 100644 --- a/migration.yml +++ b/migration.yml @@ -830,6 +830,23 @@ datasets: date: 2026-08-18 cutover: null + # The dynamic-snapshot prototype. The bytes have been in the published tree + # since the 2026-07-16 flatten (#10; originally QuantEcon/data b857c5c, + # 2025-02-16) with no manifest and no consumer — intro's business_cycle + # still makes the live wbgapi call the builder reproduces. `landed` records + # the manifest; whether the lecture adopts the snapshot is the P4 decision. + business_cycle_data.csv: + pilot: P4 + status: landed + prior_pattern: null # never read from anywhere — the lecture + # fetches live (api pattern), so there is no + # prior file read to migrate + landed: + pr: QuantEcon/data-lectures#10 + date: 2026-07-17 + repoints: [] + cutover: null + # Planned waves that have not landed anything here yet. `datasets` names the # files as the audit sees them today, so the dashboard can join the two views. # `title` is the reader-facing milestone name (the dashboard is read by people diff --git a/provenance/README.md b/provenance/README.md new file mode 100644 index 0000000..7c1231b --- /dev/null +++ b/provenance/README.md @@ -0,0 +1,26 @@ +# provenance + +Upstream metadata dumps that a builder writes **alongside** its dataset but +that are not datasets themselves. Nothing here is served: the Pages job +assembles `_site` from `site/`, `lectures/` and `audit.json` only, and no +lecture reads a file from this directory. + +The distinction this directory exists to keep is between a **published +dataset** (`lectures/`, with a sidecar manifest, a class, a licence and +a filename that is an API) and a **provenance byproduct** — the raw record of +what the upstream said about itself on the day a builder ran. The two +`business_cycle` dumps below lived in `lectures/` from the 2026-07-16 flatten +until they were moved here (QuantEcon/data-lectures#13): served at public +URLs, indistinguishable from datasets to anyone browsing the tree, and +failing every rule a dataset must meet. + +| File | Written by | What it is | +| --- | --- | --- | +| `business_cycle_metadata.md` | `builders/business_cycle.py` | `wb.series.metadata.get('NY.GDP.MKTP.KD.ZG')` — the World Bank's own record for the series: definition, source, periodicity and the `License_Type: CC BY-4.0` / `License_URL` fields that `lectures/business_cycle_data.csv.yml` cites | +| `business_cycle_info.md` | `builders/business_cycle.py` | `wb.series.info(q='GDP growth')` — the fuzzy-search listing the consuming lecture teaches; kept because the builder reproduces the lecture's own query, not because anything reads it | + +Files here carry no manifest and no hash gate. They are **regenerated on every +builder run** and are expected to drift: between the committed dump of +2025-02 and a run on 2026-09-01 the World Bank rewrote the metadata prose and +added a `Dataset: WB_WDI` line. The manifest's typed `source` and `license` +fields are the durable record; a dump is the evidence they were read from. diff --git a/lectures/business_cycle_info.md b/provenance/business_cycle_info.md similarity index 100% rename from lectures/business_cycle_info.md rename to provenance/business_cycle_info.md diff --git a/lectures/business_cycle_metadata.md b/provenance/business_cycle_metadata.md similarity index 100% rename from lectures/business_cycle_metadata.md rename to provenance/business_cycle_metadata.md diff --git a/requirements.txt b/requirements.txt index aae4e80..c8e7353 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ wbgapi==1.0.12 -pandas==2.3.3 # imported by 9 of the 10 builders — every one except - # builders/business_cycle.py. Pinned so an +pandas==2.3.3 # imported by every .py builder (business_cycle.py joined + # them with its validate() stage, 2026-09-01). Pinned so an # integrity.upstream.status of `verified` names a # reproducible environment rather than whatever python is # ambient (QuantEcon/data-lectures#85). This is the From 3770c7e8098e99d2557576bc985001ab034ecfcf Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 1 Sep 2026 10:05:11 +1000 Subject: [PATCH 2/3] AGENTS: who gets told when a snapshot breaks, refreshes, or changes shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three interfaces, three audiences. An upstream interface change fails the builder's validate(), writes nothing, and opens an issue here — consumers are unaffected by construction and the change is absorbed in pre_process. A successful refresh reaches each consumer per a new per-consumer `on_refresh` field (rebuild | review), sketched in manifest-schema.yml for the Phase 5 fan-out. A deliberate published-schema change is a new filename plus a hand-opened issue in every consumer repo. Recorded ahead of the workflows so they are written to a stated policy. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 12 ++++++++++++ manifest-schema.yml | 15 +++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 360104b..9487e23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -124,6 +124,18 @@ Builders follow four stages — **fetch → pre-process → validate → write** Live API calls are for *teaching data access*, not for getting data. Don't propose "the lecture should just call the API" as a fix — the fix is a snapshot here plus an automated refresh. +### Refresh, break, or schema change — who gets told + +A dynamic snapshot has three interfaces, and each failure mode has a different audience. Settled 2026-09-01 while retrofitting `business_cycle.py`; the workflows that act on it are PLAN Phase 5. + +| What happened | Detected by | Who is told | Consumer code changes? | +| --- | --- | --- | --- | +| **Upstream interface changed** (a renamed column, a dropped series, a units switch) | the builder's `validate()` fails; nothing is written, the last-good snapshot stays | an issue **in this repo** (the sources-alive canary). Consumers are unaffected by construction | **No.** Absorb it in the builder's `pre_process` stage so the published schema is unchanged — that adapter logic is where upstream churn is supposed to live | +| **Successful refresh** — schema intact, values revised | the refresh lands as a PR here whose body is `validate()`'s overlap summary | on merge, each repo in the manifest's `consumers` list, per its `on_refresh` (`manifest-schema.yml`): `rebuild` dispatches a build, `review` opens an issue there with the summary, for a lecture whose prose narrates a number | No, but figures and narrated numbers may need an author's eye | +| **Published schema changes deliberately** — the upstream change cannot honestly be absorbed | a decision, not a detection | an issue in every `consumers[].repo`, opened by hand as the invitation to opt in | **Yes, on the consumer's schedule:** new filename per "Corrections vs vintages"; the old file stays valid | + +The case none of this covers is a lecture where **the API call is the lesson** (`business_cycle` teaches `wb.series.info`): an upstream change to the call itself must reach the lecture, because the code on the page is the content. A snapshot twin still belongs beside it — as the `lecture-wasm` read and the fallback — but the canary only makes the break heard sooner. + ### Licensing and attribution Because this repo is a **stability cache, not a content-distribution host** (see "What this repo is"), the licence question is *"is this source OK to cache and serve publicly, with attribution?"* — not *"may we republish this as our own?"*. Attribution to the upstream source is carried in every manifest (`source`: name, url, series, citation), and that is the primary obligation. diff --git a/manifest-schema.yml b/manifest-schema.yml index b772e7b..fa5ac3e 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -216,6 +216,21 @@ consumers: [] # The generated catalog surfaces `note` beside the consumer, so a reader of # CATALOG.md's "Used by" column cannot mistake a local-copy reader for a # repointed one. + # + # Dynamic snapshots only (sketched 2026-09-01, AGENTS.md "Refresh, break, or + # schema change"; acted on by the Phase 5 consumer fan-out, not yet built). + # What a merged refresh does for this consumer: + # rebuild dispatch the repo's build; the figures re-render and nothing + # on the page narrates a specific number + # review open an issue in the repo carrying the builder's overlap + # summary, because prose quotes a value or a row count that the + # refresh may have moved + # Omit for static datasets. A refresh never changes the published schema + # (that is a new filename), so this routes VALUE changes only. + # + # - repo: QuantEcon/lecture-python-intro + # file: lectures/business_cycle.md + # on_refresh: review # --------------------------------------------------------------------------- # Builder — required for constructed and dynamic-snapshot; omit for verbatim From 57106d8d7e8a6dd75e2971f6801e4f10866a2cda Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 1 Sep 2026 10:19:15 +1000 Subject: [PATCH 3/3] Copilot review on #109: landing PR is #109, and writes are atomic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit migration.yml: `landed` means file + manifest by the file's own lifecycle definition, and the manifest arrived with #109, not with the #10 flatten that placed the bytes — the record now says so. business_cycle.py: every write goes through a same-directory temp file and os.replace(), so an interrupted run cannot leave a truncated file where the last-good snapshot was — the docstring made that promise and the direct to_csv() did not keep it. Co-Authored-By: Claude Fable 5 --- builders/business_cycle.py | 21 +++++++++++++++------ migration.yml | 13 +++++++------ 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/builders/business_cycle.py b/builders/business_cycle.py index 72db4d0..3cbad00 100644 --- a/builders/business_cycle.py +++ b/builders/business_cycle.py @@ -32,7 +32,9 @@ and the `wb.series.info(q='GDP growth')` listing the lecture teaches. Stages: fetch -> pre-process -> validate -> write. Writes only on validation -pass, so a failed refresh leaves the last-good snapshot in place. +pass, and each write goes through a temp file and os.replace(), so neither a +failed refresh nor an interrupted one can leave anything but the last-good +snapshot in place. Usage: python builders/business_cycle.py # refresh in place @@ -148,6 +150,15 @@ def validate(frame, previous=None): assert worst <= MAX_REVISION, f'revision of {worst:.3f} pp exceeds {MAX_REVISION}' +def _atomic_write(path, text): + """Write via a same-directory temp file and os.replace(), so an interrupted + run cannot leave a truncated file where the last-good snapshot was.""" + tmp = path + '.tmp' + with open(tmp, 'w') as f: + f.write(text) + os.replace(tmp, path) + + def run(out_dir=None): data_dir = out_dir or PUBLISHED_DIR prov_dir = out_dir or PROVENANCE_DIR @@ -161,11 +172,9 @@ def run(out_dir=None): os.makedirs(data_dir, exist_ok=True) os.makedirs(prov_dir, exist_ok=True) - frame.to_csv(os.path.join(data_dir, OUT_FILE)) - with open(os.path.join(prov_dir, METADATA_FILE), 'w') as f: - f.write(metadata) - with open(os.path.join(prov_dir, INFO_FILE), 'w') as f: - f.write(info) + _atomic_write(os.path.join(data_dir, OUT_FILE), frame.to_csv()) + _atomic_write(os.path.join(prov_dir, METADATA_FILE), metadata) + _atomic_write(os.path.join(prov_dir, INFO_FILE), info) years = _years(frame) print(f'wrote {OUT_FILE}: {frame.shape[0]} economies x {len(years)} years ' f'({years[0]} .. {years[-1]}) -> {data_dir}') diff --git a/migration.yml b/migration.yml index 0aae525..0cc1689 100644 --- a/migration.yml +++ b/migration.yml @@ -830,11 +830,12 @@ datasets: date: 2026-08-18 cutover: null - # The dynamic-snapshot prototype. The bytes have been in the published tree + # The dynamic-snapshot prototype. The BYTES have been in the published tree # since the 2026-07-16 flatten (#10; originally QuantEcon/data b857c5c, - # 2025-02-16) with no manifest and no consumer — intro's business_cycle - # still makes the live wbgapi call the builder reproduces. `landed` records - # the manifest; whether the lecture adopts the snapshot is the P4 decision. + # 2025-02-16), but `landed` means file + manifest (see the lifecycle above), + # and the manifest arrived with #109 — so that is the landing PR. No + # consumer: intro's business_cycle still makes the live wbgapi call the + # builder reproduces; whether it adopts the snapshot is the P4 decision. business_cycle_data.csv: pilot: P4 status: landed @@ -842,8 +843,8 @@ datasets: # fetches live (api pattern), so there is no # prior file read to migrate landed: - pr: QuantEcon/data-lectures#10 - date: 2026-07-17 + pr: QuantEcon/data-lectures#109 + date: 2026-09-01 repoints: [] cutover: null