From b2743e19c13b765ad4cc5e0d5802e73dc3bb2bc6 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 7 Sep 2026 09:23:24 +1000 Subject: [PATCH 1/2] Record the four schema and naming decisions: rename, pattern capture groups, null placement, dtype sweep The decisions ratified 2026-09-07 on #120, #121, #122 and #113, written into the manifests and manifest-schema.yml so the shared validate() (#119) has a spec to read. - #113 naming: business_cycle_data.csv -> gdp_growth_annual.csv while its consumers list is still empty (the only free moment); the six rules sit beside `filename` in manifest-schema.yml and in AGENTS.md. Every reference to the old name follows (builder table and docstring, builders/README.md, migration.yml key, PLAN.md, audit prose, CATALOG.md regenerated). - #120 pattern: `YR\d{4}` -> `YR(\d{4})` in the three World Bank manifests; the capture group is what the validator derives date_range from. The semantics (fullmatch, ordered, contiguous, >=1 match, exhaustive) are the comment on `pattern`. - #121 nulls: integers stay exact; the four dynamic snapshots gain a `nulls:` placement block (along / leading / recent / ended / inner) mirroring what the two builders already assert; `known_nulls_total` leaves the three named- column manifests and stays legal only inside header-less sheet reads. - #122 dtype: 21 `string` + 9 `object` -> `str`, 2 `datetime` -> `datetime64`, forbes `government` -> `bool`; the closed vocabulary and the family-compare rule are the comment on `dtype`. Bytes unchanged: the renamed CSV hashes to the sha256 its manifest records. Co-Authored-By: Claude Fable 5.1 --- AGENTS.md | 6 +- CATALOG.md | 2 +- PLAN.md | 4 +- builders/README.md | 2 +- builders/business_cycle.py | 11 +- lectures/ames_house_prices.csv.yml | 2 +- lectures/countries.csv.yml | 22 ++-- lectures/employ.csv.yml | 12 +- lectures/epl_match_goals.csv.yml | 8 +- lectures/forbes-billionaires.csv.yml | 2 +- lectures/fred_data.csv.yml | 2 +- ...s_cycle_data.csv => gdp_growth_annual.csv} | 0 ...data.csv.yml => gdp_growth_annual.csv.yml} | 22 +++- lectures/japan_earthquakes.csv.yml | 2 +- lectures/private_credit_to_gdp.csv.yml | 10 +- lectures/realwage.csv.yml | 8 +- lectures/test_pwt.csv.yml | 4 +- lectures/unemployment_rate_annual.csv.yml | 26 +++-- lectures/us_adult_heights.csv.yml | 2 +- lectures/us_business_cycle_monthly.csv.yml | 38 ++++-- manifest-schema.yml | 109 +++++++++++++++--- migration.yml | 6 +- scripts/audit_annotations.yml | 2 +- scripts/render_audit.py | 2 +- 24 files changed, 209 insertions(+), 95 deletions(-) rename lectures/{business_cycle_data.csv => gdp_growth_annual.csv} (100%) rename lectures/{business_cycle_data.csv.yml => gdp_growth_annual.csv.yml} (85%) diff --git a/AGENTS.md b/AGENTS.md index a05586a..13483c3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,6 +23,10 @@ Classify as exactly one of: | **constructed** | built from upstream sources by our processing | all of the above **plus the builder script, committed here** | | **dynamic snapshot** | constructed, tracking a moving source (FRED, World Bank) | all of the above **plus a refresh cadence** | +**Naming a published file** (policy settled 2026-09-07, [#113](https://github.com/QuantEcon/data-lectures/issues/113)): the tree is flat, so the filename is the URL and the future `qeld` key. Name the *variable*, add the *frequency* when the same variable exists at another frequency, name a composite for its *topic* rather than its lecture, no source prefix, never a vintage or coverage in a dynamic snapshot's name (a verbatim release keeps upstream's name). Rename freely while `consumers: []`; after that only as a new filename with the old one kept serving. The six rules with examples are in `manifest-schema.yml` beside `filename`. + +**The `schema` block is executable** (decisions of 2026-09-07 on [#120](https://github.com/QuantEcon/data-lectures/issues/120), [#121](https://github.com/QuantEcon/data-lectures/issues/121), [#122](https://github.com/QuantEcon/data-lectures/issues/122); validator in [#119](https://github.com/QuantEcon/data-lectures/issues/119)): column `pattern`s are ordered, contiguous, exhaustive and carry the date in a capture group; `known_nulls` integers are exact and a dynamic snapshot declares null *placement* under `nulls:`; dtypes use the pandas-3 names (`str`, not `string`/`object`) and are compared by family. `manifest-schema.yml` carries the rules. + A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (`sha256` plus the `upstream` verification status, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step. **Verifying `integrity.upstream`, by class** (once here, not repeated per manifest): re-fetch-and-compare for **verbatim**; re-run the builder and compare the overlap window for **constructed / dynamic**; spot-check against the cited publication for author-assembled. When verification is impossible, say so plainly — `status: unverifiable` with a one-line `note` is a known status the catalog can show; silence is not. **Migration safety** (does the file byte-match what the consuming lecture used before a repoint?) is deliberately *not* a manifest field: it is a one-time gate recorded in the repoint PR, and the manifest's `sha256` keeps it reproducible afterwards. @@ -112,7 +116,7 @@ The limits: **50 MiB** warns on push, **100 MiB** (104,857,600 B) is a hard bloc **One builder per published dataset, in `builders/`, named for the dataset it produces:** `builders/.` builds `lectures/.`. The stem is the dataset's, not the lecture's — `builders/japan_earthquakes.py` writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` field predictable and lets CI assert it. -Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — the SCF and Forbes builders each write two. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. A builder's **provenance byproducts** — upstream metadata dumps that are not datasets — go to `provenance/`, never `lectures/`: `business_cycle.py` writes one dataset there and two dumps here. +Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — the SCF and Forbes builders each write two. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. A builder's **provenance byproducts** — upstream metadata dumps that are not datasets — go to `provenance/`, never `lectures/`: `business_cycle.py` writes three datasets there and two dumps here. `scripts/` is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart. diff --git a/CATALOG.md b/CATALOG.md index 7e1653b..f450a4f 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -18,7 +18,6 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)
French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)
[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)
⚠️ BROKEN reader (measured 2026-08-19): fetches this dataset through a stale `base_url` still pointing at lecture-python-intro's deleted `datasets/` copy (french_rev.md:70-75), which serves 404 | | [**bbh_macro_quarterly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_macro_quarterly.csv)
Bhandari-Borovička-Ho replication — quarterly US macro series for the belief-wedge VAR, 1955Q1-2019Q4 | constructed | [Replication package for "Survey data and subjective beliefs in business cycle models" (Bhandari, Borovička and Ho), file `data input/FRED/data_FRED.xlsx`](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 31.5 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) | | [**bbh_michigan_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_michigan_monthly.csv)
Michigan Surveys of Consumers monthly aggregates and the US unemployment rate, 1978-01 to 2020-03 (BBH replication extract) | constructed | [Bhandari, Borovička and Ho replication package (Zenodo), carrying University of Michigan Surveys of Consumers published aggregates and a US Bureau of Labor Statistics series retrieved via FRED](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ⚠️ restricted | ✅ verified | ✅ committed | 11.9 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) | -| [**business_cycle_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/business_cycle_data.csv)
World Bank GDP growth (annual %) — nine economies, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 10.4 KB | — | | [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)
French Revolution — monthly specie value of the assignat, 1791-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)
⚠️ Reads a local `datasets/` copy, not this file
[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)
⚠️ Course fork with a live Pages site; reads its own blob-identical `datasets/` copy (french_rev.md:715-716), not this file, and its `base_url` still points at lecture-python-intro
[python-lecture-sandpit.myst · french_rev.md](https://github.com/QuantEcon/python-lecture-sandpit.myst/blob/main/lectures/french_rev.md)
⚠️ Public sandpit holding `lectures/_static/` copies | | [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)
The Ends of Four Big Inflations — appendix tables, transcribed | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)
[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md)
[lecture-intro.zh-cn · inflation_history.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inflation_history.md)
[test-actions-lecture-intro · inflation_history.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inflation_history.md) | | [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)
World Population Review — Brazilian city populations, 2023 | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | @@ -33,6 +32,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)
Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | [lecture-python.myst · mle.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/mle.md)
[lecture-python.zh-cn · mle.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/mle.md)
[lecture-stats · mle.md](https://github.com/QuantEcon/lecture-stats/blob/main/lectures/mle.md) | | [**fred_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fred_data.csv)
US Treasury yields and NBER recessions — GS1/GS5/GS10, DFII5/DFII10 and USREC, monthly 1953-04 to 2024-12 | constructed | [FRED (Federal Reserve Bank of St. Louis) — series GS1, GS5, GS10, DFII5, DFII10, USREC](https://fred.stlouisfed.org) | | ✅ permitted | ✅ verified | ✅ committed | 27.3 KB | [lecture-python-advanced.myst · risk_aversion_or_mistaken_beliefs.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/risk_aversion_or_mistaken_beliefs.md) | +| [**gdp_growth_annual.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/gdp_growth_annual.csv)
World Bank GDP growth (annual %) — nine economies, 1960 onward | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 10.4 KB | — | | [**hansen_jagannathan_1991_data.json**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_jagannathan_1991_data.json)
Hansen-Jagannathan (1991) replication — US asset returns 1891-1986, with annual consumption (three-table bundle) | constructed | [Robert J. Shiller's public data workbooks (chapt26.xlsx, ie_data.xls) and FRED](http://www.econ.yale.edu/~shiller/data.htm) | | ✅ permitted | ✅ verified | ⚠️ unrecovered | 62.0 KB | [lecture-python-advanced.myst · hansen_jagannathan_1991.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/hansen_jagannathan_1991.md) | | [**hansen_singleton_1982_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1982_data.csv)
Hansen-Singleton (1982) replication — monthly US gross real market return and consumption growth, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 11.4 KB | [lecture-python.myst · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1982.md)
[lecture-python.zh-cn · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1982.md) | | [**hansen_singleton_1983_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1983_data.csv)
Hansen-Singleton (1983) replication — monthly US returns, consumption and inflation, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 25.5 KB | [lecture-python.myst · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1983.md)
[lecture-python.zh-cn · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1983.md) | diff --git a/PLAN.md b/PLAN.md index cbe32d4..f6aa33d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -24,7 +24,7 @@ This repository is being shaped into the **single canonical repository for data ## Where we are -**Update 2026-07-16 — the layout below has been flattened (Phase 2).** The 8 in-use static files and `business_cycle_data.csv` now sit directly in `lectures/`; `scripts/` moved to the root; the 3 no-consumer files were dropped (see Phase 6). The audit that follows is retained as the record of what was migrated and what state each file was in — it is the input to Phases 6 and 7, and the `consumers` column of the manifests still has to be backfilled from it. +**Update 2026-07-16 — the layout below has been flattened (Phase 2).** The 8 in-use static files and `business_cycle_data.csv` (now `gdp_growth_annual.csv`) now sit directly in `lectures/`; `scripts/` moved to the root; the 3 no-consumer files were dropped (see Phase 6). The audit that follows is retained as the record of what was migrated and what state each file was in — it is the input to Phases 6 and 7, and the `consumers` column of the manifests still has to be backfilled from it. ### The audit (2026-07-15) @@ -340,7 +340,7 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde **The flip was the acceptance test, and it measured as one.** Dry-run locally in both directions before pushing: `landed` → exit 1 with 6 warnings; `repointed` → exit 0. So the red window was real, opened when the last consuming PR merged, and closed with the flip. **Do this both-directions dry-run on every future wave** — it converts "same-day, trust me" into a measurement. Five things P3 proved that were not on its test list. A `constructed` dataset's builder must land in the **same** PR as the data, because `check_consumed_files.py` asserts the `builder:` path resolves. `builders/README.md`'s coverage table is a real coverage report and goes stale silently. The plain-git decision costs ~10 MB of packed history for 110 MB of working tree, since CSV compresses 5-22×. The **C0 → C1 → C2 ordering worked and proved less than it looks like** — the sync PR it was designed to defuse (QuantEcon/lecture-intro.zh-cn#293) touched zero data-read lines, zero `# i18n` markers and zero protected localisations, but nothing ever asked the model to rewrite those cells, so the markers remain unexercised, prompt-level protection and **the hand-diff is what protects a localisation**. And the translation sync is **`.md`-only**, so no hand-localised `_static` asset can be created, updated or repaired by it — every `data.ipynb` copy had to be repointed by hand in all four repos, filed upstream as QuantEcon/action-translation#271 -- [ ] **P4 — dynamic snapshot twin**: originally `UNRATE` alone; **reframed 2026-09-01** as the `business_cycle` set, because the lecture that needs a twin is excluded from `lecture-wasm` for want of one and a partial twin buys it nothing. Done so far: `business_cycle_data.csv` manifested and its builder retrofitted ([#109](https://github.com/QuantEcon/data-lectures/pull/109)); the refresh-as-PR and canary workflow ([#110](https://github.com/QuantEcon/data-lectures/pull/110)); the first real refresh ([#112](https://github.com/QuantEcon/data-lectures/pull/112)); the World Bank set extended to three tables and the FRED half landed as one composite monthly file on a shared `builders/_fred.py` library ([#114](https://github.com/QuantEcon/data-lectures/pull/114)). Remaining: the `lecture-wasm` adoption ([QuantEcon/lecture-wasm#70](https://github.com/QuantEcon/lecture-wasm/issues/70) — intro keeps its live calls as the lesson), the flip with `on_refresh: rebuild`, and a canary run catching an induced failure +- [ ] **P4 — dynamic snapshot twin**: originally `UNRATE` alone; **reframed 2026-09-01** as the `business_cycle` set, because the lecture that needs a twin is excluded from `lecture-wasm` for want of one and a partial twin buys it nothing. Done so far: `business_cycle_data.csv` (renamed `gdp_growth_annual.csv` 2026-09-07 under the naming policy, [#113](https://github.com/QuantEcon/data-lectures/issues/113)) manifested and its builder retrofitted ([#109](https://github.com/QuantEcon/data-lectures/pull/109)); the refresh-as-PR and canary workflow ([#110](https://github.com/QuantEcon/data-lectures/pull/110)); the first real refresh ([#112](https://github.com/QuantEcon/data-lectures/pull/112)); the World Bank set extended to three tables and the FRED half landed as one composite monthly file on a shared `builders/_fred.py` library ([#114](https://github.com/QuantEcon/data-lectures/pull/114)). Remaining: the `lecture-wasm` adoption ([QuantEcon/lecture-wasm#70](https://github.com/QuantEcon/lecture-wasm/issues/70) — intro keeps its live calls as the lesson), the flip with `on_refresh: rebuild`, and a canary run catching an induced failure - [ ] Verify each migrated URL with a pyodide/JupyterLite fetch (CORS, meta#143) - [ ] Fold every validated decision into the draft `styleguide/datasets.md` (manual#108) as it is proven diff --git a/builders/README.md b/builders/README.md index 4aa79f8..c291b7f 100644 --- a/builders/README.md +++ b/builders/README.md @@ -64,7 +64,7 @@ re-fetched** — see `AGENTS.md`. | `fred_data.py` | `fred_data.csv` | committed — fetches six FRED series live over a pinned 1953-04..2024-12 window (yields and the recession dummy are stable history, unlike the BBH national-accounts snapshot). Reproduces its output byte for byte (2026-08-18) | | `hansen_singleton_1982_data.py` | `hansen_singleton_1982_data.csv` | committed — fetches FRED and the Ken French factors live. Reproduces its output byte for byte (2026-08-13) | | `hansen_singleton_1983_data.py` | `hansen_singleton_1983_data.csv` | committed — the same construction plus a T-bill leg, so its output is a strict superset of the 1982 file's. Reproduces its output byte for byte (2026-08-13) | -| `business_cycle.py` | `business_cycle_data.csv`, `unemployment_rate_annual.csv`, `private_credit_to_gdp.csv` (plus two dumps to `provenance/`) | committed — the World Bank half of the `business_cycle` lecture's data as three **dynamic snapshots** (`cadence: annual`), one builder writing a set. Fetches live WDI; does NOT reproduce its bytes and is not meant to — the World Bank revises the series (63 of 64 GDP-growth columns moved between the 2025-02 vintage and 2026-09-01). validate() places nulls (before an economy's first observation or in the newest two years, never inside a series), bounds each table's overlap window, and prints the summary — the review surface for a refresh PR. Validates all three before writing any | +| `business_cycle.py` | `gdp_growth_annual.csv`, `unemployment_rate_annual.csv`, `private_credit_to_gdp.csv` (plus two dumps to `provenance/`) | committed — the World Bank half of the `business_cycle` lecture's data as three **dynamic snapshots** (`cadence: annual`), one builder writing a set. Fetches live WDI; does NOT reproduce its bytes and is not meant to — the World Bank revises the series (63 of 64 GDP-growth columns moved between the 2025-02 vintage and 2026-09-01). validate() places nulls (before an economy's first observation or in the newest two years, never inside a series), bounds each table's overlap window, and prints the summary — the review surface for a refresh PR. Validates all three before writing any | | `business_cycle_fred.py` | `us_business_cycle_monthly.csv` | committed — the FRED half as one composite **dynamic snapshot** (`cadence: monthly`): six series on a monthly grid from 1919. Built on the shared `_fred.py` library; declares every structural null exactly (series starts, UMCSENT's sparse pre-1978 years, the 2025-10 shutdown hole) so a new hole fails the refresh | | `webscrape_forbes.ipynb` | `forbes-global2000.csv`, `forbes-billionaires.csv` | **committed-frozen** — an undocumented Forbes API, a spoofed user-agent and hardcoded GDPR consent cookies. Defects recorded in the two manifests rather than fixed | | `generating_mini.md` | `SCF_plus_mini.csv`, `SCF_plus_mini_no_weights.csv` | **committed-frozen** — its `to_csv` calls are commented out upstream and stay that way. As written it still fetches the `high_dim_data` URL; that URL is historical, and the input is now committed at `sources/SCF_plus.dta`. See `sources/README.md` | diff --git a/builders/business_cycle.py b/builders/business_cycle.py index 0607890..be8175b 100644 --- a/builders/business_cycle.py +++ b/builders/business_cycle.py @@ -5,7 +5,7 @@ per economy, ISO3 code as the index, country name as `Country`, one `YR` column per year from 1960): - business_cycle_data.csv NY.GDP.MKTP.KD.ZG real GDP growth, %, for + gdp_growth_annual.csv NY.GDP.MKTP.KD.ZG real GDP growth, %, for the nine economies the lecture plots (the union of its two selections) unemployment_rate_annual.csv SL.UEM.TOTL.NE.ZS unemployment, % of labour @@ -14,9 +14,10 @@ private sector, % of GDP, GBR One builder, three files: the "builder writes a set" precedent -(builders/README.md). The two new filenames are PROVISIONAL pending the naming -policy (QuantEcon/data-lectures#113); they are free to change while no lecture -reads them. +(builders/README.md). Filenames follow the naming policy settled 2026-09-07 +(QuantEcon/data-lectures#113): the variable, then frequency where the same +variable exists at another frequency. gdp_growth_annual.csv was +business_cycle_data.csv until then, renamed while nothing read it. These are DYNAMIC SNAPSHOTS (`cadence: annual`). The World Bank revises this data continuously -- national-accounts rebasing moved GDP growth by up to 1.5 @@ -69,7 +70,7 @@ # window in the series' own units (GDP growth measured at 1.5 pp routine; # the credit ratio is rebased in larger steps). TABLES = [ - {'file': 'business_cycle_data.csv', 'series': 'NY.GDP.MKTP.KD.ZG', + {'file': 'gdp_growth_annual.csv', 'series': 'NY.GDP.MKTP.KD.ZG', 'economies': ['USA', 'ARG', 'GBR', 'GRC', 'JPN', 'CHN', 'DEU', 'BRA', 'MEX'], 'band': (-50, 50), 'min_abs_max': 1.0, 'max_revision': 5.0, 'first_year_null': True}, # growth is undefined in the series' first year diff --git a/lectures/ames_house_prices.csv.yml b/lectures/ames_house_prices.csv.yml index 43d69af..d2a1bea 100644 --- a/lectures/ames_house_prices.csv.yml +++ b/lectures/ames_house_prices.csv.yml @@ -60,7 +60,7 @@ schema: - {name: bedrooms, dtype: int64, description: bedrooms above grade} - {name: living_area_sqft, dtype: int64, description: above-grade living area in square feet} - {name: year_built, dtype: int64, description: year of original construction} - - {name: neighborhood, dtype: object, description: neighborhood within Ames city limits} + - {name: neighborhood, dtype: str, description: neighborhood within Ames city limits} row_count_floor: 2930 date_range: {start: 2006, end: 2010} known_nulls: {} diff --git a/lectures/countries.csv.yml b/lectures/countries.csv.yml index f916dc3..d1e5204 100644 --- a/lectures/countries.csv.yml +++ b/lectures/countries.csv.yml @@ -58,23 +58,23 @@ schema: format: csv delimiter: ";" # read with pd.read_csv(url, sep=';') columns: - - {name: Country (en), dtype: string, description: country name in English (used by the lecture)} - - {name: Country (de), dtype: string, description: country name in German} - - {name: Country (local), dtype: string, description: local-language country name} - - {name: Country code, dtype: string, description: ISO 3166 alpha-2 code} - - {name: Continent, dtype: string, description: continent (used by the lecture)} - - {name: Capital, dtype: string, description: capital city} + - {name: Country (en), dtype: str, description: country name in English (used by the lecture)} + - {name: Country (de), dtype: str, description: country name in German} + - {name: Country (local), dtype: str, description: local-language country name} + - {name: Country code, dtype: str, description: ISO 3166 alpha-2 code} + - {name: Continent, dtype: str, description: continent (used by the lecture)} + - {name: Capital, dtype: str, description: capital city} - {name: Population, dtype: int64, description: population count} - {name: Area, dtype: int64, description: land area (km²)} - {name: Coastline, dtype: int64, description: coastline length (km)} - - {name: Government form, dtype: string, description: form of government} - - {name: Currency, dtype: string, description: currency name} - - {name: Currency code, dtype: string, description: ISO 4217 currency code} - - {name: Dialing prefix, dtype: string, description: international dialing prefix (e.g. "1-684" for territories)} + - {name: Government form, dtype: str, description: form of government} + - {name: Currency, dtype: str, description: currency name} + - {name: Currency code, dtype: str, description: ISO 4217 currency code} + - {name: Dialing prefix, dtype: str, description: international dialing prefix (e.g. "1-684" for territories)} - {name: Birthrate, dtype: float64, description: births per 1000} - {name: Deathrate, dtype: float64, description: deaths per 1000} - {name: Life expectancy, dtype: float64, description: life expectancy (years)} - - {name: Url, dtype: string, description: source page on laenderdaten.info} + - {name: Url, dtype: str, description: source page on laenderdaten.info} row_count_floor: 248 # frozen historical extract -- exact row count # date_range omitted: country-attribute table, no time dimension. known_nulls: diff --git a/lectures/employ.csv.yml b/lectures/employ.csv.yml index bb0d88f..f296838 100644 --- a/lectures/employ.csv.yml +++ b/lectures/employ.csv.yml @@ -49,13 +49,13 @@ schema: format: csv columns: - {name: "", dtype: int64, description: "pandas row index (0-based), not used by the lecture"} - - {name: GEO, dtype: string, description: geography (country or EU aggregate)} - - {name: AGE, dtype: string, description: age band (e.g. "From 15 to 24 years")} - - {name: UNIT, dtype: string, description: unit of measure (e.g. "Thousand persons")} - - {name: SEX, dtype: string, description: Total / Males / Females} - - {name: INDIC_EM, dtype: string, description: employment indicator (e.g. "Active population")} + - {name: GEO, dtype: str, description: geography (country or EU aggregate)} + - {name: AGE, dtype: str, description: age band (e.g. "From 15 to 24 years")} + - {name: UNIT, dtype: str, description: unit of measure (e.g. "Thousand persons")} + - {name: SEX, dtype: str, description: Total / Males / Females} + - {name: INDIC_EM, dtype: str, description: employment indicator (e.g. "Active population")} - {name: Value, dtype: float64, description: measured value in UNIT} - - {name: DATE, dtype: string, description: observation year as ISO date (YYYY-01-01)} + - {name: DATE, dtype: str, description: observation year as ISO date (YYYY-01-01)} row_count_floor: 15120 # frozen historical extract -- exact row count date_range: {start: 2007-01-01, end: 2016-01-01} # 1080 rows carry a row index but are otherwise entirely blank -- spacer rows diff --git a/lectures/epl_match_goals.csv.yml b/lectures/epl_match_goals.csv.yml index 41d983d..b3a71ad 100644 --- a/lectures/epl_match_goals.csv.yml +++ b/lectures/epl_match_goals.csv.yml @@ -49,10 +49,10 @@ integrity: schema: format: csv columns: - - {name: season, dtype: object, description: "season label, e.g. 2023-24"} - - {name: date, dtype: object, description: "match date, ISO 8601"} - - {name: home_team, dtype: object, description: home team name} - - {name: away_team, dtype: object, description: away team name} + - {name: season, dtype: str, description: "season label, e.g. 2023-24"} + - {name: date, dtype: str, description: "match date, ISO 8601"} + - {name: home_team, dtype: str, description: home team name} + - {name: away_team, dtype: str, description: away team name} - {name: home_goals, dtype: int64, description: full-time goals scored by the home team} - {name: away_goals, dtype: int64, description: full-time goals scored by the away team} row_count_floor: 3800 diff --git a/lectures/forbes-billionaires.csv.yml b/lectures/forbes-billionaires.csv.yml index 5eceaa7..44ffa5a 100644 --- a/lectures/forbes-billionaires.csv.yml +++ b/lectures/forbes-billionaires.csv.yml @@ -103,7 +103,7 @@ schema: - {name: rank, dtype: float64, description: annual-list rank} - {name: worth, dtype: float64, description: annual-list net worth (USD millions)} - {name: title, dtype: str, description: job title} - - {name: government, dtype: str, description: "whether the wealth is government-linked, as the string True/False"} + - {name: government, dtype: bool, description: "whether the wealth is government-linked; True/False in the bytes, which pandas reads as bool — a nullable bool column comes back as `object` under a default read and the validator compares by family (#122)"} - {name: pay, dtype: float64, description: reported pay (USD millions)} - {name: managementAssets, dtype: float64, description: assets under management (USD millions)} row_count_floor: 2935 # frozen extract — exact row count by design diff --git a/lectures/fred_data.csv.yml b/lectures/fred_data.csv.yml index 16eeec3..87120df 100644 --- a/lectures/fred_data.csv.yml +++ b/lectures/fred_data.csv.yml @@ -135,7 +135,7 @@ integrity: schema: format: csv columns: - - {name: DATE, dtype: datetime, description: "first-of-month date stamp, 1953-04-01 to 2024-12-01, 861 rows on an unbroken monthly grid. The CSV index column; the lecture reads it with parse_dates and index_col"} + - {name: DATE, dtype: datetime64, description: "first-of-month date stamp, 1953-04-01 to 2024-12-01, 861 rows on an unbroken monthly grid. The CSV index column; the lecture reads it with parse_dates and index_col"} - {name: GS1, dtype: float64, description: "1-year Treasury constant-maturity yield, percent per annum, monthly, NSA. Observed 0.05-16.72"} - {name: GS5, dtype: float64, description: "5-year Treasury constant-maturity yield, percent per annum. Observed 0.27-15.93"} - {name: GS10, dtype: float64, description: "10-year Treasury constant-maturity yield, percent per annum. Observed 0.62-15.32"} diff --git a/lectures/business_cycle_data.csv b/lectures/gdp_growth_annual.csv similarity index 100% rename from lectures/business_cycle_data.csv rename to lectures/gdp_growth_annual.csv diff --git a/lectures/business_cycle_data.csv.yml b/lectures/gdp_growth_annual.csv.yml similarity index 85% rename from lectures/business_cycle_data.csv.yml rename to lectures/gdp_growth_annual.csv.yml index 9772a64..57cf5fa 100644 --- a/lectures/business_cycle_data.csv.yml +++ b/lectures/gdp_growth_annual.csv.yml @@ -1,9 +1,14 @@ -# Manifest for business_cycle_data.csv — the repo's one DYNAMIC SNAPSHOT, and +# Manifest for gdp_growth_annual.csv — the repo's first DYNAMIC SNAPSHOT, and # the prototype for the P4 pilot (PLAN Phase 8: a maintained snapshot twin for # lectures that fetch live API data). The file has sat here since the Feb 2025 # migration (QuantEcon/data b857c5c, then the 2026-07-16 flatten in #10) # without a manifest; this one closes the Phase 6 backfill. # +# Named business_cycle_data.csv until 2026-09-07, when the naming policy +# (QuantEcon/data-lectures#113) settled: a file is named for the variable it +# holds, not the lecture that first read it. Renamed while `consumers` was +# still empty, which is the only time a rename is free (policy rule 6). +# # Read this before refreshing: the World Bank REVISES this series. A re-run of # the builder is not expected to reproduce these bytes and validate() does not # ask it to — see integrity.upstream. A refresh is a deliberate in-place @@ -11,7 +16,7 @@ # summary (PLAN Phase 5, refresh-as-PR), not a correction and not a new # vintage under a new name. -filename: business_cycle_data.csv +filename: gdp_growth_annual.csv title: World Bank GDP growth (annual %) — nine economies, 1960 onward description: > Annual real GDP growth (percent, constant local currency) for the nine @@ -115,7 +120,7 @@ schema: columns: - {name: economy, dtype: str, description: "ISO3 code — wbgapi's index, written as the CSV's first column; the lecture reads it with index_col=0"} - {name: Country, dtype: str, description: "country name, the label column wbgapi adds with labels=True"} - - {pattern: 'YR\d{4}', dtype: float64, description: "annual real GDP growth, percent, one column per year on an unbroken grid from YR1960 to the newest year the source carries (`schema.date_range.end`). The builder's validate() bounds every value at ±50 and requires at least one |value| ≥ 1, so a units switch cannot ship; the observed range is deliberately not written here, since prose in a dynamic snapshot's manifest must not embed facts a refresh can change"} + - {pattern: 'YR(\d{4})', dtype: float64, description: "annual real GDP growth, percent, one column per year on an unbroken grid from YR1960 to the newest year the source carries (`schema.date_range.end`). The builder's validate() bounds every value at ±50 and requires at least one |value| ≥ 1, so a units switch cannot ship; the observed range is deliberately not written here, since prose in a dynamic snapshot's manifest must not embed facts a refresh can change"} row_count_floor: 9 # exact by construction: the builder asks for # nine economies and validate() asserts the # set. Columns, not rows, are what grow. @@ -125,10 +130,15 @@ schema: date_range: {start: 1960, end: 2025} # Growth is undefined in the series' first year, so YR1960 is empty for all - # nine economies. validate() additionally allows a null only before an - # economy's first observation or in the newest two years, never inside a - # series — none of either exists in this vintage. + # nine economies — stable by construction, so declared as an exact count. known_nulls: {YR1960: 9} + # Placement rule (#121, 2026-09-07): a dynamic snapshot declares WHERE nulls + # may sit, not how many. Series run along the YR columns, one per + # economy row; a null is allowed before an economy's first observation or + # in the newest two years (the World Bank publishes year T from mid T+1), + # never inside a series. This is what validate() asserts today + # (MAX_TRAILING_YEARS = 2); the shared validator (#119) reads it from here. + nulls: {along: columns, leading: true, recent: 2, ended: [], inner: {}} # --------------------------------------------------------------------------- # Consumers — how a correction knows what to rebuild diff --git a/lectures/japan_earthquakes.csv.yml b/lectures/japan_earthquakes.csv.yml index 33b73f1..9a3a344 100644 --- a/lectures/japan_earthquakes.csv.yml +++ b/lectures/japan_earthquakes.csv.yml @@ -59,7 +59,7 @@ integrity: schema: format: csv columns: - - {name: time, dtype: object, description: "event origin time, ISO 8601 UTC"} + - {name: time, dtype: str, description: "event origin time, ISO 8601 UTC"} - {name: magnitude, dtype: float64, description: "reported magnitude (mixed scales, as ComCat supplies)"} - {name: latitude, dtype: float64, description: "epicentre latitude, degrees north"} - {name: longitude, dtype: float64, description: "epicentre longitude, degrees east"} diff --git a/lectures/private_credit_to_gdp.csv.yml b/lectures/private_credit_to_gdp.csv.yml index 6125ab5..7fbae8d 100644 --- a/lectures/private_credit_to_gdp.csv.yml +++ b/lectures/private_credit_to_gdp.csv.yml @@ -63,15 +63,17 @@ schema: columns: - {name: economy, dtype: str, description: "ISO3 code — wbgapi's index, written as the CSV's first column; read it with index_col=0"} - {name: Country, dtype: str, description: "country name, the label column wbgapi adds with labels=True"} - - {pattern: 'YR\d{4}', dtype: float64, description: "domestic credit to the private sector, % of GDP, one column per year on an unbroken grid from YR1960 to the newest year the source carries (`schema.date_range.end`). The builder's validate() bounds every value in [0, 400]"} + - {pattern: 'YR(\d{4})', dtype: float64, description: "domestic credit to the private sector, % of GDP, one column per year on an unbroken grid from YR1960 to the newest year the source carries (`schema.date_range.end`). The builder's validate() bounds every value in [0, 400]"} row_count_floor: 1 # exact by construction: one economy # `end` is stamped by scripts/snapshots.py on every refresh. date_range: {start: 1960, end: 2025} # The newest year may be unpublished (a trailing null); nothing else. - # validate() asserts placement, not a count, which is why this is a ceiling. - known_nulls_total: 2 + # Placement rule (#121): validate() asserts where a null may sit, not a + # count — before the first observation or in the newest two years. + known_nulls: {} + nulls: {along: columns, leading: true, recent: 2, ended: [], inner: {}} -# None yet — see business_cycle_data.csv.yml (QuantEcon/lecture-wasm#70). +# None yet — see gdp_growth_annual.csv.yml (QuantEcon/lecture-wasm#70). consumers: [] builder: builders/business_cycle.py diff --git a/lectures/realwage.csv.yml b/lectures/realwage.csv.yml index d692763..f8f6568 100644 --- a/lectures/realwage.csv.yml +++ b/lectures/realwage.csv.yml @@ -52,10 +52,10 @@ schema: format: csv columns: - {name: "", dtype: int64, description: "pandas row index (0-based), not used by the lecture"} - - {name: Time, dtype: string, description: observation year as ISO date (YYYY-01-01)} - - {name: Country, dtype: string, description: OECD country name} - - {name: Series, dtype: string, description: price basis (e.g. "In 2015 constant prices at 2015 USD PPPs")} - - {name: Pay period, dtype: string, description: Annual / Hourly} + - {name: Time, dtype: str, description: observation year as ISO date (YYYY-01-01)} + - {name: Country, dtype: str, description: OECD country name} + - {name: Series, dtype: str, description: price basis (e.g. "In 2015 constant prices at 2015 USD PPPs")} + - {name: Pay period, dtype: str, description: Annual / Hourly} - {name: value, dtype: float64, description: real minimum wage} row_count_floor: 1408 # frozen historical extract -- exact row count date_range: {start: 2006-01-01, end: 2016-01-01} diff --git a/lectures/test_pwt.csv.yml b/lectures/test_pwt.csv.yml index e4b4972..96c55ee 100644 --- a/lectures/test_pwt.csv.yml +++ b/lectures/test_pwt.csv.yml @@ -126,8 +126,8 @@ integrity: schema: format: csv columns: - - {name: country, dtype: object, description: "country name, 8 unique — Argentina, Australia, India, Israel, Malawi, South Africa, United States, Uruguay"} - - {name: country isocode, dtype: object, description: "ISO 3166-1 alpha-3 code. The SPACE in this column name is load-bearing for the lectures, which demonstrate column selection on it"} + - {name: country, dtype: str, description: "country name, 8 unique — Argentina, Australia, India, Israel, Malawi, South Africa, United States, Uruguay"} + - {name: country isocode, dtype: str, description: "ISO 3166-1 alpha-3 code. The SPACE in this column name is load-bearing for the lectures, which demonstrate column selection on it"} - {name: year, dtype: int64, description: "2000 in every row"} - {name: POP, dtype: float64, description: "population, thousands. Observed 3219.793-1006300.297"} - {name: XRAT, dtype: float64, description: "exchange rate, national currency per US dollar. Observed 0.9995-59.543808"} diff --git a/lectures/unemployment_rate_annual.csv.yml b/lectures/unemployment_rate_annual.csv.yml index e31ea97..4012d93 100644 --- a/lectures/unemployment_rate_annual.csv.yml +++ b/lectures/unemployment_rate_annual.csv.yml @@ -1,9 +1,9 @@ # Manifest for unemployment_rate_annual.csv — the second of three World Bank # tables builders/business_cycle.py writes for the intro `business_cycle` -# lecture (with business_cycle_data.csv and private_credit_to_gdp.csv). The -# FILENAME IS PROVISIONAL: it follows the strawman on the naming-policy issue -# (QuantEcon/data-lectures#113) and is free to change while `consumers` is -# empty. Landed 2026-09-01 as part of the P4 dynamic-snapshot pilot. +# lecture (with gdp_growth_annual.csv and private_credit_to_gdp.csv). Named +# under the naming policy settled 2026-09-07 (QuantEcon/data-lectures#113: +# variable, then frequency because a monthly unemployment rate exists in +# us_business_cycle_monthly.csv). Landed 2026-09-01 in the P4 pilot. filename: unemployment_rate_annual.csv title: World Bank unemployment rate (annual, % of labour force) — USA, FRA, GBR, JPN, 1960 onward @@ -46,7 +46,7 @@ license: verified: 2026-09-01 note: > World Bank open data under CC BY-4.0, the same terms as - business_cycle_data.csv (stated in the series metadata the builder dumps + gdp_growth_annual.csv (stated in the series metadata the builder dumps to provenance/ for the GDP series; WDI applies one licence across the database). @@ -70,17 +70,19 @@ schema: columns: - {name: economy, dtype: str, description: "ISO3 code — wbgapi's index, written as the CSV's first column; read it with index_col=0"} - {name: Country, dtype: str, description: "country name, the label column wbgapi adds with labels=True"} - - {pattern: 'YR\d{4}', dtype: float64, description: "unemployment, % of the labour force, one column per year on an unbroken grid from YR1960 to the newest year the source carries (`schema.date_range.end`). The builder's validate() bounds every value in [0, 60]; the observed range is deliberately not written here"} + - {pattern: 'YR(\d{4})', dtype: float64, description: "unemployment, % of the labour force, one column per year on an unbroken grid from YR1960 to the newest year the source carries (`schema.date_range.end`). The builder's validate() bounds every value in [0, 60]; the observed range is deliberately not written here"} row_count_floor: 4 # exact by construction: four economies # `end` is stamped by scripts/snapshots.py on every refresh. date_range: {start: 1960, end: 2025} - # Leading nulls only: GBR is empty 1960–1970 (11 cells) and FRA 1960–1969 - # (10), where the national series had not started. Declared as a total - # because the holes are per economy, not per column; validate() asserts - # their placement (before the first observation), not this count. - known_nulls_total: 21 + # No exact counts: the nulls are per economy, not per column (GBR is empty + # 1960–1970 and FRA 1960–1969, where the national series had not started), + # and a count would move if the source backfilled. Placement rule (#121): + # a null may sit before an economy's first observation or in the newest two + # years, never inside a series — what validate() asserts today. + known_nulls: {} + nulls: {along: columns, leading: true, recent: 2, ended: [], inner: {}} -# None yet — see business_cycle_data.csv.yml: lecture-wasm will read this +# None yet — see gdp_growth_annual.csv.yml: lecture-wasm will read this # file once business_cycle is re-enabled there (QuantEcon/lecture-wasm#70); # lecture-python-intro keeps its live call. consumers: [] diff --git a/lectures/us_adult_heights.csv.yml b/lectures/us_adult_heights.csv.yml index b61f210..ba4c2f0 100644 --- a/lectures/us_adult_heights.csv.yml +++ b/lectures/us_adult_heights.csv.yml @@ -67,7 +67,7 @@ integrity: schema: format: csv columns: - - {name: sex, dtype: object, description: "male or female (NHANES RIAGENDR)"} + - {name: sex, dtype: str, description: "male or female (NHANES RIAGENDR)"} - {name: height_cm, dtype: float64, description: standing height in centimetres (NHANES BMXHT)} row_count_floor: 10478 date_range: {start: 2015, end: 2018} diff --git a/lectures/us_business_cycle_monthly.csv.yml b/lectures/us_business_cycle_monthly.csv.yml index 726d0da..44729cb 100644 --- a/lectures/us_business_cycle_monthly.csv.yml +++ b/lectures/us_business_cycle_monthly.csv.yml @@ -1,11 +1,11 @@ # Manifest for us_business_cycle_monthly.csv — the FRED half of the intro # `business_cycle` lecture's data as one composite monthly file, built by # builders/business_cycle_fred.py on the shared builders/_fred.py library. -# The FILENAME IS PROVISIONAL: it follows the strawman on the naming-policy -# issue (QuantEcon/data-lectures#113 — name a composite for its topic, not -# its lecture) and is free to change while `consumers` is empty. Landed -# 2026-09-01 as part of the P4 dynamic-snapshot pilot; it is the first -# composite dynamic snapshot and the first file with a monthly cadence. +# Named under the naming policy settled 2026-09-07 (QuantEcon/data-lectures#113: +# a composite is named for the topic that unites its members, not for the +# lecture file). Landed 2026-09-01 as part of the P4 dynamic-snapshot pilot; +# it is the first composite dynamic snapshot and the first file with a +# monthly cadence. filename: us_business_cycle_monthly.csv title: US business-cycle indicators, monthly — unemployment, NBER recessions, consumer sentiment, core CPI, industrial production, 1919 onward @@ -97,7 +97,7 @@ integrity: schema: format: csv columns: - - {name: DATE, dtype: datetime, description: "first-of-month date stamp on an unbroken monthly grid from 1919-01-01 to the newest month (`schema.date_range.end`); the CSV index column — read it with parse_dates and index_col"} + - {name: DATE, dtype: datetime64, description: "first-of-month date stamp on an unbroken monthly grid from 1919-01-01 to the newest month (`schema.date_range.end`); the CSV index column — read it with parse_dates and index_col"} - {name: UNRATE, dtype: float64, description: "civilian unemployment rate, %, SA; empty before 1948-01 and for 2025-10; validate() bounds it in [0, 30]"} - {name: USREC, dtype: int64, description: "NBER recession indicator, 1 in recession months and 0 otherwise; complete from 1919-01"} - {name: UMCSENT, dtype: float64, description: "Michigan consumer sentiment index, 1966Q1=100; empty before 1952-11, sparse (quarterly then irregular) until 1978-01, monthly since; validate() bounds it in [20, 150]"} @@ -112,11 +112,27 @@ schema: date_range: {start: 1919-01-01, end: 2026-07-01} # Every null is structural — before a series' first observation, in # UMCSENT's sparse pre-1978 years, after the historical series' last month, - # or the 2025-10 shutdown hole in UNRATE and CPILFESL — and validate() - # asserts their PLACEMENT exactly. The totals below are the first vintage's - # (they grow only for M0892AUSM156SNBR, by one per month) and are recorded - # for the reader, not asserted. - known_nulls: {UNRATE: 349, UMCSENT: 616, CPILFESL: 457, M0892AUSM156SNBR: 1132} + # or the 2025-10 shutdown hole in UNRATE and CPILFESL. Three counts are + # stable (leading nulls plus the one hole) and are declared exact; + # M0892AUSM156SNBR's grows by one per month because the series ended in + # 1942-06, so it is covered by the placement rule alone (#121). + known_nulls: {UNRATE: 349, UMCSENT: 616, CPILFESL: 457} + # Placement rule (#121, 2026-09-07): series run down the DATE index, one per + # column. `leading`: empty before a series' first observation; `ended`: + # empty after its last; `inner`: the accepted holes inside a series, as a + # list of periods or `{sparse_until: P}` for an irregular early history; + # `recent: 0`: the newest month must be populated for every live series + # (the builder requires it). Mirrors FIRST_OBS / LAST_OBS / KNOWN_HOLES / + # MONTHLY_FROM in builders/business_cycle_fred.py. + nulls: + along: rows + leading: true + recent: 0 + ended: [M0892AUSM156SNBR] + inner: + UNRATE: [2025-10-01] + CPILFESL: [2025-10-01] + UMCSENT: {sparse_until: 1978-01-01} # None yet. Decided 2026-09-01: lecture-python-intro keeps its live # pandas_datareader calls (the API is the lesson there); lecture-wasm will diff --git a/manifest-schema.yml b/manifest-schema.yml index fa5ac3e..d6bc527 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -33,11 +33,28 @@ # Identity # --------------------------------------------------------------------------- -filename: business_cycle_data.csv # must match the sidecar's own name +# NAMING (policy settled 2026-09-07, QuantEcon/data-lectures#113). The tree is +# flat, so the filename is the whole identity: the served URL's last segment +# and the future `qeld.url('')` key. Name a file once, by these rules: +# 1. name the VARIABLE(S), not the source and not the lecture +# (gdp_growth_annual, unemployment_rate_annual, private_credit_to_gdp); +# 2. add FREQUENCY when the same variable plausibly exists at another +# frequency (unemployment_rate_annual beside a monthly one); +# 3. a COMPOSITE is named for what unites its members — if that is only one +# lecture, name the topic, not the lecture file (us_business_cycle_monthly, +# not business_cycle_data); +# 4. NO SOURCE PREFIX (no wdi_/fred_) — provenance is `source:` and CATALOG.md; +# 5. NEVER coverage or vintage in a dynamic snapshot's name (it changes on +# refresh); a verbatim release keeps the upstream's own name (mpd2020.xlsx); +# 6. rename FREELY while `consumers: []`; once adopted, only as a NEW filename +# with the old one kept serving — a rename breaks every consumer and every +# downloaded notebook. Selection ("these nine economies") lives in the +# manifest, not the name. +filename: gdp_growth_annual.csv # must match the sidecar's own name title: World Bank GDP growth (annual %) — selected economies description: > - Annual real GDP growth for USA, ARG, GBR, GRC and JPN, extracted from World - Bank series NY.GDP.MKTP.KD.ZG. + Annual real GDP growth for nine economies, extracted from World Bank series + NY.GDP.MKTP.KD.ZG. # Exactly one of: verbatim | constructed | dynamic-snapshot # verbatim third-party file republished as distributed @@ -163,29 +180,89 @@ integrity: # Shape — what a consumer can rely on; also the basis for PR validation tests # --------------------------------------------------------------------------- +# The schema block is an EXECUTABLE CONTRACT: the shared manifest-driven +# validate() (QuantEcon/data-lectures#119) reads it as its spec, both in every +# dynamic builder's validate stage and in the PR-validation workflow. The three +# decisions that make it executable were settled 2026-09-07 — #120 (`pattern`), +# #121 (nulls), #122 (dtypes) — and are recorded beside the fields below. +# +# What the validator reads: the RAW file as written, with pandas defaults — +# honouring `delimiter` (countries.csv is semicolon-separated) and `read_as` +# (positional workbook reads), with no `index_col`, and with `parse_dates` on +# every column declared `datetime64`. The manifest describes the bytes on +# disk, not the frame a particular lecture builds from them. + schema: format: csv columns: - - {name: economy, dtype: string, description: ISO3 country code} - - {name: Country, dtype: string, description: Country name} - # A column entry carries EITHER `name` (one exact column) OR `pattern` (a - # regex matching a run of columns), never both. `pattern` is for files - # whose column set grows — one column per year here — where enumerating - # them would hard-code an end year into a file whose purpose is to move. - # First used by business_cycle_data.csv.yml (2026-09-01); proposed in - # QuantEcon/data-lectures#14. - - {pattern: 'YR\d{4}', dtype: float64, description: Annual GDP growth (%), one column per year from YR1960} + # dtype (#122): the canonical vocabulary is what pandas 3 reports for a + # default read — `str`, `float64`, `int64`, `bool`, `datetime64` — plus the + # storage types a non-CSV format genuinely carries (`float32`, `int32`, + # `int8` from Stata). `string`, `object` and `datetime` are retired (swept + # 2026-09-07). The validator compares by FAMILY, not exact string: str / + # string / object are one text family, datetime64 matches any unit, and a + # nullable bool read back as object matches `bool` — so the check stays + # green on pandas-2 readers (lecture-datascience, Pyodide). A CSV carries + # no dtypes, so a declaration is a statement about a default read; a + # `datetime64` declaration is also the instruction to parse that column. + - {name: economy, dtype: str, description: ISO3 country code} + - {name: Country, dtype: str, description: Country name} + # pattern (#120): a column entry carries EITHER `name` (one exact column) + # OR `pattern` (a regex), never both. `pattern` is for files whose column + # set grows — one column per year here — where enumerating them would + # hard-code an end year into a file whose purpose is to move. Semantics: + # - Python `re`, full-match (`YR\d{4}` cannot match `YR19601`); + # - the validator walks `columns` IN ORDER and a pattern consumes the + # maximal run of CONSECUTIVE matching columns in that position — + # column order is part of the contract (positional reads exist); + # - at least ONE column must match, else "wide part missing" fails; + # - EXHAUSTIVE: every file column is claimed by exactly one entry, so an + # unexpected column fails; + # - exactly one CAPTURE GROUP marks the date token, from which the + # validator derives `date_range` for a wide file. A stricter grid + # check (no gaps) is the builder's, layered on top. + # First used by business_cycle_data.csv.yml (now gdp_growth_annual.csv.yml, + # 2026-09-01); proposed in QuantEcon/data-lectures#14. + - {pattern: 'YR(\d{4})', dtype: float64, description: Annual GDP growth (%), one column per year from YR1960} row_count_floor: 5 # PR validation fails below this. For a # frozen historical extract set it to the # exact row count by design (P1 lingcod = # 82); a loose floor is for files that grow. date_range: {start: 1960, end: null} # null end = tracks the moving source - # Nulls that are correct and must not fail validation. P1 found a real one: - # lingcod's F/Fmsy is null in its terminal assessment year, which a blanket - # no-nulls rule would reject. Declaring them makes the invariant tests - # (Phase 5) enforceable without hand-waving. + # Nulls (#121). Two forms, defaulted by `class`: + # + # `known_nulls: {column: N}` — an EXACT count; `n != N` fails. Required for + # every nulled column of a `verbatim` or `constructed` file (P1 found a real + # one: lingcod's F/Fmsy is null in its terminal assessment year, which a + # blanket no-nulls rule would reject). A dynamic snapshot may also declare + # an exact count for a column whose nulls are stable by construction + # (gdp_growth_annual's YR1960: growth is undefined in the first year). + # There is deliberately NO ceiling form ({max: N}): a ceiling silently + # accepts a column emptying out below the bound, which is the failure the + # check exists to catch. known_nulls: {} # e.g. {F_over_Fmsy: 1} + # `nulls:` — a PLACEMENT rule, required for `dynamic-snapshot` because its + # counts move on every refresh while WHERE a null may sit does not. It + # mirrors what both dynamic builders already assert by hand: + # along: rows | columns which way a series runs — down a time index + # (one series per column) or along a run of + # pattern columns (one series per row) + # leading: true | false empty before a series' first observation + # recent: N the newest N periods may be unpublished + # ended: [series, ...] series that stopped; empty after their last + # observation (a historical FRED series) + # inner: {series: [P, ...] | {sparse_until: P}} + # accepted holes INSIDE a series: listed + # periods, or an irregular early history + # A null anywhere else fails. See us_business_cycle_monthly.csv.yml for the + # full form and gdp_growth_annual.csv.yml for the wide-file form. + # nulls: {along: columns, leading: true, recent: 2, ended: [], inner: {}} + # + # `known_nulls_total: N` is legal ONLY inside a `sheets:` entry read with + # `header: null` (assignat.xlsx, dette.xlsx): a positional read has no + # column names to count against, so an exact total is the only form. It is + # not a ceiling and not for named columns. # --------------------------------------------------------------------------- # Consumers — machine-readable, so a correction knows what to rebuild diff --git a/migration.yml b/migration.yml index ffd0aba..f14f35a 100644 --- a/migration.yml +++ b/migration.yml @@ -836,7 +836,9 @@ datasets: # and the manifest arrived with #109 — so that is the landing PR. No # consumer: intro's business_cycle still makes the live wbgapi call the # builder reproduces; whether it adopts the snapshot is the P4 decision. - business_cycle_data.csv: + # Renamed from business_cycle_data.csv on 2026-09-07 under the naming policy + # (#113), while `consumers` was still empty; no URL was ever read. + gdp_growth_annual.csv: pilot: P4 status: landed prior_pattern: null # never read from anywhere — the lecture @@ -850,7 +852,7 @@ datasets: # The rest of the business_cycle set (#114): two more World Bank tables from # the same builder, and the FRED half as one composite monthly file. Names - # provisional pending #113. Consumer-to-be: lecture-wasm (lecture-wasm#70). + # settled 2026-09-07 (#113). Consumer-to-be: lecture-wasm (lecture-wasm#70). unemployment_rate_annual.csv: pilot: P4 status: landed diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index ad47c3d..f3bc2b7 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -45,7 +45,7 @@ api: lecture-python-intro:business_cycle:pandas_datareader: series: UNRATE, USREC, M0892AUSM156SNBR, UMCSENT, CPILFESL, INDPRO pedagogy: mixed - note: snapshot pipeline already exists (data-lectures business_cycle_data.csv) but is unadopted + note: snapshot pipeline already exists (data-lectures gdp_growth_annual.csv, business_cycle_data.csv until 2026-09-07) but is unadopted lecture-python-intro:commod_price:yfinance: series: CT=F pedagogy: incidental diff --git a/scripts/render_audit.py b/scripts/render_audit.py index 62f5541..b0e516f 100644 --- a/scripts/render_audit.py +++ b/scripts/render_audit.py @@ -1098,7 +1098,7 @@ def audit_rules() -> str: "lecture's wbgapi/yfinance sections) or when currency is the point. “The lecture " "needs series X” is not a reason — that's what snapshots are for."), ("Every live-API lecture gets a snapshot twin.", - "A refresh builder in data-lectures producing the snapshot (the business_cycle_data.csv " + "A refresh builder in data-lectures producing the snapshot (the gdp_growth_annual.csv " "pattern, already prototyped). Breakage becomes a one-line URL switch, and the WASM " "build always uses the twin — pyodide cannot reach the live APIs at all."), ("Prefer direct CSV endpoints over wrapper libraries.", From a2feea829c0a9ad523362e5bb46502900610d557 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 7 Sep 2026 09:34:29 +1000 Subject: [PATCH 2/2] Copilot review: clarify the forbes government dtype note; fix the stale sidecar path in provenance/README.md Co-Authored-By: Claude Fable 5.1 --- lectures/forbes-billionaires.csv.yml | 2 +- provenance/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lectures/forbes-billionaires.csv.yml b/lectures/forbes-billionaires.csv.yml index 44ffa5a..b28c31e 100644 --- a/lectures/forbes-billionaires.csv.yml +++ b/lectures/forbes-billionaires.csv.yml @@ -103,7 +103,7 @@ schema: - {name: rank, dtype: float64, description: annual-list rank} - {name: worth, dtype: float64, description: annual-list net worth (USD millions)} - {name: title, dtype: str, description: job title} - - {name: government, dtype: bool, description: "whether the wealth is government-linked; True/False in the bytes, which pandas reads as bool — a nullable bool column comes back as `object` under a default read and the validator compares by family (#122)"} + - {name: government, dtype: bool, description: "whether the wealth is government-linked. True/False in the bytes; because the column has nulls (2606 of 2935), a default pd.read_csv yields `object` holding Python bools and NaN rather than a bool dtype, so the declared family is bool and the validator compares by family (#122)"} - {name: pay, dtype: float64, description: reported pay (USD millions)} - {name: managementAssets, dtype: float64, description: assets under management (USD millions)} row_count_floor: 2935 # frozen extract — exact row count by design diff --git a/provenance/README.md b/provenance/README.md index b5a0dd3..5c9c6ae 100644 --- a/provenance/README.md +++ b/provenance/README.md @@ -16,7 +16,7 @@ failing every rule a dataset must meet. | File | Written by | What it is | | --- | --- | --- | -| `business_cycle_metadata.md` | `builders/business_cycle.py` | `wb.series.metadata.get('NY.GDP.MKTP.KD.ZG')` — the World Bank's own record for the series: definition, source, periodicity and the `License_Type: CC BY-4.0` / `License_URL` fields that `lectures/business_cycle_data.csv.yml` cites | +| `business_cycle_metadata.md` | `builders/business_cycle.py` | `wb.series.metadata.get('NY.GDP.MKTP.KD.ZG')` — the World Bank's own record for the series: definition, source, periodicity and the `License_Type: CC BY-4.0` / `License_URL` fields that `lectures/gdp_growth_annual.csv.yml` cites | | `business_cycle_info.md` | `builders/business_cycle.py` | `wb.series.info(q='GDP growth')` — the fuzzy-search listing the consuming lecture teaches; kept because the builder reproduces the lecture's own query, not because anything reads it | Files here carry no manifest and no hash gate. Runs of blank lines in the