Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ Classify as exactly one of:
| **constructed** | built from upstream sources by our processing | all of the above **plus the builder script, committed here** |
| **dynamic snapshot** | constructed, tracking a moving source (FRED, World Bank) | all of the above **plus a refresh cadence** |

**Naming a published file** (policy settled 2026-09-07, [#113](https://github.com/QuantEcon/data-lectures/issues/113)): the tree is flat, so the filename is the URL and the future `qeld` key. Name the *variable*, add the *frequency* when the same variable exists at another frequency, name a composite for its *topic* rather than its lecture, no source prefix, never a vintage or coverage in a dynamic snapshot's name (a verbatim release keeps upstream's name). Rename freely while `consumers: []`; after that only as a new filename with the old one kept serving. The six rules with examples are in `manifest-schema.yml` beside `filename`.

**The `schema` block is executable** (decisions of 2026-09-07 on [#120](https://github.com/QuantEcon/data-lectures/issues/120), [#121](https://github.com/QuantEcon/data-lectures/issues/121), [#122](https://github.com/QuantEcon/data-lectures/issues/122); validator in [#119](https://github.com/QuantEcon/data-lectures/issues/119)): column `pattern`s are ordered, contiguous, exhaustive and carry the date in a capture group; `known_nulls` integers are exact and a dynamic snapshot declares null *placement* under `nulls:`; dtypes use the pandas-3 names (`str`, not `string`/`object`) and are compared by family. `manifest-schema.yml` carries the rules.

A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (`sha256` plus the `upstream` verification status, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step.

**Verifying `integrity.upstream`, by class** (once here, not repeated per manifest): re-fetch-and-compare for **verbatim**; re-run the builder and compare the overlap window for **constructed / dynamic**; spot-check against the cited publication for author-assembled. When verification is impossible, say so plainly — `status: unverifiable` with a one-line `note` is a known status the catalog can show; silence is not. **Migration safety** (does the file byte-match what the consuming lecture used before a repoint?) is deliberately *not* a manifest field: it is a one-time gate recorded in the repoint PR, and the manifest's `sha256` keeps it reproducible afterwards.
Expand Down Expand Up @@ -112,7 +116,7 @@ The limits: **50 MiB** warns on push, **100 MiB** (104,857,600 B) is a hard bloc

**One builder per published dataset, in `builders/`, named for the dataset it produces:** `builders/<stem>.<ext>` builds `lectures/<stem>.<ext2>`. The stem is the dataset's, not the lecture's — `builders/japan_earthquakes.py` writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` field predictable and lets CI assert it.

Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — the SCF and Forbes builders each write two. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. A builder's **provenance byproducts** — upstream metadata dumps that are not datasets — go to `provenance/`, never `lectures/`: `business_cycle.py` writes one dataset there and two dumps here.
Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — the SCF and Forbes builders each write two. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. A builder's **provenance byproducts** — upstream metadata dumps that are not datasets — go to `provenance/`, never `lectures/`: `business_cycle.py` writes three datasets there and two dumps here.

`scripts/` is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart.

Expand Down
2 changes: 1 addition & 1 deletion CATALOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*
| [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)<br><sub>French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)<br>[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)<br>[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)<br>[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)<br><sub>⚠️ BROKEN reader (measured 2026-08-19): fetches this dataset through a stale `base_url` still pointing at lecture-python-intro's deleted `datasets/` copy (french_rev.md:70-75), which serves 404</sub> |
| [**bbh_macro_quarterly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_macro_quarterly.csv)<br><sub>Bhandari-Borovička-Ho replication — quarterly US macro series for the belief-wedge VAR, 1955Q1-2019Q4</sub> | constructed | [Replication package for "Survey data and subjective beliefs in business cycle models" (Bhandari, Borovička and Ho), file `data input/FRED/data_FRED.xlsx`](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 31.5 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) |
| [**bbh_michigan_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_michigan_monthly.csv)<br><sub>Michigan Surveys of Consumers monthly aggregates and the US unemployment rate, 1978-01 to 2020-03 (BBH replication extract)</sub> | constructed | [Bhandari, Borovička and Ho replication package (Zenodo), carrying University of Michigan Surveys of Consumers published aggregates and a US Bureau of Labor Statistics series retrieved via FRED](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ⚠️ restricted | ✅ verified | ✅ committed | 11.9 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) |
| [**business_cycle_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/business_cycle_data.csv)<br><sub>World Bank GDP growth (annual %) — nine economies, 1960 onward</sub> | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 10.4 KB | — |
| [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)<br><sub>French Revolution — monthly specie value of the assignat, 1791-1796</sub> | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)<br>[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)<br>[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)<br><sub>⚠️ Reads a local `datasets/` copy, not this file</sub><br>[tom-econ370-2025 · french_rev.md](https://github.com/QuantEcon/tom-econ370-2025/blob/main/lectures/french_rev.md)<br><sub>⚠️ Course fork with a live Pages site; reads its own blob-identical `datasets/` copy (french_rev.md:715-716), not this file, and its `base_url` still points at lecture-python-intro</sub><br>[python-lecture-sandpit.myst · french_rev.md](https://github.com/QuantEcon/python-lecture-sandpit.myst/blob/main/lectures/french_rev.md)<br><sub>⚠️ Public sandpit holding `lectures/_static/` copies</sub> |
| [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)<br><sub>The Ends of Four Big Inflations — appendix tables, transcribed</sub> | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)<br>[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md)<br>[lecture-intro.zh-cn · inflation_history.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inflation_history.md)<br>[test-actions-lecture-intro · inflation_history.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inflation_history.md) |
| [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)<br><sub>World Population Review — Brazilian city populations, 2023</sub> | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
Expand All @@ -33,6 +32,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*
| [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)<br><sub>Forbes Global 2000 — firm size measures</sub> | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)<br><sub>Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates</sub> | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | [lecture-python.myst · mle.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/mle.md)<br>[lecture-python.zh-cn · mle.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/mle.md)<br>[lecture-stats · mle.md](https://github.com/QuantEcon/lecture-stats/blob/main/lectures/mle.md) |
| [**fred_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fred_data.csv)<br><sub>US Treasury yields and NBER recessions — GS1/GS5/GS10, DFII5/DFII10 and USREC, monthly 1953-04 to 2024-12</sub> | constructed | [FRED (Federal Reserve Bank of St. Louis) — series GS1, GS5, GS10, DFII5, DFII10, USREC](https://fred.stlouisfed.org) | | ✅ permitted | ✅ verified | ✅ committed | 27.3 KB | [lecture-python-advanced.myst · risk_aversion_or_mistaken_beliefs.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/risk_aversion_or_mistaken_beliefs.md) |
| [**gdp_growth_annual.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/gdp_growth_annual.csv)<br><sub>World Bank GDP growth (annual %) — nine economies, 1960 onward</sub> | dynamic-snapshot | [World Bank, World Development Indicators (national accounts data, and OECD National Accounts data files)](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) | CC BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 10.4 KB | — |
| [**hansen_jagannathan_1991_data.json**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_jagannathan_1991_data.json)<br><sub>Hansen-Jagannathan (1991) replication — US asset returns 1891-1986, with annual consumption (three-table bundle)</sub> | constructed | [Robert J. Shiller's public data workbooks (chapt26.xlsx, ie_data.xls) and FRED](http://www.econ.yale.edu/~shiller/data.htm) | | ✅ permitted | ✅ verified | ⚠️ unrecovered | 62.0 KB | [lecture-python-advanced.myst · hansen_jagannathan_1991.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/hansen_jagannathan_1991.md) |
| [**hansen_singleton_1982_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1982_data.csv)<br><sub>Hansen-Singleton (1982) replication — monthly US gross real market return and consumption growth, 1959-1978</sub> | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 11.4 KB | [lecture-python.myst · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1982.md)<br>[lecture-python.zh-cn · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1982.md) |
| [**hansen_singleton_1983_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1983_data.csv)<br><sub>Hansen-Singleton (1983) replication — monthly US returns, consumption and inflation, 1959-1978</sub> | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 25.5 KB | [lecture-python.myst · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1983.md)<br>[lecture-python.zh-cn · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1983.md) |
Expand Down
Loading
Loading