Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ A constructed dataset without its committed builder is a bug. Manifest fields: `

The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108).

- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`).
- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that **accounts for the gap** — never left bare. Any resolved status does that: `unverifiable` says the vintage cannot be established at all, and `diverged` or `verified` say something stronger, because a re-fetch that hashes identically pins the vintage by content rather than by date, which is what a `retrieved` date was a proxy for. What is not acceptable is `null` beside `unverified` — that is two unanswered questions, not one answered a different way. (Amended 2026-08-13, wave B1': the rule previously named `unverifiable` alone, which was the only case that had arisen. `life-expectancy-vs-gdp-per-capita.csv` and `mpd2020.xlsx` already paired `null` with `diverged`; B1' added five files where an upstream re-fetch succeeded byte-identically.)
- **`builder_status: committed-frozen` — the builder is here, and deliberately will not run.** For a dataset built from a source that must not be refreshed: a frozen vintage, or a scraper we will not re-run. The artifact is kept as the record of what produced these bytes, so it is committed verbatim and not edited — editing it is what would destroy its value as provenance. Distinct from `committed`, which asserts a runnable four-stage builder, and from `unrecovered`, which says the builder is absent.
- **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder.

Expand Down
7 changes: 6 additions & 1 deletion CATALOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@

The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md).

**26 datasets** · 26 read by lectures today · 111.9 MB total · 21 permitted / 5 restricted redistribution
**31 datasets** · 26 read by lectures today, 5 awaiting repoint · 113.0 MB total · 26 permitted / 5 restricted redistribution

| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [**NEWQDATA.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/NEWQDATA.csv)<br><sub>Cogley-Sargent (2005) "Drifts and Volatilities" — quarterly US inflation, unemployment and T-bill rate</sub> | constructed | [Cogley and Sargent (2005), "Drifts and Volatilities" — the authors' NEWQDATA.MAT](https://doi.org/10.1016/j.red.2004.10.009) | | ✅ permitted | ✅ verified | ✅ committed | 13.4 KB | — |
| [**SCF_plus_mini.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini.csv)<br><sub>SCF+ mini — net wealth, income and survey weights, 1950-2016</sub> | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 31.3 MB | [lecture-python-intro · inequality.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inequality.md)<br>[lecture-wasm · inequality.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inequality.md)<br>[lecture-intro.zh-cn · inequality.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inequality.md)<br>[test-actions-lecture-intro · inequality.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inequality.md) |
| [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)<br><sub>SCF+ mini, weight-expanded — net wealth and income, 1950-2016</sub> | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | [lecture-python-intro · mle.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/mle.md)<br>[lecture-wasm · mle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/mle.md)<br>[lecture-intro.zh-cn · mle.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/mle.md)<br>[test-actions-lecture-intro · mle.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/mle.md) |
| [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)<br><sub>Ames, Iowa — residential house sales, 2006-2010</sub> | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
Expand All @@ -25,12 +26,16 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*
| [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)<br><sub>French Revolution — figure 3 series (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) |
| [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)<br><sub>Forbes Billionaires — individual net worth</sub> | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)<br><sub>Forbes Global 2000 — firm size measures</sub> | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)<br><sub>Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates</sub> | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | — |
| [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)<br><sub>Japan — deaths by single year of age, 2023</sub> | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)<br><sub>Japan region — earthquakes of magnitude 5 and above, 2000-2024</sub> | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)<br><sub>Japan — population by single year of age, 2024</sub> | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) |
| [**life-expectancy-vs-gdp-per-capita.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/life-expectancy-vs-gdp-per-capita.csv)<br><sub>Life expectancy vs GDP per capita — Our World in Data grapher export</sub> | verbatim | [Our World in Data — "Life expectancy vs. GDP per capita" grapher](https://ourworldindata.org/grapher/life-expectancy-vs-gdp-per-capita) | | ✅ permitted | ⇄ diverged | n/a (verbatim) | 2.0 MB | [lecture-python-intro · simple_linear_regression.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/simple_linear_regression.md)<br>[lecture-wasm · simple_linear_regression.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/simple_linear_regression.md)<br>[lecture-intro.zh-cn · simple_linear_regression.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/simple_linear_regression.md)<br>[test-actions-lecture-intro · simple_linear_regression.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/simple_linear_regression.md) |
| [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)<br><sub>Pacific Coast lingcod — biomass and fishing pressure relative to MSY</sub> | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) |
| [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)<br><sub>Price levels in four hard-currency countries, 1600-2000</sub> | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)<br>[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md) |
| [**maketable1.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable1.dta)<br><sub>Acemoglu-Johnson-Robinson (2001) colonial origins — table 1 replication data</sub> | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 1 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 14.1 KB | — |
| [**maketable2.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable2.dta)<br><sub>Acemoglu-Johnson-Robinson (2001) colonial origins — table 2 replication data</sub> | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 2 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 10.9 KB | — |
| [**maketable4.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable4.dta)<br><sub>Acemoglu-Johnson-Robinson (2001) colonial origins — table 4 replication data</sub> | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 4 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 11.7 KB | — |
| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)<br><sub>Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018</sub> | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | [lecture-python-intro · long_run_growth.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/long_run_growth.md)<br>[lecture-wasm · long_run_growth.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/long_run_growth.md) |
| [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)<br><sub>French Revolution — monthly nominal assignat balances, 1789-1796</sub> | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) |
| [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)<br><sub>OECD real minimum wages — 32 countries, 2006–2016</sub> | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
Expand Down
Loading
Loading