diff --git a/AGENTS.md b/AGENTS.md index 8e03ebb..15f353d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,7 +33,7 @@ A constructed dataset without its committed builder is a bug. Manifest fields: ` The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108). -- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). +- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that **accounts for the gap** — never left bare. Any resolved status does that: `unverifiable` says the vintage cannot be established at all, and `diverged` or `verified` say something stronger, because a re-fetch that hashes identically pins the vintage by content rather than by date, which is what a `retrieved` date was a proxy for. What is not acceptable is `null` beside `unverified` — that is two unanswered questions, not one answered a different way. (Amended 2026-08-13, wave B1': the rule previously named `unverifiable` alone, which was the only case that had arisen. `life-expectancy-vs-gdp-per-capita.csv` and `mpd2020.xlsx` already paired `null` with `diverged`; B1' added five files where an upstream re-fetch succeeded byte-identically.) - **`builder_status: committed-frozen` — the builder is here, and deliberately will not run.** For a dataset built from a source that must not be refreshed: a frozen vintage, or a scraper we will not re-run. The artifact is kept as the record of what produced these bytes, so it is committed verbatim and not edited — editing it is what would destroy its value as provenance. Distinct from `committed`, which asserts a runnable four-stage builder, and from `unrecovered`, which says the builder is absent. - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. diff --git a/CATALOG.md b/CATALOG.md index a39368b..8cf96e0 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -6,10 +6,11 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**26 datasets** · 26 read by lectures today · 111.9 MB total · 21 permitted / 5 restricted redistribution +**31 datasets** · 26 read by lectures today, 5 awaiting repoint · 113.0 MB total · 26 permitted / 5 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| [**NEWQDATA.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/NEWQDATA.csv)
Cogley-Sargent (2005) "Drifts and Volatilities" — quarterly US inflation, unemployment and T-bill rate | constructed | [Cogley and Sargent (2005), "Drifts and Volatilities" — the authors' NEWQDATA.MAT](https://doi.org/10.1016/j.red.2004.10.009) | | ✅ permitted | ✅ verified | ✅ committed | 13.4 KB | — | | [**SCF_plus_mini.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini.csv)
SCF+ mini — net wealth, income and survey weights, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 31.3 MB | [lecture-python-intro · inequality.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inequality.md)
[lecture-wasm · inequality.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inequality.md)
[lecture-intro.zh-cn · inequality.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inequality.md)
[test-actions-lecture-intro · inequality.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inequality.md) | | [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)
SCF+ mini, weight-expanded — net wealth and income, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | [lecture-python-intro · mle.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/mle.md)
[lecture-wasm · mle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/mle.md)
[lecture-intro.zh-cn · mle.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/mle.md)
[test-actions-lecture-intro · mle.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/mle.md) | | [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)
Ames, Iowa — residential house sales, 2006-2010 | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | @@ -25,12 +26,16 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)
French Revolution — figure 3 series (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) | | [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)
Forbes Billionaires — individual net worth | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | +| [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)
Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | — | | [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)
Japan — deaths by single year of age, 2023 | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)
Japan region — earthquakes of magnitude 5 and above, 2000-2024 | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | | [**life-expectancy-vs-gdp-per-capita.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/life-expectancy-vs-gdp-per-capita.csv)
Life expectancy vs GDP per capita — Our World in Data grapher export | verbatim | [Our World in Data — "Life expectancy vs. GDP per capita" grapher](https://ourworldindata.org/grapher/life-expectancy-vs-gdp-per-capita) | | ✅ permitted | ⇄ diverged | n/a (verbatim) | 2.0 MB | [lecture-python-intro · simple_linear_regression.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/simple_linear_regression.md)
[lecture-wasm · simple_linear_regression.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/simple_linear_regression.md)
[lecture-intro.zh-cn · simple_linear_regression.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/simple_linear_regression.md)
[test-actions-lecture-intro · simple_linear_regression.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/simple_linear_regression.md) | | [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | | [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)
Price levels in four hard-currency countries, 1600-2000 | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)
[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md) | +| [**maketable1.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable1.dta)
Acemoglu-Johnson-Robinson (2001) colonial origins — table 1 replication data | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 1 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 14.1 KB | — | +| [**maketable2.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable2.dta)
Acemoglu-Johnson-Robinson (2001) colonial origins — table 2 replication data | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 2 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 10.9 KB | — | +| [**maketable4.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/maketable4.dta)
Acemoglu-Johnson-Robinson (2001) colonial origins — table 4 replication data | verbatim | [Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 4 (program and data files)](https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive) | | ✅ permitted | ✅ verified | n/a (verbatim) | 11.7 KB | — | | [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | [lecture-python-intro · long_run_growth.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/long_run_growth.md)
[lecture-wasm · long_run_growth.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/long_run_growth.md) | | [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)
French Revolution — monthly nominal assignat balances, 1789-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) | | [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | diff --git a/builders/NEWQDATA.py b/builders/NEWQDATA.py new file mode 100644 index 0000000..cc9d585 --- /dev/null +++ b/builders/NEWQDATA.py @@ -0,0 +1,89 @@ +""" +Builder for lectures/NEWQDATA.csv. + +Converts Cogley and Sargent's MATLAB data file for "Drifts and Volatilities: +Monetary Policies and Outcomes in the Post WWII U.S." into the CSV the +`phillips_drifts_volatilities` lecture reads. Four quarterly series over +1948Q2-2000Q4: the quarter index, the three-month Treasury bill rate, +the civilian unemployment rate and CPI inflation. + +This is a value-preserving container conversion and nothing else -- .MAT to +.csv, no filtering, no rescaling, no reordering. That is why the published CSV +can be reproduced byte for byte from the committed input. + +READS ITS INPUT FROM sources/, WHICH IS THE EXCEPTION, NOT THE RULE. +AGENTS.md permits it only when the input cannot be re-fetched, and this one +cannot: the authors' MATLAB directory is not published by either author, by the +Review of Economic Dynamics, or by RePEc, and no Wayback capture of it exists. +Searched to exhaustion 2026-08-13 -- see sources/README.md. The only copy +located anywhere was a third-party GitHub mirror, and a builder that depends on +a stranger's repository is exactly the fragility sources/ exists to remove. + +Stages: fetch -> pre-process -> validate -> write. +""" + +import os + +import pandas as pd +from scipy.io import loadmat + +CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__)) +REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR) +PUBLISHED_DIR = os.path.join(REPO_ROOT, 'lectures') +SOURCES_DIR = os.path.join(REPO_ROOT, 'sources') + +SOURCE_FILE = 'NEWQDATA.MAT' + +# The .MAT holds four bare arrays under these names, in this order. The output +# column order is the input order -- see validate(). +COLUMNS = ['date', 'y3', 'ur', 'dp'] + +# 1948Q2 to 2000Q4 inclusive, quarterly, no gaps. +N_QUARTERS = 211 +FIRST_QUARTER = 1948.25 +LAST_QUARTER = 2000.75 + +OUT_FILE = 'NEWQDATA.csv' + + +def fetch(): + return loadmat(os.path.join(SOURCES_DIR, SOURCE_FILE)) + + +def pre_process(raw): + # Each array is (211, 1); ravel to 1-D so the frame is 211 rows, not 211 + # columns of one element. + return pd.DataFrame({name: raw[name].ravel() for name in COLUMNS}) + + +def validate(df): + """Refuse to write anything that is not the shape we expect.""" + assert list(df.columns) == COLUMNS + assert len(df) == N_QUARTERS, f'expected {N_QUARTERS} quarters, got {len(df)}' + assert not df.isnull().values.any() + assert (df.dtypes == 'float64').all() + + # The quarter index is the file's only self-describing column: it must be + # an unbroken 0.25 grid over the paper's sample. A silently truncated or + # re-based input fails here rather than in a lecture build. + assert df['date'].iloc[0] == FIRST_QUARTER + assert df['date'].iloc[-1] == LAST_QUARTER + assert ((df['date'].diff().iloc[1:] - 0.25).abs() < 1e-9).all() + + # Rates are stored as fractions, not percentages. A vintage that switched + # units would sail through every check above. + assert df['ur'].between(0.02, 0.12).all(), 'unemployment is not a fraction' + assert df['y3'].between(0.0, 0.06).all(), 'T-bill rate is not a fraction' + assert df['dp'].between(-0.02, 0.06).all(), 'inflation is not a fraction' + + +def run(): + df = pre_process(fetch()) + validate(df) + df.to_csv(os.path.join(PUBLISHED_DIR, OUT_FILE), index=False) + print(f'wrote {OUT_FILE}: {len(df)} quarters, ' + f'{df["date"].iloc[0]} to {df["date"].iloc[-1]}') + + +if __name__ == '__main__': + run() diff --git a/builders/README.md b/builders/README.md index 496dfae..9b7707e 100644 --- a/builders/README.md +++ b/builders/README.md @@ -39,6 +39,7 @@ re-fetched** — see `AGENTS.md`. | `japan_earthquakes.py` | `japan_earthquakes.csv` | committed | | `japan_population_by_age.py` | `japan_population_by_age.csv` | committed | | `us_adult_heights.py` | `us_adult_heights.csv` | committed | +| `NEWQDATA.py` | `NEWQDATA.csv` | committed — the **only** builder here that reads a committed input (`sources/NEWQDATA.MAT`) instead of fetching. Its upstream is published nowhere; see `sources/README.md`. Reproduces its output byte for byte | | `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | run by hand, no validate stage yet (PLAN Phase 5); its three outputs are the repo's only unmanifested files | | `webscrape_forbes.ipynb` | `forbes-global2000.csv`, `forbes-billionaires.csv` | **committed-frozen** — an undocumented Forbes API, a spoofed user-agent and hardcoded GDPR consent cookies. Defects recorded in the two manifests rather than fixed | | `generating_mini.md` | `SCF_plus_mini.csv`, `SCF_plus_mini_no_weights.csv` | **committed-frozen** — its `to_csv` calls are commented out upstream and stay that way. As written it still fetches the `high_dim_data` URL; that URL is historical, and the input is now committed at `sources/SCF_plus.dta`. See `sources/README.md` | diff --git a/lectures/NEWQDATA.csv b/lectures/NEWQDATA.csv new file mode 100644 index 0000000..66f1116 --- /dev/null +++ b/lectures/NEWQDATA.csv @@ -0,0 +1,212 @@ +date,y3,ur,dp +1948.25,0.0025062656641603454,0.03666666666666667,0.02935221201252737 +1948.5,0.0025062656641603454,0.03766666666666667,0.008230499136515444 +1948.75,0.0028078620136382515,0.03833333333333333,-0.012371291802546718 +1949.0,0.0029335807236166023,0.04666666666666667,-0.00833338155914441 +1949.25,0.0029335807236166023,0.05866666666666666,0.0 +1949.5,0.0025565191237657547,0.067,-0.004192878260035927 +1949.75,0.0026319087604962377,0.06966666666666667,-0.008438868645864595 +1950.0,0.0026821748176371063,0.064,0.0 +1950.25,0.0028832894571892975,0.05566666666666668,0.012631746905900564 +1950.5,0.0029084344599337175,0.04633333333333334,0.01659789140903783 +1950.75,0.0032857608668388494,0.042333333333333334,0.028399474521697957 +1951.0,0.003361260221742768,0.035,0.035367143837291344 +1951.25,0.003688555441246466,0.031000000000000003,0.0 +1951.5,0.003915269551249878,0.03166666666666667,0.0038535693159899723 +1951.75,0.0038648797871807528,0.033666666666666664,0.01904819497069441 +1952.0,0.003940466330346615,0.030666666666666672,-0.0037807228399060443 +1952.25,0.003940466330346615,0.029666666666666668,0.0037807228399061523 +1952.5,0.004545568698359981,0.03233333333333333,0.0037664827954768648 +1952.75,0.004369005172500273,0.028333333333333332,0.003752349618550372 +1953.0,0.00492412822831878,0.027,-0.0037523496185503527 +1953.25,0.005505140645031492,0.025666666666666667,0.007490671729157659 +1953.5,0.005126143330987931,0.027333333333333338,0.0037243990909822727 +1953.75,0.003461943705785009,0.037,0.0 +1954.0,0.0029587282483325428,0.05266666666666667,0.0 +1954.25,0.002430894920181448,0.057999999999999996,0.0 +1954.5,0.0018032458425165654,0.05966666666666666,-0.003724399090982328 +1954.75,0.0024560172422434245,0.05333333333333334,0.0 +1955.0,0.003059330959426232,0.04733333333333334,0.0 +1955.25,0.0039908636831405,0.044,-0.0037383221106072153 +1955.5,0.004016064257028162,0.040999999999999995,0.007462721201589594 +1955.75,0.0056062548709052695,0.042333333333333334,0.0 +1956.0,0.0060615206619885065,0.04033333333333334,0.0 +1956.25,0.006542526421741357,0.042,0.011090686694158138 +1956.5,0.005808544343584243,0.04133333333333333,0.007326040092072881 +1956.75,0.007302946361118057,0.041333333333333326,0.007272759329079878 +1957.0,0.007835924311522025,0.03933333333333334,0.010810916104215456 +1957.25,0.0077343612223816915,0.041,0.007142887512380204 +1957.5,0.007962906965023775,0.042333333333333334,0.007092228309491837 +1957.75,0.009030825891731098,0.04933333333333333,0.007042282625412951 +1958.0,0.006137438374081983,0.063,0.013937507843781678 +1958.25,0.0028330032341363776,0.07366666666666667,0.0 +1958.5,0.0022801874263951305,0.07333333333333333,0.0 +1958.75,0.006618516747615599,0.06366666666666668,0.0034542348680876036 +1959.0,0.007100055390502957,0.058333333333333334,0.0 +1959.25,0.00742979473617944,0.051000000000000004,0.0034423441909729197 +1959.5,0.008064516129032251,0.05266666666666667,0.006849341845574768 +1959.75,0.010228564212653168,0.056,0.0034071583216141346 +1960.0,0.010994565904208331,0.051333333333333335,0.0 +1960.25,0.008140736446808017,0.052333333333333336,0.006779686985378769 +1960.5,0.00578325370882582,0.05533333333333334,0.0 +1960.75,0.00578325370882582,0.06266666666666666,0.0067340321813441194 +1961.0,0.005631536604987941,0.06799999999999999,0.0 +1961.25,0.005757964345880051,0.07,0.0 +1961.5,0.005631536604987941,0.06766666666666667,0.006688988150796489 +1961.75,0.00578325370882582,0.062000000000000006,0.0 +1962.0,0.00684655658477662,0.056333333333333326,0.006644542718668511 +1962.25,0.006871900722430491,0.05533333333333334,0.0 +1962.5,0.00735368187770713,0.05566666666666668,0.006600684031352093 +1962.75,0.0068972461360319315,0.05533333333333334,0.0 +1963.0,0.00732831348057128,0.05766666666666667,0.0032840752011900187 +1963.25,0.007302946361118057,0.05733333333333334,0.0032733253449691085 +1963.5,0.00801370898644227,0.055,0.0032626456348163694 +1963.75,0.008700037826251394,0.05566666666666666,0.0064935293105483115 +1964.0,0.008878127522195411,0.05466666666666666,0.0 +1964.25,0.00875091418051599,0.05199999999999999,0.00323102058144654 +1964.5,0.008725475361880397,0.05,0.003220614700042157 +1964.75,0.009005372953611035,0.049666666666666665,0.006410278360919019 +1965.0,0.009616598096872764,0.048999999999999995,0.0 +1965.25,0.009922488449011446,0.04666666666666666,0.00953902304675891 +1965.5,0.009693053311793376,0.04366666666666667,0.0 +1965.75,0.010177538702426991,0.041,0.009448889197932289 +1966.0,0.011608204142535605,0.03866666666666667,0.009360442759563893 +1966.25,0.011684961303050079,0.03833333333333333,0.00619197024792098 +1966.5,0.012145748987854255,0.03766666666666667,0.012270092591814401 +1966.75,0.013556315722792478,0.037000000000000005,0.003044142381228273 +1967.0,0.011940902651285112,0.03833333333333333,0.003034903695154111 +1967.25,0.009693053311793376,0.03833333333333333,0.009049835519917856 +1967.5,0.010636953940220906,0.03799999999999999,0.008968669982760536 +1967.75,0.011531458628363245,0.039,0.011834457647002798 +1968.0,0.012658227848101111,0.03733333333333333,0.008784829555732811 +1968.25,0.013633368810501256,0.035666666666666666,0.011594332780919446 +1968.5,0.01345359649345057,0.035333333333333335,0.011461443519006598 +1968.75,0.013556315722792478,0.033999999999999996,0.014144507386164743 +1969.0,0.015589295688823501,0.033999999999999996,0.013947227480850552 +1969.25,0.015511944959252677,0.034333333333333334,0.013755375068485419 +1969.5,0.01781170483460559,0.035666666666666666,0.013568729206068796 +1969.75,0.01781170483460559,0.035666666666666666,0.016043124840575684 +1970.0,0.02006987478642297,0.041666666666666664,0.015789801732635073 +1970.25,0.01654425779562385,0.04766666666666666,0.012970350442627405 +1970.5,0.016389277093126653,0.051666666666666666,0.010256500167189282 +1970.75,0.014996574386561434,0.058333333333333334,0.01519016549397502 +1971.0,0.011224592982101322,0.059333333333333335,0.005012541823544414 +1971.25,0.009744029888423267,0.05900000000000001,0.01242251999855711 +1971.5,0.013684744044602182,0.060333333333333336,0.00738010729762246 +1971.75,0.01127572432623758,0.059333333333333335,0.0073260400920731016 +1972.0,0.00852201099288985,0.05766666666666667,0.007272759329079657 +1972.25,0.009361830982361541,0.057,0.007220247973487097 +1972.5,0.010049997474875116,0.05566666666666666,0.00954661188357991 +1972.75,0.011992106461569563,0.05366666666666666,0.009456335242035444 +1973.0,0.013710433614638085,0.04933333333333333,0.02095536517598809 +1973.25,0.01589881647787883,0.04933333333333333,0.018265347977293404 +1973.5,0.020434194749866075,0.047999999999999994,0.022372297754532984 +1973.75,0.018381791333570874,0.04766666666666666,0.024044874254002868 +1974.0,0.01980980547127964,0.051333333333333335,0.03188367840522198 +1974.25,0.02126790410294377,0.052,0.024794658613216274 +1974.5,0.01923811950566945,0.056333333333333326,0.032131278182793196 +1974.75,0.0190044326692822,0.066,0.02536721387842295 +1975.0,0.01589881647787883,0.08266666666666665,0.01719240054037277 +1975.25,0.014224498592763535,0.08866666666666667,0.013170463189745227 +1975.5,0.015563510803056957,0.08466666666666667,0.020352228848898517 +1975.75,0.015125367982945948,0.083,0.01814931850567727 +1976.0,0.012325057575987675,0.07733333333333334,0.007168489478612497 +1976.25,0.012299438173811739,0.07566666666666667,0.01242251999855711 +1976.5,0.013248220482812822,0.07733333333333334,0.015748356968139112 +1976.75,0.012453174040700565,0.07766666666666668,0.01379332213233577 +1977.0,0.011684961303050079,0.075,0.020339684237122783 +1977.25,0.011480301421129768,0.07133333333333333,0.014987790965491708 +1977.5,0.013145563688862882,0.06899999999999999,0.013136477905369981 +1977.75,0.01564086938858411,0.06666666666666667,0.016181582851241695 +1978.0,0.016363451570281518,0.06333333333333334,0.01750243564977276 +1978.25,0.015976226156307893,0.06,0.024923408452456934 +1978.5,0.017837604010280117,0.060333333333333336,0.022814677766171264 +1978.75,0.020382133108849176,0.059,0.020834086902842053 +1979.0,0.023934468194035663,0.05866666666666667,0.02902961467511403 +1979.25,0.02422287089670716,0.056999999999999995,0.03237439665901583 +1979.5,0.023646227863650227,0.05866666666666667,0.030015895940265568 +1979.75,0.030131341746072682,0.05966666666666666,0.03304993467255219 +1980.0,0.030927835051546504,0.063,0.04076997756271521 +1980.25,0.03412616339193386,0.07333333333333333,0.029522439266321834 +1980.5,0.02056437209777018,0.07666666666666666,0.0168273201325253 +1980.75,0.029919151346619266,0.074,0.02936206233684953 +1981.0,0.039015013766948936,0.07433333333333333,0.025144181801025164 +1981.25,0.03543786078537958,0.074,0.021217993094845188 +1981.5,0.03882612647708106,0.07400000000000001,0.02832433357714079 +1981.75,0.03503596749987059,0.08233333333333334,0.01068386230831261 +1982.0,0.031672340864541404,0.08833333333333333,0.006355953600698727 +1982.25,0.03279111799638512,0.09433333333333335,0.023996978311350333 +1982.5,0.02920365367297051,0.099,0.007190580545354209 +1982.75,0.019653827525555112,0.10666666666666667,0.0 +1983.0,0.020043861886061,0.10366666666666667,0.004085807522580354 +1983.25,0.020955103499323613,0.10133333333333333,0.01316474709121107 +1983.5,0.023227258774173754,0.09366666666666666,0.010010093595100425 +1983.75,0.022076860179885527,0.08533333333333334,0.00991088389945396 +1984.0,0.02275632830478136,0.07866666666666668,0.014684551682921099 +1984.25,0.024826420025108265,0.07433333333333333,0.007744472395477762 +1984.5,0.025956704627064786,0.07433333333333333,0.009597002641009484 +1984.75,0.024957720494029667,0.07300000000000001,0.007611835039630074 +1985.0,0.019783805833163415,0.07233333333333333,0.012246973609973294 +1985.25,0.02027802576202009,0.073,0.006532921041623095 +1985.5,0.01801893515219377,0.07200000000000001,0.005565877077444956 +1985.75,0.0182262498727217,0.07033333333333334,0.012867824611392983 +1986.0,0.017993026747766816,0.07033333333333334,-0.0036596564175303986 +1986.25,0.015383053256841084,0.07166666666666667,0.002745997148855844 +1986.5,0.014790572595580587,0.06966666666666667,0.005469475804535255 +1986.75,0.013119902740489353,0.06833333333333333,0.007246408520767253 +1987.0,0.013761816661175486,0.06599999999999999,0.013447087431214033 +1987.25,0.014301653311694862,0.06266666666666666,0.011509644542653354 +1987.5,0.014430270599274708,0.06,0.010507977598415165 +1987.75,0.015563510803056957,0.058333333333333334,0.0078091503141649475 +1988.0,0.01473908521271472,0.057,0.007748639716761098 +1988.25,0.014996574386561434,0.05466666666666666,0.012782449286924565 +1988.5,0.017112924962493947,0.05466666666666666,0.012621118313214792 +1988.75,0.01871896090665981,0.05333333333333334,0.009983443984183205 +1989.0,0.021111479845812076,0.052000000000000005,0.012340757192412188 +1989.25,0.022102976874920133,0.05233333333333333,0.014610649517434854 +1989.5,0.020095889013567403,0.05233333333333333,0.0064257249209704916 +1989.75,0.01947191354878175,0.05366666666666667,0.01193806458122503 +1990.0,0.01947191354878175,0.053,0.01725533009076184 +1990.25,0.01980980547127964,0.05333333333333334,0.010827638652063393 +1990.5,0.019419950048422452,0.057,0.01980262729617973 +1990.75,0.01825217014993763,0.061333333333333344,0.012739025777429932 +1991.0,0.015795621920869474,0.066,0.004457659684518188 +1991.25,0.014327374160010287,0.06833333333333333,0.008856146443710062 +1991.5,0.014147355610770296,0.06866666666666667,0.007320676910848397 +1991.75,0.012632591579960106,0.07100000000000001,0.008714652102443909 +1992.0,0.009591115598182709,0.07366666666666667,0.006486509229606632 +1992.25,0.009463722397476282,0.076,0.007158226700170449 +1992.5,0.008089921621008633,0.07633333333333332,0.007107350458652018 +1992.75,0.007201490658206078,0.07366666666666667,0.008462673918733728 +1993.0,0.007556675062972307,0.07133333333333333,0.006997929187644637 +1993.25,0.007226852667892247,0.07066666666666667,0.006949298293205944 +1993.5,0.00765820233776715,0.06799999999999999,0.0048359334314163165 +1993.75,0.007607436142878621,0.06633333333333333,0.008919441637069146 +1994.0,0.007505919097274605,0.06566666666666666,0.0054496047675644645 +1994.25,0.00928542591844983,0.062000000000000006,0.0054200674693393345 +1994.5,0.010943462986832353,0.06,0.009414998934601883 +1994.75,0.012530059486141054,0.056333333333333326,0.0053404666313172835 +1995.0,0.014481726647898707,0.05466666666666666,0.0072968814642759275 +1995.25,0.014327374160010287,0.056666666666666664,0.00789997525315593 +1995.5,0.013736124486795953,0.056666666666666664,0.00457966125902451 +1995.75,0.01337657073368459,0.05566666666666666,0.005857485019579088 +1996.0,0.012658227848101111,0.05533333333333334,0.009686869418221236 +1996.25,0.012530059486141054,0.055,0.007682496172016431 +1996.5,0.013042927694060902,0.05266666666666667,0.006357300495466498 +1996.75,0.012632591579960106,0.05333333333333334,0.008832864998508664 +1997.0,0.012735144441349933,0.05266666666666667,0.004387346429626051 +1997.25,0.013068584743187062,0.05,0.002498439775598719 +1997.5,0.012786428661855975,0.048666666666666664,0.006218925515656017 +1997.75,0.012581322937498474,0.04666666666666667,0.003712875552426268 +1998.0,0.012760785902369731,0.04666666666666667,0.0018512809977230173 +1998.25,0.012530059486141054,0.044333333333333336,0.004306373926687039 +1998.5,0.012555690562980892,0.045,0.0036764747293086273 +1998.75,0.00999899000101001,0.044333333333333336,0.006097579868118538 +1999.0,0.010969013799727145,0.043,0.003034903695154111 +1999.25,0.010841272649162237,0.04300000000000001,0.006644542718668511 +1999.5,0.011505879377923911,0.042333333333333334,0.010182777413856646 +1999.75,0.012299438173811739,0.040999999999999995,0.006534029780648527 +2000.0,0.013479274348839443,0.04033333333333333,0.013525639893887215 +2000.25,0.014353096312826441,0.04033333333333333,0.005824128285988713 +2000.5,0.015125367982945948,0.04,0.008097210232619303 +2000.75,0.015511944959252677,0.03966666666666667,0.005743841179252093 diff --git a/lectures/NEWQDATA.csv.yml b/lectures/NEWQDATA.csv.yml new file mode 100644 index 0000000..5365b10 --- /dev/null +++ b/lectures/NEWQDATA.csv.yml @@ -0,0 +1,164 @@ +# Manifest for NEWQDATA.csv — migrated out of QuantEcon/lecture-python.myst, +# where it sat at lectures/_static/lecture_specific/phillips_drifts_volatilities/ +# and was read by URL (PLAN Phase 8, wave B1'). +# +# RECLASSIFIED AT MIGRATION. scripts/audit_annotations.yml recorded this file as +# `verbatim` with no builder. It is not: it is a container conversion of the +# authors' MATLAB file, and the conversion is now recovered, committed at +# builders/NEWQDATA.py, and reproduces these bytes exactly. The annotation was +# an unevidenced guess, and it is deleted by the PR that lands this manifest. +# +# The same annotation credited the file to lecture-python.myst#1001. Also wrong: +# `git log --diff-filter=A` returns exactly one commit, 370a58f (#992). + +filename: NEWQDATA.csv +title: Cogley-Sargent (2005) "Drifts and Volatilities" — quarterly US inflation, unemployment and T-bill rate +description: > + The four quarterly series behind Cogley and Sargent's post-war US evidence, + 1948Q2-2000Q4: a decimal quarter index, the nominal three-month Treasury bill + rate, the seasonally adjusted civilian unemployment rate and CPI inflation, + all as fractions rather than percentages. `phillips_drifts_volatilities` + reads it to fit the drifting-coefficient, stochastic-volatility VAR, and + rebuilds the post-2000Q4 continuation from FRED separately — this file is the + frozen half of that pair. + +# Constructed, not verbatim: these bytes are a value-preserving conversion of +# the authors' NEWQDATA.MAT into CSV. No filtering, no rescaling, no +# reordering — which is exactly why builders/NEWQDATA.py reproduces the file +# byte for byte from the committed input. +class: constructed + +source: + name: Cogley and Sargent (2005), "Drifts and Volatilities" — the authors' NEWQDATA.MAT + url: https://doi.org/10.1016/j.red.2004.10.009 + # The ARTICLE doi. There is no dataset doi: Crossref's record carries no + # dataset component, only Elsevier text-mining licences for the article text + # (checked 2026-08-13). + doi: 10.1016/j.red.2004.10.009 + version: > + MAT-file header "MATLAB 5.0 MAT-file, Platform: PCWIN, Created on: + Sun Mar 11 12:15:22 2001" — the vintage of the authors' earlier paper + (Cogley and Sargent 2001), whose data the 2005 paper reuses. + series: date, y3, ur, dp — quarterly, 1948Q2-2000Q4 + citation: > + Cogley, Timothy, and Thomas J. Sargent. 2005. "Drifts and Volatilities: + Monetary Policies and Outcomes in the Post WWII U.S." Review of Economic + Dynamics 8 (2): 262-302. doi:10.1016/j.red.2004.10.009. Data as distributed + with the authors' MATLAB replication files (NEWQDATA.MAT). + note: > + NOT hosted by the authors or the journal. The underlying series are US + federal statistics — the BLS seasonally adjusted CPI for all urban + consumers, the BLS seasonally adjusted civilian unemployment rate, and the + three-month Treasury bill rate. The paper's own data section (§3.1) says + only: "Inflation is measured by the CPI for all urban consumers, + unemployment by the civilian unemployment rate, and the nominal interest + rate by the yield on 3-month Treasury bills. Inflation and unemployment + data are quarterly and seasonally adjusted, and Treasury bill data are the + average of daily rates in the first month of each quarter." Where the + MATLAB file itself could and could not be located is recorded in + sources/README.md, which is the audit trail for the committed input. + +license: + name: null + url: https://doi.org/10.1016/j.red.2004.10.009 + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). There is no licence to name and no place that states + # one: Review of Economic Dynamics had no data archive or data-availability + # policy in 2005 (economicdynamics.org publishes none, and Crossref shows no + # dataset component for the article), and the authors' own README.TXT beside + # NEWQDATA.MAT lists the files and states no terms. Both checked 2026-08-13. + redistribution: permitted + verified: 2026-08-13 + note: > + An inherited exposure, served publicly from lecture-python.myst since + 2026-07-23. `permitted` rests on what the numbers are rather than on a + licence grant: every value is a simple transformation of three US federal + statistical series (BLS CPI-U, BLS civilian unemployment rate, three-month + Treasury bill rate), which are US Government works in the public domain, + and the authors' contribution is the assembly, credited in full by the + citation above. No author or journal permission has been sought. + Registered for licence review on QuantEcon/data-lectures#35. + +retrieved: null # inherited from lecture-python.myst with no + # recorded retrieval date; not reconstructed + # from git history (AGENTS.md). It costs + # nothing here — the builder reproduces these + # bytes exactly from a committed input, so the + # vintage is pinned by the hash, not by a date. + # See the AGENTS.md note on this pairing. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to the copy every consumer reads today — sha256 98fb884b… +# against lecture-python.myst @ 06c784f, +# lectures/_static/lecture_specific/phillips_drifts_volatilities/. The +# lecture-python.zh-cn copy carries the identical git blob. Verified +# 2026-08-13; a repoint to this file therefore cannot change a figure. + +integrity: + sha256: 98fb884bf59fbf211a4095c558b210ff8df471f863357803cc3ad88d0066bcaf + upstream: + # `verified` in the sense AGENTS.md defines for a constructed dataset — + # "re-run the builder and compare" — not by analogy. builders/NEWQDATA.py + # was run against the committed sources/NEWQDATA.MAT on the date below and + # reproduced this file byte for byte, all 13,738 of them. + status: verified + date: 2026-08-13 + against: sources/NEWQDATA.MAT + note: > + The input hashes to + ab760926f37b41e7478c5227454f3d5c142acb4ad27428f0881ad5140dcb8162 and + holds exactly four 211-element float64 arrays named date, y3, ur, dp; all + 844 values are bitwise identical to this CSV parsed with + float_precision='round_trip'. What could NOT be established is a + retrieval date or an authors'-own download: neither tomsargent.com nor + Cogley's NYU pages (live or in the Wayback Machine), the RED/SED site, + nor the RePEc record carries the replication files today, and + ScienceDirect returns 403 to an anonymous fetch while Crossref lists no + dataset component. That search is what put the input in sources/ rather + than leaving the builder fetching a third-party mirror — see + sources/README.md. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13, and asserted by the builder's +# validate() stage on every run. + +schema: + format: csv + columns: + - {name: date, dtype: float64, description: "quarter as year + (quarter-1)/4 — 1948.25 is 1948Q2, 2000.75 is 2000Q4; an unbroken 0.25 grid with no gaps"} + - {name: y3, dtype: float64, description: "nominal three-month Treasury bill rate as a quarterly fraction, first month of the quarter. Reproduces bitwise as 1/(1 - TB3MS/400) - 1 for all 211 quarters — a discount-to-simple conversion, NOT the log the lecture prose describes"} + - {name: ur, dtype: float64, description: "seasonally adjusted civilian unemployment rate as a fraction, quarterly mean of the three monthly readings; the lecture applies 0.01*log[u/(1-u)] before estimation"} + - {name: dp, dtype: float64, description: "inflation as the log difference of the seasonally adjusted CPI for all urban consumers, point sampled in the third month of the quarter"} + row_count_floor: 211 # frozen 2001 vintage — exact row count by + # design, one row per quarter + date_range: {start: 1948.25, end: 2000.75} # decimal quarters, as the + # `date` column encodes them + known_nulls: {} # genuinely none — a complete quarterly panel + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at `landed`, populated by the flip PR once the repoints merge. +# Two reference-holders will be recorded: lecture-python.myst +# (phillips_drifts_volatilities.md:103-105) and lecture-python.zh-cn +# (phillips_drifts_volatilities.md:111-113), the latter outside the audit's +# SCAN_REPOS and so recorded here or nowhere. The generated +# lecture-python.notebooks mirror self-heals after a publish tag and is +# deliberately not listed, per the wave A4 precedent. +# +# BOTH reads build the URL by string concatenation across three source lines, +# and this is the only one of the seven Track B files to use the +# raw.githubusercontent.com form. A repoint must COLLAPSE the continuation, not +# patch the stem line — patching line 1 of 3 leaves 'NEWQDATA.csv' appended to +# a path that no longer exists, and the flat published tree has no +# _static/lecture_specific// segment to append it to. +consumers: [] + +builder: builders/NEWQDATA.py +builder_status: committed diff --git a/lectures/fp.dta b/lectures/fp.dta new file mode 100644 index 0000000..3696f14 Binary files /dev/null and b/lectures/fp.dta differ diff --git a/lectures/fp.dta.yml b/lectures/fp.dta.yml new file mode 100644 index 0000000..f544258 --- /dev/null +++ b/lectures/fp.dta.yml @@ -0,0 +1,248 @@ +# Manifest for fp.dta — migrated out of QuantEcon/lecture-python.myst, where it +# sat at lectures/_static/lecture_specific/mle/ and was read by URL +# (PLAN Phase 8, wave B1'). +# +# THE FILENAME IS TWO LETTERS AND SAYS NOTHING — deliberately kept at migration +# (decision 2026-08-13; a batch rename across the five generic names in +# .dev/qeld/migration-catalog.md §5.1 is proposed separately, and any rename +# has to be paired with a prose edit because mle.md names `mle/fp.dta` in +# text). Until then the title and description below are the only thing that +# tells a reader what these bytes are. +# +# NOT the same dataset as forbes-billionaires.csv in this same tree: that one +# is a list of PEOPLE (the Forbes person-level ranking, read by heavy_tails in +# lecture-python-intro); this one is a panel of COUNTRIES. They share neither +# rows nor columns. + +filename: fp.dta +title: Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates +description: > + The country-year panel from Daniel Treisman's AER Papers & Proceedings paper + "Russia's Billionaires": 219 countries over 1990-2015, pairing Forbes-derived + counts and net worth of billionaires per country with World Bank, EFW/KPMG + and Worldwide Governance Indicators covariates — GDP, population, market + capitalization, top marginal income tax rate, rule of law, natural-resource + rents, years in GATT/WTO and privatization proceeds. `mle` uses `numbil0` + (billionaires per country, 0 where a country has none in the Forbes list) to + motivate a Poisson model, histograms the 2008 cross-section, and replicates + the first three columns of the paper's table 1. + +# Verbatim: the author's own Stata file, republished exactly as distributed in +# the AER replication package — same 1,024,085 bytes, same sha256. The package +# zip was re-downloaded 2026-08-13 and its fp.dta member hashes to the sha256 +# below exactly, so no QuantEcon processing can have been applied. +class: verbatim + +source: + name: Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings) + url: https://www.aeaweb.org/articles?id=10.1257/aer.p20161068 + file_url: https://assets.aeaweb.org/asset-server/articles-attachments/aer/data/10605/P2016_1068_data.zip + doi: 10.3886/E116312V1 # the replication package, at openICPSR + version: > + P2016_1068_data — the AEA package id. fp.dta is one of its five files, + alongside `aer russian billionaires.do` (5,257 B), billionaires.xlsx + (63,880 B), Readme.pdf (313,778 B) and table3data.dta (1,286,094 B). + citation: > + Treisman, Daniel. 2016. "Russia's Billionaires." American Economic Review + 106 (5): 236-241. https://doi.org/10.1257/aer.p20161068. Data: Treisman, + Daniel. Replication data for: Russia's Billionaires. Nashville, TN: + American Economic Association [publisher], 2016. Ann Arbor, MI: + Inter-university Consortium for Political and Social Research + [distributor], 2019-12-07. https://doi.org/10.3886/E116312V1 + note: > + The package's Readme.pdf documents fp.dta as the "Stata data file + containing country year panel", lists every variable, and states "Source of + data: Forbes." for the billionaire series; the covariates name their own + upstreams in the variable labels (WDI, World Bank Privatization Database, + Privatization Barometer, OECD, Boardman and Vining (2012), EFW and KPMG, + World Bank Governance Indicators). This file is therefore itself a + composite assembled by the author — the citation above is the whole + provenance chain there is, and the right one to reproduce. + +license: + name: null + url: https://www.aeaweb.org/journals/data/data-code-policy + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). No licence can be named: the artifact actually fetched + # and compared — the AEA-hosted package zip — carries no licence file, and + # DataCite records an EMPTY rightsList for 10.3886/E116312V1. The openICPSR + # deposit does hold a LICENSE.txt (14.6 KB, 2019-12-07, visible in a 2025 + # Wayback capture of the project page), but www.openicpsr.org answers every + # automated request with a Cloudflare 403, so its text was NOT read. That + # gap is recorded here rather than papered over. + redistribution: permitted + verified: 2026-08-13 + note: > + `permitted` on the gate, not on a named licence: the AEA publishes this + package openly (the article page links it; the asset server serves the zip + with no login), its policy states that authors must permit others to use + all files in the deposit for the purpose of replication, and nothing found + imposes non-commercial or no-redistribution terms. Two honest caveats: this + is a pre-2019 deposit migrated to openICPSR, and the AEA Data Editor's FAQ + says such migrated materials stay copyrighted by the original authors + unless the deposit's LICENSE file states otherwise — the file that could + not be read. Distinct from forbes-billionaires.csv's `restricted`: that is + Forbes' own editorial product scraped from their API, this is a scholar's + replication deposit derived from published counts. Inherited exposure — + served publicly from python.quantecon.org for years. Registered for licence + review on QuantEcon/data-lectures#35. + +retrieved: null # inherited from lecture-python.myst with no + # recorded retrieval date; not reconstructed + # from git history (AGENTS.md). It costs + # nothing here — upstream is frozen and was + # re-fetched byte-identical below, so the + # vintage is pinned by the hash, not by a date. + # See the AGENTS.md note on this pairing. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to every copy in play — lecture-python.myst @ 06c784f +# lectures/_static/lecture_specific/mle/fp.dta, the published copy at +# python.quantecon.org/_static/lecture_specific/mle/fp.dta, the archived +# QuantEcon/lecture-python copy that lecture-stats reads, and lecture-dp's and +# lecture-python.zh-cn's orphan copies. All hash to the sha256 below (verified +# 2026-08-13), so a repoint cannot change a figure or a coefficient. + +integrity: + sha256: de99b9c0de6fb08cb732e306971ba87a308c5c4ef8ca3ae23fbbb50eabdecd8d + upstream: + status: verified + date: 2026-08-13 + against: https://assets.aeaweb.org/asset-server/articles-attachments/aer/data/10605/P2016_1068_data.zip + note: > + Re-fetched and byte-compared, not assumed: the AEA package zip still + resolves on the asset server, and its `P2016_1068_data/fp.dta` member is + 1,024,085 B hashing to the sha256 above. FINDING THE ARTIFACT IS THE + WORK — the article page's "Replication Package" link and the legacy + /articles/materials/5536 download both 302 to openICPSR + (doi.org/10.3886/E116312V1), which serves a Cloudflare 403 to every + automated fetch, and no Wayback capture of the download exists; the + asset-server URL above is the one automatable route left. Upstream is + frozen — a 2016 deposit migrated to openICPSR 2019-12-07 — so no vintage + drift is possible: the bytes carry their own stamp, a Stata header + timestamp of " 9 Jan 2016 23:43" and a zip member mtime of 2016-01-10. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13. 28 columns are Stata `float` +# and land as float32 under either pandas version, 7 are Stata `double` and +# land as float64 — that is why `year` and `ccode` are float32 rather than +# integers. `df["year"] == 2008` is still exact at these magnitudes, but do not +# assume int. `country` is `object` under pandas 2 and `str` under the +# anaconda=2026.07 pin (pandas 3), and is declared as the pin will read it. +# The file declares no value labels, so nothing reads back as a categorical. + +schema: + format: dta + columns: + - {name: country, dtype: str, description: "country name — the only string column"} + - {name: ccode, dtype: float32, description: "country code (label reads '(mean) ccode' — a Stata collapse artifact)"} + - {name: year, dtype: float32, description: "year, 1990-2015 (label reads '(mean) year'); null on 338 rows"} + - {name: cyear, dtype: float32, description: "country-year identifier"} + - {name: numbil, dtype: float32, description: "number of billionaires, Forbes"} + - {name: numbil0, dtype: float32, description: "number of billionaires, 0 if none in the Forbes list — THE LECTURE'S DEPENDENT VARIABLE"} + - {name: numbilall, dtype: float32, description: "number of billionaires worldwide"} + - {name: netw, dtype: float32, description: "total net worth of billionaires, bn$, Forbes"} + - {name: netw0, dtype: float32, description: "net worth of billionaires, 0 if none in the Forbes list"} + - {name: netwall, dtype: float32, description: "total net worth of billionaires worldwide, bn$, Forbes"} + - {name: gdpcurdol, dtype: float64, description: "GDP in current dollars, WDI"} + - {name: gdpbill, dtype: float32, description: "GDP, current bn dollars, WDI"} + - {name: pop, dtype: float64, description: "population, WDI"} + - {name: lnpop, dtype: float32, description: "ln population, WDI — lecture regressor"} + - {name: gdppc, dtype: float32, description: "GDP per capita, current dollars, WDI"} + - {name: lngdp, dtype: float32, description: "ln GDP in current dollars, WDI"} + - {name: lngdppc, dtype: float32, description: "ln GDP per capita, current dollars, WDI — lecture regressor"} + - {name: d7netwall, dtype: float32, description: "change in netwall from 7 years earlier, bn$"} + - {name: d7numbilall, dtype: float32, description: "change in number of billionaires from 7 years earlier"} + - {name: lowinc08, dtype: float64, description: "World Bank 2008 low-income classification"} + - {name: midinc08, dtype: float64, description: "World Bank 2008 middle-income classification"} + - {name: totppb9008, dtype: float32, description: "privatization proceeds 1990-2008, bn$, World Bank Privatization Database"} + - {name: privprocbarb, dtype: float32, description: "privatization value, bn$, Privatization Barometer"} + - {name: fullprivproc, dtype: float32, description: "estimated total privatization proceeds 1990-2008"} + - {name: wtoyear, dtype: float64, description: "year joined the WTO"} + - {name: gattwto, dtype: float64, description: "first year joined GATT or WTO"} + - {name: gattwto08, dtype: float32, description: "years in GATT or WTO as of 2008 — lecture regressor"} + - {name: mcapbdol, dtype: float32, description: "market capitalization, bn dollars, WDI"} + - {name: mcapbdol08, dtype: float32, description: "market capitalization 2008, WDI with additional sources"} + - {name: lnmcap08, dtype: float32, description: "ln market capitalization 2008 — lecture regressor (models 2 and 3)"} + - {name: topintaxnew, dtype: float32, description: "top marginal income tax rate, average 1990-2008, EFW and KPMG"} + - {name: topint08, dtype: float32, description: "top marginal income tax rate, EFW with additional sources — lecture regressor (models 2 and 3)"} + - {name: rintr, dtype: float32, description: "real interest rate, %, average 1988-2007, WDI — lecture regressor (models 2 and 3)"} + - {name: noyrs, dtype: float64, description: "number of years of real-interest-rate data available in WDI"} + - {name: roflaw, dtype: float32, description: "rule of law index, 2008, World Bank Governance Indicators — lecture regressor (model 3)"} + - {name: nrrents, dtype: float32, description: "total natural resources rents (% of GDP), WDI — lecture regressor (model 3)"} + row_count_floor: 5432 # frozen replication file — exact row count by design + date_range: {start: 1990, end: 2015} + # Nulls are extensive and correct. This is an author-assembled panel of 219 + # countries over 1990-2015 with covariates of very different coverage, and + # several columns are 2008-only or period-average constructs, so most cells + # of most columns are legitimately empty. `year` itself is null on 338 rows, + # and 253 (country, year) pairs repeat — neither reaches the lecture, which + # slices `year == 2008` (213 rows, numbil0 complete) and passes + # missing="drop" to statsmodels. A blanket no-nulls rule would reject this + # file wrongly. + known_nulls: + year: 338 + cyear: 3 + numbil: 4413 + numbil0: 1316 + numbilall: 1316 + netw: 4413 + netw0: 1316 + netwall: 1316 + gdpcurdol: 1735 + gdpbill: 1735 + pop: 1618 + lnpop: 1618 + gdppc: 1735 + lngdp: 1735 + lngdppc: 1735 + d7netwall: 2807 + d7numbilall: 2807 + lowinc08: 1000 + midinc08: 1000 + totppb9008: 2180 + privprocbarb: 4978 + fullprivproc: 1750 + wtoyear: 1998 + gattwto: 1998 + mcapbdol: 3177 + mcapbdol08: 1612 + lnmcap08: 1612 + topintaxnew: 1783 + topint08: 1658 + rintr: 885 + noyrs: 3 + roflaw: 252 + nrrents: 3664 + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at `landed`, populated by the flip PR once the repoints merge. +# Two reference-holders will be recorded: lecture-python.myst (mle.md:166) and +# lecture-python.zh-cn (mle.md:178), the latter outside the audit's SCAN_REPOS +# and so recorded here or nowhere. The generated lecture-python.notebooks +# mirror self-heals after a publish tag and is deliberately not listed, per the +# wave A4 precedent. +# +# lecture-stats is NOT listed, and the reason is worth recording because it +# looks like a consumer: its mle.md code read points at the ARCHIVED +# QuantEcon/lecture-python blob, a separate byte-identical copy that this +# migration does not touch. What it DOES share is the prose link — see below. +# +# A repoint must also edit PROSE, which no URL rewrite of a code cell touches: +# mle.md:160 in python.myst, its zh-cn twin at mle.md:172, and the identical +# line in lecture-stats/lectures/mle.md:132 all link the published-site copy at +# python.quantecon.org/_static/lecture_specific/mle/fp.dta. That URL dies when +# this file is deleted from python.myst and the site republishes, and +# lecture-stats runs a DAILY sphinx linkcheck that will go red within 24 hours. +consumers: [] + +builder: null +builder_status: not-applicable # verbatim — no builder diff --git a/lectures/maketable1.dta b/lectures/maketable1.dta new file mode 100644 index 0000000..9c4887f Binary files /dev/null and b/lectures/maketable1.dta differ diff --git a/lectures/maketable1.dta.yml b/lectures/maketable1.dta.yml new file mode 100644 index 0000000..67b41f8 --- /dev/null +++ b/lectures/maketable1.dta.yml @@ -0,0 +1,176 @@ +# Manifest for maketable1.dta — migrated out of QuantEcon/lecture-python.myst, +# where it sat at lectures/_static/lecture_specific/ols/ and was read by URL +# (PLAN Phase 8, wave B1'). +# +# One of three AJR (2001) replication tables landing together. They share a +# source, a licence position and an integrity story, and differ only in +# columns — so the source/license blocks below are deliberately identical in +# all three manifests. lecture-dp and lecture-python.zh-cn each hold a +# byte-identical orphan copy that no lecture reads (Track X, not this wave). + +filename: maketable1.dta +title: Acemoglu-Johnson-Robinson (2001) colonial origins — table 1 replication data +description: > + The 163-country cross-section behind table 1 of "The Colonial Origins of + Comparative Development": average protection against expropriation risk + 1985-95, log GDP per capita in 1995, European settler mortality and + settlement share, constraint-on-executive and democracy scores, and the + 64-country base-sample flag. `ols` reads it twice — once to fit the simple + OLS regression of `logpgp95` on `avexpr` and draw the paper's figure 2, and + again in the exercise solution that recovers the same coefficients from + `np.linalg.solve`. + +# Verbatim: the authors' own replication file as distributed inside +# maketable1.zip on Daron Acemoglu's data archive. Not merely asserted — the +# zip was re-downloaded 2026-08-13 and the enclosed .dta hashes to the sha256 +# below exactly, so no QuantEcon processing can have been applied. +class: verbatim + +source: + name: Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 1 (program and data files) + url: https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive + file_url: https://www.dropbox.com/s/e0hsiyv86vu8e96/maketable1.zip?dl=0 + doi: 10.1257/aer.91.5.1369 + version: Stata dta format 113, authored 18 Jan 2010 17:14 (from the file's own header) + citation: > + Acemoglu, Daron, Simon Johnson, and James A. Robinson. 2001. "The Colonial + Origins of Comparative Development: An Empirical Investigation." American + Economic Review 91 (5): 1369-1401. + note: > + The erratum `ols.md` cites is a table 2 note, and it does reach this file. + The lecture says "an observation was mistakenly dropped from the results in + the original paper (see the note located in `maketable2.do`)" + (ols.md:214-216). That note lives in maketable2.do and nowhere else — + maketable1.do, fetched in the same deposit, carries no such remark — and + its subject is the table 2 OLS world sample. Table 1 itself is summary + statistics and is not what the note is about. But the regression `ols.md` + runs on THIS file is table 2's column 1, and the two deposits agree exactly + where it matters: measured 2026-08-13, maketable1.dta and maketable2.dta + carry identical `shortnam`, `avexpr` and `logpgp95` over all 163 rows, both + give n=111 after dropping missings, and both give b0=4.6261, b1=0.5319, + R-squared=0.6113 — the 4.63 / 0.53 / 0.611 the lecture quotes. + +license: + name: null + url: https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). There is no licence to name: the archive page states + # only that the zipped data and program files are there to download, the + # deposit's own readme.TXT states terms nowhere, and the page carries no + # notice beyond the department footer. Nor is there an AEA deposit to fall + # back on — the AEA article page for 10.1257/aer.91.5.1369 links no + # replication package, and the current AEA data and code availability policy + # claims no retroactive reach. A 2001 paper predates that regime entirely. + redistribution: permitted + verified: 2026-08-13 + note: > + An inherited exposure: served publicly by QuantEcon since the + lecture-python.rst era, and cached here with attribution rather than left + silent. Nothing in the deposit imposes non-commercial or no-redistribution + terms — the class that earns `restricted` here — and the authors publish it + for open download. One chain deserves naming anyway, and it is shared by + all three tables: `avexpr` descends from the Political Risk Services + Group's commercial country-risk data, which the lecture links directly. + What this file carries is AJR's own 1985-95 average, not a PRS series, and + AJR have redistributed it openly for two decades; recorded `permitted` on + that basis, with the PRS chain registered for review on + QuantEcon/data-lectures#35. + +retrieved: null # inherited from lecture-python.myst with no + # recorded retrieval date; not reconstructed + # from git history (AGENTS.md). It costs + # nothing here — upstream is frozen and was + # re-fetched byte-identical below, so the + # vintage is pinned by the hash, not by a date. + # See the AGENTS.md note on this pairing. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to the copy every consumer reads today — sha256 7a8687cc… +# against lecture-python.myst @ 06c784f, +# lectures/_static/lecture_specific/ols/. The lecture-python.zh-cn and +# lecture-dp copies of the same path carry the identical git blob. Verified +# 2026-08-13; a repoint to this file therefore cannot change a figure. + +integrity: + sha256: 7a8687cc96c505a46dd24e98ab8b2332b31495003d68d476f9b280fcf002863b + upstream: + # A genuine `verified`, not an assumed one: maketable1.zip was downloaded + # from the archive page on the date below and the .dta inside hashes to + # exactly the sha256 above, at the same 14,466 bytes and the same + # 18 Jan 2010 header timestamp. + status: verified + date: 2026-08-13 + against: https://www.dropbox.com/s/e0hsiyv86vu8e96/maketable1.zip?dl=0 + note: > + Re-fetched 2026-08-13 and byte-identical. Note the two-hop path: the MIT + page is the citable landing page, but the files themselves now live on + Dropbox share links, so `file_url` will rot before `url` does — + re-derive it from the "Table 1: Summary Statistics (Program and Data + Files)" link on the archive page rather than trusting the share id. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13. The float32 columns are pinned +# by Stata storage types, not by pandas, so they read identically under either +# version; `shortnam` is `object` under pandas 2 and `str` under the +# anaconda=2026.07 pin (pandas 3), and is declared as the pin will read it. + +schema: + format: dta + columns: + - {name: shortnam, dtype: str, description: "3-letter country code, 163 distinct"} + - {name: euro1900, dtype: float32, description: "European settlers as % of population in 1900"} + - {name: excolony, dtype: float32, description: "=1 if ever a colony; 0/1 only"} + - {name: avexpr, dtype: float32, description: "average protection against expropriation risk 1985-95, 0-10 scale (descends from PRS Group data — see license.note)"} + - {name: logpgp95, dtype: float32, description: "log PPP GDP per capita in 1995, World Bank"} + - {name: cons1, dtype: float32, description: "constraint on the executive in the first year of independence, 1-7"} + - {name: cons90, dtype: float32, description: "constraint on the executive in 1990, 1-7 (the file's own variable label is empty; the name comes from maketable1.do)"} + - {name: democ00a, dtype: float32, description: "democracy score in 1900, 0-10"} + - {name: cons00a, dtype: float32, description: "constraint on the executive in 1900, 1-7"} + - {name: extmort4, dtype: float32, description: "corrected European settler mortality, deaths per 1,000"} + - {name: logem4, dtype: float32, description: "log of extmort4 — the paper's instrument"} + - {name: loghjypl, dtype: float32, description: "log GDP per worker 1988, Hall and Jones"} + - {name: baseco, dtype: float32, description: "=1 for the 64-country base sample; missing otherwise, never 0"} + row_count_floor: 163 # frozen 2001 replication deposit — exact row + # count by design, one row per country + # date_range omitted: a country cross-section, not a time series. Its + # variables are dated (1900, 1985-95, 1995) but the file has no time axis. + # + # Nulls are extensive and correct. This is the paper's full 163-country world + # frame, and the historical variables exist only for the ex-colonies; the + # lecture's own analysis is built on dropna(). A blanket no-nulls rule would + # reject this file wrongly. + known_nulls: + euro1900: 9 + excolony: 1 + avexpr: 42 + logpgp95: 15 + cons1: 75 + cons90: 75 + democ00a: 76 + cons00a: 72 + extmort4: 76 + logem4: 76 + loghjypl: 40 + baseco: 99 + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at `landed`, populated by the flip PR once the repoints merge. +# Two reference-holders will be recorded: lecture-python.myst (ols.md:94 and +# ols.md:679) and lecture-python.zh-cn (ols.md:109 and ols.md:594), the latter +# outside the audit's SCAN_REPOS and so recorded here or nowhere. The generated +# lecture-python.notebooks mirror self-heals after a publish tag and is +# deliberately not listed, per the wave A4 precedent. lecture-stats reads the +# copy in the ARCHIVED QuantEcon/lecture-python, not this one, so it is not a +# consumer of these bytes. +consumers: [] + +builder: null +builder_status: not-applicable # verbatim — no builder diff --git a/lectures/maketable2.dta b/lectures/maketable2.dta new file mode 100644 index 0000000..057d9b3 Binary files /dev/null and b/lectures/maketable2.dta differ diff --git a/lectures/maketable2.dta.yml b/lectures/maketable2.dta.yml new file mode 100644 index 0000000..9a84565 --- /dev/null +++ b/lectures/maketable2.dta.yml @@ -0,0 +1,164 @@ +# Manifest for maketable2.dta — migrated out of QuantEcon/lecture-python.myst, +# where it sat at lectures/_static/lecture_specific/ols/ and was read by URL +# (PLAN Phase 8, wave B1'). +# +# One of three AJR (2001) replication tables landing together. They share a +# source, a licence position and an integrity story, and differ only in +# columns — so the source/license blocks below are deliberately identical in +# all three manifests. lecture-dp and lecture-python.zh-cn each hold a +# byte-identical orphan copy that no lecture reads (Track X, not this wave). + +filename: maketable2.dta +title: Acemoglu-Johnson-Robinson (2001) colonial origins — table 2 replication data +description: > + The 163-country frame behind table 2's extended OLS specifications: average + protection against expropriation risk 1985-95 and log GDP per capita in 1995, + plus the continent dummies (Africa, Asia, other), absolute latitude and log + output per worker. `ols` estimates the table 2 models from it — the base + sample, the base sample with continent controls, and the world sample — + at ols.md:326. + +# Verbatim: the authors' own replication file as distributed inside +# maketable2.zip on Daron Acemoglu's data archive. Not merely asserted — the +# zip was re-downloaded 2026-08-13 and the enclosed .dta hashes to the sha256 +# below exactly, so no QuantEcon processing can have been applied. +class: verbatim + +source: + name: Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 2 (program and data files) + url: https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive + file_url: https://www.dropbox.com/s/1s1yb7a3hkjwzjh/maketable2.zip?dl=0 + doi: 10.1257/aer.91.5.1369 + version: Stata dta format 113, authored 28 Dec 2009 08:12 (from the file's own header) + citation: > + Acemoglu, Daron, Simon Johnson, and James A. Robinson. 2001. "The Colonial + Origins of Comparative Development: An Empirical Investigation." American + Economic Review 91 (5): 1369-1401. + note: > + THE DOCUMENTED UPSTREAM ERRATUM LIVES HERE. `ols.md:214-216` warns that + "an observation was mistakenly dropped from the results in the original + paper (see the note located in `maketable2.do`)", which is why the + lecture's world-sample coefficients differ slightly from the published + table. That note is in maketable2.do and nowhere else in the deposit; its + text records that 111 countries have the data to run the table 2 + regressions though the paper reports 110, and that which observation was + dropped is unknown. This is a known, deliberate divergence between the + data and the printed paper — do not "fix" it, and do not let a later + refresh from the archive silently undo the lecture's explanation of it. + The same caveat is echoed in maketable1.dta.yml, because the regression + `ols.md` runs on that file is table 2's column 1. It does not apply to + table 4. + +license: + name: null + url: https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). There is no licence to name: the archive page states + # only that the zipped data and program files are there to download, the + # deposit's own readme.TXT states terms nowhere, and the page carries no + # notice beyond the department footer. Nor is there an AEA deposit to fall + # back on — the AEA article page for 10.1257/aer.91.5.1369 links no + # replication package, and the current AEA data and code availability policy + # claims no retroactive reach. A 2001 paper predates that regime entirely. + redistribution: permitted + verified: 2026-08-13 + note: > + An inherited exposure: served publicly by QuantEcon since the + lecture-python.rst era, and cached here with attribution rather than left + silent. Nothing in the deposit imposes non-commercial or no-redistribution + terms — the class that earns `restricted` here — and the authors publish it + for open download. One chain deserves naming anyway, and it is shared by + all three tables: `avexpr` descends from the Political Risk Services + Group's commercial country-risk data, which the lecture links directly. + What this file carries is AJR's own 1985-95 average, not a PRS series, and + AJR have redistributed it openly for two decades; recorded `permitted` on + that basis, with the PRS chain registered for review on + QuantEcon/data-lectures#35. + +retrieved: null # inherited from lecture-python.myst with no + # recorded retrieval date; not reconstructed + # from git history (AGENTS.md). It costs + # nothing here — upstream is frozen and was + # re-fetched byte-identical below, so the + # vintage is pinned by the hash, not by a date. + # See the AGENTS.md note on this pairing. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to the copy every consumer reads today — sha256 7f0dc30d… +# against lecture-python.myst @ 06c784f, +# lectures/_static/lecture_specific/ols/. The lecture-python.zh-cn and +# lecture-dp copies of the same path carry the identical git blob. Verified +# 2026-08-13; a repoint to this file therefore cannot change a figure. + +integrity: + sha256: 7f0dc30d993da4aafbaf25f7705ad6cd5929ccc01ccc5e2f288d53061a60a171 + upstream: + # A genuine `verified`, not an assumed one: maketable2.zip was downloaded + # from the archive page on the date below and the .dta inside hashes to + # exactly the sha256 above. + status: verified + date: 2026-08-13 + against: https://www.dropbox.com/s/1s1yb7a3hkjwzjh/maketable2.zip?dl=0 + note: > + Re-fetched 2026-08-13 and byte-identical. MIND THE SHARE ID: the archive + page lists a `maketable2.zip` under three different papers — Colonial + Origins, Reversal of Fortune, and Income/growth/institutions — at three + different Dropbox ids. The id above is the Colonial Origins one; fetching + a sibling id returns a different paper's file of the same name, which + would compare as a spurious mismatch. Re-derive it from the "Table 2" + link in the Colonial Origins section rather than trusting the share id, + which will rot before the landing page does. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13. The float32 columns are pinned +# by Stata storage types, not by pandas, so they read identically under either +# version; `shortnam` is `object` under pandas 2 and `str` under the +# anaconda=2026.07 pin (pandas 3), and is declared as the pin will read it. + +schema: + format: dta + columns: + - {name: shortnam, dtype: str, description: "3-letter country code, 163 distinct"} + - {name: africa, dtype: float32, description: "Africa dummy"} + - {name: lat_abst, dtype: float32, description: "absolute latitude, scaled to [0,1]"} + - {name: avexpr, dtype: float32, description: "average protection against expropriation risk 1985-95, 0-10 scale (descends from PRS Group data — see license.note)"} + - {name: logpgp95, dtype: float32, description: "log PPP GDP per capita in 1995, World Bank"} + - {name: other, dtype: float32, description: "other-continent dummy"} + - {name: asia, dtype: float32, description: "Asia dummy"} + - {name: loghjypl, dtype: float32, description: "log GDP per worker 1988, Hall and Jones"} + - {name: baseco, dtype: float32, description: "=1 for the 64-country base sample; missing otherwise, never 0"} + row_count_floor: 163 # frozen 2001 replication deposit — exact row + # count by design, one row per country + # date_range omitted: a country cross-section, not a time series. + # + # Nulls are extensive and correct — the lecture's own analysis is built on + # dropna(), and it is the 111-vs-110 world sample that the erratum in + # source.note is about. A blanket no-nulls rule would reject this file. + known_nulls: + lat_abst: 1 + avexpr: 42 + logpgp95: 15 + loghjypl: 40 + baseco: 99 + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at `landed`, populated by the flip PR once the repoints merge. +# Two reference-holders will be recorded: lecture-python.myst (ols.md:326) and +# lecture-python.zh-cn (ols.md:298), the latter outside the audit's SCAN_REPOS +# and so recorded here or nowhere. The generated lecture-python.notebooks +# mirror self-heals after a publish tag and is deliberately not listed, per the +# wave A4 precedent. lecture-stats, lecture-econometrics-machine-learning and +# lecture-mapping all read the copy in the ARCHIVED QuantEcon/lecture-python, +# not this one, so none of them is a consumer of these bytes. +consumers: [] + +builder: null +builder_status: not-applicable # verbatim — no builder diff --git a/lectures/maketable4.dta b/lectures/maketable4.dta new file mode 100644 index 0000000..8892a2c Binary files /dev/null and b/lectures/maketable4.dta differ diff --git a/lectures/maketable4.dta.yml b/lectures/maketable4.dta.yml new file mode 100644 index 0000000..4382c73 --- /dev/null +++ b/lectures/maketable4.dta.yml @@ -0,0 +1,161 @@ +# Manifest for maketable4.dta — migrated out of QuantEcon/lecture-python.myst, +# where it sat at lectures/_static/lecture_specific/ols/ and was read by URL +# (PLAN Phase 8, wave B1'). +# +# One of three AJR (2001) replication tables landing together. They share a +# source, a licence position and an integrity story, and differ only in +# columns — so the source/license blocks below are deliberately identical in +# all three manifests. lecture-dp and lecture-python.zh-cn each hold a +# byte-identical orphan copy that no lecture reads (Track X, not this wave). + +filename: maketable4.dta +title: Acemoglu-Johnson-Robinson (2001) colonial origins — table 4 replication data +description: > + The 163-country frame behind table 4's instrumental-variables estimates: log + settler mortality (`logem4`, the AJR instrument), average protection against + expropriation risk 1985-95, log GDP per capita in 1995, the continent + dummies, absolute latitude, a rich-country indicator and log output per + worker. `ols` filters to `baseco == 1` — the 64 countries with complete data + — and walks through the two-stage least squares estimate by hand before + reproducing it with `IV2SLS`, at ols.md:478 and again at ols.md:606. + +# Verbatim: the authors' own replication file as distributed inside +# maketable4.zip on Daron Acemoglu's data archive. Not merely asserted — the +# zip was re-downloaded 2026-08-13 and the enclosed .dta hashes to the sha256 +# below exactly, so no QuantEcon processing can have been applied. +class: verbatim + +source: + name: Daron Acemoglu data archive — "The Colonial Origins of Comparative Development", table 4 (program and data files) + url: https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive + file_url: https://www.dropbox.com/s/mntre4yhzothfgq/maketable4.zip?dl=0 + doi: 10.1257/aer.91.5.1369 + version: Stata dta format 113, authored 28 Dec 2009 09:35 (from the file's own header) + citation: > + Acemoglu, Daron, Simon Johnson, and James A. Robinson. 2001. "The Colonial + Origins of Comparative Development: An Empirical Investigation." American + Economic Review 91 (5): 1369-1401. + note: > + The table 2 erratum recorded in maketable2.dta.yml does NOT apply to this + file — it concerns the table 2 OLS world sample, and `ols` estimates table + 4 on the 64-country base sample instead. `logem4` is the paper's excluded + instrument, and the lecture's first-stage regression of `avexpr` on it is + what motivates the whole IV discussion. + +license: + name: null + url: https://economics.mit.edu/people/faculty/daron-acemoglu/data-archive + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). There is no licence to name: the archive page states + # only that the zipped data and program files are there to download, the + # deposit's own readme.TXT states terms nowhere, and the page carries no + # notice beyond the department footer. Nor is there an AEA deposit to fall + # back on — the AEA article page for 10.1257/aer.91.5.1369 links no + # replication package, and the current AEA data and code availability policy + # claims no retroactive reach. A 2001 paper predates that regime entirely. + redistribution: permitted + verified: 2026-08-13 + note: > + An inherited exposure: served publicly by QuantEcon since the + lecture-python.rst era, and cached here with attribution rather than left + silent. Nothing in the deposit imposes non-commercial or no-redistribution + terms — the class that earns `restricted` here — and the authors publish it + for open download. One chain deserves naming anyway, and it is shared by + all three tables: `avexpr` descends from the Political Risk Services + Group's commercial country-risk data, which the lecture links directly. + What this file carries is AJR's own 1985-95 average, not a PRS series, and + AJR have redistributed it openly for two decades; recorded `permitted` on + that basis, with the PRS chain registered for review on + QuantEcon/data-lectures#35. + +retrieved: null # inherited from lecture-python.myst with no + # recorded retrieval date; not reconstructed + # from git history (AGENTS.md). It costs + # nothing here — upstream is frozen and was + # re-fetched byte-identical below, so the + # vintage is pinned by the hash, not by a date. + # See the AGENTS.md note on this pairing. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to the copy every consumer reads today — sha256 2b34cce2… +# against lecture-python.myst @ 06c784f, +# lectures/_static/lecture_specific/ols/. The lecture-python.zh-cn and +# lecture-dp copies of the same path carry the identical git blob. Verified +# 2026-08-13; a repoint to this file therefore cannot change a figure. + +integrity: + sha256: 2b34cce21b58a3a424a804e97a134106ab7010ef936bb68b3a244d3e271af404 + upstream: + # A genuine `verified`, not an assumed one: maketable4.zip was downloaded + # from the archive page on the date below and the .dta inside hashes to + # exactly the sha256 above. + status: verified + date: 2026-08-13 + against: https://www.dropbox.com/s/mntre4yhzothfgq/maketable4.zip?dl=0 + note: > + Re-fetched 2026-08-13 and byte-identical. MIND THE SHARE ID: the archive + page lists a `maketable4.zip` under three different papers — Colonial + Origins, Reversal of Fortune, and Volatility/government consumption — at + three different Dropbox ids. The id above is the Colonial Origins one; + fetching a sibling id returns a different paper's file of the same name, + which would compare as a spurious mismatch. Re-derive it from the + "Table 4" link in the Colonial Origins section rather than trusting the + share id, which will rot before the landing page does. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13. The float32 columns are pinned +# by Stata storage types, not by pandas, so they read identically under either +# version; `shortnam` is `object` under pandas 2 and `str` under the +# anaconda=2026.07 pin (pandas 3), and is declared as the pin will read it. + +schema: + format: dta + columns: + - {name: shortnam, dtype: str, description: "3-letter country code, 163 distinct"} + - {name: africa, dtype: float32, description: "Africa dummy"} + - {name: lat_abst, dtype: float32, description: "absolute latitude, scaled to [0,1]"} + - {name: rich4, dtype: float32, description: "rich-country indicator"} + - {name: avexpr, dtype: float32, description: "average protection against expropriation risk 1985-95, 0-10 scale (descends from PRS Group data — see license.note)"} + - {name: logpgp95, dtype: float32, description: "log PPP GDP per capita in 1995, World Bank"} + - {name: logem4, dtype: float32, description: "log European settler mortality — the paper's excluded instrument"} + - {name: asia, dtype: float32, description: "Asia dummy"} + - {name: loghjypl, dtype: float32, description: "log GDP per worker 1988, Hall and Jones"} + - {name: baseco, dtype: float32, description: "=1 for the 64-country base sample the lecture filters on; missing otherwise, never 0"} + row_count_floor: 163 # frozen 2001 replication deposit — exact row + # count by design, one row per country + # date_range omitted: a country cross-section, not a time series. + # + # Nulls are extensive and correct. `logem4` is missing for 76 of 163 rows + # because settler mortality exists only for the ex-colonies — that sparsity + # is precisely why the lecture filters to `baseco == 1`. A blanket no-nulls + # rule would reject this file wrongly. + known_nulls: + lat_abst: 1 + avexpr: 42 + logpgp95: 15 + logem4: 76 + loghjypl: 40 + baseco: 99 + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at `landed`, populated by the flip PR once the repoints merge. +# Two reference-holders will be recorded: lecture-python.myst (ols.md:478 and +# ols.md:606) and lecture-python.zh-cn (ols.md:421 and ols.md:526), the latter +# outside the audit's SCAN_REPOS and so recorded here or nowhere. The generated +# lecture-python.notebooks mirror self-heals after a publish tag and is +# deliberately not listed, per the wave A4 precedent. lecture-stats, +# lecture-econometrics-machine-learning and lecture-mapping all read the copy +# in the ARCHIVED QuantEcon/lecture-python, not this one, so none of them is a +# consumer of these bytes. +consumers: [] + +builder: null +builder_status: not-applicable # verbatim — no builder diff --git a/migration.yml b/migration.yml index 2177abc..b0ed8d2 100644 --- a/migration.yml +++ b/migration.yml @@ -540,6 +540,84 @@ datasets: date: 2026-08-12 cutover: null + # -------------------------------------------------------------------------- + # Wave B1' — the lecture-python.myst block, part one (Track B). + # + # Five files, three lectures, all three URL forms. Track B was planned as + # three waves and re-cut to two on 2026-08-13: a `publish*` tag rebuilds the + # whole book, so wave size does not change publish cost — only the NUMBER of + # publishes does, and each wave costs two (python.myst and python.zh-cn). + # The hansen_singleton pair follows as B2', separated by its builders, its + # READMEs and an open translation-sync PR, not by novelty. + # + # All five carry `prior_pattern: own-repo` — the lectures fetch their own + # repo's committed blobs. Consumers are recorded at the flip, and will + # include lecture-python.zh-cn, which is OUTSIDE the audit's SCAN_REPOS and + # so is recorded in the manifests or nowhere. + # -------------------------------------------------------------------------- + + maketable1.dta: + pilot: B1 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#79 + date: 2026-08-13 + repoints: [] + cutover: null + + maketable2.dta: + pilot: B1 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#79 + date: 2026-08-13 + repoints: [] + cutover: null + + maketable4.dta: + pilot: B1 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#79 + date: 2026-08-13 + repoints: [] + cutover: null + + fp.dta: + pilot: B1 + status: landed + # The one Track B file with a published-SITE prose link + # (python.quantecon.org/_static/lecture_specific/mle/fp.dta) as well as a + # code read. That link is also carried by lecture-stats, which runs a daily + # linkcheck — so deletion is gated on the lecture-stats repoint, not only + # on the two publishes. + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#79 + date: 2026-08-13 + repoints: [] + cutover: null + + NEWQDATA.csv: + pilot: B1 + status: landed + # Reclassified at migration from the `verbatim` guess in + # audit_annotations.yml: it is a container conversion of the authors' + # NEWQDATA.MAT, and the conversion is recovered and committed at + # builders/NEWQDATA.py. The only Track B file read via + # raw.githubusercontent.com, and its URL is built by string concatenation + # across three source lines — the repoint must collapse it, not patch the + # stem line. + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#79 + date: 2026-08-13 + repoints: [] + cutover: null + # Planned waves that have not landed anything here yet. `datasets` names the # files as the audit sees them today, so the dashboard can join the two views. # `title` is the reader-facing milestone name (the dashboard is read by people diff --git a/requirements.txt b/requirements.txt index 0eb37a5..481fb1f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,4 @@ wbgapi==1.0.12 PyYAML==6.0.3 # scripts/build_catalog.py — parse the sidecar manifests +scipy==1.16.3 # builders/NEWQDATA.py — loadmat, the only way to read a + # MATLAB 5.0 .MAT; pandas cannot diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index 4e02934..8bbdb7a 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -18,16 +18,6 @@ # toy invented in-lecture teaching table datasets: - NEWQDATA.csv: - description: Cogley–Sargent quarterly US data (inflation, unemployment, T-bill), 1948Q2–2000Q4 - provenance: verbatim - flags: [new-since-2026-07-15] - note: > - added by lecture-python.myst#1001; the lecture cites it as - CogleySargent2005's dataset and no builder is committed beside it. - A migration candidate — it is read from lecture-python.myst main by raw - URL, and the lecture documents every transformation and rebuilds the - post-2000Q4 continuation from FRED acs_data_summary.csv: description: ACS occupation summary provenance: constructed-lost @@ -61,9 +51,6 @@ datasets: fig_3.xlsx: description: French Revolution fiscal data provenance: author-assembled - fp.dta: - description: Treisman (2016) Russia billionaires replication data - provenance: verbatim fred_data.csv: description: FRED snapshot — GS1, GS5, GS10, DFII5, DFII10, USREC provenance: constructed-lost @@ -96,15 +83,6 @@ datasets: description: Long-run price levels (Sargent–Velde) provenance: author-assembled note: hand-transcribed by authors - maketable1.dta: - description: Acemoglu–Johnson–Robinson (2001) replication, table 1 - provenance: verbatim - maketable2.dta: - description: AJR (2001) replication, table 2 - provenance: verbatim - maketable4.dta: - description: AJR (2001) replication, table 4 - provenance: verbatim mpd2020.xlsx: description: Maddison Project Database 2020 (GDP per capita, long run) provenance: verbatim diff --git a/sources/NEWQDATA.MAT b/sources/NEWQDATA.MAT new file mode 100644 index 0000000..3d406ee --- /dev/null +++ b/sources/NEWQDATA.MAT @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ab760926f37b41e7478c5227454f3d5c142acb4ad27428f0881ad5140dcb8162 +size 7104 diff --git a/sources/README.md b/sources/README.md index 59c9a0a..2a2ba0c 100644 --- a/sources/README.md +++ b/sources/README.md @@ -11,10 +11,11 @@ The defining property is **un-refetchability**, not size. The normal case for a builder is to fetch from its third-party upstream at run time, and that is what most of them do: `jse.amstat.org`, `earthquake.usgs.gov`, -`wwwn.cdc.gov`, `stat.go.jp`, `openfootball`. **None of the six `committed` -builders in this repo has a committed input.** So `sources/` is not "where -builder inputs live" as a general rule — it is the exception layer for an input -that cannot be obtained again. +`wwwn.cdc.gov`, `stat.go.jp`, `openfootball`. **Six of the seven `committed` +builders in this repo fetch at run time and have no committed input**; the +exception is `NEWQDATA.py`, whose upstream is not published anywhere. So +`sources/` is not "where builder inputs live" as a general rule — it is the +exception layer for an input that cannot be obtained again. It is also emphatically **not "the big-file directory"**, even though `.gitattributes` LFS-tracks everything under it. A 300 MB file that can be @@ -117,3 +118,55 @@ frozen builder's corrections belong: **the input is now committed at resolve. Do not treat that as a reason to leave the dependency live in any builder that does run — and do not delete branches or rewrite history on that repo after archiving, which is what actually breaks an external reader. + +--- + +## `NEWQDATA.MAT` + +| | | +| --- | --- | +| **Origin** | Cogley and Sargent's own MATLAB replication directory for "Drifts and Volatilities". Located in 2026 only as a third-party GitHub mirror (`szokeb87/cs2005_pymc`, `matlab_files_from_cogley/`), whose README states the MATLAB files "were written entirely by Tim Cogley and Thomas Sargent" | +| **Upstream** | Cogley, Timothy, and Thomas J. Sargent (2005), *Drifts and Volatilities: Monetary Policies and Outcomes in the Post WWII U.S.*, Review of Economic Dynamics 8(2), 262-302 | +| **Upstream identifier** | DOI [10.1016/j.red.2004.10.009](https://doi.org/10.1016/j.red.2004.10.009) — the **article**. There is no dataset DOI; Crossref registers no dataset component | +| **Retrieved** | `null` — no retrieval date was recorded upstream, and the mirror's commit dates record when a third party acquired the file, not when it came from the authors. Do not promote one to the other | +| **Licence** | `null` — no licence statement exists at the journal, the authors' pages, or the mirror. Registered on [#35](https://github.com/QuantEcon/data-lectures/issues/35) | +| **`sha256`** | `ab760926f37b41e7478c5227454f3d5c142acb4ad27428f0881ad5140dcb8162` | +| **Size** | 7,104 B | +| **Consumed by** | `builders/NEWQDATA.py` (`builder_status: committed`) | +| **Produces** | `lectures/NEWQDATA.csv` | + +### Why this file is here rather than fetched at run time + +This is the repo's **first committed input to a builder that actually runs**, +so the exception deserves its reasoning in full. + +The file is un-refetchable from any authoritative source. Searched to +exhaustion on 2026-08-13: + +- **tomsargent.com** and Cogley's NYU pages — live and via the Wayback Machine + — carry no replication files for this paper; +- the **Review of Economic Dynamics / SED** site publishes no data archive, and + had no data-availability policy in 2005; +- **RePEc**'s record for the article lists no dataset; +- **ScienceDirect** returns 403 to an anonymous fetch; +- **Crossref** registers no dataset component, only Elsevier text-mining + licences for the article text. + +The single copy located anywhere was the third-party mirror above. A builder +that fetches from a stranger's repository at run time is precisely the +fragility this directory exists to remove — the same failure shape as a network +read from a retired QuantEcon repo, with less recourse. So the input is +committed here, and `builders/NEWQDATA.py` reads it from disk. + +The conversion is value-preserving — four arrays to four columns, no filtering, +no rescaling, no reordering — and the builder reproduces +`lectures/NEWQDATA.csv` **byte for byte**, all 13,738 of them. That is what +earns the manifest's `integrity.upstream.status: verified` under AGENTS.md's +definition for a `constructed` dataset ("re-run the builder and compare"), +rather than by analogy to a verbatim re-fetch. + +### Size + +7,104 B — four orders of magnitude below any limit that matters. It is here on +the un-refetchability test alone, which is the test, and a useful counterweight +to reading this directory as "the big-file directory".