From e1bb6d516e2c9f7697143c7b91836da8acc206ff Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 11:43:16 +1000 Subject: [PATCH 1/2] Add generated dataset catalog (CATALOG.md + build_catalog.py) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Realises PLAN Phase 2's "index/catalog page from the manifests — doubles as the dataset registry". Migrated-only scope: a dataset appears once it has a manifest. - scripts/build_catalog.py reads lectures/*.yml and emits CATALOG.md — one row per dataset with class, source, licence + redistribution, integrity status, builder status, size and consumers (which lectures use it, linked). Generated, not hand-maintained, so it cannot drift from the manifests; a Phase 5 CI check will assert currency via `git diff --exit-code CATALOG.md`. - CATALOG.md — first generation: the 4 migrated datasets (P1 lingcod + P2 pandas_panel trio). Surfaces the honest gaps AGENTS.md promises the catalog shows: unverifiable upstream integrity, the countries.csv restricted licence (tracked in QuantEcon/workspace-lectures#20), and unrecovered builders. - requirements.txt: add PyYAML for the parser. - PLAN.md: tick the Phase 2 catalog item. Co-Authored-By: Claude Opus 4.8 (1M context) --- CATALOG.md | 20 +++++ PLAN.md | 4 +- requirements.txt | 1 + scripts/build_catalog.py | 187 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 210 insertions(+), 2 deletions(-) create mode 100644 CATALOG.md create mode 100644 scripts/build_catalog.py diff --git a/CATALOG.md b/CATALOG.md new file mode 100644 index 0000000..9980b97 --- /dev/null +++ b/CATALOG.md @@ -0,0 +1,20 @@ + + + + +# Dataset catalog — `QuantEcon/data-lectures` + +The migrated-dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest; files not yet migrated are tracked in [PLAN.md](PLAN.md) Phase 9. + +**4 datasets migrated** · 1.8 MB total · 3 permitted / 1 restricted redistribution + +| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)
WorldData.info country reference table | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | +| [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)
Eurostat employment in Europe — by age and sex, 2007–2016 | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | +| [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | +| [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | + +--- + +**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9). diff --git a/PLAN.md b/PLAN.md index ac96525..5ed6f0b 100644 --- a/PLAN.md +++ b/PLAN.md @@ -53,11 +53,11 @@ Ordering note: phases 1–3 and 6 can proceed now; phase 4 needs the DNS questio - [x] Rename `QuantEcon/data` → `data-lectures` (GitHub redirects preserve all existing URLs, so this was non-breaking) - [x] Add repo description and topics (`quantecon`, `datasets`, `economics`, `open-data`, `teaching-materials`) -### Phase 2 — Layout (2026-07-16 — catalog outstanding) +### Phase 2 — Layout (2026-07-16) - [x] Restructure consumer-keyed tree → flat published tree — `lectures/` is the published root; no folder implies series ownership. Done while zero lectures referenced the repo, so no consumer could break; **that window is now closed** - [x] Decide where non-published assets live relative to the published tree — `scripts/` and `manifest-schema.yml` sit at the root, outside `lectures/`, and are never served. Manifests are the exception: they live *inside* `lectures/` as sidecars named `.yml`, so a dataset cannot be moved or removed without its metadata, and CI can assert the pairing with a glob -- [ ] Generate an index/catalog page from the manifests — doubles as the dataset registry +- [x] Generate an index/catalog page from the manifests — doubles as the dataset registry. `scripts/build_catalog.py` emits `CATALOG.md` (migrated-only registry) from `lectures/*.yml`; regenerate on any manifest change, and a Phase 5 CI check will assert it is current (`git diff --exit-code`). Feeds the Pages index at Phase 4 The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.yml`) because a stem-keyed sidecar collides when one dataset ships in two formats — exactly the `fig_3.xlsx` / `fig_3.ods` case this repo already had. Strawman until the pilot tests it; see `manifest-schema.yml`. diff --git a/requirements.txt b/requirements.txt index ee92909..0eb37a5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,2 @@ wbgapi==1.0.12 +PyYAML==6.0.3 # scripts/build_catalog.py — parse the sidecar manifests diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py new file mode 100644 index 0000000..b1e25bd --- /dev/null +++ b/scripts/build_catalog.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Generate CATALOG.md — the migrated-dataset registry — from the sidecar +manifests in lectures/*.yml. + +CATALOG.md is a GENERATED file: the per-dataset manifests are the source of +truth, and this script is the only thing that should write the catalog. Run it +after adding or editing any manifest: + + python scripts/build_catalog.py + +CI asserts the catalog is current with `git diff --exit-code CATALOG.md` after +regenerating, so a stale catalog fails the build (PLAN Phase 5). + +Scope: migrated-only. A dataset appears here once it has a manifest; files in +lectures/ without a manifest are not yet migrated and are tracked in PLAN +Phase 9, not here. +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import yaml + +REPO = Path(__file__).resolve().parent.parent +LECTURES = REPO / "lectures" +OUT = REPO / "CATALOG.md" + +# Interim URL form (AGENTS.md); swaps to data.quantecon.org/lectures/ at Phase 4. +RAW = "https://github.com/QuantEcon/data-lectures/raw/main/lectures" + + +def human_size(n: int) -> str: + size = float(n) + for unit in ("B", "KB", "MB", "GB"): + if size < 1024 or unit == "GB": + return f"{int(size)} {unit}" if unit == "B" else f"{size:.1f} {unit}" + size /= 1024 + return f"{size:.1f} GB" + + +def oneline(text) -> str: + """Collapse a folded/multiline scalar to a single trimmed line.""" + if text is None: + return "" + return " ".join(str(text).split()) + + +def load_manifests(): + manifests = [] + for path in sorted(LECTURES.glob("*.yml")): + with path.open() as f: + m = yaml.safe_load(f) + if not isinstance(m, dict) or "filename" not in m: + print(f"skip (not a manifest): {path.name}", file=sys.stderr) + continue + m["_manifest_path"] = path + data_file = LECTURES / m["filename"] + m["_size"] = data_file.stat().st_size if data_file.exists() else None + manifests.append(m) + return manifests + + +def fmt_consumers(consumers) -> str: + if not consumers: + return "—" + parts = [] + for c in consumers: + repo = c.get("repo", "") + short = repo.split("/")[-1] if repo else "?" + file = c.get("file", "") + stem = file.split("/")[-1] if file else "" + if repo and file: + parts.append(f"[{short} · {stem}](https://github.com/{repo}/blob/main/{file})") + else: + parts.append(f"{short} · {stem}".strip(" ·")) + return "
".join(parts) + + +def fmt_source(source) -> str: + if not isinstance(source, dict): + return oneline(source) + name = oneline(source.get("name", "")) + url = source.get("url") + return f"[{name}]({url})" if url else name + + +def fmt_redist(license) -> str: + if not isinstance(license, dict): + return "?" + r = license.get("redistribution", "?") + return {"permitted": "✅ permitted", "restricted": "⚠️ restricted"}.get(r, str(r)) + + +def fmt_integrity(integrity) -> str: + if not isinstance(integrity, dict): + return "?" + up = integrity.get("upstream") or {} + status = up.get("status", "?") + mark = {"verified": "✅", "spot-checked": "◑", "unverifiable": "⚠️", + "unverified": "…", "failing": "❌"}.get(status, "") + return f"{mark} {status}".strip() + + +def fmt_builder(m) -> str: + if m.get("class") == "verbatim": + return "n/a (verbatim)" + bs = m.get("builder_status", "?") + return {"committed": "✅ committed", "unrecovered": "⚠️ unrecovered", + "not-applicable": "n/a"}.get(bs, str(bs)) + + +def build(manifests) -> str: + total = sum(m["_size"] or 0 for m in manifests) + permitted = sum(1 for m in manifests + if (m.get("license") or {}).get("redistribution") == "permitted") + restricted = len(manifests) - permitted + + lines = [] + lines.append("") + lines.append("") + lines.append("") + lines.append("") + lines.append("# Dataset catalog — `QuantEcon/data-lectures`") + lines.append("") + lines.append( + "The migrated-dataset registry, **auto-generated** from the sidecar " + "manifests (`lectures/*.yml`). Do not edit by hand — run " + "`python scripts/build_catalog.py`. A dataset appears here once it has a " + "manifest; files not yet migrated are tracked in " + "[PLAN.md](PLAN.md) Phase 9." + ) + lines.append("") + lines.append( + f"**{len(manifests)} datasets migrated** · {human_size(total)} total · " + f"{permitted} permitted / {restricted} restricted redistribution" + ) + lines.append("") + lines.append("| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |") + lines.append("| --- | --- | --- | --- | --- | --- | --- | --- | --- |") + for m in manifests: + fn = m["filename"] + title = oneline(m.get("title", "")) + dataset = f"[**{fn}**]({RAW}/{fn})" + if title: + dataset += f"
{title}" + license = m.get("license") or {} + row = [ + dataset, + m.get("class", "?"), + fmt_source(m.get("source")), + oneline(license.get("name", "?")), + fmt_redist(license), + fmt_integrity(m.get("integrity")), + fmt_builder(m), + human_size(m["_size"]) if m["_size"] is not None else "—", + fmt_consumers(m.get("consumers")), + ] + lines.append("| " + " | ".join(row) + " |") + lines.append("") + lines.append("---") + lines.append("") + lines.append( + "**Legend** — *Integrity* is the `integrity.upstream.status` (is this " + "what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · " + "… unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as " + "inherited exposures and tracked for licence review " + "([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). " + "*Builder* ⚠️ unrecovered marks a constructed dataset whose builder was " + "never committed (PLAN Phase 9)." + ) + lines.append("") + return "\n".join(lines) + + +def main() -> int: + manifests = load_manifests() + if not manifests: + print("no manifests found under lectures/*.yml", file=sys.stderr) + return 1 + OUT.write_text(build(manifests)) + print(f"wrote {OUT.relative_to(REPO)} — {len(manifests)} datasets") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From b0dd2f3401fd4d43a0676e4b60add3c67f3f52be Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 11:55:02 +1000 Subject: [PATCH 2/2] catalog: force UTF-8 when reading manifests and writing CATALOG.md Copilot review (data-lectures#19): Path.open()/write_text() used the platform default encoding. Manifests carry non-ASCII (en-dashes, curly quotes) and the catalog emits Unicode marks (checkmark/warning), so on a non-UTF-8 locale (e.g. Windows cp1252) reads/writes could raise UnicodeDecode/EncodeError. This script is headed for CI where the runner locale is not guaranteed. No output change on UTF-8 systems. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/build_catalog.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index b1e25bd..468b5a2 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -49,7 +49,7 @@ def oneline(text) -> str: def load_manifests(): manifests = [] for path in sorted(LECTURES.glob("*.yml")): - with path.open() as f: + with path.open(encoding="utf-8") as f: m = yaml.safe_load(f) if not isinstance(m, dict) or "filename" not in m: print(f"skip (not a manifest): {path.name}", file=sys.stderr) @@ -178,7 +178,7 @@ def main() -> int: if not manifests: print("no manifests found under lectures/*.yml", file=sys.stderr) return 1 - OUT.write_text(build(manifests)) + OUT.write_text(build(manifests), encoding="utf-8") print(f"wrote {OUT.relative_to(REPO)} — {len(manifests)} datasets") return 0