From 07dd946dd0380c505bb9be594e8e67139d696089 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 7 Aug 2026 11:11:57 +1000 Subject: [PATCH 1/2] =?UTF-8?q?PLAN:=20add=20repoint=20rule=206=20?= =?UTF-8?q?=E2=80=94=20the=20media=20host=20is=20LFS-only,=20so=20a=20fold?= =?UTF-8?q?=20changes=20host?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `media.githubusercontent.com` serves only LFS objects and returns 404 for a plain-git file, while `raw.githubusercontent.com` serves it. Both send access-control-allow-origin: *, so this is host routing rather than CORS — a different failure from rule 5, and it affects CPython consumers too. This is latent in exactly one remaining piece of work, and it would break all of it. high_dim_data tracks *.csv and *.dta under a blanket LFS rule, so all 14 consuming reads go through the media host today. The storage decision lands those six datasets here as plain git, after which the media host 404s for them — so the fold must change host as well as org and repo. A mechanical org/repo swap breaks every read. Rule 6 records the measurement, enumerates the 14 reads with file and line, and gives the acceptance grep. Phase 3 gets a matching checkbox, since that is where the work happens. Nothing in CI covers this: the strict audit's URL-form check catches only the github.com/*/raw/ form, and only in lecture-wasm (#48). Two of the 14 reads are builder notebooks the audit does not scan as data reads at all. Found while preparing the french_rev set. Context in QuantEcon/workspace-lectures#23. --- PLAN.md | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/PLAN.md b/PLAN.md index e43dd2e..dff48d6 100644 --- a/PLAN.md +++ b/PLAN.md @@ -42,7 +42,7 @@ This repository is being shaped into the **single canonical repository for data ## Repoint rules -Five rules learned the hard way, three of them the hard way twice. Rules 1-3 are about *ordering* and none is enforced by CI — the strict audit catches rule 2 only after the fact, and cannot see rule 3 at all. Rule 4 is about *scope*, and rule 5 about *URL form*. +Six rules learned the hard way, three of them the hard way twice. Rules 1-3 are about *ordering* and none is enforced by CI — the strict audit catches rule 2 only after the fact, and cannot see rule 3 at all. Rule 4 is about *scope*; rules 5 and 6 about *URL form and host*, and neither is fully enforced either. ### 1. Repoint a sibling reader before deleting the file it reads @@ -117,6 +117,38 @@ Learned from the independent validation ([#45](https://github.com/QuantEcon/data Phase 4 inherits the requirement: `data.quantecon.org` must serve `access-control-allow-origin: *` before `lecture-wasm` can cut over to it — recorded as an acceptance criterion on [#37](https://github.com/QuantEcon/data-lectures/issues/37). +### 6. `media.githubusercontent.com` is LFS-only — a fold changes the *host*, not just the org + +`media.githubusercontent.com/media////` is the **LFS media endpoint**. It resolves only for paths that are LFS-tracked *in that repo*, and returns **404** for a plain-git file. Measured 2026-08-07: + +| URL | Status | +| --- | --- | +| `raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/mpd2020.xlsx` | **200** | +| `media.githubusercontent.com/media/QuantEcon/data-lectures/main/lectures/mpd2020.xlsx` | **404** | + +Both hosts send `access-control-allow-origin: *`, so this is **host routing, not CORS** — a distinct failure from rule 5, and it bites CPython consumers too, not only the browser. + +This matters for exactly one piece of remaining work, and it matters a lot. `high_dim_data` tracks `*.csv` **and** `*.dta` under a blanket LFS rule, so **every** consuming lecture reads its datasets through the media host today. The storage decision lands those six datasets here as **plain git** (both SCF minis fit under the 100 MiB blob limit). After the fold the media host will 404 for them, so **every consuming read must change host as well as org and repo** — a mechanical org/repo swap that preserves the host breaks all of them. + +**14 reads are affected**, all in Track A — 12 on the media host, 2 on the `github.com/*/raw/` redirect form: + +| Repo | File | Lines | Current host | +| --- | --- | --- | --- | +| `lecture-python-intro` | `lectures/heavy_tails.md` | 827, 854, 855, 879 | media | +| `lecture-python-intro` | `lectures/_static/lecture_specific/inequality/data.ipynb` | 37 | media | +| `lecture-python-intro` | `lectures/mle.md` | 93 | `github.com/*/raw/` | +| `lecture-python-intro` | `lectures/inequality.md` | 249 | `github.com/*/raw/` | +| `lecture-wasm` | `lectures/heavy_tails.md` | 827, 854, 855, 879 | media | +| `lecture-wasm` | `lectures/mle.md` | 95 | media | +| `lecture-wasm` | `lectures/inequality.md` | 250 | media | +| `lecture-wasm` | `lectures/_static/lecture_specific/inequality/data.ipynb` | 37 | media | + +The plain-git decision does not *dissolve* the raw-vs-media trap for the repoint — it **inverts** it. The trap stops being "consumers must know to use the media host" and becomes "consumers already on the media host must be moved off it, in the same PR as the fold." + +**Acceptance check for the fold:** `grep -rn 'media.githubusercontent.com/media/QuantEcon/data-lectures' repos/` must return nothing. This is **not** covered by CI — the strict audit's URL-form check catches only the `github.com/*/raw/` form, and only in `lecture-wasm` ([#48](https://github.com/QuantEcon/data-lectures/pull/48)). A media-host reference to a data-lectures path is invisible to every build and fails at read time, in the reader's notebook or browser. + +Note the two `_static/…/inequality/data.ipynb` reads are builder notebooks, which the audit does not scan as data reads at all — they must be changed and checked by hand. + ## Migration tracks The remaining work decomposes by **consuming series** rather than by hosting pattern, because — apart from the `intro`/`wasm` pairing — each series now owns its own data. This is the execution view; the phases below remain the machinery each track passes through. @@ -179,6 +211,7 @@ Only one file genuinely forces LFS, and it is not a dataset: - [ ] Per-path LFS via `.gitattributes`, scoped to `sources/` only — never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta` (data#1) - [ ] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints) - [ ] **Repoint `generating_mini.md`'s input URL.** The SCF builder currently reads its source over the network from the repo being retired — `pd.read_stata('https://github.com/QuantEcon/high_dim_data/blob/main/SCF_plus/SCF_plus.dta?raw=true')`. Archiving `high_dim_data` while that line stands re-introduces exactly the legacy-repo dependency this project drove to zero. Point it at `sources/` before archiving +- [ ] **Move all 14 consuming reads off `media.githubusercontent.com`** in the same set as the fold — see repoint rule 6. The six datasets are LFS-tracked in `high_dim_data` and land here as plain git, so the media host will 404 for them; changing only org and repo breaks every read. Acceptance: `grep -rn 'media.githubusercontent.com/media/QuantEcon/data-lectures' repos/` returns nothing. Not covered by CI, and two of the 14 are builder notebooks the audit never scans - [ ] Set the Pages job's checkout to `lfs: false` once the above holds — nothing under `lectures/` is an LFS object, so the 99 MiB `.dta` never needs downloading on a dashboard build (it runs on every push to `main` plus weekly) **Sequencing constraint** (still applies to anything that *does* enter LFS): enabling LFS breaks every `raw.githubusercontent.com` URL for the paths it covers — those URLs return pointer text, not data, so consumers fail with a confusing parse error rather than a 404. Do not LFS-track an existing file until its consumers use a form that survives it. Keeping the published tree plain-git means no consumer-facing path is ever affected. From 903fd5f7bc91bb492fe9402412cdac1116fcbf76 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 7 Aug 2026 11:43:03 +1000 Subject: [PATCH 2/2] PLAN: tighten rule 6's intro line and de-duplicate the Phase 3 checkbox MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses Copilot review on #50. The "Repoint rules" intro said rules 5 and 6 are "not fully enforced either" — a doubled construction, and vague where the rest of that paragraph is specific about what CI does and does not catch. Now says what CI actually covers: the github.com/*/raw/ form in lecture-wasm, and nothing else. The Phase 3 checkbox repeated the acceptance grep and both caveats already in rule 6. A grep string copied into two places drifts, and this one is load-bearing — if the copies disagree later, the checklist is what someone will act on. The box keeps the what and the why so it still means something on its own, and defers the enumerated reads, the acceptance check and the CI caveat to rule 6, which is now the single canonical place. The grep string appears once in the file. Docs only; no change to the rule's substance. --- PLAN.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/PLAN.md b/PLAN.md index dff48d6..d9d8915 100644 --- a/PLAN.md +++ b/PLAN.md @@ -42,7 +42,7 @@ This repository is being shaped into the **single canonical repository for data ## Repoint rules -Six rules learned the hard way, three of them the hard way twice. Rules 1-3 are about *ordering* and none is enforced by CI — the strict audit catches rule 2 only after the fact, and cannot see rule 3 at all. Rule 4 is about *scope*; rules 5 and 6 about *URL form and host*, and neither is fully enforced either. +Six rules learned the hard way, three of them the hard way twice. Rules 1-3 are about *ordering* and none is enforced by CI — the strict audit catches rule 2 only after the fact, and cannot see rule 3 at all. Rule 4 is about *scope*; rules 5 and 6 about *URL form and host*, and CI covers only a corner of them — the strict audit checks the `github.com/*/raw/` form in `lecture-wasm` and nothing else. ### 1. Repoint a sibling reader before deleting the file it reads @@ -211,7 +211,7 @@ Only one file genuinely forces LFS, and it is not a dataset: - [ ] Per-path LFS via `.gitattributes`, scoped to `sources/` only — never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta` (data#1) - [ ] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints) - [ ] **Repoint `generating_mini.md`'s input URL.** The SCF builder currently reads its source over the network from the repo being retired — `pd.read_stata('https://github.com/QuantEcon/high_dim_data/blob/main/SCF_plus/SCF_plus.dta?raw=true')`. Archiving `high_dim_data` while that line stands re-introduces exactly the legacy-repo dependency this project drove to zero. Point it at `sources/` before archiving -- [ ] **Move all 14 consuming reads off `media.githubusercontent.com`** in the same set as the fold — see repoint rule 6. The six datasets are LFS-tracked in `high_dim_data` and land here as plain git, so the media host will 404 for them; changing only org and repo breaks every read. Acceptance: `grep -rn 'media.githubusercontent.com/media/QuantEcon/data-lectures' repos/` returns nothing. Not covered by CI, and two of the 14 are builder notebooks the audit never scans +- [ ] **Move all 14 consuming reads off `media.githubusercontent.com`** in the same set as the fold — they are LFS-tracked in `high_dim_data` and land here as plain git, so the media host 404s for them and changing only org and repo breaks every read. See **repoint rule 6** for the enumerated reads, the acceptance check, and why CI does not cover it - [ ] Set the Pages job's checkout to `lfs: false` once the above holds — nothing under `lectures/` is an LFS object, so the 99 MiB `.dta` never needs downloading on a dashboard build (it runs on every push to `main` plus weekly) **Sequencing constraint** (still applies to anything that *does* enter LFS): enabling LFS breaks every `raw.githubusercontent.com` URL for the paths it covers — those URLs return pointer text, not data, so consumers fail with a confusing parse error rather than a 404. Do not LFS-track an existing file until its consumers use a form that survives it. Keeping the published tree plain-git means no consumer-facing path is ever affected.