AGENTS.md

AGENTS.md — working in QuantEcon/data-lectures

Guidance for coding agents (and humans) making changes in this repository. Read PLAN.md first — it holds the roadmap and links to the governing issues; this file holds the rules and traps.

What this repo is

The canonical home for data consumed by the QuantEcon lecture series (renamed from QuantEcon/data on 2026-07-16, per meta#336). Its purpose is stability: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a cache, not a content-distribution host. The published tree is flat (lectures/, since 2026-07-16) and live on GitHub Pages; consumers fetch it over the raw GitHub forms below. There is no pending host transition: the data.quantecon.org custom domain was deferred indefinitely on 2026-08-12 in favor of the qeld consumer package (PLAN-QELD-PACKAGE.md, D11) — the stable interface lectures get is a package call, qeld.url('<filename>'), not a branded host (#37, #15). The full convention lives in the draft manual page (QuantEcon.manual#108).

Rules

Layout

• Do not add new consumer-keyed directories (no lecture-xyz/ folders). New datasets go directly in the flat published tree, lectures/<filename>, with their sidecar manifest beside them.
• No folder may imply ownership by a lecture series — any lecture can consume any file.

Every dataset needs a class and a manifest

Classify as exactly one of:

Class	Meaning	Must ship with the file
verbatim	third-party file republished as distributed	source URL, citation, license, retrieval date
constructed	built from upstream sources by our processing	all of the above plus the builder script, committed here
dynamic snapshot	constructed, tracking a moving source (FRED, World Bank)	all of the above plus a refresh cadence

Naming a published file (policy settled 2026-09-07, #113): the tree is flat, so the filename is the URL and the future qeld key. Name the variable, add the frequency when the same variable exists at another frequency, name a composite for its topic rather than its lecture, no source prefix, never a vintage or coverage in a dynamic snapshot's name (a verbatim release keeps upstream's name). Rename freely while consumers: []; after that only as a new filename with the old one kept serving. The six rules with examples are in manifest-schema.yml beside filename.

The schema block is executable (decisions of 2026-09-07 on #120, #121, #122; validator in #119): column patterns are ordered, contiguous, exhaustive and carry the date in a capture group; known_nulls integers are exact and a dynamic snapshot declares null placement under nulls:; dtypes use the pandas-3 names (str, not string/object) and are compared by family. manifest-schema.yml carries the rules.

A constructed dataset without its committed builder is a bug. Manifest fields: source, license (with the verified date it was established), retrieved, integrity (sha256 plus the upstream verification status, see Phase 7), schema (including known_nulls), consumers (repo + lecture file, machine-readable), maintainer, builder / builder_status, cadence (dynamic only). manifest-schema.yml is the authoritative, commented field reference — keep it and this list in step.

Verifying integrity.upstream, by class (once here, not repeated per manifest): re-fetch-and-compare for verbatim; re-run the builder and compare the overlap window for constructed / dynamic; spot-check against the cited publication for author-assembled. When verification is impossible, say so plainly — status: unverifiable with a one-line note is a known status the catalog can show; silence is not. Migration safety (does the file byte-match what the consuming lecture used before a repoint?) is deliberately not a manifest field: it is a one-time gate recorded in the repoint PR, and the manifest's sha256 keeps it reproducible afterwards.

Capture what the source gives you; never let a missing field block a useful dataset. Rich provenance — DOI, upstream version, exact retrieval date, licence id — is always welcome and worth recording whenever it is available, because it makes the data auditable years later at almost no ongoing cost. But effort scales with what the source actually provides: where a field is genuinely unavailable, record it as an explicit, reasoned gap (see the inherited-file states below) rather than fabricating it or refusing the file. A clean, well-documented source should produce a short manifest; only genuinely messy provenance earns a long one.

Two inherited-file states that look like violations but are tracked, not hidden

The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap explicitly — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot (meta#338), to be folded into manual#108.

• retrieved: null — inherited-undated bytes. retrieved is required, but may be null when the bytes were inherited (e.g. from a lecture repo) with no recorded upstream-retrieval date. Do not reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null retrieved must be paired with an integrity.upstream entry that accounts for the gap — never left bare. Any resolved status does that: unverifiable says the vintage cannot be established at all, and diverged or verified say something stronger, because a re-fetch that hashes identically pins the vintage by content rather than by date, which is what a retrieved date was a proxy for. What is not acceptable is null beside unverified — that is two unanswered questions, not one answered a different way. (Amended 2026-08-13, wave B1': the rule previously named unverifiable alone, which was the only case that had arisen. life-expectancy-vs-gdp-per-capita.csv and mpd2020.xlsx already paired null with diverged; B1' added five files where an upstream re-fetch succeeded byte-identically.)
• builder_status: committed-frozen — the builder is here, and deliberately will not run. For a dataset built from a source that must not be refreshed: a frozen vintage, or a scraper we will not re-run. The artifact is kept as the record of what produced these bytes, so it is committed verbatim and not edited — editing it is what would destroy its value as provenance. Distinct from committed, which asserts a runnable four-stage builder, and from unrecovered, which says the builder is absent.
• builder_status: unrecovered — constructed without a recoverable builder. A constructed dataset ships its builder, and one that omits it silently is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep class: constructed — reclassifying to verbatim to dodge the rule is misclassification — set builder: null and builder_status: unrecovered, and the gap stays visible for Phase 9 to recover. unrecovered is for inherited files only; never introduce a new constructed file without its builder.

Repointing a lecture — four ordering traps

All cheap to follow and expensive to discover. PLAN.md carries the reasoning and the current counts.

• Never delete a file a sibling repo reads. lecture-wasm fetches lecture-python-intro's committed blobs by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set.

• Repoint every consumer of a dataset together. The strict audit has no green state for a partially-repointed dataset — pending/landed fails once any consumer reads data-lectures, and repointed/final fails while any consumer still does not. Land the lecture repoints first, then flip migration.yml; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. This binds the lecture PRs too: merging one half of a set while the other sits open opens the same window.

• Repoint, publish, then delete — the published site lags main. A lecture repo that publishes on a tag (lecture-python-intro uses publish*) does not refresh its site when a repoint merges, so the already-published notebooks keep the old URL. Delete the file in the same PR and that URL 404s for every reader who downloads or opens the lecture in Colab, until someone tags a publish. Rendered HTML is unaffected — figures are baked at build time — so nothing will alert you. Split it: repoint the URLs and keep the files, publish, then delete in a follow-up PR. Repos that publish on push to main (lecture-wasm) self-heal and need no split — and neither does deleting a copy that no lecture reads, such as one a repo committed while its lecture fetches another repo's copy by URL (a mirror-orphan).

• Deleting a file does not un-publish it — the build reuses a cached _build. In lecture-python.myst, ci.yml and publish.yml both restore the build-cache artifact produced by cache.yml and then build over it. Sphinx copies html_static_path into _build/html/_static but never prunes assets that have since been removed from source, so a deleted _static file survives in the restored tree and is deployed again. Only cache.yml builds from a clean checkout, and it runs weekly (cron: '0 3 * * 1'). So a deletion merged on a Tuesday keeps being served until the next Monday rebuild and a publish after it. Verify a deletion against the published URL, never against main — the repo, migration.yml and CATALOG.md will all say "deleted" while readers still get the bytes. Measured 2026-08-13 on lecture-python.myst#1035: its preview served every deleted file at HTTP 200 with correct content-type and byte count, while a never-existed control path 404'd. To close the window deliberately, workflow_dispatch cache.yml before tagging the publish — it is a ~1h50m GPU build, which is exactly why the cache exists.

Cross-repo repoints are worked from QuantEcon/workspace-lectures — same branch name in each repo, one PR per repo, no aggregate PR.

Repointing a lecture — one scope rule

• A migration moves bytes; it does not update them. Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in integrity.upstream — the manifest is the register of record, and the generated catalog shows the delta from the day it is found — and leave the decision for after the migration. The upstream-delta register issue #39 was that register while the migration ran; it closed on 2026-09-07 once the migration completed (40 of 40 repointed 2026-08-18, strict audit green) and its review step had run (2026-09-01: mpd2020.xlsx and life-expectancy-vs-gdp-per-capita.csv kept deliberately, business_cycle_data.csv refreshed by #112). Its history is the worked example of the three steps; new deltas need no issue — set integrity.upstream in the same PR that finds them, and the catalog is where a reviewer looks. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a new filename, never a silent replacement.

Corrections vs vintages

• Corrections (bad parse, wrong units, corrupt rows): fix in place, same filename — every consumer should get the fix. Use the manifest's consumers list to know which lectures to rebuild/review.
• New vintages (e.g. Maddison 2020 → 2023): new filename — the old vintage stays valid; consumers opt in.
• Never delete or rename a published file without checking consumers (and, until manifests are backfilled, grepping the lecture repos).

URL forms — the host is a function of storage and of runtime

When writing or reviewing URLs that fetch from this repo (in docs, tests, or lecture repoints):

Form	LFS-tracked file	plain-git file	fetchable from a browser
raw.githubusercontent.com/…	❌ 200 with pointer text	✅	✅ access-control-allow-origin: *
github.com/{org}/{repo}/raw/{ref}/…	✅	✅	❌ 302 with an empty access-control-allow-origin
media.githubusercontent.com/media/…	✅	❌ 404	✅

There is no form that is safe in every case, and that is the whole trap. The github.com/…/raw/ form resolves whatever the storage, but only because the server resolves storage on your behalf — and it is exactly the form a browser rejects.

• CPython consumers (site notebooks, Colab, every series except lecture-wasm): https://github.com/QuantEcon/data-lectures/raw/main/lectures/<file>.
• lecture-wasm (code cells execute under Pyodide in the reader's browser — repoint rule 5): https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/<file>, and nothing else. The strict audit hard-fails any lecture-wasm code-cell read via a github.com/… form. {download} targets and prose links are navigations and are CORS-exempt, so any resolving form is fine there.
• Never media.githubusercontent.com for a path in this repo (repoint rule 6). It is the LFS media endpoint and routes per path, so it 404s everything lectures/ publishes — which is all of it, since the published tree is 100% plain git. The strict audit fails on this too.
• There is no pending final-form swap — the two direct forms above are standing (decision 2026-08-12, PLAN-QELD-PACKAGE.md D11; data.quantecon.org deferred, #37). Once the qeld package ships, lecture code reads qeld.url('<filename>'), which emits exactly these forms per runtime; prose links, {download} targets and URL-is-the-lesson reads keep the direct form permanently.
• Never reference a non-default branch in a published URL — the audit fails on any ref that is not main.

The failure modes are silent in both directions, which is why all three are machine-checked: a raw read of an LFS path returns HTTP 200 with ~133 bytes of pointer text, and pd.read_csv on it raises nothing and yields a 2×1 frame.

LFS, and sources/ vs lectures/

The published tree is plain git. Do not put an LFS object in lectures/ (settled 2026-08-06, PLAN Phase 3). Every published dataset fits comfortably in plain git — the largest, SCF_plus_mini_no_weights.csv, is 72.4 MiB against GitHub's 100 MiB limit. Keeping it that way means no consumer can ever hit the raw-vs-media trap above; the hazard is removed rather than managed.

LFS exists here for one purpose: upstream inputs that builders consume and no lecture reads, which live in sources/ and are never served.

• lectures/<file> — a published dataset. Plain git, sidecar manifest required, its filename is an API.
• sources/<file> — a builder input. Per-path LFS, no manifest, not served, recorded instead in sources/README.md — the audit trail: origin, retrieval date, licence, upstream identifier (DOI where one exists), sha256, and the builder that consumes it. CI enforces the last of those: check_consumed_files.py requires every file here to be captured by the LFS rule and to hash to a sha256 recorded under a ## <filename> heading in that README, and fails on a README entry with no file. It reads the pointer's oid rather than the object, so it costs no LFS bandwidth.

Rules that still apply:

• LFS is per-path, opt-in, large binaries only. Never a blanket rule like high_dim_data's *.csv and *.dta.
• Do not LFS-track an existing file until you've confirmed no consumer fetches it via raw.githubusercontent.com — converting silently turns their download into pointer text.
• A builder must read its input from sources/, never over the network from another QuantEcon repo. That is how a retired repo becomes load-bearing again.
• Two workflows check this repo out, and both now say lfs: false — .github/workflows/audit-dashboard.yml (the Pages deploy) and .github/workflows/consumed-file-check.yml (every pull request). Leave them that way: lfs: false is the assertion that nothing published is an LFS object. If a lectures/ file is ever tracked by mistake, the checker hashes the pointer and goes red, and Pages deploys the same pointer bytes a reader would get from raw.githubusercontent.com — whereas lfs: true fetches the real bytes, passes green, and publishes a file that works only from Pages. It also keeps sources/ (a 99 MiB LFS object) off every run; LFS bandwidth is an org-wide quota.
• git check-attr filter -- sources/<file> must print filter: lfs before you git add anything to sources/. SCF_plus.dta is 103,934,093 B against GitHub's 104,857,600 B hard limit, so a mis-scoped rule does not error — the push succeeds as plain git and the blob is in history permanently. CI asserts this too, but only after the fact: by the time a PR goes red the blob is already in the branch's history, so run it yourself first.

When a published file approaches the 100 MiB blob limit

The limits: 50 MiB warns on push, 100 MiB (104,857,600 B) is a hard block, 2 GiB per push. Work down this ladder in order — every rung but the last keeps a browser consumer working with at most a change of filename. Reasoning and measurements: #58.

1. Subset it (any published file over 50 MiB). Justify the size in the PR: does the lecture read all of it? The house precedent is the SCF chain — a 103,934,093 B .dta became a 75,902,999 B mini and then a no-weights mini. A derived teaching extract is a dataset in its own right and gets its own manifest.
2. Plain git, up to ~90 MiB. Served gzipped by both raw and Pages, packed compressed in git, no metered quota.
3. gzip it in place, over ~90 MiB. Lossless, so it does not violate "a migration moves bytes; it does not update them" — SCF_plus.dta goes to 17,854,577 B (5.82×), and pandas infers decompression from the .gz suffix, so the only consumer change is the filename. Both hosts return application/gzip with identity encoding, so there is no double-decompression trap.
4. Split into ≤50 MiB parts, if compression is not enough.
5. Publish from outside git via the Pages artifact. This repo is already build_type: workflow; the blob limit binds only because the workflow copies lectures/ out of the checkout. Fetching a file during the workflow escapes it entirely — at the cost that the published file is no longer reproducible from a checkout, and the hash gate needs another source.

Never put an LFS object under lectures/, and never reach for GitHub release assets: they send no access-control-allow-origin on any hop, so a browser cannot read them. Parquet is not a size remedy here either — pyarrow is absent from the Pyodide lecture-wasm pins, and gzipped CSV is smaller than Parquet on this data anyway.

Builders

One builder per published dataset, in builders/, named for the dataset it produces: builders/<stem>.<ext> builds lectures/<stem>.<ext2>. The stem is the dataset's, not the lecture's — builders/japan_earthquakes.py writes lectures/japan_earthquakes.csv. That makes the manifest's builder: field predictable and lets CI assert it.

Where one builder produces a set of files, name it for the set and let each file's manifest point at the same path — the SCF and Forbes builders each write two. The stem rule is the default, not an invariant; what CI asserts is that every builder: path exists, and that a dataset claiming a builder names one. A builder's provenance byproducts — upstream metadata dumps that are not datasets — go to provenance/, never lectures/: business_cycle.py writes three datasets there and two dumps here.

scripts/ is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart.

Where a builder reads its input from. The normal case is the third-party upstream, fetched at run time: eight of the nine committed builders here do that, and it is the fetch stage of the contract below. A builder reads from sources/ only when the input cannot be re-fetched — the upstream is gone, unlocatable, or was inherited with no recoverable source. sources/ is that exception layer, not a general input tree, and it is emphatically not "the big-file directory": the defining property is un-refetchability, not size. What it must never be is a network read from another QuantEcon repo — that is how a retired repo becomes load-bearing again.

Builders follow four stages — fetch → pre-process → validate → write — and only write on validation pass. The validate stage is shared: builders/_validate.py reads the manifest's schema block as its spec (columns and pattern runs, dtype families, exact known_nulls, the nulls: placement rule, row_count_floor, date_range) and measures the overlap window against the previous vintage; a builder calls validate(frame.reset_index(), manifest, previous) and layers on only what the schema cannot say — value bands, a grid check, the per-series revision bound (a tracking snapshot is revised by its source, so the test is a tolerance plus a printed summary, never equality). The same function runs over every committed CSV on every PR (scripts/validate_datasets.py, validate-datasets.yml), so a manifest that drifts from its bytes fails the PR, not the next refresh. A dynamic builder also exposes check_committed() — its own validate() on the committed bytes, no network — which the same workflow runs (--builders) under both pandas majors; a builder-specific check that breaks on a pandas change fails the PR that introduces it (#128 was a week-old pandas-3 break the shared layer could not see). Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build.

One builder per source, composite files where a lecture reads series together, shared fetch libraries (decided 2026-09-01, #26): builders/_fred.py is the Fred class for FRED reads — no pandas_datareader in a builder — and a lecture's FRED data is one file with the lecture's own variable names as columns, not six. A builder that writes a set validates every file before writing any.

A dynamic snapshot's builder also honours the refresh contract that .github/workflows/refresh-snapshots.yml and scripts/snapshots.py rely on — copy builders/_template.py: --out-dir (dry run for the weekly canary), --summary-json (the run summary the manifest stamp and the refresh PR body are built from), writes through a temp file and os.replace(), and exit code 2 for a ValidationError against 1 for a fetch failure, which is how the canary issue tells "the data broke the contract" from "the network was down". The manifest fields the workflow stamps (retrieved, integrity.sha256, integrity.upstream.*, schema.date_range.end) must be single-line values with their reasoning in comments above them, not beside — the stamp replaces the line. And no prose in the manifest may embed a fact a refresh can change — an end year, an observed range, a row count — because nothing re-writes prose: the first refresh PR (#112) shipped a column description still saying "YR2023 in the committed bytes".

Live APIs

Live API calls are for teaching data access, not for getting data. Don't propose "the lecture should just call the API" as a fix — the fix is a snapshot here plus an automated refresh.

Refresh, break, or schema change — who gets told

A dynamic snapshot has three interfaces, and each failure mode has a different audience. Settled 2026-09-01 while retrofitting business_cycle.py; the workflows that act on it are PLAN Phase 5.

What happened	Detected by	Who is told	Consumer code changes?
Upstream interface changed (a renamed column, a dropped series, a units switch)	the builder's validate() fails; nothing is written, the last-good snapshot stays	an issue in this repo (the sources-alive canary). Consumers are unaffected by construction	No. Absorb it in the builder's pre_process stage so the published schema is unchanged — that adapter logic is where upstream churn is supposed to live
Successful refresh — schema intact, values revised	the refresh lands as a PR here whose body is validate()'s overlap summary	on merge, each repo in the manifest's consumers list, per its on_refresh (manifest-schema.yml): rebuild dispatches a build, review opens an issue there with the summary, for a lecture whose prose narrates a number	No, but figures and narrated numbers may need an author's eye
Published schema changes deliberately — the upstream change cannot honestly be absorbed	a decision, not a detection	an issue in every consumers[].repo, opened by hand as the invitation to opt in	Yes, on the consumer's schedule: new filename per "Corrections vs vintages"; the old file stays valid

The case none of this covers is a lecture where the API call is the lesson (business_cycle teaches wb.series.info): an upstream change to the call itself must reach the lecture, because the code on the page is the content. A snapshot twin still belongs beside it — as the lecture-wasm read and the fallback — but the canary only makes the break heard sooner.

Licensing and attribution

Because this repo is a stability cache, not a content-distribution host (see "What this repo is"), the licence question is "is this source OK to cache and serve publicly, with attribution?" — not "may we republish this as our own?". Attribution to the upstream source is carried in every manifest (source: name, url, series, citation), and that is the primary obligation.

For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a known yes, recorded once per source — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's redistribution field as a cheap binary gate (permitted / restricted): a fast permitted for public statistics agencies, restricted blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file.

Licensing does not gate migration (settled 2026-08-06, #35). A file the lectures have already served publicly migrates here with its licence recorded as found — including redistribution: restricted and a null licence name where that is the honest answer — and is logged in the inventory (#35, feeding workspace-lectures#20) with a note. Resolve it (permission, an open replacement, or removal) before this repo is ever promoted as a branded public open-data host: that promotion is the gate, not the file's move — and with the custom domain deferred indefinitely (2026-08-12, D11), no promotion is scheduled, so the gate looms over nothing while the #35 inventory stays tracked. Rehosting the same bytes with better provenance and an explicit licence field improves on the status quo, so a licence question is never a reason to stall a migration.

This covers inherited data only. A genuinely new dataset — one with no prior life in a lecture repo — still has its licence established before it lands, as the P5 additions all did.

The audit dashboard stays truthful

The generated dashboard (scripts/build_audit.py, #20) verifies its three inputs against a fresh scan of the lecture repos, and the strict build fails when they drift. Keep them current in the same PR as the change that moves reality:

• Landing or repointing a dataset → update its migration.yml record (status, PR refs, dates). A dataset marked repointed whose consumers still read an old URL — or the reverse — is a build failure, by design.
• A new manifest (lectures/*.yml) → add its migration.yml record; delete any stale entry for the same file in scripts/audit_annotations.yml (manifested datasets must not be annotated there).
• The weekly scan flags an unannotated reference (a lecture repo started reading a new file) → classify it and add an entry to scripts/audit_annotations.yml; that file holds judgment (description, provenance, why-live), never facts the scan can derive.

site/ and audit.json are generated — never commit them; CI rebuilds and deploys on every push to main.

Cross-repo hygiene

• Changes here often pair with PRs in lecture repos and issues in QuantEcon/meta. In commit messages and PR bodies, never place a GitHub closing keyword (fixes, closes, resolves, …) immediately before a cross-repo reference like QuantEcon/meta#336 — GitHub will auto-close the referenced issue when the commit lands on the default branch. Write "See QuantEcon/meta#336" or "Part of QuantEcon/meta#336".
• When a decision marked (open) in PLAN.md gets settled upstream, update PLAN.md and this file in the same PR that acts on it.

Repo map

lectures/            # the published tree — flat, live on Pages; read via raw URLs
                     #   today, qeld.url() once the package ships (PLAN-QELD-PACKAGE.md)
                     #   44 datasets, 44 manifests (complete since 2026-09-01;
                     #   the business_cycle set added three on 2026-09-01, #114).
                     #   Manifests are sidecars: <filename>.yml
builders/            # one builder per published dataset — NOT published
                     #   builders/<stem>.py builds lectures/<stem>.<ext>
sources/             # inputs a builder cannot re-fetch — NOT published, per-path LFS
                     #   no manifests; sources/README.md is the audit trail
provenance/          # upstream metadata dumps a builder writes beside its data —
                     #   NOT published, no manifests, regenerated every run (#13)
scripts/             # repo tooling — NOT published, produces no dataset
  build_catalog.py   #   generates CATALOG.md from the manifests
  build_audit.py     #   the audit dashboard: scan lecture repos → audit.json → site/
  render_audit.py    #   its render stage
  snapshots.py       #   dynamic snapshots: which are due, stamp a manifest after
                     #   a refresh, render the refresh PR body
  audit_annotations.yml  # curated judgment for not-yet-migrated data refs
migration.yml        # migration lifecycle tracker (status + PR provenance per dataset)
manifest-schema.yml  # per-dataset manifest schema (strawman)
requirements.txt
PLAN.md              # roadmap — start here
AGENTS.md            # this file

The Feb 2025 consumer-keyed layout (lecture-python-intro/{static,dynamic,scripts}/) was flattened into this tree on 2026-07-16, while nothing referenced the repo.

That freedom is now spent — the repo is live. The first repoint merged on 2026-07-17 (P1: msy_fishery in lecture-python-intro reads lectures/lingcod_msy_recovery.csv from raw/main), so every move or rename in lectures/ is a breaking change for a live lecture build. Treat published filenames as an API: corrections in place, new vintages under new names, and check consumers before touching anything. Enforced by the protect-main ruleset (PRs only, no force-push, and the consumed-files check is required — a PR that breaks a consumed file cannot merge).

The git history is complete — it does not begin at #57. The root commit is 77ece40 ("Initial commit", 2025-02-09) and main carried 104 commits at 47017ea on 2026-09-07, so the Feb 2025 migration from QuantEcon/data (c0adb7a) and the 2026-07-16 flatten (#10, 52dbb89) are both in git, and git log --follow crosses them. A clone reporting a single root at 931d626 with 50 commits is shallow, not authoritative: git clone --depth=50 at 818811b reproduces that signature exactly, because a shallow boundary hides its parents and so is indistinguishable from a root. Run git rev-parse --is-shallow-repository before concluding history is missing — the 2026-09-07 validation did not, and refuted a checklist box on the artefact.

check_consumed_files.py hashes every file whose manifest records an integrity.sha256, with or without consumers. That is the deliberate rekeying in #56 — manifests land ahead of their repoints, so keying the hash on consumers made the one PR that introduces new bytes the one PR that never verified them. The consequence for a validation checklist is that consumed-files goes red for any change to a published file's bytes: measured on #130, a manifest-only mutation left it green with only validate red, while appending a column to gdp_growth_annual.csv turned both red. A check that wants the schema gate isolated must mutate a manifest, not the data.