P4: the business_cycle snapshot set — three World Bank tables and a FRED composite

AGENTS.md

@@ -120,6 +120,8 @@ Where one builder produces a **set** of files, name it for the set and let each
 
 Builders follow four stages — **fetch → pre-process → validate → write** — and only write on validation pass (expected columns/dtypes, row-count floor, recency of date range, no all-NaN columns, and a **bounded** overlap window against the previous vintage — a tracking snapshot is revised by its source, so the test is a tolerance plus a printed summary, never equality). Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build.
 
+**One builder per source, composite files where a lecture reads series together, shared fetch libraries** (decided 2026-09-01, #26): `builders/_fred.py` is the `Fred` class for FRED reads — no `pandas_datareader` in a builder — and a lecture's FRED data is one file with the lecture's own variable names as columns, not six. A builder that writes a set validates every file before writing any.
+
 **A dynamic snapshot's builder also honours the refresh contract** that `.github/workflows/refresh-snapshots.yml` and `scripts/snapshots.py` rely on — copy `builders/_template.py`: `--out-dir` (dry run for the weekly canary), `--summary-json` (the run summary the manifest stamp and the refresh PR body are built from), writes through a temp file and `os.replace()`, and exit code **2** for a `ValidationError` against **1** for a fetch failure, which is how the canary issue tells "the data broke the contract" from "the network was down". The manifest fields the workflow stamps (`retrieved`, `integrity.sha256`, `integrity.upstream.*`, `schema.date_range.end`) must be single-line values with their reasoning in comments **above** them, not beside — the stamp replaces the line. And **no prose in the manifest may embed a fact a refresh can change** — an end year, an observed range, a row count — because nothing re-writes prose: the first refresh PR (#112) shipped a column description still saying "YR2023 in the committed bytes".
 
 ### Live APIs

PLAN.md

@@ -340,7 +340,7 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde
   **The flip was the acceptance test, and it measured as one.** Dry-run locally in both directions before pushing: `landed` → exit 1 with 6 warnings; `repointed` → exit 0. So the red window was real, opened when the last consuming PR merged, and closed with the flip. **Do this both-directions dry-run on every future wave** — it converts "same-day, trust me" into a measurement.
 
   Five things P3 proved that were not on its test list. A `constructed` dataset's builder must land in the **same** PR as the data, because `check_consumed_files.py` asserts the `builder:` path resolves. `builders/README.md`'s coverage table is a real coverage report and goes stale silently. The plain-git decision costs ~10 MB of packed history for 110 MB of working tree, since CSV compresses 5-22×. The **C0 → C1 → C2 ordering worked and proved less than it looks like** — the sync PR it was designed to defuse (QuantEcon/lecture-intro.zh-cn#293) touched zero data-read lines, zero `# i18n` markers and zero protected localisations, but nothing ever asked the model to rewrite those cells, so the markers remain unexercised, prompt-level protection and **the hand-diff is what protects a localisation**. And the translation sync is **`.md`-only**, so no hand-localised `_static` asset can be created, updated or repaired by it — every `data.ipynb` copy had to be repointed by hand in all four repos, filed upstream as QuantEcon/action-translation#271
-- [ ] **P4 — dynamic snapshot twin**: `UNRATE`, consumed today by 4 lectures across 3 repos via 2 access methods. Tests: the full dynamic template — manifest, four-stage builder, refresh-as-PR, canary catching an induced failure — plus the documented live-call ↔ snapshot switch mechanism
+- [ ] **P4 — dynamic snapshot twin**: originally `UNRATE` alone; **reframed 2026-09-01** as the `business_cycle` set, because the lecture that needs a twin is excluded from `lecture-wasm` for want of one and a partial twin buys it nothing. Done so far: `business_cycle_data.csv` manifested and its builder retrofitted ([#109](https://github.com/QuantEcon/data-lectures/pull/109)); the refresh-as-PR and canary workflow ([#110](https://github.com/QuantEcon/data-lectures/pull/110)); the first real refresh ([#112](https://github.com/QuantEcon/data-lectures/pull/112)); the World Bank set extended to three tables and the FRED half landed as one composite monthly file on a shared `builders/_fred.py` library ([#114](https://github.com/QuantEcon/data-lectures/pull/114)). Remaining: the `lecture-wasm` adoption ([QuantEcon/lecture-wasm#70](https://github.com/QuantEcon/lecture-wasm/issues/70) — intro keeps its live calls as the lesson), the flip with `on_refresh: rebuild`, and a canary run catching an induced failure
 - [ ] Verify each migrated URL with a pyodide/JupyterLite fetch (CORS, meta#143)
 - [ ] Fold every validated decision into the draft `styleguide/datasets.md` (manual#108) as it is proven
 

builders/README.md

@@ -27,7 +27,12 @@ upstream outage may fail a refresh, it must never break a lecture build. The
 architecture discussion is in
 [#14](https://github.com/QuantEcon/data-lectures/issues/14); the copy-able
 template is [`_template.py`](_template.py) (not a builder — the underscore
-keeps it out of any manifest).
+keeps it out of any manifest). Shared fetch code lives beside it under the
+same convention: [`_fred.py`](_fred.py) is the `Fred` class every FRED
+builder should use (`fred.series('UNRATE')`, `fred.frame([...])`), so a fetch
+stage is a line and `validate()` is the only thing worth reading. One
+builder per **source** for a lecture's data, writing a composite file where
+the lecture reads the series together (decided 2026-09-01 on #26).
 
 A **dynamic snapshot's** builder additionally honours the refresh contract
 (`.github/workflows/refresh-snapshots.yml`, `scripts/snapshots.py`):
@@ -59,7 +64,8 @@ re-fetched** — see `AGENTS.md`.
 | `fred_data.py` | `fred_data.csv` | committed — fetches six FRED series live over a pinned 1953-04..2024-12 window (yields and the recession dummy are stable history, unlike the BBH national-accounts snapshot). Reproduces its output byte for byte (2026-08-18) |
 | `hansen_singleton_1982_data.py` | `hansen_singleton_1982_data.csv` | committed — fetches FRED and the Ken French factors live. Reproduces its output byte for byte (2026-08-13) |
 | `hansen_singleton_1983_data.py` | `hansen_singleton_1983_data.csv` | committed — the same construction plus a T-bill leg, so its output is a strict superset of the 1982 file's. Reproduces its output byte for byte (2026-08-13) |
-| `business_cycle.py` | `business_cycle_data.csv` (plus two dumps to `provenance/`) | committed — the repo's one **dynamic snapshot** (`cadence: annual`), retrofitted to the four-stage contract 2026-09-01. Fetches live WDI; does NOT reproduce its bytes and is not meant to — the World Bank revises the series (63 of 64 year columns moved between the 2025-02 vintage and 2026-09-01). validate() bounds the overlap window at 5 pp and prints the revision summary, which is the review surface for a refresh PR |
+| `business_cycle.py` | `business_cycle_data.csv`, `unemployment_rate_annual.csv`, `private_credit_to_gdp.csv` (plus two dumps to `provenance/`) | committed — the World Bank half of the `business_cycle` lecture's data as three **dynamic snapshots** (`cadence: annual`), one builder writing a set. Fetches live WDI; does NOT reproduce its bytes and is not meant to — the World Bank revises the series (63 of 64 GDP-growth columns moved between the 2025-02 vintage and 2026-09-01). validate() places nulls (before an economy's first observation or in the newest two years, never inside a series), bounds each table's overlap window, and prints the summary — the review surface for a refresh PR. Validates all three before writing any |
+| `business_cycle_fred.py` | `us_business_cycle_monthly.csv` | committed — the FRED half as one composite **dynamic snapshot** (`cadence: monthly`): six series on a monthly grid from 1919. Built on the shared `_fred.py` library; declares every structural null exactly (series starts, UMCSENT's sparse pre-1978 years, the 2025-10 shutdown hole) so a new hole fails the refresh |
 | `webscrape_forbes.ipynb` | `forbes-global2000.csv`, `forbes-billionaires.csv` | **committed-frozen** — an undocumented Forbes API, a spoofed user-agent and hardcoded GDPR consent cookies. Defects recorded in the two manifests rather than fixed |
 | `generating_mini.md` | `SCF_plus_mini.csv`, `SCF_plus_mini_no_weights.csv` | **committed-frozen** — its `to_csv` calls are commented out upstream and stay that way. As written it still fetches the `high_dim_data` URL; that URL is historical, and the input is now committed at `sources/SCF_plus.dta`. See `sources/README.md` |
 | `usa-gini-nwealth-tincome-lincome.ipynb` | `usa-gini-nwealth-tincome-lincome.csv` | **committed-frozen** — three independent reasons, any one sufficient: no validate stage; it raises under the pinned pandas 3 (`np.asarray` of a Series is read-only under copy-on-write, so `rd.shuffle` fails — the lecture got the `.copy()` fix in QuantEcon/lecture-python-intro#776, this notebook did not); and it is non-deterministic, so it cannot reproduce its own bytes. It is also the only builder here whose input is **another file in this repo** |

builders/_fred.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""
+Shared FRED fetch library for builders -- the `Fred` class.
+
+One place for the things every FRED read has to get right, so a builder's
+fetch stage is a line or two and its validate() is the only thing worth
+reading. Plain HTTP GET against `fredgraph.csv`, no `pandas_datareader` (the
+decision on QuantEcon/data-lectures#26: a wrapper library in the builder
+re-creates the single-point fragility Phase 5 exists to remove).
+
+    from _fred import Fred
+    fred = Fred()
+    unrate = fred.series('UNRATE')                          # full history
+    yields = fred.frame(['GS1', 'GS10'], start='1953-04-01', end='2024-12-01')
+    tips   = fred.series('DFII5', freq='Monthly', agg='avg') # daily -> monthly
+
+What it normalises:
+
+- the date column, which fredgraph titles `observation_date` today and
+  `DATE` in older exports -- every Series/DataFrame comes back with a
+  DatetimeIndex named `DATE`, so committed files keep the header the
+  lectures expect
+- FRED's `.` for a missing observation -> NaN
+- a User-Agent header, which fred.stlouisfed.org has been seen to require
+- one series per request, aligned with an outer join in frame(), so a
+  series that starts later is simply empty before its first observation
+
+Not a builder: the leading underscore keeps it out of any manifest's
+`builder:` field. Import it as `from _fred import Fred` (builders run with
+their own directory as sys.path[0]).
+"""
+import io
+import urllib.parse
+import urllib.request
+
+import pandas as pd
+
+FREDGRAPH = 'https://fred.stlouisfed.org/graph/fredgraph.csv'
+
+
+class Fred:
+    def __init__(self, user_agent='qeld-builder', timeout=60):
+        self.user_agent = user_agent
+        self.timeout = timeout
+
+    def _get(self, params):
+        query = urllib.parse.urlencode({k: v for k, v in params.items() if v is not None})
+        request = urllib.request.Request(f'{FREDGRAPH}?{query}',
+                                         headers={'User-Agent': self.user_agent})
+        with urllib.request.urlopen(request, timeout=self.timeout) as response:
+            return response.read()
+
+    def series(self, sid, start=None, end=None, freq=None, agg=None):
+        """One FRED series as a pd.Series named `sid`, indexed by DATE.
+
+        `start`/`end` are ISO dates (fredgraph's `cosd`/`coed`); `freq` and
+        `agg` request a server-side frequency change, e.g. freq='Monthly',
+        agg='avg' for the monthly mean of a daily series (fredgraph's
+        `fq`/`fam`)."""
+        payload = self._get({'id': sid, 'cosd': start, 'coed': end, 'fq': freq, 'fam': agg})
+        frame = pd.read_csv(io.BytesIO(payload), index_col=0, parse_dates=True, na_values='.')
+        if frame.shape[1] != 1:
+            raise ValueError(f'{sid}: expected one value column, got {list(frame.columns)}')
+        s = frame.iloc[:, 0].rename(sid)
+        s.index.name = 'DATE'
+        return s
+
+    def frame(self, sids, start=None, end=None):
+        """Several series on one DATE index (outer join, sorted)."""
+        out = pd.concat([self.series(sid, start, end) for sid in sids], axis=1).sort_index()
+        out.index.name = 'DATE'
+        return out