Manifest-driven validate(): the schema block is the spec, shared by the builders and a PR check

[mmcky]

The manifest's schema block becomes the spec a validator executes, as #119 asked: one function, builders/_validate.py, used unchanged by the two dynamic builders and by a new PR check. The four decisions #124 wrote into the manifests (#120 pattern, #121 nulls, #122 dtypes, #113 naming) are what make it executable; nothing here changes a data file.

What lands

Piece	What it does
builders/_validate.py	validate(frame, manifest, previous): walks columns in order (a pattern claims a contiguous run, at least one, exhaustive, capture group → date_range); compares dtypes by family (str/string/object one family, datetime64 any unit, a nullable bool read as object matches bool, an all-null column matches anything); exact known_nulls, with an undeclared column in a static file meaning zero; the nulls: placement rule (along, leading, recent, ended, inner); row_count_floor; date_range.start (and .end for non-dynamic files). With previous, it measures the overlap window — no series or period may disappear, no populated cell may go empty — and returns revisions for the builder to bound. read_raw() reads a CSV as written: delimiter honoured, parse_dates derived from datetime64 declarations, no index column.
scripts/validate_datasets.py + .github/workflows/validate-datasets.yml	On every PR and push to main: a conformance pass over all 44 manifests (canonical dtype names, name-xor-pattern with exactly one capture group, integer known_nulls, a nulls: block on every dynamic snapshot, known_nulls_total only inside header: null sheet entries) and the byte-level validate() over every CSV. Non-CSV formats get conformance only and the log says so. Runs on pandas 2.3.3, one major behind the lectures, as the standing check that family comparison keeps both green.
business_cycle.py, business_cycle_fred.py, _template.py	Call the shared validator on frame.reset_index() and keep only what a schema cannot say: the year grid, the economy set, value bands, recency, each FRED series' first observation, USREC 0/1, the per-series revision bound. The FRED builder's MONTHLY_FROM / LAST_OBS / KNOWN_HOLES constants are removed; the manifest's nulls: block is now the single source.
Docs	PLAN Phase 5's PR-validation box ticked; manifest-schema.yml header says the schema block is validated; AGENTS.md Builders and builders/README.md describe the two-layer shape; consumed-file-check.yml's header names its sibling instead of promising to be subsumed.

Two contract fixes the validator found

The FRED refresh would have failed its first monthly run. Dry-running business_cycle_fred.py against live FRED on 2026-09-07 returned a frame to 2026-08 with only UNRATE and USREC populated for that month; UMCSENT, CPILFESL and INDPRO release around the 15th. The hand-written check asserted zero trailing nulls for every live series, so the September canary would have gone red on a publication lag, not a data problem. us_business_cycle_monthly.csv.yml now says recent: 1 and carries no exact counts, since every count in that file moves with the newest month; placement is the contract, which is what #121 decided.

Namibia. countries.csv has one null the manifest did not declare: the ISO alpha-2 code NA, which a default read_csv parses as missing. The schema describes a default read (#122), so it is declared as Country code: 1 with the reason, and a note that a lecture wanting Namibia's code must pass keep_default_na=False.

Verification

• scripts/validate_datasets.py: 44 of 44 pass under pandas 2.3.3 and under 3.0.5.
• Both builders dry-run green against the live sources with --out-dir, reproduce the committed vintages (0 cells revised in every overlap window), and the FRED summary feeds snapshots.py pr-body unchanged.
• Mutation tests on the committed frames (script in the PR description's source session, not committed): 16 mutations caught — an unclaimed column, a mistyped year column, a wrong dtype, a changed exact count, a hole inside a series, a null too far from the end, a moved start, too few rows, a cell going empty, a period disappearing, a new FRED hole, a hole in the ended series, a non-integer USREC, a wrong static count. The four not caught are the builder layer's (an interior year dropped → the grid check; INDPRO's first observation → FIRST_OBS), both confirmed caught there, or permitted by design (a lagging newest month; a backfilled hole).
• check_consumed_files.py: 44 manifests, 47 files hashed, 0 errors. CATALOG.md regenerated with no diff.

Not in this PR

Reading xlsx/xls/dta/npy/json ranges the way each lecture does, so those 13 manifests are conformance-only for now. Making validate-datasets a required status check is a branch-protection setting. The mutation script could become a committed test once the repo has a test runner.

Closes #119. With it, every child of #14 is closed.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The builders/business_cycle.py refactor drops a non-null assertion for the Country label column, which is not otherwise enforced by the shared validator for dynamic snapshots and can allow malformed refreshes to pass validation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR makes the manifest schema block an executable validation contract by introducing a shared validator (builders/_validate.py) and a CI workflow that runs it on every PR, then refactors the dynamic-snapshot builders to delegate schema-derived checks to that shared implementation.

Changes:

• Added builders/_validate.py (manifest-driven read_raw() + validate()), and a new PR check (scripts/validate_datasets.py + .github/workflows/validate-datasets.yml) that runs conformance for all manifests and byte-level validation for CSVs.
• Refactored dynamic builders (builders/business_cycle.py, builders/business_cycle_fred.py) and the builder template to use the shared schema validator and keep only dataset-specific checks locally.
• Updated manifests/docs to reflect the executable-schema contract and refine null placement rules for dynamic snapshots.

File summaries

File	Description
scripts/validate_datasets.py	New CLI used by CI to enforce manifest conformance and CSV byte-level schema validation.
PLAN.md	Marks PR validation as landed and documents the new validator + workflow behavior.
manifest-schema.yml	Updates documentation to reflect that the schema block is now validated.
lectures/us_business_cycle_monthly.csv.yml	Adjusts null contract to placement-based (including recent: 1) to handle publication lag.
lectures/countries.csv.yml	Declares a known_nulls entry for Namibia’s NA being parsed as missing by default read_csv.
builders/README.md	Documents shared validator usage across builders and CI.
builders/business_cycle.py	Switches schema/invariant checks to the shared validator and keeps builder-specific validation.
builders/business_cycle_fred.py	Switches schema/invariant checks to the shared validator and keeps builder-specific validation.
builders/_validate.py	New shared, manifest-driven validator implementing schema rules and overlap-window measurement.
builders/_template.py	Updates the template to call the shared validator as the base validation layer.
AGENTS.md	Documents the shared validate stage and its role in builders and PR validation.
.github/workflows/validate-datasets.yml	New CI job running the schema validator on PRs and pushes to main.
.github/workflows/consumed-file-check.yml	Clarifies relationship to the new schema-validation workflow.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.