Correct two manifest records and pin the builder environment - #89
Merged
Conversation
Three defects the #84 validation found in records written during waves B1' and B2', plus the schema/practice gap bundled with them. The 1983 manifest described consumption_per_capita as "Observed 2.14e-4 to 2.66e-4". Re-derived from the committed bytes with the stdlib csv module: the minimum is 2.1254904619e-4 at 1960-12-31, which rounds to 2.13e-4, and the second-smallest value rounds there too — so 2.14e-4 matched no observed value. Both bounds now carry the date they occur on. Both hansen manifests quoted the Ken French zip header for the T-bill leg's lineage but stopped mid-sentence, inside quotation marks. Re-fetched 2026-08-17, unchanged at the 202606 CRSP vintage; the quote is now complete and each manifest says plainly that the ICE BofA clause applies to no row in a sample ending 1978. The issue also expected this quote in a builder docstring — it is not there; both builders reference the zip without quoting its header, so the manifests were the only place to fix. requirements.txt pinned wbgapi, PyYAML and scipy but not pandas, which 9 of the 10 builders import. Both hansen manifests carry integrity.upstream.status: verified, which was therefore measured against an unrecorded ambient environment. Pinned at 2.3.3 — the version they were verified against — with a comment recording that they also reproduce byte-identically under 3.0.5, so the pin documents the measurement rather than guarding a known sensitivity. manifest-schema.yml omitted source.doi (16 of 33 manifests), source.version (10), source.note (20) and license.note (16). All four are documented, and the header now states what the file is: a record of the fields that recur, not a closed schema, with extensions expected and documented back once they earn their place. That gives a field-by-field conformance check a defined answer and names the two existing extensions (schema.sheets, schema.read_as). Verified: all 33 manifests and the schema parse; CATALOG.md regenerates unchanged, so no consumer of these fields exists yet. Closes #85 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR corrects provenance/schema metadata for the Hansen–Singleton datasets and improves builder reproducibility by pinning a key dependency, aligning the repo’s dataset manifests and schema documentation with re-validated primary evidence.
Changes:
- Pin
pandas==2.3.3inrequirements.txtto make builder verification environments more reproducible. - Fix/complete Ken French zip-header quotations in the 1982/1983 Hansen–Singleton manifests.
- Correct the documented observed lower bound for
consumption_per_capitain the 1983 manifest and document additional commonly-used manifest-schema fields (source.doi,source.version,source.note,license.note).
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| requirements.txt | Pins pandas to record the builder verification environment. |
| manifest-schema.yml | Documents additional commonly-used manifest fields and clarifies schema completeness expectations. |
| lectures/hansen_singleton_1983_data.csv.yml | Corrects the observed range for consumption_per_capita and completes the Ken French provenance quote. |
| lectures/hansen_singleton_1982_data.csv.yml | Completes the Ken French provenance quote. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Closed
40 tasks
Closed
41 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All four items from #85, each re-derived from primary evidence rather than taken from the issue text.
1. The 1983 manifest's lower bound was wrong.
consumption_per_capitawas described as "Observed 2.14e-4 to 2.66e-4". Re-derived from the committed CSV with the stdlibcsvmodule (no pandas, so the measurement does not depend on the thing this PR is pinning): the minimum is 2.1254904619e-4 at 1960-12-31, which rounds to 2.13e-4 at three significant figures, and the second-smallest value rounds there too — so 2.14e-4 matched no observed value. Both bounds now carry the date they occur on, which is cheaper to re-check than a bare number.2. The Ken French quote was truncated mid-sentence inside quotation marks. Re-fetched
F-F_Research_Data_Factors_CSV.zipon 2026-08-17: still the 202606 CRSP vintage, header unchanged. The full sentence is now quoted in both manifests, and each says plainly that the ICE BofA clause applies to no row in a sample ending 1978.One correction to the issue: it expected this quote in a builder docstring as well. It is not there. Both builders reference the zip but neither quotes its header (
grepforIbbotson|ICE BofA|CRSP databaseoverbuilders/exits 1, with a passing positive control onF-F_Research_Data_Factors), so the manifests were the only place to fix.3. pandas is now pinned. It is imported by 9 of the 10 builders — every one except
builders/business_cycle.py. Pinned at 2.3.3, the version both hansen builders were verified against on 2026-08-13, with a comment recording that they also reproduce byte-identically under 3.0.5. The pin documents the measurement rather than guarding a known sensitivity, which is what makes theintegrity.upstream.status: verifiedclaims reproducible against a recorded environment.4. The four undocumented fields are documented, and the header now says what the file actually is.
manifest-schema.ymldescribed itself as a strawman but read as a closed schema, which is what made "is this field allowed?" undecidable. It now states that it records the fields that recur, that a dataset needing a field not shown should add one rather than distort itself to fit, and that a field earning its place in several manifests gets documented back. It also names the two extensions already in wide use but described nowhere —schema.sheetsandschema.read_as— which were the subject of a separate open question inPLAN-QELD-PACKAGE.md.Verification
All 33 manifests and the schema parse under
yaml.safe_load.scripts/build_catalog.pyregeneratesCATALOG.mdunchanged, which also establishes that nothing currently consumes the four newly documented fields, so this cannot move the generated catalog.Not addressed here, and left where the issue put it: the Ken French developer-comment finding belongs to the licence review on #35, not to this PR.
Closes #85
🤖 Generated with Claude Code