Add machine-readable paper card checks - #4
Merged
Conversation
There was a problem hiding this comment.
🟡 Not ready to approve
The new verifier has path/robustness issues (cwd-relative resolution and exception behavior) and a couple of correctness/efficiency/documentation mismatches that should be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds a lightweight “evidence contract” layer so replication work has both human-readable cards and machine-readable paper metadata that can be statically verified.
Changes:
- Introduces a paper card schema (
schemas/paper_card_schema.json) plus an initial machine-readable card (paper_cards/autoalpha.json). - Extends
scripts/verify_cards.pyto validate both replication cards and paper metadata cards, and adds a unit test for the new contract. - Updates README to document the evidence contract layout and how to run verification.
File summaries
| File | Description |
|---|---|
tests/test_cards.py |
Adds a regression test asserting the paper metadata card validates cleanly. |
scripts/verify_cards.py |
Adds paper-card schema validation and extends the CLI verifier to check paper_cards/*.json. |
schemas/paper_card_schema.json |
Defines required fields and allowed enumerations for paper metadata cards. |
README.md |
Documents the new evidence contract layout and verification commands. |
paper_cards/autoalpha.json |
Adds the first machine-readable paper metadata card pointing at the replication card as evidence. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 3
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Comment on lines
+20
to
+36
| def validate_paper_card(card: Path, schema: Path = Path("schemas/paper_card_schema.json")) -> list[str]: | ||
| payload = json.loads(card.read_text(encoding="utf-8")) | ||
| rules = json.loads(schema.read_text(encoding="utf-8")) | ||
| errors: list[str] = [] | ||
| for key in rules["required"]: | ||
| if key not in payload: | ||
| errors.append(f"missing key: {key}") | ||
| if payload.get("implementation_status") not in rules["allowed_status"]: | ||
| errors.append("invalid implementation_status") | ||
| if payload.get("replication_status") not in rules["allowed_status"]: | ||
| errors.append("invalid replication_status") | ||
| if payload.get("claim_ceiling") not in rules["allowed_claim_ceiling"]: | ||
| errors.append("invalid claim_ceiling") | ||
| for evidence in payload.get("evidence_files", []): | ||
| if not Path(evidence).exists(): | ||
| errors.append(f"missing evidence file: {evidence}") | ||
| return errors |
Comment on lines
43
to
+47
| failures = {str(card): validate(card) for card in cards if validate(card)} | ||
| paper_cards = sorted(Path("paper_cards").glob("*.json")) | ||
| if not paper_cards: | ||
| raise SystemExit("no machine-readable paper cards found") | ||
| paper_failures = {str(card): validate_paper_card(card) for card in paper_cards if validate_paper_card(card)} |
| ``` | ||
|
|
||
| The tests check that each card has source, scope, deviation, and claim-boundary sections. They do not validate trading performance. | ||
| The checks validate that each public card has source, scope, deviation, and claim-boundary sections. They also validate that each machine-readable paper card links to existing evidence. They do not validate trading performance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a lightweight evidence-contract layer:
Local verification:
C:\qds\Scripts\python.exe -m unittest discover -s tests -vandC:\qds\Scripts\python.exe scripts\verify_cards.pypassed.