Repository navigation
Conversation
doublewhy
marked this pull request as ready for review
October 13, 2026 11:13
doublewhy
marked this pull request as draft
October 13, 2026 19:39
…invalid yaml PyYAML's SafeLoader resolves an unquoted 2026-02-30 to a YAML 1.1 timestamp and lets datetime's ValueError escape from parse_experiment_spec and load_experiment_spec. Map it to the existing invalid-yaml ExperimentSpecValidationError with the fixed message, so callers such as the MCP experiment_validate tool report a validation error instead of a traceback. Closes #1563
Collected on this branch's source (dev 348c490 plus the fix) with the shared collector: 6 pinned artifacts through the production pipeline (42 stage results), 10 retained SDL cases replayed with replay_case, 4 production cases run with their raes processor argv, and 14 participant test functions (18 passed). Recorded with python -m tools.research_capture record against dev's newest manifests.
doublewhy
force-pushed
the
1563-experiment-spec-invalid-date
branch
from
October 13, 2026 23:08
cfa1f71 to
e4f577b
Compare
doublewhy
marked this pull request as ready for review
October 14, 2026 00:08
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parse_experiment_specandload_experiment_speclet a bareValueErrorescape for an unquoted invalid date. The experiment loader is a PyYAMLSafeLoader(YAML 1.1), sodescription: 2026-02-30resolves to!!timestampandconstruct_yaml_timestampraisesValueError: day is out of range for monthunhandled; likewise for2026-13-01,0000-01-01, an out-of-range hour or offset, and an invalid date as a nested mapping key. The MCPexperiment_validatetool, which catches onlyExperimentSpecValidationError, raised a traceback too.The loader now maps a
ValueErrorraised while reading the YAML to the existinginvalid-yamlExperimentSpecValidationError, with every YAML failure's fixed message ("Experiment spec YAML is invalid."). The rejected value and Python's message never appear indetails; the original error stays on__cause__. Valid unquoted dates are accepted and typed exactly as before.Requirement UIDs
## Requirementssection and no requirement scope file.Related Issues
Closes #1563
ADR Impact
Sources read: YAML 1.1 timestamp type (https://yaml.org/type/timestamp.html; its regexp accepts impossible dates), YAML 1.2.2 section 10.3 (https://yaml.org/spec/1.2.2/; no timestamp tag in the Core schema, so SDL is unaffected), PyYAML 6.0.3
constructor.py(pinned).Changes
raes_contracts/experiment_spec.py:parse_experiment_spec: theyaml.YAMLErrorarm around_load_yamlbecomesexcept (yaml.YAMLError, ValueError) as exc:(same code, message andpath, chainedfrom exc). Neither PyYAML error subclassesValueError, so no existing mapping changes;load_experiment_specdelegates and gets its"<file>: "prefix. Design deviations, recorded in the design note: one widened arm instead of two identical arms; content-addressed captures (since feat(contracts): publish inject readback, correlation and backend declaration contracts #1449) instead of numbered republishing.tools.specification_coverage._artifacts._execute_artifact(42 stage results: 38 passed, 4 preregistered non-execution), thenrecompute_analysis(10 of 10 load-bearing passed). Formal validation: the 10 retained SDL cases replayed withreplay_case, the 4 production cases ran with theirraes processor satisfiability/exploit-pathargv, and the protocol's 14 participant test functions ran with pytest (18 passed); 20 of 20 observations equal their expected outcome. Recorded withpython -m tools.research_capture recordagainst dev's newest manifests: specificationcapture-961ac2d4...(templatecapture-c9ca186a...) and formalcapture-68f6095a...(templatecapture-3bad3945...), each with itscapture-recordsreceipt. Drift from the templates: the implementation-surface digest (this fix), the formalretained-case-replayargv andversions.python(3.12.3). No outcome changed.Done-when matrix (tests in
test_issue_1563_experiment_spec_invalid_date.py, CI lanecanonical / test-shard):parse_experiment_specandload_experiment_specraiseExperimentSpecValidationErrorfor an unquoted invalid date such as2026-02-30.experiment_spec.py:parse_experiment_spectest_parse_rejects_unquoted_invalid_timestamp_as_invalid_yaml(6 cases),test_load_rejects_unquoted_invalid_date_with_path,test_mcp_validate_reports_invalid_date_as_validation_error: all 8 fail with a bareValueErrortest_valid_unquoted_date_still_reaches_contract_validationpasses either waytools/check_specification_coverage.pyandtools/check_formal_semantic_validation.pypass.git merge-tree --write-tree origin/dev HEADtreeTest Plan
canonical / gatepassed)canonical / coverage-reducepassed)uv run --project implementations/python --frozen --all-extras python -m pytest implementations/python/tests/test_issue_1563_experiment_spec_invalid_date.py -q -p no:cacheprovider: 9 passed. Parse cases assert codeinvalid-yamland aValueErrorcause, the load casepath, the MCP caseVALIDATION ERROR; all assert the fixed details and no rejected value. The control (2026-02-28) still fails withcontract-invalid.Fail-without-change: with
git stash push implementations/python/packages/raes_contracts/experiment_spec.py(restored afterwards), 8 fail and the control passes. Every failure is a bareValueErrorfrom thedatetimeortimedeltaconstructors (for exampleday is out of range for month,year 0 is out of range).test_experiment_authoring.py,test_example_schema_conformance.pyand the new module: 86 passed (no integration-marked tests).tools/check_example_library.pypassed. The five evidence modules (test_formal_semantic_validation*.py,test_specification_coverage*.py,test_research_capture.py): 173 passed, and 20 with-m integration.nox -s contractsandnox -s verify-fast-feedback(both-- --base-rev origin/dev),nox -s lint,make policyand ruff (fromimplementations/python) passed.CI (checked for this edit): head e4f577b on
dev348c490; run 38482420601 passed every job after one rerun ofgeneric-tool-local-inputs, whose Isabelle mirror download failed its exact payload verification; the same commit passed on rerun;deployskipped.Ground Control Checks
The fail-without-change run is the test-quality evidence; no Ground Control pre-push review was used for this lane.
Traceability
implementations/python/packages/raes_contracts/experiment_spec.pyimplementations/python/tests/test_issue_1563_experiment_spec_invalid_date.pyChecklist
fix, patch release).Documentation
Verified unchanged: the error's docstring already covers a specification that "failed to parse or validate"; no doc describes the loader's exception types.