Skip to content

fix(contracts): report unquoted invalid dates in experiment specs as invalid yaml - #1597

Open
doublewhy wants to merge 2 commits into
devfrom
1563-experiment-spec-invalid-date
Open

doublewhy wants to merge 2 commits into
devfrom
1563-experiment-spec-invalid-date

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

parse_experiment_spec and load_experiment_spec let a bare ValueError escape for an unquoted invalid date. The experiment loader is a PyYAML SafeLoader (YAML 1.1), so description: 2026-02-30 resolves to !!timestamp and construct_yaml_timestamp raises ValueError: day is out of range for month unhandled; likewise for 2026-13-01, 0000-01-01, an out-of-range hour or offset, and an invalid date as a nested mapping key. The MCP experiment_validate tool, which catches only ExperimentSpecValidationError, raised a traceback too.

The loader now maps a ValueError raised while reading the YAML to the existing invalid-yaml ExperimentSpecValidationError, with every YAML failure's fixed message ("Experiment spec YAML is invalid."). The rejected value and Python's message never appear in details; the original error stays on __cause__. Valid unquoted dates are accepted and typed exactly as before.

Requirement UIDs

Related Issues

Closes #1563

ADR Impact

  • No ADR required; no ADR text changes. The fix stays inside ADR-074's thin Pydantic loader (section 4), does not echo rejected values (issue-274 preflight, item 2), and adds no exception, diagnostic type or code (issue-164 and issue-652 preflights). The YAML 1.2 Core profile (issue-721 preflight) is SDL-only.

Sources read: YAML 1.1 timestamp type (https://yaml.org/type/timestamp.html; its regexp accepts impossible dates), YAML 1.2.2 section 10.3 (https://yaml.org/spec/1.2.2/; no timestamp tag in the Core schema, so SDL is unaffected), PyYAML 6.0.3 constructor.py (pinned).

Changes

  • raes_contracts/experiment_spec.py:parse_experiment_spec: the yaml.YAMLError arm around _load_yaml becomes except (yaml.YAMLError, ValueError) as exc: (same code, message and path, chained from exc). Neither PyYAML error subclasses ValueError, so no existing mapping changes; load_experiment_spec delegates and gets its "<file>: " prefix. Design deviations, recorded in the design note: one widened arm instead of two identical arms; content-addressed captures (since feat(contracts): publish inject readback, correlation and backend declaration contracts #1449) instead of numbered republishing.
  • Research captures, collected on a clean checkout of this branch (dev 348c490 plus the fix). The earlier pair, copied from dev's capture, is dropped. Specification coverage: all 6 pinned artifacts ran through tools.specification_coverage._artifacts._execute_artifact (42 stage results: 38 passed, 4 preregistered non-execution), then recompute_analysis (10 of 10 load-bearing passed). Formal validation: the 10 retained SDL cases replayed with replay_case, the 4 production cases ran with their raes processor satisfiability / exploit-path argv, and the protocol's 14 participant test functions ran with pytest (18 passed); 20 of 20 observations equal their expected outcome. Recorded with python -m tools.research_capture record against dev's newest manifests: specification capture-961ac2d4... (template capture-c9ca186a...) and formal capture-68f6095a... (template capture-3bad3945...), each with its capture-records receipt. Drift from the templates: the implementation-surface digest (this fix), the formal retained-case-replay argv and versions.python (3.12.3). No outcome changed.

Done-when matrix (tests in test_issue_1563_experiment_spec_invalid_date.py, CI lane canonical / test-shard):

Done-when (verbatim) Change Test; result without the fix Evidence
parse_experiment_spec and load_experiment_spec raise ExperimentSpecValidationError for an unquoted invalid date such as 2026-02-30. experiment_spec.py:parse_experiment_spec test_parse_rejects_unquoted_invalid_timestamp_as_invalid_yaml (6 cases), test_load_rejects_unquoted_invalid_date_with_path, test_mcp_validate_reports_invalid_date_as_validation_error: all 8 fail with a bare ValueError 9 passed locally; CI run 38482420601
A regression test fails without the change and passes with it. same the 8 above; the control test_valid_unquoted_date_still_reaches_contract_validation passes either way stash run
tools/check_specification_coverage.py and tools/check_formal_semantic_validation.py pass. the two captures both tools exit 0 at the head and on the git merge-tree --write-tree origin/dev HEAD tree

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Completion command passes (canonical / gate passed)
  • No coverage regression (canonical / coverage-reduce passed)

uv run --project implementations/python --frozen --all-extras python -m pytest implementations/python/tests/test_issue_1563_experiment_spec_invalid_date.py -q -p no:cacheprovider: 9 passed. Parse cases assert code invalid-yaml and a ValueError cause, the load case path, the MCP case VALIDATION ERROR; all assert the fixed details and no rejected value. The control (2026-02-28) still fails with contract-invalid.

Fail-without-change: with git stash push implementations/python/packages/raes_contracts/experiment_spec.py (restored afterwards), 8 fail and the control passes. Every failure is a bare ValueError from the datetime or timedelta constructors (for example day is out of range for month, year 0 is out of range).

test_experiment_authoring.py, test_example_schema_conformance.py and the new module: 86 passed (no integration-marked tests). tools/check_example_library.py passed. The five evidence modules (test_formal_semantic_validation*.py, test_specification_coverage*.py, test_research_capture.py): 173 passed, and 20 with -m integration. nox -s contracts and nox -s verify-fast-feedback (both -- --base-rev origin/dev), nox -s lint, make policy and ruff (from implementations/python) passed.

CI (checked for this edit): head e4f577b on dev 348c490; run 38482420601 passed every job after one rerun of generic-tool-local-inputs, whose Isabelle mirror download failed its exact payload verification; the same commit passed on rerun; deploy skipped.

Ground Control Checks

  • Repository policy command passes

The fail-without-change run is the test-quality evidence; no Ground Control pre-push review was used for this lane.

Traceability

  • IMPLEMENTS: implementations/python/packages/raes_contracts/experiment_spec.py
  • TESTS: implementations/python/tests/test_issue_1563_experiment_spec_invalid_date.py

Checklist

  • Code follows the project coding standards
  • FM0: parser-level error mapping; parser unit tests are the required artifact. No executable or runtime adoption is claimed.
  • No model or schema changed.
  • PR title is a Conventional Commit (fix, patch release).

Documentation

Verified unchanged: the error's docstring already covers a specification that "failed to parse or validate"; no doc describes the loader's exception types.

@doublewhy
doublewhy marked this pull request as ready for review October 13, 2026 11:13
@doublewhy
doublewhy marked this pull request as draft October 13, 2026 19:39
…invalid yaml

PyYAML's SafeLoader resolves an unquoted 2026-02-30 to a YAML 1.1 timestamp and lets datetime's ValueError escape from parse_experiment_spec and load_experiment_spec. Map it to the existing invalid-yaml ExperimentSpecValidationError with the fixed message, so callers such as the MCP experiment_validate tool report a validation error instead of a traceback.

Closes #1563
Collected on this branch's source (dev 348c490 plus the fix) with the shared collector: 6 pinned artifacts through the production pipeline (42 stage results), 10 retained SDL cases replayed with replay_case, 4 production cases run with their raes processor argv, and 14 participant test functions (18 passed). Recorded with python -m tools.research_capture record against dev's newest manifests.
@doublewhy
doublewhy force-pushed the 1563-experiment-spec-invalid-date branch from cfa1f71 to e4f577b Compare October 13, 2026 23:08
@doublewhy
doublewhy marked this pull request as ready for review October 14, 2026 00:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant