Skip to content

fix(conformance): validate the published snapshot carriers in target conformance - #1599

Open
doublewhy wants to merge 2 commits into
devfrom
1551-target-snapshot-published-carriers
Open

doublewhy wants to merge 2 commits into
devfrom
1551-target-snapshot-published-carriers

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Target conformance's target-snapshot case validated a hand-built payload that omitted realization_envelope, realization_provenance and the other carriers the /snapshot route publishes, and skipped its credential redaction (RS-1 of the #1344 fixture-provenance audit). The case now validates the payload the route's own serializer publishes, with semantic checks only after the schema check passes, so an invalid published envelope or provenance entry fails target-snapshot with the existing conformance.schema-invalid diagnostic. Report shape and wire output are unchanged.

Requirement UIDs

Related Issues

Closes #1551

Context, not closed here: #1344, #1461 (open audit record), #1553 (RS-2) and #1555 (RS-3).

ADR Impact

Changes

  • raes_runtime/control_plane_api_models.py:published_snapshot_payload (new, public) holds _snapshot_model's former body, redaction included; _snapshot_model validates its result, so the route is unchanged. tools/policy/adr_policy.yaml adds raes_runtime.control_plane_api_models to raes_conformance's public imports. control_plane.py is unchanged (see design deviations).
  • raes_conformance/conformance/target_probes.py:_hermetic_snapshot_case validates published_snapshot_payload(control_plane.get_snapshot()), runs semantic checks only on a schema-valid payload (as semantics.py::_fixture_case_diagnostics does), and the hand-built _hermetic_snapshot_payload is deleted.

Done-when matrix (bullets verbatim; tests in test_issue_1551_target_snapshot_published_carriers.py, selected by canonical / test-shard (0-3); evidence: CI run 38482282315):

Done-when Change Test, and its result with the two source files reverted to dev
A conformance test fails when the published snapshot's realization_envelope or realization_provenance is invalid. target_probes.py:_hermetic_snapshot_case, control_plane_api_models.py:published_snapshot_payload test_target_snapshot_rejects_invalid_published_carrier[envelope], [provenance] fail (target-snapshot.passed is True); test_target_snapshot_validates_the_route_payload fails (the case never reads the published snapshot).
If #1461 has merged, the RS-1 entry in docs/research/fixture-provenance/index.md and its row in inventory.md record the fix. Does not apply: #1461 is open (checked 2026-10-13). -

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Completion command passes
  • No coverage regression

pytest implementations/python/tests/test_issue_1551_target_snapshot_published_carriers.py passed (3 passed).

With the two source files reverted to dev, all 3 tests fail (matrix); without the schema-first order, both carrier cases raise pydantic_core.ValidationError from the semantic step.

Regression: the 81 modules referencing run_target_conformance, _snapshot_model, target_probes or raes_runtime.control_plane gave 2085 passed, 2 skipped. RAES_DOCKER_INTEGRATION_REQUIRED=1 pytest -m docker test_reference_backend_docker_integration.py: 2 passed, with the real-container backend's target-snapshot on the published payload.

Evidence, collected on 04d003e (dev 348c490 plus this fix, clean tree): the 6 pinned specification artifacts ran through _execute_artifact (42 stages: 38 passed, 4 preregistered non-execution) and recompute_analysis (10 of 10 load-bearing passed); replay_case ran the 10 retained SDL cases, raes processor the 4 production cases and pytest the 14 participant functions (all 20 observations as expected). research_capture record wrote capture-908b9484... and capture-15e1b6e9... (templates c9ca186a, 3bad3945). Drift: only the formal first command (retained-case-replay, as run) and Python 3.12.3. Both checkers pass at the head and on the merge-tree result; the five evidence test modules gave 173 passed; nox -s contracts passed.

nox -s verify-fast-feedback -- --base-rev origin/dev, nox -s lint and nox -s docs-local passed; the full completion suite is required in CI before merge. CI at head 26613f0: 32 checks pass, Docs deploy skipped (run 38482282315).

Ground Control Checks

  • Repository policy command passes
  • Pre-push code review and test-quality review completed, or not run for this lane

make policy passed, including the module-boundary check for the new public import.

Traceability

  • IMPLEMENTS: implementations/python/packages/raes_runtime/control_plane_api_models.py, implementations/python/packages/raes_conformance/conformance/target_probes.py
  • TESTS: implementations/python/tests/test_issue_1551_target_snapshot_published_carriers.py

Checklist

  • Code follows the project coding standards
  • FM level classified if semantic change
  • Published contract schemas regenerated if models changed
  • PR title is a Conventional Commit (release-please derives the version and CHANGELOG.md from it)
  • Architectural docs updated if applicable

FM0 (structural: an existing published payload validated against its existing schema). No executable or runtime adoption is claimed. No contract model or schema changed. Design deviations: content-addressed captures (since #1449); a policy public import instead of a control_plane.py re-export (#1581's multi-rev1 crossing bundle pins its digest).

Documentation

Updated: docs/explain/reference/backend-conformance.md ("Target Conformance Reference Scenario").

@doublewhy
doublewhy force-pushed the 1551-target-snapshot-published-carriers branch from 4186b7b to 0bb2a44 Compare October 13, 2026 12:14
@doublewhy
doublewhy marked this pull request as ready for review October 13, 2026 12:38
@doublewhy
doublewhy marked this pull request as draft October 13, 2026 19:39
@doublewhy
doublewhy force-pushed the 1551-target-snapshot-published-carriers branch from 0bb2a44 to b107f8d Compare October 13, 2026 22:34
…conformance

The target-snapshot case validated a hand-built payload that dropped
realization_envelope, realization_provenance and the other carriers the
/snapshot route publishes. It now validates published_snapshot_payload(),
the route's own serializer (a public import of raes_runtime.control_plane_api_models
for raes_conformance in adr_policy.yaml; control_plane.py stays unchanged because
the #1581 multi-rev1 crossing bundle pins its digest),
and runs semantic checks only on a schema-valid payload.

Closes #1551
Collected on 04d003e (origin/dev 348c490 plus the #1551 fix) with the production artifact pipeline, replay_case on the 10 retained SDL cases, the 4 production processor cases and the 14 participant test functions, then recorded with tools.research_capture record.
@doublewhy
doublewhy force-pushed the 1551-target-snapshot-published-carriers branch from b107f8d to 26613f0 Compare October 13, 2026 23:07
@doublewhy
doublewhy marked this pull request as ready for review October 14, 2026 03:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant