Skip to content

fix(runtime): order workflow compensation by completion instant, not timestamp text - #1464

Open
doublewhy wants to merge 3 commits into
devfrom
1344-compensation-order
Open

doublewhy wants to merge 3 commits into
devfrom
1344-compensation-order

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

specs/formal/workflows/compensation.md fixes compensation order to reverse_completion (line 28): registration follows actual completion order (line 52), and compensation runs in strict reverse completion order (line 63). maybe_apply_compensation (raes_runtime/control_plane_workflows.py:52 on dev) sorted the completed steps by the text of their step_completed timestamps. RAES does not emit step_completed itself. The producers are LilRAE's workflow engine and external orchestrators. LilRAE stamps each event 1 ms after the previous one with datetime.isoformat() (src/aptl/core/runtime/workflow_engine.py:363 on LilRAE dev, via _next_timestamp). RAES formats its own events the same way (control_plane_execution._utc_now). isoformat() drops the fraction on a whole second, so a step completed at ...:01.001000Z sorted before one completed at ...:01Z, and compensation ran the steps in the order they had run. Text order also misorders completions stamped with different UTC offsets.

This PR sorts by the parsed instant. It uses the parser that the workflow history contract checks already use (raes_runtime/diagnostics._parse_timestamp, called from workflow_result_contract_checks/_context.py:170). Like those checks, it reads an offset-less timestamp as UTC, so compensation can order every history the checks admit. The #1344 fixture-provenance audit found the gap: the compensation tests had one compensable step, so nothing observed the order.

Because this changes source that the research evidence binds, both captures are republished at the next free releases (specification coverage 69.0.0, formal validation 70.0.0). Retained outcomes, classifications and claim limits are unchanged. Other open pull requests carry the same release numbers, so whichever merges later regenerates its captures.

Requirement UIDs

  • None. SEM-204 governs compensation ordering but is not phase-mapped in tools/policy/requirement_order.yaml, so this PR does not activate it.

Related Issues

Closes #1462

ADR Impact

  • None.

Changes

  • implementations/python/packages/raes_runtime/control_plane_workflows.py: maybe_apply_compensation orders completed compensable steps by _parse_timestamp(event.timestamp) from raes_runtime/diagnostics.py, newest first. A comment records why the text order is wrong. Steps with equal instants keep their history order, as before. The sort can raise only on a timestamp that parser cannot read, and the history contract checks reject such a timestamp before a backend history is admitted (backend_apply_results._backend_snapshot_contract_diagnostics).
  • implementations/python/tests/test_issue_1462_compensation_order.py (new): a workflow with two compensable steps, compensated on cancellation and on timeout. It runs once per trigger (RuntimeControlPlane.cancel_workflow and reconcile_workflow_timeouts) and once per completion timestamp pair: the LilRAE isoformat() form 1 ms apart across a whole second, an offset-less pair, and a +02:00/Z pair. Every seeded instant is in 2000, so the cancellation or timeout that the runtime stamps afterwards keeps the history monotonic. The test checks that the seeded history passes workflow_result_contract_diagnostics. After the trigger, it expects compensation for the later step to be registered first and the resulting history to pass the same checks.
  • Specification-coverage release 69.0.0 (execution-snapshot-v69.json, analysis-v69.json, issue-1462 bundle) replays the retained matrix against this branch's source.
  • Formal-validation release 70.0.0 (execution-snapshot-v70.json, analysis-v70.json, retest-v70.json) replays the retained formal cases with baseline 69.0.0.
  • Revision pins advanced in tools/check_specification_coverage.py, tools/formal_semantic_validation/, and the three evidence test modules. Both research indexes record the new releases. The corpus-v4 release set in _baseline.py gains 69.0.0, the baseline, which feat: carry authored execution and recovery requirements #1413 published without adding to the set. It also gains 70.0.0, as fix: republish research evidence for the merged control-plane boundary source #1404, fix(deps): upgrade pyjwt to 2.15.1 and urllib3 to 2.8.0 for published advisories #1405, fix: reject unprovable required evidence media types #1403 and chore(deps-dev): bump hatchling from 1.32.0 to 1.32.4 in /implementations/python in the python-minor-patch group across 1 directory #1392 each added the release they published (65.0.0 to 68.0.0). The set lists every corpus-v4 release, including 46.0.0 to 48.0.0 and 60.0.0, which never served as a baseline.

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Full completion suite required in CI before merge
  • No coverage regression

uv run --project implementations/python --frozen --all-extras python -m pytest on the new module and the runtime modules that exercise compensation and workflow timeouts: 284 passed. The modules are test_runtime_control_plane_api.py, test_runtime_workflow_timeout_reconciliation.py, test_runtime_control_plane.py, test_runtime_manager.py, test_issue_1348_operation_supervision.py, test_issue_1361_execution_policy.py and test_runtime_result_envelope_properties.py. The same selection with -m integration: 284 deselected (none are integration-marked).

With control_plane_workflows.py restored to origin/dev, the new module fails 4 of 6 cases: the whole-second and explicit-offset pairs under both triggers fail assert ['a', 'b'] == ['b', 'a']. With the strict parse_timestamp sort of this PR's first revision, the offset-less pair fails under both triggers with ValueError: timestamp must be an ISO-8601 value with an explicit UTC offset. With this change, all 6 pass.

Both evidence checker CLIs pass (tools/check_specification_coverage.py and tools/check_formal_semantic_validation.py). The three evidence modules pass with default markers (209 passed, 19 deselected) and with -m integration (19 passed, 209 deselected). The regenerated captures differ from this PR's previous captures only in captured_at, the source revision and the implementation digest. There are no formal deviations (execution-snapshot-v70.json records "deviations": []) and no changed artifact pins.

nox -s verify-fast-feedback -- --base-rev origin/dev passed. Ruff format and check are clean from implementations/python.

No coverage regression: the SonarCloud PR quality gate passed on head e8ab47c6 (analysis 2026-10-09T13:16:53Z) with new-code coverage 100.0% (7 new lines to cover, 0 uncovered), 0 new issues and 0.0% duplication on new lines. CI run 37933866019 passed every job on this head, including canonical / coverage-reduce, the four test shards, canonical / integration and canonical / checks (which runs both evidence steps). In total 32 checks passed, and only deploy was skipped.

Ground Control Checks

  • Repository policy command passes
  • Regression test fails on the pre-fix code and passes after

make policy passed; requirement governance was skipped automatically because no requirement UID or scope file resolves from this branch. nox -s lint passed, and so did nox -s contracts -- --base-rev origin/dev (14 of 14 steps, including both evidence steps). The regression evidence is in the Test Plan above.

Traceability

Checklist

  • Code follows the project coding standards
  • FM level classified if semantic change: FM not applicable. No semantic rule, state machine or contract changes; compensation now follows the reverse-completion rule in specs/formal/workflows/compensation.md for every timestamp form the workflow history contract checks admit. No executable or runtime adoption is claimed.
  • Published contract schemas regenerated if models changed: no model or schema changes.
  • PR title is a Conventional Commit (release-please derives the version and CHANGELOG.md from it)
  • No release gate weakened.

…timestamp text

Producers format step completion times with isoformat(), which drops the
fraction on a whole second, so a later completion stamped
...:01.001000Z sorted before an earlier one stamped ...:01Z and
compensation ran in completion order. Sort completed steps by the parsed
instant.
…ring compensation

The compensation sort used the strict parse_timestamp, which raises on a
timestamp without an offset. The workflow history contract checks admit
such a value and read it as UTC, so cancelling or timing out a workflow
with an admitted offset-less completion raised instead of compensating.
Sort with the parser those checks use (diagnostics._parse_timestamp).

The regression test now seeds every instant in the past, so the history
stays monotonic after the cancellation or timeout. It covers both
compensation triggers and three admitted timestamp forms.
@doublewhy
doublewhy force-pushed the 1344-compensation-order branch from dc3c3ab to e8ab47c Compare October 9, 2026 13:00
@doublewhy
doublewhy marked this pull request as ready for review October 9, 2026 13:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant