Repository navigation
Conversation
…timestamp text Producers format step completion times with isoformat(), which drops the fraction on a whole second, so a later completion stamped ...:01.001000Z sorted before an earlier one stamped ...:01Z and compensation ran in completion order. Sort completed steps by the parsed instant.
…ring compensation The compensation sort used the strict parse_timestamp, which raises on a timestamp without an offset. The workflow history contract checks admit such a value and read it as UTC, so cancelling or timing out a workflow with an admitted offset-less completion raised instead of compensating. Sort with the parser those checks use (diagnostics._parse_timestamp). The regression test now seeds every instant in the past, so the history stays monotonic after the cancellation or timeout. It covers both compensation triggers and three admitted timestamp forms.
doublewhy
force-pushed
the
1344-compensation-order
branch
from
October 9, 2026 13:00
dc3c3ab to
e8ab47c
Compare
doublewhy
marked this pull request as ready for review
October 9, 2026 13:37
9 of 11 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
specs/formal/workflows/compensation.mdfixes compensation order toreverse_completion(line 28): registration follows actual completion order (line 52), and compensation runs in strict reverse completion order (line 63).maybe_apply_compensation(raes_runtime/control_plane_workflows.py:52ondev) sorted the completed steps by the text of theirstep_completedtimestamps. RAES does not emitstep_completeditself. The producers are LilRAE's workflow engine and external orchestrators. LilRAE stamps each event 1 ms after the previous one withdatetime.isoformat()(src/aptl/core/runtime/workflow_engine.py:363on LilRAEdev, via_next_timestamp). RAES formats its own events the same way (control_plane_execution._utc_now).isoformat()drops the fraction on a whole second, so a step completed at...:01.001000Zsorted before one completed at...:01Z, and compensation ran the steps in the order they had run. Text order also misorders completions stamped with different UTC offsets.This PR sorts by the parsed instant. It uses the parser that the workflow history contract checks already use (
raes_runtime/diagnostics._parse_timestamp, called fromworkflow_result_contract_checks/_context.py:170). Like those checks, it reads an offset-less timestamp as UTC, so compensation can order every history the checks admit. The #1344 fixture-provenance audit found the gap: the compensation tests had one compensable step, so nothing observed the order.Because this changes source that the research evidence binds, both captures are republished at the next free releases (specification coverage 69.0.0, formal validation 70.0.0). Retained outcomes, classifications and claim limits are unchanged. Other open pull requests carry the same release numbers, so whichever merges later regenerates its captures.
Requirement UIDs
tools/policy/requirement_order.yaml, so this PR does not activate it.Related Issues
Closes #1462
ADR Impact
Changes
implementations/python/packages/raes_runtime/control_plane_workflows.py:maybe_apply_compensationorders completed compensable steps by_parse_timestamp(event.timestamp)fromraes_runtime/diagnostics.py, newest first. A comment records why the text order is wrong. Steps with equal instants keep their history order, as before. The sort can raise only on a timestamp that parser cannot read, and the history contract checks reject such a timestamp before a backend history is admitted (backend_apply_results._backend_snapshot_contract_diagnostics).implementations/python/tests/test_issue_1462_compensation_order.py(new): a workflow with two compensable steps, compensated on cancellation and on timeout. It runs once per trigger (RuntimeControlPlane.cancel_workflowandreconcile_workflow_timeouts) and once per completion timestamp pair: the LilRAEisoformat()form 1 ms apart across a whole second, an offset-less pair, and a+02:00/Zpair. Every seeded instant is in 2000, so the cancellation or timeout that the runtime stamps afterwards keeps the history monotonic. The test checks that the seeded history passesworkflow_result_contract_diagnostics. After the trigger, it expects compensation for the later step to be registered first and the resulting history to pass the same checks.execution-snapshot-v69.json,analysis-v69.json, issue-1462 bundle) replays the retained matrix against this branch's source.execution-snapshot-v70.json,analysis-v70.json,retest-v70.json) replays the retained formal cases with baseline 69.0.0.tools/check_specification_coverage.py,tools/formal_semantic_validation/, and the three evidence test modules. Both research indexes record the new releases. The corpus-v4 release set in_baseline.pygains 69.0.0, the baseline, which feat: carry authored execution and recovery requirements #1413 published without adding to the set. It also gains 70.0.0, as fix: republish research evidence for the merged control-plane boundary source #1404, fix(deps): upgrade pyjwt to 2.15.1 and urllib3 to 2.8.0 for published advisories #1405, fix: reject unprovable required evidence media types #1403 and chore(deps-dev): bump hatchling from 1.32.0 to 1.32.4 in /implementations/python in the python-minor-patch group across 1 directory #1392 each added the release they published (65.0.0 to 68.0.0). The set lists every corpus-v4 release, including 46.0.0 to 48.0.0 and 60.0.0, which never served as a baseline.Test Plan
uv run --project implementations/python --frozen --all-extras python -m pyteston the new module and the runtime modules that exercise compensation and workflow timeouts: 284 passed. The modules aretest_runtime_control_plane_api.py,test_runtime_workflow_timeout_reconciliation.py,test_runtime_control_plane.py,test_runtime_manager.py,test_issue_1348_operation_supervision.py,test_issue_1361_execution_policy.pyandtest_runtime_result_envelope_properties.py. The same selection with-m integration: 284 deselected (none are integration-marked).With
control_plane_workflows.pyrestored toorigin/dev, the new module fails 4 of 6 cases: the whole-second and explicit-offset pairs under both triggers failassert ['a', 'b'] == ['b', 'a']. With the strictparse_timestampsort of this PR's first revision, the offset-less pair fails under both triggers withValueError: timestamp must be an ISO-8601 value with an explicit UTC offset. With this change, all 6 pass.Both evidence checker CLIs pass (
tools/check_specification_coverage.pyandtools/check_formal_semantic_validation.py). The three evidence modules pass with default markers (209 passed, 19 deselected) and with-m integration(19 passed, 209 deselected). The regenerated captures differ from this PR's previous captures only incaptured_at, the source revision and the implementation digest. There are no formal deviations (execution-snapshot-v70.jsonrecords"deviations": []) and no changed artifact pins.nox -s verify-fast-feedback -- --base-rev origin/devpassed. Ruff format and check are clean fromimplementations/python.No coverage regression: the SonarCloud PR quality gate passed on head
e8ab47c6(analysis 2026-10-09T13:16:53Z) with new-code coverage 100.0% (7 new lines to cover, 0 uncovered), 0 new issues and 0.0% duplication on new lines. CI run 37933866019 passed every job on this head, includingcanonical / coverage-reduce, the four test shards,canonical / integrationandcanonical / checks(which runs both evidence steps). In total 32 checks passed, and onlydeploywas skipped.Ground Control Checks
make policypassed; requirement governance was skipped automatically because no requirement UID or scope file resolves from this branch.nox -s lintpassed, and so didnox -s contracts -- --base-rev origin/dev(14 of 14 steps, including both evidence steps). The regression evidence is in the Test Plan above.Traceability
implementations/python/packages/raes_runtime/control_plane_workflows.pyimplementations/python/tests/test_issue_1462_compensation_order.pyChecklist
specs/formal/workflows/compensation.mdfor every timestamp form the workflow history contract checks admit. No executable or runtime adoption is claimed.CHANGELOG.mdfrom it)