Skip to content

fix(lab): close the first runtime manager before the start retry - #1427

Closed
doublewhy wants to merge 3 commits into
devfrom
1424-close-runtime-owner-before-retry
Closed

doublewhy wants to merge 3 commits into
devfrom
1424-close-runtime-owner-before-retry

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When the first backend apply fails with aptl.provisioner.backend-start-failed and the startup adapter admits BEFORE_BACKEND_RETRY (the bundled TechVault does), lilrae lab start waits 60 s and crashes with RuntimeError: backend already has a runtime owner. --json prints nothing. raes 7.0.0 lets one RuntimeManager own a target's components until it is closed, and the retry built a second manager while the first attempt's was still open.

The retry now closes the first attempt's manager before the hook runs, then applies the same admitted plan. If RAES refuses the close, the retry is skipped and the result keeps the first error with the reason. If the retry also fails, its error ends with Initial attempt: and the first attempt's diagnostics. The retryable set does not change.

Requirement UIDs

  • None (bug fix in the start handoff; no requirement statement changes).

Related Issues

Closes #1424

Shares test_raes_backend.py and runtime-architecture.md with #1369 (#1325) in disjoint hunks; either merge order is clean, and #1369's retry test relies on this PR's _FakeRuntimeManager.close().

ADR Impact

None. Start still applies through RAES's own RuntimeManager.apply() (ADR-046, runtime service addendum).

Changes

  • src/aptl/backends/_raes_start_retry.py (new, 79 lines): apply_with_single_retry. It catches only what RAES close() raises (RuntimeError, TimeoutError).
  • src/aptl/backends/raes.py: _apply_with_backend_retry keeps its signature and delegates (499 → 493 lines).
  • tests/test_raes_start_retry.py (new, 7 tests): the deployment backend and the retry hook are modelled. The runtime manager, its RAES claim, and close() are real; the skip test subclasses the real manager only to leave a clock driver that does not stop. The CLI test replaces only the lab steps around the handoff. Owner checks expect a second AptlRuntimeManager(target) to raise "already has a runtime owner".
  • tests/test_raes_backend.py: the shared _FakeRuntimeManager gets a no-op close(), which the existing retry tests now call.
  • docs/sdl/runtime-architecture.md: the retry paragraph names the admitting hook (not the soc profile), the close, the skip and the suffix.
  • docs/reviews/962-lilrae-readiness/tracked-file-inventory.tsv: rows for the two new files.
Done when Change Test; result without the change
Clean retry: after a failed first apply, a successful retry starts the lab and its manager stays open. apply_with_single_retry, _release_runtime_owner test_retry_releases_the_first_runtime_owner_and_starts: RuntimeError: backend already has a runtime owner
Clean failure: a failed retry gives --json ok=false with both attempts' diagnostics. apply_with_single_retry (Initial attempt:) test_failed_retry_reports_both_attempts: same RuntimeError; test_lab_start_json_reports_both_attempts_after_a_failed_retry: JSONDecodeError on empty stdout
Clean failure: a first owner that cannot be released skips the retry and keeps the first diagnostics. _release_runtime_owner, _RETRY_SKIPPED test_unreleasable_first_owner_skips_the_retry (RAES's TimeoutError from the real close()): same RuntimeError
No close without a retry: first-attempt success, a non-retryable failure and a missing hook keep the first owner. early return in apply_with_single_retry test_first_attempt_without_retry_keeps_its_owner (3 cases): pass without the change; fail under a close-on-every-path mutation

All run in the "Python tests + coverage" lane.

Test Plan

  • bash tools/run-targeted-tests.sh tests/test_raes_start_retry.py: 7 passed. Without the change: 4 failed (three RuntimeError, one empty --json), 3 passed (no-retry cases).
  • Mutations: closing on every path fails the 3 no-retry cases (DID NOT RAISE RuntimeError); closing the retry's manager fails the retry-success test.
  • bash tools/run-targeted-tests.sh (both touched test files): 128 passed. pytest -k "not integration" on the 25 modules that reference aptl.backends.raes, test_cli.py and test_raes_participant_delivery.py: 1004 passed; test_lilrae_rename_boundary.py: 5 passed.
  • pre-commit run --files <changed files> (including the pack boundary), ruff check, and Vale on the doc: passed.
  • Live, bundled TechVault, lilrae lab start --scenario techvault --clean --yes --json on Linux (uid 1001), where wazuh-dashboard exits with EACCES (Fix the wazuh-dashboard config read failure (EACCES) #1422) and fails the first apply. On dev (cad3c6e): exit 1 after 695 s, empty start.json, RuntimeError: backend already has a runtime owner in stderr. On this branch: exit 1 after 781 s, both applies ran, no runtime owner line, start.json parses with ok: false, outcome: failed, and error carries, after the retry's diagnostics, Initial attempt: RAES runtime handoff failed: aptl.provisioner.backend-start-failed … Node wazuh-dashboard … required IPv4 address was not observed.

raes 7.0.0 lets one RuntimeManager own a target's components until it is
closed. The automatic backend retry kept the first attempt's manager alive
and built a second one for the same target, so the retry raised "backend
already has a runtime owner" and `lilrae lab start --json` printed nothing.

The retry now closes the first manager before the admitted hook runs. If
RAES refuses to release it, the retry is skipped and the first attempt's
error is kept with the reason. If the retry also fails, its error ends with
"Initial attempt:" and the first attempt's diagnostics. The retryable set
is unchanged.
@doublewhy
doublewhy marked this pull request as ready for review October 25, 2026 11:41
@doublewhy

Copy link
Copy Markdown
Contributor Author

Closing in favor of #1442, which fixes #1424 the same way and also closes the planning manager that admission left open.

@doublewhy doublewhy closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant