Repository navigation
Failed aptl lab start leaves the whole range running with no teardown guidance #952
Description
Finding
When the RAES handoff fails after containers exist, aptl lab start prints the error and exits, leaving the full range running. Observed 2026-09-04 on a clean packaged lab: SEM-218 backend-contract-invalid on shuffle-backend after 11m52s, with all 32 containers, five networks, and every volume still up. Nothing in the output says so. The next aptl lab start refuses with [lifecycle-range-present] until the user runs aptl lab stop. On a laptop this silently holds more than 20 GB of RAM.
Leaving the range up is useful for diagnosis, so the default should not change silently. The user must be told what is running and how to recover.
Required work
- When a start fails after the project has been mutated (project containers or networks exist), the failure summary must name the residue (container count, networks) and print the exact recovery commands:
aptl lab stopandaptl lab stop -vwith their data consequences. - Add
--teardown-on-failure(off by default) that runs the project-scoped stop without volume removal after a failed start, under the same lifecycle lock as the start (fix: harden lab lifecycle recovery #933). - Keep
_step_reject_preexisting_rangesemantics; the message it prints should match the recovery guidance.
Acceptance criteria
- A failed start with residue prints the residue and the recovery commands.
-
aptl lab start --teardown-on-failureleaves zero project containers and networks after a failed start; volumes are kept. - Unit tests cover both paths and the lock ordering.
Relationships
- Follows lab teardown/boot robustness: a partially-up or lingering range causes confusing re-boot failures #905 (closed) and fix: harden lab lifecycle recovery #933.
- Related: Fix TechVault Orborus socket lowering and offline child-image preparation #949 (the Shuffle failure that surfaced this).
Activity
Brad-Edwards commentedon Oct 14, 2026
🛠️ Picked up by /implement - driver codex, branch
952-start-failure-recovery, 2026-10-14T08:28:47.736Z.Brad-Edwards commentedon Oct 14, 2026
gc workflow phase recorded:
preflight(issue #952). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.Brad-Edwards commentedon Oct 14, 2026
gc workflow phase recorded:
plan(issue #952). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.The original #952 implementation shipped in #1266 (4617180), but architecture preflight identified remaining acceptance gaps. This PR will complete those gaps on the existing core/backend boundaries.
Likely changes: src/aptl/core/lab.py, lab_types.py, ephemeral_containers.py; deployment/compose{stop,project_cleanup,resource_resolution,owned_start}.py; cli/_start_residue.py; focused tests in tests/test_lab.py, test_cli.py, test_compose_resource_ownership.py, test_deployment_backend.py and test_stranded_helper_cleanup.py, plus the preflight docs and mkdocs.yml. No new dependencies, persistent configuration, API/web controls, or Compose assets.
Security: preserve strict configuration, canonical lifecycle-root and path validation, receipt/native-ID deletion authority and original Docker/SSH endpoint binding. Normalize only bounded error classifications and never render raw backend stderr, environment or exception repr. No foreign-resource deletion or wider cleanup selectors. Maintainability/extensibility: extend the existing removal policy, owned-stop seam, result types and recovery renderer; keep helper disposal distinct from node data retention. Whole-repository impact: CLI/core/backend consumers, appliance supervision, documentation navigation and targeted pytest/hygiene; hosted CI owns full suites and lifecycle qualification.
Verification: observe focused regression failures before each fix, run exact affected pytest paths via bash tools/run-targeted-tests.sh, then staged pre-commit, bounded review, hosted CI and SonarCloud. Existing baseline: 33 focused acceptance/ownership-lock tests pass after synchronizing the local environment to uv.lock (removed duplicate aptl-labs adapter registration). No requirement UIDs are attached; no structural gate or new requirement is introduced. Proposed Conventional Commit title: fix: preserve volumes and complete failed-start recovery.
Brad-Edwards commentedon Oct 14, 2026
gc_codex_review — sanitized deferred publication for issue #952, cycle 1 of 1
Reviewed revision:
a18fd4622b302f78c40fd8fc920c17e3c53cf3e7c4dc925564942cc1341010a9Architectural read
The core review confirms reuse of lifecycle, receipt-owned cleanup, shared recovery guidance and volume policy boundaries. The security review found no concrete exploitable regression in the reviewed diff. One existing guest-access test must exercise the separated readiness and supervision phases.
Verdict:
ship-with-fixesFindings
core-F1 — Update the guest-access contract test for separate startup phases
one-offfixtests/test_seat_image_runtime.py:test_readiness_calls_the_current_guest_access_contractBrad-Edwards commentedon Oct 14, 2026
gc_codex_review pre-push cycle 1 of 1 complete for issue #952 on branch '952-start-failure-recovery'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next
gc_codex_review(uncommitted) invocation to count cycles.Brad-Edwards commentedon Oct 14, 2026
Review decision record — codex cycle 1 (issue #952)
Reviewer: codex
Cycle: 1
Verdict:
ship-with-fixesArchitectural read:
Blocking findings: 1
Finding 1 —
one-offcore-F1tests/test_seat_image_runtime.py:test_readiness_calls_the_current_guest_access_contractBrad-Edwards commentedon Oct 14, 2026
Pre-PR base synchronization
refs/remotes/origin/devat6e92fcc6e61de1b04368949ebce66491eca3eb37merged_clean622232051fd7f8cc1176ff618785f6cf5b39b0c58bf5bcdfc81318d3c875db220b0f74d9b21ab2f9Brad-Edwards commentedon Oct 14, 2026
Pre-PR base synchronization
refs/remotes/origin/devat6e92fcc6e61de1b04368949ebce66491eca3eb37already_current4285e8ea1c6f2c731ccef05eb3e7e1afb98624599bd5b6e5d369d4f768b60ec29a8cf07811b69bd1Brad-Edwards commentedon Oct 14, 2026
Pre-PR base synchronization
refs/remotes/origin/devat6e92fcc6e61de1b04368949ebce66491eca3eb37already_current2a97f5efc1b2594db0e06d7e67a24fb629fcbf91ccb4c711774bada0d2ef7d2940b2156a186394d2Brad-Edwards commentedon Oct 15, 2026
Pre-PR base synchronization
refs/remotes/origin/devat096d32d13f98c63583703cb488b8c85dbe22bc9cmerged_clean83cabb380a57de023006ccf507bd712c3d493780ae4c606f05c3f7c17bedee38929079c8eb14af60Brad-Edwards commentedon Oct 15, 2026
Execution obligation RECOVERY-CLEAN-TARGET — Opened
Category: defect
Observed state: The pre-existing-range refusal qualifies both stop commands, but its alternative aptl lab start --clean example still omits the selected external project.
Impact: Following the clean-start example from another directory can select the wrong project for destructive cleanup.
Current obligation: Qualify the clean-start alternative through the shared lifecycle command formatter and extend the regression test.
Evidence
Brad-Edwards commentedon Oct 15, 2026
Pre-PR base synchronization
refs/remotes/origin/devat096d32d13f98c63583703cb488b8c85dbe22bc9calready_currente5e8cc7e52eda284d84a976bca464d5566c34fcd36c42429d9327f347cfa444fbbecc6f88933a946Brad-Edwards commentedon Oct 15, 2026
Execution obligation RECOVERY-CLEAN-TARGET — Resolved
Category: defect
Observed state: The preexisting-range refusal's clean-start alternative lacked the external project's --project-dir argument.
Impact: Following the recovery command from another directory could clean the wrong project.
Current obligation: Qualify the clean-start alternative with the same canonical project targeting as stop recovery commands.
Evidence
Disposition: fix
Corrective action: Published common lifecycle recovery command formatting and qualified the clean-start alternative in commit e5e8cc7.
Verification
Brad-Edwards commentedon Oct 15, 2026
Ready for review — issue #952
PR: #1387
Plan: #952 (comment)
Outcome
Failed lab starts report what remains and provide recovery commands for the correct project. Optional failure teardown and backend compensation preserve node volumes, while startup readiness failures are handled under the lifecycle lock.
Complete failed-start recovery, volume preservation and correctly targeted cleanup guidance. Fixed all reported quality findings; 29 focused regression checks and staged hygiene passed.
Files changed
Added:
docs/architecture/issue-952-start-failure-recovery-preflight.mdModified:
docs/architecture/issue-905-lab-lifecycle-robustness-preflight.mddocs/reference/cli.mddocs/reviews/962-lilrae-readiness/tracked-file-inventory.tsvmkdocs.ymlsrc/aptl/cli/_start_residue.pysrc/aptl/cli/lab_render.pysrc/aptl/core/deployment/_compose_owned_start.pysrc/aptl/core/deployment/_compose_project_cleanup.pysrc/aptl/core/deployment/_compose_resource_resolution.pysrc/aptl/core/deployment/_compose_stop.pysrc/aptl/core/ephemeral_containers.pysrc/aptl/core/lab.pysrc/aptl/core/lab_types.pysrc/aptl/core/lifecycle_guard.pytests/test_cli.pytests/test_compose_resource_ownership.pytests/test_deployment_backend.pytests/test_lab.pytests/test_seat_image_runtime.pytests/test_stranded_helper_cleanup.pyReviews
Traceability reconciliation
Proposed in this PR — verified against the merged tree in Phase E.
Status
Brad-Edwards commentedon Oct 15, 2026
Delivery readiness recorded — Phase E finalizes issue #952 automatically when PR #1387 merges.
github-actions commentedon Oct 15, 2026
Phase E could not finalize PR #1387
The merge was detected and the delivery-readiness record was read, but finalization did
not complete. This issue is intentionally left open.
delivery_readiness_head_mismatchre_record_phase_d_readiness_for_the_merged_head_and_rerun_phase_eRepair the cause, then re-run the Ground Control Phase E workflow for this pull request.