test(runtime-host): assert owned Host exit against the kernel's shutdown contract - #4784
Closed
UncertaintyDeterminesYou4ndMe wants to merge 1 commit into
Conversation
…own contract Both owned-lifecycle tests asserted that a Host process had exited against a 5 s wall clock, which is shorter than the exit the kernel guarantees, and one of them started that clock from an event the Host does not control. `an authority-supervised Candidate exits if its launch owner is killed` waited on `connected.connection.closed` and then gave the process 5 s. That promise is the Client's own transport, and the Client aborts the transport when a liveness probe goes unanswered for two seconds, so a Host that is merely busy resolves it while still running and the exit budget starts at an unrelated moment. The Host also cannot exit as early as the bound assumed: the owner-loss close is armed by `launchOwnerGuard.bind(...)` only after `startExecutionRuntimeHostCandidate` resolves, so composition startup and recovery must finish first, and the shutdown that follows is bounded by `shutdownGraceMs` (10 s). Wait for the process instead, with a 20 s bound that sits above that grace and below the launcher fixture's idle grace, and keep the connection-closed assertion after the exit, where it is a consequence rather than a gate. The fixture's idle grace moves from 10 s to 60 s so a Candidate that exits because it went idle can never satisfy an owner-loss bound, with an explicit 10 s first-connection deadline so a Candidate no Client reaches still exits. `owned Host exits promptly after its first connection closes` allowed the whole shutdown 5 s against a kernel that allows itself 10 s, and conflated the two claims in its name. The promptness claim is the owned launch's idleGraceMs of 0 against a 30 s default, which the draining registration the kernel publishes as its first shutdown step reports directly; the exit is a second claim, now bounded by the kernel's grace. Refs apache#4776 Generated-by: Claude Code
Contributor
Author
|
Closing this one myself: it is a test-only stability change, and with the project actively reducing surface area right now I would rather not add review load for it. The root-cause analysis and the fault-injection evidence in the description stand on their own if anyone wants to pick #4776 up later; the branch stays on my fork. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Both owned-lifecycle tests assert that a Host process has exited, against a 5 s wall clock that is shorter than the exit the kernel guarantees, and one of them starts that clock from an event the Host does not control. Under suite load they fail on a Host that is working correctly. Each assertion now observes the Host and is bounded by the kernel's own shutdown grace, so it can only fail when the Host really does not exit.
Fixes #4776
Root cause
an authority-supervised Candidate exits if its launch owner is killedmeasured from the wrong object. It waited onconnected.connection.closed, then gave the process 5 s. That promise is the Client's own transport —RuntimeHostConnectionassignsclosedstraight fromtransport.closed, and its#failpath ends intransport.abort(). An unanswered liveness probe (2 s interval, 2 s timeout) therefore resolves it while the Host is still running, and the exit budget starts at a moment unrelated to the Host's shutdown.The Host also cannot exit as early as that bound assumed. The owner-loss close is armed by
launchOwnerGuard.bind(...)incandidate-entry.ts, which runs only afterstartExecutionRuntimeHostCandidateresolves, so composition startup and recovery must finish before the Candidate can begin closing. Instrumenting the test showed the SIGKILL landing while startup was still in progress even on an idle machine, becauseretryConnectreturns as soon as the handshake is admitted and the kernel admits Clients while it is stillrecovering. The shutdown that follows is then bounded byshutdownGraceMs(10 s), after which the kernel force-terminates.owned Host exits promptly after its first connection closesbounded shutdown tighter than the kernel does.settle(5_000)allowed the whole shutdown 5 s against a kernel that allows itself 10 s, and it conflated the two claims in the test's name: that the owned launch'sidleGraceMsis 0 rather than 30 s, and that the process exits cleanly.settlealso SIGKILLs on timeout, so a Host that was still shutting down was reported as an unclean exit rather than as a slow one.Evidence
I could not reproduce either failure naturally on this machine: 42 focused repetitions and 19 full-suite runs of each test, including runs with two extra suites as background load, produced zero failures. The mechanism was established by fault injection against the built kernel, which reproduces both reported failures exactly.
Stalling the Host's main thread during composition startup — the shape of the synchronous store work profiled in #4032 — reproduces
Error: process <pid> did not exit, with the merged Client, entry and kernel timeline showing why:Stalling composition close for 6.5 s reproduces
false !== truewithexit: { code: null, signal: 'SIGKILL' }—settle's own kill of a Host that was mid-shutdown.What changed
Verification
bindto a no-op) fails the launch-owner test at 20.5 s. Stalling shutdown past the kernel's own 10 s grace still fails the owned-exit test.@maka/runtime-hostsuite, 5 runs at default concurrency: both tests pass in all 5.npm run format:check,npm run lint: clean.npm run typecheck: clean once every workspace dependency'sdistis current (a stale@maka/uibuild reported 42 errors inapps/desktopthat a rebuild cleared, none of them in files this PR touches).Unrelated, and worth its own issue:
WorkHub correction replaces its link without stopping a shared manual Turn(execution-composition.test.ts) failed in 10 of 13 full-suite runs before this change and 2 of 5 after it, onmainand on this branch alike. It is the flake this machine reproduces, and it is not the one this PR addresses. Filed as #4785: the WorkHub candidate-set identity digests the Session name, so the automatic title commit invalidates the snapshot the delegation was built on.AI use
Select exactly one:
Tool(s) and scope: Claude Code read the lifecycle paths, wrote and ran the fault-injection harness and the instrumented timeline, made the test changes, and drafted this PR body. The commit carries a
Generated-by: Claude Codetrailer. I reviewed the diff and the evidence and remain the contributor of record.Checklist
Does this PR entail a change in behavior?