Skip to content

test(runtime-host): assert owned Host exit against the kernel's shutdown contract - #4784

Closed
UncertaintyDeterminesYou4ndMe wants to merge 1 commit into
apache:mainfrom
UncertaintyDeterminesYou4ndMe:fix/4776-owned-host-exit-flake
Closed

test(runtime-host): assert owned Host exit against the kernel's shutdown contract#4784
UncertaintyDeterminesYou4ndMe wants to merge 1 commit into
apache:mainfrom
UncertaintyDeterminesYou4ndMe:fix/4776-owned-host-exit-flake

Conversation

@UncertaintyDeterminesYou4ndMe

@UncertaintyDeterminesYou4ndMe UncertaintyDeterminesYou4ndMe commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Both owned-lifecycle tests assert that a Host process has exited, against a 5 s wall clock that is shorter than the exit the kernel guarantees, and one of them starts that clock from an event the Host does not control. Under suite load they fail on a Host that is working correctly. Each assertion now observes the Host and is bounded by the kernel's own shutdown grace, so it can only fail when the Host really does not exit.

Fixes #4776

Root cause

an authority-supervised Candidate exits if its launch owner is killed measured from the wrong object. It waited on connected.connection.closed, then gave the process 5 s. That promise is the Client's own transport — RuntimeHostConnection assigns closed straight from transport.closed, and its #fail path ends in transport.abort(). An unanswered liveness probe (2 s interval, 2 s timeout) therefore resolves it while the Host is still running, and the exit budget starts at a moment unrelated to the Host's shutdown.

The Host also cannot exit as early as that bound assumed. The owner-loss close is armed by launchOwnerGuard.bind(...) in candidate-entry.ts, which runs only after startExecutionRuntimeHostCandidate resolves, so composition startup and recovery must finish before the Candidate can begin closing. Instrumenting the test showed the SIGKILL landing while startup was still in progress even on an idle machine, because retryConnect returns as soon as the handshake is admitted and the kernel admits Clients while it is still recovering. The shutdown that follows is then bounded by shutdownGraceMs (10 s), after which the kernel force-terminates.

owned Host exits promptly after its first connection closes bounded shutdown tighter than the kernel does. settle(5_000) allowed the whole shutdown 5 s against a kernel that allows itself 10 s, and it conflated the two claims in the test's name: that the owned launch's idleGraceMs is 0 rather than 30 s, and that the process exits cleanly. settle also SIGKILLs on timeout, so a Host that was still shutting down was reported as an unclean exit rather than as a slow one.

Evidence

I could not reproduce either failure naturally on this machine: 42 focused repetitions and 19 full-suite runs of each test, including runs with two extra suites as background load, produced zero failures. The mechanism was established by fault injection against the built kernel, which reproduces both reported failures exactly.

Stalling the Host's main thread during composition startup — the shape of the synchronous store work profiled in #4032 — reproduces Error: process <pid> did not exit, with the merged Client, entry and kernel timeline showing why:

     0 ms  test    SIGKILL to launcher
     4 ms  entry   candidate observes IPC disconnect
    62 ms  kernel  composition startup still running (bind has not happened)
  1262 ms  kernel  main thread blocked
  4006 ms  test    connection.closed resolves — Client's liveness timeout; process alive
  9022 ms  test    process <pid> did not exit

Stalling composition close for 6.5 s reproduces false !== true with exit: { code: null, signal: 'SIGKILL' }settle's own kill of a Host that was mid-shutdown.

What changed

  • The launch-owner test waits for the process, bounded at 20 s: above the kernel's 10 s shutdown grace, below the launcher fixture's idle grace. The connection-closed assertion stays, now after the exit, where it is a consequence rather than a gate.
  • The launcher fixture's idle grace goes from 10 s to 60 s, so a Candidate that exits because it went idle can never satisfy an owner-loss bound, with an explicit 10 s first-connection deadline so a Candidate no Client reaches still exits on its own.
  • The owned-exit test observes the draining registration the kernel publishes as the first step of shutdown, which is the idle-grace claim directly, then bounds the exit at 15 s.

Verification

  • Under the same fault injection, both tests pass (13.9 s and 7.2 s).
  • The bounds still bite. Neutering the owner-loss close (bind to a no-op) fails the launch-owner test at 20.5 s. Stalling shutdown past the kernel's own 10 s grace still fails the owned-exit test.
  • @maka/runtime-host suite, 5 runs at default concurrency: both tests pass in all 5.
  • npm run format:check, npm run lint: clean. npm run typecheck: clean once every workspace dependency's dist is current (a stale @maka/ui build reported 42 errors in apps/desktop that a rebuild cleared, none of them in files this PR touches).
  • The fault-injection harness is not committed.

Unrelated, and worth its own issue: WorkHub correction replaces its link without stopping a shared manual Turn (execution-composition.test.ts) failed in 10 of 13 full-suite runs before this change and 2 of 5 after it, on main and on this branch alike. It is the flake this machine reproduces, and it is not the one this PR addresses. Filed as #4785: the WorkHub candidate-set identity digests the Session name, so the automatic title commit invalidates the snapshot the delegation was built on.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code read the lifecycle paths, wrote and ran the fault-injection harness and the instrumented timeline, made the test changes, and drafted this PR body. The commit carries a Generated-by: Claude Code trailer. I reviewed the diff and the evidence and remain the contributor of record.

Checklist

  • Tests cover the change and fail without it — this PR is the test change; the deterministic seam is the uncommitted fault-injection harness, whose before/after results are above, together with the mutation check that the new bounds still fail a broken guard
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

…own contract

Both owned-lifecycle tests asserted that a Host process had exited against a
5 s wall clock, which is shorter than the exit the kernel guarantees, and one
of them started that clock from an event the Host does not control.

`an authority-supervised Candidate exits if its launch owner is killed` waited
on `connected.connection.closed` and then gave the process 5 s. That promise is
the Client's own transport, and the Client aborts the transport when a liveness
probe goes unanswered for two seconds, so a Host that is merely busy resolves it
while still running and the exit budget starts at an unrelated moment. The Host
also cannot exit as early as the bound assumed: the owner-loss close is armed by
`launchOwnerGuard.bind(...)` only after `startExecutionRuntimeHostCandidate`
resolves, so composition startup and recovery must finish first, and the
shutdown that follows is bounded by `shutdownGraceMs` (10 s). Wait for the
process instead, with a 20 s bound that sits above that grace and below the
launcher fixture's idle grace, and keep the connection-closed assertion after
the exit, where it is a consequence rather than a gate.

The fixture's idle grace moves from 10 s to 60 s so a Candidate that exits
because it went idle can never satisfy an owner-loss bound, with an explicit
10 s first-connection deadline so a Candidate no Client reaches still exits.

`owned Host exits promptly after its first connection closes` allowed the whole
shutdown 5 s against a kernel that allows itself 10 s, and conflated the two
claims in its name. The promptness claim is the owned launch's idleGraceMs of 0
against a 30 s default, which the draining registration the kernel publishes as
its first shutdown step reports directly; the exit is a second claim, now
bounded by the kernel's grace.

Refs apache#4776

Generated-by: Claude Code
@UncertaintyDeterminesYou4ndMe

Copy link
Copy Markdown
Contributor Author

Closing this one myself: it is a test-only stability change, and with the project actively reducing surface area right now I would rather not add review load for it. The root-cause analysis and the fault-injection evidence in the description stand on their own if anyone wants to pick #4776 up later; the branch stays on my fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/S Under 100 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flaky(runtime-host): owned Host lifecycle tests intermittently fail to observe process exit

1 participant