ci(windows): remaining RPC named-pipes job flake variants (host-ensure concurrent-start SIGTERM, lifecycle hostAlive probe)

[code-yeongyu]

Summary

The RPC named pipes (Windows) CI job (runs test/rpc-host-ensure.test.ts and test/rpc-host-lifecycle.test.ts on windows-latest) fails intermittently with several independent variants. Two were fixed at the root today; two remain and are tracked here so they are not mistaken for regressions in unrelated PRs.

Fixed at the root (2026-09-02)

• fix(rpc): size the ensure-host lock wait to the startup critical section #1288 - serializes concurrent starts for one socket across agent directories -> Error: database is locked: ensure-lock wait (10 s) shorter than the holder's critical section.
• test(rpc): tolerate an already-exited host in Windows supervisor teardown #1284 - starts a fresh host transparently on the next ensure after an idle exit -> Stop-Process on an already idle-exited pid / ENOENT host.pid: teardown and pidfile read raced the 600 ms idle exit.

Still open

1. rpc-host-ensure.test.ts > serializes concurrent starts for one socket across agent directories -> Error: RPC socket host exited with code null (SIGTERM) before answering get_protocol_info, followed by ENOTEMPTY ... rmdir senpi-host-ensure-cross-agent-race-* in cleanup. Seen: run 33602751743 attempt 1 (PR fix(rpc): scope multi-session socket events to attached connections #1277), run 33615244048 attempt 2 job 100202119649 (PR fix(rpc): scope multi-session socket events to attached connections #1277 at 44c7414, which does not touch host-ensure). The second caller ends up spawning (its existing-host probe did not reuse the first caller's fixture host) and its spawned host is SIGTERM'd before readiness - candidate causes: the 10 s existing-host probe timing out on the runner and stopManagedHost terminating the live fixture, or pidfile identity confusion between the two agent dirs.
2. rpc-host-lifecycle.test.ts > does not exit while a turn is active even with no connections; exits after the turn settles -> expect(await hostAlive(entry.pidFile)).toBe(true) false at line 201. Seen: run 33602751743 attempt 2 job 100164143208 and run 33615244048 attempt 1 job 100199445430 (both PR fix(rpc): scope multi-session socket events to attached connections #1277; the same file passes on macOS/Linux with the same tree). hostAlive goes through the 1 s PowerShell CIM identity probe (readProcessStartTime) with no retry, so a slow probe reads as "host gone"; production callers got retry in 1640d9b, this test helper did not.

Suggested next steps

• Variant 2: give the test's hostAlive the same bounded retry the production probe has (or reuse waitForStartTime), and capture supervisor stderr on failure to distinguish a real exit from a probe timeout.
• Variant 1: log which branch ensureHostLocked took for the second caller (reuse / stop+spawn) and the probe timing; on Windows consider lengthening EXISTING_HOST_PROBE_TIMEOUT_MS or serializing the fixture's readiness with the lock release.

[code-yeongyu]

Update after two consecutive reruns of #1277 (run 33615244048 attempts 2 and 3) failing on variant 1 only: the failure message comes from the spawn path (spawned RPC socket host exited ... before answering get_protocol_info), so the SECOND ensureHost caller (different agentDir) did not reuse the first caller's live fixture host - its existing-host probe failed - then spawned its own host whose readiness failed and was torn down with SIGTERM. Hypothesis: on Windows the named-pipe handshake secret / pidfile live under the caller's agentDir, so a caller with a different agentDir cannot authenticate to (or identify) a host started by another agentDir even though both target the same socket; the new spawn then collides on the pipe. Before #1288 this test usually failed earlier with database is locked, which masked this path. Needs a Windows repro with supervisor stderr captured.

[code-yeongyu]

Observation from this PR's own RPC named pipes (Windows) run (33619854835): the same test failed again with hostAlive -> false, WITH the retry in place — so in this variant readProcessStartTime did not throw; the CIM probe returned __SENPI_ABSENT__ (or a mismatched identity) for a live pid. The retry-on-throw in this PR is therefore necessary but not sufficient; #1290 variant 2 needs the probe itself to treat a transient ABSENT for a pid that was just observed alive as suspect (e.g. confirm absent twice before reporting dead), or the test to capture the probe result kind on failure. Leaving this PR as-is for the throw path and updating #1290.

[code-yeongyu]

Independent corroboration from PR #1276's CI runs (no new variant)

While validating an unrelated PR (#1276 — provider retry/timeout defaults) this job failed 5 times across successive runs, with the failing case rotating between the two steps. Recording it here since the issue exists to keep these from being misread as regressions in unrelated PRs.

Run	Failing step	Case	Error
33598824099	Run RPC host ensure tests	serializes concurrent starts for one socket across agent directories	database is locked
33602094330	Run RPC host lifecycle tests (ensure step passed)	does not exit while a turn is active even with no connections	Timed out waiting for RPC record
33602848864	Run RPC host lifecycle tests	does not exit while a client is attached, then exits after it detaches	(same step)
33605500664	Run RPC host ensure tests	serializes concurrent starts... + starts a missing host and reuses it on the second call	database is locked
33622766958	Run RPC host lifecycle tests	(lifecycle variant)	—

Mapping to this issue: rows 1 and 4 are the variant fixed by #1288; rows 2, 3 and 5 are the "Still open" lifecycle variants, with the hostAlive probe one addressed by #1291.

Two extra data points that may help whoever picks up the remaining variants:

1. The lifecycle cases fail on unmodified origin/main too. I ran does not exit while a turn is active... and does not exit while a client is attached... on a pristine origin/main worktree and on a feature branch side by side; both failed identically, which is what let me rule my own change out.
2. A distinct startup symptom appears off-CI: in a fresh checkout the same cases fail with RPC socket host exited with code 1 before answering get_protocol_info rather than the record timeout. That looks like the SIGTERM/startup family already described in "Still open" item 1 rather than the 15 s record wait — the lifecycle test spawns the supervisor via process.execPath on src/modes/rpc/host-lifecycle.ts with no TS runtime flag and no built entry, so a checkout without the right runtime/build state exits immediately. Worth separating from the timeout variant when fixing.

Also confirming for anyone triaging: RPC named pipes (Windows) is not a required status check (required contexts are Check and test, Changelog gate, Review claim gate); it blocks merges only transitively through the Check and test roll-up.

[code-yeongyu]

Variant 2 RECLASSIFIED: real product bug, not observer flake

PR #1291's new expectHostAlive() diagnostic (attaches supervisor stderr) caught it red-handed at 2c6e385 (job 100223762670):

Error: host 7504 reported dead
[supervisor stderr]
senpi rpc listening on unix://\\.\pipe\senpi-rpc-internal-9b257e7b-...
senpi rpc host ready on unix://...senpi-hlc-turn-a7EXH8\rpc.sock (coldStart=transient, idleExitMs=800)
senpi rpc host supervisor: idle shutdown

The supervisor logged idle shutdown — the host GENUINELY exited while a turn was active (the test had already awaited agent_start and the fake model holds the response open). The two-consecutive-read probe agreed the pid was gone. So on windows-latest the active-turn occupancy signal does not always reach (or hold) the host-lifecycle idle clock after the last connection drops: candidates are the busy/occupancy propagation in host-lifecycle.ts / the lifecycle observer of agent_settled, or a Windows-specific race where the connection drop's release path runs before the turn's busy state is registered.

Variant 2's earlier "hostAlive false on a live pid" reading is withdrawn — the pid really was dead. The remaining probe-hardening in #1291 stays useful (throw-retry, two-read confirmation, stderr diagnostics) but the fix for this variant belongs in the runtime idle-exit gating, not in the test.

[code-yeongyu]

The merged hostAlive retry (#1291) does not clear this variant

Observed on an unrelated PR (#1293 — root package.json script shapes) that was rebased onto cc0ed64d9, i.e. with #1291's fix present in the tree (verified: expectHostAlive appears 4x in the test file on that branch).

Run 33626348905, step Run RPC host lifecycle tests:

FAIL  test/rpc-host-lifecycle.test.ts > ensureHost-spawned host lifecycle
      > does not exit while a turn is active even with no connections; exits after the turn settles
 ❯ expectHostAlive test/rpc-host-lifecycle.test.ts:675:20
 673| async function expectHostAlive(qa: Scratch, pidFile: { pid: number; pr…

The failure moved into the new retry helper rather than disappearing, which is consistent with the reading I posted on #1291 before it merged: if the supervisor is already gone when the probe runs, retrying the probe cannot make it report alive — it just spends the retry budget and reports false.

Supporting evidence gathered earlier in this investigation:

• The same two lifecycle cases fail identically on a pristine origin/main worktree and on a feature branch when run off-CI, so the cause is not branch-specific.
• Off-CI the symptom presents as RPC socket host exited with code 1 before answering get_protocol_info rather than a probe/record timeout. The lifecycle test spawns the supervisor via process.execPath on src/modes/rpc/host-lifecycle.ts with no TS-runtime flag and no built entry, so a checkout lacking the right runtime/build state exits immediately — the SIGTERM/startup family in "Still open" item 1.

Suggested next probe for whoever picks this up: capture why the supervisor is not alive at that moment (exit code / stderr from the spawned process) instead of only re-asking whether it is alive. If it exited, the probe budget is the wrong lever.

No competing PR from me — this issue and its PRs own the fix.

[code-yeongyu]

Merged #1296 — root cause of variant 2 confirmed and fixed: PR #1277's attached-only event delivery starved the supervisor's lifecycle observer (it never sends open_session), so busySessions never saw agent_start and the supervisor idle-exited a host mid-turn. The fanout now broadcasts an exact content-free lifecycle allowlist (agent_start, agent_settled, agent_idle, session_opened, session_closed) to every registered connection; all other records stay attached/requester scoped (isolation suite still pins that). Windows proof: run 33626085975 attempt 2, RPC named pipes (Windows) SUCCESS with the target test passing; regression 5 files / 87 tests + biome + tsc clean. Strict review: APPROVE, no blockers.

Reviewer note kept open here for follow-up (pre-existing, not introduced by #1296): host-lifecycle.ts ~545-560 marks a RECONNECTED observer healthy without replaying lifecycle state, so an observer that drops mid-turn re-enters with an empty busy map; a reconnect should either replay current busy state (e.g. list_sessions with status) or stay "unhealthy" (fail-safe alive) until the next agent_start/settled pair.

Variant 1 (host-ensure concurrent-start on Windows: second caller cannot reuse the first's fixture host, then its own spawn is SIGTERM'd) remains open in this issue.

[code-yeongyu]

Recurrence data from PR #1295, which touches no RPC files at all (six files: prompt.ts, tools/monitor.ts, docs/terminal-tools.md, two changelogs, one test). Both variants in this issue's title reproduced there, so the lane is still taxing unrelated PRs after #1294.

Observed on PR #1295 (5 occurrences, 2 variants):

Head	Failing step	Variant
98d7b3454	Run RPC host lifecycle tests	rpc-host-lifecycle.test.ts > does not exit while a turn is active even with no connections — host NNNN reported dead
3d4fd5d86	Run RPC host ensure tests	rpc-host-ensure.test.ts > serializes concurrent starts for one socket across agent directories — RPC socket host exited with code null (SIGTERM) before answering get_protocol_info
9dd8260a8	Run RPC host ensure tests	same host-ensure SIGTERM
d3f0c2049	Run RPC host ensure tests	same host-ensure SIGTERM

Base rate on main itself (last 6 CI runs of the same job): 4 pass / 1 fail / 1 cancelled — d1e2f0cd0 failed the identical job, so it is not PR-specific.

The lifecycle variant appears fixed by a223381c8 (feed lifecycle observer records); the host-ensure concurrent-start SIGTERM variant is still live and is the one currently blocking. It reaches merge protection through the Check and test fan-in, so an unrelated PR cannot go green while it trips.

For whoever picks this up: in rpc-host-ensure.test.ts:35 the second ensureHost runs against a different agentDir but the same socket while the first holds the lock via _test.afterLockAcquired; the spawned fixture host takes a SIGTERM before answering get_protocol_info. No reproduction attempted here — there is no Windows host available in this fleet, so I did not push a speculative fix.