fix(daemon): probe a live daemon again before replacing it as unreachable - #3050
okwasniewski wants to merge 5 commits into
Conversation
…able A 500 ms connect probe measures wall-clock time on the client's event loop. A client that stalls past it (large sync parse, GC on a loaded CI host) reads a listening daemon as unreachable, and the takeover kills it, ending every live session. The next command then meets no session: 2 devices match equally, Daemon request timed out, Invalid daemon response. While the recorded daemon process is still ours, probe up to three more times before replacing it.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
On Linux the stalled-client test's first probe still connects, so the retry never ran under coverage. A mocked miss exercises it on every host; the stall test skips with the reason where the host outruns it.
|
Pushed The remaining red job, iOS Smoke Tests, is unrelated: |
Under the load that stalls the probe, ps misses its deadline too and the identity read fails closed, ending the retries. The takeover still proves identity before it signals.
|
This PR is ready. I reviewed 058b06a and found nothing that needs to change before merge. All 14 checks pass on that commit, and there are no conflicts. The iOS smoke failure you saw on an earlier head exercises simctl openurl, which does not overlap the daemon-client probe path. Not blocking, and you can take or leave these: canReachReusableDaemon runs before the version and code identity check, so a live but unreachable daemon with a mismatch waits up to about 3.6 s for an answer the decision never reads, and making viaClientTransport lazy like onAnyAdvertisedTransport would avoid that; no test pins the liveness gate or the daemon_probe_recovered diagnostic, and asserting that a dead pid issues exactly one probe would cover it; the Atomics.wait stall test in daemon-client-stalled-probe.test.ts patches the global net.createConnection and skips when the connect finishes first, so would it be simpler to drop it or keep it only as a documented local reproduction? On evidence: I did not run the new tests, so the claim that they fail without the fix comes from reading the pre-change route. I could not reproduce the CI benchmark stall, and whether 3 x 200 ms retries recover a client whose stalls keep recurring is unmeasured. The PR body's validation names ed3b8e6, not 058b06a, so only CI covers the head. Nothing else needs to happen before a maintainer merges. |
thymikee
added
the
ready-for-human
…he takeover A version or code mismatch decides alone, so the patient probe no longer waits on a daemon about to be replaced anyway. Pins a dead pid's single probe and the recovery diagnostic, and drops the stall test that patched the global createConnection and skipped where the host outran it.
|
Thanks @thymikee. Took all three suggestions in
|
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
|
Can you fix the coverage pls? |
… cannot reuse The fresh stand-in binds before the dead port is picked, and the test asserts the replacement spawn ran and the one dead probe failed; it skips when the host recycled the exited pid.
|
@thymikee the Coverage failure isn't from this diff. The failing test is To check this diff's own coverage, I merged
|
|
The delta since 058b06a looks good at 1aeb7b9. The re-probe now reuses the launch spec's daemon identity before it replaces anything, so a live daemon that stalled on one probe is kept, and a dead daemon still fails after one bounded extra probe inside the caller budget. The new stalled-probe tests cover both cases. CI is green and there are no conflicts. |
|
Correction to my previous comment on 1aeb7b9: I called it clean too early. A closer read found one defect. The Coverage failure from the earlier head is gone, and all 14 checks pass on 1aeb7b9. One defect remains in the newer-daemon path. The delta made the client-transport check lazy, but it left |
thymikee
removed
the
ready-for-human
Summary
A client replaced a live daemon it could not reach within one 500 ms probe. That budget is wall-clock time on the client's own event loop: a client that stalls past it (a large synchronous parse, a GC pause on a loaded host) sees a listening daemon as unreachable. The takeover then kills it, and every session it held goes with it.
Downstream, the e2e mobile benchmark drives two iOS simulators through one daemon. In CI we saw
Replacing daemon (pid N, v0.21.16) ...: unreachablemid-run. The other worker then failed its next command in one of three ways:snapshot failed: 2 devices match this request equally(its session was gone),Daemon request timed out, orInvalid daemon response.readReusableLocalDaemonnow probes up to three more times, 200 ms apart, before it decides a daemon is unreachable. Reachability on the client's transport is asked last, only when version and code identity leave the decision to it, and only while the recorded pid is alive (a signal-0 check, since thepsidentity read misses its deadline under the same load). The takeover still proves identity before it signals. A dead daemon is still replaced at once. A recovered probe emits adaemon_probe_recovereddiagnostic.Touched: 2 source files (
daemon-client-lifecycle.ts,daemon-launch-spec.ts), 1 new test file, 2 updated test files.Validation
Tested commit
1aeb7b91d; on its merge withmain,pnpm test:coverage:cipassed and the changed-line gate passed (92.3%).daemon-client-stalled-probe.test.ts: a first probe forced to miss keeps the live stand-in daemon, anddaemon_probe_recoverednames it. A daemon whose pid is gone gets exactly one probe before it is replaced.daemon-launch-spec.test.ts: a version mismatch never asks client-transport reachability. Each assertion fails when its line of the fix is removed.pnpm check:affected --run: passed.