You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A worker never redials a gateway that vanished without closing the uplink #941
A worker stays "connected" to a gateway that is gone when the gateway disappears without closing the TCP connection: the gateway host loses power or freezes, the network splits, or a NAT or Tailscale path starts dropping traffic. The worker never redials, not even after the gateway is back, so it drops out of the fleet until the OS gives up on the socket, which can take many minutes.
A clean gateway restart is fine: the socket closes, GatewayUplink (src/daemon/gateway-uplink.ts) sees close and redials with its capped, jittered backoff (e2e gateway-fleet.e2e.test.ts:498).
Diagnosis: the worker learns a link died only from the socket's close event. WebSocketUplinkConnection (src/ports/uplink-websocket.ts) sends no WebSocket pings and sets no idle timeout or TCP keepalive. Over the uplink only the gateway sends requests, so the worker writes nothing that would fail. The gateway side does detect a dead worker (WORKER_CALL_TIMEOUT_MS and the status.get timeout limit in src/gateway/worker-link.ts), but the worker never finds out.
A second, smaller gap: WebSocketUplinkConnector.connect sets no handshakeTimeout, so a dial into a host that drops packets waits for the OS connect timeout (about 75s on macOS) before the next backoff step.
Fix, worker side only, in src/ports/uplink-websocket.ts:
While the uplink is open, ping every 15s. If no pong arrived since the previous ping, terminate() the socket. The resulting close drives the existing redial. ws answers pings by itself (autoPong defaults to true), so the gateway needs no change.
Pass handshakeTimeout: 10_000 to the outbound WebSocket.
Both values are named constants, not config keys. No public surface changes.
Tests (fast lane):
A server that stops answering pings: the worker terminates the link within two intervals, and GatewayUplink redials.
A healthy link stays open across several ping rounds.
A server that accepts TCP but never completes the handshake: the dial fails with an UplinkError at the timeout.
Activity
fix(ports): a worker terminates an uplink whose gateway stops answeri…
test: a healthy uplink stays open across several ping rounds (#941)
fix(ports): a stalled uplink handshake fails after 10s instead of the…
chore(ports): keep the handshake timeout constant private (#941)
test: a silent gateway link fails on the closed assertion, not a time…
fix(ports): a worker redials a gateway that vanished without closing …
V3RON commentedon Oct 23, 2026
Delivery run
Merged: #942 (#941)
Parked: none
Waiting: none
Decided: none
Written by an agent.