Skip to content

A worker never redials a gateway that vanished without closing the uplink #941

Closed

Description

@V3RON

A worker stays "connected" to a gateway that is gone when the gateway disappears without closing the TCP connection: the gateway host loses power or freezes, the network splits, or a NAT or Tailscale path starts dropping traffic. The worker never redials, not even after the gateway is back, so it drops out of the fleet until the OS gives up on the socket, which can take many minutes.

A clean gateway restart is fine: the socket closes, GatewayUplink (src/daemon/gateway-uplink.ts) sees close and redials with its capped, jittered backoff (e2e gateway-fleet.e2e.test.ts:498).

Diagnosis: the worker learns a link died only from the socket's close event. WebSocketUplinkConnection (src/ports/uplink-websocket.ts) sends no WebSocket pings and sets no idle timeout or TCP keepalive. Over the uplink only the gateway sends requests, so the worker writes nothing that would fail. The gateway side does detect a dead worker (WORKER_CALL_TIMEOUT_MS and the status.get timeout limit in src/gateway/worker-link.ts), but the worker never finds out.

A second, smaller gap: WebSocketUplinkConnector.connect sets no handshakeTimeout, so a dial into a host that drops packets waits for the OS connect timeout (about 75s on macOS) before the next backoff step.

Fix, worker side only, in src/ports/uplink-websocket.ts:

  • While the uplink is open, ping every 15s. If no pong arrived since the previous ping, terminate() the socket. The resulting close drives the existing redial. ws answers pings by itself (autoPong defaults to true), so the gateway needs no change.
  • Pass handshakeTimeout: 10_000 to the outbound WebSocket.
  • Both values are named constants, not config keys. No public surface changes.

Tests (fast lane):

  • A server that stops answering pings: the worker terminates the link within two intervals, and GatewayUplink redials.
  • A healthy link stays open across several ping rounds.
  • A server that accepts TCP but never completes the handshake: the dial fails with an UplinkError at the timeout.

Written by an agent.

Activity

  1. self-assigned this
    on Oct 23, 2026
  2. added a commit that references this issue on Oct 23, 2026
    cac23e0
  3. V3RON commented on Oct 23, 2026

    @V3RON
    ContributorAuthor

    Delivery run

    Merged: #942 (#941)
    Parked: none
    Waiting: none
    Decided: none

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug:readyAn agent may fix it.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions