tonic-xds/xds-client: client permanently stuck on stale endpoints after backend pod rotation — no keepalives or connect timeouts anywhere in the xDS path

[aryehlev]

Labels: bug, tonic-xds, xds-client

Environment

• tonic-xds 0.1.0-alpha.2 / xds-client 0.1.0-alpha.2
• Istio istiod as the xDS server (proxyless gRPC), SotW ADS v3
• Bootstrap (file-based, not via istio-agent):

{
  "xds_servers": [{
    "server_uri": "https://istiod.istio-system.svc:15010",
    "channel_creds": [{"type": "insecure"}],
    "server_features": ["xds_v3"]
  }],
  "node": {
    "id": "sidecar~10.44.113.2~filtration-67556869bd-2dsxf.default~cluster.local",
    "metadata": {"GENERATOR": "grpc"}
  }
}

Symptom

The client works fine until the backend server deployment does a rolling restart. Once the old pods are replaced, the client never routes to the new pods: every request hangs until the caller's deadline and times out. The client never recovers without a restart.

Root cause analysis

The xDS protocol machinery itself looks correct (ACK/NACK, version/nonce, reconnect-with-resubscribe, EDS-not-full-state per gRFC A53, endpoint diffing, health-status filtering). The failure is that dead connections are never detected, in two places:

1. ADS channel to the xDS server has no keepalives, no read deadline, no connect timeout

TonicTransportBuilder::build (xds-client/src/transport/tonic.rs) constructs the channel with just Endpoint::from_shared(...).connect() — no http2_keep_alive_interval, no keep_alive_timeout, no keep_alive_while_idle, no connect_timeout. The worker reads the stream with a bare stream.recv() inside tokio::select! with no timeout arm (xds-client/src/client/worker.rs, run_connected).

The ADS stream is mostly idle from the client's perspective, so if the connection goes half-open during cluster churn (istiod restart where the GOAWAY/RST is lost, dropped conntrack/NAT entry, LB reset) — tonic gets no error and no bytes, recv() pends forever, and the otherwise-correct reconnect + re-subscribe logic never fires. The client then serves its last-known EDS endpoints indefinitely. Combined with (2), that's a permanent outage.

2. Data-plane endpoint channels can never fail

Per-endpoint channels are created with connect_lazy() and no connect timeout or keepalive (PlaintextConnector / TlsConnector in tonic-xds/src/xds/cluster_discovery.rs). The production balancer is raw tower::balance::p2c::Balance (tonic-xds/src/client/cluster.rs), and a lazy tonic channel's poll_ready reflects buffer capacity, not TCP connectivity — so every endpoint is "ready" the moment it's inserted and stays ready forever. Kubernetes black-holes deleted pod IPs (SYNs dropped, no RST), so a request picked onto a dead endpoint hangs until the caller's deadline. Nothing ejects the endpoint.

Notably, tonic-xds/src/client/loadbalance/ contains a full connectivity-aware load balancer with outlier detection (gRFC A50), but the module is #[allow(dead_code)] and never constructed by the production stack.

3. (Secondary) https:// server_uri + insecure creds is passed through unreconciled

With insecure channel creds no TLS is configured, but ensure_secure_server_uri (xds-client/src/transport/tonic.rs) only adds an https:// scheme on the secure path — it never strips one on the insecure path. So the bootstrap above hands tonic an https URI with no TLS backend. Per xDS bootstrap semantics, channel_creds should decide transport security, not the URI scheme.

Failure sequence

Deployment rolls → old pod IPs become black holes → requests picked onto them hang (no connect timeout, no ejection) → the ADS connection is also disturbed by the churn and goes half-open → no EDS update ever arrives to replace the endpoint set (no keepalive to detect the dead stream) → client permanently routes 100% of traffic to dead IPs.