network: detect a wedged discv5 socket (operator health check + boot-node /healthz) - #2982
network: detect a wedged discv5 socket (operator health check + boot-node /healthz)#2982iurii-ssv wants to merge 6 commits into
Conversation
Greptile SummaryThe PR adds socket-read tracking so operator and boot-node health checks can detect wedged discv5 listeners.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains established. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| network/discovery/timed_conn.go | Adds an atomic timestamping wrapper around successful discv5 UDP reads. |
| network/discovery/dv5_service.go | Routes the post-fork listener through TimedConn and exposes discovery staleness. |
| network/p2p/p2p.go | Extends operator health reporting to reject stale discovery sockets. |
| utils/boot_node/health.go | Implements boot-node health evaluation from routing-table occupancy and socket-read freshness. |
| utils/boot_node/node.go | Wires TimedConn into the boot-node listener and serves the new health endpoint. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
UDP["discv5 UDP socket"] --> TC["TimedConn records successful reads"]
TC --> OP["Operator DiscoveryStale"]
OP --> PH["p2p Healthy"]
PH --> OW["Operator watchdog restart"]
TC --> BH["Boot-node health check"]
RT["Routing table state"] --> BH
BH --> HZ["/healthz"]
HZ --> BL["Boot-node liveness restart"]
Reviews (2): Last reviewed commit: "utils/boot_node: add /healthz tied to di..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
7bc0d99 to
c9d0cc1
Compare
|
@greptile pls re-review |
There was a problem hiding this comment.
Sound, well-reasoned design: the socket-read timestamp is a genuinely cause-agnostic wedge signal, it is wired to the only listener that actually drains the socket, and greptile's 'quiet socket' P1 is properly fixed by the populated-table gate (verified against geth's 3s revalidation interval, not just the comment thread). The one thing worth changing before merge is the operator path, which cannot distinguish 'wedged' from 'never received any UDP' and will crash-loop a node that restarts while its bootnodes are unreachable; the rest are minor robustness/observability/test-wiring points. [verdict: with_fixes]
c9d0cc1 to
b9f0d08
Compare
b9f0d08 to
a16e6da
Compare
| // loads seed nodes from the persistent enode DB straight into the table, so | ||
| // AllNodes() can be >0 before the socket has been read once — and StaleFor stays | ||
| // disarmed until that first read. Those seeds then fail revalidation and age out, | ||
| // the table empties, and the empty-table grace trips instead. A runtime wedge |
There was a problem hiding this comment.
I fear the fallback described here doesn't hold under a dispatch-loop wedge.
Table entries are only deleted in tableRevalidation.handleResponse, and revalidation itself goes through the wedged dispatch goroutine (initCall blocks on t.callCh, which that goroutine drains), so the seeds never age out, AllNodes() stays >0, lastNonEmpty keeps refreshing, and /healthz stays green indefinitely.
The dispatch-wedge case is still caught (a prior read has armed StaleFor), but a stall where the socket never yields a read (kernel/conntrack-level) lands exactly in populated-table + disarmed-socket = healthy forever. Worth correcting the comment at minimum, or considering a self-probe: write a junk packet to the socket's own LocalAddr() and require LastRead() to advance, which covers it deterministically and independently of peers.
There was a problem hiding this comment.
Partly agreed — the comment now states the residual precisely — but the named case (kernel/conntrack-level stall, socket never read) is caught: under it the dispatch loop is alive, so revalidation pings still time out via dispatch's own timers and failed peers are dropped (livenessChecks /= 3 → delete at 0; DB seeds start at 0), the table empties within minutes, and the empty-table grace trips.
"Seeds never age out" requires dispatch itself to be wedged — but geth's read loop is one-packet-at-a-time flow-controlled (readNextCh), so any packet arriving before/as dispatch wedges arms the staleness check first. The genuinely uncaught combination is: dispatch wedged from boot and zero packets ever read and persistent-DB seeds. No known cause produces it since #2980, so it's documented as an accepted blind spot rather than closed with the self-probe — writing to LocalAddr() of a 0.0.0.0-bound socket is platform-fragile, and it adds a moving part on the hot discovery port for a no-known-cause case. Happy to revisit the self-probe as a follow-up if you feel strongly.
| // A wedged discv5 socket leaves discovery silently dead while bootstrap keeps | ||
| // looping; surface it so the hprobe watchdog restarts the node. n.disc is nil | ||
| // in tests and briefly at startup, hence the guard. | ||
| if n.disc != nil && n.disc.DiscoveryStale(discoveryStaleGrace) { |
There was a problem hiding this comment.
Could this lead to a restart loop on a healthy node that just has no inbound UDP?
After a single successful read ever, 3 minutes of inbound silence fails Healthy: e.g. inbound UDP blocked upstream (firewall/NAT/conntrack change) while the node keeps operating fine over its established libp2p TCP peers; the restart drops every peer connection and can't fix the network.
A total blackhole self-limits (the arm resets on restart), but a partial one the odd scan packet or NAT-refreshed reply arriving each boot re-arms every cycle and loops indefinitely.
Maybe stamp writes in TimedConn too and only declare a wedge when reads AND writes are both stale: all v5 sends funnel through the same dispatch goroutine, so a real wedge freezes both, while an inbound blackhole leaves writes flowing (nursery bootnodes are re-pinged every refresh).
There was a problem hiding this comment.
Addressed via the peer-set gate (details in the sibling thread): staleness alone no longer restarts — only staleness plus a degraded peer set does — so the partial-blackhole loop on a node happily serving over TCP is gone.
Went with impact corroboration rather than write-stamping: all v5 sends funnel through the dispatch loop, so a genuine dispatch wedge freezes writes too (your point), but a stall confined to the read side would keep writes flowing and be missed by a reads-AND-writes condition — and an inbound blackhole also leaves writes flowing, so writes don't separate the two cases we need separated.
| func (h *bootNodeHealth) check() error { | ||
| now := h.now() | ||
| if len(h.lister.AllNodes()) > 0 { | ||
| h.lastNonEmpty.Store(now.UnixNano()) |
There was a problem hiding this comment.
Seems that the empty-table clock is driven by probe arrivals rather than table state: if probing pauses (kubelet restart, or the livenessProbe manifest lands in a later rollout than the image), the first resumed check compares now against a stale observation and can fail with no effective grace.
Restarting a healthy pod. Conversely, a table flapping in and out of empty faster than the probe period resets the clock every time, so the grace never accumulates. Might be worth sampling the table from a small owned ticker and having check() read only the sampled state.
There was a problem hiding this comment.
Done — table state is now sampled by an owned 1s ticker (sample/start), and check() reads only the sampled atomics. The empty-table clock advances with the table itself: a probe pause while the table was populated can no longer turn a stale observation into an instant past-grace verdict, and the 1s granularity removes the probe-instant aliasing on a flapping table.
Side benefit for your hardening thread: /healthz requests no longer touch the discv5 table mutex or allocate. Added TestBootNodeHealth_ClockIsSampleDriven and TestBootNodeHealth_CheckReadsSamplesOnly.
| return false | ||
| } | ||
| age, ok := dvs.socketConn.ReadStaleness() | ||
| recordDiscoveryReadStaleness(dvs.ctx, int64(age.Seconds())) |
There was a problem hiding this comment.
Should we distinguish the never-read state in the gauge?
Right now "read 200ms ago" and "never read since boot" both record 0 and never-read is precisely the state the health check deliberately ignores, so it's the one you'd want visible to a human for alerting.
A sentinel (-1) or a companion read-ever metric would keep it distinguishable. Also, since this only records when Healthy() reaches the staleness check, the series goes quiet whenever isReady/discoveryFailed short-circuit earlier: i.e. exactly when discovery is broken.
There was a problem hiding this comment.
Done, both points. The gauge is now an Int64ObservableGauge registered against the socket at listener init and sampled at scrape time, so the series keeps reporting no matter what the health-check path does — including when isReady/discoveryFailed short-circuit before the staleness step (previously it went quiet exactly when discovery broke). Never-read reports -1 instead of 0, keeping the state the health check deliberately ignores visible for alerting (description updated). DiscoveryStale is a pure predicate again, and the callback registration is released on Close.
| // shows up as a last-read timestamp that stops advancing. StaleFor turns that | ||
| // into a liveness signal, used by both the operator and boot nodes. | ||
| // | ||
| // Only ReadFromUDPAddrPort is overridden; writes, Close and LocalAddr fall |
There was a problem hiding this comment.
Might be worth stating the scope here: only the post-fork listener reads through TimedConn, the pre-fork listener reads SharedUDPConn's buffer, so a pre-fork dispatch stall moves neither health signal.
The known blocking cause is gone post-#2980, but as written the doc reads as if the wedge signal covers discovery generally.
There was a problem hiding this comment.
Done — the doc now scopes the signal: only the post-fork listener reads the socket through TimedConn; the pre-fork listener consumes decode-rejected packets relayed through SharedUDPConn's buffer, so a pre-fork-only stall moves neither the timestamp nor the socket — and since the relay drops on a full buffer (#2980), it can't back up into the post-fork reader either.
| func (h *bootNodeHealth) handler() http.HandlerFunc { | ||
| return func(w http.ResponseWriter, _ *http.Request) { | ||
| if err := h.check(); err != nil { | ||
| http.Error(w, err.Error(), http.StatusServiceUnavailable) |
There was a problem hiding this comment.
Do we need /healthz unauthenticated on the public ENR-advertised port?
Every request takes the discv5 table mutex via AllNodes() and allocates the full node slice, there's no method filtering, and the error body discloses internal state.
Maybe memoize check() for ~1s, reject non-GET/HEAD with 405, and return a fixed body with the reason logged instead — or serve it on a pod-internal listener.
There was a problem hiding this comment.
Done, proportionately: non-GET/HEAD now gets 405 (on both /healthz and /p2p), the 503 body is a fixed unhealthy with the reason logged rather than disclosed, and the per-request cost is gone — check() reads sampled atomics only (see the sampler thread), so no memoization is needed.
Kept the port: the AWS LB check and the charts livenessProbe target it, and /p2p on the same mux — which dumps the entire table to anyone — is the materially larger pre-existing surface; worth its own follow-up if we want to lock that down.
ovidiu-ssv-labs
left a comment
ovidiu-ssv-labs
left a comment
There was a problem hiding this comment.
🔁 Re-review vs c9d0cc1: 2 fixed, 1 partial, 0 still open, 3 new. All three prior findings were addressed — the never-read arming guard, the ListenV5 wiring test, and the staleness gauge + warn log are all in and the packages build and test green. Two things remain: the operator's arming latch is permanent, so momosh-ssv's restart-loop case (blocked inbound UDP after a prior read) is genuinely still open; and the boot node's empty-table branch ignores the socket signal, turning a discv5 protocol-ID/config mismatch into a permanent CrashLoopBackOff that also destroys the /p2p debug surface. The core design — a cause-agnostic socket-read timestamp wired to the only listener that actually drains the socket — is sound and well documented. [verdict: with_fixes]
| // A wedged discv5 socket leaves discovery silently dead while bootstrap keeps | ||
| // looping; surface it so the hprobe watchdog restarts the node. n.disc is nil | ||
| // in tests and briefly at startup, hence the guard. | ||
| if n.disc != nil && n.disc.DiscoveryStale(discoveryStaleGrace) { |
There was a problem hiding this comment.
Finding 1 · [IMPORTANT] [still open] The operator's arming latch is permanent, so lost inbound UDP still produces a self-sustaining restart cycle
Status: independently verified as still open. This confirms momosh-ssv's question on this line, with the mechanism traced end to end.
Mechanism. read is a one-way latch with no reset path:
// network/discovery/timed_conn.go:50-57
func (c *TimedConn) ReadFromUDPAddrPort(b []byte) (n int, addr netip.AddrPort, err error) {
n, addr, err = c.UDPConn.ReadFromUDPAddrPort(b)
if err == nil {
c.lastReadUnixNano.Store(c.nowFn().UnixNano())
c.read.Store(true) // set once, never cleared for the process lifetime
}
return n, addr, err
}Once set, ReadStaleness() returns ok=true forever, so Healthy() at p2p.go:568 fails on any 3-minute inbound-UDP drought, regardless of cause. startHealthProber treats a failed round as terminal — it returns an error, the node exits non-zero, and the orchestrator restarts it.
Why the fix for prior finding 1 does not cover this. The never-read guard only exonerates a process that has never read a packet. A node running normally for hours has read=true. If inbound UDP then stops — upstream firewall/security-group change, NAT/conntrack rotation killing reply flows, an ISP-level UDP filter — the node keeps operating perfectly over its established libp2p TCP peer set, but exits within ~5 minutes.
Why it self-sustains rather than self-limiting. The stamp is taken at the socket read, before any decode: geth's UDPv5.readLoop reads every datagram on the port and only then attempts decode. So a port-scan probe, a stray discv4 packet, or a single NAT-refreshed reply is enough to flip read to true. Anything short of a perfect blackhole re-arms every boot, settling into a ~5-minute exit cycle.
Why it matters. Each cycle tears down every libp2p connection, forces full re-handshake/re-subscription, and risks missed attestation/sync-committee duties — direct penalty exposure on a validator node. The restart cannot fix an upstream UDP filter, so the loop is unbounded.
On the suggested fix in the thread (stamp writes). Does not discriminate — in the #2980 wedge, discv5 keeps sending (dispatch retries) even though reads are stale; in the blocked-inbound-UDP case, discv5 also keeps sending. Writes advancing with reads stale is the signature of both.
What does discriminate: not the socket — the impact. A genuine discv5 wedge progressively costs the node peers (churn with no replacement); an upstream UDP filter on a node with a warm, healthy TCP peer set does not.
Suggested fix: Require corroborating evidence that the node is actually impaired before restarting:
if n.disc != nil && n.disc.DiscoveryStale(discoveryStaleGrace) {
peers := len(n.host.Network().Peers())
if peers < n.cfg.MinPeers {
n.logger.Warn("discv5 socket wedged and peer set degraded", ...)
return fmt.Errorf("discv5 socket not drained for >%s and only %d peers (discovery wedged)", discoveryStaleGrace, peers)
}
n.logger.Error("discv5 socket not drained, but peer set still healthy — not restarting", zap.Duration("grace", discoveryStaleGrace), zap.Int("peers", peers))
}Alternative: raise discoveryStaleGrace substantially (discv5 revalidates at a 3s PingInterval, so even 30 minutes is a 600x margin) and cap self-inflicted restarts. Either way, add a test pinning that a permanent external UDP fault does not restart-loop.
There was a problem hiding this comment.
Done — adopted the impact gate. Healthy() now treats a stale socket as fatal only when the connected-peer count is below a floor (discoveryStalePeerFloor = 10); at or above it the node logs an error and stays up on its established connections, failing the probe later only if they erode. So a permanent external UDP fault on a working node no longer restart-loops — the loop needed the unconditional exit, and that path now requires corroborating impact.
Agreed on write-stamping: it doesn't discriminate — a stall confined to the read side keeps writes flowing and would slip past a reads-AND-writes-stale condition, and the blocked-inbound case keeps writes flowing too. Peer count keys on impact regardless of wedge shape.
There's no cfg.MinPeers, so the floor is a named constant with the rationale in its doc. Added the test you asked for — discovery wedged but peer set healthy → probe passes — plus a boundary case at floor−1 and a nil-host guard case.
| } | ||
| return nil | ||
| } | ||
| if emptyFor := now.Sub(time.Unix(0, h.lastNonEmpty.Load())); emptyFor > h.emptyTableGrace { |
There was a problem hiding this comment.
Finding 2 · [IMPORTANT - debatable] Boot node: the empty-table branch ignores the socket signal, so a protocol-ID/config mismatch becomes a permanent CrashLoopBackOff
Mechanism. check() splits on table population; the empty-table branch never consults h.socket:
// utils/boot_node/health.go:74-87
if len(h.lister.AllNodes()) > 0 {
h.lastNonEmpty.Store(now.UnixNano())
if h.socket.StaleFor(h.readStaleGrace) { ... } // socket consulted
return nil
}
if emptyFor := now.Sub(...); emptyFor > h.emptyTableGrace {
return fmt.Errorf("discv5 routing table empty for >%s", h.emptyTableGrace) // socket ignored
}So AllNodes() == 0 for 10 minutes fails closed even when the socket is demonstrably being drained.
Concrete break. The boot node's table is populated exclusively by inbound traffic (no Bootnodes configured), and geth only adds an inbound peer after a completed handshake; a V5ProtocolID mismatch (wrong Network option, or a protocol-ID rollout landing on the boot node ahead of the fleet) makes every packet decode-fail before reaching the table. The socket is read normally (TimedConn is fresh), but AllNodes() stays 0 forever. /healthz then 503s every 10 minutes and the livenessProbe restarts the pod indefinitely — a restart cannot fix a config mismatch.
Why this is worse than the pre-PR silent failure. The boot node's only diagnostic surface is /p2p on the same HTTP server. Under CrashLoopBackOff that endpoint is unreachable for most of the cycle, so the operator is left with a flapping pod and no way to observe why the table is empty. A fresh deployment whose ENR hasn't been distributed yet hits the same flap, and CrashLoopBackOff backoff (up to 5 min) means it may be down at the moment the first cold-bootstrapping node arrives — the exact failure #2979 exists to prevent.
Ruled out: 'populated table full of dead peers during a network outage' does NOT false-positive — geth's tableRevalidation drains failed peers well inside the 3-minute read grace. Only the empty-table branch has this gap.
Suggested fix: Fold the socket signal into the empty-table branch — fail closed only when BOTH the table is empty AND the socket is undrained:
if emptyFor := now.Sub(time.Unix(0, h.lastNonEmpty.Load())); emptyFor > h.emptyTableGrace {
if age, everRead := h.socket.ReadStaleness(); everRead && age <= h.readStaleGrace {
h.logger.Error("discv5 routing table empty while socket is being drained — check DiscoveryProtocolID / network config", zap.Duration("empty_for", emptyFor), zap.Duration("read_age", age))
return nil
}
return fmt.Errorf("discv5 routing table empty for >%s and socket undrained", h.emptyTableGrace)
}Needs socketDrainState widened to expose ReadStaleness() (time.Duration, bool) — *TimedConn already has it. Add two test cases: empty table + fresh socket past the empty grace (healthy, with error log), and empty table + never-read past the empty grace (503).
There was a problem hiding this comment.
Done — the empty-table branch now consults the socket: past the grace it fails closed only when the socket is undrained (never read, or read then gone stale); when the socket is actively drained it stays healthy and logs an error pointing at DiscoveryProtocolID/network config, since that state isn't restart-fixable and crash-looping would also take down the /p2p debug surface. socketDrainState widened to ReadStaleness(). Both suggested tests added — empty+drained past grace → 200 with the error log (asserted via a zap observer), empty+never-read past grace → 503 — plus empty+stale → 503.
One caveat to keep on record: a mismatch-broken boot node now stays green on /healthz, so anything routing on it would keep the node in rotation. If that matters we can split liveness from readiness (/readyz: empty table → out of rotation) as a follow-up; /healthz here carries k8s liveness semantics, which is what this PR's infra chain consumes.
Async sampling keeps the series reporting even when Healthy short-circuits before the staleness check, and -1 keeps never-read distinguishable from just-read. Also scope the TimedConn doc to the post-fork listener, and deduplicate the test host address (goconst).
…egraded Staleness alone can't tell a wedged read loop from inbound UDP lost upstream, which a restart can't fix — and any stray packet re-arms the check after boot, so restarting unconditionally would turn one external UDP fault into a fleet-correlated restart loop. Requiring a degraded peer set keeps every real wedge in scope: discovery going dead erodes the peer set, which then trips the probe.
…t is undrained An empty table past grace with an actively drained socket is a config/compatibility mismatch (e.g. DiscoveryProtocolID) a restart can't fix, so it's logged instead of crash-looping the pod. Table state is now sampled by an owned ticker, making the empty-table clock probe-independent and /healthz requests cheap; both endpoints are GET/HEAD-only and the 503 body is fixed with the reason logged. Also lowercase the remaining log messages.
3 participants
Problem
A discv5 socket can wedge — stay bound but stop being drained — leaving discovery silently dead while the process still looks healthy. #2979 documents a boot node that sat like this for ~20 days undetected. Neither node type surfaces it today: the operator's
p2pNetwork.Healthy()only checks a discovery-bootstrap flag (a runtime wedge never trips it, since the bootstrap loop keeps running and just yields nothing), and the boot node's HTTP handler always returns200.#2980 removes one cause of the wedge on the operator (the blocking
Unhandledsend); this PR adds the missing detection, so a wedge from any cause becomes a self-correcting restart.Approach
One shared, cause-agnostic signal — "how long since the discv5 socket was last read" — with actuation scoped per node type.
TimedConn(network/discovery): wraps the socket and stamps the last successful read;StaleFor(d)is the wedge signal, armed only once the socket has been read at least once (a node that never had inbound UDP isn't wedged — a restart can't fix that). Both node types wrap the conn they hand toListenV5. Read staleness is also exported as an async gauge (ssv.p2p.discovery.socket.read_staleness, sampled at scrape time;-1= never read).DiscoveryStalefeedsp2pNetwork.Healthy(), and the existinghprobewatchdog restarts the node on a wedge — but only while the connected-peer set is degraded (< 10 peers). Staleness alone can't distinguish a wedged read loop from inbound UDP lost upstream, which a restart can't fix; a well-connected node logs an error and stays up, tripping the probe later only if its peers erode. No routing-table check here — a stale-but-populated table would mask it on an operator./healthzreturns non-200 when the socket has gone unread while the routing table is populated (discv5 revalidates its peers, so a populated-but-unread socket is a wedge), or when the table has been empty past a cold-start grace while the socket is undrained. Empty-table is the boot node's definitional health (the 0-vs-72/102 datapoint in boot-node: discv5 can wedge silently — no health signal tied to discovery actually working #2979) — with one deliberate exception: an empty table while the socket is actively drained is a config/compatibility mismatch (e.g.DiscoveryProtocolID) a restart can't fix, so it stays 200 with a loud error log instead of crash-looping the pod. Table state is sampled by an owned 1s ticker (probe-independent empty-table clock, cheap requests); both HTTP endpoints are GET/HEAD-only and the 503 body is fixed, with the reason logged.Grace values: 3 min read-staleness (both node types); 10 min empty-table cold-start and 1s table sampling (boot node); 10-peer floor gating the operator restart.
Notes
initDiscV5Listener, which network/discovery: stop undecodable packets from wedging discv5 #2980 rewrote); network/discovery: stop undecodable packets from wedging discv5 #2980 has merged and this PR targetsstagedirectly./debug/pprof, and a secondsepoliaboot node.livenessProbeon/healthzis added by the downstream PRs below.Tests
TimedConn: seeded-not-stale, never-read-never-stale, stale boundary (injected clock, no real sleeps), read-stamps, errored-read-doesn't-stamp, gauge flattening (-1sentinel).DiscoveryStale;TestP2PNetwork_Healthycovers the wedge across peer-set states — degraded / nil-host / floor-boundary fail the probe, wedged-but-well-connected passes it (no restart loop on an external UDP fault) — plusready with live discovery; the existing nil-disccases stay green.TestInitDiscV5Listener_WrapsPostForkConnpins the wiring the detector rests on./healthzacross populated+fresh, populated+wedged, empty-within-grace (incl. the quiet-socket regression), and empty-past-grace undrained (503) vs drained (200 + error log, asserted via a zap observer); the sampler clock is probe-independent andcheckreads samples only; method filter (405, HEAD ok) and fixed 503 body.Closes #2979. Full discovery + p2p + boot_node suites pass, discovery also under
-race.Downstream infra PRs for the boot-node side of this (what makes the boot node self-healing)
livenessProbeto theboot-nodechart (inert by default)./healthz).Merge order: #2980 → this → build image → ssvlabs/charts#183 → ssvlabs/gitops-production#881.