Symptom (live, 2026-08-22)
On the production dmsg discovery (022e607e…), /debug/log over a 7.6-min window (15:20:20→15:27:58Z):
- 674 of 1376 log lines (49%) are
422: entry validation error: sequence field of new entry is not sequence of old entry + 1 — a sustained ~1.5/sec.
- Only 8 successful
Updating entry in the same window; 301 server-registrations; 5× 500 something unexpected happened.
- The discovery runs at ~half its usual steady-state load, and
server-clients shows 173 "servers" vs 9 registered (164 stale references) — consistent with clients unable to reliably refresh entries, so entries go stale/expire (slow removed 1 stale set member GC).
Mechanism
updateClientEntryOnEndpoint (pkg/dmsg/dmsg/entity_common.go:949) does GET (line 950) → PutEntry. PutEntry (pkg/dmsg/disc/client.go:236) takes updateMux and posts entry.Sequence+1, but the GET is outside updateMux, so the read→write window lets a concurrent trigger (startup synchronous publish at client.go:319, the loop tick, and a session nudge — or a restart whose in-memory Sequence reset to 0) POST a stale sequence → 422. Each rejected POST is a full Noise handshake on the discovery (the exact cost #50's bounded-retry aimed to bound). #50's mitigations (bounded retry + backoff, debounce, updatedWithin/SamePubKeys throttles) reduce but don't eliminate it.
Fix direction
Make the per-entity read-modify-write atomic: hold the update lock across GET→POST (not just the POST), and/or ensure a single in-flight update per entity so the startup publish and loop can't race. Also worth investigating the 5× 500s (real server errors, not sequence conflicts).
Diagnosed via /debug/log + /debug/pprof over the survey-gated debug surface.
Symptom (live, 2026-08-22)
On the production dmsg discovery (
022e607e…),/debug/logover a 7.6-min window (15:20:20→15:27:58Z):422: entry validation error: sequence field of new entry is not sequence of old entry + 1— a sustained ~1.5/sec.Updating entryin the same window; 301 server-registrations; 5×500 something unexpected happened.server-clientsshows 173 "servers" vs 9 registered (164 stale references) — consistent with clients unable to reliably refresh entries, so entries go stale/expire (slowremoved 1 stale set memberGC).Mechanism
updateClientEntryOnEndpoint(pkg/dmsg/dmsg/entity_common.go:949) does GET (line 950) → PutEntry.PutEntry(pkg/dmsg/disc/client.go:236) takesupdateMuxand postsentry.Sequence+1, but the GET is outsideupdateMux, so the read→write window lets a concurrent trigger (startup synchronous publish at client.go:319, the loop tick, and a session nudge — or a restart whose in-memorySequencereset to 0) POST a stale sequence →422. Each rejected POST is a full Noise handshake on the discovery (the exact cost #50's bounded-retry aimed to bound). #50's mitigations (bounded retry + backoff, debounce,updatedWithin/SamePubKeysthrottles) reduce but don't eliminate it.Fix direction
Make the per-entity read-modify-write atomic: hold the update lock across GET→POST (not just the POST), and/or ensure a single in-flight update per entity so the startup publish and loop can't race. Also worth investigating the 5×
500s (real server errors, not sequence conflicts).Diagnosed via
/debug/log+/debug/pprofover the survey-gated debug surface.