Fetching a dmsg server’s /health over dmsg is far slower than a deployment service’s, and the slowest one gates Visor.ServiceHealth().
Measured (live, from a visor’s survey proxy)
| dmsg server |
host |
warm /health |
0371ab4bcff7… |
45.79.213.251 (dmsgd/SD/reward) |
steady ~5.7s |
0281a102c828… |
139.162.160.227 (RF/AR/TPD) |
~15.8s |
0326978f5a53… |
standalone edge |
12s cold → 0.6s warm (route setup) |
Deployment services (Config/TPD/DMSGD/AR/RF/SD) answer /health in ~300ms by comparison. None of the servers are down — all return 200 with correct versions; they are just slow to answer.
Why
A dmsg server serves /health on a secondary transit dmsg client (an extra relay hop), not its primary relay listener. The two slow servers are colocated on the busy deployment-service hosts (the dmsg-server consolidation), so that secondary path competes with RF/AR/TPD/CXO/reward on those loaded boxes. Part of the first-hit cost is also cold route setup (see the edge server, 12s→0.6s).
Impact
ServiceHealth() fans every probe out in parallel and wg.Wait()s, so the slowest server gated the whole call — and the curated visor state snapshot that folds it in — at ~10s. Under load that tipped over callers’ RPC deadlines and the snapshot came back empty, which reads downstream as "zero routes". Bounded in #4241 (per-probe 4s timeout), but that only caps the blast radius: slow servers now show no version, and the underlying latency is unaddressed.
Suggested directions
- Cache the per-server version fetch (it changes only on deploy) so
ServiceHealth never re-probes it in the hot path.
- Investigate why the colocated transit-client
/health is starved on the loaded hosts — is the transit client under-prioritised, or is /health doing real work?
- Consider serving dmsg-server
/health on a faster/primary path, or pre-warming routes to server transit clients.
Fetching a dmsg server’s
/healthover dmsg is far slower than a deployment service’s, and the slowest one gatesVisor.ServiceHealth().Measured (live, from a visor’s survey proxy)
/health0371ab4bcff7…0281a102c828…0326978f5a53…Deployment services (Config/TPD/DMSGD/AR/RF/SD) answer
/healthin ~300ms by comparison. None of the servers are down — all return 200 with correct versions; they are just slow to answer.Why
A dmsg server serves
/healthon a secondary transit dmsg client (an extra relay hop), not its primary relay listener. The two slow servers are colocated on the busy deployment-service hosts (the dmsg-server consolidation), so that secondary path competes with RF/AR/TPD/CXO/reward on those loaded boxes. Part of the first-hit cost is also cold route setup (see the edge server, 12s→0.6s).Impact
ServiceHealth()fans every probe out in parallel andwg.Wait()s, so the slowest server gated the whole call — and the curatedvisor statesnapshot that folds it in — at ~10s. Under load that tipped over callers’ RPC deadlines and the snapshot came back empty, which reads downstream as "zero routes". Bounded in #4241 (per-probe 4s timeout), but that only caps the blast radius: slow servers now show no version, and the underlying latency is unaddressed.Suggested directions
ServiceHealthnever re-probes it in the hot path./healthis starved on the loaded hosts — is the transit client under-prioritised, or is/healthdoing real work?/healthon a faster/primary path, or pre-warming routes to server transit clients.