feat(cxo): consolidate tp-list onto the telemetry feed (one visor↔TPD connection) - #4184
Merged
Merged
Conversation
… connection) The dedicated tp-list discovery feed (skycoin#4152, a second CXO node on DmsgVisorTPListCXOPort) meant every visor held TWO CXO connections to TPD — one per aggregator — doubling TPD's per-conn goroutines and the per-session dmsg handshake load on the dmsg servers. Publish the tp-list leaf on the telemetry feed by default so a visor holds ONE CXO node/conn to TPD. TPD needs no change: its port-50 aggregator's targeted discovery-leaf fetch already extracts the small tp-list leaf from the combined feed, and the port-69 aggregator keeps serving visors still on the dedicated feed during rollout. Safe to retry now that skycoin#4179 fixed the big-Root fill that forced the earlier revert (skycoin#4174 of skycoin#4171): the flat 90s whole-Root fill cap is replaced by a stall timer that resets on every fetched object plus a hard ceiling, so a large-but-progressing combined Root completes instead of breaking mid-fill. Measured churn on the busy dev hub dropped to ~0.86%/interval. Discovery itself does not wait on that fill — the targeted tp-list fetch lands the inlined leaf independently. Reversible: Stats.DedicatedTPListFeed=true restores the separate dedicated feed if a busy hub is ever seen under-filling on the combined feed.
0pcom
added a commit
that referenced
this pull request
Aug 25, 2026
…e busy-hub Root) (#4190) Replace the N per-transport transports/<uuid>/current JSON telemetry leaves with 16 fixed sharded binary leaves at transports/telemetry/<sh> (shard = uuid[0]>>4). A busy hub's ~851 per-transport leaves (~1700 CXO objects) collapse to a handful, so TPD's whole-Root fill completes over the short announce conn instead of undercounting. The byte layout lives in a single shared codec (pkg/telemetrywire) imported by both the visor publisher (pkg/visor/stats) and the TPD reader (pkg/deployment/tpd/cxoaggregator) so the two sides can't drift. Version byte 0x02 gates the format; entries are 53 bytes and carry the transport's passively-observed peak-goodput throughput estimate (GetThroughputBps) alongside bandwidth/latency, which TPD records per-transport. TPD reads the shards and applies each row exactly as a current snapshot; it keeps the old transports/<uuid>/current path as a fallback for not-yet-upgraded visors, preferring shards when a Root carries them. Deploy is TPD-FIRST: TPD (shards + old-current fallback) must ship before visors switch to shard-only publishing. Also re-consolidates the tp-list discovery leaf onto this telemetry feed (one visor<->TPD CXO connection), redoing the reverted #4184 correctly: the now-small sharded Root is what makes the single combined feed fillable. The dedicated port-69 feed remains available behind Stats.DedicatedTPListFeed as a transitional escape hatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishes the tp-list discovery leaf on the visor's telemetry CXO feed by default, so a visor holds ONE CXO connection to TPD instead of two (the dedicated port-69 tp-list feed from #4152 was a second node/conn under the same PK). This halves each visor's TPD connection count and the per-session dmsg handshakes.
TPD needs no change: its port-50 aggregator already extracts the small tp-list leaf via its targeted discovery-leaf fetch (runs on every OnRootReceived, independent of the whole-Root telemetry fill), and the port-69 aggregator keeps serving visors still on the dedicated feed during rollout.
Safe to retry now: this reverts the revert (#4174) of #4171. That revert happened because a busy hub's big combined Root couldn't finish its fill under the old flat 90s cap, so TPD's transport count churned. #4179 replaced that cap with a stall timer that resets on every fetched object plus a hard ceiling, so a large-but-progressing Root now completes — measured churn on the busy dev hub dropped to ~0.86%/interval. Discovery does not even wait on that fill; the targeted tp-list fetch lands the inlined leaf in a handful of objects.
Escape hatch retained:
Stats.DedicatedTPListFeed=truerestores the separate dedicated feed without a redeploy if a busy hub is ever seen under-filling on the combined feed.Depends on / unblocked by #4179. Live busy-hub agreement will be confirmed post-deploy.