test(inspector): probe the fan-out under a paused clock - #4241
Draft
haiyuechen-nearone wants to merge 15 commits into
Draft
test(inspector): probe the fan-out under a paused clock#4241haiyuechen-nearone wants to merge 15 commits into
haiyuechen-nearone wants to merge 15 commits into
Conversation
Spawns the probe detached, so it reports which network each provider serves without delaying startup or gating anything. A status carries no provider text and no auth material, so it is logged whole.
Publishes `mpc_foreign_chain_rpc_providers_configured` and `mpc_foreign_chain_rpc_providers_healthy`, labelled by chain rather than by provider, since a provider name is operator chosen and would put an unbounded label on a time series. `ForeignChain::label()` is the one place the chain label lives, so a metric label and a config key cannot drift apart. An e2e test points one node at a mock serving the expected chain id and another at the same mock while expecting a different network, so the gauges separate a healthy provider from one on the wrong network end to end. The EVM mock answers `eth_chainId` to make that possible. Leaves a provider no probe covers out of the `x/y providers healthy` summary. It counted toward the denominator but could never reach the numerator, so a node configuring a chain without a probe read as permanently degraded.
A chain no probe covers no longer publishes gauges, and an empty foreign chains config no longer warns on every boot. The panic guard is gone: the deployed binary aborts on panic, so it never ran. The summary is a value now, asserted directly instead of through log lines, and a node config test pins every chain's config key to its label.
A node is expected to configure the chains the code supports, so an empty section is a misconfiguration rather than a quiet path. The startup probe is the only place that observes it.
The warning named our own mechanism and left the operator to work out which of their chains went unchecked; it now lists them. The docs claimed a `ton` section exists, and that chains without an inspector report a status name an operator never sees.
`ProbeReport` gains `From<Vec<ProviderHealth>>`, so the gauge tests no longer open sockets and drive another crate's probe just to reach this crate's publication. They now cover several chains in one report, and pin the label of the one chain whose key is not its variant name. Log lines carry the chain's config key rather than its variant name, and a healthy or uncheckable provider logs at info, so the documented line per provider appears at the level operators run. Also drops metric labels from the truncation rationale: no metric carries a fingerprint.
Keep only the claim the fixture does not make, and leave the Given and When markers bare.
Covers the avalanche and adi sections main added to ForeignChainsConfig, and applies review feedback from the earlier PRs in the stack: named structs in place of positional tuples in the probe tests, bare Given/When/Then markers, and comments that no longer restate the code.
check_all_providers is slated for removal, so routing its chain names through ForeignChain::label was churn in a file on its way out. Also drops two comments that restated the code beside them.
haiyuechen-nearone
force-pushed
the
4043-paused-clock-fanout-tests
branch
from
August 25, 2026 17:01
a2e2273 to
406fbe2
Compare
haiyuechen-nearone
changed the base branch from
3764-node-startup-probe
to
4043-inject-probe-inspectors
August 25, 2026 17:01
Closes #4043 The probe built its own RPC clients, so its tests stood up HTTP servers and waited on the wall clock: a second for the timeout case and a fifth of a second per retry case, with no way to go faster while the timeout is configured in whole seconds. The node's startup path now hands the inspector constructor in, and a scripted inspector answers those cases from a queue of replies whose delays are virtual timers. Reaching that seam needed the chain dispatch untangled first. Dispatch, client construction and fingerprint canonicalization were spread across a thirteen arm match; one registry entry per chain now binds a chain's canonical form to the constructor of its inspector, so the two cannot drift apart, and the inspectors share one type. Canonicalizing an operator written fingerprint is a property of the chain rather than of an inspector value, so it moves off NetworkFingerprintInspector, leaving the trait with the single capability it is named for. The tests that still exercise real clients keep covering the wiring the scripted ones cannot: auth splicing, fingerprint bounding and the per chain requests.
haiyuechen-nearone
force-pushed
the
4043-paused-clock-fanout-tests
branch
from
August 25, 2026 17:16
406fbe2 to
53fadf2
Compare
The chain registry paired each chain with a canonicalizer, which only existed because an aggregate inspector could not answer for a chain it was not told about. Taking the fingerprint by reference to self lets every inspector answer for itself, so the aggregate delegates like it does for the fingerprint it reports, the registry drops to the constructor alone, and a chain cannot be paired with another chain's normalization. The EVM constructors share one generic function instead of a macro; the variant each arm names fixes the marker type.
haiyuechen-nearone
force-pushed
the
4043-paused-clock-fanout-tests
branch
from
August 25, 2026 21:24
53fadf2 to
5d522c6
Compare
The startup probe and the transaction validation path each turned a provider's config into a client: resolve the auth token, splice `Path` and `Query` auth into the URL, then either build the JSON-RPC client or peel the header off for the Aptos and Sui clients. Two implementations of the same three constructors, with the header case written out four times. They now share one set, next to the auth conversion they all start from, so the code that carries a token to the wire has a single definition to read and to keep from leaking it.
haiyuechen-nearone
force-pushed
the
4043-paused-clock-fanout-tests
branch
from
August 25, 2026 21:44
5d522c6 to
7ae915f
Compare
The crate turns a provider's configuration into a client that can reach it. Auth is a step in that, not the whole job, so the name said less than the crate did once the client constructors moved in. `RpcAuthentication` and `build_http_client` follow it. Neither inspects a chain, and the inspector crate never used them: they describe how to reach a provider, which is what this crate is now for. The inspector crate is left with the chain protocols alone.
The fan-out's own retry, backoff and timeout tests waited on the wall clock through a real socket. Driving them with the scripted inspector pins the same behaviour deterministically, and lets the assertions state the exact elapsed time.
haiyuechen-nearone
force-pushed
the
4043-paused-clock-fanout-tests
branch
from
August 25, 2026 22:22
b6ee447 to
90574ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4043. Stacked on #4071.
The fan-out's retry, backoff and timeout tests waited on the wall clock through a real socket. A scripted inspector answers from a queue whose delays are virtual timers, so the same behaviour is pinned deterministically and the assertions can be exact.
Notes for review
test-utilsfeature, so the probe's tests can use it from the next PR in the stack.