Skip to content

refactor: inject the inspectors into RPC probing path - #4244

Merged
haiyuechen-nearone merged 14 commits into
mainfrom
4043-inject-probe-inspectors
Sep 1, 2026
Merged

refactor: inject the inspectors into RPC probing path#4244
haiyuechen-nearone merged 14 commits into
mainfrom
4043-inject-probe-inspectors

Conversation

@haiyuechen-nearone

@haiyuechen-nearone haiyuechen-nearone commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Closes #4043.

probe_all_providers now takes its inspectors as a dependency via the BuildInspectors trait. Production path uses a new struct InspectorFactory and tests can re-implement this trait and utilize MockInspector

foreign-chain-rpc-auth crate is renamed foreign-chain-rpc-factory, since it now includes code to resolve credentials and build inspectors.

SIde notes:

  • timeout_of moved onto ForeignChainConfig::timeout_duration().
  • My intension is that foreign-chain-rpc-factory will contain all client and inspector construction methods. To restrict the scope of this PR I purposefully tried not to touch the TX verification route.

@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch from df5f6e2 to 2b5d1b2 Compare August 26, 2026 13:18
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch from 2b5d1b2 to 83aaaea Compare August 26, 2026 13:30
@haiyuechen-nearone haiyuechen-nearone changed the title refactor(probe): inject the inspectors into the startup probe refactor(probe): inject the inspectors into the foreign chain probe Aug 26, 2026
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch from 712f054 to 788dd53 Compare August 26, 2026 13:53
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch 2 times, most recently from 04e2103 to 5e32044 Compare August 27, 2026 17:47
@haiyuechen-nearone haiyuechen-nearone changed the title refactor(probe): inject the inspectors into the foreign chain probe feat: inject the clients a chain's inspector is built over Aug 27, 2026
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch from 0931f52 to 1f6729a Compare August 27, 2026 21:59
@haiyuechen-nearone haiyuechen-nearone changed the title feat: inject the clients a chain's inspector is built over feat: inject the inspectors into the startup probe Aug 27, 2026
Base automatically changed from 3764-node-startup-probe to main August 28, 2026 10:08
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch from 28d6bd7 to d8d8a48 Compare August 28, 2026 10:08
BuildRpcClients is the seam: choosing it chooses the transport, and by
extension every inspector built over it, so a caller can put the whole set
on clients it controls. RpcClients reaches the real providers.

Nothing uses it yet. The probe and transaction verification each build their
own inspectors today, and move across separately.
Building the client that talks to a provider and building the inspector over
that client are separate concerns, so they get a module each. The crate
resolved credentials and nothing else, so it is renamed for what it builds now.
Free functions can only be called. As trait methods they can be implemented,
so a caller can answer for the inspectors themselves rather than only for the
transport beneath them.

A blanket implementation over BuildRpcClients keeps both levels open: choosing
the clients settles the inspectors, and a caller that needs to supply its own
implements BuildInspectors directly and builds no client at all.
A caller that spans chains, as the probe does, needs a single type to hold
them behind. BuildInspectors::any answers with it, or with None where no
inspector exists, and its match is the only list of covered chains.

Reaching that needed canonical_fingerprint to take self: with no receiver
there is nothing to dispatch on, so no erased inspector could implement the
trait. Every implementation already ignored the distinction.
probe_all_providers takes a BuildInspectors, so the chain dispatch it used
to carry moves behind the factory and its coverage becomes one list rather
than two. InspectorFactory holds the clients it builds over, so choosing
the clients and choosing the inspectors stay separate decisions.

The tests that were bounded by the wall clock now script an inspector under
paused time, taking the timeout case from a second to microseconds. The ones
that still stand up a server keep covering what a script cannot: real request
shapes, auth splicing and client setup.
BuildInspectors is now the only seam: it lives in foreign-chain-inspector, and RpcInspector is a concrete enum over the real transports. The BuildRpcClients trait and RpcClientFactory had no consumers and only made the transport a detail the caller had to pick; InspectorFactory becomes a unit struct. The verify path's duplicated construction is left for a follow-up.
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 4043-inject-probe-inspectors branch from d8d8a48 to 9a40dfe Compare August 28, 2026 13:20
@haiyuechen-nearone haiyuechen-nearone changed the title feat: inject the inspectors into the startup probe refactor: inject the inspectors into the startup probe Aug 28, 2026
@haiyuechen-nearone haiyuechen-nearone changed the title refactor: inject the inspectors into the startup probe refactor: inject the inspectors into RPC probing path Aug 28, 2026
@haiyuechen-nearone
haiyuechen-nearone marked this pull request as ready for review August 28, 2026 13:38
@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown

Pull request overview

probe_all_providers no longer hard-codes per-chain inspector construction: it takes a BuildInspectors dependency, with InspectorFactory in production and a MockInspector-backed impl in tests. To make one generic probe path possible, canonical_fingerprint becomes an instance method and a new RpcInspector enum unifies the per-chain inspector types. foreign-chain-rpc-auth is renamed foreign-chain-rpc-factory to host the factory next to the existing credential resolution, and timeout_of moves onto ForeignChainConfig::timeout_duration(). Closes the TODO for issue 4043 on probe_all_providers.

Changes:

  • New BuildInspectors trait plus ChainInspector bound alias; probe_all_providers/probe_chain generic over it, 13-arm match chain gone from probe.rs.
  • canonical_fingerprint from associated fn to &self, across every inspector impl and the new RpcInspector dispatch.
  • New mock module behind a test-utils feature; four httpmock probe tests replaced by in-process doubles on a paused clock.
  • Crate rename plus inspectors::InspectorFactory; ForeignChainConfig::timeout_duration() replaces the private timeout_of.

Reviewed changes

Per-file summary
File Description
Cargo.toml / Cargo.lock Workspace member and dependency rename
foreign-chain-inspector/Cargo.toml test-utils feature, anyhow, mpc-node-config
foreign-chain-inspector/src/lib.rs canonical_fingerprint(&self, ..); ChainInspector and BuildInspectors; new module decls
foreign-chain-inspector/src/rpc_inspector.rs RpcInspector enum, one variant per probeable chain
foreign-chain-inspector/src/mock.rs MockInspector/MockReply double with a call counter
aptos, bitcoin, evm, starknet, sui inspector.rs Receiver change only
foreign-chain-rpc-factory/Cargo.toml Rename; new deps plus two dev-dependencies
foreign-chain-rpc-factory/src/lib.rs Declares inspectors; drops rationale comments from four tests
foreign-chain-rpc-factory/src/inspectors.rs New InspectorFactory
foreign-chain-health-check/Cargo.toml Dep rename; dev-dep on test-utils
foreign-chain-health-check/src/lib.rs Uses cfg.timeout_duration()
foreign-chain-health-check/src/probe.rs Factory injected; Ok(None) handling; tests split between mock and factory
node-config/src/foreign_chains.rs Adds timeout_duration()
node Cargo.toml, foreign_chain_probe.rs, verify_foreign_tx.rs Wiring

Findings

Blocking (must fix before merge):

  • crates/foreign-chain-rpc-factory/src/inspectors.rs:29 - credentials are resolved before the chain-support check, so an unprobeable chain can now be reported as unhealthy. Self::authenticate(provider)? runs ahead of match chain, so for Solana/Ton/Fogo a provider whose TokenConfig::Env var is unset (or whose AuthConfig::Query URL is unparseable) yields Err instead of Ok(None). probe_chain turns that into AuthTokenUnresolved/ClientSetupFailed rather than ProbeNotImplemented, which makes the chain count as probed at crates/node/src/foreign_chain_probe.rs:47. Result: a FOREIGN_CHAIN_RPC_PROVIDERS_HEALTHY gauge of 0 for solana and a warn! every tick - precisely what publish_metrics__should_publish_no_counts_for_an_unprobeable_chain (foreign_chain_probe.rs:249) exists to prevent. Before this PR the _ => rows_of(.., ProbeNotImplemented) arm was unconditional. Solana/Ton/Fogo credentials are resolved nowhere else, so an unset env var there is exactly the case that stays silent until the probe misreports it. Return Ok(None) from chain alone before authenticating, and add a regression test (unprobeable chain plus unresolvable auth gives ProbeNotImplemented); none of the current tests cover it.

  • crates/foreign-chain-health-check/src/probe.rs:125 - same root cause, surfacing as order-dependence: Ok(None) returns from inside the provider loop and discards the rows gathered for earlier providers. A chain with one resolvable and one unresolvable provider reports ProbeNotImplemented for both, while the same chain with only the failing provider reports AuthTokenUnresolved - the verdict depends on BTreeMap provider-name ordering. Deciding Ok(None) from chain alone removes the mixed case; hoisting the check out of the loop would make it structurally impossible.

  • crates/foreign-chain-rpc-factory/Cargo.toml:18-19 - near-mpc-bounded-collections and tokio are added to the dev-dependencies table, but nothing in the crate references them (src/lib.rs tests are all sync, and src/inspectors.rs has no tests). cargo make check-all-fast runs cargo shear --deny-warnings (Makefile.toml:67), invoked by CI at .github/workflows/ci.yml:260, so this should fail the build. Drop both, or add the tests they were added for.

Non-blocking (nits, follow-ups, suggestions):

  • crates/foreign-chain-inspector/src/lib.rs:240 - BuildInspectors::build is a new public trait method with no docs, and two of its contracts are not inferable from the signature: Ok(None) means no inspector exists for this chain (vs Err, this provider could not be set up), and the Sui arm needs a tokio runtime (GrpcSuiClient::new, crates/foreign-chain-rpc-interfaces/src/sui.rs:82) even though build is sync. Documenting both would also retire the "Inspector not implemented for the chain" comment at probe.rs:124.
  • crates/foreign-chain-inspector/src/lib.rs:235 - "hold, clone and share across chains" is inaccurate: an inspector targets one chain; the bounds are about sharing across tasks/threads. Also, ChainInspector is a new agent-noun trait vs docs/engineering-standards.md section "Name capability traits after the action" - a judgement call given it sits next to the pre-existing NetworkFingerprintInspector.
  • crates/foreign-chain-inspector/Cargo.toml:19 - pulling mpc-node-config into foreign-chain-inspector just so BuildInspectors can name ForeignChainProviderConfig cuts against the layering the auth_config_to_rpc_auth doc comment describes (foreign-chain-rpc-factory/src/lib.rs:10-13), and drags near-indexer-primitives/clap/serde_yaml into the chain-abstraction tree. Both the implementor and the only consumer already depend on foreign-chain-rpc-factory, so the trait could live there.
  • crates/foreign-chain-health-check/src/probe.rs:1074 - probe_all_providers__should_retry_a_provider_that_refused_with_a_rate_limit_code no longer involves a rate-limit code; it injects a generic RpcRequestFailed. should_retry_a_transient_failure would match. (The -32005 mapping stays covered in foreign-chain-inspector, so naming only.)
  • crates/foreign-chain-health-check/src/probe.rs:210-242 - MockInspectors and its impl sit between two use groups, with use super::*; at line 242 after the items that depend on it. Compiles, but reads as if the imports were forgotten.
  • crates/foreign-chain-rpc-factory/src/lib.rs:141,186,205,227 - the removed "Given:" text named the real providers motivating each case (Tatum/NowNodes raw token, Helius query-param key, dRPC network plus dkey). That is the why a reader cannot reconstruct from the code, which "Write helpful code comments" asks to keep; only paraphrasing is the target. Also unrelated to this PR.
  • crates/foreign-chain-inspector/src/lib.rs:686 - moving let error = transport(..) from Given into When puts input construction in the action step; the action is the classify_rpc_client_error call.
  • crates/foreign-chain-rpc-factory/src/inspectors.rs:30 - Self::auth_header(auth.clone()) clones the resolved credential for every chain, but only Aptos and Sui use it; the other eleven pay a HeaderValue clone of secret material per provider per tick.
  • crates/foreign-chain-health-check/src/probe.rs:88 - the generic parameter InspectorFactory shadows the concrete InspectorFactory the tests in this same file import and pass; Inspectors would read better.
  • crates/foreign-chain-inspector/src/rpc_inspector.rs:19-31 - the nine EVM variants differ only by a phantom Chain that neither network_fingerprint nor canonical_fingerprint reads, so a mis-wired arm (ForeignChain::Base => RpcInspector::Bnb(..)) would compile and pass every test. Harmless today, but 26 lines of dispatch that grow per EVM chain and carry no information.
  • crates/foreign-chain-rpc-factory/src/inspectors.rs:72 - non_exhaustive forces the wildcard to exist; it does not stop Solana | Ton | Fogo => return Ok(None) from being written next to it. The hazard worth naming is the reverse: a new probeable chain silently falls into _ and reports ProbeNotImplemented instead of failing to compile.
  • crates/foreign-chain-health-check/src/lib.rs:141-160 - prepare_jsonrpc/prepare_aptos/prepare_sui now duplicate InspectorFactory::authenticate/auth_header nearly line for line. Defensible given the TODO for issue 3969, but worth noting so it does not outlive the golden-vector path.

⚠️ Issues found

@haiyuechen-nearone

Copy link
Copy Markdown
Contributor Author

@claude review

@claude

claude Bot commented Aug 31, 2026

Copy link
Copy Markdown

Pull request overview

probe_all_providers no longer hard-codes per-chain inspector construction: it takes a BuildInspectors dependency, with InspectorFactory in production and a MockInspector-backed impl in tests. canonical_fingerprint becomes an instance method and a new RpcInspector enum unifies the per-chain inspector types so one generic probe path is possible. foreign-chain-rpc-auth is renamed foreign-chain-rpc-factory and absorbs build_http_client / resolve_provider_auth from foreign-chain-inspector, and timeout_of moves onto ForeignChainConfig::timeout_duration(). Closes the TODO for #4043.

Since the previous review round, the two functional blockers are resolved: InspectorFactory::build now matches on chain first, so Solana/Ton/Fogo reach _ => Ok(None) without touching credentials (which also removes the provider-order dependence in probe_chain), and the orphaned near-mpc-bounded-collections / tokio dev-dependencies are gone. The latest commit additionally de-duplicates prepare_jsonrpc/prepare_aptos/prepare_sui against the factory, which addresses an earlier non-blocking note.

Changes:

  • New BuildInspectors trait plus ChainInspector bound alias; probe_all_providers/probe_chain generic over it, the 13-arm match chain gone from probe.rs.
  • canonical_fingerprint from associated fn to &self, across every inspector impl and the new RpcInspector dispatch.
  • New mock module behind a test-utils feature; four httpmock probe tests replaced by in-process doubles on a paused clock.
  • Crate rename; RpcAuthentication, build_http_client and the new resolve_provider_auth/InspectorFactory all live in foreign-chain-rpc-factory; ForeignChainConfig::timeout_duration() replaces the private timeout_of.

Reviewed changes

Per-file summary
File Description
Cargo.toml / Cargo.lock Workspace member and dependency rename
crates/foreign-chain-health-check/Cargo.toml Dep rename; dev-dep on foreign-chain-inspector with test-utils
crates/foreign-chain-health-check/src/lib.rs cfg.timeout_duration(); prepare_jsonrpc/prepare_aptos deleted in favour of the factory's build_http_client/resolve_provider_auth
crates/foreign-chain-health-check/src/probe.rs Factory injected; Ok(None)ProbeNotImplemented; tests split between MockInspectors and InspectorFactory
crates/foreign-chain-inspector/Cargo.toml test-utils feature; anyhow + mpc-node-config deps; http demoted to dev; new dev-dep on foreign-chain-rpc-factory
crates/foreign-chain-inspector/src/lib.rs canonical_fingerprint(&self, ..); new ChainInspector/BuildInspectors; RpcAuthentication and build_http_client removed
crates/foreign-chain-inspector/src/rpc_inspector.rs New RpcInspector enum, one variant per probeable chain
crates/foreign-chain-inspector/src/mock.rs MockInspector/MockReply double with a call counter
aptos, bitcoin, evm, starknet, sui inspector.rs Receiver change only
crates/foreign-chain-inspector/tests/*.rs build_http_client call sites rewritten around ForeignChainProviderConfig
crates/foreign-chain-rpc-factory/Cargo.toml Rename; foreign-chain-rpc-interfaces, jsonrpsee, near-mpc-contract-interface
crates/foreign-chain-rpc-factory/src/lib.rs RpcAuthentication (now pub(crate)), resolve_provider_auth, build_http_client
crates/foreign-chain-rpc-factory/src/inspectors.rs New InspectorFactory
crates/foreign-chain-rpc-interfaces/src/aptos.rs Comment tweak
crates/node-config/src/foreign_chains.rs Adds timeout_duration()
crates/node Cargo.toml, foreign_chain_probe.rs, verify_foreign_tx.rs Wiring

Findings

Blocking (must fix before merge):

  • crates/foreign-chain-rpc-factory/src/inspectors.rs:76the fix for the "credentials resolved before the chain-support check" defect has no regression test, and the suite cannot detect a reintroduction. Both ProbeNotImplemented tests (probe.rs:800, probe.rs:819) configure Solana via one_provider(..), which hard-codes auth: AuthConfig::None (probe.rs:359). Under AuthConfig::None, resolve_provider_auth always succeeds, so moving authentication back above match chain would still leave both tests green — while the shipped Solana config (crates/node-config/src/foreign_chains.rs:291-298, alchemy with token: { env: ALCHEMY_API_KEY }) would report AuthTokenUnresolved, count as probed at crates/node/src/foreign_chain_probe.rs:42, and emit a warn! plus a 0 healthy gauge every tick. Per docs/engineering-standards.md §Add tests, please pin the behaviour the whole Ok(None) design exists to protect:

    #[tokio::test]
    async fn probe_all_providers__should_report_an_unprobeable_chain_as_not_implemented_before_resolving_auth() {
        // Given
        let config = solana_only(chain_config(
            Some(ANY_FINGERPRINT),
            NonEmptyBTreeMap::new(
                "alchemy".to_string().into(),
                ForeignChainProviderConfig {
                    rpc_url: CLOSED_PORT_URL.to_string(),
                    auth: AuthConfig::Header {
                        name: http::HeaderName::from_static("authorization"),
                        scheme: Some("Bearer".to_string()),
                        token: TokenConfig::Env { env: "PROBE_TEST_TOKEN_THAT_IS_NOT_SET".to_string() },
                    },
                },
            ),
        ));
    
        // When
        let report = probe_all_providers(&config, &InspectorFactory).await;
    
        // Then
        assert_eq!(
            status_of(&report, ForeignChain::Solana, "alchemy"),
            ProviderStatus::ProbeNotImplemented
        );
    }

Non-blocking (nits, follow-ups, suggestions):

  • crates/foreign-chain-inspector/Cargo.toml:29 — the new dev-dependency on foreign-chain-rpc-factory closes a cycle in the crate graph (foreign-chain-rpc-factoryforeign-chain-inspector → dev → foreign-chain-rpc-factory). Cargo permits dev-dep cycles, so this builds, but the only thing the inspector's tests need from the factory is an unauthenticated HttpClient, and the ceremony it costs is visible: every manual test now spells out mpc_node_config::ForeignChainProviderConfig { rpc_url: .., auth: AuthConfig::None } where it used to pass a URL. jsonrpsee is already a direct dependency, so HttpClientBuilder::default().build(URL) gives the same client with no extra edge. Combined with the lib's new mpc-node-config dependency (needed only so BuildInspectors can name ForeignChainProviderConfig), moving BuildInspectors into foreign-chain-rpc-factory — as suggested last round — would drop both edges.
  • crates/foreign-chain-health-check/src/probe.rs:135 — the expected_network_fingerprint guard moved from above the provider loop to below it. The reported statuses are unchanged, but a chain configured without a fingerprint now reads env tokens and constructs an HTTP or gRPC client per provider on every tick, only to discard the rows and return MissingExpectedFingerprint. Hoisting the guard back above the loop restores the old short-circuit; canonical_fingerprint can stay where it is now that it needs an instance.
  • crates/foreign-chain-rpc-factory/src/lib.rs:67resolve_provider_auth is public with no doc comment and returns a bare (String, Option<(HeaderName, HeaderValue)>). A consumer cannot tell from the signature that the String is rpc_url with Path/Query tokens already substituted and that None means the credential went into the URL rather than "no credential". The contract used to be documented on auth_config_to_rpc_auth, which is now private.
  • crates/node/src/providers/verify_foreign_tx.rs:65 — still Duration::from_secs(c.timeout_sec.get()) after this PR introduces ForeignChainConfig::timeout_duration() and converts the five call sites in foreign-chain-health-check. This file is already touched by the diff.

⚠️ Issues found

@haiyuechen-nearone

Copy link
Copy Markdown
Contributor Author

crates/foreign-chain-rpc-factory/src/inspectors.rs:76 — the fix for the "credentials resolved before the chain-support check" defect has no regression test, and the suite cannot detect a reintroduction.

Our intension is to have inspectors for and probe all chains that are defined in our code base. Not supporting a chain is not the final state, I feel adding a test to check that "we are still in progress" is overkill.

@haiyuechen-nearone
haiyuechen-nearone requested review from kevindeforth and removed request for anodar August 31, 2026 09:02

@netrome netrome left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor considerations from my side.

Comment on lines +50 to +51
replies: Arc<Mutex<VecDeque<MockReply>>>,
calls: Arc<AtomicUsize>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: calls can be derived from replies if the initial length is known so we could drop the atomic here.

@haiyuechen-nearone haiyuechen-nearone Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to use calls here to enable the test cases to verify that "Inspector was called x times", which is sort of independent from number of MockReplies configured.

.expect("call beyond the queued replies");
match reply {
MockReply::Answer { delay, fingerprint } => {
tokio::time::sleep(delay).await;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of adding a delay here? This feels strange.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not used in this PR directly, but I added this so we can test Fanout concurrency. Aka given a Fanout with 1 inspector taking 1 sec and another taking 3 sec, Fanout should return after 3 sec not 4 sec.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay and how do you test that reliably? With tokio paused time and advance? That also reminds me, seems like we're not running these tests with paused tokio time - something we probably should do if we let tokio act as the time abstraction without controlling it ourselves.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes my plan is to use tokio paused time. That is the next PR in the stack.

@kevindeforth kevindeforth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thank you!

@haiyuechen-nearone
haiyuechen-nearone added this pull request to the merge queue Sep 1, 2026
Merged via the queue into main with commit 355fe15 Sep 1, 2026
14 of 15 checks passed
@haiyuechen-nearone
haiyuechen-nearone deleted the 4043-inject-probe-inspectors branch September 1, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inject the inspectors into the foreign chain startup probe

3 participants