Skip to content

feat(gateway): V2 scale-hardening — nonce-manager + RPC failover + batch attest (#126 P2) - #157

Open
LamaSu wants to merge 4 commits into
masterfrom
feat/v2-scale-hardening
Open

feat(gateway): V2 scale-hardening — nonce-manager + RPC failover + batch attest (#126 P2)#157
LamaSu wants to merge 4 commits into
masterfrom
feat/v2-scale-hardening

Conversation

@LamaSu

@LamaSu LamaSu commented Jun 15, 2026

Copy link
Copy Markdown
Owner

V2 scale-hardening for the settlement path (board #126 P2-SCALE — concurrent-create race #154 flagged as separate P2; single-settle already works live). Additive, flag-gated, V1/V2 intact.

ADDS: (1) Nonce manager (nonce-manager.ts) — per-signer serialized queue, monotonic nonces, self-healing re-seed; process-wide registry keyed by signer so all 14 V1+V2 escrow-client writes + paid-job-flow create share ONE queue. Consolidates #145 inline logic. Kill-switch PCC_NONCE_MANAGER_DISABLED. (2) RPC failover+retry (rpc-transport.ts) — viem fallback() primary+fallback, retry/backoff/timeout; getRpcUrls() (PCC_RPC_URLS); oracle verifyWithOracle -> fetchWithRetry. (3) Batch attest (eas-client.ts) — EAS multiAttest + submitMilestoneAttestationsBatch, flag PCC_EAS_MULTIATTEST, no-op <2.

Tests +60. On Spark: gateway 1661 pass/4 skip, contracts 250 pass, build 52/52, typecheck 0.

⚠️ Needs rebase before merge: off 799eff6 (pre-#152); conflicts with merged #152 in escrow-client.ts + oracle-client.ts (both touch them). Other 13 files clean — resolve preserving BOTH.

LamaSu and others added 4 commits June 15, 2026 14:10
Adds the single source of truth for RPC failover ordering, consumed by the
gateway's fallback() transport so chain reads/writes rotate off a laggy or
rate-limited endpoint instead of stalling/dropping a settlement tx.

- ChainDeployment.rpcUrls?: ordered fallback list (primary first).
- base-sepolia + sepolia get public no-key fallbacks (sepolia.base.org was the
  single hardcoded endpoint behind the P0-1 nonce/milestone races).
- getRpcUrls(network): env override (PCC_RPC_URLS full / PCC_RPC_URL single +
  network fallbacks) → chain-config list → default; always deduped + non-empty.
- Exported from the @pcc/contracts barrel. 5 unit tests.

Additive — existing single-rpcUrl consumers and PCC_RPC_URL keep working.

Board #126 P2-SCALE (RPC failover, part 1/2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n reads + oracle (P2-SCALE)

The gateway's chain reads and the oracle call used a single bare http(rpcUrl)
(hardcoded sepolia.base.org) / a bare fetch — no retry, no timeout, no failover.
A rate-limited or briefly-down endpoint then stalled or dropped settlement.

- rpc-transport.ts: buildRpcTransport(urls) → viem fallback() over per-URL http
  transports with retry + exponential backoff + per-request timeout (env-tunable
  PCC_RPC_TIMEOUT_MS / PCC_RPC_RETRY_COUNT). rank:false keeps the explicit
  primary-first order (canonical RPC first, fallbacks only on failure).
- chain-client.ts: read public client now uses the fallback transport.
- oracle-client.ts: verifyWithOracle goes through fetchWithRetry — AbortController
  timeout + retry/backoff on network errors, aborts, and transient HTTP (429/5xx);
  deterministic 4xx returns immediately. Env: PCC_ORACLE_TIMEOUT_MS / _RETRY_COUNT.

Happy path is unchanged (primary tried first, identical when healthy). 12 tests.

Board #126 P2-SCALE (RPC failover + retry, part 2/2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…consolidate #145 (P2-SCALE)

The gateway settles from a single hot signer. createEscrowV2/addMilestone/
fundEscrow/submitAttestation/release all went through viem auto-nonce, so two
concurrent settlements (or a create+addMilestone on a laggy RPC) could be handed
the same nonce — the second tx dropped/replaced and a milestone/release silently
vanished. PR #145 pinned nonces inline for ONE route; this consolidates that into
a reusable primitive every write routes through.

- nonce-manager.ts: NonceManager serializes submission per signer (promise-chain
  tail), seeds the nonce once from getTransactionCount("pending"), pins it
  monotonically, advances only on a confirmed broadcast, and re-seeds from chain
  on a nonce-shaped error (self-healing desync). Process-wide registry keyed by
  signer address → escrow-client writes and paid-job-flow create share ONE queue,
  so cross-path collisions on the hot key are removed. Kill-switch:
  PCC_NONCE_MANAGER_DISABLED falls back to viem auto-nonce (reversible, no deploy).
- escrow-client.ts: one wrap of the wallet client routes ALL 14 V1+V2 writes
  through the manager (zero churn per call site); also switched to the failover
  transport; exports writeWithSigner for sibling modules.
- paid-job-flow.ts: replaced #145's inline `let nonce; nonce++` with the shared
  manager + failover transport (same monotonic guarantee, now cross-path safe).

On-chain effects are identical when enabled — just race-free. 13 nonce-manager
tests + nonce-pinning assertions added to the V2 settlement wiring suite.

Board #126 P2-SCALE (nonce manager). Keeps V1/V2 paths intact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a job's milestones settle together, minting their evidence attestations one
EAS.attest() at a time is N txs / N nonces. EAS.multiAttest([...]) mints them in
one. Adds the primitive + the integration seam, flag-gated so the live flow is
unchanged.

- eas-client.ts: EAS multiAttest ABI fragment; pure encoders (encodeMultiAttest,
  buildEvidenceMultiAttestRequest, groupEvidenceAttestationsBySchema); multiAttestEAS
  write through escrow-client's writeWithSigner (shares the one signer + nonce
  queue, so a batch mint can't race an escrow release); submitMilestoneAttestationsBatch
  high-level helper that skips (caller falls back to per-milestone) unless
  PCC_EAS_MULTIATTEST is set AND >=2 items.

Additive and gated: the live /complete flow settles a single milestone (index 0)
and lets the off-chain oracle mint, so nothing here is auto-engaged — this is the
gateway-as-attester / multi-milestone-settle seam. 10 tests (encode round-trips +
mocked on-chain dispatch).

Board #126 P2-SCALE (batch attestations). Keeps V1/V2 paths intact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@LamaSu

LamaSu commented Jul 2, 2026

Copy link
Copy Markdown
Owner Author

Agreed on all four — the failover path is the right thing to be careful about, since it sits directly on paid writes. Confirming each against the current rpc-transport.ts / nonce-manager.ts:

1. Writes must not double-send; already known / nonce too low are idempotent results, not failures.
viem's fallback re-sends the same serialized tx on eth_sendRawTransaction failover, so a rotated retry is the identical hash (the network dedupes → already known), not a second spend — good. The gap is in handling: NonceManager.run rethrows on any isNonceError, so a benign already known (the tx already reached the mempool via the primary) surfaces to the caller as a settlement failure — which then either retries with a fresh nonce (the real double-send) or marks the milestone failed (the "silently vanishes" case). Fix:

  • already knownsuccess: resolve with the tx hash (compute it from the signed tx before broadcast, or read it back), advance the cursor, don't rethrow.
  • nonce too lowambiguous: look up our pre-computed tx hash on-chain first — if it's mined/pending it's idempotent success; only if a different tx took the nonce is it a real failure (then re-seed + surface). Never assume success without confirming the hash.
  • Only ever re-broadcast the byte-identical signed tx; never re-sign / rebuild on the retry path.

2. chainId gate before a URL enters fallback().
buildRpcTransport wraps each URL in http() with no chain check, so a misconfigured PCC_RPC_URLS entry (or a URL silently serving another chain) can enter the rotation. Add a one-time eth_chainId assertion per URL against the expected getChainId(network) and drop / hard-fail mismatches before they're eligible for a write. (EIP-155 signed txs carry chainId, so a wrong-chain node rejects at broadcast — but catching it at transport-build time beats catching it mid-settlement.)

3. verifyWithOracle → fetchWithRetry must reject stale block data, not only transport errors.
Retry currently triggers on transport/timeout only; a healthy-but-lagging endpoint returning an old block passes. Add a freshness gate: capture the block number, require it monotonic vs last-seen and within N blocks of the max seen across endpoints, and treat a stale read as a retriable failure (rotate), not a valid response. Otherwise verification can pass against outdated escrow state.

4. PCC_NONCE_MANAGER_DISABLED must not let the retry path mask a nonce race.
With the manager bypassed we're on viem auto-nonce + per-endpoint retry. Nonce errors are deterministic, not transient — ensure the disabled path surfaces an isNonceError loudly instead of retrying/failing-over as if transient, so the kill-switch can't hide a real race.

Tests — your shape plus a few:

// reads fail over; a paid write is retried only with the same signed tx
// RPC-A: eth_call times out, then returns a stale block number
// RPC-B: healthy, expected chainId, current block
// expect: reads rotate to B; sendRawTransaction re-sends identical bytes;
//         `already known` from B resolves to the original tx hash (no throw)
  • chainId-mismatch URL is rejected at buildRpcTransport (never eligible for a write)
  • stale-block oracle read rotates instead of being accepted
  • with PCC_NONCE_MANAGER_DISABLED, a nonce error surfaces (is not retried/masked)

I can implement these on a follow-up branch off this one and run the suite on Spark (this checkout is mid-work on another branch, so I'm keeping clear of it directly). Want me to take that?

LamaSu added a commit that referenced this pull request Jul 6, 2026
…review point 1)

Under viem `fallback`, a paid write's byte-identical signed tx is re-broadcast
on failover; a node that already saw it reports `already known`. The nonce
manager rethrew that as a settlement failure, which risks a real double-send
(retry with a fresh nonce) or a vanished milestone.

Fix per the review:
- `already known` -> success: resolve with the tx hash computed from the signed
  tx BEFORE broadcast; advance the cursor; don't rethrow.
- `nonce too low` -> ambiguous: confirm the pre-computed hash on-chain first;
  our tx pending/mined = idempotent success, a different tx on the nonce = real
  failure (re-seed + surface). Never assume success without confirming.
- Only ever re-broadcast the byte-identical signed tx; never re-sign.

Managed path signs locally (prepareTransactionRequest -> signTransaction ->
sendRawTransaction) so the hash is known pre-broadcast. Capability-checked:
clients that only implement writeContract (test mocks / exotic wrappers) fall
back to the legacy broadcast, keeping the settlement/EAS suites green; prod viem
clients take the idempotent path. New pure DI module idempotent-write.ts +
isAlreadyKnownError, fully unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CWkSqFVjmKusApvz1AH2Eu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant