feat(gateway): V2 scale-hardening — nonce-manager + RPC failover + batch attest (#126 P2) - #157
feat(gateway): V2 scale-hardening — nonce-manager + RPC failover + batch attest (#126 P2)#157LamaSu wants to merge 4 commits into
Conversation
Adds the single source of truth for RPC failover ordering, consumed by the gateway's fallback() transport so chain reads/writes rotate off a laggy or rate-limited endpoint instead of stalling/dropping a settlement tx. - ChainDeployment.rpcUrls?: ordered fallback list (primary first). - base-sepolia + sepolia get public no-key fallbacks (sepolia.base.org was the single hardcoded endpoint behind the P0-1 nonce/milestone races). - getRpcUrls(network): env override (PCC_RPC_URLS full / PCC_RPC_URL single + network fallbacks) → chain-config list → default; always deduped + non-empty. - Exported from the @pcc/contracts barrel. 5 unit tests. Additive — existing single-rpcUrl consumers and PCC_RPC_URL keep working. Board #126 P2-SCALE (RPC failover, part 1/2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n reads + oracle (P2-SCALE) The gateway's chain reads and the oracle call used a single bare http(rpcUrl) (hardcoded sepolia.base.org) / a bare fetch — no retry, no timeout, no failover. A rate-limited or briefly-down endpoint then stalled or dropped settlement. - rpc-transport.ts: buildRpcTransport(urls) → viem fallback() over per-URL http transports with retry + exponential backoff + per-request timeout (env-tunable PCC_RPC_TIMEOUT_MS / PCC_RPC_RETRY_COUNT). rank:false keeps the explicit primary-first order (canonical RPC first, fallbacks only on failure). - chain-client.ts: read public client now uses the fallback transport. - oracle-client.ts: verifyWithOracle goes through fetchWithRetry — AbortController timeout + retry/backoff on network errors, aborts, and transient HTTP (429/5xx); deterministic 4xx returns immediately. Env: PCC_ORACLE_TIMEOUT_MS / _RETRY_COUNT. Happy path is unchanged (primary tried first, identical when healthy). 12 tests. Board #126 P2-SCALE (RPC failover + retry, part 2/2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…consolidate #145 (P2-SCALE) The gateway settles from a single hot signer. createEscrowV2/addMilestone/ fundEscrow/submitAttestation/release all went through viem auto-nonce, so two concurrent settlements (or a create+addMilestone on a laggy RPC) could be handed the same nonce — the second tx dropped/replaced and a milestone/release silently vanished. PR #145 pinned nonces inline for ONE route; this consolidates that into a reusable primitive every write routes through. - nonce-manager.ts: NonceManager serializes submission per signer (promise-chain tail), seeds the nonce once from getTransactionCount("pending"), pins it monotonically, advances only on a confirmed broadcast, and re-seeds from chain on a nonce-shaped error (self-healing desync). Process-wide registry keyed by signer address → escrow-client writes and paid-job-flow create share ONE queue, so cross-path collisions on the hot key are removed. Kill-switch: PCC_NONCE_MANAGER_DISABLED falls back to viem auto-nonce (reversible, no deploy). - escrow-client.ts: one wrap of the wallet client routes ALL 14 V1+V2 writes through the manager (zero churn per call site); also switched to the failover transport; exports writeWithSigner for sibling modules. - paid-job-flow.ts: replaced #145's inline `let nonce; nonce++` with the shared manager + failover transport (same monotonic guarantee, now cross-path safe). On-chain effects are identical when enabled — just race-free. 13 nonce-manager tests + nonce-pinning assertions added to the V2 settlement wiring suite. Board #126 P2-SCALE (nonce manager). Keeps V1/V2 paths intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a job's milestones settle together, minting their evidence attestations one EAS.attest() at a time is N txs / N nonces. EAS.multiAttest([...]) mints them in one. Adds the primitive + the integration seam, flag-gated so the live flow is unchanged. - eas-client.ts: EAS multiAttest ABI fragment; pure encoders (encodeMultiAttest, buildEvidenceMultiAttestRequest, groupEvidenceAttestationsBySchema); multiAttestEAS write through escrow-client's writeWithSigner (shares the one signer + nonce queue, so a batch mint can't race an escrow release); submitMilestoneAttestationsBatch high-level helper that skips (caller falls back to per-milestone) unless PCC_EAS_MULTIATTEST is set AND >=2 items. Additive and gated: the live /complete flow settles a single milestone (index 0) and lets the off-chain oracle mint, so nothing here is auto-engaged — this is the gateway-as-attester / multi-milestone-settle seam. 10 tests (encode round-trips + mocked on-chain dispatch). Board #126 P2-SCALE (batch attestations). Keeps V1/V2 paths intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Agreed on all four — the failover path is the right thing to be careful about, since it sits directly on paid writes. Confirming each against the current 1. Writes must not double-send;
2. chainId gate before a URL enters 3. 4. Tests — your shape plus a few: // reads fail over; a paid write is retried only with the same signed tx
// RPC-A: eth_call times out, then returns a stale block number
// RPC-B: healthy, expected chainId, current block
// expect: reads rotate to B; sendRawTransaction re-sends identical bytes;
// `already known` from B resolves to the original tx hash (no throw)
I can implement these on a follow-up branch off this one and run the suite on Spark (this checkout is mid-work on another branch, so I'm keeping clear of it directly). Want me to take that? |
…review point 1) Under viem `fallback`, a paid write's byte-identical signed tx is re-broadcast on failover; a node that already saw it reports `already known`. The nonce manager rethrew that as a settlement failure, which risks a real double-send (retry with a fresh nonce) or a vanished milestone. Fix per the review: - `already known` -> success: resolve with the tx hash computed from the signed tx BEFORE broadcast; advance the cursor; don't rethrow. - `nonce too low` -> ambiguous: confirm the pre-computed hash on-chain first; our tx pending/mined = idempotent success, a different tx on the nonce = real failure (re-seed + surface). Never assume success without confirming. - Only ever re-broadcast the byte-identical signed tx; never re-sign. Managed path signs locally (prepareTransactionRequest -> signTransaction -> sendRawTransaction) so the hash is known pre-broadcast. Capability-checked: clients that only implement writeContract (test mocks / exotic wrappers) fall back to the legacy broadcast, keeping the settlement/EAS suites green; prod viem clients take the idempotent path. New pure DI module idempotent-write.ts + isAlreadyKnownError, fully unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CWkSqFVjmKusApvz1AH2Eu
V2 scale-hardening for the settlement path (board #126 P2-SCALE — concurrent-create race #154 flagged as separate P2; single-settle already works live). Additive, flag-gated, V1/V2 intact.
ADDS: (1) Nonce manager (nonce-manager.ts) — per-signer serialized queue, monotonic nonces, self-healing re-seed; process-wide registry keyed by signer so all 14 V1+V2 escrow-client writes + paid-job-flow create share ONE queue. Consolidates #145 inline logic. Kill-switch PCC_NONCE_MANAGER_DISABLED. (2) RPC failover+retry (rpc-transport.ts) — viem fallback() primary+fallback, retry/backoff/timeout; getRpcUrls() (PCC_RPC_URLS); oracle verifyWithOracle -> fetchWithRetry. (3) Batch attest (eas-client.ts) — EAS multiAttest + submitMilestoneAttestationsBatch, flag PCC_EAS_MULTIATTEST, no-op <2.
Tests +60. On Spark: gateway 1661 pass/4 skip, contracts 250 pass, build 52/52, typecheck 0.