Skip to content

rpc: separate overlay-backed and committed read paths - #22533

Merged
yperbasis merged 69 commits into
mainfrom
yperbasis/rpc-view-consistency
Aug 25, 2026
Merged

rpc: separate overlay-backed and committed read paths#22533
yperbasis merged 69 commits into
mainfrom
yperbasis/rpc-view-consistency

Conversation

@yperbasis

@yperbasis yperbasis commented Jul 16, 2026

Copy link
Copy Markdown
Member

During forkchoice flush and commit, the published BlockOverlay can be one block ahead of the MDBX snapshot held by an RPC request. Both are valid views, but combining an overlay block or transaction with committed temporal state or history can produce a response that never existed on-chain.

This PR applies two explicit policies to the migrated handlers:

Read kind Selected view
Head-sensitive, block-table-only reads One selected overlay generation, when present
Replay, proof, witness, and history reads One committed database transaction

Behavior

  • Overlay-backed helpers preserve an already selected overlay generation across nested calls. Their non-owning read views can be closed without closing the shared overlay or the caller's transaction.
  • Migrated committed paths keep selector resolution and state or history reads on one transaction. Immutable headers may use the block LRU after their exact hash is resolved. Receipt regeneration is the remaining exception described below.
  • Migrated state-backed hash selectors reject non-canonical blocks, and targets above execution progress fail before reading incomplete state or history.
  • Tracing, simulation, proof, witness, eth_call, eth_createAccessList, and GraphQL call reject pending because they cannot acquire state matching the requested pending block.
  • The raw debug block family consistently returns null when it cannot fully represent a published pending block.
  • Migrated tracing, simulation, and Otterscan replay paths avoid state caches that are not bound to the selected transaction. Request-scoped proof and simulation domains also avoid the process-wide commitment branch cache.

Scope

This PR preserves an overlay generation after it has been selected. Atomic acquisition of the database snapshot and overlay, including pinning the absence of an overlay, remains in #22987. Until then, the temporary IsOverlayReadView marker prevents nested helpers from replacing an existing overlay view. Adoption by the remaining composite and independently transacted RPC paths is tracked by #23416.

Receipt generation remains part of #23416: the generator can still select the live overlay internally and use a process-wide state cache instead of remaining bound to the transaction selected by its caller.

This PR does not change eth_estimateGas. Consistent EstimateGas view selection and safe pending-template handling remain in stacked draft #23485.

The generic embedded-daemon latest-state view remains tracked by #21314. Remote block readers also require snapshot propagation or result validation, tracked by #23416. overlay_getLogs selects its range from committed progress, but its parallel workers still open independent transactions under the same follow-up.

Related follow-ups cover witness branch-cache isolation (#22198), selector and error-policy consolidation (#23424 and #23428), coherent replay-cache reuse (#23425), transaction-index validation (#23431), and call-many state-context validation (#23444). Overlay-history fallback when in-memory history is disabled is tracked by #23500; the historical state boundary for ad-hoc trace bundles is tracked by #23501.

Review map

  • View primitives: db/kv/membatchwithdb/memory_mutation.go, rpc/rpchelper/filters.go, and rpc/jsonrpc/eth_api.go
  • Overlay-backed migrations: selected handlers in the raw-debug, block-count, transaction-lookup, receipt, uncle, Erigon, GraphQL, Otterscan, and overlay families
  • Committed migrations: tracing, logs, simulation, proof, witness, and debug state endpoints
  • Regression coverage: rpc/jsonrpc/overlay_race_test.go and rpc/jsonrpc/trace_view_consistency_test.go

Tests

Regression tests exercise overlay publication, replacement, and unpublish windows; reorgs; execution progress; pending selectors; missing data; cache isolation; and endpoint-compatible error behavior.

RPC integration coverage uses the pinned rpc-tests release, which includes the corresponding test correction from erigontech/rpc-tests#588.

Split out of #21293. Overlay-view methods pin one BlockOverlay read view
per request; committed-view methods resolve tags with nil filters so the
bounds agree with the temporal data they scan. eth_getProof keeps all
reads on the caller's single RO snapshot.
awskii added 2 commits August 10, 2026 22:54
…nsistency

# Conflicts:
#	rpc/jsonrpc/eth_call.go
#	rpc/jsonrpc/eth_call_test.go
trace_block, trace_call, trace_callMany, trace_replayBlockTransactions,
trace_rawTransaction and debug_traceCall{,Many} resolved head-sensitive
block tags through the overlay while replaying on the committed tx, so a
"latest" trace could run block N's transactions against state ending at
N-1. On a mainnet archive node at the tip that made trace_block("latest")
fail 129 of 4134 calls (3.1%) with "nonce too high: tx X state X-1", and
occasionally "insufficient funds"; debug_traceBlockByNumber, which already
builds its context from the plain tx, failed 0 of 4134 over the same run.

The replay reads SD-temporal data, which the overlay does not serve:
OverlayTemporalReadView.GetLatest delegates straight to the committed tx,
so the state side cannot follow the overlay head. Resolving these tags on
the committed view is what makes the bounds and the scan agree; pinning a
single overlay view instead needs the SD-aware temporal view from #21314.

Extends the same treatment already applied to trace_filter and
debug_traceBlockBy*.
@awskii

awskii commented Aug 10, 2026

Copy link
Copy Markdown
Member

Merged main (was ~350 commits behind, conflicting) and extended the committed-view rule to the remaining tracing replay paths.

Merge conflicts resolved: dropped the blockNumber == 0 genesis rejection in GetProof (main removed it in #22552, and its new TestGetProofGenesisPrunedCommitmentHistory asserts genesis works), kept both sides' new tests, and updated NewPrivateDebugAPI call sites to the *rpccfg.DebugApiConfig signature. Two of those were silent — git merged them cleanly into code that did not compile.

The added commit applies the same nil filters to trace_block, trace_call, trace_callMany, trace_replayBlockTransactions, trace_rawTransaction and debug_traceCall{,Many}. Reason to widen the scope: trace_block is the method that actually fails today. On a mainnet archive node at the tip:

build trace_block("latest") debug_traceBlockByNumber("latest")
main 56684f48 129 / 4134 errors (3.1%) 0 / 4134
this branch 86071289 0 / 4512 0 / 4512

Failures were txIndex N: nonce too high: address 0x…, tx: X state: X-1 (125 of 129) and insufficient funds (4 of 129) — state exactly one block behind. Same 25-minute alternating loop for both runs.

Worth noting for #22969: that panic is behind dbg.AssertEnabled (ERIGON_ASSERT, default false), so CI crashes on the assert while a production node returns the diverged trace with no error.

I did not touch the state-reader path — pinning a single overlay view instead would still read N-1, since OverlayTemporalReadView.GetLatest delegates straight to the committed tx. That needs the SD-aware temporal view from #21314, so committed-view resolution is the correct fix for these sites today.

Happy to split the added commit into its own PR if you would rather keep this one at its original scope.

awskii added 3 commits August 11, 2026 01:00
The BaseAPI helpers derived the overlay view twice — once to resolve the
block tag, once to read the header or body — so an overlay unpublished
between the two calls dropped the read onto the older MDBX snapshot while
the number came from the newer one. headerByNumber, headerByNumberOrHash,
blockByNumberWithSenders and blockByHashWithSenders now derive it once and
thread it through, and erigon_getBlockByTimestamp reuses the view its
search bounds came from.

Also corrects a comment that claimed the overlay exposes block tables
only: MemoryMutation.GetAsOf and HistorySeek do consult the SharedDomains
set by InitBlockOverlay. GetLatest and RangeAsOf are the reads that stay
on the committed backing tx, which is what the committed-view resolution
depends on.
Resolving the blockHash filter through HeaderNumber alone accepts any
header the header-number index knows, including side-chain and
header-only ones, while the log scan that follows is by block number. A
non-canonical hash therefore returned the canonical block's logs instead
of an error. Gate the resolved number on the canonical hash matching.
The tracing methods resolve and replay on the committed view, which holds
no pending block, so rpchelper.GetBlockNumber silently resolved "pending"
to the latest executed block: a caller asking to trace pending got a trace
of a different block, reported as that block, with no error.

Reject the tag instead, matching go-ethereum, which answers
"tracing on top of pending is not supported" rather than substituting a
block. Covers debug_traceBlockByNumber/ByHash, debug_traceCall,
debug_traceCallMany, trace_block, trace_replayBlockTransactions,
trace_call and trace_callMany.
@awskii

awskii commented Aug 11, 2026

Copy link
Copy Markdown
Member

Added an explicit rejection of the pending tag in the tracing methods.

With nil filters, _GetBlockNumber resolves pending to the latest executed block (helper.go, the rpc.PendingBlockNumber case falls through to plainStateBlockNumber). For the tracing methods that is a wrong answer rather than a lag: the caller asked to trace pending and gets a trace of a different block, reported as that block, with no error. The PR body lists this as an intended compatibility choice, which is right for the state-reading methods, but tracing has no equivalent to "read the newest state you have".

go-ethereum rejects it outright — eth/tracers/api.go, "tracing on top of pending is not supported" — and derives state and block context from one resolved block so the two can never disagree. This follows that, covering debug_traceBlockByNumber/ByHash, debug_traceCall, debug_traceCallMany, trace_block, trace_replayBlockTransactions, trace_call and trace_callMany. TestTracingRejectsPendingTag pins it; all seven subtests fail without the guard.

Separately, parity_listStorageKeys and eth_getProof had state-version bugs unrelated to view consistency, so those went to #23165 rather than widening this PR further.

debug_traceBlockByNumber("pending") answered before this series and the RPC
integration suite pins that (debug_traceBlockByNumber/test_25), so rejecting
the tag there broke mainnet-rpc-integ-tests. go-ethereum draws the line in
the same place: it refuses to execute a call on top of pending, but traces a
pending block rather than erroring.

Keep the rejection on debug_traceCall, debug_traceCallMany, trace_call and
trace_callMany; drop it from debug_traceBlockBy*, trace_block and
trace_replayBlockTransactions.
@awskii
awskii marked this pull request as ready for review August 11, 2026 06:20
@awskii
awskii requested a review from lupin012 as a code owner August 11, 2026 06:20
awskii added a commit that referenced this pull request Aug 11, 2026
… read

The execution gate read the plain roTx while the block tag was still
resolved through the overlay, so during a commit an overlay-resolved head
could be reported as not executed. Resolve on the committed view instead,
which is also where the commitment-history reads happen. This overlaps the
same change in #22533; whichever lands first, the other is a trivial
conflict.

rawdb.ReadCurrentBlockNumber returns nil when no head header is set, so
listStorageKeys dereferenced a nil pointer instead of returning an error.
TestTraceBlockAcceptsPendingTag asserted NotErrorIs, which passes on any
unrelated error; all three methods return nil there, so assert NoError.
The test now fails if the rejection is widened back onto block tracing,
which is the regression that broke mainnet-rpc-integ-tests.

trace_call and trace_callMany still advertised 'pending' as an accepted
tag, so update those two parameter lists. trace_block and
trace_replayBlockTransactions keep accepting it and are left alone.
Versioned docs describe shipped releases and are not touched.
Completes the rejection started for the call methods. debug_traceBlockBy*,
trace_block and trace_replayBlockTransactions resolve tags on the committed
view, where "pending" falls through to the latest executed block, so they
answered for the head block and reported it as the pending request.
go-ethereum either traces a real pending block or errors; it never
substitutes a different one, so answering for latest matches neither branch.

Real pending-block tracing needs a pending state source and is left for
later; until then an explicit error beats a wrong block.

CI note: this changes debug_traceBlockByNumber/test_25 in the rpc-tests
suite, updated in erigontech/rpc-tests#588. mainnet-rpc-integ-tests stays
red until that merges and RPC_VERSION is bumped.
@awskii

awskii commented Aug 11, 2026

Copy link
Copy Markdown
Member

Extended the pending rejection to the block-tracing methods — debug_traceBlockBy*, trace_block, trace_replayBlockTransactions — so it now covers all seven.

The earlier split (reject on the call methods, accept on block tracing) was based on go-ethereum tracing a pending block rather than erroring. That is only half of what geth does: EthAPIBackend.BlockByNumber returns miner.Pending() for PendingBlockNumber or errors when none is available. It never substitutes a different block. Erigon on the committed view has no pending source at all, so pending fell through to the latest executed block — matching neither geth branch and returning a full trace of the head block for a request that asked for pending.

Real pending-block tracing needs a pending state source and is a separate change; until then an explicit error beats a wrong block.

CI dependency: this changes debug_traceBlockByNumber/test_25, which pins the old result: null. Fixture update is erigontech/rpc-tests#588. mainnet-rpc-integ-tests will stay red on this PR until that merges, a release is cut, and RPC_VERSION is bumped in .github/workflows/scripts/rpc_version.env (currently v2.24.0). Happy to do the bump as a follow-up commit once the tag exists.

Docs updated for all four affected trace_* methods; the versioned v3.3/v3.4 docs describe shipped releases and are left alone.

pull Bot pushed a commit to Dustin4444/erigon that referenced this pull request Aug 11, 2026
…n execution (erigontech#23165)

Two state-version bugs in the RPC layer, both independent of each other
and of the view-consistency work in erigontech#22533.

`parity_listStorageKeys` reads the account with a latest-state reader —
the state after head block `bn` — but scanned its storage at `Min(bn)`,
the first txNum of `bn`, which is the state after `bn-1`. The account
and its storage therefore came from different blocks: a slot written in
the head block was missing from the listing, and a slot deleted in it
was still listed. `state.Dumper`, the equivalent path, uses
`Min(blockNumber+1)`.

`eth_getProof` resolved a block by canonical hash alone. Canonical
hashes exist for blocks the header stage has downloaded but execution
has not reached, so a request for one walked the history path and
surfaced a `PrunedError` or a root-hash mismatch instead of reporting
that the block is not executed yet.

## Changes

- `parity_api.go` — `Min(bn)` → `Min(bn+1)` so the storage scan matches
the account read.
- `eth_call.go` — gate `GetProof` on `rpchelper.CheckBlockExecuted`
after resolution.
@yperbasis
yperbasis marked this pull request as draft August 15, 2026 07:18
@yperbasis

yperbasis commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

@AskAlexSharov Follow-up to #22533 (comment), rechecked against current head 6d331e5:

The remaining composite and independently acquired view propagation is tracked by #23416. Thanks for the detailed review.

@yperbasis
yperbasis marked this pull request as ready for review August 24, 2026 09:48
@yperbasis
yperbasis requested review from AskAlexSharov and a balanced review from Copilot August 24, 2026 09:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 4 comments.

Suppressed comments (1)

rpc/jsonrpc/eth_receipts.go:586

  • The selector at line 573 may resolve through overlay generation A, but this independently selects the current overlay for the body, and getReceipts receives the raw transaction and can select another generation. A publish/unpublish between these steps can combine different blocks and receipts. Capture one temporal overlay view before selector resolution and pass it through block and receipt reads.
	block, err := api.blockWithSenders(ctx, api.filters.WithOverlay(tx), blockHash, blockNum)

Comment thread rpc/jsonrpc/graphql_api.go
Comment thread rpc/jsonrpc/eth_call.go
Comment thread rpc/jsonrpc/eth_receipts.go Outdated
Comment thread rpc/jsonrpc/otterscan_search_v3.go Outdated
@yperbasis
yperbasis marked this pull request as draft August 24, 2026 10:16
@yperbasis
yperbasis marked this pull request as ready for review August 24, 2026 12:47
@yperbasis
yperbasis requested a balanced review from Copilot August 24, 2026 12:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants