Skip to content

op-reth,kona-node!: build derived payloads at forkchoiceUpdated - #23447

Closed
sebastianst wants to merge 3 commits into
developfrom
seb/op-reth-derived-fcu-build
Closed

sebastianst wants to merge 3 commits into
developfrom
seb/op-reth-derived-fcu-build

Conversation

@sebastianst

@sebastianst sebastianst commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Alternative: #23454 builds derived payloads in a payload service wrapper instead, with no reth changes.

op-reth only found out that a derived block's transactions were invalid in its asynchronous payload build, after engine_forkchoiceUpdated had already answered VALID. An invalid forced transaction was skipped, so op-reth built a block that the fault proof rejects. An invalid 0x7D failed the build instead, so engine_getPayload returned an error, the rollup node retried the same attributes forever and the safe head stalled.

op-reth now builds the payload of noTxPool attributes that carry a non-deposit transaction exactly once inside engine_forkchoiceUpdated. It does so in OpEngineApi with op-reth's payload builder, whatever payload service the node runs, so op-reth-premium is covered too, and answers -38003 when the transactions make the block invalid. op-node and kona-node (fixed here) then replace the block with a deposits-only one. Deposit-only noTxPool attributes still go to the payload service.

  • engine_getPayload hands the executed block to the engine tree through a new reth hook, PayloadBuilderHandle::publish_built_payload (feat(payload): allow publishing payloads built outside a payload job reth#16), so engine_newPayload doesn't execute it a second time.
  • Attributes reth rejects keep reth's exact response. A build that fails for a node-local reason (missing state, provider error, panic) is never -38003; it takes the payload-service path as before.
  • A retried request joins the running build or gets the built payload, and forkchoice updates are handled one at a time, so a fallback can't overwrite a newer forkchoice.
  • For derived attributes with an invalid transaction, debug_executePayload returns the witness up to that transaction, where the fault proof program stops, instead of the witness of a block without it.

Latency for 30M-gas blocks (release build, median of 9, FCU + getPayload + newPayload + FCU), before → after: ETH transfers 120 → 95 ms, SSTORE 14 → 24 ms, KECCAK256 165 → 159 ms, ecrecover 388 → 389 ms, all within run-to-run noise (±10–20 ms). An invalid block is rejected at FCU after executing up to the failing transaction.

Supersedes #23188, whose kona-node fix and acceptance test this carries. Compared with it and with an earlier private fix (optimism-private@ed9e4d8):

#23188 ed9e4d8 this PR
Where it checks payload job generator, at job creation OpEngineValidator::validate_payload_attributes_against_header OpEngineApi, in engine_forkchoiceUpdated
Derived attributes covered 0x7D only all with a non-deposit tx all with a non-deposit tx (one-line switch to all noTxPool)
Executions per derived block 1 2 (validate, then build) 1
op-reth-premium not covered (own job generator) covered covered
Invalid forced tx skipped (fault-proof mismatch) yes fixed fixed
kona-node fix + acceptance test yes no yes
Upstream changes none none reth hook (ethereum-optimism/reth#16)
Costs blocks the payload service while building; copy of reth's payload service builder double execution; breaking Evm parameter on OpEngineValidator copies reth's attribute checks; FCU serialized by a lock

Before merge: move the reth pin from the head of ethereum-optimism/reth#16 to its commit on optimism, and re-tag the stale UPSTREAM-MIRROR reth tags (just mirrors stale); the only reth change is the payload-service hook.

Review agents (Rust, Go, derivation-batch, security) ran; their findings are fixed except:

  • Dismissed, to confirm: no timeout on a derived build that holds the forkchoice lock. Builds are bounded by the block gas limit, a stuck state provider stalls reth's engine tree as well, and a request the rollup node abandons releases the lock.
  • Not changed here, as kona-node with op-geth already behaves so: before Holocene, kona-node retries a -38003 for derived attributes instead of dropping them; kona-node and kona-client gate the deposits-only replacement on the L2 timestamp, op-node on the L1 origin's; a transient error while building the replacement is critical.

BREAKING CHANGE: op-reth answers engine_forkchoiceUpdated with -38003 when the transactions of derived attributes make the block invalid. op-node handles this. kona-node needs the change in this PR, otherwise it retries and stalls the safe head, so upgrade both together. Rust callers of OpEngineApi::new must pass the provider, validator and DerivedPayloads it builds derived payloads with, and OpEngineApiBuilder requires the node's pool, provider and EVM to fit op-reth's payload builder.

🤖 Co-created with Claude Opus 5.5

@sebastianst
sebastianst force-pushed the seb/op-reth-derived-fcu-build branch from 7cf5123 to 862e945 Compare October 15, 2026 11:49
sebastianst and others added 3 commits October 15, 2026 14:21
OpEngineApi builds the payload of derived (noTxPool) attributes that carry a
non-deposit transaction while it answers engine_forkchoiceUpdated, with
op-reth's payload builder whatever payload service the node runs. If the
transactions make the block invalid it answers -38003, so the rollup node
replaces the block with a deposits-only one or drops the batch. Before, the
payload service either skipped the invalid transaction and built a block the
fault proof rejects, or failed the build at getPayload, which the rollup node
retries forever. The builder no longer skips invalid transactions of noTxPool
attributes.

getPayload serves the payload and publishes it to the payload service, whose
subscribers include the engine tree, so newPayload does not execute the block
again. That needs PayloadBuilderHandle::publish_built_payload, so reth is
pinned to the fork commit that adds it.

Attributes reth rejects, and builds that fail for a node-local reason, take
reth's path as before. Requests for one payload share its build, and
forkchoice updates are handled one at a time, so a fallback cannot overwrite
a newer forkchoice.

The witness of derived attributes with an invalid transaction covers the
state read up to that transaction, where the fault proof program stops.

BREAKING CHANGE: op-reth answers engine_forkchoiceUpdated with -38003 when the
transactions of derived attributes make the block invalid. The rollup node
must then replace the block with a deposits-only block or drop the batch.
op-node does; kona-node needs the matching change, or it retries the
attributes and stalls the safe head, so upgrade both together. Downstream
crates that call OpEngineApi::new must pass the provider, validator and
DerivedPayloads it builds derived payloads with; OpEngineApiBuilder requires
the node's pool, provider and EVM to fit op-reth's payload builder.

Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the execution layer rejects derived attributes at forkchoiceUpdated with
-38003 (invalid payload attributes), kona-node builds the deposits-only
replacement under Holocene, as op-node does, instead of retrying the
attributes forever. The engine queue drops a task that ends in a flush rather
than retrying it, since retrying would replace the block again.

The forkchoiceUpdated and newPayload rejections share one deposits-only
replacement, which now logs why a failed replacement is critical.

Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Posts a batch whose 0x7D over-refunds a transaction and checks that the
sequencer and the verifier both replace the block with the same deposits-only
block, and that the safe head keeps advancing.

Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sebastianst

Copy link
Copy Markdown
Member Author

We chose #23454

@sebastianst
sebastianst deleted the seb/op-reth-derived-fcu-build branch October 21, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant