Skip to content

op-reth,kona-node!: build derived payloads in a payload service wrapper - #23454

Merged
sebastianst merged 8 commits into
developfrom
seb/op-reth-derived-payload-service
Oct 20, 2026
Merged

sebastianst merged 8 commits into
developfrom
seb/op-reth-derived-payload-service

Conversation

@sebastianst

@sebastianst sebastianst commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Alternative to #23447: same fix, built in a payload service wrapper instead of OpEngineApi.

Also tried and dropped: forwarding derived attributes to the wrapped service and verifying its payload (0e32d682ee). It only kept op-reth-premium's subblocks for derived builds, which a sequencer only runs in recovery.

op-reth only found out that a derived block's transactions were invalid in its asynchronous payload build, after engine_forkchoiceUpdated had already answered VALID. An invalid forced transaction was skipped, so op-reth built a block that the fault proof rejects. An invalid 0x7D failed the build instead, so engine_getPayload returned an error, the rollup node retried the same attributes forever and the safe head stalled.

DerivedPayloadServiceBuilder wraps a payload service builder. For noTxPool attributes that carry a non-deposit transaction, the service it spawns builds the payload exactly once, with op-reth's payload builder and the engine tree's resources, before it answers the engine's BuildNewPayload command. The engine awaits that answer in engine_forkchoiceUpdated. If the transactions make the payload invalid, the service rejects the attributes and the engine answers -38003 with the build error in data, so op-node and kona-node (fixed here) replace the block with a deposits-only one. Every other command goes to the wrapped service. OpNode wraps its payload service; op-reth-premium opts in with a one-line wrap.

reth is pinned to the fork tag op-v2.6.0-1, which adds ethereum-optimism/reth#19, a backport of paradigmxyz/reth#28015, which keeps a payload job's start error: -38003 only when the payload builder rejects the attributes, -32603 for any other failure. Before, reth answered -38003 for every job start failure, including failures of the node.

  • The build runs on a blocking task. The engine tree doesn't wait for it; only the forkchoiceUpdated response does.
  • engine_getPayload resolves the built payload like a payload job's and emits Events::BuiltPayload on the first resolution, as reth's service does, so engine_newPayload doesn't execute the block a second time.
  • A build that fails for a node-local reason (missing state, provider error, panic) is never -38003; it goes to the wrapped service with the engine's leases. If that fails too, the answer is -32603, which op-node and kona-node treat as temporary and retry.
  • A retried request joins the running build or gets the built payload.
  • For derived attributes with an invalid transaction, debug_executePayload returns the witness up to that transaction, where the fault proof program stops, instead of the witness of a block without it.
  • op-reth and the fault proof program decide which execution errors make a derived payload invalid with one function, alloy_op_evm::block::is_invalid_payload_error (from alloy-op-evm,kona,op-reth!: share the rules for invalid derived payloads #23509).
  • kona-node handles attributes rejected at forkchoiceUpdated and at newPayload with one rule, op-node's onBuildInvalid (from alloy-op-evm,kona,op-reth!: share the rules for invalid derived payloads #23509). The seal task no longer replaces a sequencer's own rejected payload with a deposits-only block, which it never gossiped.

Supersedes #23188, whose kona-node fix and acceptance test this carries. Compared with it, with an earlier private fix (optimism-private@ed9e4d8) and with #23447:

#23188 ed9e4d8 #23447 this PR
Where it checks payload job generator, at job creation OpEngineValidator::validate_payload_attributes_against_header OpEngineApi, in engine_forkchoiceUpdated payload service wrapper, at BuildNewPayload
Derived attributes covered 0x7D only all with a non-deposit tx all with a non-deposit tx (one-line switch to all noTxPool) all with a non-deposit tx (one-line switch to all noTxPool)
Executions per derived block 1 2 (validate, then build) 1 1
op-reth-premium not covered (own job generator) covered covered covered with a one-line wrap
Invalid forced tx skipped (fault-proof mismatch) yes fixed fixed fixed
kona-node fix + acceptance test yes no yes yes
Upstream changes none none reth hook (ethereum-optimism/reth#16) reth keeps payload job start errors (ethereum-optimism/reth#19)
Costs blocks the payload service while building; copy of reth's payload service builder double execution; breaking Evm parameter on OpEngineValidator copies reth's attribute checks; FCU serialized by a lock payload services must opt in

Downsides of this approach:

  • A payload service that isn't wrapped isn't covered: premium and third-party node builders must opt in.
  • It proxies reth's PayloadServiceCommand enum; a new command upstream needs a match arm here (a compile error, not a silent gap).
  • Events::BuiltPayload must fire exactly when reth's service fires it, or engine_newPayload re-executes the block.
  • After a node-local failure, the wrapped service's fallback build runs without the tree's execution cache and state root task (it keeps the leases).

Review agents (Rust, security) ran; their findings are fixed except, to confirm:

  • The switch to all noTxPool attributes stays a const, as that choice is still open.
  • A payload id collision with different attributes is only partly handled: a request joins a build or a built payload only if the attributes match too, but engine_getPayload looks payloads up by id. Ids hash every attribute field, and reth trusts them the same way.
  • Events::Attributes is emitted when a derived build starts, as reth does when it creates a job, so after a node-local failure the wrapped service emits it again. Nothing in reth consumes it.
  • The wrapper requirement is stated on OpNode::components and on its return type, and on OpNode::payload_builder and on its return type, so a node author finds it wherever they start.
  • The kona-node tests for -32603 and for a transport failure at forkchoiceUpdated stay two tests.
  • Undecodable transactions and malformed 0x7D payloads are classified as invalid by their error type (alloy_rlp::Error, PostExecPayloadValidationError). Today only attribute decoding produces those types; tagging the errors where the attributes are decoded would be more robust.
  • Not changed here, as in op-reth,kona-node!: build derived payloads at forkchoiceUpdated #23447: before Holocene, kona-node retries a -38003 for derived attributes instead of dropping them, so it stalls on a pre-Holocene batch with an invalid transaction; a transient error while kona-node builds the deposits-only replacement is critical; kona-node gates the replacement on the L2 timestamp, op-node on the L1 origin's.

The kona-node commit adds a test that -32603 at forkchoiceUpdated stays temporary; otherwise it and the acceptance-test commit are the ones from #23447, where the Go and derivation-batch reviewers ran.

BREAKING CHANGE: op-reth answers engine_forkchoiceUpdated with -38003 when the transactions of derived attributes make the block invalid. op-node handles this. kona-node needs the change in this PR, otherwise it retries and stalls the safe head, so upgrade both together. A payload job that fails to start for any other reason now gets -32603 instead of -38003. OpNodeComponentBuilder's default payload service builder is now DerivedPayloadServiceBuilder<OpPayloadServiceBuilder>; nodes that replace the payload service must wrap theirs in DerivedPayloadServiceBuilder to be covered. alloy-op-evm's OpBlockExecutionError no longer has the LoadCacheAccount and GetJovianDaFootprintScalar variants; nothing returned them, so code that matches on them only drops those arms.

🤖 Co-created with Claude Opus 5.5

Closes https://github.com/ethereum-optimism/protocol-team/issues/368
Fixes ethereum-optimism/protocol-team#341
Fixes ethereum-optimism/protocol-team#308
Fixes ethereum-optimism/protocol-team#309

nonsense
nonsense previously approved these changes Oct 20, 2026
sebastianst and others added 7 commits October 20, 2026 13:31
DerivedPayloadServiceBuilder wraps a payload service builder. The service it
spawns builds the payload of derived (noTxPool) attributes that carry a
non-deposit transaction before it answers the engine's BuildNewPayload, with
op-reth's payload builder and the engine's resources, on a blocking task off
the engine tree. If the transactions make the block invalid it rejects the
attributes, which the engine answers with -38003 and the build error, so the
rollup node replaces the block with a deposits-only one or drops the batch.
Before, the payload service either skipped the invalid transaction and built
a block the fault proof rejects, or failed the build at getPayload, which the
rollup node retries forever. The builder no longer skips invalid transactions
of noTxPool attributes.

Built payloads are kept for getPayload and served like a payload job's:
resolving one emits the BuiltPayload event, so the engine inserts the executed
block and newPayload does not execute it again. Requests for one payload share
its build. Every other command, and a derived build that fails for a
node-local reason, goes to the wrapped service. If that fails too, the answer
is an internal error (-32603), which the rollup node retries. OpNode wraps its
payload service builder; nodes that bring their own payload service opt in by
wrapping it.

reth is pinned to a fork commit that keeps a payload job's start error and
maps it to -38003 only when the payload builder rejects the attributes.

The witness of derived attributes with an invalid transaction covers the
state read up to that transaction, where the fault proof program stops.

BREAKING CHANGE: op-reth answers engine_forkchoiceUpdated with -38003 when the
transactions of derived attributes make the block invalid. The rollup node
must then replace the block with a deposits-only block or drop the batch.
op-node does; kona-node needs the matching change, or it retries the
attributes and stalls the safe head, so upgrade both together. A payload job
that fails to start for any other reason is now -32603 instead of -38003. The
default payload service builder of OpNodeComponentBuilder is now
DerivedPayloadServiceBuilder<OpPayloadServiceBuilder>.

Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the execution layer rejects derived attributes at forkchoiceUpdated with
-38003 (invalid payload attributes), kona-node builds the deposits-only
replacement under Holocene, as op-node does, instead of retrying the
attributes forever. Any other error, such as an internal error (-32603),
stays temporary and is retried. The engine queue drops a task that ends in a
flush rather than retrying it, since retrying would replace the block again.

The forkchoiceUpdated and newPayload rejections share one deposits-only
replacement, which now logs why a failed replacement is critical.

Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Posts a batch whose 0x7D over-refunds a transaction and checks that the
sequencer and the verifier both replace the block with the same deposits-only
block, and that the safe head keeps advancing.

Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The derived payload service forwarded `BestPayload`, `PayloadTimestamp` and
`Resolve` for jobs it doesn't own from spawned tasks, so the wrapped service
could receive a later `BuildNewPayload` before them. It now sends each
command before it returns and only awaits the answer on a task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fier

The fault proof program and op-reth's payload builder must agree on which
failed block executions make a derived payload invalid. Such a payload is
replaced with a deposits-only block; any other failure is retried. Each held
its own copy of the same exhaustive match, and if the copies drift, op-reth
and the fault proof replace different blocks. The match now lives once, in
alloy_op_evm::block::is_invalid_payload_error, and kona-executor and op-reth
call it. The result is the same for every error.

OpBlockExecutionError loses LoadCacheAccount and GetJovianDaFootprintScalar.
No code constructs them: the executor reports a failed database read through
BlockExecutionError::other. The enum now states that each of its variants is
caused by the block's transactions.

BREAKING CHANGE: alloy-op-evm's OpBlockExecutionError no longer has the
LoadCacheAccount and GetJovianDaFootprintScalar variants. Code that matches
on them must drop those arms. Nothing returned them, so behaviour does not
change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The consolidate task (rejection at forkchoiceUpdated) and the seal task
(rejection at newPayload) each held their own copy of the deposits-only
rules. The seal task's copy did not check whether the attributes were
derived. A sequencer whose own payload the engine rejected under Holocene
therefore built a deposits-only block, made it its unsafe head and never
gossiped it. op-node replaces only derived blocks.

Both tasks now call handle_rejected_attributes, which follows op-node's
onBuildInvalid: rejected deposits-only attributes are critical, rejected
derived attributes are replaced with their deposits-only block under
Holocene, and any other rejection keeps the task's own error. The sequencer
then drops the payload and builds the next one, as op-node's does.
build_and_seal now only builds derived attributes, so it no longer takes
is_attributes_derived.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
witness_covers_a_derived_block_up_to_its_invalid_transaction passed and
then aborted the test process with `pthread lock: Invalid argument` in
two CI runs in a row. The test returned about 0.2 s after it launched a
node, while the node's engine thread could still be closing RocksDB, and
the process exit destroys the static mutexes that RocksDB's close path
locks. The pinned reth acknowledges engine termination before the engine
tree drops its providers, so a graceful shutdown in the test would not
remove that race.

The witness needs only a provider, so the test now builds on a provider
factory with the genesis state. No node thread outlives the test, and the
provider, with its database, is dropped before the test returns.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sebastianst
sebastianst force-pushed the seb/op-reth-derived-payload-service branch from 72d3896 to 168b762 Compare October 20, 2026 11:32
@geoknee

geoknee commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

nonsense
nonsense previously approved these changes Oct 20, 2026
ethereum-optimism/reth#19 merged into op-v2.6.0 as 89e32ef290, tagged
op-v2.6.0-1. It has the tree of the fork commit the reth crates were pinned
to by rev, plus the fork's CI change #21, which only touches a workflow file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sebastianst
sebastianst added this pull request to the merge queue Oct 20, 2026
Merged via the queue into develop with commit 53ad9f4 Oct 20, 2026
120 of 123 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants