Repository navigation
op-reth,kona-node!: build derived payloads in a payload service wrapper - #23454
Merged
Merged
Conversation
sebastianst
force-pushed
the
seb/op-reth-derived-payload-service
branch
from
October 15, 2026 15:35
3d7c776 to
3bf37ce
Compare
sebastianst
force-pushed
the
seb/op-reth-derived-payload-service
branch
2 times, most recently
from
October 16, 2026 18:40
3964882 to
5cdfbbe
Compare
sebastianst
force-pushed
the
seb/op-reth-derived-payload-service
branch
2 times, most recently
from
October 19, 2026 14:42
0e32d68 to
542ddae
Compare
sebastianst
marked this pull request as ready for review
October 19, 2026 15:00
nonsense
previously approved these changes
Oct 20, 2026
DerivedPayloadServiceBuilder wraps a payload service builder. The service it spawns builds the payload of derived (noTxPool) attributes that carry a non-deposit transaction before it answers the engine's BuildNewPayload, with op-reth's payload builder and the engine's resources, on a blocking task off the engine tree. If the transactions make the block invalid it rejects the attributes, which the engine answers with -38003 and the build error, so the rollup node replaces the block with a deposits-only one or drops the batch. Before, the payload service either skipped the invalid transaction and built a block the fault proof rejects, or failed the build at getPayload, which the rollup node retries forever. The builder no longer skips invalid transactions of noTxPool attributes. Built payloads are kept for getPayload and served like a payload job's: resolving one emits the BuiltPayload event, so the engine inserts the executed block and newPayload does not execute it again. Requests for one payload share its build. Every other command, and a derived build that fails for a node-local reason, goes to the wrapped service. If that fails too, the answer is an internal error (-32603), which the rollup node retries. OpNode wraps its payload service builder; nodes that bring their own payload service opt in by wrapping it. reth is pinned to a fork commit that keeps a payload job's start error and maps it to -38003 only when the payload builder rejects the attributes. The witness of derived attributes with an invalid transaction covers the state read up to that transaction, where the fault proof program stops. BREAKING CHANGE: op-reth answers engine_forkchoiceUpdated with -38003 when the transactions of derived attributes make the block invalid. The rollup node must then replace the block with a deposits-only block or drop the batch. op-node does; kona-node needs the matching change, or it retries the attributes and stalls the safe head, so upgrade both together. A payload job that fails to start for any other reason is now -32603 instead of -38003. The default payload service builder of OpNodeComponentBuilder is now DerivedPayloadServiceBuilder<OpPayloadServiceBuilder>. Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the execution layer rejects derived attributes at forkchoiceUpdated with -38003 (invalid payload attributes), kona-node builds the deposits-only replacement under Holocene, as op-node does, instead of retrying the attributes forever. Any other error, such as an internal error (-32603), stays temporary and is retried. The engine queue drops a task that ends in a flush rather than retrying it, since retrying would replace the block again. The forkchoiceUpdated and newPayload rejections share one deposits-only replacement, which now logs why a failed replacement is critical. Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Posts a batch whose 0x7D over-refunds a transaction and checks that the sequencer and the verifier both replace the block with the same deposits-only block, and that the safe head keeps advancing. Co-authored-by: Anton Evangelatov <anton.evangelatov@oplabs.co> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The derived payload service forwarded `BestPayload`, `PayloadTimestamp` and `Resolve` for jobs it doesn't own from spawned tasks, so the wrapped service could receive a later `BuildNewPayload` before them. It now sends each command before it returns and only awaits the answer on a task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fier The fault proof program and op-reth's payload builder must agree on which failed block executions make a derived payload invalid. Such a payload is replaced with a deposits-only block; any other failure is retried. Each held its own copy of the same exhaustive match, and if the copies drift, op-reth and the fault proof replace different blocks. The match now lives once, in alloy_op_evm::block::is_invalid_payload_error, and kona-executor and op-reth call it. The result is the same for every error. OpBlockExecutionError loses LoadCacheAccount and GetJovianDaFootprintScalar. No code constructs them: the executor reports a failed database read through BlockExecutionError::other. The enum now states that each of its variants is caused by the block's transactions. BREAKING CHANGE: alloy-op-evm's OpBlockExecutionError no longer has the LoadCacheAccount and GetJovianDaFootprintScalar variants. Code that matches on them must drop those arms. Nothing returned them, so behaviour does not change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The consolidate task (rejection at forkchoiceUpdated) and the seal task (rejection at newPayload) each held their own copy of the deposits-only rules. The seal task's copy did not check whether the attributes were derived. A sequencer whose own payload the engine rejected under Holocene therefore built a deposits-only block, made it its unsafe head and never gossiped it. op-node replaces only derived blocks. Both tasks now call handle_rejected_attributes, which follows op-node's onBuildInvalid: rejected deposits-only attributes are critical, rejected derived attributes are replaced with their deposits-only block under Holocene, and any other rejection keeps the task's own error. The sequencer then drops the payload and builds the next one, as op-node's does. build_and_seal now only builds derived attributes, so it no longer takes is_attributes_derived. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
witness_covers_a_derived_block_up_to_its_invalid_transaction passed and then aborted the test process with `pthread lock: Invalid argument` in two CI runs in a row. The test returned about 0.2 s after it launched a node, while the node's engine thread could still be closing RocksDB, and the process exit destroys the static mutexes that RocksDB's close path locks. The pinned reth acknowledges engine termination before the engine tree drops its providers, so a graceful shutdown in the test would not remove that race. The witness needs only a provider, so the test now builds on a provider factory with the genesis state. No node thread outlives the test, and the provider, with its database, is dropped before the test returns. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sebastianst
force-pushed
the
seb/op-reth-derived-payload-service
branch
from
October 20, 2026 11:32
72d3896 to
168b762
Compare
Contributor
nonsense
previously approved these changes
Oct 20, 2026
ethereum-optimism/reth#19 merged into op-v2.6.0 as 89e32ef290, tagged op-v2.6.0-1. It has the tree of the fork commit the reth crates were pinned to by rev, plus the fork's CI change #21, which only touches a workflow file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nonsense
self-requested a review
October 20, 2026 15:22
nonsense
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to #23447: same fix, built in a payload service wrapper instead of
OpEngineApi.Also tried and dropped: forwarding derived attributes to the wrapped service and verifying its payload (
0e32d682ee). It only kept op-reth-premium's subblocks for derived builds, which a sequencer only runs in recovery.op-reth only found out that a derived block's transactions were invalid in its asynchronous payload build, after
engine_forkchoiceUpdatedhad already answered VALID. An invalid forced transaction was skipped, so op-reth built a block that the fault proof rejects. An invalid0x7Dfailed the build instead, soengine_getPayloadreturned an error, the rollup node retried the same attributes forever and the safe head stalled.DerivedPayloadServiceBuilderwraps a payload service builder. FornoTxPoolattributes that carry a non-deposit transaction, the service it spawns builds the payload exactly once, with op-reth's payload builder and the engine tree's resources, before it answers the engine'sBuildNewPayloadcommand. The engine awaits that answer inengine_forkchoiceUpdated. If the transactions make the payload invalid, the service rejects the attributes and the engine answers-38003with the build error indata, so op-node and kona-node (fixed here) replace the block with a deposits-only one. Every other command goes to the wrapped service.OpNodewraps its payload service; op-reth-premium opts in with a one-line wrap.reth is pinned to the fork tag
op-v2.6.0-1, which adds ethereum-optimism/reth#19, a backport of paradigmxyz/reth#28015, which keeps a payload job's start error:-38003only when the payload builder rejects the attributes,-32603for any other failure. Before, reth answered-38003for every job start failure, including failures of the node.engine_getPayloadresolves the built payload like a payload job's and emitsEvents::BuiltPayloadon the first resolution, as reth's service does, soengine_newPayloaddoesn't execute the block a second time.-38003; it goes to the wrapped service with the engine's leases. If that fails too, the answer is-32603, which op-node and kona-node treat as temporary and retry.debug_executePayloadreturns the witness up to that transaction, where the fault proof program stops, instead of the witness of a block without it.alloy_op_evm::block::is_invalid_payload_error(from alloy-op-evm,kona,op-reth!: share the rules for invalid derived payloads #23509).onBuildInvalid(from alloy-op-evm,kona,op-reth!: share the rules for invalid derived payloads #23509). The seal task no longer replaces a sequencer's own rejected payload with a deposits-only block, which it never gossiped.Supersedes #23188, whose kona-node fix and acceptance test this carries. Compared with it, with an earlier private fix (optimism-private@ed9e4d8) and with #23447:
OpEngineValidator::validate_payload_attributes_against_headerOpEngineApi, inengine_forkchoiceUpdatedBuildNewPayload0x7DonlynoTxPool)noTxPool)Evmparameter onOpEngineValidatorDownsides of this approach:
PayloadServiceCommandenum; a new command upstream needs a match arm here (a compile error, not a silent gap).Events::BuiltPayloadmust fire exactly when reth's service fires it, orengine_newPayloadre-executes the block.Review agents (Rust, security) ran; their findings are fixed except, to confirm:
noTxPoolattributes stays aconst, as that choice is still open.engine_getPayloadlooks payloads up by id. Ids hash every attribute field, and reth trusts them the same way.Events::Attributesis emitted when a derived build starts, as reth does when it creates a job, so after a node-local failure the wrapped service emits it again. Nothing in reth consumes it.OpNode::componentsand on its return type, and onOpNode::payload_builderand on its return type, so a node author finds it wherever they start.-32603and for a transport failure at forkchoiceUpdated stay two tests.0x7Dpayloads are classified as invalid by their error type (alloy_rlp::Error,PostExecPayloadValidationError). Today only attribute decoding produces those types; tagging the errors where the attributes are decoded would be more robust.-38003for derived attributes instead of dropping them, so it stalls on a pre-Holocene batch with an invalid transaction; a transient error while kona-node builds the deposits-only replacement is critical; kona-node gates the replacement on the L2 timestamp, op-node on the L1 origin's.The kona-node commit adds a test that
-32603at forkchoiceUpdated stays temporary; otherwise it and the acceptance-test commit are the ones from #23447, where the Go and derivation-batch reviewers ran.BREAKING CHANGE: op-reth answers
engine_forkchoiceUpdatedwith-38003when the transactions of derived attributes make the block invalid. op-node handles this. kona-node needs the change in this PR, otherwise it retries and stalls the safe head, so upgrade both together. A payload job that fails to start for any other reason now gets-32603instead of-38003.OpNodeComponentBuilder's default payload service builder is nowDerivedPayloadServiceBuilder<OpPayloadServiceBuilder>; nodes that replace the payload service must wrap theirs inDerivedPayloadServiceBuilderto be covered. alloy-op-evm'sOpBlockExecutionErrorno longer has theLoadCacheAccountandGetJovianDaFootprintScalarvariants; nothing returned them, so code that matches on them only drops those arms.🤖 Co-created with Claude Opus 5.5
Closes https://github.com/ethereum-optimism/protocol-team/issues/368
Fixes ethereum-optimism/protocol-team#341
Fixes ethereum-optimism/protocol-team#308
Fixes ethereum-optimism/protocol-team#309