Repository navigation
op-reth: count every structural post-exec rejection on the payload-builder path - #23339
Conversation
080d676 to
47f832d
Compare
einar-oplabs
left a comment
There was a problem hiding this comment.
Fable 5.1 (xhigh): Review of the attributes-path counting. CI is green and the seam does what the description says, but two findings are functional gaps and two change what the P1 series means. Most severe first; inline comments carry the details.
- Counting is a silent no-op unless another crate unifies
stdin (inline onpayload.rs). Reproducible:cargo nextest run -p reth-optimism-payload-builderfails the newrejected_attributes_count_the_failuretest withNoneinstead ofSome(1.0). - A second rejection point on the same path is still uncounted.
parse_embedded_post_execincrates/payload/src/builder.rs(reached viactx.post_exec_mode()?intry_build) rejects a0x7Danchored to the wrong block number or present before Lagoon and maps it straight toPayloadBuilderError::otheratdebug!level, with nosdm_metricscall. That is the CL/EL disagreement the P1 alert exists for, soblock_number_mismatchstays flat on this path. - Caller-supplied RPC input now bumps the P1 series (inline on
lib.rs):debug_executePayloadreachesfrom_rpc_attrsthroughAttrs::try_new. - One rejected attributes set counts at least twice even without interval retries (inline on
lib.rs): job creation, theresolve_kindre-spawn ongetPayload, and the empty-payload fallback each runconvert_build_args. - No CI gate builds
reth-optimism-evmwithoutstd.just check-no-std(rust/justfile) omits the crate and every lint/test target runs--all-features, so the std/no_std import split this PR hand-fixed can regress silently. A host-targetcargo check -p reth-optimism-evm --no-default-featuresinjust lintwould cover it.
One change resolves 3 and 4's documentation together with two smaller notes: move the counting call from from_rpc_attrs into convert_build_args, which is engine-only, owns the per-attempt behavior, and holds the parent header, so the next block number is available and report_post_exec_validation_failure can be reused outright.
Fixes for all of the above are being pushed to this branch.
d7927df to
68cdbd5
Compare
The Engine API newPayload preflight and block import count SDM post-exec structural rejections, but the forkchoiceUpdated-with-transactions path validated the same 0x7D rules and rejected without a log line or a metric, so the P1 post-exec alert could not see a CL/EL disagreement that surfaced only as a failed getPayload. OpPayloadBuilderAttributes::from_rpc_attrs now runs the encoded-transaction preflight through the shared sdm_metrics counters before decoding, and the builder's parse of the decoded attributes reports the activation and anchor rules the preflight cannot check through the same block-keyed report. The payload crate depends on the evm crate with std, which the counters need, and check-no-std gains a host-target no_std check of the evm crate. Counting is per conversion: op-reth's payload job converts on every build attempt and premium's generator once per job, so a persistent rejection keeps the series moving for as long as the attempts continue. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
68cdbd5 to
62af7ef
Compare
|
Fable 5.1 (xhigh): Additional details: the gap was on the execution layer side, in op-reth. The consensus layer never counted these; the metric the P1 alert watches is op-reth's own series, and the consensus client only sees the result as an Engine API error. A post-exec transaction reaches op-reth through two kinds of Engine API call:
The second path is the one that matters most for the alert. It is exactly where a disagreement between consensus and execution about the post-exec transaction shows up: the consensus client believes the derived block is valid and op-reth refuses to build it. That is the chain-split-risk condition the alert was written for, and on that path it was invisible. This PR counts both stages of that path. The attribute conversion reports the five structural rules it can check on raw bytes, and the builder's later parse reports the two it cannot, Lagoon activation and the block-number anchor. |
geoknee
left a comment
There was a problem hiding this comment.
The std change is fine. reth-optimism-evm and the payload builder aren't in kona-client's dependency graph or any other proof program's. In the op-reth binary, reth-optimism-node and reth-optimism-post-exec-replay already turn on the evm crate's std. So the new edge only affects standalone builds of the payload crate, and there it's what makes the counters real. The added host-target --no-default-features check is a good addition.
My main concern is how this lands next to the derived-payload fix, whichever of #23447 and #23454 we take. Both build noTxPool attributes with a non-deposit tx inside engine_forkchoiceUpdated and answer -38003 when they're invalid. That covers every derived attributes set that carries a 0x7D. Neither touches payload.rs or the evm crate. Both go through OpPayloadBuilder::try_build, then convert_build_args, then from_rpc_attrs, so the counters here still fire. Merging is easy: each conflicts with this PR only where both append a test to the end of payload/src/builder/tests.rs, and the resolution is to keep both. I merged this PR on top of each one locally: payload and evm tests pass, 95/95 on each, and reth-optimism-node and reth-optimism-rpc compile with their tests.
Some of what this PR says about counting stops being true once either lands:
-
The per-attempt rationale no longer holds for derived attributes. The module doc in
evm/src/sdm_metrics.rs(L8–11) says a persistent rejection "keeps the series moving for as long as the attempts continue". The doc onconvert_build_argsinpayload/src/builder.rs(L384–385) makes the same per-attempt point, and so does the PR body ("which the alert relies on"). With either fix, a rejected derived attributes set is converted once at FCU, retries join that build, and the rollup node moves on to a deposits-only block. The counter ticks about once per bad batch. The alert still fires, since it'sincrease(...[5m]) > 0withfor: 0m, but it resolves about 5 minutes later. I'd reword both comments to say counting is per conversion, and that a rejection is repeated only by builders that keep reconverting. Then nothing claims the alert needs a moving series. -
There will be two counters for the same event, and they cover different sets. #23447 and #23454 each add derived-payload counters:
built,invalidandfailed, underoptimism_rpc.engine.derived_payloadsoroptimism_payload_builder.derivedrespectively. For a structural0x7Drejection,invalidand this PR's…validation_failure_total{reason}both increment. That's fine, sincereasonis the useful part here. But onlyinvalidsees execution-level rejections, including the H1 case itself (refund larger than gas used). So this PR's series, the one the P1 alert watches, never fires for the bug that started this. Not a blocker for this PR. Once one of the fixes merges, we'll want an alert on itsinvalidcounter in k8s. -
The title overpromises. "count every post-exec rejection" covers the structural rules: the five preflight reasons, activation, and the anchor. Could we say "structural" in the title, or note in the body that execution-level rejections are counted by the derived-payload fix?
-
The alert text will be wrong for this path. It says the node "returned INVALID for the block… stalls safe-head progress". After the fix, a rejection on the attributes path means a deposits-only replacement, not a stall. That's a k8s annotation change and out of scope here. Mentioning it so the PR body doesn't lean on it.
Placing the preflight in from_rpc_attrs instead of convert_build_args still makes sense with either fix, because premium's generator calls it directly. Happy to approve once the wording in (1) is updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Fable 5.1 (xhigh): Thanks for merging it against both fix shapes locally.
|
…crates Both crates are std-only and depend on reth-optimism-evm through the workspace entry, which turns default features off, so a standalone build of either resolved the evm crate without std and its SDM counters compiled to a no-op. The op-reth binary was unaffected because the node crate unifies std in; the edge is now declared where it is needed, as the payload crate already does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes-path-counting # Conflicts: # rust/op-reth/crates/payload/src/builder/tests.rs
geoknee
left a comment
There was a problem hiding this comment.
Thanks, all four points are addressed. The per-conversion wording and the Scope section read right, and making std explicit in txpool and flashblocks is the better fix for the implicit unification you spotted. Those crates are host-only, so the fault proof is still unaffected. The merge resolution keeps both tests and moves the with_local_recorder import to file scope, which is correct. One nit: the body still names #23188 as "the behavior fix", though #23447 and #23454 both supersede it. LGTM.
Why
The Engine API
newPayloadpreflight and block import count SDM post-exec structural rejections inreth_op_sdm_post_exec_validation_failure_total, the series the P1 post-exec alert watches. TheforkchoiceUpdated-with-transactions path applied the same0x7Drules but rejected without a log line or a metric. That path is where a node rebuilds a block op-node derived, so a CL/EL disagreement about the post-exec transaction surfaced only as a failedgetPayload, invisible to the alert.Residual of protocol-team#376 / #23149. Closes #23338.
Consensus impact
The underlying bug is the SDM-H1 audit finding, protocol-team#368 (High, blocking). The Lagoon and Holocene derivation specs replace an invalid derived payload with a deposits-only block and drop the rest of the channel, which the Kona proof program does. An op-reth-backed op-node or kona-node instead retries the same attributes forever, because op-reth only fails inside the async build and
getPayloadreturns -38001, which neither client treats as a deposits-only trigger. Every node stalls at that block while the proof system attests to a chain that keeps going. There is no split among nodes, so the direct effect is liveness: the safe chain stops advancing, deposits included. The trigger needs the batcher key or an honest producer batching a verifier-rejected0x7D(FMA FM1a).This PR changes none of that behavior. The error returned to the consensus client is identical; only the log line and the P1 alert's counter are new. The behavior fix is #23188, which replaces invalid SDM derived blocks with deposits-only in op-reth and kona; #23447 and #23454 are alternative shapes of that fix which build derived payloads inside
forkchoiceUpdatedand answer -38003. Both still pass throughfrom_rpc_attrs, so the counters here keep firing after either lands.Scope
This PR counts the structural rules: the five preflight reasons, Lagoon activation, and the block-number anchor. Execution-level rejections, including the H1 case itself (a refund larger than the gas used), fail later in the build and are not seen by this series. The derived-payload fix adds an
invalidcounter that covers those; it needs its own alert in k8s once it merges, and the existing alert text ("stalls safe-head progress") will need revisiting for the attributes path at the same time.What
OpPayloadBuilderAttributes::from_rpc_attrsruns the encoded-transaction preflight (validate_post_exec_attributesinreth-optimism-evm) before decoding, reporting through the shared counters (…validation_failure_total{reason}and…validation_total{result="fail"}); the engine-facing error is unchanged. It counts here rather than in op-reth's engine-onlyconvert_build_argsbecause op-reth-premium's payload generator callsfrom_rpc_attrsdirectly and would otherwise lose coverage. The log is keyed by the parent hash, the only block identity the function and premium's call carry.parse_embedded_post_exec) reports the two rules the preflight cannot check, Lagoon activation and the block-number anchor, through the block-keyedreport_post_exec_validation_failure, now exported from the evm crate. Without it an op-node0x7Danchored to the wrong height still rejected invisibly.std, as the node and post-exec-replay crates already do. The counters are std-only, so without it they compiled to a no-op whenever one of those crates was built outside the node binary; the payload crate's own test run failed with the counter absent, and a feature-resolution sweep of the workspace found the other two resolving the evm crate the same way.just check-no-stdgains a host-targetcargo check -p reth-optimism-evm --no-default-features: no lint or test target exercised the evm crate's std/no_std import split, so the requiredrust-check-no-stdjob now does.ok.Counts are per conversion into builder attributes. Today op-reth's
BasicPayloadJobconverts on job creation, again ongetPayload, in the empty-payload fallback, and once per interval tick after a failed attempt, so one rejected attributes set counts several times; premium's generator converts once per job, and once derived payloads are built atforkchoiceUpdatedop-reth converts a derived set once as well. The P1 alert isincrease(...[5m]) > 0, so it fires on a single rejection and resolves once the node stops rejecting; nothing here depends on the series continuing to move. Counting once perforkchoiceUpdatedin the engine validator instead would turn the asyncgetPayloadfailure into a synchronousINVALID_PAYLOAD_ATTRIBUTES, which is the behavior change the fix PRs own. Replays throughdebug_executePayloadanddebug_executionWitnessshare the conversion and count too.Tests
rejected_transactions_count_at_every_preflight(five reasons × Engine API / attributes entry points, one table for both seams) andaccepted_transactions_count_nothing_at_any_preflight.rejected_attributes_count_the_failure,accepted_attributes_count_nothing(asserts absent-or-zero counters, not an absent metric name), andrejected_embedded_post_exec_counts_under_the_failed_rule(wrong anchor →block_number_mismatch, pre-activation →unexpected_post_exec_tx).93/93 across both crates, 35/35 for the payload crate alone, clippy clean with all features and targets, host no_std check green.
Line breakdown
🤖 Generated with Claude Code