Skip to content

op-reth: count every structural post-exec rejection on the payload-builder path - #23339

Merged
einar-oplabs merged 4 commits into
developfrom
einar/23338-attributes-path-counting
Oct 20, 2026
Merged

einar-oplabs merged 4 commits into
developfrom
einar/23338-attributes-path-counting

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

The Engine API newPayload preflight and block import count SDM post-exec structural rejections in reth_op_sdm_post_exec_validation_failure_total, the series the P1 post-exec alert watches. The forkchoiceUpdated-with-transactions path applied the same 0x7D rules but rejected without a log line or a metric. That path is where a node rebuilds a block op-node derived, so a CL/EL disagreement about the post-exec transaction surfaced only as a failed getPayload, invisible to the alert.

Residual of protocol-team#376 / #23149. Closes #23338.

Consensus impact

The underlying bug is the SDM-H1 audit finding, protocol-team#368 (High, blocking). The Lagoon and Holocene derivation specs replace an invalid derived payload with a deposits-only block and drop the rest of the channel, which the Kona proof program does. An op-reth-backed op-node or kona-node instead retries the same attributes forever, because op-reth only fails inside the async build and getPayload returns -38001, which neither client treats as a deposits-only trigger. Every node stalls at that block while the proof system attests to a chain that keeps going. There is no split among nodes, so the direct effect is liveness: the safe chain stops advancing, deposits included. The trigger needs the batcher key or an honest producer batching a verifier-rejected 0x7D (FMA FM1a).

This PR changes none of that behavior. The error returned to the consensus client is identical; only the log line and the P1 alert's counter are new. The behavior fix is #23188, which replaces invalid SDM derived blocks with deposits-only in op-reth and kona; #23447 and #23454 are alternative shapes of that fix which build derived payloads inside forkchoiceUpdated and answer -38003. Both still pass through from_rpc_attrs, so the counters here keep firing after either lands.

Scope

This PR counts the structural rules: the five preflight reasons, Lagoon activation, and the block-number anchor. Execution-level rejections, including the H1 case itself (a refund larger than the gas used), fail later in the build and are not seen by this series. The derived-payload fix adds an invalid counter that covers those; it needs its own alert in k8s once it merges, and the existing alert text ("stalls safe-head progress") will need revisiting for the attributes path at the same time.

What

  • OpPayloadBuilderAttributes::from_rpc_attrs runs the encoded-transaction preflight (validate_post_exec_attributes in reth-optimism-evm) before decoding, reporting through the shared counters (…validation_failure_total{reason} and …validation_total{result="fail"}); the engine-facing error is unchanged. It counts here rather than in op-reth's engine-only convert_build_args because op-reth-premium's payload generator calls from_rpc_attrs directly and would otherwise lose coverage. The log is keyed by the parent hash, the only block identity the function and premium's call carry.
  • The builder's parse of the decoded attributes (parse_embedded_post_exec) reports the two rules the preflight cannot check, Lagoon activation and the block-number anchor, through the block-keyed report_post_exec_validation_failure, now exported from the evm crate. Without it an op-node 0x7D anchored to the wrong height still rejected invisibly.
  • The payload, txpool, and flashblocks crates depend on the evm crate with std, as the node and post-exec-replay crates already do. The counters are std-only, so without it they compiled to a no-op whenever one of those crates was built outside the node binary; the payload crate's own test run failed with the counter absent, and a feature-resolution sweep of the workspace found the other two resolving the evm crate the same way.
  • just check-no-std gains a host-target cargo check -p reth-optimism-evm --no-default-features: no lint or test target exercised the evm crate's std/no_std import split, so the required rust-check-no-std job now does.
  • A pass counts nothing; the import path owns ok.

Counts are per conversion into builder attributes. Today op-reth's BasicPayloadJob converts on job creation, again on getPayload, in the empty-payload fallback, and once per interval tick after a failed attempt, so one rejected attributes set counts several times; premium's generator converts once per job, and once derived payloads are built at forkchoiceUpdated op-reth converts a derived set once as well. The P1 alert is increase(...[5m]) > 0, so it fires on a single rejection and resolves once the node stops rejecting; nothing here depends on the series continuing to move. Counting once per forkchoiceUpdated in the engine validator instead would turn the async getPayload failure into a synchronous INVALID_PAYLOAD_ATTRIBUTES, which is the behavior change the fix PRs own. Replays through debug_executePayload and debug_executionWitness share the conversion and count too.

Tests

  • evm: rejected_transactions_count_at_every_preflight (five reasons × Engine API / attributes entry points, one table for both seams) and accepted_transactions_count_nothing_at_any_preflight.
  • payload: rejected_attributes_count_the_failure, accepted_attributes_count_nothing (asserts absent-or-zero counters, not an absent metric name), and rejected_embedded_post_exec_counts_under_the_failed_rule (wrong anchor → block_number_mismatch, pre-activation → unexpected_post_exec_tx).

93/93 across both crates, 35/35 for the payload crate alone, clippy clean with all features and targets, host no_std check green.

Line breakdown

Category Added Removed Net
Implementation 36 6 30
Docs and comments 20 6 14
Tests 182 28 154
Config (Cargo.toml, justfile) 11 3 8
Blank 22 0 22
Total 271 43 228

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 12, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23338-attributes-path-counting branch from 080d676 to 47f832d Compare October 12, 2026 04:58
@einar-oplabs
einar-oplabs requested a review from geoknee October 12, 2026 06:52
@einar-oplabs
einar-oplabs marked this pull request as ready for review October 12, 2026 14:09
@einar-oplabs
einar-oplabs requested a review from a team as a code owner October 12, 2026 14:09

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fable 5.1 (xhigh): Review of the attributes-path counting. CI is green and the seam does what the description says, but two findings are functional gaps and two change what the P1 series means. Most severe first; inline comments carry the details.

  1. Counting is a silent no-op unless another crate unifies std in (inline on payload.rs). Reproducible: cargo nextest run -p reth-optimism-payload-builder fails the new rejected_attributes_count_the_failure test with None instead of Some(1.0).
  2. A second rejection point on the same path is still uncounted. parse_embedded_post_exec in crates/payload/src/builder.rs (reached via ctx.post_exec_mode()? in try_build) rejects a 0x7D anchored to the wrong block number or present before Lagoon and maps it straight to PayloadBuilderError::other at debug! level, with no sdm_metrics call. That is the CL/EL disagreement the P1 alert exists for, so block_number_mismatch stays flat on this path.
  3. Caller-supplied RPC input now bumps the P1 series (inline on lib.rs): debug_executePayload reaches from_rpc_attrs through Attrs::try_new.
  4. One rejected attributes set counts at least twice even without interval retries (inline on lib.rs): job creation, the resolve_kind re-spawn on getPayload, and the empty-payload fallback each run convert_build_args.
  5. No CI gate builds reth-optimism-evm without std. just check-no-std (rust/justfile) omits the crate and every lint/test target runs --all-features, so the std/no_std import split this PR hand-fixed can regress silently. A host-target cargo check -p reth-optimism-evm --no-default-features in just lint would cover it.

One change resolves 3 and 4's documentation together with two smaller notes: move the counting call from from_rpc_attrs into convert_build_args, which is engine-only, owns the per-attempt behavior, and holds the parent header, so the next block number is available and report_post_exec_validation_failure can be reused outright.

Fixes for all of the above are being pushed to this branch.

Comment thread rust/op-reth/crates/evm/src/lib.rs Resolved
Comment thread rust/op-reth/crates/evm/src/lib.rs Resolved
@einar-oplabs
einar-oplabs force-pushed the einar/23338-attributes-path-counting branch from d7927df to 68cdbd5 Compare October 12, 2026 15:57
The Engine API newPayload preflight and block import count SDM post-exec
structural rejections, but the forkchoiceUpdated-with-transactions path
validated the same 0x7D rules and rejected without a log line or a metric, so
the P1 post-exec alert could not see a CL/EL disagreement that surfaced only
as a failed getPayload.

OpPayloadBuilderAttributes::from_rpc_attrs now runs the encoded-transaction
preflight through the shared sdm_metrics counters before decoding, and the
builder's parse of the decoded attributes reports the activation and anchor
rules the preflight cannot check through the same block-keyed report. The
payload crate depends on the evm crate with std, which the counters need, and
check-no-std gains a host-target no_std check of the evm crate.

Counting is per conversion: op-reth's payload job converts on every build
attempt and premium's generator once per job, so a persistent rejection keeps
the series moving for as long as the attempts continue.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23338-attributes-path-counting branch from 68cdbd5 to 62af7ef Compare October 12, 2026 16:04
@einar-oplabs einar-oplabs changed the title op-reth: count post-exec rejections on the payload-builder attributes path op-reth: count every post-exec rejection on the payload-builder path Oct 12, 2026
@einar-oplabs
einar-oplabs requested a review from nonsense October 13, 2026 05:28
@einar-oplabs

einar-oplabs commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Fable 5.1 (xhigh): Additional details: the gap was on the execution layer side, in op-reth. The consensus layer never counted these; the metric the P1 alert watches is op-reth's own series, and the consensus client only sees the result as an Engine API error.

A post-exec transaction reaches op-reth through two kinds of Engine API call:

  • newPayload carries a complete block, usually one the node received over gossip or during sync. op-reth checks the 0x7D structure before decoding and again when it builds the execution context, and both checks logged a warning and incremented the failure counter when they rejected.
  • forkchoiceUpdated with attributes carries transactions the consensus client, op-node or kona-node, derived from L1 batches, including the 0x7D it embedded, and asks op-reth to rebuild that block itself. op-reth ran the same structural checks when converting the attributes, but on rejection it only returned an error. The forkchoiceUpdated call still answered with a payload id, every build attempt failed quietly, and the consensus client eventually got a failed getPayload. No warning, no counter.

The second path is the one that matters most for the alert. It is exactly where a disagreement between consensus and execution about the post-exec transaction shows up: the consensus client believes the derived block is valid and op-reth refuses to build it. That is the chain-split-risk condition the alert was written for, and on that path it was invisible.

This PR counts both stages of that path. The attribute conversion reports the five structural rules it can check on raw bytes, and the builder's later parse reports the two it cannot, Lagoon activation and the block-number anchor.

Comment thread rust/op-reth/crates/payload/Cargo.toml Resolved

@geoknee geoknee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The std change is fine. reth-optimism-evm and the payload builder aren't in kona-client's dependency graph or any other proof program's. In the op-reth binary, reth-optimism-node and reth-optimism-post-exec-replay already turn on the evm crate's std. So the new edge only affects standalone builds of the payload crate, and there it's what makes the counters real. The added host-target --no-default-features check is a good addition.

My main concern is how this lands next to the derived-payload fix, whichever of #23447 and #23454 we take. Both build noTxPool attributes with a non-deposit tx inside engine_forkchoiceUpdated and answer -38003 when they're invalid. That covers every derived attributes set that carries a 0x7D. Neither touches payload.rs or the evm crate. Both go through OpPayloadBuilder::try_build, then convert_build_args, then from_rpc_attrs, so the counters here still fire. Merging is easy: each conflicts with this PR only where both append a test to the end of payload/src/builder/tests.rs, and the resolution is to keep both. I merged this PR on top of each one locally: payload and evm tests pass, 95/95 on each, and reth-optimism-node and reth-optimism-rpc compile with their tests.

Some of what this PR says about counting stops being true once either lands:

  1. The per-attempt rationale no longer holds for derived attributes. The module doc in evm/src/sdm_metrics.rs (L8–11) says a persistent rejection "keeps the series moving for as long as the attempts continue". The doc on convert_build_args in payload/src/builder.rs (L384–385) makes the same per-attempt point, and so does the PR body ("which the alert relies on"). With either fix, a rejected derived attributes set is converted once at FCU, retries join that build, and the rollup node moves on to a deposits-only block. The counter ticks about once per bad batch. The alert still fires, since it's increase(...[5m]) > 0 with for: 0m, but it resolves about 5 minutes later. I'd reword both comments to say counting is per conversion, and that a rejection is repeated only by builders that keep reconverting. Then nothing claims the alert needs a moving series.

  2. There will be two counters for the same event, and they cover different sets. #23447 and #23454 each add derived-payload counters: built, invalid and failed, under optimism_rpc.engine.derived_payloads or optimism_payload_builder.derived respectively. For a structural 0x7D rejection, invalid and this PR's …validation_failure_total{reason} both increment. That's fine, since reason is the useful part here. But only invalid sees execution-level rejections, including the H1 case itself (refund larger than gas used). So this PR's series, the one the P1 alert watches, never fires for the bug that started this. Not a blocker for this PR. Once one of the fixes merges, we'll want an alert on its invalid counter in k8s.

  3. The title overpromises. "count every post-exec rejection" covers the structural rules: the five preflight reasons, activation, and the anchor. Could we say "structural" in the title, or note in the body that execution-level rejections are counted by the derived-payload fix?

  4. The alert text will be wrong for this path. It says the node "returned INVALID for the block… stalls safe-head progress". After the fix, a rejection on the attributes path means a deposits-only replacement, not a stall. That's a k8s annotation change and out of scope here. Mentioning it so the PR body doesn't lean on it.

Placing the preflight in from_rpc_attrs instead of convert_build_args still makes sense with either fix, because premium's generator calls it directly. Happy to approve once the wording in (1) is updated.

@einar-oplabs einar-oplabs changed the title op-reth: count every post-exec rejection on the payload-builder path op-reth: count every structural post-exec rejection on the payload-builder path Oct 16, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Fable 5.1 (xhigh): Thanks for merging it against both fix shapes locally.

  1. Reworded in abe61d0: the sdm_metrics module doc now says attributes count once per conversion and a rejection repeats only when a builder reconverts the same attributes; the convert_build_args doc says each call counts, so a job that converts more than once counts each conversion. Nothing claims the alert needs a moving series. The PR body's counting paragraph is rewritten the same way and states the alert fires on a single rejection and resolves once the node stops rejecting.
  2. Agreed, and noted in a new Scope section of the body: the H1 refund case is execution-level and only the derived-payload invalid counter sees it, so that counter needs its own k8s alert once the fix merges.
  3. Title now reads "count every structural post-exec rejection on the payload-builder path".
  4. Noted in the same Scope section as a follow-up to the k8s annotation; the body no longer leans on the current alert text.

…crates

Both crates are std-only and depend on reth-optimism-evm through the workspace
entry, which turns default features off, so a standalone build of either
resolved the evm crate without std and its SDM counters compiled to a no-op.
The op-reth binary was unaffected because the node crate unifies std in; the
edge is now declared where it is needed, as the payload crate already does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs requested a review from geoknee October 20, 2026 13:41
…utes-path-counting

# Conflicts:
#	rust/op-reth/crates/payload/src/builder/tests.rs

@geoknee geoknee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all four points are addressed. The per-conversion wording and the Scope section read right, and making std explicit in txpool and flashblocks is the better fix for the implicit unification you spotted. Those crates are host-only, so the fault proof is still unaffected. The merge resolution keeps both tests and moves the with_local_recorder import to file scope, which is correct. One nit: the body still names #23188 as "the behavior fix", though #23447 and #23454 both supersede it. LGTM.

@einar-oplabs
einar-oplabs added this pull request to the merge queue Oct 20, 2026
Merged via the queue into develop with commit 219842a Oct 20, 2026
111 checks passed
@einar-oplabs
einar-oplabs deleted the einar/23338-attributes-path-counting branch October 20, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: count post-exec rejections on the payload-builder attributes path

2 participants