Skip to content

op-acceptance-tests,op-devstack: cover forkchoice updates with a stale safe or finalized hash - #23440

Closed
einar-oplabs wants to merge 1 commit into
einar/23398-withdrawals-root-persistencefrom
einar/23399-fcu-stale-safe-finalized
Closed

einar-oplabs wants to merge 1 commit into
einar/23398-withdrawals-root-persistencefrom
einar/23399-fcu-stale-safe-finalized

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

reth 2.6 refuses a forkchoice update whose safe or finalized hash is off the head's chain before applying anything, and develop's suites only touched that path incidentally.

Four tests in op_reth_stale_forkchoice: an L1 reorg rewinding a follower's safe head, a resync from a wiped datadir, a raw update finalizing a reorged-out block, and a raw update to a persisted block reorged out below the in-memory window. Adds a datadir wipe, a sibling-block builder and a -38002 assertion to the devstack.

Stacked on #23438. Closes #23399. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 174 21 +153
Tests 518 0 +518
Docs 0 0 0

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 15, 2026
@wiz-0f98cca50a

wiz-0f98cca50a Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 3 Medium
Software Management Finding Software Management Findings -
Total 3 Medium

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

n.p.Require().NotEmptyf(n.dataDirPath, "op-reth %s has no datadir to wipe", n.name)
n.p.Logger().Info("Wiping op-reth datadir", "name", n.name, "chain", n.chainID, "datadir", n.dataDirPath, "proofs_store", n.proofsHistoryDir)
n.p.Require().NoError(os.RemoveAll(n.dataDirPath), "must remove datadir")
n.p.Require().NoError(os.MkdirAll(n.dataDirPath, 0o755), "must create datadir")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium SAST Finding

Overly permissive directory permissions in Go (CWE-732)

More Details

This rule detects cases where Go applications create directories with overly permissive permissions, potentially allowing unauthorized access or modification of files within those directories. Assigning excessively open permissions to directories can lead to security vulnerabilities, as it may enable malicious actors or other processes to read, write, or execute files they should not have access to.

Consequences of this issue can include data leaks, data tampering, or even remote code execution, depending on the contents and purpose of the affected directories. To mitigate this risk, it is crucial to assign the least permissive access rights necessary for the application's intended functionality.

Attribute Value
Impact Medium
Likelihood Medium

Remediation

The vulnerability arises from assigning overly permissive permissions to directories created by the application. This can potentially allow unauthorized access or modification of files within those directories, leading to security risks such as data leaks, data tampering, or even remote code execution, depending on the contents and purpose of the affected directories.

To fix this issue securely, you should assign the least permissive access rights necessary for the application's intended functionality. Specifically, you should use the os.Mkdir or os.MkdirAll functions with a permission mask that restricts access to only the necessary users or processes.

Code examples

// VULNERABLE CODE - Assigns overly permissive permissions (0777) to the created directory
err := os.Mkdir("/path/to/dir", 0777)
// SECURE CODE - Assigns more restrictive permissions (0750) to the created directory
err := os.Mkdir("/path/to/dir", 0750)

Additional recommendations

  • Follow the principle of least privilege and assign the minimum required permissions for the application's functionality.
  • Consider using a more restrictive permission mask, such as 0700 (read, write, execute for the owner only), if the directory does not need to be accessed by other users or processes.
  • Regularly review and audit directory permissions to ensure they remain appropriate for the application's security requirements.
  • Adhere to relevant security standards and best practices, such as the OWASP Top 10 and CWE guidelines, which emphasize the importance of proper access control and resource protection.
  • If the application requires more granular access control, consider implementing additional security measures, such as file system access control lists (ACLs) or mandatory access control (MAC) systems.

Rule ID: WS-I011-GO-00011


To ignore this finding as an exception, reply to this conversation with #wiz_ignore reason

If you'd like to ignore this finding in all future scans, add an exception in the .wiz file (learn more) or create an Ignore Rule (learn more).


To get more details on how to remediate this issue using AI, reply to this conversation with #wiz remediate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude: #wiz_ignore Test harness code: this recreates a devstack op-reth datadir inside the test's temporary directory, and 0o755 matches every other devstack datadir (op-devstack/sysgo/l1_runtime.go, mixed_runtime.go). Nothing outside the test reads or writes it.

"--datadir=" + dataDirPath,
"--chain=" + chainConfigPath,
}
initOut, err := exec.CommandContext(t.Ctx(), execPath, initArgs...).CombinedOutput()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium SAST Finding

OS Command Injection (CWE-78)

More Details

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system.

The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application. This can occur when user-supplied data is passed directly to functions that execute OS commands, such as exec.Command, exec.CommandContext, syscall.ForkExec, or syscall.StartProcess.

If an attacker can inject malicious code into these commands, they can potentially execute any command on the system, including installing malware, stealing data, or causing a denial of service. This vulnerability can have severe consequences, including data breaches, system compromise, and unauthorized access to sensitive information.

To avoid this vulnerability, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code.

Attribute Value
Impact Medium
Likelihood Medium

Remediation

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system. The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application.

To fix this issue, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code. Additionally, it is recommended to use language-specific facilities for executing commands, which provide better input validation and sanitization.

Code examples

// VULNERABLE CODE - User input is directly used in constructing the command
userInput := getInput() // Assume this comes from user input
cmd := exec.Command("/bin/sh", "-c", userInput)
err := cmd.Run()
// SECURE CODE - Command and arguments are hardcoded, and user input is validated
userInput := getInput() // Assume this comes from user input
if !isValidInput(userInput) {
    // Handle invalid input
    return
}
cmd := exec.Command("/bin/sh", "-c", "echo", userInput)
err := cmd.Run()

Additional recommendations

  • Follow the principle of least privilege and run the application with the minimum required permissions.
  • Use input validation and sanitization techniques to ensure that user input does not contain malicious code.
  • Implement secure coding practices and follow security standards such as OWASP Top 10 and CWE.
  • Consider using language-specific facilities for executing commands, such as os/exec in Go, which provide better input validation and sanitization.
  • Regularly update and patch the application and its dependencies to address known vulnerabilities.
  • Implement logging and monitoring mechanisms to detect and respond to potential security incidents.

Rule ID: WS-I011-GO-00026


To ignore this finding as an exception, reply to this conversation with #wiz_ignore reason

If you'd like to ignore this finding in all future scans, add an exception in the .wiz file (learn more) or create an Ignore Rule (learn more).


To get more details on how to remediate this issue using AI, reply to this conversation with #wiz remediate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude: #wiz_ignore Test harness code: the command runs the op-reth binary the devstack resolved for this node, through exec.CommandContext without a shell, with flags the harness builds from its own generated datadir and chain config paths. No user or network input reaches the binary path or its arguments.

"--chain=" + chainConfigPath,
"--proofs-history.storage-path=" + storagePath,
}
initOut, err := exec.CommandContext(t.Ctx(), execPath, initProofsArgs...).CombinedOutput()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium SAST Finding

OS Command Injection (CWE-78)

More Details

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system.

The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application. This can occur when user-supplied data is passed directly to functions that execute OS commands, such as exec.Command, exec.CommandContext, syscall.ForkExec, or syscall.StartProcess.

If an attacker can inject malicious code into these commands, they can potentially execute any command on the system, including installing malware, stealing data, or causing a denial of service. This vulnerability can have severe consequences, including data breaches, system compromise, and unauthorized access to sensitive information.

To avoid this vulnerability, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code.

Attribute Value
Impact Medium
Likelihood Medium

Remediation

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system. The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application.

To fix this issue, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code. Additionally, it is recommended to use language-specific facilities for executing commands, which provide better input validation and sanitization.

Code examples

// VULNERABLE CODE - User input is directly used in constructing the command
userInput := getInput() // Assume this comes from user input
cmd := exec.Command("/bin/sh", "-c", userInput)
err := cmd.Run()
// SECURE CODE - Command and arguments are hardcoded, and user input is validated
userInput := getInput() // Assume this comes from user input
if !isValidInput(userInput) {
    // Handle invalid input
    return
}
cmd := exec.Command("/bin/sh", "-c", "echo", userInput)
err := cmd.Run()

Additional recommendations

  • Follow the principle of least privilege and run the application with the minimum required permissions.
  • Use input validation and sanitization techniques to ensure that user input does not contain malicious code.
  • Implement secure coding practices and follow security standards such as OWASP Top 10 and CWE.
  • Consider using language-specific facilities for executing commands, such as os/exec in Go, which provide better input validation and sanitization.
  • Regularly update and patch the application and its dependencies to address known vulnerabilities.
  • Implement logging and monitoring mechanisms to detect and respond to potential security incidents.

Rule ID: WS-I011-GO-00026


To ignore this finding as an exception, reply to this conversation with #wiz_ignore reason

If you'd like to ignore this finding in all future scans, add an exception in the .wiz file (learn more) or create an Ignore Rule (learn more).


To get more details on how to remediate this issue using AI, reply to this conversation with #wiz remediate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude: #wiz_ignore Test harness code: the command runs the op-reth binary the devstack resolved for this node, through exec.CommandContext without a shell, with flags the harness builds from its own generated datadir and chain config paths. No user or network input reaches the binary path or its arguments.

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 30, 2026
einar-oplabs added a commit that referenced this pull request Nov 4, 2026
… source

A reset can lower the safe head below a finalization target that the
node's own L2Finalizer requested from L1 finality, and the by-number
target then failed with BlockNotSafe and stopped the node (seen in
#23440's datadir-wipe CI run). Like op-node's promoteFinalized, skip a
target above the safe head with a warning whatever its source; the next
L1 finality update requests a newer block. Finality still never moves
above the safe head.

BREAKING CHANGE: FinalizeTaskError::BlockNotSafe is removed: a
finalization target above the safe head, by hash or by number, is now
logged and skipped instead of returning a Critical error. Remove any
match arm for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
einar-oplabs added a commit that referenced this pull request Nov 5, 2026
A finalization target the engine cannot apply yet no longer stops
kona-node. A target above the safe head is skipped with a warning,
whatever its source, as op-node's promoteFinalized does: a delegate can
run ahead of the local safe head, and a reset can lower the safe head
below a target that was safe when requested (#23757, #23440). A
hash-pinned target the execution layer has not downloaded yet is
skipped the same way. Neither sends a forkchoice update; the delegate
poll resubmits its target every 4s and the next L1 finality update
requests a newer block, so unsafe imports keep running while the
execution layer catches up.

Pinned lookups still never fall back to the local block at the same
height, and a returned block whose hash or number differs from the
target fails with the new BlockIdentityMismatch. A by-number target
whose block is missing still fails with BlockNotFound.

Tests cover the finalize task over missing, present, mismatched and
above-safe targets for both lookup kinds, recovery once a missing block
arrives, a queue that keeps importing while a target is missing, the
real EngineActor across two resets below a pending target, and the
delegated derivation actor's polling and L1 validation.

BREAKING CHANGE: This affects code that uses the kona-engine crate and
matches FinalizeTaskError exhaustively. BlockNotSafe is removed: a
target above the safe head is logged and skipped instead of returning a
Critical error. BlockIdentityMismatch { actual, expected } is added,
with Critical severity, for a pinned hash the engine answers with a
different block, including a height mismatch that previously returned
BlockNotFound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
einar-oplabs added a commit that referenced this pull request Nov 5, 2026
A finalization target the engine cannot apply yet no longer stops
kona-node. A target above the safe head is skipped with a warning,
whatever its source, as op-node's promoteFinalized does: a delegate can
run ahead of the local safe head, and a reset can lower the safe head
below a target that was safe when requested (#23757, #23440). A
hash-pinned target the execution layer has not downloaded yet is
skipped the same way. Neither sends a forkchoice update; the delegate
poll resubmits its target every 4s and the next L1 finality update
requests a newer block, so unsafe imports keep running while the
execution layer catches up.

Pinned lookups still never fall back to the local block at the same
height, and a returned block whose hash or number differs from the
target fails with the new BlockIdentityMismatch. A by-number target
whose block is missing still fails with BlockNotFound.

Tests cover the finalize task over missing, present, mismatched and
above-safe targets for both lookup kinds, recovery once a missing block
arrives, a queue that keeps importing while a target is missing, the
real EngineActor across two resets below a pending target, and the
delegated derivation actor's polling and L1 validation.

BREAKING CHANGE: This affects code that uses the kona-engine crate and
matches FinalizeTaskError exhaustively. BlockNotSafe is removed: a
target above the safe head is logged and skipped instead of returning a
Critical error. BlockIdentityMismatch { actual, expected } is added,
with Critical severity, for a pinned hash the engine answers with a
different block, including a height mismatch that previously returned
BlockNotFound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
einar-oplabs added a commit that referenced this pull request Nov 6, 2026
A finalization target the engine cannot apply yet no longer stops
kona-node or holds the engine queue. A target above the safe head
fails with `BlockNotSafe`, now of `Skip` severity, whatever its source,
as op-node's promoteFinalized does: a delegate can run ahead of the
local safe head, and a reset can lower the safe head below a target
that was safe when requested (#23757, #23440). A hash-pinned target
the execution layer has no block for yet fails with the new skipped
`NotYetAvailable`, whether the lookup returns null or an error
response, so a retried lookup cannot hold the queue ahead of the
imports that let the execution layer catch up. The delegate poll
resubmits its target every 4s.

A by-number target from L1 finality that a reset skipped is no longer
lost until L1 finality moves again: like op-node's retry when
derivation goes idle, the finalizer keeps the highest finalized L1
block and finalizes newly safe blocks against it on each L1 head
update. A by-number target whose block is missing still fails with
`BlockNotFound`.

BREAKING CHANGE: `FinalizeTaskError` has a new variant,
`NotYetAvailable { target }`; exhaustive matches must handle it.
`BlockNotSafe` keeps its name but has `Skip` severity instead of
`Critical`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
einar-oplabs added a commit that referenced this pull request Nov 7, 2026
A finalization target the engine cannot apply yet no longer stops
kona-node or holds the engine queue. A target above the safe head
fails with `BlockNotSafe`, now of `Skip` severity, whatever its source,
as op-node's promoteFinalized does: a delegate can run ahead of the
local safe head, and a reset can lower the safe head below a target
that was safe when requested (#23757, #23440). A hash-pinned target
the execution layer has no block for yet fails with the new skipped
`NotYetAvailable`, which carries the answer, whether the lookup
returned null or an error response, so a retried lookup cannot hold
the queue ahead of the imports that let the execution layer catch up.
The delegate poll resubmits its target every 4s.

A by-number target from L1 finality that a reset skipped is no longer
lost until L1 finality moves again: like op-node's retry when
derivation goes idle, the finalizer keeps the highest finalized L1
number and finalizes newly safe blocks against it on each L1 head
update, which batches the blocks confirmed since the last update. A
by-number target whose block is missing still fails with
`BlockNotFound`.

Tests cover the finalizer, a block derived again after a reset, a
delegated poll during execution-layer sync at the actor level, and
the error variant of each unavailable-block answer;
`test_utils::expect_sync_start` replaces two copies of the reset's RPC
script.

BREAKING CHANGE: `FinalizeTaskError` has a new variant,
`NotYetAvailable { engine_answer, target }`; exhaustive matches must
handle it. `BlockNotSafe` keeps its name but has `Skip` severity
instead of `Critical`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 530218f to 458fd57 Compare November 9, 2026 21:32
@einar-oplabs
einar-oplabs force-pushed the einar/23399-fcu-stale-safe-finalized branch 2 times, most recently from c7e6250 to a3c78e4 Compare November 10, 2026 13:38
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 458fd57 to 2fc40d6 Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into a3c78e4. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


reth 2.6 (#23046) carries reth#27247 and reth#27429: a forkchoice update whose safe or finalized hash is not on the head's chain is refused with -38002 before anything is applied, and a persisted header is canonical only when it matches the canonical hash at its height. On develop the green acceptance suites only touch these paths incidentally. This adds the four scenarios of #23399 as acceptance tests, written for the bump's documented behavior, with the cells develop's reth 2.5.2 answers differently skipped naming #23046 and the observed answer quoted.

What the tests cover

All in op-acceptance-tests/tests/op_reth_stale_forkchoice, on the mixed runtime at a 1s block time with an op-reth sequencer and an op-reth follower, both without proofs history and both with --engine.persistence-threshold and --engine.memory-block-buffer-target=5 explicit. The follower's consensus client runs the cell's sync mode (sysgo.L2CLVerifierSyncMode, from #23421); kona-node has no CL sync mode, so its cl_sync cells skip with that reason, as in #23421.

  1. TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt (cl_sync, el_sync): with the test sequencer building L1 one block at a time (dsl.TestSequencer.SequenceBlocksUntil, new, the mechanics of sync/elsync/reorg as a DSL wait), waits until the follower's safe head derives from an L1 block past the takeover, reorgs that L1 block out from under it while the follower's consensus client keeps running, and requires the follower to replace the safe block that derived from it (ReorgTriggered), to have a safe head deriving from the replacing L1 block (WaitL1OriginHash), to be safe five blocks past the pre-reorg safe height again, and to be in sync with the sequencer at the safe and unsafe heads.
  2. TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads (cl_sync, el_sync): waits for the follower to finalize a block above genesis, stops its op-reth, wipes its datadir (dsl.L2ELNode.WipeDatadir, new) and starts it again while its consensus client keeps running, so the first forkchoice updates the empty node receives name a head, a safe block and a finalized block it has not downloaded, re-peers it with the sequencer and requires it to hold the sequencer's unsafe head and to be safe and finalized at least where it was before the wipe, with its consensus client in sync with the sequencer's at the safe head. Under kona-node the cell skips before the finalized requirement; see CI follow-up.
  3. TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected (head_already_canonical, head_reorgs_to_sibling): stops the batcher, lets the follower get one block past the sequencer's head at that moment (so the safe head stays below the block it replaces, which both tests assert), stops the follower's consensus client, builds a sibling of the follower's head through the engine API (dsl.L2ELNode.BuildSiblingBlock, new), inserts it, and sends a forkchoice update that names the replaced head as finalized: in the first cell after the sibling has been made the head, in the second as the update that would make it the head. Requires the engine API's -38002 error with its Invalid forkchoice state message (ForkchoiceUpdateResult.IsInvalidForkchoiceState, new) and the head, safe and finalized labels unchanged.
  4. TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid (threshold_7, threshold_50): stops the batcher, lets the follower get threshold + 5 + 3 blocks past the sequencer's head at that moment (so nothing the batcher had loaded reaches the block it replaces), stops its consensus client, builds a sibling of the block threshold + 5 + 2 below the head (persisted and dropped from reth's memory on either schedule), reorgs to it, so every replaced block is on disk until reth's disk reorg removes it, and sends a forkchoice update back to the replaced block. The replaced block is not canonical, so the engine must not take it for the head it already has: the answer must be VALID with the head already moved back, or SYNCING, which makes the replaced block a sync target the follower downloads from the sequencer; either way the follower must end on the replaced block (ReorgExact).

Matrix: sync mode {CL, EL} for 1 and 2, threshold {7, 50} for 4, CL {op-node, kona-node} through DEVSTACK_L2CL_KIND (CI runs both). Tests 3 and 4 stop the consensus client before the raw updates, so the sync mode does not reach them. Every test skips on DEVSTACK_L2EL_KIND=op-geth.

Local verification

Debug op-reth built from this branch's Rust tree (identical to #23438's), kona-node release binary from develop's rust/kona, -parallel 2, under caffeinate (the machine slept through one earlier run):

cd <worktree> && DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> RUST_BINARY_PATH_KONA_NODE=<release kona-node> \
  go test -count=1 -timeout 40m -parallel 2 -run 'TestForkchoiceUpdate' -v ./op-acceptance-tests/tests/op_reth_stale_forkchoice/
# then the same with -run 'TestFollower'

op-node (default):

--- PASS: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_already_canonical (18.02s)
--- SKIP: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_reorgs_to_sibling (0.00s)
--- SKIP: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_7 (0.00s)
--- SKIP: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_50 (0.00s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   19.150s
--- PASS: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/cl_sync (55.58s)
--- PASS: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/el_sync (55.57s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/cl_sync (149.03s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync (141.75s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   205.657s

The three skipped cells, run on develop with their skips neutralized (same command), fail as the skip messages quote: head_reorgs_to_sibling at the labels-unchanged assertion (17.70s), threshold_7 and threshold_50 at a VALID answer must mean the head is the requested block (31.63s, 74.01s).

kona-node (DEVSTACK_L2CL_KIND=kona-node, same commands):

--- PASS: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_already_canonical (18.13s)
--- SKIP: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_reorgs_to_sibling (0.00s)
--- SKIP: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_7 (0.00s)
--- SKIP: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_50 (0.00s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   19.189s
--- SKIP: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/cl_sync (0.00s)
--- SKIP: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/el_sync (0.00s)
--- SKIP: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/cl_sync (0.00s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync (437.27s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   438.317s

The kona-node datadir-wipe cell spends 52s reaching 60 blocks, 124s on the first finalized block, 17s on the resync and 223s on being finalized again (see CI follow-up).

Against #23046's op-reth (reth 2.6.0, the debug binary built from the bump head 93d9ec9 for #23436), with the three develop-only skips neutralized, op-node:

--- PASS: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_already_canonical (18.35s)
--- PASS: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_reorgs_to_sibling (18.27s)
--- PASS: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_7 (31.93s)
--- PASS: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_50 (73.60s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   93.103s
--- PASS: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/cl_sync (55.22s)
--- PASS: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/el_sync (55.06s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/cl_sync (148.61s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync (141.48s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   204.675s

Against the bump the follower's op-node also resets on reset: detected L1 reorg in both sync modes and recovers inside the same bounds; no -38002 was answered. An earlier run of this cell against the bump died in WaitL1OriginHash on a one-shot safe lookup that answered not found for a moment right after the L2 reorg; the wait now retries its lookups (see Findings).

Against the bump under kona-node (same, DEVSTACK_L2CL_KIND=kona-node):

--- PASS: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_already_canonical (18.06s)
--- PASS: TestForkchoiceUpdateNamingReorgedOutFinalizedIsRejected/head_reorgs_to_sibling (17.77s)
--- PASS: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_7 (32.11s)
--- PASS: TestForkchoiceUpdateToPersistedReorgedOutHeadIsAppliedWhenReportedValid/threshold_50 (74.69s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   75.757s
--- SKIP: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/cl_sync (0.00s)
--- SKIP: TestFollowerSafeHeadAdvancesAgainAfterL1ReorgRewindsIt/el_sync (0.00s)
--- SKIP: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/cl_sync (0.00s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync (495.84s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   496.917s

The bump answered the persisted-head update VALID with the head already on the replaced block in all four cells (status=VALID latest_valid_hash=af2652…597aea head=af2652…597aea:11 at threshold 7 under op-node). Under kona-node the re-finalization after the wipe took 282s against the bump and 223s on develop; the bound is 7 minutes.

Unit tests (go test -run 'Sibling' ./op-devstack/dsl/, go test -run 'OpReth' ./op-devstack/sysgo/): ok. go vet, the custom golangci lint on op-devstack/... and the new package, and semgrep on the changed files: clean.

Skipped cells

Findings

  • develop (reth 2.5.2) applies a reorg before it checks the finalized hash: in head_reorgs_to_sibling the update naming the replaced head as finalized is refused with -38002 Invalid forkchoice state, but the head has moved to the sibling (a refused forkchoice update must leave the head, safe and finalized blocks as they were: Not equal: expected head 8f37be…9f697c:13, actual 0db6cc…2c2a2a:13). This is what reth#27247 fixes; the cell is skipped naming rust!: update reth to v2.6.0 #23046.

  • develop takes a persisted, reorged-out block for canonical as long as it is still on disk: in both threshold_7 and threshold_50 the update back to the replaced block is answered VALID with the replaced block as latestValidHash while the follower's head stays on the sibling (status=VALID latest_valid_hash=8f9698…014777 head=42103d…200c51:1 at threshold 7, status=VALID latest_valid_hash=07774d…0a8473 head=aa0354…d2d950:2 at threshold 50), deterministic in three runs. This is reth#27429's case; the test is skipped naming rust!: update reth to v2.6.0 #23046.

  • Against rust!: update reth to v2.6.0 #23046's op-reth the same update is answered VALID with the head moved back to the replaced block in the same call (reth logs the sibling committed, then the replaced block committed again): the deep reorg loads the replaced blocks back into reth's tree state, so find_canonical_header no longer takes the disk copy for canonical and on_new_head reorgs back, as reth#27429 describes. The issue's "pinning that it is not reported as valid" assumed the answer would be SYNCING; the test pins what both honest answers share, that a VALID answer moves the head and a SYNCING answer is followed by the sync-back, which develop fails and the bump passes.

  • With the sibling already the head, develop refuses the stale finalized hash correctly (head_already_canonical passes): the finalized check happens before the safe check and nothing has been applied yet, so the rule itself is in place and only its ordering relative to the head change is wrong.

  • The L1 reorg under a running follower produces no forkchoice rejection on develop in either sync mode: op-node logs L1 head signal indicates a possible L1 re-org, resets with reset: detected L1 reorg, and its subsequent updates move the safe hash back below the unsafe chain the engine still holds, which reth accepts; the follower replaces its safe block and is in sync with the sequencer again within the bounds. reth logs Changeset cache MISS in range, falling back to aggregate DB-based computation (WARN) while it recomputes state for the reorg; expected.

  • The empty datadir under a running consensus client causes no forkchoice rejection on develop either: reth answers the first update, whose head, safe and finalized hashes it has not downloaded, with SYNCING (it downloads the safe block first, then the head), op-node logs temp: cannot prepare unsafe chain for new payload: ... err: updated forkchoice, but node is syncing and then resets with expected engine was synced and had unsafe block to reconcile, but cannot find the block: not found, re-reads its heads from the empty engine and syncs the gap (through its peers in CL sync, through the engine in EL sync); the follower was back on the sequencer's head about six seconds after the wipe in both modes, with safe and finalized back at their pre-wipe heights. In el_sync the finalized label only returns once L1 finality catches up again, since op-node's post-EL-sync shortcut (finalized = the EL-sync tip) does not apply to a reset.

  • Under kona-node on develop, the L1 reorg is not survived: the kona-node sequencer resets its derivation (Derivation pipeline is being reset: L1 reorg detected) but keeps building on the unsafe chain whose L1 origin was reorged out (its op-reth commits blocks 42, 43, … on the same chain after the reset), so nothing replaces that chain, every node rejects the old batches (batch is for different L1 chain, epoch hash does not match l1_block_number=4) and buffers the rest (received out-of-order batch for future processing), and the follower's safe block is never replaced (operation failed permanently after 90 attempts: expected head to reorg 0x3f26…d122:20, but got 0x3f26…d122:20). The L1-reorg test skips on kona-node naming kona: Check L2 reorg due to L1 reorg #18676 (kona: check L2 reorg due to L1 reorg), the same reason the sync/elsync/reorg tests skip there (op-acceptance-tests: run TestUnsafeGapFillAfterSafeReorg against kona-node #22275). A kona-node follower under an op-node sequencer is not covered by this matrix.

  • dsl.L2ELNode.WaitL1OriginHash looked blocks up with the one-shot, test-failing lookups inside its retry loop; against the bump the follower's safe label answered not found for a moment right after the L2 reorg and the wait failed the test instead of retrying. The wait now uses the non-fatal lookups like the other pollers (blockRefByNumber is new, next to blockRefByLabel).

  • Under kona-node the follower is not reliably finalized again after the wipe, and in CI its kona-node exited on Finalize(BlockNotSafe); see CI follow-up.

Seams added

  • stack.DatadirWipeable with sysgo.OpReth.WipeDatadir (op-devstack/sysgo/l2_el_opreth.go), forwarded by the preset EL frontend like SwitchBinary, and dsl.L2ELNode.WipeDatadir: empties the datadir and, when enabled, the proofs store, and runs op-reth init (and proofs init) again, so the next Start runs the same binary and arguments on an empty chain. The node must be stopped; its p2p key, logs and genesis live outside the datadir, so its identity survives. The two init invocations the mixed runtime made inline are now initOpRethDatadir and initOpRethProofsStore, run under the test context and reporting their output on failure.
  • dsl.L2ELNode.GetPayload(info): engine_getPayload for the job a forkchoice update with attributes started.
  • dsl.L2ELNode.BuildSiblingBlock(number): has the engine build a block that can replace the canonical block at number on the same parent from the attributes that block was built from (same transactions, timestamp, fee recipient, gas limit, EIP-1559 parameters and, post-Jovian, minimum base fee, decoded from the block's extraData), differing in its prevRandao. The build is started by a forkchoice update naming the parent as head with zero safe and finalized hashes, since reth 2.6 refuses a nonzero safe or finalized hash above the head it is given; the method requires the head not to have moved. The payload is returned uninserted. siblingPayloadAttributes has unit tests for pre-Holocene, Holocene and Jovian extraData.
  • dsl.ForkchoiceUpdateResult.IsInvalidForkchoiceState(): requires the engine API's -38002 error with its Invalid forkchoice state message and no payload status.
  • dsl.TestSequencer.SequenceBlocksUntil(t, chainID, attempts, reached): builds one block every 2s until a CheckFunc passes, retrying failed builds and checks; the L1-advance loop of sync/elsync/reorg as a bounded DSL wait.

Notes on the issue text

  • Case 2 asks for an EL sync from scratch "where the first forkchoice update carries safe and finalized hashes reth has not downloaded". op-node never sends such an update to a fresh node on its own: its EL-sync start takes safe and finalized from the empty engine (genesis), and --syncmode.offset-el-safe only applies at completion, from blocks already downloaded; kona-node likewise starts from the engine's heads. A consensus client only names undownloaded safe and finalized blocks when its heads are ahead of its engine, so the test produces that state by wiping the datadir under a running consensus client. Under cl_sync op-node closes the gap through its peers rather than through EL sync, which is the recovery the matrix cell can have.
  • Case 4 asks to pin "that it is not reported as valid". On the bump the answer is SYNCING, and because the replaced block is then a sync target the follower downloads it from the sequencer and makes it canonical again; the test pins both, which also separates the bump from develop's VALID no-op deterministically.
  • The sync-mode dimension does not reach cases 3 and 4: both stop the follower's consensus client before the raw updates, so the cells are head_already_canonical/head_reorgs_to_sibling and threshold_7/threshold_50.
  • On develop the L1-reorg and datadir-wipe scenarios produce no forkchoice rejection at all: op-node's reset paths keep the safe and finalized hashes on the engine's chain (see Findings). The tests still pin the recovery, which the bump's stricter checks must not break.

CI follow-up

memory-all-kona-op-reth-fusaka (CircleCI job 5705185) failed TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync under kona-node at the finalized wait (datadir_wipe_test.go:58): operation failed permanently after 210 attempts: expected head for label=finalized to advance to target=59, but got current=0, 621 s into the cell. The pre-wipe labels (head 183, safe 179, finalized 59), the resync of the unsafe head and the safe head were as expected.

What the follower's kona-node does after the wipe (the CI per-test log carries only its WARN lines; a local run with its INFO lines fills in the rest):

  • The first consolidation after the wipe finds no block in the empty engine (Received `None` block for 186, Consolidate(MissingUnsafeL2Block(186))) and resets. The reset reads the wiped engine's labels (Loaded current L2 EL forkchoice state unsafe=0 safe=0 finalized=0) and the sync-start safe search walks back to the finalized block, genesis, so derivation restarts at L1 genesis and the node rebuilds the chain through derivation (Built and imported new safe block 1, 2, …: about 185 blocks in 13 s) rather than through EL sync.
  • Whenever the derived safe head catches up with the engine's canonical tip while the gossip-fed unsafe head is ahead of it, the next consolidation finds None again and resets again; every reset walks back to genesis (Found suitable L2 safe block l2_safe=0 is_finalized=true is_genesis=true) and clears the finalization queue (L2Finalizer::clear on Signal::Reset). Locally the loop ran for the rest of the cell (93 and 94 MissingUnsafeL2Block resets in four minutes in the two runs quoted above, every 4 to 8 s in CI), so the finalized label stays at genesis until a finalize task lands between two resets, which locally took 223 s and 282 s.
  • In CI the finalize task that landed found the safe head reset below the block it was to finalize (FinalizeTaskError::BlockNotSafe), which kona-node treats as Critical: at 07:04:01 the follower's kona-node exited with Failed to start rollup node service: EngineTask(Finalize(BlockNotSafe)), and nothing advanced for the remaining four minutes of the bound.

Both look like kona-node bugs worth an issue: a reset after an EL datadir wipe restarts derivation at genesis and then loops on MissingUnsafeL2Block instead of letting the EL sync, and a finalize request for a block above the reset safe head is fatal instead of being dropped.

The change: #23399's criterion for this scenario is that the sync completes. The cell keeps requiring, under both consensus clients and bounded, that the follower holds the sequencer's unsafe head and is safe at least where it was before the wipe. Being finalized at least where it was before the wipe, which is the consensus client's finalization after its reset rather than the EL's sync, and the consensus-client in-sync check that follows it now run only under op-node; under kona-node the cell skips at that point with a message naming #23399 and quoting the failure (the kona-node defect needs an issue of its own, which the skip should then name instead). The op-node cells run the same sequence as before.

The follower's op-reth is not blameless by inspection alone: in the CI log it answers null by number for block 348 at 07:04:00 while it had received payload 361, and its blocks become canonical in bursts of about six; op-node drives the same EL through the same scenario and passes, so the loop is kona-node's, but whoever takes the issue should know the EL answers null below the highest payload it has received.

Local runs of the committed test (debug op-reth built from this branch, kona-node release binary from develop's rust/kona, -parallel 2, under caffeinate):

DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> \
  go test -count=1 -timeout 40m -parallel 2 -run TestFollowerResyncsFromScratch -v ./op-acceptance-tests/tests/op_reth_stale_forkchoice/
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync (141.66s)
--- PASS: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/cl_sync (148.57s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   149.648s

# the same with DEVSTACK_L2CL_KIND=kona-node RUST_BINARY_PATH_KONA_NODE=<release kona-node>
--- SKIP: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/cl_sync (0.00s)
--- SKIP: TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync (200.31s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_stale_forkchoice   201.370s

Under op-node the follower was finalized again about 70 s after the wipe in el_sync and at the first poll in cl_sync, and its op-node was in sync with the sequencer's at the safe head right after. Under kona-node the el_sync cell held the sequencer's unsafe head and was safe where it was 7.5 s after the wipe, then skipped. An earlier pair of runs with the consensus-client check ahead of the finalized wait passed the same way (op-node 141.20 s and 151.73 s; kona-node in sync at the safe head 19 s after the wipe, then skipped).

Stacked on #23438, #23436, #23431, #23427, #23421 and #23418.

Closes #23399

@einar-oplabs
einar-oplabs requested a review from aakoshh November 10, 2026 14:07
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 2fc40d6 to f4c7010 Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23399-fcu-stale-safe-finalized branch 2 times, most recently from 6cabe52 to eb73ef7 Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from f4c7010 to be963b4 Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23399-fcu-stale-safe-finalized branch from eb73ef7 to 574c473 Compare November 11, 2026 13:59
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch 2 times, most recently from 5b3e133 to 63096fc Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23399-fcu-stale-safe-finalized branch 2 times, most recently from 4b951df to a6fb532 Compare November 11, 2026 15:28
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 63096fc to 5fe7487 Compare November 11, 2026 15:28
@einar-oplabs
einar-oplabs force-pushed the einar/23399-fcu-stale-safe-finalized branch from a6fb532 to c4f5482 Compare November 11, 2026 16:28
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 5fe7487 to 4c4f1fb Compare November 11, 2026 16:28
…e safe or finalized hash

Four acceptance tests for the forkchoice updates a consensus client
sends when its safe or finalized hash is not on the head's chain, which
reth 2.6 rejects before applying anything (reth#27247, reth#27429): an
L1 reorg that rewinds a running follower's safe head, a resync from an
empty datadir under a consensus client that kept its heads, a raw update
naming a reorged-out block as finalized, and a raw update back to a
block reth reports persisted after a sibling reorged it out, where the
reorg to the sibling must report the sibling as the latest valid block.
Under kona-node the L1-reorg test skips naming #18676, and the
datadir-wipe test requires the follower's unsafe and safe heads but
skips the finalized requirement naming #24066.

Devstack seams: an op-reth datadir wipe (stack.DatadirWipeable),
engine_getPayload and a sibling-block builder on the L2 EL DSL, an
assertion for the engine API's -38002 refusal, and non-fatal lookups
inside WaitL1OriginHash's retry loop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23399-fcu-stale-safe-finalized branch from c4f5482 to 9ac60ed Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 4c4f1fb to 19b45d1 Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 13:18
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 13:18
@joshklop joshklop assigned joshklop and unassigned einar-oplabs and joshklop Nov 21, 2026
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants