Skip to content

op-devstack,op-acceptance-tests: cover the production proofs init path with backfill - #23429

Closed
einar-oplabs wants to merge 3 commits into
einar/23383-proofs-reorg-reexecfrom
einar/23389-proofs-init-path
Closed

einar-oplabs wants to merge 3 commits into
einar/23383-proofs-reorg-reexecfrom
einar/23389-proofs-init-path

Conversation

@einar-oplabs

Copy link
Copy Markdown
Contributor

Why

Every production proofs-history store is created by op-reth proofs init on a node that synced without proofs history, with a backward backfill through the snapshot path or through reth's changesets. Devstack only ever ran proofs init on a genesis datadir, so the backfill, the snapshot path and the init guards #23046 adds (ensure_source_state_trie_durable) had no end-to-end coverage.

What

op-acceptance-tests/tests/proofs_history_init (sequencer without proofs history as the eth_getProof reference, follower started without proofs history, 1s L2 blocks, both op-reths with --engine.persistence-threshold and --engine.memory-block-buffer-target=5 passed explicitly, window W = 30):

  • TestFollowerServesTheWholeWindowAfterProofsInitWithBackfill: the operator path. Alice and bob transfer back and forth and bob writes to a deployed contract until the chain is past W + 5 and the follower has imported it; the follower's CL and op-reth are stopped cleanly; op-reth proofs init --proofs-history.window=30 runs against the datadir, with the default snapshot backfill or --proofs-history.use-snapshot=false; op-reth restarts with --proofs-history. Before the CL is back, so nothing extends or prunes the store, the store must report exactly [tip - W, tip] for reth's datadir tip and VerifyHistoricalProofsMatch sweeps every block of it for the four accounts against the sequencer. The CL then comes back, traffic runs the chain W + 5 past the anchor, WaitForProofsStorePrunedPast(anchor) requires pruning to have moved the earliest block past the init's anchor, and a second sweep covers the window the store serves then, which is entirely live-indexed. Matrix: threshold {7, 50} x proofs path {index, execute} x backfill {snapshot, plain}, eight cells.
  • TestProofsInitSeedsAConsistentStoreFromAKilledNodeDatadir: the follower's op-reth is killed (StopUngracefully) once the chain is past a window, so its datadir trails the head by the blocks reth held in memory (the test requires that gap); proofs init must succeed, the restarted node must serve exactly [persisted tip - W, persisted tip] and that window must verify before the CL is back; after the CL re-feeds the lost blocks and ten more blocks of traffic, a second sweep covers the live window.
  • TestProofsInitAgainstARunningNodeLeavesAStoreThatMatchesTheChain: proofs init runs against the datadir while the follower is running and persisting. The test accepts two outcomes: exit 0, or a refusal whose message carries reth database tip moved (rust!: update reth to v2.6.0 #23046's InitializeSourceTipMoved); any other failure fails the test. After a clean stop (and a second init if the first was refused) the restarted node's store must catch up with the datadir tip and every proof of the window it serves must verify and match the sequencer, before and after the CL is back.
  • Guard matrix: threshold {7, 50} x masking {0, 30}; the masking cells skip naming ethereum-optimism/protocol-team#457 and say what rust!: update reth to v2.6.0 #23046 must assert there (see below). The guard tests drive the chain to threshold + W + 5 before the kill or the init, since reth persists nothing until more than the threshold of blocks sit in memory and its datadir then trails the head by at most the threshold.

op-devstack seam (general; #23390 reuses the window field and the init action):

  • stack.ProofsHistoryInit{PlainBackfill, Window} (zero value = the operator's plain invocation, snapshot backfill and the node's configured window), stack.ProofsHistoryInitReport{Duration, Output} and stack.ProofsHistoryInitializer (op-devstack/stack/proofs_history.go).
  • sysgo.OpReth.InitProofsHistory(init): runs op-reth proofs init --chain --datadir --proofs-history.storage-path --proofs-history.window=W [--proofs-history.use-snapshot=false] against the node's datadir whether or not the node is running (the running case is what the guard test is about), returns the duration and output, and on success makes the next Start pass the proofs-history flags with that window. Window zero keeps the node's configured window (OpRethWithProofsHistoryWindow or the runtime's 10000). OpReth now keeps its node arguments without the proofs-history flags and appends them in Start while proofs history is enabled; buildMixedOpRethNode seeds the genesis store through the same method, so the default path is unchanged except that it now passes the node's window explicitly (a no-op at genesis). OpRethWithoutProofsHistory is the way to build a node whose store a test seeds later.
  • Preset forwarding in l2ELFrontend; DSL L2ELNode.InitProofsHistory(opts...) (requires a stopped node, fails the test on error, returns the report), InitProofsHistoryRaw(opts...) (runs whether or not the node is running and returns the outcome) and WaitForProofsStorePrunedPast(block, timeout).
  • Unit tests: TestProofsHistoryNodeArgs, TestProofsHistoryInitArgs, TestOpRethNodeArgsCarryProofsHistoryOnlyOnceEnabled, TestInitProofsHistoryEnablesProofsHistoryOnSuccessOnly (fake op-reth script).

Findings

  • On develop proofs init against a running node's datadir exits 0 in every run (4 of 4 cells, refused=false): the command opens reth's MDBX read-only, anchors the store at reth's Finish checkpoint and copies the hashed-state and trie tables in a second read transaction. The copy matched the anchor every time, and after the restart the store caught up from the anchor to the datadir tip ([15, 45] at threshold 7, [55, 85] at 50) and every proof verified. Whether that copy can describe another block than the anchor depends on a persistence commit landing between the two read transactions, a few milliseconds apart, which is the same window rust!: update reth to v2.6.0 #23046's ensure_source_state_trie_durable closes with InitializeSourceTipMoved. So the issue's "on the bump the running-node case must fail with the tip-moved error" is a race on the bump too (its unit tests cover it with a hand-moved checkpoint); this test accepts the refusal when it happens, retries on the stopped node, and asserts the end-to-end property, which is what an operator needs. The retry is sound on the bump because ensure_source_state_trie_durable runs before set_initial_state_anchor, so a refused store holds no anchor.
  • A clean stop (SIGINT) makes reth persist everything it held in memory: in all eight operator-path cells the datadir tip after the restart equaled the head at the stop, including the threshold-50 cells, where reth had persisted nothing before the stop. The guard tests, which kill the node, therefore drive the chain to threshold + W + 5 before the kill, so reth has persisted at least a window; a first version drove to W + 5 and at threshold 50 the kill left the datadir at genesis (store window [0, 0], the init anchored at block 0).
  • A kill loses more at threshold 50 than the restart test's model suggests: at head 85 the datadir tip was 46 (39 blocks), at head 45 and threshold 7 it was 39 (6 blocks). The init seeded [16, 46] and [9, 39] and the follower re-imported the rest from its CL and served all of it.
  • Init plus a 30-block backfill takes about half a second on a devnet chain (0.49-0.72 s across all cells), snapshot and plain alike, so the per-cell durations say nothing about the two backfill paths beyond both producing the same proofs; a chain with real state is needed to compare them.
  • rust/op-reth/tests/proofs/prune/ (dormant: CI runs proofs/core only, and its hard-coded window of 200 never prunes against the runtime's 10000) is superseded by the prune step of the operator-path test. I was not permitted to delete it from this environment; it should be removed in this PR or right after.

Local verification

Debug op-reth built from this branch, op-node: DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> go test -count=1 -timeout 40m -parallel 2 -v ./op-acceptance-tests/tests/proofs_history_init/:

--- PASS: TestFollowerServesTheWholeWindowAfterProofsInitWithBackfill
    threshold_7-index-backfill_snapshot    (82.89s)  init 0.54s  anchor 35  seeded [5, 35]   after prune [40, 70]
    threshold_7-index-backfill_plain       (83.86s)  init 0.52s  anchor 37  seeded [7, 37]   after prune [42, 72]
    threshold_7-execute-backfill_snapshot  (84.01s)  init 0.53s  anchor 37  seeded [7, 37]   after prune [42, 72]
    threshold_7-execute-backfill_plain     (84.14s)  init 0.51s  anchor 37  seeded [7, 37]   after prune [42, 72]
    threshold_50-index-backfill_snapshot   (77.27s)  init 0.51s  anchor 35  seeded [5, 35]   after prune [40, 70]
    threshold_50-index-backfill_plain      (82.89s)  init 0.72s  anchor 36  seeded [6, 36]   after prune [41, 71]
    threshold_50-execute-backfill_snapshot (81.81s)  init 0.53s  anchor 35  seeded [5, 35]   after prune [40, 70]
    threshold_50-execute-backfill_plain    (83.70s)  init 0.51s  anchor 37  seeded [7, 37]   after prune [42, 72]
--- PASS: TestProofsInitSeedsAConsistentStoreFromAKilledNodeDatadir
    threshold_7-masking_0    init 0.52s  head at kill 45  datadir tip 39  seeded [9, 39]
    threshold_50-masking_0   init 0.53s  head at kill 85  datadir tip 46  seeded [16, 46]
--- PASS: TestProofsInitAgainstARunningNodeLeavesAStoreThatMatchesTheChain
    threshold_7-masking_0    init 0.52s exit 0 at head 44   datadir tip after the stop 45  store caught up to [15, 45]
    threshold_50-masking_0   init 0.52s exit 0 at head 85   datadir tip after the stop 85  store caught up to [55, 85]
    4 masking_30 cells skipped (protocol-team#457)
ok      op-acceptance-tests/tests/proofs_history_init   474.692s (operator path, first full run; the guard cells above are from the rerun after the drive-target fix, 236.546s)

Every sweep compared every block of its range for the four accounts (pruned_during_sweep=0). The eight operator-path cells sum to about 11 minutes back to back and the four guard cells to about 5, so the package stays under 20 minutes with -parallel 1.

kona-node (DEVSTACK_L2CL_KIND=kona-node RUST_BINARY_PATH_KONA_NODE=<release kona-node built from develop today>, same command): all twelve cells pass, ok ... 578.047s at -parallel 2; the follower EL-syncs there, and the threshold-50 guard cells take about 125 s because the chain runs to block 85 before the kill or the init (killed node: head 87, datadir tip 46, seeded [16, 46]; running node: init exit 0 at head 87, store caught up to [57, 87]).

After the review changes (option inversion, exec.CommandContext, renames) one cell per test was run again under op-node on the final commit: threshold_7-index-backfill_plain (84.89s), TestProofsInitSeedsAConsistentStoreFromAKilledNodeDatadir/threshold_7-masking_0 (78.00s), TestProofsInitAgainstARunningNodeLeavesAStoreThatMatchesTheChain/threshold_50-masking_0 (123.70s), ok ... 163.987s.

go vet ./op-devstack/... ./op-acceptance-tests/tests/proofs_history_init/      # ok
./linter/bin/op-golangci-lint run ./op-devstack/... ./op-acceptance-tests/tests/proofs_history_init/...   # 0 issues
go test -count=1 ./op-devstack/sysgo/ -run 'TestProofsHistory|TestOpRethNodeArgs|TestInitProofsHistory|TestOpRethWith'   # ok
go test -count=1 ./op-devstack/dsl/ ./op-devstack/presets/ ./op-devstack/stack/   # ok

Skipped cells

  • All masking_30 guard cells: ethereum-optimism/protocol-team#457. On rust!: update reth to v2.6.0 #23046 with masking on, the init against a killed node must fail with InitializeSourceStateTrieLagging until a clean restart repairs the datadir; the skip message says so.
  • The bump-only guard assertions ("the running-node case must fail with the tip-moved error") are not asserted on develop, which has no guard; see Findings for why they cannot be asserted deterministically on the bump either.

Stacked on #23428, #23421 and #23418.

Closes #23389

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 14, 2026
@wiz-0f98cca50a

wiz-0f98cca50a Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 1 Medium
Software Management Finding Software Management Findings -
Total 1 Medium

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

logger := n.p.Logger().New("name", n.name, "chain", n.chainID, "running", n.sub != nil)
logger.Info("Running op-reth proofs init", "cmd", n.execPath, "args", args)
started := time.Now()
output, err := exec.CommandContext(n.p.Ctx(), n.execPath, args...).CombinedOutput()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium SAST Finding

OS Command Injection (CWE-78)

More Details

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system.

The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application. This can occur when user-supplied data is passed directly to functions that execute OS commands, such as exec.Command, exec.CommandContext, syscall.ForkExec, or syscall.StartProcess.

If an attacker can inject malicious code into these commands, they can potentially execute any command on the system, including installing malware, stealing data, or causing a denial of service. This vulnerability can have severe consequences, including data breaches, system compromise, and unauthorized access to sensitive information.

To avoid this vulnerability, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code.

Attribute Value
Impact Medium
Likelihood Medium

Remediation

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system. The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application.

To fix this issue, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code. Additionally, it is recommended to use language-specific facilities for executing commands, which provide better input validation and sanitization.

Code examples

// VULNERABLE CODE - User input is directly used in constructing the command
userInput := getInput() // Assume this comes from user input
cmd := exec.Command("/bin/sh", "-c", userInput)
err := cmd.Run()
// SECURE CODE - Command and arguments are hardcoded, and user input is validated
userInput := getInput() // Assume this comes from user input
if !isValidInput(userInput) {
    // Handle invalid input
    return
}
cmd := exec.Command("/bin/sh", "-c", "echo", userInput)
err := cmd.Run()

Additional recommendations

  • Follow the principle of least privilege and run the application with the minimum required permissions.
  • Use input validation and sanitization techniques to ensure that user input does not contain malicious code.
  • Implement secure coding practices and follow security standards such as OWASP Top 10 and CWE.
  • Consider using language-specific facilities for executing commands, such as os/exec in Go, which provide better input validation and sanitization.
  • Regularly update and patch the application and its dependencies to address known vulnerabilities.
  • Implement logging and monitoring mechanisms to detect and respond to potential security incidents.

Rule ID: WS-I011-GO-00026


To ignore this finding as an exception, reply to this conversation with #wiz_ignore reason

If you'd like to ignore this finding in all future scans, add an exception in the .wiz file (learn more) or create an Ignore Rule (learn more).


To get more details on how to remediate this issue using AI, reply to this conversation with #wiz remediate

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 29, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
einar-oplabs and others added 3 commits November 9, 2026 20:39
…fill

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 8e22ead)
(cherry picked from commit dde6412af5e5d2ec17988d5cf4d7291421ccd1fb)
…rst persistence

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit c9576b8)
(cherry picked from commit ce3a7c42a019dff799492ffbc051fd66c8edcb39)
…am and tests

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit a5c44cf)
(cherry picked from commit 21eb4663e3b719779d601419af2fdec85a8c0e8a)
@einar-oplabs
einar-oplabs force-pushed the einar/23389-proofs-init-path branch from a5c44cf to f4bf9f5 Compare November 9, 2026 22:29
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: acceptance test for the production proofs init path with backfill

3 participants