Skip to content

op-reth,op-devstack: behavioral tests for the proofs CLI subcommands - #23437

Closed
einar-oplabs wants to merge 7 commits into
einar/23390-proofs-rpc-edgesfrom
einar/23393-proofs-cli-tests
Closed

einar-oplabs wants to merge 7 commits into
einar/23390-proofs-rpc-edgesfrom
einar/23393-proofs-cli-tests

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

No op-reth proofs subcommand had a behavioral test, only the CLI snapshot. proofs unwind --target T documented removing every block after T but passed T to the store's unwind_history, which removes T as well, so the documented recovery for ethereum-optimism/protocol-team#464 (proofs unwind --target <reth tip>) left the store one block short of reth.

What

op-reth proofs unwind now matches its documentation. The store's unwind_history(to) removes to.block.number..=latest and keeps (to.block.number - 1, to.parent) as the tip (the trait doc now states that contract); the command passes target + 1 with the target's hash as the parent, so the target stays and becomes the store's latest block. A target equal to the earliest block is allowed and leaves [earliest, earliest]. A target below the earliest block is refused with a non-zero exit (it was a silent exit 0 with "Nothing to unwind" while the store kept history above the target); the message points at a backfill past the target, or at a re-init when reth no longer holds those blocks. A target at or above the latest block stays a no-op, and so does an empty store. The refusals are a typed UnwindError; the pruned-body refusal is unchanged.

op-reth proofs prune rejects --proofs-history.prune-batch-size=0 at parse time and exits non-zero when the pruner fails: OpProofStoragePruner::run logs and swallows errors, so the command reported success for a prune it did not do; the command now calls the new try_run. An empty store is nothing to prune, as it is nothing to unwind. In the trie crate, with_batch_size(0) is raised to one (the pruner could not advance with it), and PrunerError renders through thiserror, so BlockNotFound names the block instead of printing the variant name.

Behavior changes an operator can see

  • proofs unwind --target T leaves the store at T; it left it at T - 1.
  • proofs unwind with a target below the store's earliest block exits non-zero; it exited 0.
  • proofs prune exits non-zero when the pruner fails and rejects a batch size of 0; it exited 0 in both cases. On an empty store it exits 0 instead of failing with NoBlocksFound.

The first is the fix the #464 recovery depends on; the others turn silent failures into exits a runbook can check. None of them changes a flag, so this is not marked op-reth!:; say so if you want the BREAKING CHANGE paragraph for the release notes.

Rust: rust/op-reth/crates/cli/tests/proofs_cli/ (40 tests)

Every test seeds a real datadir through the CLI's own EnvironmentArgs (genesis from a JSON file with every OP fork active, storage v2) with a short chain committed the way reth's persistence commits blocks (append_blocks_with_state plus write_trie_updates; every block changes an EOA balance and a contract storage slot, so every block has account and storage changesets), then drives Cli in-process the way the binary does and asserts on get_proof_window and the snapshot anchor read from the MDBX store. Refusals are asserted on the typed errors (UnwindError, PrunerError, SnapshotError, BackfillError, clap::Error). After every unwind and prune the store's state at the new tip or earliest block is checked against reth's header by building the snapshot there (SnapshotInitJob validates the state root), so a store that moved its pointers without its state would fail.

subcommand cases
init fresh with snapshot backfill, plain backfill and a window reaching below genesis; --proofs-history.skip-backfill leaves [tip, tip]; idempotent; a second init only extends the backfill (a wider window extends, a narrower one never prunes)
unwind (store [3, 8]) target {earliest, earliest + 1, latest - 1, latest, latest + 1} with the resulting window and the target's reth hash as the tip; below earliest refused and the window untouched; pruned body refused; store ahead of reth (a block reth does not hold appended to the store) unwound to reth's tip, the #464 recovery; a target reth does not hold refused; empty store is a no-op
prune (store [2, 8]) window 3 with the default batch size, batch size 1 and 2; a window covering or wider than the store leaves it; batch size 0 rejected; pruning up to a block reth does not hold fails with BlockNotFound(9) and leaves the window; an empty store is a no-op
backfill (store [8, 8]) snapshot and plain paths within the chain and reaching below genesis; batch size 1; a covered window is a no-op; a wider window extends an existing window further back; an empty store fails with NoBlocksFound
snapshot init|drop (store [4, 8] without a snapshot) init at the earliest block by default and at a target block; targets outside the window refused; a second init at the same anchor keeps it; another anchor refused while a snapshot is ready; drop removes it and dropping nothing succeeds; drop lets a snapshot backfill continue after a plain backfill moved the window off the snapshot's anchor

e2e: rust/op-reth/tests/proofs/core/unwind_test.go

TestProofsFollowerServesTheWindowAfterItsStoreIsUnwoundAndRestarted (threshold 7, index path, 20-block window): the follower is stopped cleanly, op-reth proofs unwind --target <head - 5> runs against its datadir, op-reth restarts with its CL still down and must re-index the datadir up to its tip (the ExEx's startup sync_to), every proof of the window is swept against the sequencer, then the CL comes back, traffic runs ten more blocks and the sweep repeats over the live window.

Seams

  • stack.ProofsHistoryUnwind{Target}, stack.ProofsHistoryUnwinder, and ProofsHistoryInitReport renamed to ProofsHistoryCommandReport since both subcommands report the same thing.
  • sysgo.OpReth.UnwindProofsHistory (shares runProofsHistorySubcommand with the init), forwarded by the preset frontend; DSL L2ELNode.UnwindProofsHistory(target) requires a stopped node and fails the test on error.
  • OpProofStoragePruner::try_run returns the pruner's result instead of logging it; the trie crate's strum dependency is now optional under metrics, its only user.

Follow-ups, not in this PR

  • OpProofsProviderRw::unwind_history takes a BlockWithParent of the first removed block but no implementation reads its hash; the CLI passes B256::ZERO there (reth does not hold that block when the store is ahead). Narrowing the parameter to the new tip's BlockNumHash would remove the sentinel. The in-memory implementation also lacks the MDBX guards (no-op above latest, error at or below earliest).
  • docs/public-docs/node-operators/tutorials/reth-historical-proofs.mdx now says the unwind target stays and a target below the earliest block fails; the generated CLI reference is rebuilt at release time.

Matrix

Storage version v2 only (the CLI's --storage.v2 default; v1 datadirs are legacy). Unwind targets as listed above. The bump-only init guards (lagging trie, moved tip) do not exist on develop and are not tested here.

Local verification

Rust, from rust/ (debug op-reth test profile, shared target dir):

cargo test -p reth-optimism-cli --test proofs_cli      # 40 passed (38 before the review changes, three consecutive runs, 37-44 s each)
cargo test -p reth-optimism-trie --lib prune::         # 7 passed (the pruner's own tests through try_run, plus the two new ones)
cargo clippy -p reth-optimism-cli -p reth-optimism-trie --all-targets   # no warnings
mise exec -- zepter run check                          # ok (alloy-eips/serde propagated)
just fmt-fix                                           # applied

Finding while stabilizing the target: with every test seeding its own datadir in parallel threads of one process, one or two tests per run failed in EnvironmentArgs::init or MdbxProofsStorage::new with Cannot allocate memory (12) (3 of 4 runs of the compiled binary). reth and the proofs storage each map an 8 TB MDBX geometry, and a dozen of them exhaust the process's address space. CI's nextest runs every test in its own process, but cargo test does not, so the fixture holds a process-wide lock for the lifetime of each seeded datadir; the target then passed three runs in a row.

Go:

go build ./op-devstack/... ./rust/op-reth/tests/proofs/...          # ok
go vet ./op-devstack/{sysgo,dsl,presets,stack}/ ./rust/op-reth/tests/proofs/core/   # ok
go test -count=1 ./op-devstack/sysgo/ -run 'TestProofsHistory|TestOpRethNodeArgs|TestInitProofsHistory|TestUnwindProofsHistory|TestOpRethWith'   # ok
op-golangci-lint run (the same packages)                             # 0 issues

e2e (rust/op-reth/tests/proofs/core, TestProofsFollowerServesTheWindowAfterItsStoreIsUnwoundAndRestarted, run with the recipe's env vars by hand against a debug op-reth built from this branch, since just test-e2e-sysgo hard-codes a release binary):

DISABLE_OP_E2E_LEGACY=true DEVSTACK_ORCHESTRATOR=sysgo OP_RETH_ENABLE_PROOF_HISTORY=true RUST_BINARY_PATH_OP_RETH=<debug op-reth> \
  go test -count=1 -timeout 40m -parallel 2 -v -run TestProofsFollowerServesTheWindowAfterItsStoreIsUnwoundAndRestarted ./proofs/core/
--- PASS: TestProofsFollowerServesTheWindowAfterItsStoreIsUnwoundAndRestarted (62.01s)
ok      github.com/ethereum-optimism/optimism/rust/op-reth/tests/proofs/core    63.018s

The test took head 24 and target 19 while the store already served [5, 25]; proofs unwind --target=19 took 0.52 s and removed 20..25; the restarted follower's datadir tip was 25, the store's first re-executed block was 20 and it was back at [5, 25] ten seconds after the start with the CL still down; the first sweep covered [5, 25] and the second, after ten more blocks of traffic, [35, 52], every block for the four accounts, pruned_during_sweep=0.

Not run locally: the op-geth validator variant (OP_DEVSTACK_PROOF_VALIDATOR_EL=op-geth skips the test) and a kona-node CL (the preset runs op-node).

Notes for the #464 and Notion recovery text

The recovery proofs unwind --target <reth tip> now does what the text says: the store ends at reth's tip. The text does not need to change; the previous behavior left the store at tip - 1. A target below the store's earliest block now fails instead of exiting 0, which the runbook should mention if it ever lists it.

Stacked on #23434, #23429, #23428, #23421 and #23418.

Closes #23393

🤖 Generated with Claude Code

CI follow-up: the prune-ahead-of-reth case is split by pruner behavior

prune_fails_when_the_new_earliest_block_is_not_in_reth is pin-specific: the pruner of the reth 2.6 bump with #23358 caps the target at reth's last block, so there the prune succeeds at block 8 instead of failing on block 9. No consistent datadir makes block_hash(target) None while last_block_number() >= target (both read the static-file headers segment on both pins), so one failing case cannot hold on both. The scenario now lives in seeded_storage_ahead_of_reth with a test per expectation: the failing one stays active on develop, and prune_stops_at_reths_last_block_when_the_storage_is_ahead_of_reth is added #[ignore]d naming #23358; when this PR lands on the bump, drop the #[ignore] from the capped test and delete the failing one, which pins a failure #23358 removes (no pin sniffing). Verified with cargo test -p reth-optimism-cli --test proofs_cli prune: on develop the active test passes and the ignored one fails under --ignored; on the bump evaluation tree the reverse (the ignored one passes under --ignored, the active one fails).

@einar-oplabs einar-oplabs self-assigned this Oct 15, 2026
@wiz-0f98cca50a

wiz-0f98cca50a Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 1 Medium
Software Management Finding Software Management Findings -
Total 1 Medium

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

logger := n.p.Logger().New("name", n.name, "chain", n.chainID, "running", n.sub != nil, "subcommand", subcommand)
logger.Info("Running op-reth proofs subcommand", "cmd", n.execPath, "args", args)
started := time.Now()
output, err := exec.CommandContext(n.p.Ctx(), n.execPath, args...).CombinedOutput()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium SAST Finding

OS Command Injection (CWE-78)

More Details

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system.

The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application. This can occur when user-supplied data is passed directly to functions that execute OS commands, such as exec.Command, exec.CommandContext, syscall.ForkExec, or syscall.StartProcess.

If an attacker can inject malicious code into these commands, they can potentially execute any command on the system, including installing malware, stealing data, or causing a denial of service. This vulnerability can have severe consequences, including data breaches, system compromise, and unauthorized access to sensitive information.

To avoid this vulnerability, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code.

Attribute Value
Impact Medium
Likelihood Medium

Remediation

OS command injection is a critical vulnerability that allows an attacker to execute arbitrary commands on the system. This can lead to a full system compromise, as the attacker can potentially gain complete control over the application and the underlying system. The vulnerability arises when user input is used to construct commands or command arguments that are then executed by the application.

To fix this issue, user input should never be used directly in constructing commands or command arguments. Instead, the application should use a hardcoded set of arguments and validate any user input to ensure it does not contain malicious code. Additionally, it is recommended to use language-specific facilities for executing commands, which provide better input validation and sanitization.

Code examples

// VULNERABLE CODE - User input is directly used in constructing the command
userInput := getInput() // Assume this comes from user input
cmd := exec.Command("/bin/sh", "-c", userInput)
err := cmd.Run()
// SECURE CODE - Command and arguments are hardcoded, and user input is validated
userInput := getInput() // Assume this comes from user input
if !isValidInput(userInput) {
    // Handle invalid input
    return
}
cmd := exec.Command("/bin/sh", "-c", "echo", userInput)
err := cmd.Run()

Additional recommendations

  • Follow the principle of least privilege and run the application with the minimum required permissions.
  • Use input validation and sanitization techniques to ensure that user input does not contain malicious code.
  • Implement secure coding practices and follow security standards such as OWASP Top 10 and CWE.
  • Consider using language-specific facilities for executing commands, such as os/exec in Go, which provide better input validation and sanitization.
  • Regularly update and patch the application and its dependencies to address known vulnerabilities.
  • Implement logging and monitoring mechanisms to detect and respond to potential security incidents.

Rule ID: WS-I011-GO-00026


To ignore this finding as an exception, reply to this conversation with #wiz_ignore reason

If you'd like to ignore this finding in all future scans, add an exception in the .wiz file (learn more) or create an Ignore Rule (learn more).


To get more details on how to remediate this issue using AI, reply to this conversation with #wiz remediate

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 31, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
einar-oplabs and others added 7 commits November 9, 2026 20:41
`op-reth proofs unwind --target T` documented removing every block after T
but passed T to the store's `unwind_history`, which removes T too, so the
documented recovery for ethereum-optimism/protocol-team#464 left the store
one block short of reth. The command now keeps the target as the store's
latest block, allows a target equal to the earliest block, and refuses a
target below it instead of exiting 0.

`op-reth proofs prune` rejects a batch size of 0 and exits non-zero when
the pruner fails, through the new `OpProofStoragePruner::try_run`;
`PrunerError::BlockNotFound` names the block.

New integration test target `rust/op-reth/crates/cli/tests/proofs_cli/`
seeds a real datadir through the CLI's own `EnvironmentArgs` and drives
`Cli` in-process for init, unwind (every target edge, a pruned body, a
store ahead of reth), prune, backfill and snapshot init|drop.

Devstack: `stack.ProofsHistoryUnwind`, `sysgo.OpReth.UnwindProofsHistory`
and DSL `L2ELNode.UnwindProofsHistory` for a stopped node, with an e2e
smoke test in `rust/op-reth/tests/proofs/core` that unwinds a stopped
follower, restarts it, and requires the store to resume right above the
target before sweeping the window against the sequencer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 697689c)
(cherry picked from commit 7baa99d5b53a1ba8f2b91044d29fb2755be7a70b)
`proofs unwind` refusals are a typed `UnwindError`, with the doc and
message for a target below the earliest block pointing at a backfill
rather than at a re-init; `proofs prune` treats an empty storage as
nothing to prune, like unwind; `OpProofStoragePruner::with_batch_size`
raises a zero batch to one and `PrunerError` renders through thiserror
with the block it could not find; `strum` is only needed by the trie
crate's metrics. `alloy-eips/serde` is propagated for zepter.

The CLI tests assert on the typed errors, verify the storage's state
root at the new tip after an unwind and at the new earliest block after
a prune through `snapshot init`, and pin prune and backfill on an empty
storage. The pruner tests cover a block reth does not hold and the batch
size clamp. The public historical-proofs tutorial now says the unwind
target stays.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 6f87a18)
(cherry picked from commit 1fbdf0e48824f018c9ca90f58ad5a74e4d5af282)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 35c09c3)
(cherry picked from commit 40ac8fe7664e72aea0b376ca9338a82855120be3)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 2d9becf)
(cherry picked from commit 25eee69e7412a21d1951f02c618d4bc2b6612bc0)
…runer behavior

`prune_fails_when_the_new_earliest_block_is_not_in_reth` expects the pruner
to target `latest - window` and fail on a block reth does not hold. The reth
2.6 pin's pruner (#23358) caps the target at reth's last block, prunes up to
it and succeeds. No consistent datadir satisfies both: `last_block_number`
and `block_hash` read the same static-file headers segment, so a block at or
below the last block number always has a hash.

The scenario now lives in one fixture helper with a test per expectation:
the failing one stays active on this pin, and the capped one is ignored
naming #23358, which also makes the failing one obsolete.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit a0d5d60)
(cherry picked from commit 9acfb95d21d749000055317e22978d9e36d81b55)
The pruner resolves the new earliest block through the chain reader that also
reads the Finish checkpoint, so the block-not-in-reth test mocks that reader,
and the bump's pruner tests take the renamed try_run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r has

Since #23358 the pruner caps the new earliest block at reth's Finish checkpoint
instead of failing on a block reth does not hold, so the capped case runs and the
failing case goes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23393-proofs-cli-tests branch from a0d5d60 to 3328f77 Compare November 9, 2026 22:30
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested review from a team as code owners November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: behavioural tests for the proofs CLI subcommands

3 participants