Skip to content

docs,op-acceptance-tests,op-devstack,op-service: cover an op-reth upgrade and rollback over a live datadir - #23427

Closed
einar-oplabs wants to merge 1 commit into
einar/23380-proofs-unclean-shutdownfrom
einar/23395-reth-upgrade-rollback
Closed

einar-oplabs wants to merge 1 commit into
einar/23380-proofs-unclean-shutdownfrom
einar/23395-reth-upgrade-rollback

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Operators upgrade op-reth over a live datadir and proofs store and roll back when the upgrade misbehaves; nothing tested that path.

op_reth_upgrade switches a node between the previous release and this tree's op-reth over {follower, sequencer} x {clean stop, SIGKILL} x {upgrade, rollback} and requires it to resume, keep up and serve matching proofs. It skips unless RUST_BINARY_PATH_OP_RETH_PREVIOUS is set; the follower rollback cells skip naming ethereum-optimism/protocol-team#482 (ExEx WAL decode). OpReth.Start now fails as soon as the process exits.

Stacked on #23421. Closes #23395. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 374 31 +343
Tests 586 0 +586
Docs 16 0 +16

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 30, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from c474193 to 35282e5 Compare November 9, 2026 19:35
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch 2 times, most recently from 0e56ee4 to 82fc120 Compare November 10, 2026 13:38
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 35282e5 to 2d72947 Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into 82fc120. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


Every operator upgrades op-reth over a node's live datadir and proofs store, and rolls back when the upgrade misbehaves; nothing tested that path. op-acceptance-tests/tests/op_reth_upgrade now runs, for each of {follower, sequencer} x {clean stop, SIGKILL} x {upgrade, rollback}, a node on the previous op-reth release, switches it in place to this tree's op-reth (and back, for the rollback cells), and requires it to resume on the chain its datadir holds, keep up, serve historical proofs that match the sequencer's for the blocks every binary wrote, and log no error. Each cell gets its own system rather than the single four-phase system the issue sketches, so a kill in one cell cannot taint another and a cell can be rerun alone; each phase spans the bump's persistence threshold plus the in-memory buffer, so every binary has both persisted and unpersisted blocks when it is stopped.

The previous release is not built by this repo, so the package skips unless RUST_BINARY_PATH_OP_RETH_PREVIOUS names it (see docs/ai/acceptance-tests.md). CI skips it until the job provides that binary.

With previous = this branch's base (reth 0fbe428) and current = the reth 2.6 bump (#23046 head, reth 8d359d01), under op-node and under kona-node alike: every upgrade cell and both sequencer rollback cells pass. Both follower rollback cells fail: the previous release aborts at startup with failed to decode notification N from <datadir>/exex/wal/N.wal: wrong msgpack marker FixArray(14), because the bump changes the serde shape of the sorted trie updates and hashed storage inside every ExEx notification. A proofs-history node therefore cannot roll back across the bump without first deleting exex/wal. Per #23395 that is a release-notes item, not a reason to loosen the test; once it has a tracking issue, the two cells should skip naming it.

The devstack gains the seams the test needs (a per-node binary switch, a recorder of a node's process logs, a skip for binaries no package here builds) and a fix the test exposed: sysgo.OpReth.Start waited for the process to announce its RPC endpoints and, when the binary aborted first, blocked until the test deadline (33 minutes per cell here). It now fails as soon as the process exits, naming the node and its exit status.

go-code-reviewer was run and every finding was addressed, none dismissed; the general and security reviews were skipped because the change is test infrastructure.

Part of #23395.

@einar-oplabs einar-oplabs changed the title op-acceptance-tests,op-devstack: cover an op-reth upgrade and rollback over a live datadir docs,op-acceptance-tests,op-devstack,op-service: cover an op-reth upgrade and rollback over a live datadir Nov 10, 2026
@einar-oplabs
einar-oplabs requested a review from aakoshh November 10, 2026 14:07
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 2d72947 to a1ecc68 Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from 82fc120 to 09e9bee Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from a1ecc68 to af31ccb Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from 09e9bee to 44d938b Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from af31ccb to 8f5a66c Compare November 11, 2026 13:59
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch 2 times, most recently from 604eed8 to fffdfb3 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 8f5a66c to 272c097 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from fffdfb3 to faa379b Compare November 11, 2026 16:28
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 272c097 to 259913d Compare November 11, 2026 16:28
…rade and rollback over a live datadir

Switch a node between the previous op-reth release and this tree's
op-reth over its datadir and proofs store, after a clean stop or a
SIGKILL, and require it to resume, keep up, serve historical proofs that
match the sequencer's from the first pre-switch block on, and log no
error, the replaced binary's shutdown included. The follower rollback
cells are skipped naming ethereum-optimism/protocol-team#482: the
previous op-reth aborts at startup decoding the ExEx WAL the reth 2.6
binary wrote. In the sequencer cells the batcher is stopped, so
derivation cannot pull the sequencer back, and after a kill the follower
is moved onto the rebuilt chain through the engine API before it must
keep up over gossip.

The devstack gains a binary switch that takes a path resolved before the
chain starts and a recorder of a node's process logs. OpReth.Start fails
as soon as the process exits, before or right after it announces its RPC
servers, quoting its last unstructured stderr lines.
logpipe.ParseRustStderr raises an unstructured stderr line to ERROR only
when it looks like a Rust panic or abort, for op-reth, kona-node and
op-zk-proposer.

Part of #23395.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 259913d to 7f21b30 Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from faa379b to 3b2c3bc Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 13:18
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 13:18
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants