Skip to content

kona-node: keep the safe head across L1 reorgs that leave its L1 data canonical - #23634

Draft
teddyknox wants to merge 20 commits into
developfrom
claude/happy-payne-c7b612
Draft

teddyknox wants to merge 20 commits into
developfrom
claude/happy-payne-c7b612

Conversation

@teddyknox

@teddyknox teddyknox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes ethereum-optimism/protocol-team#328.

Today any L1 reorg makes kona-node run the cold-start reset, which moves the safe head back a full sequencing window. It stops early only at the finalized head. On glamsterdam-defense-2, L1 finality had stalled, so replacing a single tip block rewound the safe head ~72k blocks.

What changes

  • Reorg reset. Derivation remembers, for its last 64 safe heads, which L1 block each one depends on. On a reorg, the engine keeps the newest safe head whose L1 block is still canonical. It only drops unsafe blocks built on replaced L1 blocks. A tip reorg usually moves nothing. The full reset still runs on cold start, or if the reorg is older than what was remembered.
  • Cold start. It checked an unsafe block's L1 origin by hash, which only proves the block exists, so it could keep an unsafe head built on a reorged-out L1 block. It now checks by number, shares the walk with the reorg reset, and never walks below the finalized head.
  • Cold-start safe head. The search for a safe head a sequencing window behind the unsafe head could stop above the previous safe head. That marked blocks safe that derivation never produced, whenever the unsafe head was more than a window ahead. Now the safe head is at or below the previous one, as in op-node's FindL2Heads.
  • Safe head DB. After a reset, it could record a block at an L1 block below its parent's. That made it answer a safe head for a height where it wasn't yet safe. It now records the same L1 block as the reorg reset's memory.

Why keeping the safe head is safe. Derivation is deterministic. If the L1 data a safe head was derived from is still canonical, the new L1 chain derives the same safe head. Restarting from it relies on nothing new: every reset already restarts from a safe head it trusts.

Limits

  • After a restart, the memory is empty, so reorgs take the full reset for about one sequencing window.
  • After Holocene, the DB's recorded L1 block is a lower bound rather than exact. That can differ from op-node; this needs a decision from protocol owners.
  • op-node has the same full-rewind behaviour and is not changed.

Review notes. rust-code-reviewer and a safedb-reviewer-style review ran, and their findings are applied. Plain cargo test can fail test_chain_label_metrics and a middleware test because of shared global state; both pass in isolation, and the middleware one also fails on develop. Not yet run against a real L1 reorg on a devnet.

🤖 Generated with Claude Code

teddyknox and others added 5 commits October 26, 2026 21:25
… canonical

Every L1 reorg that derivation detected triggered a full engine reset,
which re-runs find_starting_forkchoice and moves the safe head back a
full sequencing window, stopping early only at the finalized head. A
one-block replacement of the L1 tip cost as much as a cold start, and
while L1 finality was stalled it rewound the safe head ~72k L2 blocks
(ethereum-optimism/protocol-team#328).

Derivation now records its recent safe heads, each with an L1 block that
all of the L1 data the safe head depends on is at or below, and sends
them with the reset request. The engine keeps the newest one whose L1
block is still canonical, and falls back to the full reset when none
qualifies, when it is off the engine's chain, or when the unsafe head's
L1 origin was reorged out. A tip reorg usually keeps the current safe
head with no forkchoice update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… head

The engine reports its safe head only when it changes, so derivation now
resumes on its own when a reset keeps the safe head it has confirmed,
instead of the engine resending an unchanged safe head. That removes the
resend and the guard against it racing an L1 head. Also inlines the
pending-reset merge rule and consolidates overlapping tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…origin

A reorg that replaced the unsafe head's L1 origin fell back to the full
reset. Sequencers pick origins only 4-5 L1 blocks behind the tip, so that
covered every reorg deeper than about 5 blocks, the ones most likely
during a finality stall. The engine now keeps the safe head it chooses
and walks the unsafe chain back to its last block with a canonical L1
origin, never past that safe head. find_reorg_safe_head becomes
find_reorg_forkchoice and returns both heads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reset the pipeline replays from before the safe head's L1 origin
and can produce the next block from an L1 block below the one its parent
was recorded at. The in-memory SafeHeadHistory skipped such a block; the
safe head database recorded it there, out of ascending L1 order, so it
answered that block for L1 heights where its parent was not yet safe.

The history now raises such a block to its parent's L1 block and returns
that block, which the database records too, so the two records agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cold-start reset kept an unsafe head whenever its L1 origin could be
fetched by hash. An L1 node serves a block by hash after a reorg replaces
it at the same height, so an unsafe head built on a replaced L1 block
survived a cold start. find_starting_forkchoice and the L1-reorg reset
now share one walk: it checks each origin by number, checks a replaced
origin once, and never walks past a floor, the finalized head on a cold
start and the kept safe head on an L1 reorg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teddyknox
teddyknox force-pushed the claude/happy-payne-c7b612 branch from f51ee1e to 60e4baa Compare October 26, 2026 20:26
teddyknox and others added 3 commits October 27, 2026 01:07
Resolves the conflict with #23258, which replaced SynchronizeTask with the
shared synchronize step: the L1-reorg reset now calls synchronize, as the
full reset does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflicts with #23590, which split the derivation actor into
actor/core.rs and widened its fields to pub(super) for the tests. The actor
keeps both #23590's genesis record and this branch's safe head history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestLeadershipTransferMovesActiveSequencer failed in
memory-all-kona-op-reth-fusaka on a Raft/unsafe-head race during the
transfer, a path this branch does not touch; its sibling conductor tests
are flagged flaky in the same job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teddyknox
teddyknox marked this pull request as ready for review October 27, 2026 13:58
@teddyknox
teddyknox requested a review from a team as a code owner October 27, 2026 13:58
teddyknox and others added 5 commits October 27, 2026 19:27
Resolves the conflicts with #23223, #23229 and #23581. Import lists keep
both CanonicalizeRequest and ResetKind. The engine actor's reset keeps the
L1-reorg dispatch and, from #23223, publishes the unsafe head after either
kind of reset. The admin-reset test asserts #23223's published heads against
this branch's TestChain fixture: unsafe stays at 26, the safe heads walk back
to finalized 18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflicts with #23628, which has derivation adopt the head a
reset signal carries and resume after every reset, and drops the engine's
separate safe-head report after a reset. That subsumes this branch's resume
after a reset that keeps the confirmed safe head, so the merge takes #23628's
version.

Tests:
- The resume test no longer has a case that waits after a reset; it now
  checks that the block derived after a reset to the confirmed safe head is
  recorded.
- The history-clearing test asserts on the history directly, since
  derivation now resumes on the reset itself.
- #23628's invalid-forkchoice test expects no L1 call, as this branch's
  unsafe-chain walk makes none when the unsafe head is the finalized head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cold-start safe head search walked down from the unsafe head and stopped
at the first block more than a sequencing window behind it, without checking
the previous safe head. When the unsafe head is more than a window ahead of
derivation, that marks blocks safe that derivation never produced, and that a
batcher outage may have made invalid. op-node's FindL2Heads also requires the
safe head to be at or below the previous one; do the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflict in kona-node-service's export list: develop added
RunningDiscovery (#23753) next to this branch's ResetKind; both are kept.

Tests: fmt, clippy and rustdoc clean; 563/563 nextest in kona-engine,
kona-node-service, kona-rpc and kona-node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous-safe-head cap from the last commit regressed the reset after EL
sync. There the engine's safe and finalized heads are still at genesis, so the
capped search walked every L2 block back to genesis and restarted derivation
from there. Like op-node's RecoverMinSeqWindows check, skip the cap when the
safe and finalized heads are at genesis and the unsafe head's L1 origin is more
than 14 sequencing windows past L1 genesis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teddyknox
teddyknox marked this pull request as draft October 29, 2026 15:16
@teddyknox

Copy link
Copy Markdown
Contributor Author

Parking this in favour of #23696 + #23548. Together they cover the case behind protocol-team#328 (a shallow tip reorg rewinding the safe head) and keep kona's reset matching op-node.

The one case this PR still handles differently is an L1 reorg deeper than the verifier confirmation depth while L1 finality is stalled. There #23548, like op-node, rolls the safe head back about a sequencing window, and this PR keeps it. If that turns out to matter, it should be a spec change made in both clients, not kona-only behaviour.

Two things from here are worth carrying over:

  • Cap on the previous safe head in the cold-start safe walk. kona-engine,kona-node!: align reset recovery with op-node #23548 has it, but it also needs op-node's RecoverMinSeqWindows exception. Without it, the reset after EL sync, where safe and finalized are still at genesis, walks the whole chain back to genesis. See the last two commits here for the fix and its test.
  • Safe head DB ordering. After a reset, the safe head DB can record the next block at an L1 block below its parent's entry: before Holocene, its batch can land first and be buffered. op-node records it the same way, so this needs a decision for both clients rather than a kona-only fix.

…claude/happy-payne-c7b612

# Conflicts:
#	rust/kona/crates/node/service/src/actors/derivation/actor/core.rs
#	rust/kona/crates/node/service/src/actors/derivation/actor/tests.rs
…claude/happy-payne-c7b612

# Conflicts:
#	rust/kona/crates/node/engine/src/task_queue/core.rs
#	rust/kona/crates/node/service/src/actors/engine/actor.rs
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Agent approval check

Head branch claude/happy-payne-c7b612 matches agent prefix claude/

Current head: b48da2fce90a791b03f9803b5643b3fbc9e4f6d1
Human approvals: 0/2

A write-capable human can approve by submitting a native review or by commenting:

/approve b48da2fce90a791b03f9803b5643b3fbc9e4f6d1

teddyknox and others added 5 commits November 17, 2026 17:39
…claude/happy-payne-c7b612

# Conflicts:
#	rust/kona/crates/node/service/src/lib.rs
…claude/happy-payne-c7b612

# Conflicts:
#	rust/kona/crates/node/engine/src/task_queue/core.rs
…claude/happy-payne-c7b612

# Conflicts:
#	rust/kona/crates/node/engine/src/sync/error.rs
#	rust/kona/crates/node/engine/src/sync/mod.rs
#	rust/kona/crates/node/engine/src/test_utils/rpc.rs
#	rust/kona/crates/node/service/src/actors/engine/tests.rs
Adopts #23848/#23849's shared L1 client: the reorg path's canonical
L1 check now uses EngineClient::l1_block_hash_by_number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant