Skip to content

kona-node: stop derivation when pipeline signals fail - #23815

Merged
einar-oplabs merged 7 commits into
developfrom
einar/kona-failed-reset
Nov 4, 2026
Merged

einar-oplabs merged 7 commits into
developfrom
einar/kona-failed-reset

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

A failed pipeline reset is currently acknowledged as processed, so derivation can resume using retained stage data after SafeDB has already been truncated. A reset now adopts the engine's head and resumes derivation only once the pipeline reset has succeeded.

A temporary pipeline failure during the reset, such as an L2 RPC error while the pipeline walks back to its L1 origin, schedules a retry with exponential backoff (100 ms doubling to 10 s), and derivation holds until the reset succeeds, whichever actor requested it. A reset head the pipeline can no longer walk back from, because the L2 chain reorged under it, asks the engine for a fresh head instead. Only a critical failure stops the node. This matches op-node, whose deriver turns a temporary initialReset error into EngineTemporaryErrorEvent and lets the step scheduler retry it, turns a reset error into ResetEvent, and only shuts down on ErrCritical. Activation and channel flush failures remain non-fatal. A gauge of pending reset attempts makes a node stuck retrying visible on dashboards.

Closes ethereum-optimism/protocol-team#586.

Related: #23473. Acceptance/documentation is in #23787; additional mixed-node parity validation is in #23818.

  • op-node parity tests: a temporary reset failure neither stops the actor nor lets derivation resume on an L1 head or EL sync completion during the backoff, and the scheduled retry adopts the reset head and resumes; a reorged reset head requests a fresh engine reset and holds derivation until it succeeds. All three fail without the derivation hold and pass with it.
  • Critical reset-error actor regressions fail without the fix and pass with it, forbidding stale payload/finalization submission.
  • Two-actor service-supervisor/RPC test for the critical error: RPC responds before injection, the error reaches the production supervisor, cancellation fires, and the server stops.
  • All 269 node-service unit tests pass locally, with zero skips.
  • Targeted clippy and pinned-nightly formatting pass.
  • CI green for 5e439c02b7.
  • Human approval and merge.

The shutdown test exercises the production supervisor and RPC actor with a real HTTP endpoint, not a full rollup topology with an injected L1/provider failure.

🤖 Generated with Claude Code

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. Stopping derivation from resuming on a pipeline that only partly reset is the right fix. My main concern is how far it goes: a temporary RPC error during a reset now shuts down the whole node. Inline comments cover that, the tests, and a few smaller points.

einar-oplabs and others added 4 commits November 1, 2026 13:13
Only a failed reset stops the derivation actor, and it now increments the
critical-error metric. Activation and channel flush failures stay non-fatal.
Test pipelines share one signal-failure injection helper, reset failures use
provider errors, and the supervisor test no longer depends on the error's
Debug output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A temporary reset failure must be retried and a critical one must stop the
actor, matching op-node's deriver and step scheduler.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A reset whose pipeline signal fails temporarily, such as on an L2 RPC error
during the pipeline's initial reset, is retried with exponential backoff
while the state machine keeps awaiting the signal. Only a critical failure
stops the actor, matching op-node's deriver and step scheduler.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs requested a review from joshklop November 1, 2026 20:36

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes again. Thanks for working through the earlier round: the retry, the reset-only fatal path, the metric, and the test cleanups all look good. Two problems remain in the new retry path. A temporary reset failure doesn't stop derivation from resuming when the reset came from the engine rather than from derivation, and a Reset-kind error from the pipeline still shuts the node down. Inline comments cover both, plus a few smaller points.

One correction to my own earlier comment: I said that skipping SignalProcessed keeps the state machine in AwaitingSignal. That only holds when derivation requested the reset, and the first issue below follows from that.

A temporary reset failure only kept derivation parked when derivation had
requested the reset. An engine-initiated reset leaves the state machine in
AwaitingL1Data or AwaitingELSyncCompletion, so an L1 head or EL sync
completion during the backoff window stepped a pipeline that never reset
after SafeDB was already truncated. attempt_derivation now yields while a
reset is pending, whichever actor requested it.

A Reset-kind pipeline error, such as the reset head's parent being reorged
out during the walk-back, was treated as critical and stopped the node.
Like op-node's ResetEvent, it now asks the engine for a fresh reset head
and holds derivation until that reset succeeds.

Also: expose pending reset attempts as a gauge, forward non-reset signals
through a pipeline-only helper instead of a signal() that special-cased
Reset, and cover both engine-initiated reset shapes and the reorged head
in the op-node parity tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@joshklop

joshklop commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@einar-oplabs this PR fixes ethereum-optimism/protocol-team#586 (a failed pipeline reset resumes derivation on a stale pipeline), which is now tracked under ethereum-optimism/protocol-team#588. Could you add Closes ethereum-optimism/protocol-team#586 to the description? I closed #23858, which fixed the same bug, in favor of this PR, and rebased the reset-epoch change (#23859) onto your branch.

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Both blocking issues from the last round are fixed: attempt_derivation holds while a reset is pending in every state, whichever actor requested it, and a Reset-kind failure asks the engine for a fresh head instead of stopping the node. I walked the state machine for each resting state (including AwaitingELSyncCompletion at startup) and the retry's SignalProcessed then NewAttributesConfirmed sequence is valid from all of them. The engine releases reset waiters before sending the new signal, so the fresh-head request can't deadlock, and re-running truncate_safe_db on each attempt is idempotent (including the genesis re-record).

Verified locally at 5e439c02b7: the 94 derivation tests pass, and with the hold removed all three parity tests fail.

One non-blocking suggestion inline on repeated Reset-kind failures, plus two nits. Fine as a follow-up.

@einar-oplabs
einar-oplabs disabled auto-merge November 3, 2026 23:38
@einar-oplabs
einar-oplabs added this pull request to the merge queue Nov 3, 2026
Merged via the queue into develop with commit 1920e48 Nov 4, 2026
124 checks passed
@einar-oplabs
einar-oplabs deleted the einar/kona-failed-reset branch November 4, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants