Repository navigation
kona-node: stop derivation when pipeline signals fail - #23815
Conversation
joshklop
left a comment
There was a problem hiding this comment.
Requesting changes. Stopping derivation from resuming on a pipeline that only partly reset is the right fix. My main concern is how far it goes: a temporary RPC error during a reset now shuts down the whole node. Inline comments cover that, the tests, and a few smaller points.
Only a failed reset stops the derivation actor, and it now increments the critical-error metric. Activation and channel flush failures stay non-fatal. Test pipelines share one signal-failure injection helper, reset failures use provider errors, and the supervisor test no longer depends on the error's Debug output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A temporary reset failure must be retried and a critical one must stop the actor, matching op-node's deriver and step scheduler. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A reset whose pipeline signal fails temporarily, such as on an L2 RPC error during the pipeline's initial reset, is retried with exponential backoff while the state machine keeps awaiting the signal. Only a critical failure stops the actor, matching op-node's deriver and step scheduler. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
joshklop
left a comment
There was a problem hiding this comment.
Requesting changes again. Thanks for working through the earlier round: the retry, the reset-only fatal path, the metric, and the test cleanups all look good. Two problems remain in the new retry path. A temporary reset failure doesn't stop derivation from resuming when the reset came from the engine rather than from derivation, and a Reset-kind error from the pipeline still shuts the node down. Inline comments cover both, plus a few smaller points.
One correction to my own earlier comment: I said that skipping SignalProcessed keeps the state machine in AwaitingSignal. That only holds when derivation requested the reset, and the first issue below follows from that.
A temporary reset failure only kept derivation parked when derivation had requested the reset. An engine-initiated reset leaves the state machine in AwaitingL1Data or AwaitingELSyncCompletion, so an L1 head or EL sync completion during the backoff window stepped a pipeline that never reset after SafeDB was already truncated. attempt_derivation now yields while a reset is pending, whichever actor requested it. A Reset-kind pipeline error, such as the reset head's parent being reorged out during the walk-back, was treated as critical and stopped the node. Like op-node's ResetEvent, it now asks the engine for a fresh reset head and holds derivation until that reset succeeds. Also: expose pending reset attempts as a gauge, forward non-reset signals through a pipeline-only helper instead of a signal() that special-cased Reset, and cover both engine-initiated reset shapes and the reorged head in the op-node parity tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
@einar-oplabs this PR fixes ethereum-optimism/protocol-team#586 (a failed pipeline reset resumes derivation on a stale pipeline), which is now tracked under ethereum-optimism/protocol-team#588. Could you add |
joshklop
left a comment
There was a problem hiding this comment.
Approving. Both blocking issues from the last round are fixed: attempt_derivation holds while a reset is pending in every state, whichever actor requested it, and a Reset-kind failure asks the engine for a fresh head instead of stopping the node. I walked the state machine for each resting state (including AwaitingELSyncCompletion at startup) and the retry's SignalProcessed then NewAttributesConfirmed sequence is valid from all of them. The engine releases reset waiters before sending the new signal, so the fresh-head request can't deadlock, and re-running truncate_safe_db on each attempt is idempotent (including the genesis re-record).
Verified locally at 5e439c02b7: the 94 derivation tests pass, and with the hold removed all three parity tests fail.
One non-blocking suggestion inline on repeated Reset-kind failures, plus two nits. Fine as a follow-up.
A failed pipeline reset is currently acknowledged as processed, so derivation can resume using retained stage data after SafeDB has already been truncated. A reset now adopts the engine's head and resumes derivation only once the pipeline reset has succeeded.
A temporary pipeline failure during the reset, such as an L2 RPC error while the pipeline walks back to its L1 origin, schedules a retry with exponential backoff (100 ms doubling to 10 s), and derivation holds until the reset succeeds, whichever actor requested it. A reset head the pipeline can no longer walk back from, because the L2 chain reorged under it, asks the engine for a fresh head instead. Only a critical failure stops the node. This matches op-node, whose deriver turns a temporary
initialReseterror intoEngineTemporaryErrorEventand lets the step scheduler retry it, turns a reset error intoResetEvent, and only shuts down onErrCritical. Activation and channel flush failures remain non-fatal. A gauge of pending reset attempts makes a node stuck retrying visible on dashboards.Closes ethereum-optimism/protocol-team#586.
Related: #23473. Acceptance/documentation is in #23787; additional mixed-node parity validation is in #23818.
5e439c02b7.The shutdown test exercises the production supervisor and RPC actor with a real HTTP endpoint, not a full rollup topology with an injected L1/provider failure.
🤖 Generated with Claude Code