Repository navigation
kona-node: back off repeated pipeline reset failures of any kind - #23901
einar-oplabs wants to merge 1 commit into
Conversation
A reset head that the pipeline could not walk back from asked the engine for a fresh head immediately, and the engine reset immediately. An EL that keeps answering "not found" for a canonical block, as a lagging replica behind a load balancer does, drove a hot loop of engine forkchoice resets and L2 walk-backs, invisible on dashboards because the pending reset gauge was zeroed on the way. The pending reset now carries one consecutive-failure count across retries of a signal and fresh-head requests, and the next step runs after the same 100 ms to 10 s backoff whichever kind the last failure was, as op-node routes ResetEvent through its step scheduler. The gauge stays at the failure count until a reset succeeds. Also: forward_non_reset_signal asserts it is not handed a reset, and the derivation hold logs the failure count instead of the whole pending reset. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
f6214c0 to
5670ed7
Compare
joshklop
left a comment
There was a problem hiding this comment.
Approving. I found no correctness bugs.
- Held derivation:
attempt_derivationholds whenever any reset is pending. The pending reset is cleared only on success, which is also the only place the gauge goes back to 0. - Resets from other actors: a reset signal from the engine or sequencer that arrives during any step either clears the pending reset or adds to the shared count.
AwaitFreshHeadcan't hang: the engine replies only after its reset succeeds, and it always sends the reset signal afterwards.- Timer: the timer in
stepis correct. - Tests: the 93
actors::derivationtests pass locally on5670ed7.
The inline comments are about whether the op-node comparisons are accurate and about gaps in the tests.
One follow-up outside the diff, so it's here rather than inline. The title says "of any kind", but reset errors raised during derivation still skip the backoff. produce_next_attributes (core.rs:302-331) calls reset_engine_forkchoice() immediately and never touches pending_reset. Because a successful reset clears the count, this loop never backs off:
reset succeeds (count → 0) → derivation step returns Reset → immediate engine reset → reset succeeds → …
The PR's own lagging-replica scenario can trigger it on the L1 side. A beacon replica that returns 404 produces BlobNotFound, which becomes ResetError::BlobsUnavailable (protocol/derive/src/errors/sources.rs). L1 BlockNotFound and BlobsUnderFill behave the same way. op-node still backs off at least about 2 s here, because AttemptStep counts the failed step before ResetEvent reschedules it. This predates the PR, so I'd file it as a follow-up issue or narrow the title, not block on it.
Generated by Claude Code
| impl PendingReset { | ||
| /// op-node retries a failed reset, temporary or not, through its exponential step backoff, | ||
| /// capped at ten seconds. | ||
| const MAX_DELAY: Duration = Duration::from_secs(10); |
There was a problem hiding this comment.
This schedule isn't op-node's, although this comment and the PR body say it is. op-node's step backoff is retry.Exponential() (op-service/retry/strategies.go:49): Min(0) + 2^attempt × 1s, capped at 10 s, plus 0–250 ms of jitter. AttemptStep raises stepAttempts to 1 before the first failure, so op-node waits about 2 s, 4 s, 8 s, then 10 s. This code waits 100 ms, 200 ms, 400 ms and so on, without jitter:
- the first wait is 20× shorter;
- it makes about 7 attempts in the first 13 s, where op-node makes 3;
- without jitter, nodes that share a flaky EL behind a load balancer retry in lockstep.
The 100 ms floor came from #23815, but this PR now applies it to fresh-head requests too. Could we either match op-node (2 s base plus jitter) or reword this comment to claim only the 10 s cap?
Generated by Claude Code
| /// RPC error while the pipeline walks back to its L1 origin, schedules a retry with backoff, | ||
| /// and a reset head that was reorged out asks the engine for a fresh one. Only a critical | ||
| /// failure stops the actor. | ||
| /// and a reset head that was reorged out asks the engine for a fresh one after that same |
There was a problem hiding this comment.
Nit on the op-node comparison. op-node's onResetEvent (op-node/rollup/driver/sync_deriver.go) sends ResetEngineRequestEvent straight away. It delays only the next step, and the walk-back runs in that step. This PR instead delays the engine request itself.
Both limit how fast the loop can run, so the behaviour is fine. The visible difference is that kona leaves the engine forkchoice un-reset for up to 10 s, where op-node resets it immediately. Could the wording here, and in the PR body ("routes ResetEvent through RequestStep"), say that both limit the loop rate rather than imply the mechanism is the same?
Generated by Claude Code
| async fn retry_pending_reset(&mut self) -> Result<(), DerivationError> { | ||
| let Some(PendingReset::Retry(retry)) = &self.pending_reset else { | ||
| fn set_pending_reset(&mut self, attempts: u32, next_step: NextResetStep) { | ||
| metrics::gauge!(Metrics::DERIVATION_PENDING_RESET_ATTEMPTS).set(f64::from(attempts)); |
There was a problem hiding this comment.
This gauge fix is the change dashboards depend on, but no test reads DERIVATION_PENDING_RESET_ATTEMPTS. If it regressed to 0 on the fresh-head path, nothing would fail. Other kona crates already check metrics in tests with metrics_util::debugging::DebuggingRecorder and metrics::with_local_recorder (see gossip/src/driver/tests/mod.rs). Could repeated_reorged_reset_heads_back_off_like_op_node assert that the gauge reads 1, then 2, and goes back to 0 after the successful reset?
Generated by Claude Code
| self.reset_pipeline_and_resume(reset).await | ||
| let failed_attempts = pending.attempts; | ||
| match pending.next_step { | ||
| NextResetStep::AwaitFreshHead => Ok(()), |
There was a problem hiding this comment.
Nit: this arm can't be reached, and neither can the None early return above. due_at() returns None for both cases, so step never selects the timer branch for them. A short comment saying so, or debug_assert!(false, ...), would make the intent explicit.
Generated by Claude Code
| /// Runs the scheduled reset step and returns how long the actor waited before it. | ||
| async fn step_when_due(node: &mut RecordingActor) -> Duration { | ||
| let started = Instant::now(); | ||
| node.actor.step().await.expect("the scheduled reset step must succeed"); |
There was a problem hiding this comment.
Suppose the fresh-head request went back to being sent immediately. pending_reset would then be AwaitFreshHead with no deadline, so this step() would wait on recv forever. request_tx stays alive, and the paused clock has no timer to advance to. The suite would hang instead of failing. Wrapping the call in tokio::time::timeout(Duration::from_secs(11), ...) would turn that hang into a test failure.
Two orders and limits aren't covered:
- a temporary failure followed by a reorged-head failure (only the reverse order is tested);
- the 10 s cap.
Generated by Claude Code
Follow-up to #23815 for the three non-blocking items from its approval review.
A reset head that the pipeline could not walk back from asked the engine for a fresh head immediately, and
begin_resetset the engine's retry time to now, so an EL that keeps answering "not found" for a canonical block (a lagging replica behind a load balancer, or history pruned inside the channel-timeout window) drove a hot loop of engine forkchoice resets and L2 walk-backs. The pending reset gauge was zeroed on that path, so the loop was invisible on dashboards, and the gauge's description was wrong while a fresh head was awaited.The pending reset now carries one consecutive-failure count across retries of a signal and fresh-head requests, and the next step runs after the same 100 ms to 10 s backoff whichever kind the last failure was, as op-node routes
ResetEventthroughRequestStepand its step backoff. The gauge stays at the failure count until a reset succeeds.Also:
forward_non_reset_signalasserts in debug builds that it is not handed aReset, so the contract matches the name.The derivation hold logs the failure count instead of the whole pending reset with its raw
Instant.repeated_reorged_reset_heads_back_off_like_op_node: two consecutive reorged-head failures wait 100 ms then 200 ms before the engine is asked, and the third reset resumes derivation.a_temporary_failure_after_a_fresh_head_continues_the_backoff: a temporary failure of the fresh reset waits 200 ms, not 100 ms.The existing reorged-head parity test steps through the scheduled request.
All 285 node-service unit tests pass locally on the rebased branch; clippy and pinned-nightly formatting pass.
CI green.
🤖 Generated with Claude Code