Skip to content

kona-node: back off repeated pipeline reset failures of any kind - #23901

Closed
einar-oplabs wants to merge 1 commit into
developfrom
einar/kona-reset-backoff
Closed

einar-oplabs wants to merge 1 commit into
developfrom
einar/kona-reset-backoff

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #23815 for the three non-blocking items from its approval review.

A reset head that the pipeline could not walk back from asked the engine for a fresh head immediately, and begin_reset set the engine's retry time to now, so an EL that keeps answering "not found" for a canonical block (a lagging replica behind a load balancer, or history pruned inside the channel-timeout window) drove a hot loop of engine forkchoice resets and L2 walk-backs. The pending reset gauge was zeroed on that path, so the loop was invisible on dashboards, and the gauge's description was wrong while a fresh head was awaited.

The pending reset now carries one consecutive-failure count across retries of a signal and fresh-head requests, and the next step runs after the same 100 ms to 10 s backoff whichever kind the last failure was, as op-node routes ResetEvent through RequestStep and its step backoff. The gauge stays at the failure count until a reset succeeds.

Also:

  • forward_non_reset_signal asserts in debug builds that it is not handed a Reset, so the contract matches the name.

  • The derivation hold logs the failure count instead of the whole pending reset with its raw Instant.

  • repeated_reorged_reset_heads_back_off_like_op_node: two consecutive reorged-head failures wait 100 ms then 200 ms before the engine is asked, and the third reset resumes derivation.

  • a_temporary_failure_after_a_fresh_head_continues_the_backoff: a temporary failure of the fresh reset waits 200 ms, not 100 ms.

  • The existing reorged-head parity test steps through the scheduled request.

  • All 285 node-service unit tests pass locally on the rebased branch; clippy and pinned-nightly formatting pass.

  • CI green.

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Nov 3, 2026
Base automatically changed from einar/kona-failed-reset to develop November 4, 2026 00:16
A reset head that the pipeline could not walk back from asked the engine
for a fresh head immediately, and the engine reset immediately. An EL that
keeps answering "not found" for a canonical block, as a lagging replica
behind a load balancer does, drove a hot loop of engine forkchoice resets
and L2 walk-backs, invisible on dashboards because the pending reset gauge
was zeroed on the way.

The pending reset now carries one consecutive-failure count across retries
of a signal and fresh-head requests, and the next step runs after the
same 100 ms to 10 s backoff whichever kind the last failure was, as
op-node routes ResetEvent through its step scheduler. The gauge stays at
the failure count until a reset succeeds.

Also: forward_non_reset_signal asserts it is not handed a reset, and the
derivation hold logs the failure count instead of the whole pending reset.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/kona-reset-backoff branch from f6214c0 to 5670ed7 Compare November 4, 2026 12:17
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 5, 2026 10:01
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 5, 2026 10:01

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I found no correctness bugs.

  • Held derivation: attempt_derivation holds whenever any reset is pending. The pending reset is cleared only on success, which is also the only place the gauge goes back to 0.
  • Resets from other actors: a reset signal from the engine or sequencer that arrives during any step either clears the pending reset or adds to the shared count.
  • AwaitFreshHead can't hang: the engine replies only after its reset succeeds, and it always sends the reset signal afterwards.
  • Timer: the timer in step is correct.
  • Tests: the 93 actors::derivation tests pass locally on 5670ed7.

The inline comments are about whether the op-node comparisons are accurate and about gaps in the tests.

One follow-up outside the diff, so it's here rather than inline. The title says "of any kind", but reset errors raised during derivation still skip the backoff. produce_next_attributes (core.rs:302-331) calls reset_engine_forkchoice() immediately and never touches pending_reset. Because a successful reset clears the count, this loop never backs off:

reset succeeds (count → 0) → derivation step returns Reset → immediate engine reset → reset succeeds → …

The PR's own lagging-replica scenario can trigger it on the L1 side. A beacon replica that returns 404 produces BlobNotFound, which becomes ResetError::BlobsUnavailable (protocol/derive/src/errors/sources.rs). L1 BlockNotFound and BlobsUnderFill behave the same way. op-node still backs off at least about 2 s here, because AttemptStep counts the failed step before ResetEvent reschedules it. This predates the PR, so I'd file it as a follow-up issue or narrow the title, not block on it.


Generated by Claude Code

impl PendingReset {
/// op-node retries a failed reset, temporary or not, through its exponential step backoff,
/// capped at ten seconds.
const MAX_DELAY: Duration = Duration::from_secs(10);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This schedule isn't op-node's, although this comment and the PR body say it is. op-node's step backoff is retry.Exponential() (op-service/retry/strategies.go:49): Min(0) + 2^attempt × 1s, capped at 10 s, plus 0–250 ms of jitter. AttemptStep raises stepAttempts to 1 before the first failure, so op-node waits about 2 s, 4 s, 8 s, then 10 s. This code waits 100 ms, 200 ms, 400 ms and so on, without jitter:

  • the first wait is 20× shorter;
  • it makes about 7 attempts in the first 13 s, where op-node makes 3;
  • without jitter, nodes that share a flaky EL behind a load balancer retry in lockstep.

The 100 ms floor came from #23815, but this PR now applies it to fresh-head requests too. Could we either match op-node (2 s base plus jitter) or reword this comment to claim only the 10 s cap?


Generated by Claude Code

/// RPC error while the pipeline walks back to its L1 origin, schedules a retry with backoff,
/// and a reset head that was reorged out asks the engine for a fresh one. Only a critical
/// failure stops the actor.
/// and a reset head that was reorged out asks the engine for a fresh one after that same

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit on the op-node comparison. op-node's onResetEvent (op-node/rollup/driver/sync_deriver.go) sends ResetEngineRequestEvent straight away. It delays only the next step, and the walk-back runs in that step. This PR instead delays the engine request itself.

Both limit how fast the loop can run, so the behaviour is fine. The visible difference is that kona leaves the engine forkchoice un-reset for up to 10 s, where op-node resets it immediately. Could the wording here, and in the PR body ("routes ResetEvent through RequestStep"), say that both limit the loop rate rather than imply the mechanism is the same?


Generated by Claude Code

async fn retry_pending_reset(&mut self) -> Result<(), DerivationError> {
let Some(PendingReset::Retry(retry)) = &self.pending_reset else {
fn set_pending_reset(&mut self, attempts: u32, next_step: NextResetStep) {
metrics::gauge!(Metrics::DERIVATION_PENDING_RESET_ATTEMPTS).set(f64::from(attempts));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gauge fix is the change dashboards depend on, but no test reads DERIVATION_PENDING_RESET_ATTEMPTS. If it regressed to 0 on the fresh-head path, nothing would fail. Other kona crates already check metrics in tests with metrics_util::debugging::DebuggingRecorder and metrics::with_local_recorder (see gossip/src/driver/tests/mod.rs). Could repeated_reorged_reset_heads_back_off_like_op_node assert that the gauge reads 1, then 2, and goes back to 0 after the successful reset?


Generated by Claude Code

self.reset_pipeline_and_resume(reset).await
let failed_attempts = pending.attempts;
match pending.next_step {
NextResetStep::AwaitFreshHead => Ok(()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this arm can't be reached, and neither can the None early return above. due_at() returns None for both cases, so step never selects the timer branch for them. A short comment saying so, or debug_assert!(false, ...), would make the intent explicit.


Generated by Claude Code

/// Runs the scheduled reset step and returns how long the actor waited before it.
async fn step_when_due(node: &mut RecordingActor) -> Duration {
let started = Instant::now();
node.actor.step().await.expect("the scheduled reset step must succeed");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suppose the fresh-head request went back to being sent immediately. pending_reset would then be AwaitFreshHead with no deadline, so this step() would wait on recv forever. request_tx stays alive, and the paused clock has no timer to advance to. The suite would hang instead of failing. Wrapping the call in tokio::time::timeout(Duration::from_secs(11), ...) would turn that hang into a test failure.

Two orders and limits aren't covered:

  • a temporary failure followed by a reorged-head failure (only the reverse order is tested);
  • the 10 s cap.

Generated by Claude Code

@joshklop joshklop closed this Nov 21, 2026
auto-merge was automatically disabled November 21, 2026 22:41

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants