Skip to content

Caught-up fix - #2472

Merged
kushti merged 1 commit into
v6.0.4from
caughtup-fix
Aug 18, 2026
Merged

Caught-up fix#2472
kushti merged 1 commit into
v6.0.4from
caughtup-fix

Conversation

@kushti

@kushti kushti commented Aug 18, 2026

Copy link
Copy Markdown
Member

Fix ExtraIndexer stalling after "Deferring catch-up": add a FullBlockApplied handler for the !caughtUp state that reschedules Index(), so the indexer resumes catch-up when the chain heals instead of waiting for a Rollback. Includes a regression test that fails without the fix.

@kushti
kushti merged commit e764689 into v6.0.4 Aug 18, 2026
14 of 16 checks passed
@a-shannon

Copy link
Copy Markdown
Contributor

Post-merge review note: the wake-up addresses the reported stall, but I think two follow-ups are needed before shipping 6.0.4.

  1. While caughtUp == false, every successful catch-up Index() already enqueues its successor. The new FullBlockApplied case enqueues another Index() for every applied full block, so pending tokens grow linearly during sync and drain only after catch-up. This needs a coalesced/single-flight wake-up. (current path)

  2. Deferred recovery still reads blockCache by height before history and does not bind the cached BlockTransactions to the selected next header. A body prefetched from branch B can therefore be indexed while the state records branch A's header ID. (cache path, index path)

Both safeguards already exist in #2465: scheduleRetry() uses one generation-guarded timer, and getBlockTransactionsAt(height, header) accepts a cached section only when its headerId matches before falling back to the section identified by header.transactionsId. (retry, cache binding)

#2465 is broader and intentionally moves the extra-index schema to version 7. For 6.0.4, a narrow backport of only the single-flight retry and cache/header binding would preserve the no-reindex goal; otherwise #2465 is the existing follow-up rather than a new parallel PR.

Two focused regressions would pin the boundary:

  • multiple FullBlockApplied events while one block behind produce one terminal catch-up;
  • a branch-stale cached BlockTransactions is rejected for the selected header.

@odiseusme

Copy link
Copy Markdown

Confirmed the mechanism from ExtraIndexer.loaded(): with caughtUp=false and no rollback in progress, FullBlockApplied reaches the no-op catch-all and is dropped; once "Deferring catch-up because the next header does not extend the indexed tip" returns, no continuing self-schedule remains, and only a qualifying Rollback/RemoveAfter path automatically restarts catch-up. A headers-only fork that briefly becomes best and then loses does not itself produce the full-chain switch that publishes Rollback, so the indexer can remain deferred indefinitely unless a later Rollback, another Index() trigger, or a restart wakes it. Rescheduling Index() on FullBlockApplied is the minimal fix.

Mailbox serialization avoids reentrancy: each Index() observes the latest state installed with context.become and, while the required block is available, advances one block. Once caughtUp=true, surplus Index() messages only repeat blockCache.clear(), caughtUpHook() and the caught-up log; saveProgress runs only for the first such message if modCount > 0. While the mismatch persists, each FullBlockApplied schedules another Index(), so this can produce one info-level deferral log and one queued retry per applied block; harmless at live-chain cadence, a bit noisy during fast block application, not a correctness issue.

Test: it sets caughtUp=false while the next best header still extends the indexed tip, so it proves the reschedule fires but not the deferred-then-healed transition. GenerateBetterChainTip() exists but builds a competing full block via generateBetter, so a stronger case would need a headers-only better tip, drive Index() into the deferral branch, then extend the original chain past it with a full block and assert the indexer resumes without a Rollback.

Field note: my 6.0.3 node with extraIndex on did not hit this today; no "Deferring catch-up" in its log 04:34-12:00 UTC (it was offline before that) and the indexer buffered 1853474 within a second of the block applying at 06:57:59. So the fork storm was not sufficient on that node; the source path additionally requires caughtUp=false with the next best header not extending the indexed tip, followed by no effective wake-up message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants