Fix dropping from SYNCING to IN_SYNC while far behind the head - #11219
Open
zilm13 wants to merge 5 commits into
Open
Fix dropping from SYNCING to IN_SYNC while far behind the head#11219zilm13 wants to merge 5 commits into
zilm13 wants to merge 5 commits into
Conversation
zilm13
marked this pull request as ready for review
September 2, 2026 18:58
rolfyone
reviewed
Sep 2, 2026
Comment on lines
+47
to
+48
| * Peer count from which we stop trusting a single peer's claimed head and require a second peer | ||
| * to corroborate it. |
Contributor
There was a problem hiding this comment.
so, really what we're saying is we need the 'MIN_PEERS_FOR_AGREEMENT' to agree, where as the comment suggests 2...
rolfyone
reviewed
Sep 2, 2026
Comment on lines
+120
to
+122
| this.maxSlotsBehindHead = | ||
| UInt64.valueOf( | ||
| (long) genesisSpec.getSlotsPerEpoch() * genesisSpec.getConfig().getMaxSeedLookahead()); |
Contributor
There was a problem hiding this comment.
given we have RecentChainData anyway, maybe just refactor RecentChainData to allow us to query the value - in this way we're not duplicating this computation if we're wanting them to be the same
its also specVersion dependent so potentially we should be just calling RecentChainData for the right value rather than storing
rolfyone
reviewed
Sep 2, 2026
|
|
||
| private void updateCurrentState() { | ||
| final SyncState previousState = currentState; | ||
| boolean heldBehindHead = false; |
Contributor
There was a problem hiding this comment.
what do we mean 'heldBehindHead'? that is a confusing term to me
Contributor
Author
|
@rolfyone addressed feedback, simplified the things |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Description
Issue: how we drop to
IN_SYNCwhile far behindSyncStateTracker.updateCurrentState()choseIN_SYNCpurely from!syncActive— it neverchecked how far behind the head actually was. Forward sync deactivates whenever it has no target
chain to work on, which includes "all peers dropped" and "stalled, waiting to retry". Both OOM
logs show exactly that: sync deactivates with the head still 850–1100 slots back, and the node
logs
Syncing completedand reportsIN_SYNC.What it leads to
SyncState.isInSync()→ForkChoiceNotifier.inSync = true→ProposersDataManager.calculatePayloadBuildingAttributespasses its!inSyncguard and asks forthe state at
currentSlot + 1on top of a head ~1000 slots older.StateAtSlotTask.regenerateFromStatereplays that gap: ~27–34 epoch transitions, eachmaterialising ~1M
ValidatorSSZ views plusValidatorStatusobjects.It also runs many times over:
streamIntermediateSteps()only looks back 640 slots, so a larger gap defeats rebasing anddeduplication entirely;
CachingTaskQueueallowsavailableProcessorsof them concurrently.9 GB heap exhausted →
OutOfMemoryError.The fix
Judge behind-ness against the head our peers report, not the wall-clock slot.
SyncStateTrackerimplementsSlotEventsChannel, re-evaluates each slot, and holdsSYNCINGwhile our head is more than
slotsPerEpoch * MAX_SEED_LOOKAHEAD(128 slots) behind thepeer-reported head from
Eth2Peer.getStatus().getHeadSlot().startupTargetPeerCount == 0startupTargetPeerCount > 0Requiring two peers to agree once 3+ are connected means one peer overstating its head can't
wedge us. Sync retry is untouched — a restart just moves the node from held-
SYNCINGtoactive-
SYNCING.Using peer heads rather than the current slot is the load-bearing detail: the obvious version
(head vs current slot) deadlocks on a chain with >128 consecutive empty slots, because the held
SYNCINGmakesValidatorApiHandler.isSyncActive()reject every duty, so nobody proposes and thehead never advances. Peer heads are equally old during such an outage, so the comparison stays
correct.
Fixed Issue(s)
Documentation
doc-change-requiredlabel to this PR if updates are required.Changelog