Gloas payload-by-root sync retries forever when peer selection comes up empty

[nflaig]

Observed on a local gloas kurtosis devnet while e2e testing #9832: a node missing the payload envelope for an early slot never recovered it and stayed on its own fork, retrying the same download for the rest of the run with an ever-growing delay

Aug-15 18:44:55.082[sync] debug: Ignoring unknown payload root after failed download slot=1, root=0x9b7c...dc29, pendingPayloads=1, delaySec=48.06 - Error fetching payload by root slot=1 root=0x9b7c...dc29 after 2: cannot find peer with needed columns=[]

pendingColumns was empty and filterPeers treats an empty set as matching any peer, so the balancer's peer list must have been empty or every peer was excluded, rate limited or at its concurrent request cap. The node was otherwise participating in the network, so it had connected peers.

Two things worth looking at:

• why peer selection comes up empty while peers are connected, peersMeta may not track the peers that could serve the request
• the retry loop never recovered even over minutes, an eventual attempt against fresh peers should succeed

[nflaig]

@twoeths this might already be resolved I think, but worth to double check

[nflaig]

my claude thinks this is still an issue 👇

---

Reproduced locally on a fresh minimal-preset gloas kurtosis devnet (gloas_fork_epoch: 0), with a second variant of the symptom: every node queues the genesis block root as a pending payload right after startup and retries the download until the entry is pruned at finalization, 60+ attempts over ~3 minutes.

Ignoring unknown payload root after failed download slot=0, root=0x17a1b0..., pendingPayloads=1, delaySec=186.5 - Error fetching payload by root slot=0 root=0x17a1b0... after 5: no peer available to download pending payload

On some nodes peer selection comes up empty as described above, on others the requests go out and every peer responds with Missing execution payload envelope for root=..., which can never succeed since genesis has no envelope.

The trigger is searchUnknownEnvelope: gossip messages referencing the genesis root pass the chain.seenPayloadEnvelope(root) check because the anchor seeding in the chain constructor only adds the anchor bid to the envelope cache for anchorBlockSlot > 0, so the genesis root is never considered seen and gets queued for an impossible download.

[nflaig]

@lodekeeper is that related to what you already fixed here #9281, if that's the case, can you please revisit your pr

[lodekeeper]

Checked this against fix/genesis-envelope-sync / #9281.

The fresh-genesis variant looks like the same class of impossible target that #9281 is meant to catch. On unstable, searchUnknownEnvelope() only has the seen-cache / awaiting / per-slot dedupe checks, so if chain.seenPayloadEnvelope(genesisRoot) is false it can enqueue the genesis root. On #9281, the same function does one more check before enqueueing:

const knownBlock = this.chain.forkChoice.getBlockHexDefaultStatus(root);
if (!canKnownBlockRequireExecutionPayloadEnvelope(..., knownBlock)) return;

That does not depend on the seenPayloadEnvelopeInputCache seeding path. Fork choice is initialized before the seen-envelope cache, and ProtoArray.initialize() inserts the anchor block. For a fresh gloas_fork_epoch: 0 genesis, the genesis block is therefore a known fork-choice block with slot 0, and the helper returns false for known genesis (knownBlock.slot > GENESIS_SLOT is required). The current test on #9281 covers exactly that decision: returns false for genesis even when current fork is gloas-era.

Verification I ran locally after rebuilding the worktree:

CI=true pnpm vitest run --project unit packages/beacon-node/test/unit/network/processor.executionPayloadEnvelopeEligibility.test.ts
# 1 file passed, 4 tests passed

So I do not think there is a new #9281 patch needed for the genesis-root retry loop; I would retest this repro against the #9281 branch/head.

I would keep the original slot=1 / cannot find peer with needed columns=[] symptom separate for now. #9281 prevents impossible known roots (genesis / known pre-Gloas) from entering payload-by-root sync, but it does not change the peer balancer retry path for a real post-Gloas block that genuinely needs an envelope. In that path an empty pendingColumns set still means “any peer can serve the envelope”, and bestPeerForPendingColumns() can still return null if no tracked peer is eligible / all are excluded, rate-limited, or at request cap. I do not see evidence from this check that that part is fixed.

[twoeths]

I found 2 issues with BlockInputSync when investigating this @nflaig

• need to feed PeerSyncMeta initially when we subscribe to network, have fix: feed peer sync meta to UnknownBlockPeerBalancer on subscribeToNetwork #9924 for it
• need to enhance UnknownBlockPeerBalancer for gloas, tracked in Enhance UnknownBlockPeerBalancer for gloas #9923

[lodekeeper]

Thanks @twoeths — those two line up exactly with the peer-selection follow-up I flagged above (the path #9281 doesn't touch).

Mapping them back to the original slot=1 symptom (cannot find peer with needed columns=[], empty pendingColumns matching any peer, yet no peer found):

• fix: feed peer sync meta to UnknownBlockPeerBalancer on subscribeToNetwork #9924 (feed initial PeerSyncMeta on subscribeToNetwork) matches the issue-body hypothesis directly — "the balancer's peer list must have been empty" / "peersMeta may not track the peers that could serve the request." As fix: feed peer sync meta to UnknownBlockPeerBalancer on subscribeToNetwork #9924 puts it, UnknownBlockPeerBalancer only accrues sync meta from peerConnected events, so peers already connected before it subscribes are never seeded. columns=[] then matches "any peer," but the tracked set is still empty → cannot find peer. That's the empty-selection case.
• Enhance UnknownBlockPeerBalancer for gloas #9923 (know which peers can actually serve a payload / columns for a given block root) covers the deeper selection correctness and the second bullet — "an eventual attempt against fresh peers should succeed." There's currently no per-root notion of who can serve an unknown-block payload (and for columns, custody advertisement ≠ having the columns for that specific root), so even with peers tracked, selection can't reliably pick an eligible one.

So yes — I think #9923 + #9924 explain the original empty-peer-selection symptom (pending the fixes landing + a re-test on the gloas devnet).

One thing to keep separate: the genesis-envelope variant I looked at (slot=0 genesis root queued as a pending payload) is not subsumed by #9923/#9924. That one isn't a peer-selection failure — on some nodes it even shows peers responding Missing execution payload envelope for root=…, i.e. selection worked fine, the target was just impossible (genesis has no envelope). That's the impossible-target enqueue path #9281 guards against (fork-choice known-block check before enqueue). So the two are complementary:

• Enhance UnknownBlockPeerBalancer for gloas #9923 / fix: feed peer sync meta to UnknownBlockPeerBalancer on subscribeToNetwork #9924 → fix peer selection for real post-Gloas targets that genuinely need an envelope.
• fix: skip impossible payload envelope sync targets #9281 → stop genesis / known pre-Gloas roots from ever entering payload-by-root sync in the first place.

Happy to help review #9923/#9924 or re-run the kurtosis repro against them once they're up.