Summary
On glamsterdam-devnet-7 (Gloas/ePBS, v1.44.0/94954a2, glamsterdam-devnet-7 branch), lodestar proposers fall back to self-building on ~20% of proposals even though a correctly-keyed builder bid was on the wire ~390 ms before slot start. The cause is measurable end-to-end: the bid's gossip validation frequently completes after produceBlockV4 has already done its single, synchronous bid-pool read at slot start. The bid loses a race it was never given a chance to re-enter.
Full write-ups (interactive charts, per-slot evidence, reproduction queries):
Mechanism (at 94954a2)
produceBlockV4 runs at slot start and calls executionPayloadBidPool.getBestBid(slot, bidParentBlockHash, parentBlockRootHex) once — a synchronous exact-triple map read with no wait, no re-check, and no pool-insert wake-up. null → self-build (there is no value comparison yet; the TODO GLOAS: respect builderSelection… above it).- The only path into that pool is
validateGossipExecutionPayloadBid, which includes a per-bid chain.regen.getBlockSlotState(..., {dontTransferCache: true}) plus BLS verification.
- buildoor (the devnet builder) publishes its first bid at −400 ms and its last at −100 ms before slot start (its defaults), so the entire budget between bid arrival and the pool read is ~390 ms — all of which is consumed by queueing + validation when the tail bites.
Evidence
Bid pipeline latency per implementation — each node's own beacon-event emission (fires post-validation) minus the earliest emission network-wide for the same bid; same bids, same mesh, 2.58 M observations, 2026-07-22 → 07-31:
| CL |
p50 |
p90 |
p99 |
| prysm / grandine / lighthouse / teku / nimbus |
4–7 ms |
8–13 ms |
29–85 ms |
| lodestar |
22 ms |
751 ms |
1.36 s |
The distribution is bimodal: a fast mode like every other client, plus a second population (~16% of bids) at 400–1,500 ms — a hump, not a tail, suggesting discrete stalls (event-loop pause / queue behind an expensive item / regen miss) rather than uniformly slower validation.
Per-slot causality on 7,850 lodestar proposals where the proposing node's own bid timeline is measurable:
| outcome |
own validation of the bid (median) |
n |
| bid taken |
−365 ms (before slot start) |
6,461 — 1 late |
| self-built |
+510 ms (after the decision; p90 +1,034 ms) |
1,389 — 1,267 late |
The daily fraction of late-validated bids tracks the daily self-build rate almost 1:1 (e.g. 24.1%/20.8%, 28.0%/21.3%, 24.0%/25.0%). The excess on 07-23 is the BuilderCircuitBreaker (working as intended during a payload non-reveal incident). A node restart on 07-28 did not reset the latency, so it is workload-shaped, not accumulated process state; what fattened the tail from 07-25 onward is still open and likely needs your internal gossip-queue/regen metrics.
Impact
On the devnet this is benign (the self-built fallback works and reveals early — every one of those payloads landed). But with the current one-shot read, a validator forfeits the builder bid on every slot the race is lost — the bids left on the table in this window were worth ~2× the local block. It also means lodestar's effective bid acceptance depends on its own validation latency, which seems unintended.
Possible fixes (any one would eliminate the effect)
- Re-check the pool (or wait briefly / wake on pool insert) before committing to the local payload in
produceBlockV4 — even a bounded ~500 ms grace re-read would recover nearly all of the 400–1,500 ms hump.
- Get the per-bid state regen out of the gossip-validation hot path (cache the prepared parent-branch state for the expected next slot, as bid validation for slot N almost always needs the same state).
- A metric on bid gossip-validation queue+processing time would make the tail visible without external tooling.
Happy to provide raw data, per-slot samples, or run instrumented builds on the devnet.
Summary
On glamsterdam-devnet-7 (Gloas/ePBS,
v1.44.0/94954a2,glamsterdam-devnet-7branch), lodestar proposers fall back to self-building on ~20% of proposals even though a correctly-keyed builder bid was on the wire ~390 ms before slot start. The cause is measurable end-to-end: the bid's gossip validation frequently completes afterproduceBlockV4has already done its single, synchronous bid-pool read at slot start. The bid loses a race it was never given a chance to re-enter.Full write-ups (interactive charts, per-slot evidence, reproduction queries):
Mechanism (at
94954a2)produceBlockV4runs at slot start and callsexecutionPayloadBidPool.getBestBid(slot, bidParentBlockHash, parentBlockRootHex)once — a synchronous exact-triple map read with no wait, no re-check, and no pool-insert wake-up.null→ self-build (there is no value comparison yet; theTODO GLOAS: respect builderSelection…above it).validateGossipExecutionPayloadBid, which includes a per-bidchain.regen.getBlockSlotState(..., {dontTransferCache: true})plus BLS verification.Evidence
Bid pipeline latency per implementation — each node's own beacon-event emission (fires post-validation) minus the earliest emission network-wide for the same bid; same bids, same mesh, 2.58 M observations, 2026-07-22 → 07-31:
The distribution is bimodal: a fast mode like every other client, plus a second population (~16% of bids) at 400–1,500 ms — a hump, not a tail, suggesting discrete stalls (event-loop pause / queue behind an expensive item / regen miss) rather than uniformly slower validation.
Per-slot causality on 7,850 lodestar proposals where the proposing node's own bid timeline is measurable:
The daily fraction of late-validated bids tracks the daily self-build rate almost 1:1 (e.g. 24.1%/20.8%, 28.0%/21.3%, 24.0%/25.0%). The excess on 07-23 is the
BuilderCircuitBreaker(working as intended during a payload non-reveal incident). A node restart on 07-28 did not reset the latency, so it is workload-shaped, not accumulated process state; what fattened the tail from 07-25 onward is still open and likely needs your internal gossip-queue/regen metrics.Impact
On the devnet this is benign (the self-built fallback works and reveals early — every one of those payloads landed). But with the current one-shot read, a validator forfeits the builder bid on every slot the race is lost — the bids left on the table in this window were worth ~2× the local block. It also means lodestar's effective bid acceptance depends on its own validation latency, which seems unintended.
Possible fixes (any one would eliminate the effect)
produceBlockV4— even a bounded ~500 ms grace re-read would recover nearly all of the 400–1,500 ms hump.Happy to provide raw data, per-slot samples, or run instrumented builds on the devnet.