Skip to content

[WIP] Backfill Gloas execution payload envelopes alongside blocks - #17394

Draft
satushh wants to merge 4 commits into
developfrom
envelope-backfill-minimal
Draft

[WIP] Backfill Gloas execution payload envelopes alongside blocks#17394
satushh wants to merge 4 commits into
developfrom
envelope-backfill-minimal

Conversation

@satushh

@satushh satushh commented Aug 21, 2026

Copy link
Copy Markdown
Member

What type of PR is this?

Feature

What does this PR do? Why is it needed?

Adds one stage to the existing backfill service so a checkpoint-synced node also fills in execution payload envelopes for the historical slots it backfills blocks for. Envelopes are fetched from CL peers, verified statelessly, cross-checked against the local EL, and persisted blinded alongside the blocks.

No new services, no coverage state, and no changes to serving or availability advertisement.

Why blocks aren't enough any more

Pre-Gloas a block contains its payload — backfill downloads the block and is done. Gloas splits them into two independently-transmitted objects:

 Gloas slot N
 ┌───────────────────────────────┐          ┌───────────────────────────────────┐
 │ BeaconBlock(N)                │          │ SignedExecutionPayloadEnvelope(N) │
 │  signed_execution_payload_bid │ commits  │  Payload{...}                     │
 │    .block_hash    = H_N       │───────►  │  ExecutionRequests                │
 │    .builder_index = B         │  to      │  signature (by B, or by proposer) │
 └───────────────────────────────┘          └───────────────────────────────────┘
         always exists                        separate object, MAY NEVER EXIST

That leaves two questions a block cannot answer about itself.

Should this slot have an envelope at all? The builder may have withheld the payload.
Only the child block's bid testifies:

   block N                   block N+1
 bid.block_hash = H_N      bid.parent_block_hash
                                  │
                                  ├─ == H_N  → N REVEALED  → expect an envelope
                                  └─ != H_N  → N WITHHELD  → expect nothing, ever

Whose key signs it? builder_index is a reusable registry slot, so the current occupant of index B could validly sign a historical envelope and a passing signature would prove nothing. The origin-state snapshot occupant is trusted only when deposit_epoch < envelope_epoch; otherwise the slot is unverifiable and is never even requested. Self-built payloads use the historical block's proposer key instead (validator indices are never reused, so a snapshot key is always safe there).

Verification is stateless, mirroring the existing block-backfill verifier: all key material comes from the checkpoint origin state, with a dedicated domainCache instance for DOMAIN_BEACON_BUILDER. Backfill only descends below the origin, so a pre-Gloas origin implies no expectations at all and a Gloas origin's registries are cumulative below it.

One batch, end to end

Blocks inside a batch are in ascending slot order, so the batch's highest block is the tail and its child sits in the batch above, which descent has already imported.

 origin
   │           ┌──── already imported ────┐
   ▼           ▼                          ▼
 ... [5456 ──────────── 5472) [5440 ──────────── 5456)        descent
                               ▲▲▲ current batch ▲▲▲   ───────────────►
                               5440 ..........  5455 = tail
                                                   │
                               child of tail = 5456 = status().LowRoot
 1. EXPECT    each block asks its child "was I revealed?"; the tail asks the already-imported 
                      batch above via the boundary-child lookup.  
                       Revealed → resolve the verification key → pending{...}
                                  │
 2. ASK       fixed 16-slot pages, one RPC per page to the batch's assigned peer.
              Re-asks ONLY still-pending slots. Empty/short responses are
              protocol-legal and never downscored. Per page: ≤3 attempts and an
              elapsed budget of 3 × RespTimeout, whichever expires first.
                                  │
 3. VERIFY    three gates over the whole page, cheap → expensive:
                a. bind  every retained field vs the block and its bid
                b. BLS   ONE aggregate verify for the page
                c. EL    ONE batched reconstruct (per-hash fallback on error),
                         then HTR(peer payload) == HTR(EL payload)
              Each gate runs only if the previous passed for every candidate.
                                  │
 4. HOLD      held[slot] = BlindEnvelope(env). Payload bytes are discarded here:
              the EL can always regenerate them, so only the signature, execution
              requests and roots are durable.
                                  │
 5. FINALIZE  at import, atomic with the blocks: classify the tail (its child is
              now certain), then persist every held envelope.

Gate ordering matters because the EL call is the only one that can fail for our reasons rather than the peer's, so it goes last and its failure never downscores. A page of 16 where one envelope binds to the wrong block root is abandoned at gate (a) with zero BLS and zero EL work; the other 15 stay pending for another peer.

Envelopes are held in memory and written only at import, so a batch is all-or-nothing and a re-run is idempotent — already-stored envelopes are skipped before any request.

Edge cases

Case How it's detected Outcome
withheld, mid-batch next block in the batch skips its payload expects nothing; no metric
withheld, at the tail boundary child skips it; decided at import, not at fetch expects nothing
empty (missed) slot no block ⇒ no expectation invisible
EL down or pruned bind ✓ BLS ✓ reconstruct ✗ el_failed skip; blocks still import
peer lies about bytes HTR(peer) ≠ HTR(EL) downscore, stay pending, retry
reused builder index deposit_epoch ≥ envelope_epoch, before any request sig_unverifiable; never fetched
peer drought page out of attempts or elapsed budget peer_exhausted
retention floor moves Env floor advances with the wall clock mid-run bottom batches expire, dropped

The key asymmetry: withheld and empty slots produce no metric; every genuine failure produces a typed skip. So backfill_envelope_slots_skipped having no series at all means "nothing failed", not "nothing known".

New metrics: backfill_envelopes_download_count, backfill_envelopes_verified_count, backfill_envelope_slots_skipped{reason}, backfill_envelope_conflicts_kept.

Deliberate limitations

  1. Skips are terminal. All retries live inside the batch — 3 page attempts with peer rotation, a 3 × RespTimeout budget, 3 local retries, per-hash EL fallback. Once the batch imports, the outcome is permanent and is counted but not recorded durably. On the devnet, a 60-second EL outage left 496 slots permanently without envelopes. Durable repair is follow-up work. Operators should watch backfill_envelope_slots_skipped; it is the only signal, and it is process-lifetime only.
  2. Serving is unchanged, so a gap truncates. The by-range handler walks the EL parent-hash chain and breaks at the first missing envelope, so a request spanning a gap returns short with no signal to the requester. This is pre-existing behaviour on develop, where a checkpoint-synced node holds zero envelopes below its origin — this PR strictly increases what is servable. An honest coverage gate that refuses rather than truncating, plus tying earliest_available_slot to EL availability, is separate work and is needed independently of this PR.
  3. Fresh checkpoint syncs only. A node that already finished block backfill never runs batches again, so it gets no envelopes without a resync.

Other notes for review

Acknowledgements

  • I have read CONTRIBUTING.md.
  • I have included a uniquely named changelog fragment file.
  • I have added a description with sufficient context for reviewers to understand this PR.
  • I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

  - Assign blocks and stage state together in handleBlocks, and nil-guard columnsNeeded: a setup error left b.blocks set with a nil columnSync, which transitionToNext dereferenced on the retry path.
  - Defer an EL cross-check failure on an unclassified batch tail to import-time classification, so a revealed tail reports el_failed
rather than peer_exhausted and a withheld tail stays silent.
  - Publish skip counters only after the batch status write succeeds, so a retried or expired batch cannot leave phantom or duplicate gap counts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant