cl: implement Gloas devnet 8

[domiwei]

Summary

• align Gloas state upgrades, transition processing, progressive metadata, and consensus spectests with consensus-specs v1.7.0-alpha.14
• implement execution-payload-bid and proposer-preferences gossip validation, timing, compatibility, first-valid retention, and fork-boundary behavior
• fix Gloas fork-choice payload availability, execution-status transitions, head selection, checkpoint recovery, persistence, and pruning lifecycle
• make Gloas block production use a coherent fork-choice head/status snapshot and safe REST/P2P first-valid handoff
• implement the devnet-8 CL-facing Builder/Beacon API flow: dynamic builder transport and configuration, proposer preferences, execution payload bids/envelopes, strict Gloas publish parsing and validation, state builders, and Gloas event streams
• add focused reverse, boundary, overflow, cache-eviction, persistence-failure, cancellation, and concurrency regressions

Validation

• make erigon integration
• go test -tags=spectest ./cl/spectest -count=1
• go test ./cl/beacon/handler ./cl/beacon/builder/... ./cl/beacon/beaconevents ./cl/phase1/network/services ./cl/phase1/forkchoice/... ./cl/phase1/stages ./common/event ./cmd/caplin/caplin1 -count=1
• targeted race tests for fork choice, fork graph, gossip reservation handoff, builder transport, events, pruning, and payload-status lifecycle
• git diff --check
• local lint reports zero issues for this PR worktree; the repository-wide local harness also scans an unrelated sibling worktree with pre-existing errcheck findings
• live Glamsterdam devnet 8 checkpoint sync and head following with an external Geth EL: distance 0, non-optimistic, Gloas FCU and payload imports advancing
• no test skips added
• multi-agent spec, standards, and adversarial review converged CLEAN on c24c661c92

Scope and explicit policy follow-ups

This PR is limited to the devnet 8 consensus-layer and CL-facing API implementation. Remaining execution-layer work, including full EIP-8070 engine_getBlobsV4 / FCU custody-column wiring and native Erigon EL devnet 8 execution issues, is intentionally out of scope.

The following external-builder behaviors remain disabled pending explicit operator/security-policy approval:

• forwarding a signed Gloas block to an arbitrary request-supplied Eth-Builder-Url
• allowing private/loopback builder targets in production (needed by some Docker/Kurtosis layouts)
• falling back from a failed local EL build to an external builder bid

Spec: https://github.com/ethereum/consensus-specs/releases/tag/v1.7.0-alpha.14
Devnet: https://notes.ethereum.org/@ethpandaops/glamsterdam-devnet-8

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[yperbasis]

CI is red

[Copilot]

Pull request overview

Copilot reviewed 109 out of 111 changed files in this pull request and generated 5 comments.

Files not reviewed (2)

• cl/abstract/mock_services/beacon_state_mock.go: Generated file
• cl/beacon/builder/mock_services/builder_client_mock.go: Generated file

Suppressed comments (1)

cl/cltypes/gloas_api.go:88

• These three fields are dynamic, so the encoded container also contains three 4-byte offsets. The current result under-reports the SSZ size by 12 bytes.

[yperbasis]

Requesting changes for the following blockers:

1. Persisted envelopes bypass validation for later submissions. cl/phase1/forkchoice/on_execution_payload.go returns success when HasEnvelope(root) is true, before bid, BLS, DA, or EL checks. The P2P service and REST consensus path interpret nil as successful validation and may propagate the submitted object. Return an explicit duplicate/ignore result and ensure callers never emit or propagate an unvalidated duplicate.

2. The self-build path persists a SignedExecutionPayloadEnvelope with InfiniteSignature. This marks the payload present and prevents the later VC-signed envelope from replacing it. The invalid placeholder can also be served through REST and req/resp. Keep local payload data separate and wait for the proposer-signed envelope before setting HasEnvelope.

3. storeBlockAndBlobs calls OnBlock before the database write, while OnBlock emits the block SSE event immediately. This matches the failing caplin-minimal serial job: assertoor receives a block event and immediately gets 404 for that root. Emit the public block-import event only after persistence commits.

4. caplin.builder.allow-private-urls also applies to a raw request-supplied Eth-Builder-Url. ClaimOrAdd does not bind that route to an authenticated builder configuration, so enabling the flag permits request headers to target private or loopback services. Restrict private targets to trusted configured builder entries.

5. cl/beacon/handler/epbs.go emitFullHeadV2 only rechecks root and slot. Gloas EMPTY and FULL nodes share the same root, and optimism can also change. Mirror the payload-status and optimism checks already used by execution_payload_service.go.

6. The SSZ payload-attestation endpoint rejects a zero-byte body, although that is the valid encoding of an empty List[PayloadAttestationMessage, PTC_SIZE] and JSON [] succeeds.

There is also a scope mismatch: the PR description says arbitrary Eth-Builder-Url forwarding, private targets, and external-bid fallback after local EL failure remain disabled pending approval, but all three paths are enabled in the current code. Please gate them or record the approval and update the stated scope. The private-target trust-boundary issue still needs correction either way.

The focused CL test set and git diff --check pass. Current CI remains red from the Caplin minimal ordering race above and a likely unrelated Sonar failure in execution/commitment.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[domiwei]

Addressed the requested blockers in 279c1c23c2 (with the block/header visibility seam included in the preceding commits):

1. Persisted execution-payload envelopes no longer bypass validation or propagate caller-derived duplicates. Duplicate handling is explicit, persisted identity is checked, forged/mismatched submissions are rejected, and a retry after transient gossip failure can finish dissemination without duplicating import events.
2. The self-build path no longer constructs or persists an InfiniteSignature envelope. The unsigned payload remains in the bounded self-build cache until the validator fetches, signs, and publishes the canonical envelope.
3. The block-event-to-REST visibility race is closed for the exact block and header routes: durable storage remains the fast path, with a validated forkchoice fallback during the pre-persistence window. Unknown, unvalidated, mismatched, and pruned roots remain 404; optimistic/canonical metadata is sampled after visibility.
4. Request-supplied/unbound Eth-Builder-Url forwarding is strict-public only. Private targets require a locally configured trusted route. Trusted and public calls now use separate pinned transports, so keep-alive reuse cannot cross the policy boundary; redirects and special-purpose/private address ranges are rejected on the public path.
5. REST and service head_v2 emission now revalidate the coherent root/slot/payload-status/optimistic tuple, covering same-root EMPTY/FULL transitions.
6. Zero-byte SSZ is accepted as the canonical empty payload-attestation list.

The scope text has also been reconciled with the approved implementation: multi-BN forwarding is supported through the echoed URL, but remains strict-public unless the operator explicitly enables private configured builders; direct builder fallback uses the frozen requested parent and keeps gossip-only policy separate.

Additional review-driven hardening in the same commit covers payload-attestation REST/P2P first-valid and publish-retry ownership, SSE idempotency, forward-sync envelope reconciliation, finalized-checkpoint ancestry, and public-builder DNS/connection-pool boundaries.

Validation:

• go test ./cl/beacon/handler -count=1
• go test ./cl/phase1/network/services ./cl/beacon/builder ./cl/phase1/stages ./cmd/caplin/caplin1 -count=1
• targeted coordinator and builder transport tests under -race
• make erigon integration
• git diff --check
• two lint runs report 0 issues for this worktree (the aggregate Make target subsequently scans an unrelated sibling worktree and reports its pre-existing errcheck findings)

All inline review threads are resolved. Re-review requested.