ePBS devnet e2e — glamsterdam-devnet (SIP-94, #2901)

[iurii-ssv]

Tracking issue for the public-devnet e2e validation of the node-side ePBS implementation (#2901, SIP ssvlabs/SIPs#94) — the external-interop bar: a real 512-member PTC and multi-client BNs we don't control.

This is one of two independent validation initiatives. The other — the hermetic local Gloas net (local_testnet_gloas, we own the whole validator set so PTC hits every slot) — is done and green (automated Loki-based (ptc) suite, tracked in ssvlabs/aetheria#125); keep re-running it on branch tips as the fast regression signal. Mock-green ≠ local-green ≠ interop-green: this issue owns the interop half.

Status

• ✅ Devnet picked & verified live (2026-06-30): glamsterdam-devnet-6. ⚠️ Devnets reset frequently and the README status table lags — probe https://glamsterdam-devnet-N.ethpandaops.io/ (→ 200) to find the live one and re-check before every run (it already moved 5→6; devnet-0..4 are torn down).
• ✅ networkconfig stub committed (glamsterdam-devnet in ePBS (EIP-7732 / Gloas) — SIP-94 #2901) — header pinned to devnet-6 (chain 7052886157, genesis 1782386940), TotalEthereumValidators: 3909 from the verified live count. RegistryContractAddr/RegistrySyncOffset/Bootnodes are TODO(e2e) (steps 3–5 below); the node refuses to start on the zero contract address until they're filled.
• ✅ A (proposer) run on the local Gloas net already validated the §4 pipeline end-to-end (duty dispatched → block produced → signed → landed on-chain) and shook out the fixes now on the branch (graffiti padding, local Gloas block signing + slashing protection, the §2 attestation-index fetch). The open devnet-specific items are V1–V11 below.

Devnet-6 facts (verified 2026-06-30 — re-verify each run)

• Open read endpoints (no auth): CL https://beacon.glamsterdam-devnet-6.ethpandaops.io (Prysm), EL https://rpc.glamsterdam-devnet-6.ethpandaops.io (chainId 0x1a462808d).
• Per-client BNs bn-<cl>-<el>-1.srv.glamsterdam-devnet-6.ethpandaops.io (+ EL rpc-…) need basic-auth (401 without; creds from the ethpandaops devnet spec — or run your own BN). Lighthouse/Lodestar preferred for the full PTC endpoint set (both confirmed to serve them; verify the specific combo used).
• Spec matches the PR's wire values: GLOAS_FORK_EPOCH: 30, PTC_SIZE: 512, PAYLOAD_ATTESTATION_DUE_BPS: 7500, MAX_PAYLOAD_ATTESTATIONS: 4, domains PTC_ATTESTER: 0x0c / BEACON_BUILDER: 0x0b / PROPOSER_PREFERENCES: 0x0d. POST /eth/v1/validator/duties/ptc/{epoch} → 200.
• ~3909 active validators ⇒ PTC picks 512/slot ⇒ each validator draws PTC duty roughly every 8 slots — frequent even with a handful of SSV validators.
• Config sources: glamsterdam-devnets/network-configs/devnet-6/ + config.glamsterdam-devnet-6.ethpandaops.io/api/v1/nodes/inventory.

Checklist

• 1 · Pick & verify the live devnet (sanity: GET /eth/v1/config/spec shows GLOAS_FORK_EPOCH in the past; a PTC endpoint answers)
• 2 · networkconfig stub committed & refreshed to devnet-6
• 3 · Deploy the SSV contracts on the devnet EL + register 4 operators → record RegistryContractAddr + deployment block
• 4 · Validators — deposit (devnet deposit contract 0x00000000219ab540356cBB839Cbe05303d7705Fa) → await activation → split keys into 4 shares → register validators+shares on the SSV contract
• 5 · Fill the stub TODOs — RegistryContractAddr + RegistrySyncOffset (from 3), Bootnodes (operator ENRs from 6)
• 6 · Run 4 operators (env-var config below); harvest their ENRs into the stub Bootnodes (or pin NETWORK_PRIVATE_KEY per node for stable ENRs)
• 7 · Verify PTC across all 4 operators via the greppable logs (signal table below); cross-check the BN payload_attestations pool
• 8 · Work the devnet-verify items V1–V11 below

Known risks: devnet resets/instability; validator activation latency; SSV contract deploy on a non-standard chain; per-client beacon-API PTC completeness.

Operator run config (env vars; no config file needed)

The config is cleanenv-based — a node started without --config reads purely from env.

Env var	Value / note
NETWORK	glamsterdam-devnet (beacon genesis/fork schedule incl. GLOAS_FORK_EPOCH come from the BN at runtime)
BEACON_NODE_ADDR	required — CL HTTP URL(s); Lighthouse/Lodestar preferred; ;-separated for multiple
ETH_1_ADDR	required — EL WS URL(s); ;-separated
OPERATOR_KEY	operator private key (or PRIVATE_KEY_FILE). Local signing only — remote-signer (Web3Signer) can't sign the new Gloas duties (bounded by f), don't use it for this test
DB_PATH	per-operator DB dir
LOG_LEVEL	debug — the greppable ePBS logs are the pass/fail signal
NETWORK_PRIVATE_KEY	optional stable P2P identity (for stable Bootnodes ENRs)
METRICS_API_PORT / EVENTS_PATH	optional

Pre-flight (T-minus a few epochs)

1. Every BN reports the same, non-far-future GLOAS_FORK_EPOCH — the node cannot self-check BN disagreement (it logs the resolved epoch at startup: Gloas (ePBS) fork scheduled); if a BN doesn't schedule it, the node silently stays pre-Gloas (no error, nothing ePBS fires).
2. Every BN serves the 8 Gloas routes (block produce/publish, envelope get/publish, PTC duties/data/submit, proposer dependent-root).
3. ProposerDelayEPBS ≤ 1s (else boot-abort).
4. The validator set actually draws proposer + PTC selections inside the observation window.
5. Capture a pre-fork baseline (thirds timing, zero ePBS roles) for a clean delta.

Pass/fail log signals (per SIP section)

Section	Healthy	Red flag
§1 timing	attestation submit rate holds across the /3→/4 retiming	⚠️ late duty execution bursts
§2 attestation	built gloas attestation vote with payload_status_index	value-check rejects (attestation data index out of range, 120B-vs-112B decode errors)
§3 PTC	fetched PTC duties → ✔️ successfully submitted payload attestation; occasional abstaining…no beacon block (missed slots) is expected	constant abstains; failed to fetch PTC duties / PTC attestation failed…
§4 proposer	🧊 got gloas beacon block proposal → ✅ successfully submitted block proposal; decided gloas block build source	could not submit gloas beacon block
§5 prefs	emitted proposer preferences duties → built proposer preferences (with dependent_root/fee_recipient/target_gas_limit) → ✔️ successfully submitted proposer preferences	could not submit proposer preferences / proposer preferences failed: could not build
§6 envelope	built execution payload envelope → ✅ published execution payload envelope (builder); did not build the decided envelope, skipping publication (others)	could not submit execution payload envelope; no decided block root recorded
cross-role	duty concluded (Debug, succeeded/not_required)	⚠️ duty failed / ⚠️ duty did not complete before slot end (likely stuck)

Known boundary quirks (expected, not bugs)

• The first Gloas slot may return CL 500 BeaconStateError(IncorrectStateVariant) (state not yet in the Gloas variant) — the per-slot duty refetch absorbs it the next slot; no code change needed.
• A missed proposal slot → CL 404 No block on the PTC data fetch → the runner fails safe (abstain); with per-epoch PTC assignment the duty lands within an epoch or two.

Devnet-verify items

• V1 — §4 submit on devnet-6 (was: invalid SSZ). An earlier run produced+signed but devnet-6 rejected the block encoding; root cause identified as the Gloas ExecutionRequests layout (the EIP-8282 five-list variant vs Electra's three) and fixed on the branch. Re-verify the §4 submit lands on devnet-6, and while there confirm the all-operators duplicate submit is deduped by root by the BN (the pre-Gloas redundancy assumption). → hermetic Lodestar answered the verify; node fix tracked in ePBS §4: handle BLOCK_ERROR_ALREADY_KNOWN on the redundant all-operator block submit #2922.
• V2 — §6 envelope produce 404. GET /eth/v1/validator/execution_payload_envelopes/{slot}/{root} 404'd even where the §4 block landed. Needs devnet correlation: the exact path the BN echoed, the response body, and whether the §4 block was imported at that moment (timing vs endpoint-unsupported). Can't be resolved node-side blind.
• V3 — §2 Fulu-tagged attestation acceptance (HIGH if it fails). Gloas reuses the Electra/Fulu attestation wire format, so the node submits with Eth-Consensus-Version: fulu on Gloas slots (BeaconForkAtEpoch caps at Fulu). Confirm a Gloas BN accepts it; if rejected, every attestation fails → extend BeaconForkAtEpoch with DataVersionGloas (the existing TODO(gloas) in networkconfig/beacon.go).
• V4 — §2 BN-supplied payload-status index. The Gloas vote takes attData.Index from the BN's attestation_data response. Confirm the response index is the payload-status value (0=EMPTY / 1=FULL), not an echo of the requested committee index and not a constant 0 — otherwise every vote carries EMPTY and FULL payloads are never attested.
• V5 — §2 aggregate index across BNs. computeAttestationDataRoot (aggregation path) re-fetches attestation data from the aggregator's own BN and keeps that BN's payload-status index, not the QBFT-decided one the committee signed. Confirm the aggregate fetch matches the signed index across BNs (low risk — payload status should be settled by aggregation time — but a cross-BN mismatch silently misses the aggregate).
• V6 — §4/§6 stateless …Contents body. SSV publishes the §6 envelope as the blinded/stateful body (the producing BN un-blinds from cache). Confirm no devnet BN requires the payload-stateless SignedExecutionPayloadEnvelopeContents (envelope + blobs + KZG); wire it only if one does (that also un-defers the §4 blob plumbing). → hermetic Lodestar confirmed a BN requires it; node fix tracked in ePBS §6: publish the stateless SignedExecutionPayloadEnvelopeContents envelope body #2921.
• V7 — §5 dependent_root tracking through E−1. Once the SIP-94 §5 re-emission rule lands (discussion), confirm re-emission tracks the dependent_root as it settles through the pre-epoch, and evaluate the optional "don't publish in the first jittery slots of E−1" heuristic.
• V8 — QuickTimeout under Gloas. The 2s per-round budget is deliberately not retimed, making the Gloas proposer effectively round-1-must-succeed. Collect real round-trip data and decide whether to shrink the budget to restore a round-2 fallback inside the ~quarter-slot deadline.
• V9 — telemetry gauging. Gauge PTC non-convergence frequency (currently surfaces only as the generic "likely stuck" at slot end) and add a distinct outcome/log if warranted; collect real local-build % and reconstruction-miss rates to revisit §6 priority and any consensus tuning.
• V10 — consensus-specs pin drift watch. Re-verify the node's Gloas SSZ layouts against the live devnet's spec at every devnet reset (V1 is this risk materialized once already; the spec is pre-final).
• V11 — Web3Signer fork_info on Gloas (cross-system). The remote path now sends the correct Gloas fork in fork_info; Web3Signer's compute_domain is generic over the version bytes, so this should suffice without Web3Signer Gloas support — confirm against a live Web3Signer (the only non-in-repo fact in that fix).

local_testnet residuals (tracked in ssvlabs/aetheria#125 — duplicated here so they don't slip)

• Flip monitor.enabled: true in ssv-mini params-gloas.yaml once the ethereum2-monitor Gloas block-decode (ssvlabs/ethereum2-monitor#504, merged) is in the built monitor image — deliberately deferred out of ssv-mini#34 so it merged standalone; this is the easy-to-forget step that turns E2M attestation validation back on for the Gloas chain.
• ssvlabs/aetheria#128 — kurtosis-native Loki, so the (ptc) suites run on local_testnet too.

Post-run bookkeeping

• Record RegistryContractAddr + RegistrySyncOffset + Bootnodes in networkconfig/glamsterdam-devnet.go; refresh TotalEthereumValidators and the header whenever the devnet resets or moves (probe for the live N).
• Fold confirmed answers from V1–V11 back into ePBS (EIP-7732 / Gloas) — SIP-94 #2901 (or follow-up PRs) and check them off here.

[iurii-ssv]

Hermetic validation against Lodestar (the live devnet-6 CL) — answers V1, V2, V6

Ran the hermetic (proposer) suite (aetheria local_testnet_gloas, ssvlabs/aetheria#125) with the CL swapped from lighthouse to Lodestar v1.43.0 — the same client the live devnet-6 runs — so this is a close preview of the §4/§6 wire interop, before the full live bring-up.

V2 — §6 envelope produce: RESOLVED (endpoint-unsupported, not timing). On lighthouse v8.2.0 the produce GET /eth/v1/validator/execution_payload_envelopes/{slot}/{root} returns a generic route-not-found 404 (byte-identical to a bogus path) even for a canonical block — the endpoint simply isn't in that build. On Lodestar it works: every operator's produce call resolves and builds the envelope (built execution payload envelope on all 4 nodes). So the earlier 404 was endpoint-unsupported (a lighthouse build gap), not a §4-import timing issue.

V6 — stateless …Contents body: a devnet BN requires it. With produce working, §6 reaches the publish, and Lodestar rejects it:
POST /eth/v1/beacon/execution_payload_envelopes → 400 {"code":400,"message":"Offset out of bounds 2425415465 > 128"}
The node sends the blinded SignedBlindedExecutionPayloadEnvelope (~128 B); Lodestar SSZ-decodes it as the full envelope → garbage offset. So Lodestar requires the payload-stateless SignedExecutionPayloadEnvelopeContents body — V6's "wire it only if a devnet BN requires it" is answered: one does. (This also un-defers the §4 blob plumbing.)

V1 — §4 submit + all-operators dedup: submit lands, dedup FAILS. The §4 block lands on Lodestar (the EIP-8282 five-list ExecutionRequests fix holds — an operator logs ✅ successfully submitted block proposal). But the BN does not dedupe the redundant all-operators submits:
POST /eth/v2/beacon/blocks → 500 {"code":500,"message":"BLOCK_ERROR_ALREADY_KNOWN"} (the non-leader operators).
So the pre-Gloas "BN dedupes by root" redundancy assumption does not hold on Lodestar — the node should treat BLOCK_ERROR_ALREADY_KNOWN as success (or submit leader-only).

Caveats: hermetic net (Gloas fork @ epoch 2, small validator set); the run also hit the ~2-epoch metadata-load lag on §4 (a known test-timing artifact, not a node fault). But the §4/§6 wire behaviors above are CL-driven and are what the live devnet's Lodestar will do.

Net: V2 resolved (good); V6 answered — node must wire the stateless body; V1 dedup answered — node must handle already known. Two new node tasks, both surfaced hermetically.

[iurii-ssv]

Closing — the public glamsterdam-devnet path this issue tracked is superseded, so keeping it open would only track dead devnet mechanics (contract deploy on devnet-N, validator seeding, ENR harvesting, the devnet-reset watch).

What shipped instead:

• The hermetic half is done and green — local_testnet_gloas with the automated Loki suites (ptc) / (proposer) / (transition) (ssvlabs/aetheria#125, ssvlabs/aetheria#128, ssvlabs/aetheria#141). The §1–§6 pass/fail log-signal table from this issue now lives in code there.
• Every node fix the hermetic Lodestar run surfaced is merged: ePBS §6: publish the stateless SignedExecutionPayloadEnvelopeContents envelope body #2921 (§6 stateless body), ePBS §4: handle BLOCK_ERROR_ALREADY_KNOWN on the redundant all-operator block submit #2922 (§4 ALREADY_KNOWN), ePBS §6: treat EXECUTION_PAYLOAD_ENVELOPE_ERROR_ALREADY_KNOWN as success on the envelope publish (§6 analog of #2922) #2923 (§6 ALREADY_KNOWN), ePBS §5: indices-change re-emits all lookahead proposer-preferences, not just new validators #2934 (§5 re-emission) — resolving V1 / V2 / V6 / V7.

Live confidence is now split across the successors (nothing lost):

• Lightweight manual Sepolia log-check across the Glamsterdam fork → ePBS/Gloas: manual Loki log-check on live Sepolia SSV clusters (pre/at/post-fork) #2953
• Remaining node-design verification items V5 / V8 / V9 / V11 → ePBS/Gloas: node-side interop verification items (V5/V8/V9/V11) carried over from #2920 #2954
• Full Aetheria harness on Hoodi post-fork → ssvlabs/aetheria#139
• Automated fork-transition coverage → ssvlabs/aetheria#141

Follow-up: the networkconfig/glamsterdam-devnet stub committed for this effort can be cleaned up if the devnet target is truly retired.