diff --git a/docs/analytics/attester/README.md b/docs/analytics/attester/README.md new file mode 100644 index 0000000000..231066bb73 --- /dev/null +++ b/docs/analytics/attester/README.md @@ -0,0 +1,107 @@ +# Committee-duty QBFT decision timing vs attestation outcomes + +*June 2026 — SSV mainnet, 35-day window (2026-05-07 → 2026-06-10), network-wide. Rates computed on a stratified sample of 36 epochs (one random epoch per day): 216,243 committee duties covering 3.89M validator-attestations. Complete enumeration of cluster-failure events over the full window (7,732 events) provides the failure forensics.* + +This is the committee-duty (attestation) companion to the [proposer analysis](../proposer/README.md), asking the same question for attestations: how does the timing of the committee QBFT decision relate to whether the cluster's attestations land on chain? + +## TL;DR — the opposite of the proposer story + +For block proposals, decision time is destiny (100% missed past 4.0s into the slot). For committee duties, decision time is **almost irrelevant to inclusion**: + +- **0 of the 19,463 attestations from the 2,826 round-2-decided duties were missed** (CI95 ≤ 0.02%); likewise 0 of 128 from round-3+ decisions, and 0 of 355 from the 61 duties decided 7–8.5s into the slot. Round-1 decisions miss 0.0011%. Even the handful of duties decided past 10s were included, ~13 slots late. +- Of the 1,500 missed validator-attestations in the sample, 97% came from duties that never produced a decision: 86% where consensus messages were observed but a decide quorum never formed, and 11% where the exporter saw no consensus activity for the duty at all. The remaining 3% (41 misses, from just four duties) completed consensus early — between 2.3s and 5.1s — and failed after it. + +![Where missed attestations come from](miss-sources.png) + +What lateness does cost is **quality, not inclusion**: + +![Attestation quality vs QBFT decision time](quality-vs-decision-time.png) + +- **Inclusion in the first possible block** holds ≥97% through ~7.1s, then falls off sharply: roughly 70% at ~7.4s and ~56% past 7.5s (small samples there — see the error bars) — late attestations increasingly miss the slot+1 aggregate but still get included a slot or two later. +- **Head-vote correctness** dips to ~96.4% for round-1 decisions at 4.0–5.0s — the "no block seen by 4s" fallback path votes for a head that often turns out stale — and *recovers to 99.6%+ for round-2 decisions at 6–7s*. The round-change retry waits long enough for the late block to arrive: **round changes heal head votes**. +- For scale, decided duties split by round as **98.67% round 1** (212,457 duties), **1.31% round 2** (2,826 duties / 19,463 attestations), and **0.013% round 3+** (28 duties / 128 attestations) — everything right of ~6s on this chart describes a small minority of duties. + +Conditioning on the decided round separates the two effects — i.e. what the curves would look like if every duty decided in round K: + +![Round-1 decisions only](quality-vs-decision-time-round1.png) + +![Round-2 decisions only](quality-vs-decision-time-round2.png) + +Round-1 decisions carry the entire 4–5s head-vote dip (the fallback path) but never pay an inclusion-optimality price — they complete well before the danger zone. Round-2 decisions are the mirror image: head votes near-perfect throughout, and the only cost is first-block inclusion once the decision slips past ~7.2s. + +## Headline numbers (sampled epochs) + +| Question | Result | +|---|---| +| Committee QBFTs that touched round ≥2 | 4,207 of 216,076 traced duties = **1.95%** | +| Decided in round 2 | 2,826 of 215,311 decided duties = **1.31%** (round 3+: 28 duties = 0.013%) | +| Round-1 decision → attestation missed | 41 of 3,870,361 attestations = **0.0011%** | +| Round-2 decision → attestation missed | **0** of 19,463 attestations (CI95 ≤ 0.02%) | +| QBFT never decided → missed | 1,283 of 1,283 attestations = **100%** | +| Overall validator-attestation miss rate | 1,500 of 3,891,411 attestations = **0.039%** | + +The first two rows count committee duties (one QBFT each); the miss-rate rows count validator-attestations, since one duty carries the attestations of many validators. + +Decision-time distribution: **p50 = 2.42s, p90 = 3.84s, p99 = 6.07s, max = 23.7s** into the slot. The committee QBFT starts when the slot's beacon block arrives (typically ~2s into the slot), with a 4s fallback; round timeouts are slot-aligned — round 1 ends at 6s, round 2 at 8s (`protocol/v2/qbft/roundtimer/timer.go`). Post-consensus signing quorum trails the decision by only 16ms median (p99 ~200ms), so decision time ≈ submission time. + +### Why round changes are ~7× more common here than for proposals + +The [proposer analysis](../proposer/README.md) found only 0.29% of QBFTs touching round ≥2, vs 1.95% here. The gap is real, and mostly by design rather than a difference in cluster health: + +- **The two round timers measure different things.** Proposer round timeouts run relative to QBFT *instance start* ([#2429](https://github.com/ssvlabs/ssv/issues/2429)): the slow upstream step (relay payload fetch) delays the start without consuming round budget, and once started, a healthy cluster decides in ~0.2s against a 2s timer — it almost never fires. Committee round timeouts are *slot-aligned* while the instance start is gated on beacon-block arrival, so block lateness eats the round-1 budget directly. The data shows exactly that: **72% of round-2-touching committee duties never produced a round-1 proposal at all** — the leader was still waiting for the block when the 6s timer fired. +- **Late or contested blocks hit every cluster at once.** Half of all round-2 touches occur in slots where ≥5 clusters (up to 59) moved to round 2 simultaneously — a network-level event. Proposer duties run at most one per slot, so this amplification channel does not exist for them. +- **Different weighting of unhealthy clusters.** Committee statistics count every active cluster once per slot, so a few chronically-degraded clusters with small validator counts weigh heavily: the top-10 round-2-touching clusters (individual touch rates of 7–28%) contribute 29% of all touches, and the rate excluding them is still 1.43%. Proposer statistics weight clusters by validator count, diluting the same clusters. + +Given the quality results above — round-2 decisions land reliably and carry *better* head votes — the higher committee round-change rate is the slot-aligned timer doing its job when the chain's block is late, not a reliability deficit. + +## Full-window cluster-failure forensics + +All events where ≥5 validators of one cluster missed together, over the full 35 days (7,732 events; 121 excluded as suspect near a monitoring ingestion gap): + +| Category | n | share | +|---|---|---| +| QBFT never decided | 6,275 | 82.4% | +| Decided round 1 (on time), failed downstream | 1,000 | 13.1% | +| No consensus messages observed | 196 | 2.6% | +| Decided round 2 | 133 | 1.7% | +| Decided round 3+ | 7 | 0.1% | +| *Roll-up: decided ≥7.0s into the slot (across the three "decided" rows above)* | **15** | 0.2% | + +Of the 1,140 failure events that did reach a decision (rounds 1, 2, and 3+ combined), only 15 decided at 7.0s or later — 98.7% decided with comfortable margin before the aggregation deadline, so their failures happened after consensus, not because of its timing. + +Failures are heavily concentrated: **five chronically-failing clusters account for 82% of all cluster-failure events** (the worst single cluster alone is 39%). Attestation reliability on mainnet is an operator-health problem, not a consensus-timing problem. + +## Method + +- **Duties, rounds, timing**: exporter committee traces (`/v1/exporter/traces/committee`), one request per committee × epoch, reduced to compact per-duty rows at fetch time. Committee IDs are computed locally from operator sets (sha256 over sorted uint32-LE operator IDs, per `ssv-spec` `GetCommitteeID`). Trace coverage: 99.92%. The trace response's schedule provides assigned validators per slot, which also yields a per-epoch validator→cluster map. +- **Outcomes**: SSV Labs' internal duty monitor (e2m) per-validator attestation rows — included/missed, inclusion slot (delay), head-vote correctness — for the sampled epochs; `success=false` enumeration over the full window (335,077 misses = 0.041% of ~808M validator-duties) for the forensics. +- **False-negative guard**: the monitor's known whole-committee false-negative mode is triggered by block-ingestion gaps in its database. A full-window scan comparing its block table against the canonical chain (via a public beacon API) found **exactly one** gap slot in 35 days — the previously documented incident — affecting zero sampled misses; forensics events near it were excluded. +- **Decision time** = arrival of the earliest decided-quorum message at the exporter, minus slot start (P2P skew typically ≤300ms). + +### Caveats + +- Round-3+ is thin in the sample (28 duties / 128 attestations); conclusions there lean on the full-window forensics, which point the same way. +- Zero observed misses in round 2/3+ bounds the rate by sample size, not to literal zero. +- The 41 round-1-decided misses (the 0.0011%) come from four duties that decided on time and failed downstream; where ≥5 of a cluster's validators missed together, those duties also appear in the full-window forensics under "decided round 1 (on time), failed downstream" — so these sample misses overlap the forensics set rather than fall outside it. + +## Implications + +- **There is no attestation analog of the proposer 4s cliff** within the operationally reachable range — the aggregation pipeline tolerates decisions at least up to ~8.5s, with quality (inclusion optimality, hence rewards) degrading from ~7s. Consensus-timing tuning has little to gain for attestations. +- **Round changes are benign-to-positive for committee duties**: they cost inclusion nothing in this window, and they systematically improve head-vote correctness versus the 4s no-block fallback. +- **Reliability work should target cluster health**: the dominant failure mode is consensus never completing — concentrated in a handful of operators — followed by on-time decisions failing at submission. Monitoring and operator outreach beat protocol-timing changes here. + +## Reproduction + +The pipeline lives in the internal `ssv-scout` repository under `analysis/committee-qbft-timing/` (see its README for the full runbook). The window and sample are set in one place (`common.py`): full forensics window plus a deterministic stratified sample — one uniform-random epoch per 225-epoch (one-day) stratum, fixed seed — so a re-run of the same window reproduces the same sample. Seven resumable steps: + +1. **`fetch_committees.py`** — committees active in the window from the duty monitor's environment endpoint, with committee IDs computed locally from operator sets (derivation in Method above). +2. **`fetch_traces.py`** — `POST /v1/exporter/traces/committee`, one request per (committee × sampled epoch), reduced at fetch time to per-duty rows (rounds, `t_decided`, attester-signature quorum time, round-1 leader) — raw traces are not stored. The response's per-slot schedule yields both assigned-validator counts and a per-epoch validator→cluster map (the validator set drifts over weeks, hence per-epoch maps). ~17K requests ≈ 1h. +3. **`fetch_misses.py`** — all `success=false` attester duties over the full window (50-epoch chunks, the server's cap for attest ranges). +4. **`gap_scan.py`** — the false-negative guard: the monitor's block table (100-epoch chunks) vs `GET /eth/v1/beacon/headers/{slot}` on a public node for every monitor-absent slot, separating genuinely skipped slots from ingestion gaps. Misses with a gap inside `(slot, slot+32]` are excluded from rate numerators as suspects. +5. **`fetch_quality.py`** — per-validator outcome rows (inclusion slot, earliest possible inclusion slot, head vote) for each sampled epoch; one request per epoch with automatic pagination fallback. +6. **`fetch_failure_traces.py`** — full-window forensics: misses grouped into (committee, slot) events via the validator maps; one committee trace per event with ≥5 validators missing together. +7. **`analyze.py` + `render_charts.py`** — joins, rates, and the figures above. Rates are validator-attestation-weighted and computed strictly within the sampled epochs; the forensics set is full-window and non-rate. Wilson 95% CIs throughout; chart points are drawn at ≥25 included attestations per 0.25s bucket. + +Key definitions: `t_decided` = earliest decided-quorum arrival at the exporter minus slot start; signature quorum = (2f+1)-th distinct operator's attester partial-sig arrival (quorum = n − (n−1)//3); missed = monitor `InclusionSlot == 0`; inclusion delay = `InclusionSlot − slot`; optimal inclusion = `InclusionSlot == EarliestInclusionSlot`; head correctness defined only for included attestations. + +Constraints when re-running: exporter trace retention (~5 weeks at the time of writing) bounds the window — probe an old slot first; end the window ≥1–2h before the present so outcomes are settled; the exporter is a production observer, so keep fetch concurrency moderate. diff --git a/docs/analytics/attester/miss-sources.png b/docs/analytics/attester/miss-sources.png new file mode 100644 index 0000000000..31dbdec47d Binary files /dev/null and b/docs/analytics/attester/miss-sources.png differ diff --git a/docs/analytics/attester/quality-vs-decision-time-round1.png b/docs/analytics/attester/quality-vs-decision-time-round1.png new file mode 100644 index 0000000000..5522becf34 Binary files /dev/null and b/docs/analytics/attester/quality-vs-decision-time-round1.png differ diff --git a/docs/analytics/attester/quality-vs-decision-time-round2.png b/docs/analytics/attester/quality-vs-decision-time-round2.png new file mode 100644 index 0000000000..94dfd2c89e Binary files /dev/null and b/docs/analytics/attester/quality-vs-decision-time-round2.png differ diff --git a/docs/analytics/attester/quality-vs-decision-time.png b/docs/analytics/attester/quality-vs-decision-time.png new file mode 100644 index 0000000000..bb0d8d895d Binary files /dev/null and b/docs/analytics/attester/quality-vs-decision-time.png differ diff --git a/docs/analytics/proposer/README.md b/docs/analytics/proposer/README.md new file mode 100644 index 0000000000..220b429e58 --- /dev/null +++ b/docs/analytics/proposer/README.md @@ -0,0 +1,90 @@ +# Proposer QBFT decision timing vs block acceptance + +*June 2026 — SSV mainnet, 35 days of data (2026-05-07 → 2026-06-10), 44,838 proposer duties across the whole network.* + +This analysis measures the relationship between **when the proposer-duty QBFT instance decides** (seconds into the slot) and **whether the produced block is accepted by the Ethereum mainnet chain**. The motivating questions: + +1. What share of proposer-duty QBFT instances reach round 2? +2. What share decide in round 2? +3. How does the block miss rate change as the QBFT decision time moves toward the 4s into-the-slot mark (the attestation deadline)? + +## TL;DR + +Round 2 is rare (~0.3% of proposer QBFTs), but deciding there is dangerous: **11.4%** of round-2 decisions resulted in a missed block vs **0.049%** for round-1 decisions — a ~230× relative risk. The dominant variable is not the round number, though; it is **wall-clock decision time relative to the 4s attestation deadline**: + +![Miss rate vs QBFT decision time](miss-rate-vs-decision-time.png) + +- decided ≤ 3.0s into the slot → ~0.05% missed (background rate, unrelated to timing) +- decided 3.0–3.5s → ~1.9% missed +- decided 3.5–4.0s → 30% missed +- decided ≥ 4.0s → **100% missed (8/8)** — by then attesters have voted for the parent block, so the late block loses the fork-choice race essentially without exception + +A round-2 decision that still lands by ~3.5s is almost always fine (2/96 missed). Past the 4s mark the block is effectively guaranteed to be rejected regardless of round. + +## Headline numbers + +| Question | Result | +|---|---| +| Proposer QBFTs that touched round ≥2 | 129 of 44,808 traced duties = **0.29%** — 115 of those (0.26%) progressed to a round-2 proposal or decision; the remaining 14 only had round-change messages observed | +| Decided in round 2 | 114 of 44,803 decided duties = **0.25%** (no round-3+ decisions occurred in 35 days) | +| Round-2 decision → block accepted | 101 / 114 = **88.6%** (missed 13 = 11.4%, CI95 [6.8%, 18.5%]) | +| Round-1 decision → block accepted | 44,667 / 44,689 = **99.951%** (missed 22 = 0.049%, CI95 [0.03%, 0.07%]) | +| Overall miss rate (canonical-chain-verified) | 41 / 44,838 = **0.091%** | + +Decision-time distribution across all decided duties: p50 = 1.40s, p90 = 2.07s, p99 = 2.25s, p99.9 = 3.36s, max = 4.94s into the slot. Round-2 decisions cluster at 3.0–3.5s, which follows from the proposer round-1 timeout being 2s **from QBFT instance start** rather than from slot start ([#2429](https://github.com/ssvlabs/ssv/issues/2429)) — the instance itself starts only after RANDAO pre-consensus and payload fetch (~1.2s median). + +### Within round-2 decisions only + +| Decided at | n | missed | miss rate | +|---|---|---|---| +| < 3.0s | 7 | 0 | 0% | +| 3.0–3.5s | 89 | 2 | 2.25% | +| 3.5–4.0s | 10 | 3 | 30% | +| ≥ 4.0s | 8 | 8 | 100% | + +Round 2 per se is survivable — *late* round 2 is not. + +## What the misses actually are (41 total) + +| Category | n | Notes | +|---|---|---| +| Decided round 2, late | 13 | Decided 3.37–4.94s into the slot; these are the timing-driven misses | +| Decided round 1, on time | 22 | Decided 1.10–2.44s; 21 of 22 also reached post-consensus quorum within ~2.5s, i.e. the block was signed and submitted promptly but never became canonical — relay / payload / submission / orphaning failures downstream of consensus | +| QBFT never decided | 5 | Consensus failure, no block produced at all | +| No trace observed | 1 | | + +Seven committees account for 16 of the 41 misses — repeat offenders point at operator-specific infrastructure issues rather than random timing bad luck. Notably, the on-time round-1 misses are a *different failure class* from the timing cliff: deciding early does not protect against a failing relay or beacon submission path. + +## Method + +- **Duty enumeration and acceptance status**: SSV Labs' internal duty monitor (e2m), which tracks every assigned duty network-wide — including missed proposals, which the canonical chain alone cannot attribute to a proposer. +- **QBFT rounds and timing**: the exporter's duty traces (`/v1/exporter/traces/validator`). The exporter passively observes all P2P traffic, so coverage spans all operators, not only SSV Labs nodes. Trace coverage was 44,808 / 44,838 = 99.93%. +- **Acceptance ground truth**: every miss reported by the monitor was cross-checked against the canonical chain via a public beacon API (`/eth/v1/beacon/headers/{slot}`). One false miss was found (a monitoring ingestion gap) and corrected. "Missed" therefore means *no canonical block at the duty slot* — orphaned blocks count as missed. +- **Decision time** (`t_decided`) = arrival time of the earliest decided-quorum message at the exporter, minus slot start. Exporter timestamps are arrival times, so P2P propagation (typically well under ~300ms) shifts measurements slightly right; the same curve computed on post-consensus quorum time (the closest observable proxy to block broadcast) is identical in shape, shifted ~40ms. + +### Caveats + +- The round-1 vs round-2 comparison is correlational: whatever broke round 1 (a slow relay, a slow leader) may independently raise miss odds. The time-bucketed curves are the cleaner read, and both show the deadline effect dominating. +- The tail buckets are small (n=10 at 3.5–4.0s, n=8 at ≥4.0s), so the exact rates there carry wide confidence intervals — but the 0% → 30% → 100% gradient is unambiguous. + +## Implications + +- **`ProposerDelay` tuning** (see [MEV considerations](../../MEV_CONSIDERATIONS.md)): the delay inserts itself before QBFT starts, shifting the whole decision-time distribution right. With the current p50 ≈ 1.4s and p99 ≈ 2.25s, the data supports the existing guidance that ~1.2s is the upper bound of reasonable on mainnet: it keeps p99 decisions under the ~3s knee where the miss rate is still at background level. Values pushing decisions past ~3s buy MEV at a steep and sharply nonlinear cost. +- **Round changes are an early-warning signal, not a death sentence**: a round-2 decision still lands reliably if it completes by ~3.5s. Efforts to shorten the path to a round-2 decision (leader fail-over speed, payload re-fetch) have measurable value; efforts to salvage duties past 4s do not. + +## Reproduction + +The pipeline lives in the internal `ssv-scout` repository under `analysis/proposer-qbft-timing/` (see its README for the full runbook). Four steps, all parameterized by an epoch window and resumable after interruption: + +1. **`fetch_duties.py`** — enumerate proposer duties with outcomes from the duty monitor: `GET /api/duties?types=propose&with_committee`, chunked 200 epochs/request (the server caps propose-only ranges at 14 days/request). Yields `{slot, validator, committee operator IDs, success}` per duty, including missed ones. +2. **`fetch_traces.py`** — exporter QBFT traces: `POST /v1/exporter/traces/validator` with `roles=["PROPOSER"]`, one request per 5-epoch batch carrying that batch's proposer indices (~15 min for 35 days). +3. **`analyze.py`** — joins and metrics. Every monitor-reported miss is re-verified against the canonical chain (`GET /eth/v1/beacon/headers/{slot}` on a public node; 404 = no canonical block = real miss), which also neutralizes the monitor's known block-ingestion false-negative mode. Time buckets `[0, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 6, 8, 12]`s with Wilson 95% CIs. +4. **`render_chart.py`** — the figure above, regenerated from the analysis output. + +Key definitions: `t_decided` = earliest decided-quorum message arrival at the exporter minus slot start (genesis 1606824023 + slot × 12s); decided round = lowest round with a decided aggregate; post-consensus quorum = arrival of the (2f+1)-th distinct operator's partial signature, with quorum = n − (n−1)//3 for an n-operator committee. + +Constraints to respect when re-running: exporter trace retention bounds the window (~5 weeks at the time of writing — probe an old slot first); end the window an hour or more before the present so duty outcomes are settled. + +## See also + +[Committee-duty QBFT decision timing vs attestation outcomes](../attester/README.md) — the attestation companion to this analysis, with the opposite conclusion: attestation inclusion is insensitive to decision time, and failures are binary and cluster-concentrated. diff --git a/docs/analytics/proposer/miss-rate-vs-decision-time.png b/docs/analytics/proposer/miss-rate-vs-decision-time.png new file mode 100644 index 0000000000..e34513b590 Binary files /dev/null and b/docs/analytics/proposer/miss-rate-vs-decision-time.png differ