eth: verify registry-event completeness after optimistic historical sync (#2990)

[iurii-ssv]

Closes #2990.

TL;DR

• eth_getLogs is served from the EL's derived, non-consensus log index, which can silently return fewer logs than a block holds; historical sync trusted it, so registry events could be lost permanently and invisibly (Registry events silently and permanently lost: historical log sync skips bloom verification but advances lastProcessedBlock anyway #2990).
• Now: restart catch-ups (up to ~a week of blocks) are verified inline at boot; large cold syncs run optimistically and are verified in the background against per-block digests, with disagreements settled by receipts — the index-independent source of truth.
• A receipts-confirmed miss triggers an automatic drop-and-resync on the next start — resumable if interrupted, rate-limited against loops.
• No config changes, no DB migration. Ops: ssv.event_syncer.verify.misses and .parked should stay 0; a confirmed miss shows up as a loud warning + self-restart into a rebuild.

Problem

Historical registry-event sync reads logs via eth_getLogs. The log index serving it can return fewer logs than a block actually holds — while still building (snap sync), pruned, mid-rebuild, or buggy (e.g. the geth log-index bug) — with no error. Because the sync advanced its last-processed marker regardless, events dropped this way were silently and permanently lost.

A field analysis of an affected node (full-keyspace diff of the drifted DB against a canonical resync) confirmed the blast radius: a dropped window loses whatever registry events it contained — operator registrations, fee-recipient updates, owner-nonce bumps, cluster liquidations — not just the OperatorAdded events the issue was first noticed by.

Verifying every block inline (header bloom + receipts) is too costly to run across a full cold sync on the startup critical path (see Costs), so the work is split into an optimistic hot path plus background verification with an automatic, guarded repair.

How it works

1. Optimistic hot path — the historical sync goes optimistic only when it's large: a cold sync from the registry offset, or a very prolonged catch-up. A normal restart catch-up (up to maxInlineVerifyCatchUp = 50k blocks ≈ a week) is instead bloom-checked inline, so the node starts with guaranteed-complete state and no window. On the optimistic path, eth_getLogs is used plain (fast), and for each non-empty block the handler records a digest of the logs it received — sha256 over each log's (TxIndex, Index) — and extends an "unverified range", in the same transaction that advances the marker, so a hard crash can't leave a processed block outside the range the verifier will later check.

2. Background verifier — off the startup critical path (concurrent with ongoing sync), each journalled range is re-fetched (VerifyLogs) and compared, block by block, to the recorded digests. On a disagreement it resolves the block against receipts and flags a resync only when receipts confirm the sync genuinely missed events (eth_getLogs can only ever return a subset, so a recorded digest short of receipts is provably a miss). When the verify-time fetch merely blipped (or a different EL under MultiClient returned less), receipts vindicate the recorded digest and the block is retired clean. If there is no authoritative source for the block, the range is parked — left pending and visible on pending_ranges, re-checked on the next start — rather than resynced on unauthoritative evidence or retired as verified. "No authoritative source" covers an EL without eth_getBlockReceipts (under MultiClient the receipts resolution first fails over across clients, so one client's gap doesn't park a mixed fleet) and a self-inconsistent response: zero receipts for a block evidenced to hold logs is a broken receipt store, never proof the block is empty. Progress is persisted per chunk (resumable), and transient failures retry in-process with capped backoff.

3. Repair — registry events are order-dependent and carry per-owner nonces, so a detected miss cannot be patched in place. The verifier persists a resync flag and returns a sentinel that terminates the node; the next start drops registry state and resyncs from the registry offset with inline verification. The repair is resumable (a resync-in-progress marker means an interrupted repair resumes from the last-processed marker — its progress was inline-verified — instead of re-dropping) and rate-limited (defaultResyncCooldown = 6h: further confirmed misses within the window are parked and logged, not acted on, so a common-mode EL fault can't fatal+wipe+resync a large fraction of the operator set in a loop).

Guarantees

• On the optimistic path, the marker never advances past a block the verifier won't later check (journaling is atomic with the marker).
• A resync is triggered only by a receipts-confirmed miss — never on unauthoritative evidence; unresolvable disagreements park the range visibly instead of retiring it as verified.
• Repairs are resumable and rate-limited.
• Near-head streaming stays inline-verified exactly as before; it journals nothing and needs no background pass.

Residual blind spot: if the sync and the verify-time re-fetch drop the same logs, the digests agree and receipts are never consulted — an identical double-drop goes undetected. Resolving every block against receipts would close it but is far more expensive, and the partial within-block drop it guards against has not been observed.

Operator impact

• No config changes, no migration. New storage keys under the operator/ prefix (unverified ranges, per-block digests, resync flags), absent on existing databases and created on demand. Downgrade-safe: older versions ignore the new keys; ranges synced by an older version are simply not verified (the pre-fix status quo).
• Normal boots are unchanged: catch-ups up to maxInlineVerifyCatchUp are verified inline in minutes (see Costs), so there is no unverified window after ordinary restarts.
• Cold sync: startup speed stays ~pre-fix; the background verifier then re-reads the synced range while the node is already running — expect a few hours of extra EL traffic (see Costs) and pending_ranges > 0 until it finishes. During that window a missed Added under-serves (a validator whose data isn't present simply doesn't start, then starts once reconciled), while a missed liquidation/removal briefly over-serves (duties keep running that should have stopped — field-observed, never slashing-relevant) until verification catches it.
• On a confirmed miss: a warning ("background verification found the optimistic sync missed registry events; a full resync will run on the next start"), then the node terminates with registry resync required; the next start drops registry state and rebuilds with inline verification — validator downtime for the duration of the rebuild.
• The repair is slashing-safe. DropRegistryData never touches signer/slashing-protection storage, and the resync's replayed AddShare is idempotent over the key manager's existing accounts and only ever bumps slashing protection. Field-corroborated: the drifted node's post-resync slashing values had strictly advanced.
• Pre-existing drift is not retroactively detected. Syncs performed by pre-fix versions journalled nothing, so this PR cannot verify them after the fact. A node suspected of historical drift needs a one-time manual resync — and note that an operator-table diff understates the damage: fee recipients, nonces and liquidation flags can be stale too (see the field report).
• Alerting: ssv.event_syncer.verify.misses and ssv.event_syncer.verify.parked should stay at zero. parked > 0 → the EL can't authoritatively verify a block (usually missing eth_getBlockReceipts) — investigate the EL. suppressed > 0 → a repair is being deferred by the rate limit. pending_ranges / cursor show verification progress. Genuine receipts recoveries count on the existing ssv.el.bloom.checks metric.

Costs (why the split)

Measured on real infra (local Besu/hoodi + two mainnet block samples via public RPC), using the same bloom-filter test the fix uses:

measurement	hoodi	mainnet (2023 blocks)	mainnet (2024 blocks)
bloom false-positive rate (empty block flagged → drives a receipts fetch)	0.018%	3.99%	5.00%
header payload (getBlockByNumber(false), incl. tx-hash list)	5.4 KB	12.5 KB	18.5 KB
getBlockReceipts payload	77 KB	417 KB	457 KB

Two cost drivers: an O(N) header/bloom screen (one header per empty block, regardless of FP rate) and O(FP) receipts (bloom-positive-empty blocks). Projected:

• Restart catch-up (every boot, the common path): trivial — even a week of downtime (~50k blocks) is <1 GB headers + ~1 GB receipts in minutes, so full inline verification is effectively free.
• Mainnet cold sync (~8.2M blocks from the sync offset, once per node lifetime): header screen ≈ ~120 GB / ~1–2 h, receipts ≈ ~160 GB / ~1–2 h — data-volume-dominated and local-EL-bound.

So verifying every block inline across a cold sync would add ~1–2 h to the startup critical path. Hence the split: ≤ threshold verifies fully inline (effectively free); > threshold goes optimistic and moves the unavoidable verification off the critical path into the background — startup stays ~pre-fix speed while the gap is still closed (eventually), not skipped.

Review guide

Suggested reading order:

1. operator/storage/verification.go — journal + flags storage
2. eth/executionclient/logs.go — the per-block digest
3. eth/eventhandler/event_handler.go — same-transaction journaling
4. eth/eventsyncer/verifier.go — digest comparison, receipts resolution, park/rate-limit
5. cli/operator/eventsync.go — boot wiring: inline/optimistic split + resumable repair
6. eth/executionclient/bloom.go — bloom→re-request→receipts completeness check, VerifyLogs, BlockContractLogs
7. eth/executionclient/eth_client.go — bounded batched RPC with sequential fallback

The rest of the diff is mostly mechanical: the two regenerated mock files, and test files updated for the new parameter on SyncHistory / FetchHistoricalLogs / HandleBlockEventsStream.

Also included

• Offset-0 journaling fix. The journal signal is a *uint64, not a 0 sentinel, so a network with RegistrySyncOffset == 0 (local-testnet, custom YAML) journals instead of silently disabling verification.
• Bounded batched RPC. HeadersByNumbers/SingleBlockLogs cap each batch (~100, matching common hosted-provider limits) with per-element sequential fallback, and remember a provider that rejects batching wholesale (latched only when the sequential fallback succeeds — and never on a batch timeout/cancellation — so neither an outage nor a slow provider false-latches).
• Receipts self-consistency guard. Zero receipts for a block evidenced to hold logs is treated as an unreliable receipt store (park / fall back to re-requests), never as an authoritative empty block; and -32004 ("method not supported", EIP-1474) is recognized alongside -32601 when detecting a missing eth_getBlockReceipts.
• Response-size-aware subdivision. subdivideLogFetch also splits on response-size failures (websocket read limit, HTTP 413, "response too large"), not just the -32005 query-limit code, so a wide verify chunk can't wedge the verifier in a no-progress retry loop.
• Staleness-guard underflow fix. ensureBlockAboveThreshold compared in unsigned space; a large staleness threshold wrapped the cast and flagged every block as too old. Now compared in signed space.

Accepted tradeoffs (feedback welcome)

• Unverified window on the optimistic path only (cold sync, or a very prolonged catch-up) until the background pass clears it — for a fresh cold sync, potentially hours. The window cuts both ways (missed additions under-serve, missed liquidations/removals over-serve; see Operator impact), but normal restarts avoid it entirely, and it's strictly better than the status quo, where a miss was permanent and silent.
• Resync blast radius. A confirmed miss costs a full DropRegistryData + resync-from-offset (validator downtime for the rebuild). Misses should be rare (EL bug/corruption only), a full rebuild is the only correct remedy given the nonce ordering, and the repair is resumable + rate-limited to bound the cost. A surgical replay-from-affected-block is deliberately left as follow-up.

Testing

• Unit — digest properties (order-independence, subset ⇒ different digest); storage journal/flag/timestamp round-trips; verifier state machine (clean range, receipts-confirmed missing-digest and subset misses, receipts-vindicated verify-time drop, park-on-no-receipts, rate-limit suppression, cursor resume, chunking, retry wrapper); handler journaling (per-block digest + range on the optimistic path, offset-0, range journalled up to the last processed block even when the stream aborts); boot resync preparation (first-attempt drop + mark-in-progress, resume-from-marker when already in progress); batch-error → sequential fallback and the batching-unsupported memo.
• Integration — the full miss → flag → drop → verified resync → clear loop with the real handler + storage and a mocked EL.
• End-to-end against a real go-ethereum RPC stack (simulated backend + an HTTP proxy that induces the incomplete eth_getLogs) — exercises the real ExecutionClient (bloom/receipts/batching), not mocks: (a) healthy EL verifies clean with no false resync; (b) a persistently-dropped block is silently missed by the optimistic sync, recovered from receipts by the background verifier, and repaired by the resync so the DB ends complete — the Registry events silently and permanently lost: historical log sync skips bloom verification but advances lastProcessedBlock anyway #2990 loop; (c) an unresolvable disagreement with eth_getBlockReceipts unavailable parks the range instead of resyncing; (d) the field report's full blast radius — a dropped block carrying a fee-recipient update, a cluster liquidation and an owner-nonce bump drifts all three and the repair restores them, with the verified resync replaying ValidatorAdded over the key manager's existing share account; (e) an empty-receipts response parks (no false resync) and the parked range retires clean once receipts return.
• The existing eth_getLogs-drop recovery regression test for Registry events silently and permanently lost: historical log sync skips bloom verification but advances lastProcessedBlock anyway #2990 is retained.
• go build ./..., pinned golangci-lint (0 issues), and -race runs of the affected packages pass.

End-to-end validation on real Besu

Beyond the in-process E2E above, the fix was validated end-to-end against a real, external EL exhibiting genuine on-disk log-index corruption — the actual #2990 mechanism, not a simulated drop. This harness is intentionally not committed / CI-run (it needs an external synced node); the methodology is recorded here so it can be reproduced.

Rig. Besu (hoodi, snap sync) + Lighthouse (hoodi, checkpoint sync), synced past block 2,746,590 — where operator 542 registered on the hoodi SSV contract 0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8.

Fault injection. Besu serves eth_getLogs from per-block log-bloom cache files (caches/logBloom-<segment>.cache, 256 bytes/block, no checksums). Zeroing operator 542's slot — segment 2746590 / 100000 = 27, offset (2746590 % 100000) * 256 = 11,927,040, 256 bytes — makes Besu silently return incomplete logs for that block while its receipts stay intact. Verified directly against the running node: eth_getLogs at block 2,746,590 dropped from 2 logs to 0 (no error), while eth_getBlockReceipts still returned both.

Result. Driving the real ExecutionClient → EventSyncer → EventHandler → BadgerDB pipeline against the corrupted node reproduced the full loop:

1. optimistic SyncHistory silently missed operator 542 — Registry events silently and permanently lost: historical log sync skips bloom verification but advances lastProcessedBlock anyway #2990;
2. the background verifier recovered the block from receipts, logged the operator-actionable warning (recovered contract logs from block receipts that the execution client's log index did not return … consider resyncing), and flagged a resync;
3. the drop + verified resync recovered operator 542.

Confirms the detect→repair path works against a real EL with a genuinely corrupt log index, not just a simulated one.

Out of scope / follow-up

• Surgical replay-from-affected-block instead of full resync on a confirmed miss. Per the field data, nonce reconciliation is the hard part: a missed event can leave a lower persisted nonce, so a replay must reconcile per-owner nonces, not just insert events.
• A config opt-out for auto-repair (the rate-limit is in; a hard off-switch is not).
• Closing the identical-double-drop blind spot (see Guarantees) would require resolving every block against receipts — far more expensive, not observed in practice.

[greptile-apps]

Greptile Summary

The PR adds deferred completeness verification and guarded automatic repair for large optimistic registry-event syncs while retaining inline verification for normal catch-ups.

• Atomically journals optimistic ranges and per-block log digests with sync progress.
• Rechecks journaled ranges in the background and resolves disagreements through block receipts.
• Persists resumable, rate-limited resync state when receipts confirm missing events.
• Adds bounded batched RPC fallbacks, verification metrics, and end-to-end coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains from the previous review thread.

The previously reported large-sync verification gap is invalidated by atomic range journaling and the background verifier’s bloom-and-receipts completeness pass.

Important Files Changed

Filename	Overview
eth/eventsyncer/verifier.go	Implements resumable background comparison, receipt-based mismatch resolution, range parking, and rate-limited repair signaling.
eth/eventhandler/event_handler.go	Journals optimistic verification state atomically with registry updates and the processed-block marker.
eth/executionclient/bloom.go	Extends completeness checks with bounded header/log requests and receipt-derived recovery.
cli/operator/eventsync.go	Selects inline versus optimistic startup sync and runs background verification alongside ongoing event synchronization.
operator/storage/verification.go	Adds durable verification ranges, block digests, and resumable repair flags.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Historical registry sync] --> B{Catch-up size}
  B -->|Small or repair| C[Inline bloom and receipts verification]
  B -->|Large| D[Optimistic eth_getLogs sync]
  D --> E[Atomically persist marker, range, and digests]
  E --> F[Start ongoing sync]
  E --> G[Background verifier]
  G --> H{Digests agree}
  H -->|Yes| I[Retire verified range]
  H -->|No| J[Resolve block through receipts]
  J -->|Recorded state is complete| I
  J -->|Receipts unavailable| K[Park range]
  J -->|Confirmed miss| L{Resync cooldown}
  L -->|Expired| M[Persist resync-required and terminate]
  L -->|Active| K
  M --> N[Next start drops registry state]
  N --> C

Loading

_{Reviews (2): Last reviewed commit: "eth: harden verify/repair robustness fro..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 74.69244% with 144 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.5%. Comparing base (48d4f3a) to head (f91cd47).
⚠️ Report is 17 commits behind head on stage.

Files with missing lines	Patch %	Lines
cli/operator/eventsync.go	31.2%	38 Missing and 6 partials ⚠️
eth/executionclient/bloom.go	72.4%	26 Missing and 9 partials ⚠️
eth/eventsyncer/verifier.go	79.7%	16 Missing and 12 partials ⚠️
operator/storage/verification.go	72.7%	9 Missing and 9 partials ⚠️
eth/executionclient/multi_client.go	63.3%	11 Missing ⚠️
eth/eventhandler/event_handler.go	60.0%	2 Missing and 2 partials ⚠️
eth/executionclient/eth_client.go	94.8%	2 Missing and 2 partials ⚠️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[iurii-ssv]

@greptile pls re-review