Registry events silently and permanently lost: historical log sync skips bloom verification but advances lastProcessedBlock anyway

[ovidiu-ssv-labs]

Summary

Registry events can be silently and permanently lost during historical event sync. FetchHistoricalLogs fetches logs with bloom verification disabled, then advances lastProcessedBlock regardless of whether the execution client actually returned the logs for those blocks. Because the block marker and the events commit in the same transaction and nothing ever revisits a processed block, an incomplete eth_getLogs response becomes a permanent gap in the node's registry state.

We found this in production-like conditions on our hoodi-stage testnet: one node had been running for ~3.5 months missing an operator, and two others are missing 32 operators and 2 validator shares each. All three report healthy: duties execute, consensus works, metrics are green, and lastProcessedBlock is level with their peers. The only way we detected it was by copying the BadgerDB files off the pods and diffing the operator tables.

This is not fork-related and is present on main.

Impact

• Silent registry divergence from chain state, undetectable through logs, metrics, or health checks.
• Only repairable by a full resync from RegistrySyncOffset — there is no reconciliation or self-heal path.
• Historical sync runs on every startup (from the stored lastProcessedBlock, cli/operator/eventsync.go), so this is the routine post-restart catch-up path, not a rare cold-start edge case. Every restart is an opportunity to lose events.
• Applies to mainnet, same code path. A node missing operator data for a committee it serves will fail to start those validators (operator/validator/controller.go, "operator data not found, validator will only start if the number of available operators is greater than or equal to the committee quorum"). We got lucky: in our cases the missing operators happened not to be in the affected nodes' own committees, so duties were unaffected. That is chance, not design.
• Bloom verification was added in feat(el): bloom filter cross-check for log integrity #2722 and correctly guards the streaming path. The historical path was intentionally left unguarded, so the protection does not cover the path that runs on every boot.

Mechanism

All references at main @ a1bcd468f, eth/executionclient/execution_client.go:

1. :143 — historical fetch disables the check:

   logsCh, errsCh = ec.fetchLogsInBatches(ctx, fromBlock, toBlock, false)

   versus :506 for streaming, which enables it (..., true).

2. :192-194 — the skip is deliberate and documented:

   // Verify each block's logs against its bloom filter.
   // Only enabled for streaming (near chain tip) where the Geth bug is most impactful.
   // Skipped during historical sync to avoid ~200 extra header RPCs per batch.
   if verifyBloom {

   eth/executionclient/bloom_test.go documents the same ("FetchHistoricalLogs skips bloom checks"). So the EL returning incomplete logs is a known failure mode — it just isn't guarded here.

3. :240-243 — progression is emitted whether or not logs arrived:

   // Emit an empty BlockLogs to indicate progression to the next block.
   if highestBlock < toBlock {
       logCh <- BlockLogs{BlockNumber: toBlock}
   }

4. eth/eventhandler/event_handler.go:172-191 — processBlockEvents writes whatever events arrived and SaveLastProcessedBlock in one atomic transaction and commits. With zero or partial logs, the marker still advances and the transaction still commits successfully.

5. Nothing re-reads a block below lastProcessedBlock; in fact processBlockEvents treats a lower block as a hard error (ErrInferiorBlock). The gap is therefore permanent.

Net effect: a single bad batch response silently drops every registry event in it, and the node cannot tell that it happened.

Evidence

Canonical chain state for our hoodi-stage registry is 587 operators / 43,304 shares. We verified this independently two ways: a node that did a full clean resync from RegistrySyncOffset produced exactly that, and replaying the event log gives 751 OperatorAdded − 172 OperatorRemoved = 587, set-identical to that node's DB.

We then surveyed DB creation era across all 111 of our stage nodes, identified the 24 whose DBs predate the incident window, and read the operator tables directly out of copies of those BadgerDBs (opened via operatorstorage.NewNodeStorage → ListOperatorsAll, i.e. the node's own code path, validated against known-good counts before trusting any diff).

Node A — DB created 2026-04-21, reports 586 operators / 43,304 shares:

• Missing exactly one row: operator 542, registered in block 2,746,590 (2026-05-04), tx 0xfcc6172ff307011100f2e5f179f4dac6c74a5d3bf657543af15a809c40b80dec, never removed.
• That block contains exactly 2 SSV logs, both from that tx (OperatorAdded + OperatorPrivacyStatusUpdated) — consistent with the whole block's log set being lost.
• Registration-order neighbours 535-541 and 548-549 are all present; of the 62 operators owned by the same address, only 542 is missing. Reverse diff empty; the other 586 rows byte-identical to canonical.
• Survived a genuine pod restart still reporting 586, so it is persistent state, not a read artifact.

Nodes B and C — report 555 operators / 43,302 shares, byte-identical to each other:

• Missing all 32 canonical operators with IDs 550-629, and nothing outside that range. Since IDs are sequential, this maps to a contiguous block window: operator 549 @ 2,805,409 (present) → 550 @ 2,819,474 (missing) → 629 @ 2,905,681 (missing) → 641 @ 2,936,696 (present). Roughly 86k blocks / ~12 days, mid-May 2026.
• Also missing 2 validator shares whose committees they otherwise know, so those are independent event losses rather than knock-on effects.
• lastProcessedBlock is 3,397,122 / 3,397,128 — level with healthy nodes. Not sync lag; permanent.

The remaining 21 nodes in the cohort were clean (587 / 43,304). No node anywhere held a row that canonical lacks, and there were no owner/pubkey mismatches — the loss is strictly one-directional.

Why other explanations were excluded

• Not a torn/partial commit — events and the block marker share one transaction (point 4 above).
• Not the MalformedEventError skip path (event_handler.go:215-218, which does silently drop an event while committing the block). That requires a pre-existing conflicting row per event; 32 operators plus 2 unrelated shares in one contiguous window cannot plausibly be 34 separate ID/pubkey conflicts. Operator 542's pubkey is also unique across all 587, and no conflicting row exists.
• Not a global/chain-level miss — all these nodes read from one shared Besu execution client, and 23 of the 24 nodes in the cohort have operator 542. The damage is per-node and per-request, which is what an unverified eth_getLogs batch response looks like.
• Not sync lag or image skew — markers are current, and the affected pods were started months after the loss windows, so the gaps live in persistent DB state.

Suggested fix

Enabling verifyBloom unconditionally would cost the ~200 header RPCs per batch the comment is trying to avoid. A cheaper option that still closes the hole: only pay for verification when a batch looks suspicious — e.g. when a block range returns zero logs (or fewer than expected), fetch the headers for that range and check the blooms; re-request if the bloom indicates logs should exist. That confines the extra RPCs to the rare suspicious case rather than the common path.

Whatever the shape, the important properties seem to be:

1. Do not advance lastProcessedBlock past a range whose log completeness was never established.
2. Fail loudly (or retry) instead of silently emitting empty progression on a suspicious batch.
3. Consider a way to detect/repair existing divergence, since affected nodes today have no signal at all and full resync is the only remedy. A periodic or on-demand reconciliation of the operator/share set against chain state would also let operators find out whether they are already affected.

Caveats / what we could not prove

• We could not confirm that the affected nodes restarted at the specific moments in question — our log retention does not reach back to early/mid-May 2026. The historical-sync path is therefore a strongly supported inference from the code and the damage pattern, not a directly observed event. It is conceivable the same incomplete-response problem struck via a different path.
• We did not confirm that the 2 missing shares' ValidatorAdded blocks fall inside the 86k-block window (the scan was too slow to finish). It is consistent with the window, not proven.
• 81 of our nodes have DBs created after the incident window and were not censused, so we do not know the true fleet-wide prevalence. Note there is no cheap way to check: ssv_validator_validators_per_status counts only a node's own validators, not the global registry set, so detecting this currently requires reading the DB.

Happy to provide the operator-set dumps or more detail on any of the above.

[iurii-ssv]

We reproduced this end-to-end against a real geth (v1.16.4, --dev) to confirm the mechanism before settling on the fix shape (#2991). Summary of the path and the evidence below.

Setup

• Real geth v1.16.4 dev chain (1s period), and the SSV simulator contract (eth/simulator/simcontract) emitting real OperatorAdded events — 5 operators registered, one block each (operator 3 landed in block 110).
• The sync side is the exact production pipeline the node wires at startup — ExecutionClient → EventSyncer → EventHandler → BadgerDB on disk, resuming from stored lastProcessedBlock+1 like cli/operator does. No beacon needed (registry sync is EL-only).
• Between the node and geth: a JSON-RPC proxy simulating a misbehaving EL by dropping block 110's logs from eth_getLogs responses. Two modes: wide lies only on multi-block queries (an intermittent drop — single-block re-requests get the truth), all lies on every eth_getLogs shape (a persistently broken log index, the suspected Nodes B/C state). Headers and receipts always pass through honestly.

Run 1 — pre-fix code (stage), lying EL: silent loss

The proxy dropped exactly one log from one range query:

[proxy] LIED: dropped 1 log(s) of block 110 from eth_getLogs [107..146] response

The node accepted the short response as routine progress — its own fetch log reports 4 events where 5 exist on chain:

INFO  ExecutionClient  fetched registry events  {"from_block": 107, "to_block": 146, "progress": "100.00%", "events": 4, ...}

Result — sync "succeeds", the marker advances past the dropped block, operator 3 is missing, and there are zero WARN/ERROR lines in the entire run:

SYNC RESULT: ok, lastProcessedBlock=146
DB STATE: 4 operators, lastProcessedBlock=146
  operator id=1 ... id=2 ... id=4 ... id=5   (id=3 absent)

Run 2 — restart the same DB against a now-honest EL: permanent

Mirrors a node restart after the EL recovers:

resuming from stored lastProcessedBlock+1 = 147 (node-restart behavior)
SYNC RESULT: ok, lastProcessedBlock=158
DB STATE: 4 operators, lastProcessedBlock=158

Operator 3 never comes back — the node never asks about block 110 again. This is the silent, permanent divergence described in this issue, reproduced deterministically.

Run 3 — fixed code (#2991), same intermittent liar: recovered

WARN  ExecutionClient  recovered contract logs the execution client omitted on first request
      {"block": "110", "recovered_events": 1, "recommendation": "events were not lost, but the EL returned incomplete logs ..."}
DB STATE: 5 operators

Run 4 — fixed code, persistent liar (every eth_getLogs shape lies): recovered via receipts

The proxy lied twice (range query, then the single-block re-request); the receipts cross-check settled it, since eth_getBlockReceipts reads the receipt store rather than the log index:

WARN  ExecutionClient  recovered contract logs from block receipts that the execution client's log index did not return
      {"block": "110", "recovered_events": 1, "recommendation": "... its log index may be corrupted, consider resyncing or updating your execution client"}
DB STATE: 5 operators

(Pre-fix code against the same persistent liar loses the operator, as in Run 1.)

Takeaways

• The loss mechanism claimed here is real and deterministic: one incomplete eth_getLogs response during historical sync → silent, permanent registry gap, invisible to logs/metrics/health checks. Confirmed on the exact production components.
• eth: verify registry-event completeness after optimistic historical sync (#2990) #2991 closes it on every sync path. Near-head streaming and normal restart catch-ups are verified inline (bloom → single-block re-request → eth_getBlockReceipts), recovering both observed classes — intermittent drops (re-request) and persistent log-index breakage (receipts) — with loud, operator-actionable warnings. A large cold sync is synced optimistically for speed, then checked in the background against block receipts; a receipts-confirmed miss triggers a full, inline-verified resync. (The fix shape evolved after this write-up — the background verifier is what closes the cold-sync case; see the PR for the final design.)
• Residuals, both documented on the PR: an EL that supports neither eth_getBlockReceipts nor a correct log index has no authoritative source, so verification parks the range (kept visible, not resynced on a guess) until the EL is fixed; and a drop that hits the sync and the background re-fetch identically would agree and go undetected (never observed).
• Detection/repair for nodes already diverged under the pre-fix code (point 3 of this issue) remains open — the background verifier only checks ranges the new optimistic sync journals, not pre-existing gaps.

Happy to contribute the reproduction harness (a small Go tool: deploy / sync / dump / lying proxy subcommands) if it's useful as a regression rig.

[iurii-ssv]

Follow-up: reproduced the end-to-end failure against real Besu by inducing a realistic cache corruption

Correction (edited): an earlier version of this comment said this was reproduced "organically" against an "unmodified Besu" with "no synthetic lie in the data path." That over-claimed and I've corrected it. The Besu binary is unmodified and every log/header/receipt value it serves is real, but we edit its on-disk log-index cache by hand to induce the incomplete responses (step 3). This is an induced reproduction of a realistic mechanism and its end-to-end consequence — not a catch of Besu misbehaving on its own.

The earlier write-up proved the conditional (given an incomplete eth_getLogs, the node loses events) with a proxy injecting the lie. Here we take it end-to-end on a real, unmodified Besu binary on real hoodi, for the real hoodi-stage contract and the exact incident block: we induce the incomplete-response state by corrupting Besu's on-disk log-bloom cache the way a crash-torn write would, then confirm the pre-fix node loses operator 542 and #2991 recovers it.

Scope — what this shows and doesn't: it reproduces the mechanism and its end-to-end consequence + fix on real infrastructure. It does not show Besu spontaneously corrupting itself: we edit its derived log-index cache to simulate corruption. Whether an unmodified Besu reaches this state on its own (e.g. a crash mid-cache-write leaving a served-partial file, or the historical index still backfilling after snap sync) is a separate, still-open question we're investigating. Note the geth variant of incomplete getLogs was already observed organically in the wild (#2722); only the Besu-specific trigger remains modeled rather than caught.

The corruption mechanism (the realistic part)

Besu serves eth_getLogs from derived per-block bloom cache files (caches/logBloom-<segment>.cache, a raw 256-byte bloom per block, no checksums). A crash mid-write leaves zeroed/partial regions — a documented real-world failure class. We reproduced exactly that state by zeroing one block's cache entry:

• segment = blockNumber / 100000, offset within file = (blockNumber % 100000) * 256
• for block 2,746,590 (operator 542's registration): segment 27, offset 11,927,040

Nuance: the segment file must exist first (either wait for Besu's background cache-gen to reach it, or force it with admin_generateLogBloomCache). Besu must be stopped before editing the file and restarted after.

Step-by-step

1. Sync Besu on hoodi (--network=hoodi --sync-mode=SNAP) + any CL (we used Lighthouse checkpoint-sync). Nuance: Besu needs a heap cap (e.g. -Xmx3g) or it OOMs during hoodi snap-sync on a ~10GB host. Side finding: Besu reports eth_syncing=false throughout the import — a misleading health signal in its own right.

2. Confirm ground truth (via any healthy RPC): block 2,746,590 contains exactly 2 SSV logs — OperatorAdded(542) + OperatorPrivacyStatusUpdated.

3. Induce the corruption (edit Besu's on-disk cache): stop Besu; dd if=/dev/zero of=caches/logBloom-27.cache bs=1 seek=11927040 count=256 conv=notrunc; start Besu.

4. Observe the resulting incomplete responses (unmodified Besu binary, direct RPC):

   • eth_getLogs single-block [2746590,2746590] → [] (wrong, 2 logs exist)
   • eth_getLogs range [2746401,2746600] → returns block 2746498's logs but silently omits 2746590 — a partial range response, HTTP 200, no error. This is the Registry events silently and permanently lost: historical log sync skips bloom verification but advances lastProcessedBlock anyway #2990 failure verbatim.
   • eth_getBlockReceipts(2746590) → 44 receipts, 2 SSV logs intact (receipts come from the receipt store, not the poisoned log index).

   Nuance: the cache is consulted for both single-block and range queries, so both lie — this models persistent structural corruption (only receipts recover it), which is stronger than the intermittent Geth drop from feat(el): bloom filter cross-check for log integrity #2722 (which a single-block re-request alone recovers). The header bloom (in the block header) is separate and intact, so the completeness check's bloom screen still flags the block.

5. Run the node's real sync pipeline (ExecutionClient → EventSyncer → EventHandler → BadgerDB) against this Besu, offset just below the corrupted block. We bounded the catch-up window to ~43k blocks (simulating a node down ~6 days) by capping the reported head via a thin passthrough — this only bounds how far the sync runs; every log/header/receipt byte is real Besu.

Result

	operators stored	operator 542
pre-fix (stage)	5 (544–548)	absent — silently lost
fix (#2991)	6	present, real RSA pubkey recovered

The pre-fix/fix DB delta is exactly {542}. The fix logged:

WARN recovered contract logs from block receipts that the execution client's log index
     did not return  {"block": "2746590", "recovered_events": 2, "contract": "0xc07B..."}

Nuances: (a) the run ends with a cosmetic block ... is too old liveness-guard error — it fires after events commit, so the DB state is authoritative; (b) one unrelated real event fails to unpack under the test harness's static ABI bindings — identical in both runs, hence not the cause of the {542} delta.

The one honest caveat

This recovered because the bounded window was inline-verified (a restart catch-up under the inline cap). A full cold sync (above the cap) is synced optimistically, so the inline path alone would still lose operator 542 — which is why #2991 also adds a background verifier that checks optimistically-synced ranges against block receipts and repairs a confirmed gap with a full, inline-verified resync. This exercise validates the inline path for the common every-restart catch-up and motivates the background verifier for the cold-sync case.