EKM: whole HD-wallet index rewritten on every account add — O(n²) writes, retained versions up to 99% of the node DB

[ovidiu-ssv-labs]

Summary

SaveWallet persists the entire HD-wallet index after every account add. The index grows linearly with validator count, so creating n accounts writes O(n²) bytes to a single BadgerDB key. Each write leaves a superseded MVCC version, and because the resulting version chain lands in one oversized bottom-level (L6) table, nothing reclaims it: we measured DBs where this one key is 66–99.5% of the physical database, with physical size 9.5–21.6× the logical content.

This is a performance/storage defect, not a correctness bug — nothing is lost or wrong, and persisting the index per add is semantically necessary for crash-safety. The defect is that it persists the whole index rather than the delta.

Measured across 15 node databases on our hoodi-stage testnet (surveyed all 56 live nodes to select them), spanning 0.7–49.6 days of age and ~1,170–4,470 accounts.

Mechanism

eth2-key-manager v1.5.6, wallets/hd/wallet.go:167-183 — every account add re-saves the whole wallet:

func (wallet *Wallet) AddValidatorAccount(account core.ValidatorAccount) error {
	validatorPublicKey := hex.EncodeToString(account.ValidatorPublicKey())
	wallet.indexMapper[validatorPublicKey] = account.ID()
	if err := wallet.context.Storage.SaveAccount(account); err != nil { return err }
	err := wallet.context.Storage.SaveWallet(wallet)   // <-- entire wallet, every time
	...
}

wallets/hd/wallet_marshalable.go:11 serialises {id, type, indexMapper}, where indexMapper maps every validator pubkey (96 hex chars) → account UUID (36 chars). Measured at 138.0–139.0 bytes per entry on all 13 sampled nodes that have a wallet.

ssvsigner/ekm/signer_storage.go:117-126 (main) writes that blob to one fixed key — prefix <network>signer_data-wallet-, key wallet:

data, err := json.Marshal(wallet)
...
return s.db.Set(nil, s.objPrefix(walletPrefix), []byte(walletPath), data)

So with n accounts: final blob = O(n); cumulative written = O(n²). SSV uses badger.DefaultOptions (storage/badger/badger.go:41), so NumVersionsToKeep=1 — superseded versions are discarded only when a compaction rewrites the table holding them.

Side note: unlike SaveAccount, which encrypts (signer_storage.go:195), the wallet blob is stored as plaintext JSON. Not secret material (a pubkey→UUID map), but worth knowing.

Evidence

The quadratic law fits with no free parameters

Against the analytic prediction 138.03 · V² / 2, where V = observed wallet write count:

V (writes)	retained	predicted	ratio
1,606	178.0 MB	178.0 MB	1.000
1,645	186.7 MB	186.8 MB	1.000
1,646 (×3 nodes)	187.0 MB	187.0 MB	1.000
1,708	200.8 MB	201.3 MB	0.997
1,918	253.9 MB	253.9 MB	1.000
2,872	547.8 MB	569.3 MB	0.962

Log-log regression over these 8 nodes: exponent 1.935, R² 0.9999. V spans 1.79× while retained bytes span 3.08× — a linear law predicts 1.79×, quadratic 3.20×. (The 0.962 outlier is expected: that node's indexMapper had shrunk from ~2,871 to 2,215 entries through removals, so its later writes were smaller than a pure ramp.)

Retained versions do not decay with time

The cleanest datapoint is an accidental controlled experiment: three operators in the same committee, with an identical validator set, whose DBs were created 0.7 d, 19.9 d and 39.7 d ago. All three measure 1,646 versions and 186,957,xxx bytes — byte-for-byte identical. ~40 days of continuous duty produced zero reclamation. Two further DBs at 49.0 d and 49.6 d are likewise fully intact (1,708 and 1,606 versions, ratio 1.00 to the model).

On every retaining node the chain sits in a single oversized L6 table (e.g. L6: 5 tables / 533 MB, L6: 21 tables / 220 MB). Bottom-level residency explains the non-reclamation: an L5→L6 compaction only rewrites that table if L5 happens to overlap this one key.

Magnitude

Every node's logical live DB is ~25 MB.

	physical	amplification	wallet key share
retaining nodes (8)	237–551 MB	9.5–21.6×	66–99.5%
collapsed nodes (4)	45–61 MB	1.7–2.3×	0.7–2.4%

Compression doesn't help — the blob is high-entropy hex, Snappy ratio ≈1.02.

Worst case in the sample: a node retaining 328.7 MB against a 10.6 KB live blob (~31,000×). It once held ~2,600 validators; removals shrank the live blob without touching the retained chain, and it has 76 active validators today. So bloat cannot be predicted from current validator count.

Reclamation is event-driven, not time-driven

Four nodes had collapsed to 1–3 versions — one in under a day, while another has not collapsed at 49.6 days. Age predicts nothing. What correlates is the size the chain would have reached, with a sharp boundary between a node at ~569 MB (wholly intact) and one at ~599 MB (wholly collapsed). Above that, behaviour is a sawtooth: some compaction eventually rewrites the giant table and collapses it, then it regrows — one node retains only the last 2,373 of 6,069 writes.

Consequence: retained space appears to saturate around 0.3–0.55 GB rather than growing without bound. Cumulative writes are the unbounded cost — one node has written ~2.5 GB to this single key.

I have not established the mechanism for that ~0.6 GB boundary. It is a four-point empirical observation, not a proven badger constant, and should not be relied on as a guaranteed ceiling.

Impact

• Write amplification — O(n²) bytes to one key, paid whenever accounts are created in bulk (fresh node, new DB path, wiped volume, EKM rebuild). ~0.6–2.5 GB of writes observed on higher-count nodes. Relevant to IO time and SSD wear.
• Space amplification — a ~25 MB logical DB occupying 237–551 MB, dominated by one key, persisting indefinitely below the collapse boundary.
• Both worsen with validator count, so larger operators are disproportionately affected.

Not affected: nodes using a remote signer. One sampled node with 1,910 validators had zero EKM accounts and no wallet key at all — its EKM state lives in ssv-signer. We checked SSV_SIGNER_ENABLED across all measured nodes to rule this out as a confound; it is a genuine mitigation for local-signer deployments.

Not amplified by the #2990/#2991 repair path: DropRegistryData (operator/storage/storage.go) doesn't drop EKM data, and saveAccount is guarded by if acc == nil (ssvsigner/ekm/local_key_manager.go:312), so a repair-resync re-processing ValidatorAdded finds existing accounts and performs no wallet rewrites. We verified this specifically because it looked like a plausible interaction.

Possible fixes

1. Batch the wallet save across bulk account creation — add all accounts, persist the index once. Removes the quadratic burst with no storage-format change, and SSV controls the call site (ssvsigner/ekm/local_key_manager.go:354). Lowest-risk meaningful win.
2. Store the index per-key (wallet-index-<pubkey> → uuid) instead of one blob, making an add O(1). Cleanest, but a format change requiring a migration, and it touches eth2-key-manager.
3. Drop the index as stored state — accounts are already stored under their UUID, so the pubkey→UUID map could be rebuilt by scanning accounts, or accounts keyed by pubkey directly.

A NumVersionsToKeep/compaction tweak would only address the space symptom, not the O(n²) writes.

Method and caveats

• Databases were copied off the pods (md5-verified file-by-file after repair rounds) and read offline with badger AllVersions=true iteration; nothing was opened in place. Live-DB memtable WALs were discarded from copies — immaterial for a key last written days earlier.
• ls sizes are badly misleading here and any reproduction should use actual disk blocks: badger preallocates sparsely, e.g. .vlog showing 209.7 MB apparent vs 8 KB actual.
• The n range is narrow and clustered — 6 of 8 fitted points sit near V≈1,650, and no node on this network has fewer than ~1,170 accounts, so the low end is untested. The exponent is supported mainly by the absolute zero-parameter fit holding across the span, not by range alone.
• Confounds checked and cleared: retention and collapse both occur within the same node image (so it is not a code difference), and both DB_PATH variants present were measured.
• Measured on a testnet only; not reproduced on mainnet, though the code path is identical.

Happy to share the per-node measurements or the measurement tooling.