Beacon node can deadlock at startup when bulk state deletion runs concurrently with a nested DB read

[james-prysm]

Background

Discovered in winter 2025 while recovering a node with a corrupted DB: the restore attempt used pruning, and the node hung on startup.

Summary

Bulk state deletion at startup runs in bolt write transactions. A delete-heavy commit can force bbolt to remap the data file, which waits on all open read transactions. A nested db.View on the startup path then blocks behind that pending remap while holding the read transaction the remap is waiting for — a circular wait that freezes the entire DB and hangs the node.

Sequence

1. Blockchain service starts: StartFromSavedState calls stategen.Resume, which fires CleanUpDirtyStates in a background goroutine.

   • beacon-chain/blockchain/service.go:279
   • beacon-chain/state/stategen/service.go:155-159

2. CleanUpDirtyStates collects every state violating the archive-interval rules and deletes them one by one, each in its own db.Update write transaction. (The --beacon-db-pruning service does the same in 32-slot batches after sync.)

   • beacon-chain/db/kv/state.go:1007-1078 (CleanUpDirtyStates)
   • beacon-chain/db/kv/state.go:562-573 (DeleteStates → DeleteState, one db.Update each)
   • beacon-chain/db/pruner/pruner.go:212-237, beacon-chain/db/kv/blocks.go:470-553 (pruner path)

3. Deleting states frees many pages, growing bbolt's freelist. At commit, the freelist is rewritten; if no contiguous free run is available, bbolt must remap the data file: db.mmap() → mmaplock.Lock(). This write lock waits for every open read transaction, since each read tx holds mmaplock.RLock() for its whole lifetime.

4. Concurrently, the main startup goroutine reaches setupForkchoiceRoot, which calls LastValidatedCheckpoint. That opens a db.View, and when lastValidatedCheckpointKey is absent (e.g. freshly checkpoint-synced DB) it calls FinalizedCheckpoint inside the open View — opening a second read transaction in the same goroutine.

   • beacon-chain/blockchain/setup_forkchoice.go:143-147
   • beacon-chain/db/kv/validated_checkpoint.go:18-38 (nested call at line 23)
   • beacon-chain/db/kv/checkpoint.go:41 (inner db.View)

5. Go's sync.RWMutex blocks new readers while a writer is pending, so the nested FinalizedCheckpoint read blocks behind the pruner's pending remap. The pruner's remap waits for the outer LastValidatedCheckpoint read tx to close; that tx can't close until the nested read completes; the nested read waits on the pruner. Circular wait, no timeout.

6. Everything else queues behind the two: all writers block on bolt's rwlock (held by the delete tx), all new readers block behind the pending remap lock, and db.Close() blocks too — so the node hangs on startup and can't even shut down cleanly.