Repository navigation
Engine review: crash-recovery fixes and on-disk format v2 - #8
Merged
Merged
Conversation
Durability and recovery - WAL open truncates a torn or zero-filled tail. Records appended after a torn tail used to land behind it, where the next replay never reached them, losing acknowledged writes. Bytes that are not a plain torn tail are saved to <wal>.discarded before the cut. - WAL records checksum their header as well as the payload (flag bit in the type byte; records without it still decode). - A halted engine no longer checkpoints, snapshots or checkpoints on Close, so the next open redoes the WAL as the halt message promises. - Followers WAL-log replicated batches; they were applied in memory only. - A blank page 0 in front of real data fails the open with ErrMetaPageDamaged instead of deleting and recreating the data file. - A failed secondary index rebuild fails the open. B+ tree - Leaf merges remove the parent entry by child pointer. Removing by the leaf's first key left the parent routing to the dead leaf once that key had been deleted, so scans returned deleted and stale entries. The right sibling now comes from the parent, not the leaf chain. Other - gRPC Scan reads in chunks and releases the engine lock before sending, so a slow client cannot block writes. - GetTuple returns ErrCorruptPage for slots outside the page. - Replication catch-up stops at the leader's fsynced LSN.
On-disk format version 2 - Every page ends with a CRC-32C trailer over its page ID and contents, stamped when a page goes to the journal or spill file and verified on every read. Damaged or misdirected pages surface as ErrPageChecksum instead of being parsed. ReadPage now zeroes the part of the buffer past the end of the file instead of leaving the caller's old bytes there. - Version 1 files are migrated on first open: records of the last checkpoint image are copied into a new file, the WAL tail is redone, and the new file replaces the old one in a single rename (old files kept as *.v1.bak). LSNs continue past the old ones. The v0 migration uses the same crash-safe install step; before, a crash between its renames left no data file at all. Page reuse - A free list threaded through freed pages, with its head saved in the meta page at each checkpoint. Merged-away leaves, internal nodes left without keys (including root collapse) and data pages whose last tuple is deleted go on it; NewPage takes from it before growing the file. A list that does not check out is abandoned, never followed. Saved maps - The free-space map is saved at every checkpoint, so space freed before a restart is still reused after it. - A clean shutdown saves the search index; the next open loads it instead of scanning every record. After a crash it is rebuilt as before. WAL - Opening fails with wal.ErrCorruptLog when an intact record follows a damaged one (mid-log damage, not a torn tail). -wal-truncate-corrupt / BEASTDB_WAL_TRUNCATE_CORRUPT cuts the log at the damage instead. Also - A failure while opening (index rebuild, WAL redo) closes the files without a checkpoint instead of persisting half-recovered state. - beastctl backup verify checks every page checksum of a v2 bundle.
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
The WAL is open with O_APPEND, and on Windows an append-only handle may not change the file size: trimming a torn tail failed with "Access is denied", so a database whose WAL had a torn tail could not be opened. Truncate and sync through a separate read-write handle instead.
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes from a review of the storage engine, plus a new on-disk format (version 2). Two commits; reviewing them in order is easiest.
1.
fix: crash recovery and B+ tree mergeClose,Checkpointand snapshots now refuse after a halt, so the next open replays the WAL as the halt message says.ErrMetaPageDamagedunless the whole file is blank.Scanreads in chunks and no longer holds the engine read lock while streaming to a slow client.ErrCorruptPageinstead of panicking; replication catch-up stops at the fsynced LSN.2.
feat(storage): format v2 with page checksums, page reuse and saved mapsErrPageChecksum).beastctl backup verifychecks every page.*.v1.bak. The v0 migration now uses the same crash-safe install.wal.ErrCorruptLog.-wal-truncate-corrupt/BEASTDB_WAL_TRUNCATE_CORRUPT=truecuts the log at the damage instead.Docs:
docs/architecture.md(format and invariants) anddocs/runbooks/disaster-recovery.md(handling damaged files).Reviewer notes
*.v1.bakcopies.Testing
go build ./... && go vet ./... && go test ./... -racepass.