Skip to content

Engine review: crash-recovery fixes and on-disk format v2 - #8

Merged
ChromaBeast merged 3 commits into
mainfrom
fix/engine-review-findings
Nov 16, 2026
Merged

ChromaBeast merged 3 commits into
mainfrom
fix/engine-review-findings

Conversation

@ChromaBeast

Copy link
Copy Markdown
Owner

Summary

Fixes from a review of the storage engine, plus a new on-disk format (version 2). Two commits; reviewing them in order is easiest.

1. fix: crash recovery and B+ tree merge

  • Torn WAL tail lost later writes. Recovery stopped at a torn record but never removed it, so records appended afterwards sat behind it and the next replay never reached them (acknowledged writes lost on the second crash). WAL open now truncates to the last valid record. A zero-filled tail no longer makes the database unopenable.
  • WAL header checksum. Records now checksum the header (LSN, type, lengths) as well as the payload. Old records still decode.
  • Leaf merge removed the wrong separator. Merges deleted the parent entry by the leaf's first key, which stops matching once that key is deleted; the parent kept routing to the dead leaf, and scans returned deleted and stale entries. Entries are now removed by child pointer.
  • Halted engine checkpointed anyway. Close, Checkpoint and snapshots now refuse after a halt, so the next open replays the WAL as the halt message says.
  • Followers did not log replicated batches in their own WAL, so they were lost on a follower crash.
  • A blank page 0 deleted the data file. It now fails with ErrMetaPageDamaged unless the whole file is blank.
  • gRPC Scan reads in chunks and no longer holds the engine read lock while streaming to a slow client.
  • Corrupt slots return ErrCorruptPage instead of panicking; replication catch-up stops at the fsynced LSN.

2. feat(storage): format v2 with page checksums, page reuse and saved maps

  • Page checksums: a CRC-32C trailer on every page, verified on every read (ErrPageChecksum). beastctl backup verify checks every page.
  • Migration: version 1 databases are converted on first open; the new file is swapped in with one rename and the old files are kept as *.v1.bak. The v0 migration now uses the same crash-safe install.
  • Page reuse: a free list saved with each checkpoint; merged leaves, emptied index nodes (including root collapse) and empty data pages are reused before the file grows.
  • Saved maps: the free-space map is saved at every checkpoint; the search index is saved on clean shutdown and loaded instead of a full scan.
  • Mid-log WAL damage (intact records after a damaged one) now fails startup with wal.ErrCorruptLog. -wal-truncate-corrupt / BEASTDB_WAL_TRUNCATE_CORRUPT=true cuts the log at the damage instead.
  • Startup failures no longer checkpoint half-recovered state.

Docs: docs/architecture.md (format and invariants) and docs/runbooks/disaster-recovery.md (handling damaged files).

Reviewer notes

  • No downgrade. Older binaries cannot open a version 2 file; the way back is the *.v1.bak copies.
  • Followers need a resync after a leader migrates. LSNs continue past the old ones, so a follower reports a gap (loudly) and must be restored from a backup.
  • Migration is tested on synthetic version 1 files only (including large chained values and a WAL tail). Please try it on a copy of a real data directory before deploying.
  • gRPC scans are no longer one consistent snapshot across chunks.
  • The search index is still memory-resident at runtime; saving it removes the startup scan after a clean shutdown, at the cost of a longer shutdown.
  • Sparse (non-empty) internal nodes are not merged; only empty ones are removed.

Testing

  • go build ./... && go vet ./... && go test ./... -race pass.
  • New tests reproduce each fixed bug, plus: page checksum detection (bit flip, misdirected page, damaged page 0), free list reuse and damaged-list handling, page chains, randomized insert/delete against a model, file size under delete/refill churn, saved-map and search-index restore (clean shutdown and crash), v1→v2 migration, and mid-log WAL damage.

Durability and recovery
- WAL open truncates a torn or zero-filled tail. Records appended after a
  torn tail used to land behind it, where the next replay never reached
  them, losing acknowledged writes. Bytes that are not a plain torn tail
  are saved to <wal>.discarded before the cut.
- WAL records checksum their header as well as the payload (flag bit in
  the type byte; records without it still decode).
- A halted engine no longer checkpoints, snapshots or checkpoints on
  Close, so the next open redoes the WAL as the halt message promises.
- Followers WAL-log replicated batches; they were applied in memory only.
- A blank page 0 in front of real data fails the open with
  ErrMetaPageDamaged instead of deleting and recreating the data file.
- A failed secondary index rebuild fails the open.

B+ tree
- Leaf merges remove the parent entry by child pointer. Removing by the
  leaf's first key left the parent routing to the dead leaf once that key
  had been deleted, so scans returned deleted and stale entries. The right
  sibling now comes from the parent, not the leaf chain.

Other
- gRPC Scan reads in chunks and releases the engine lock before sending,
  so a slow client cannot block writes.
- GetTuple returns ErrCorruptPage for slots outside the page.
- Replication catch-up stops at the leader's fsynced LSN.
On-disk format version 2
- Every page ends with a CRC-32C trailer over its page ID and contents,
  stamped when a page goes to the journal or spill file and verified on
  every read. Damaged or misdirected pages surface as ErrPageChecksum
  instead of being parsed. ReadPage now zeroes the part of the buffer past
  the end of the file instead of leaving the caller's old bytes there.
- Version 1 files are migrated on first open: records of the last
  checkpoint image are copied into a new file, the WAL tail is redone, and
  the new file replaces the old one in a single rename (old files kept as
  *.v1.bak). LSNs continue past the old ones. The v0 migration uses the
  same crash-safe install step; before, a crash between its renames left
  no data file at all.

Page reuse
- A free list threaded through freed pages, with its head saved in the
  meta page at each checkpoint. Merged-away leaves, internal nodes left
  without keys (including root collapse) and data pages whose last tuple
  is deleted go on it; NewPage takes from it before growing the file. A
  list that does not check out is abandoned, never followed.

Saved maps
- The free-space map is saved at every checkpoint, so space freed before
  a restart is still reused after it.
- A clean shutdown saves the search index; the next open loads it instead
  of scanning every record. After a crash it is rebuilt as before.

WAL
- Opening fails with wal.ErrCorruptLog when an intact record follows a
  damaged one (mid-log damage, not a torn tail). -wal-truncate-corrupt /
  BEASTDB_WAL_TRUNCATE_CORRUPT cuts the log at the damage instead.

Also
- A failure while opening (index rebuild, WAL redo) closes the files
  without a checkpoint instead of persisting half-recovered state.
- beastctl backup verify checks every page checksum of a v2 bundle.
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-16T14:46:19.507333Z
We couldn’t complete this review. Comment @codex review to try again.
7c2e17f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

The WAL is open with O_APPEND, and on Windows an append-only handle may not
change the file size: trimming a torn tail failed with "Access is denied",
so a database whose WAL had a torn tail could not be opened. Truncate and
sync through a separate read-write handle instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant