Serve: auto-recover LMDB format corruption (sequential wipe + rebuild) - #260
Merged
Merged
Conversation
… + rebuild Symbol-rebuild failures of the MDB_BAD_VALSIZE class (data written by an older storage-major, observed on all C# repos after the arroy 0.8/heed 0.22 upgrades) now queue the repo for automatic recovery: evict stores (remove_repo's sequence, minus unregister), wipe the DB dir with the bounded lock-retry, and force-reindex through the TUI machinery whose store-open path recreates fresh formats. Recoveries are processed strictly one repo at a time (single worker, flag+queue under one lock so no lost wake-up); read-only repos are skipped with a pointer to the owning writer. Detection helper is unit-tested; queue dedup + worker start pinned by test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the dependency wave (#249: arroy 0.5→0.8, heed 0.20→0.22). After deploying a binary built on the new storage majors, every repo whose on-disk DB was written by the old binary fails its symbol rebuild with
MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size— observed on all C# repos. The tantivy FTS side already self-heals (graceful reset, #253); the vector/symbol stores had no such recovery and stayed red until an operator force-reindexed by hand.What
ServeState::is_lmdb_format_corruptionclassifies the error chain (MDB_BAD_VALSIZE/Unsupported size of key/wrong DUPFIXED size, case-insensitive) in the symbol-rebuild failure path.remove_repo(stop FSW → evict → await watcher/index shutdown) minus the unregister, wipes the DB dir with the same bounded lock-retry, and starts a force reindex viaspawn_force_reindex— whosetry_open_storesrecreates fresh stores on the new formats.is_indexing, generous cap) before starting the next, because every rebuild is a full CPU-bound embed pass. Queue + started-flag are checked under one lock so a worker exit can never race an enqueue into a lost wake-up.Tests
is_lmdb_format_corruption_matches_known_lmdb_errors— table-driven detection (real LMDB message, heed-wrapped, false positives excluded).enqueue_format_recovery_dedupes_and_starts_single_worker— duplicate enqueues collapse to one entry; worker flag set.Changelog
### Fixedentry under the pending version describing the auto-recovery.