Skip to content

Serve: auto-recover LMDB format corruption (sequential wipe + rebuild) - #260

Merged
flupkede merged 1 commit into
developfrom
fix/lmdb-format-auto-rebuild
Sep 16, 2026
Merged

flupkede merged 1 commit into
developfrom
fix/lmdb-format-auto-rebuild

Conversation

@flupkede

Copy link
Copy Markdown
Owner

Follow-up to the dependency wave (#249: arroy 0.5→0.8, heed 0.20→0.22). After deploying a binary built on the new storage majors, every repo whose on-disk DB was written by the old binary fails its symbol rebuild with MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size — observed on all C# repos. The tantivy FTS side already self-heals (graceful reset, #253); the vector/symbol stores had no such recovery and stayed red until an operator force-reindexed by hand.

What

  • Detection: ServeState::is_lmdb_format_corruption classifies the error chain (MDB_BAD_VALSIZE / Unsupported size of key / wrong DUPFIXED size, case-insensitive) in the symbol-rebuild failure path.
  • Recovery: the affected alias is queued; the recovery worker evicts it exactly like remove_repo (stop FSW → evict → await watcher/index shutdown) minus the unregister, wipes the DB dir with the same bounded lock-retry, and starts a force reindex via spawn_force_reindex — whose try_open_stores recreates fresh stores on the new formats.
  • Sequential, 1 per 1: a single worker drains the queue; it waits for each rebuild to finish (poll is_indexing, generous cap) before starting the next, because every rebuild is a full CPU-bound embed pass. Queue + started-flag are checked under one lock so a worker exit can never race an enqueue into a lost wake-up.
  • Read-only repos are skipped with a pointer to the owning writer (same guard as TUI force reindex).

Tests

  • is_lmdb_format_corruption_matches_known_lmdb_errors — table-driven detection (real LMDB message, heed-wrapped, false positives excluded).
  • enqueue_format_recovery_dedupes_and_starts_single_worker — duplicate enqueues collapse to one entry; worker flag set.

Changelog

### Fixed entry under the pending version describing the auto-recovery.

… + rebuild

Symbol-rebuild failures of the MDB_BAD_VALSIZE class (data written by an
older storage-major, observed on all C# repos after the arroy 0.8/heed
0.22 upgrades) now queue the repo for automatic recovery: evict stores
(remove_repo's sequence, minus unregister), wipe the DB dir with the
bounded lock-retry, and force-reindex through the TUI machinery whose
store-open path recreates fresh formats.

Recoveries are processed strictly one repo at a time (single worker,
flag+queue under one lock so no lost wake-up); read-only repos are
skipped with a pointer to the owning writer. Detection helper is
unit-tested; queue dedup + worker start pinned by test.
@flupkede
flupkede merged commit 74851a5 into develop Sep 16, 2026
7 checks passed
@flupkede
flupkede deleted the fix/lmdb-format-auto-rebuild branch September 16, 2026 21:35
@flupkede flupkede mentioned this pull request Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant