Repository navigation
Closed
3 / 33 of 3 issues completed
Copy link
Labels
area:ingestIngest pipeline: fetch, parse, chunk, embed, indexIngest pipeline: fetch, parse, chunk, embed, indexarea:opsOps, observability, backup, cutover, infraOps, observability, backup, cutover, infrasev:mediumMedium severityMedium severitytype:findingAdversary/critique finding to remediateAdversary/critique finding to remediate
Description
Severity medium · Subsystem ingest / ops
Problem
ops/cutover.py promotes a staging corpus by swapping the LanceDB directory (os.rename(active → rollback), then os.rename(staging → active)), and rollback is the inverse swap. That makes cutover atomic for everything inside the dataset: chunks, definitions, equations. It does nothing for derived indexes that live outside it:
- Theorem names:
index/sqlite/theorem_names.dbis one global file.ingest.index_theorem_names.default_db_path(staging)resolves to the same path as for the live dataset, because both share the parentindex/. - BM25: pickles live under
index/bm25/v<corpus_version>/. Corpus versions are per-dataset MVCC, not globally unique, so staging and live can carry the same number.
So no safe point exists to build these for a staged corpus:
- Before cutover, they overwrite what the live server reads, so it serves an index describing chunks it does not have.
- After cutover, a rollback restores the old chunks but not the old indexes.
This is why #310's second half (#549) builds only the definitions index during notebook ingest, and leaves the staging bulk path alone.
Suggested remedy
Pick one of these, as a design in this issue first:
- Move them inside the dataset directory. For example
lancedb/../derived/swapped as a unit, or a sibling directory renamed in the same cutover step. Then cutover and rollback cover them as they cover definitions. - Key them by something globally unique. For example a content hash of the corpus marker, not the per-dataset version number. Then staging and live can never collide, and the server picks by key.
- Have cutover rebuild them after the swap, and rollback rebuild them again. Simplest, but it puts a long rebuild inside the cutover window.
Related: #300 (theorem-name index is global, not per notebook) and #360 (the full-table scan when that index is absent).
Done when
run_bulk_ingestcan build every derived index for a staged corpus without touching what the live server reads.- A cutover-then-rollback test leaves the live server's indexes exactly as they were before the cutover.
Metadata
Metadata
Assignees
Labels
area:ingestIngest pipeline: fetch, parse, chunk, embed, indexIngest pipeline: fetch, parse, chunk, embed, indexarea:opsOps, observability, backup, cutover, infraOps, observability, backup, cutover, infrasev:mediumMedium severityMedium severitytype:findingAdversary/critique finding to remediateAdversary/critique finding to remediate
Activity
chris-dare-dev commentedon Nov 15, 2026
Decision: option 1, each derived index lives inside the dataset directory it was derived from
Cutover, rollback, backup and notebook isolation then all come from the
os.renamethe system already does, with no new mechanism. Analysis againstmainat1a87ac1.Where each index lives today
lancedb/(LanceDB tables)corpus-version.jsonlancedb/index/bm25/v<N>, ornotebooks/<slug>/index/bm25/v<N>index/sqlite/theorem_names.db, globalindex/kuzuFacts that decide it
derived/bm25/v3/directory and aderived/theorem_names.dbfile sat next to achunkstable.list_tables()returned onlychunks, andadd()plusoptimize()left both untouched.corpus-version.jsonalready relies on this.tools/notebook_cutover.pyrecords a live collision: the shared corpus'sv49against a notebook'sv49in the global BM25 root. Per-notebook BM25 roots were the patch.server/,ingest/,tools/orops/. Aderived/subdirectory cannot be mistaken for a table.The three options compared
prevkeeps its indexes)lancedb/andnotebooks/are already in the backup setOption 2 is correct, but it adds two mechanisms (key lookup and GC) and still leaves notebook isolation unsolved. Option 3 keeps the mismatch window and turns rollback into a rebuild.
Shape of the change
<lancedb>/derived/theorem_names.dband<lancedb>/derived/bm25/v<N>/. Per-version BM25 directories stay valid within a dataset. Config derives both paths fromlancedb_path.python -m ingest.index_theorem_namesonce, and the oldindex/sqlite/andindex/bm25/become deletable.backup_set_databases(ops/checkpoint_notebooks_db.py) must also checkpoint*.dbfiles underlancedb/derived/.Phases
Each phase is filed as a sub-issue:
Generated by Claude Code