Skip to content

Derived indexes outside the LanceDB dir are not covered by cutover #551

Closed
3 of 3 issues completed
Closed

Description

@chris-dare-dev

Severity medium · Subsystem ingest / ops

Problem

ops/cutover.py promotes a staging corpus by swapping the LanceDB directory (os.rename(active → rollback), then os.rename(staging → active)), and rollback is the inverse swap. That makes cutover atomic for everything inside the dataset: chunks, definitions, equations. It does nothing for derived indexes that live outside it:

  • Theorem names: index/sqlite/theorem_names.db is one global file. ingest.index_theorem_names.default_db_path(staging) resolves to the same path as for the live dataset, because both share the parent index/.
  • BM25: pickles live under index/bm25/v<corpus_version>/. Corpus versions are per-dataset MVCC, not globally unique, so staging and live can carry the same number.

So no safe point exists to build these for a staged corpus:

  • Before cutover, they overwrite what the live server reads, so it serves an index describing chunks it does not have.
  • After cutover, a rollback restores the old chunks but not the old indexes.

This is why #310's second half (#549) builds only the definitions index during notebook ingest, and leaves the staging bulk path alone.

Suggested remedy

Pick one of these, as a design in this issue first:

  1. Move them inside the dataset directory. For example lancedb/../derived/ swapped as a unit, or a sibling directory renamed in the same cutover step. Then cutover and rollback cover them as they cover definitions.
  2. Key them by something globally unique. For example a content hash of the corpus marker, not the per-dataset version number. Then staging and live can never collide, and the server picks by key.
  3. Have cutover rebuild them after the swap, and rollback rebuild them again. Simplest, but it puts a long rebuild inside the cutover window.

Related: #300 (theorem-name index is global, not per notebook) and #360 (the full-table scan when that index is absent).

Done when

  • run_bulk_ingest can build every derived index for a staged corpus without touching what the live server reads.
  • A cutover-then-rollback test leaves the live server's indexes exactly as they were before the cutover.

Activity

  1. added
    type:findingAdversary/critique finding to remediate
    area:ingestIngest pipeline: fetch, parse, chunk, embed, index
    area:opsOps, observability, backup, cutover, infra
    on Nov 15, 2026
  2. chris-dare-dev commented on Nov 15, 2026

    @chris-dare-dev
    OwnerAuthor

    Decision: option 1, each derived index lives inside the dataset directory it was derived from

    Cutover, rollback, backup and notebook isolation then all come from the os.rename the system already does, with no new mechanism. Analysis against main at 1a87ac1.

    Where each index lives today

    Index Derived from Location Built by Moved by cutover?
    definitions, equations chunks dataset inside lancedb/ (LanceDB tables) offline CLIs (#541, #549) yes
    corpus-version.json chunks dataset inside lancedb/ ingest yes
    BM25 chunks dataset index/bm25/v<N>, or notebooks/<slug>/index/bm25/v<N> the server auto-builds a missing version at startup or rebind no
    theorem names chunks dataset index/sqlite/theorem_names.db, global offline CLI only (after #552) no
    Kùzu graph OpenAlex / INSPIRE, not the dataset index/kuzu graph ingest no

    Facts that decide it

    1. LanceDB ignores foreign entries in a dataset directory. Measured with the pinned lancedb: a derived/bm25/v3/ directory and a derived/theorem_names.db file sat next to a chunks table. list_tables() returned only chunks, and add() plus optimize() left both untouched. corpus-version.json already relies on this.
    2. Version numbers are unique only within one dataset. tools/notebook_cutover.py records a live collision: the shared corpus's v49 against a notebook's v49 in the global BM25 root. Per-notebook BM25 roots were the patch.
    3. No code lists a LanceDB directory's entries, in any of server/, ingest/, tools/ or ops/. A derived/ subdirectory cannot be mistaken for a table.

    The three options compared

    1. Inside the dataset dir 2. Content-hash keys 3. Rebuild after swap
    Cutover atomic yes (same rename) yes no: mismatch window
    Rollback exact yes (prev keeps its indexes) only until the old key is GC'd no: rebuild again
    Version collisions gone gone unchanged
    Notebook isolation (#300, theorem names) free still a global namespace unchanged
    Backup (#360) fixed: lancedb/ and notebooks/ are already in the backup set new paths needed unchanged
    Garbage collection pruned with the dataset new GC needed (none exists; #309) none needed
    §4.8 / #359 (server writes nothing) unchanged unchanged puts a rebuild in the cutover window or in the server

    Option 2 is correct, but it adds two mechanisms (key lookup and GC) and still leaves notebook isolation unsolved. Option 3 keeps the mismatch window and turns rollback into a rebuild.

    Shape of the change

    Phases

    Each phase is filed as a sub-issue:

    1. Theorem-name index moves into the dataset directory. This also fixes Notebook isolation is partial: the Kùzu graph and the theorem-name index are global while everything else is per-notebook #300 for theorem names and The absent-index fallback for find_lemma_by_name materializes the entire chunks table into memory on every call, and theorem_names.db is outside the backup set that would prevent reaching it #360's backup half.
    2. BM25 index roots move into the dataset directory.
    3. Offline builds write into staging before promotion, plus a cutover-then-rollback test.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ingestIngest pipeline: fetch, parse, chunk, embed, indexarea:opsOps, observability, backup, cutover, infrasev:mediumMedium severitytype:findingAdversary/critique finding to remediate

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions