Skip to content

#551 phase 2: move BM25 index roots into the dataset dir #554

Closed

Description

@chris-dare-dev

Part of #551 (design in its comment).

Change

  • The default BM25 root moves from index/bm25 (shared) and notebooks/<slug>/index/bm25 (per notebook) to <lancedb_path>/derived/bm25, keeping the v<N> subdirectories.
  • Config.bm25_index_root derives from lancedb_path unless set explicitly.
  • tools/notebook_ingest.py builds into the same place.

Once a version number is scoped to the dataset that holds it, it is unique, so the shared-v49 versus notebook-v49 collision recorded in tools/notebook_cutover.py becomes impossible. Server auto-build on a missing version is unchanged; it simply writes into the new root.

Done when

  • The server reads and auto-builds under <lancedb>/derived/bm25.
  • Two datasets at the same version cannot see each other's index.
  • A notebook cutover moves the BM25 index with the chunks, so notebook_cutover.py no longer needs its "BM25 is NOT built here" carve-out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ingestIngest pipeline: fetch, parse, chunk, embed, indexarea:retrievalRetrieval lanes and the 3-tier cachetype:taskConcrete unit of work under an epic/milestone

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions