Skip to content

feat(ingest): keep BM25 index in its dataset - #557

Merged
chris-dare-dev merged 1 commit into
mainfrom
claude/bm25-in-dataset
Nov 15, 2026
Merged

chris-dare-dev merged 1 commit into
mainfrom
claude/bm25-in-dataset

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

What

Moves every BM25 root to <lancedb>/derived/bm25. Before, the shared root was index/bm25 and each notebook's was notebooks/<slug>/index/bm25. A corpus_version is unique only within its dataset, so the new layout makes the shared-v49 versus notebook-v49 collision impossible. Cutover and rollback now move the index with its chunks. This is phase 2 of #551.

Closes #554

Verification

  • ruff check . is clean.
  • These tests fail on main and pass here:
    • Two datasets at the same version keep separate indexes.
    • The shared and notebook-routed Config roots derive from the dataset.
    • Notebook startup auto-builds inside the served dataset.
    • The installed-mode ApplicationPaths roots.
    • Notebook ingest builds into the dataset.
  • The cutover travel test now also checks that a derived/bm25/v<N> directory swaps on cutover and is restored on rollback.
  • Full pytest on Linux has 1 failure, test_symlink_loop_is_rejected, which also fails on main (Python 3.13). No BM25 artifacts appear under the repo's var/ after the run, so removing the conftest hook leaks nothing.

Notes

  • Removed BM25_INDEX_ROOT and its autouse conftest hook. Each test dataset now carries its own index, so the hook had nothing left to isolate.
  • Removed test_cutover_builds_no_global_bm25: the global root it guarded no longer exists.
  • Upgrade: indexes under the old roots are no longer read, and the server rebuilds the missing version on its next start. The old index/bm25 and notebooks/*/index/bm25 directories can be deleted by hand.
  • Next: #551 phase 3: build derived indexes into staging before cutover #555 builds the derived indexes into staging before the swap.

Generated by Claude Code

BM25 indexes lived under index/bm25/v<N> for the shared corpus and
notebooks/<slug>/index/bm25/v<N> per notebook. A corpus_version is
only unique within its dataset, so a root shared across datasets could
serve one dataset's index for another's version, and a notebook cutover
swapped the chunks while leaving their BM25 index behind.

The root is now <lancedb>/derived/bm25 for every dataset. Config and
ApplicationPaths derive it from the dataset unless
ARXMCP_BM25_INDEX_ROOT is set; build_bm25_index and the server's
auto-build default to it through resolve_bm25_root. The global
BM25_INDEX_ROOT, and the test hook that patched it, are gone: each test
dataset now carries its own index.

Upgrade: indexes under the old roots are no longer read. The server
rebuilds the missing version on its next start.

Closes #554
@chris-dare-dev
chris-dare-dev merged commit 4394078 into main Nov 15, 2026
1 check passed
@chris-dare-dev
chris-dare-dev deleted the claude/bm25-in-dataset branch November 15, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

#551 phase 2: move BM25 index roots into the dataset dir

2 participants