Repository navigation
feat(tools): build definitions during notebook ingest - #549
Merged
Merged
Conversation
get_definitions reads a notebook's own LanceDB dataset, but notebook ingest built only BM25 there, so every freshly ingested notebook answered index_status="absent" until someone ran the indexer by hand. tools/notebook_ingest.py now builds the definitions table right after BM25, over every paper in the notebook's chunks table. The "every paper in the chunks table" logic moves out of the index_definitions CLI into index_definitions_for_corpus, which both the CLI and notebook ingest call. The other derived indexes are deliberately not built here: - Theorem names live in one global SQLite file (#300). Building it from a notebook would mix that notebook's chunk ids into the shared index. - Equation trees need an equations table, and no ingest path extracts equations yet. - The staging bulk-ingest path is unchanged. Cutover swaps only the LanceDB directory, so an index built beside staging (theorem names, or BM25 under a per-dataset version number) would change what the live server reads before cutover, and rollback would not undo it. A new test drives notebook_ingest.run end to end with bulk ingest faked to leave a real chunks table and marker. It checks that the definitions table holds both papers; the unfixed run leaves no table. The existing tests that stub the BM25 build now stub this step too. Refs #310.
This was referenced Nov 15, 2026
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Refs #310, its second half: ingest should build the derived indexes itself.
get_definitionsreads a notebook's own LanceDB dataset, buttools/notebook_ingest.pybuilt only BM25 there. Every freshly ingested notebook therefore answeredindex_status="absent"until someone ranpython -m ingest.index_definitionsby hand. Notebook ingest now builds the definitions table right after BM25, over every paper in the notebook's chunks table.The "every paper in the chunks table" logic moves out of the
index_definitionsCLI (#541) intoindex_definitions_for_corpus, which both the CLI and notebook ingest call.Why this is not the whole of #310
The issue suggests building BM25, definitions, theorem names and equations at the end of
run_bulk_ingest. As written that is unsafe, and two indexes have nothing to build from yet:index/sqlite/theorem_names.db; see Notebook isolation is partial: the Kùzu graph and the theorem-name index are global while everything else is per-notebook #300). Building it from a notebook would mix that notebook's chunk ids into the shared index.ingest/extract_equations.pyis standalone). Filed as No ingest path extracts equations, so find_equation's TED route never fires #550.ops/cutover.pyswaps only the LanceDB directory. Theorem names live outside it, and BM25 pickles are keyed by a corpus version that is per-dataset, not unique. Building either from staging would change what the live server reads before cutover, and a rollback would not undo it. Filed as Derived indexes outside the LanceDB dir are not covered by cutover #551.#310 stays open until #550 and #551 land.
Verification
Run on Ubuntu (Linux) at
c2b39ec, on top ofmainate2357bd.tests/test_notebook_ingest.pydrivesnotebook_ingest.runend to end. Bulk ingest is faked to leave what a successful run leaves (a real chunks table and its corpus-version marker), and the BM25 build runs for real. The test expects adefinitionstable holding both papers. The unfixed run fails withTable 'definitions' was not found.tests/tools/test_notebook_scripts.pystubbuild_bm25_index, and their fakes leave no chunks table. Those places now stub the definitions step the same way, since those tests are about BM25.tests/tools/and the new file run 99 tests with 0 failures, 4 skipped as before.tests/test_index_clis.pyandtests/test_definitions_index.pypass after the refactor.ruff check .is clean. Fullpytesthere shows no failures beyond the known model-download set.make test.Notes
run, as a failing BM25 build already did, so it is never swallowed into a successful exit.ingest.preamble.PREAMBLE_DIR, which is keyed bypaper_idand shared across notebooks, as before.