Skip to content

feat(tools): build definitions during notebook ingest - #549

Merged
chris-dare-dev merged 1 commit into
mainfrom
claude/bulk-ingest-derived-indexes
Nov 15, 2026
Merged

chris-dare-dev merged 1 commit into
mainfrom
claude/bulk-ingest-derived-indexes

Conversation

@chris-dare-dev

@chris-dare-dev chris-dare-dev commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

What

Refs #310, its second half: ingest should build the derived indexes itself.

get_definitions reads a notebook's own LanceDB dataset, but tools/notebook_ingest.py built only BM25 there. Every freshly ingested notebook therefore answered index_status="absent" until someone ran python -m ingest.index_definitions by hand. Notebook ingest now builds the definitions table right after BM25, over every paper in the notebook's chunks table.

The "every paper in the chunks table" logic moves out of the index_definitions CLI (#541) into index_definitions_for_corpus, which both the CLI and notebook ingest call.

Why this is not the whole of #310

The issue suggests building BM25, definitions, theorem names and equations at the end of run_bulk_ingest. As written that is unsafe, and two indexes have nothing to build from yet:

#310 stays open until #550 and #551 land.

Verification

Run on Ubuntu (Linux) at c2b39ec, on top of main at e2357bd.

  • New test: tests/test_notebook_ingest.py drives notebook_ingest.run end to end. Bulk ingest is faked to leave what a successful run leaves (a real chunks table and its corpus-version marker), and the BM25 build runs for real. The test expects a definitions table holding both papers. The unfixed run fails with Table 'definitions' was not found.
  • Existing tests: five places in tests/tools/test_notebook_scripts.py stub build_bm25_index, and their fakes leave no chunks table. Those places now stub the definitions step the same way, since those tests are about BM25. tests/tools/ and the new file run 99 tests with 0 failures, 4 skipped as before.
  • tests/test_index_clis.py and tests/test_definitions_index.py pass after the refactor.
  • ruff check . is clean. Full pytest here shows no failures beyond the known model-download set.
  • Before merge: a workstation make test.

Notes

  • Failure behaviour: a failing definitions build propagates out of run, as a failing BM25 build already did, so it is never swallowed into a successful exit.
  • Preamble path: definitions read preambles from ingest.preamble.PREAMBLE_DIR, which is keyed by paper_id and shared across notebooks, as before.

get_definitions reads a notebook's own LanceDB dataset, but notebook
ingest built only BM25 there, so every freshly ingested notebook
answered index_status="absent" until someone ran the indexer by hand.
tools/notebook_ingest.py now builds the definitions table right after
BM25, over every paper in the notebook's chunks table.

The "every paper in the chunks table" logic moves out of the
index_definitions CLI into index_definitions_for_corpus, which both
the CLI and notebook ingest call.

The other derived indexes are deliberately not built here:

- Theorem names live in one global SQLite file (#300). Building it
  from a notebook would mix that notebook's chunk ids into the shared
  index.
- Equation trees need an equations table, and no ingest path extracts
  equations yet.
- The staging bulk-ingest path is unchanged. Cutover swaps only the
  LanceDB directory, so an index built beside staging (theorem names,
  or BM25 under a per-dataset version number) would change what the
  live server reads before cutover, and rollback would not undo it.

A new test drives notebook_ingest.run end to end with bulk ingest
faked to leave a real chunks table and marker. It checks that the
definitions table holds both papers; the unfixed run leaves no table.
The existing tests that stub the BM25 build now stub this step too.

Refs #310.
@chris-dare-dev
chris-dare-dev merged commit d1906ba into main Nov 15, 2026
1 check passed
@chris-dare-dev
chris-dare-dev deleted the claude/bulk-ingest-derived-indexes branch November 15, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants