Skip to content

feat(ingest): give the derived indexes working CLIs - #541

Merged
chris-dare-dev merged 1 commit into
mainfrom
claude/derived-index-clis
Nov 15, 2026
Merged

chris-dare-dev merged 1 commit into
mainfrom
claude/derived-index-clis

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

What

The server tells operators to run python -m ingest.index_theorem_names when the theorem-name store is empty or was dropped by a schema bump (server/resources.py:2162, server/theorem_names_store.py:249), and python -m ingest.index_equations to re-index stale equation trees (server/retrieval/equations.py:86). None of the three derived-index modules had an entry point, so both commands imported the module and exited 0 having done nothing.

Each module now runs as a command, defaulting to the LanceDB root the server reads (ingest.store.DEFAULT_LANCEDB_PATH):

  • index_theorem_names rebuilds the SQLite index from the chunks table, at the path the server opens by default (sqlite/theorem_names.db beside the LanceDB root). --db-path overrides it.
  • index_definitions indexes every paper in the chunks table, or only the ids given.
  • index_equations fills missing MathML trees. A new --rebuild flag recomputes every tree. That is what the stale-tree advice needs: without it the indexer skipped any row that already had a tree, so a normalizer bump could not be repaired even from code. The server comment now names the flag.

Refs #310. It is not closed: bulk ingest still builds none of these indexes itself, which is the issue's second half (a post-loop build seam in run_bulk_ingest) and a larger change.

Verification

Run on Ubuntu (Linux) at c3cb306, on top of main at d9ddaa8.

  • New tests: tests/test_index_clis.py, 5 tests. All of them fail on the unfixed modules.
    • The real python -m ingest.index_theorem_names builds the index from a temporary chunks table at the path the server reads, with the expected theorem names.
    • The real python -m ingest.index_equations fills a missing tree and leaves an existing one alone. --rebuild replaces a stale tree.
    • python -m ingest.index_definitions --help runs, and the definitions CLI (called in-process with the preamble directory patched) indexes every paper in the chunks table.
  • Existing tests: tests/test_theorem_names.py, test_definitions_index.py, test_notebook_durability.py, test_equation_index.py and test_equation_latex_route.py pass with 0 failures.
  • ruff check . is clean.
  • Full pytest: the same 214 failures and errors as plain main in this sandbox, and none new. All 214 come from BAAI/bge-m3 being unreachable here. A workstation make test is still needed before merge.

Notes


Generated by Claude Code

The server tells operators to run `python -m ingest.index_theorem_names`
when the theorem-name store is empty or was dropped by a schema bump,
and `python -m ingest.index_equations` to re-index stale equation
trees. None of the three derived-index modules had an entry point, so
both commands imported the module and exited 0 having done nothing.

Each module now runs as a command, defaulting to the LanceDB root the
server reads:

- index_theorem_names rebuilds the SQLite index from the chunks table,
  at the path the server opens by default.
- index_definitions indexes every paper in the chunks table, or the
  ids given.
- index_equations fills missing MathML trees. A new --rebuild flag
  recomputes every tree, which is what the stale-tree advice needs:
  without it the indexer skipped any row that already had a tree. The
  server comment now names the flag.

New tests run the real `python -m` commands against temporary LanceDB
data and check what they wrote; all of them fail on the unfixed
modules.

Refs #310: bulk ingest still builds none of these indexes itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants