Repository navigation
feat(ingest): give the derived indexes working CLIs - #541
Merged
Merged
Conversation
The server tells operators to run `python -m ingest.index_theorem_names` when the theorem-name store is empty or was dropped by a schema bump, and `python -m ingest.index_equations` to re-index stale equation trees. None of the three derived-index modules had an entry point, so both commands imported the module and exited 0 having done nothing. Each module now runs as a command, defaulting to the LanceDB root the server reads: - index_theorem_names rebuilds the SQLite index from the chunks table, at the path the server opens by default. - index_definitions indexes every paper in the chunks table, or the ids given. - index_equations fills missing MathML trees. A new --rebuild flag recomputes every tree, which is what the stale-tree advice needs: without it the indexer skipped any row that already had a tree. The server comment now names the flag. New tests run the real `python -m` commands against temporary LanceDB data and check what they wrote; all of them fail on the unfixed modules. Refs #310: bulk ingest still builds none of these indexes itself.
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The server tells operators to run
python -m ingest.index_theorem_nameswhen the theorem-name store is empty or was dropped by a schema bump (server/resources.py:2162,server/theorem_names_store.py:249), andpython -m ingest.index_equationsto re-index stale equation trees (server/retrieval/equations.py:86). None of the three derived-index modules had an entry point, so both commands imported the module and exited 0 having done nothing.Each module now runs as a command, defaulting to the LanceDB root the server reads (
ingest.store.DEFAULT_LANCEDB_PATH):index_theorem_namesrebuilds the SQLite index from the chunks table, at the path the server opens by default (sqlite/theorem_names.dbbeside the LanceDB root).--db-pathoverrides it.index_definitionsindexes every paper in the chunks table, or only the ids given.index_equationsfills missing MathML trees. A new--rebuildflag recomputes every tree. That is what the stale-tree advice needs: without it the indexer skipped any row that already had a tree, so a normalizer bump could not be repaired even from code. The server comment now names the flag.Refs #310. It is not closed: bulk ingest still builds none of these indexes itself, which is the issue's second half (a post-loop build seam in
run_bulk_ingest) and a larger change.Verification
Run on Ubuntu (Linux) at
c3cb306, on top ofmainatd9ddaa8.tests/test_index_clis.py, 5 tests. All of them fail on the unfixed modules.python -m ingest.index_theorem_namesbuilds the index from a temporary chunks table at the path the server reads, with the expected theorem names.python -m ingest.index_equationsfills a missing tree and leaves an existing one alone.--rebuildreplaces a stale tree.python -m ingest.index_definitions --helpruns, and the definitions CLI (called in-process with the preamble directory patched) indexes every paper in the chunks table.tests/test_theorem_names.py,test_definitions_index.py,test_notebook_durability.py,test_equation_index.pyandtest_equation_latex_route.pypass with 0 failures.ruff check .is clean.pytest: the same 214 failures and errors as plainmainin this sandbox, and none new. All 214 come fromBAAI/bge-m3being unreachable here. A workstationmake testis still needed before merge.Notes
mainand touches none of fix: eight small findings from the architecture review #538's files, so the two merge in either order.index_definitionsreads preambles fromingest.preamble.PREAMBLE_DIR, a module constant under the repo'svar/, the same as before. It has no notebook-aware path yet (see Theparsed_dirfootgun the CLI comment claims to have closed is still live in the Python API: ar5iv writes to the caller's parsed_dir while chunk_paper reads a module constant #362 for that class of problem).index_theorem_names_all_papersmaterializes the full chunks table, embeddings included, before grouping by paper. For an offline command that is tolerable, but it is the same pattern as get_paper materializes up to 10,000 full chunk rows — every column, including the fixed-size 1024-float embedding columns — on EVERY call, to read four scalars #353 and worth a projection later.Generated by Claude Code