Skip to content

feat(ingest): build derived indexes into staging - #558

Merged
chris-dare-dev merged 1 commit into
mainfrom
claude/derived-indexes-in-staging
Nov 15, 2026
Merged

chris-dare-dev merged 1 commit into
mainfrom
claude/derived-indexes-in-staging

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

What

Every producer of a staged dataset now builds the dataset's BM25, definitions and theorem-name indexes inside it before it reports the dataset ready:

Producer When it builds
run_bulk_ingest once at least one paper landed
run_re_embed before the complete sentinel
run_delta before it advances its harvest state

The two cutover scripts promote and roll back the indexes together with the chunks, without knowing the indexes exist. This is phase 3 of #551, which it completes.

Closes #555
Closes #310
Closes #551

Verification

  • ruff check . is clean.
  • tests/test_derived_indexes.py:
    • A staged dataset ends up with BM25, definitions and theorem names inside it, and nothing beside it.
    • Rebuilding gives the same result.
    • A cutover then rollback with ops/cutover.py leaves the live dataset's indexes byte-identical to before, and the server config still points at them.
  • Wiring tests:
    • bulk ingest builds into the dataset it wrote, and builds nothing when no paper landed;
    • re-embed builds while the sentinel still says in_progress;
    • delta builds before last_successful_run_utc is written.
  • Full pytest on Linux has 1 failure, test_symlink_loop_is_rejected, which also fails on main (Python 3.13).

Notes


Generated by Claude Code

The producers of a staged dataset wrote chunks only. Its BM25,
definitions and theorem-name indexes were built later, if at all, so a
cutover could promote chunks that nothing indexed.

ingest.derived_indexes.build_derived_indexes builds all three inside
the dataset for the version in its marker. run_bulk_ingest calls it
once a paper has landed, run_re_embed before it marks the staging run
complete, and run_delta before it advances its harvest state. Because
the indexes now live in the dataset, ops/cutover.py and
tools/notebook_cutover.py promote and roll them back with the chunks
without knowing they exist. notebook_ingest drops its own index builds,
which run_bulk_ingest now makes. python -m ingest.derived_indexes
rebuilds them by hand.

The equations table is still produced by extraction, not here.

Closes #555
Closes #310
Closes #551
@chris-dare-dev
chris-dare-dev merged commit 433ba41 into main Nov 15, 2026
1 check passed
@chris-dare-dev
chris-dare-dev deleted the claude/derived-indexes-in-staging branch November 15, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants