Skip to content

#551 phase 1: move the theorem-name index into the dataset dir #553

Closed

Description

@chris-dare-dev

Part of #551 (design in its comment).

Change

  • theorem_names.db moves from the global index/sqlite/theorem_names.db to <lancedb_path>/derived/theorem_names.db. Config derives it from lancedb_path unless ARXMCP_THEOREM_NAMES_DB_PATH is set explicitly, as it already does for the notebook cache and BM25 roots.
  • python -m ingest.index_theorem_names defaults to the same path.
  • Notebook ingest builds it into the notebook dataset, alongside definitions (feat(tools): build definitions during notebook ingest #549). This is safe now that each dataset has its own index.
  • ops/checkpoint_notebooks_db.backup_set_databases also checkpoints *.db files under a derived/ directory inside a backed-up dataset.

Fixes along the way

Done when

  • A notebook-routed server reads the theorem names of its own dataset only.
  • Notebook ingest produces the index.
  • Cutover moves the index with the chunks: a swap test sees the index travel with its dataset.
  • An absent index still takes in_memory_scan_fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:graphKuzu citation graph and its ingestarea:ingestIngest pipeline: fetch, parse, chunk, embed, indextype:taskConcrete unit of work under an epic/milestone

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions