Skip to content

#551 phase 3: build derived indexes into staging before cutover #555

Closed

Description

@chris-dare-dev

Part of #551 (design in its comment). Depends on phases 1 and 2.

Change

Once every derived index lives inside its dataset directory, the offline paths that produce a staged dataset can build the indexes into staging before promotion:

  • run_bulk_ingest;
  • tools/re_embed_all.py, for the shared corpus and per notebook.

ops/cutover.py and tools/notebook_cutover.py then promote chunks and indexes together, and rollback restores both. Nothing in either cutover script needs to know the indexes exist.

Done when

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ingestIngest pipeline: fetch, parse, chunk, embed, indexarea:opsOps, observability, backup, cutover, infratype:taskConcrete unit of work under an epic/milestone

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions