Repository navigation
Indexing pipeline + incremental maintenance #6
Description
Refs #1.
ir.index.build(source):
- load ledger; for each
(id, raw)in scope computeversion = change_signal; skip if unchanged (content-hash CRUD per ir_02 / ScaleMCP). - else
decompose-> surfaces; embed (cached); upsert records into store; update ledger. - deletions: artifacts in ledger but not in scope -> drop their records (full-refresh mode).
Light pipeline (no artifact-graph at these sizes; note the ef.artifact_graph upgrade path). Reuse ef.content_hash. Idempotent; re-run is a near-no-op.
Activity
thorwhalen commentedon Jun 5, 2026
Decisions (done in PR #14):
ef.artifact_graph: at these corpus sizes the graph is overkill; each artifact'sef.content_hashis compared to the ledger and only new/changed/re-modeled artifacts are decomposed + embedded. Idempotent; re-run is a near-no-op. The artifact-graph remains the documented upgrade for corpora large enough that recomputation dominates.embedder_idper artifact; a mismatch marks the artifact changed (delete-then-write its records). No global clear needed.chunk_size, hard-split oversized paragraphs) → 16,836. Regression-tested.Review fixes: strategy-change detection, USE_TF, dim guard, DI