Foundation: corpus-source substrate, persistence, retrieval (3 corpora live)

[thorwhalen]

What this lands

The foundation of ir as an agentic information-retrieval substrate composed on ef + vd — and three of the target corpora working end-to-end. Implements the plan in #1.

Architecture (compose, don't reinvent)

• ef → embedders (HashingEmbedder, sentence_transformers_embedder, CachedEmbedder), content_hash.
• vd → Mongo-style metadata matches_filter (hard filters), RRF (reserved for hybrid).
• dol → the key-value persistence layer (repository pattern).
• ir adds the genuinely-new band: the IndexingStrategy seam, source adapters, retrieval/dedupe.

Modules

module	role
config	XDG dirs (config/data/cache), env-overridable
base	Artifact/Surface/IndexPlan/Record/SearchHit; filter-fields vs surfaces split
store	dol-backed CorpusStore (meta/vectors/ledger/config) + in-memory; brute-force matrix
embed	decent-local MiniLM default + light hashing fallback + disk cache
strategy	IndexingStrategy: WholeText/Chunked/Skill/Package; paragraph-packing chunker
sources	CorpusSource = scope + change-signal + strategy + embedder; from_* constructors
index	incremental content-hash CRUD build + full-refresh prune
retrieve	hard metadata filter + dense brute force + per-artifact dedupe

Defining a corpus source (the core ask)

A source is an abstract strategy + parameters with smart defaults:

ir.CorpusSource.from_md_reports()                 # files → Chunked
ir.CorpusSource.from_skills()                      # priv.skills_index → Skill
ir.CorpusSource.from_packages()                    # .pth manifest → Package (multi-surface)
ir.CorpusSource.from_mapping(d, name="x")          # any mapping → WholeText
corpus = ir.build(source)                          # incremental, idempotent
ir.search(corpus, "query", filter={"owner": "ours"})

Embedding

Default = local all-MiniLM-L6-v2 (384-d, USE_TF=0 to dodge a TF/numpy ABI crash), wrapped in a disk cache keyed by (model, content_hash). embedder="light" → numpy-only hashing for fast tests.

Persistence

Repository pattern over dol MutableMappings under XDG dirs; swap the store to change where/how data persists. Default = local files; brute-force cosine (exact, instant at these sizes).

Validation

MiniLM end-to-end, top-1 correct on sample intents:

• skills (157): "download academic references" → acquire-references; "deploy web app" → tw-deploy.
• packages (1508 surfaces): "facade over vector databases" → vd; name-filter → dol.
• md-reports (16836 chunks): the three ir_0x queries resolve correctly.

22 hermetic tests pass (light embedder; source-adapter tests skip without priv/$PP).

Deferred (tracked)

Hybrid BM25+RRF and reranking seams (#7), selection stage (#11), eval harness (#12), CLI + named-corpus registry (#13). AI-authored synopsis / problem-class surfaces for packages are a documented Package-strategy extension.

Design rationale: misc/docs/ir_04. Closes #2–#10; refs #1, #11, #12, #13.

[thorwhalen]

Adversarial review pass

Ran a 5-dimension code-review workflow (incremental-maintenance · embedding/cache · retrieval math · persistence · API-reuse), each finding verified by an independent skeptic. 15 findings surfaced; triaged against the actual code and applied 5, rejected the rest as verifier over-reach:

Applied (commit ee0c34c):

1. Strategy-change detection (real bug) — the ledger now records a strategy_id; rebuilding a corpus under a different IndexingStrategy on unchanged content was being skipped. Now re-decomposes. +regression test.
2. USE_TF assignment — setdefault couldn't override a shell USE_TF=1; now a hard assignment.
3. Query/index dim guard — mismatched embedder dims now raise a clear error instead of a cryptic shape crash.
4. from_skills(fetcher=...) — injectable for testability (DI). +test.
5. best_per_artifact docstring correction.

Rejected after inspection: the "fewer-surfaces leak" (the delete loop uses prev.record_ids, which already covers it — only non-atomic crash recovery remains, a v1 non-issue, noted for later); the skills/packages whole-mapping hash (it keeps filter fields fresh and re-embeds are cache hits — current behavior is safer); and ~8 ef-internal / intended-behavior / doc-nit findings.

24 tests pass.