Epic: evolve ir toward the ir_09 Composable Search Agent — layering decision + role map

[thorwhalen]

Decision

Layering: hybrid. A new orchestration layer on top owns the agent; ir grows a tight set of retrieval-adjacent seams that each independently improve ir's own non-agentic search. ir stays the retrieval substrate ('retrieval-first, extensible into RAG without being one by default'); the new layer depends ON ir and ir never imports it.

The decision rule for every seam: does this make ir's single-shot, agent-free search measurably better? Yes -> it lands in ir. If it is the back-edge / cross-corpus reasoning that makes a system an agent rather than a DAG -> it lives in the new layer.

Role map (ir_09 §3) vs ir 0.1.11 (verified this session)

Role	ir_09 §3	Status in ir	Where it goes
Retriever (one query, one corpus)	dense/BM25/hybrid/graph/SQL/web	present — ir.retrieve.search (dense/lexical/hybrid)	ir (leaf); graph/SQL/web retrievers -> new layer
Reranker (final ordering)	swappable	present — search(rerank=ef.Reranker), per-corpus	ir at leaf; cross-source rerank at fan-in -> new layer
Evaluator/Critic	relevant + sufficient + refinement	partial — Selection has relevant/abstain + signals	ir owns verdict SHAPE (+ derived sufficient); the loop/refinement -> new layer
Formulator	rewrite/decompose/HyDE/filter-synth	absent — query embedded as-is	ir (identity-default seam; improves ir's own recall)
Planner/Orchestrator	decompose + source-select + budget + terminate	absent	new layer
Citer/Verifier	confirm pointer supports claim	absent (needs generation ir omits)	new layer (srag)

Value types: Result already = SearchHit+Disclosure (pointer-passing, §5, implemented in ir). Query/SubTask/LowLevelQuery/Judgement are agent-layer types (ir keeps a plain str query + a derived sufficient signal).

What ir owns (this epic's ir-side issues)

1. Formulator seam — injectable formulate=, identity default; make_llm_formulator mirroring make_llm_selector. Add an injectable Formulator seam to ir.search / ir.discover (identity default; opt-in rewrite/expand/HyDE) #32
2. ir.as_retriever(corpus) — the corpus->Retriever adapter matching ir_09's leaf contract. Publish a stable Retriever adapter — ir.as_retriever(corpus) -> Callable matching ir_09's leaf contract #33
3. registry.retrievers() view + multi-corpus discover — a read-projection Mapping[name, Retriever] + federated single-shot search. Project the corpus registry into a Mapping[name, Retriever] view + multi-corpus discover (single-shot, no loop) #34
4. sufficient signal on Selection — derived, model-free; refinement stays OUT. Extend Selection toward a Judgement-shaped result — add a derived sufficient signal (keep refinement OUT) #35
5. Injectable resource store on disclose + SearchHit.to_dict() — formalize §5 pointer-passing over dol/URLs; serialization-clean leaf. Make disclosure's resource-store contract explicit and injectable (BodyLoader -> Mapping resource store) + SearchHit.to_dict() #36

Each is strictly additive, keyword-only, defaults to today's behavior, verified against the existing eval harness. None breaks backcompat.

What the new layer owns (stub issue)

Planner/Orchestrator, the control loop with the back-edge (evaluator->reformulate), the budget governor, the live Mapping[name, Retriever] source registry across heterogeneous backends, Citer/Verifier, and the SingleContext-vs-MultiAgent orchestrator split. New-layer stub: #37.

Two load-bearing boundaries (guard in every review)

• Formulate returns QUERIES, never SubTasks. Decomposition into goal+sources is the Planner -> agent layer.
• ir gains a sufficient SIGNAL, never a refinement directive or a loop. The back-edge (ir_09 §4) is what makes it an agent; it must not live in ir.

Build order

Matches ir_09 §8: (1) Retriever adapter + registry view [#2, #3]; (2) new-layer single-context orchestrator with trivial planner + pass-through evaluator wrapping today's search/select/disclose [stub]; (3) Formulator + an LLM Evaluator returning sufficient/refinement, turn on the back-edge [#1 in ir; loop in new layer]; (4) move rerank to fan-in + Citer in srag [new layer]; (5) budget governor + run-log [new layer]; (6) multi-agent orchestrator if breadth justifies [new layer].

Links

• ir_09 design doc: misc/docs/ir_09 -- A Composable Search Agent ...md
• ir-side issues: Add an injectable Formulator seam to ir.search / ir.discover (identity default; opt-in rewrite/expand/HyDE) #32, Publish a stable Retriever adapter — ir.as_retriever(corpus) -> Callable matching ir_09's leaf contract #33, Project the corpus registry into a Mapping[name, Retriever] view + multi-corpus discover (single-shot, no loop) #34, Extend Selection toward a Judgement-shaped result — add a derived sufficient signal (keep refinement OUT) #35, Make disclosure's resource-store contract explicit and injectable (BodyLoader -> Mapping resource store) + SearchHit.to_dict() #36
• New-layer stub: New: orchestration layer on top of ir — the Composable Search Agent (Planner + back-edge loop + multi-corpus + Citer) #37

[thorwhalen]

ir-side seams all shipped ✅ (epic complete on the ir side):

• Publish a stable Retriever adapter — ir.as_retriever(corpus) -> Callable matching ir_09's leaf contract #33 as_retriever + Make disclosure's resource-store contract explicit and injectable (BodyLoader -> Mapping resource store) + SearchHit.to_dict() #36 SearchHit.to_dict / disclose(store=) — PR feat: agent-ready retrieval seams — as_retriever, SearchHit.to_dict, disclose(store=) #39 (ir 0.1.12)
• Project the corpus registry into a Mapping[name, Retriever] view + multi-corpus discover (single-shot, no loop) #34 registry.retrievers() view + Extend Selection toward a Judgement-shaped result — add a derived sufficient signal (keep refinement OUT) #35 Selection.sufficient — PR feat: registry.retrievers() Mapping view + Selection.sufficient signal #40 (ir 0.1.13)
• Add an injectable Formulator seam to ir.search / ir.discover (identity default; opt-in rewrite/expand/HyDE) #32 formulate= seam + make_llm_formulator — PR feat(retrieve): injectable query-formulation seam — formulate= + make_llm_formulator (#32) #41

The agent layer foundation landed in raglab (thorwhalen/raglab#1, raglab 0.2.2): value types + role Protocols + SingleContextAgent (with the back-edge) + a no-LLM thin slice running end-to-end over ir.as_retriever. Remaining epic work is raglab-side (thorwhalen/raglab#2): wire the LLM Formulator (consume ir.make_llm_formulator) + LLM Evaluator (consume Selection.sufficient) to turn on the back-edge, then reranker-at-fan-in, budget/observability, and the MultiAgentAgent. Cross-source merge stays raglab's fan-in (not ir).

[thorwhalen]

Boundary revision: the traversal primitive moves to ir (maintainer-confirmed, 2026-06-12)

The role map assigned "graph/SQL/web retrievers → new layer". Revised for graph specifically, recorded in ADR discussion #43: the traversal primitive — traverse(query, store, policy) over a typed-edge links view, with safety (visited-set, depth cap, node budget) enforced by the operator — lands in ir (#46, #47). It passes this epic's own decision rule: single-shot, agent-free search over linked corpora (synopsis→chunk routing) improves without any loop, exactly like fuse_hits did for federation.

What stays in the agent layer, unchanged: when to traverse, LLM-in-the-loop routing/decomposition, and any policy that loops on model calls. The two load-bearing boundaries are untouched (formulate returns queries; ir emits signals, never directives). SQL/web retrievers remain new-layer.

Related: purpose-centric memory was decided into raglab (thorwhalen/raglab#6 — raglab is the only layer that sees purpose), and retrieval-time context expansion into ir as a hit operation (#44, #45). Full rationale + rejected alternatives in #43.

[thorwhalen]

Progress — ir-side capabilities for the arc are COMPLETE (ir 0.1.22, 2026-06-13).

Both ir-side capabilities the agent consumes are shipped and on PyPI:

• Cap 2 — context expansion: ir#44 (SearchHit.surface_index + ledger-backed sibling addressing, 0.1.17) → ir#45 (expand(hit, corpus, *, policy=) + NeighborhoodPolicy, 0.1.19). ✅
• Cap 1 — linked retrieval / graph v1: ir#46 (typed-edge links view + GraphStore protocol + EdgeExtractor, 0.1.20) → ir#47 (traverse(query, store, policy) + WalkPolicy, operator-enforced safety: visited-set/depth/budget; collapsed_tree_policy first policy, 0.1.21) → ir#48 (with_synopsis — LLM synopsis surfaces as routing fuel; staleness via recursive _strategy_id; lazy oa, 0.1.22). ✅

Boundary held as decided here (#43 decision 2): the traversal primitive lives in ir; when-to-traverse + LLM-in-loop routing stay in raglab.

Remaining arc work is raglab-side (per #43 decision 8 sequencing): budget governor + run-log/observability (raglab#2 item 5, folds in raglab#5) → PurposeStore v0 (raglab#6, consumes the run-log schema). ir cap-3 (purpose memory) deliberately stays out of ir (signal-only substrate).