Eval harness (ir_03) [later]

[thorwhalen]

Refs #1.

Capability-discovery eval on top of ef.evaluation: recall@k / NDCG@k (retrieval), conditional selection accuracy, distractor-robustness curve, failure-mode taxonomy. Back-translation case generation from the registry with function-masking. Inspect-AI-shaped scorers; deepdiff for params.

[thorwhalen]

Dev journal — eval harness, PR 1 (scoring harness, offline)

Direction confirmed: retrieval-centric, reuse ef, scoring-harness-first (no LLM, CI-safe). Case generation (back-translation + name-masking) is deferred to a follow-up; this PR scores a given set of cases.

Decisive design finding

The ir_03 research doc is written for a typed @command registry (name + docstring + typed params → deepdiff/AST/BFCL scoring). ir's corpora are documents (skills/packages/reports) joined on artifact_id, with no argument signatures. So the eval that fits is retrieval of the right artifact, not argument-dict scoring. The deepdiff parameter scorer / BFCL fork are intentionally not built — they belong with a typed-tool registry ir doesn't have (more aligned with the selection issue).

Reuse stance (compose, don't reinvent)

ef.evaluation already provides every retrieval metric as pure, tested functions (recall_at_k, ndcg_at_k, precision_at_k, reciprocal_rank→MRR, average_precision→MAP) plus the BEIR driver evaluate_retrieval, Qrels (native multi-gold), and RetrievalEvalReport. ir.eval imports these — it does not reimplement them. ir's search already returns artifact_id-keyed hits, so the bridge is a ~3-line adapter (query → [artifact_id,…], which ef consumes as bare doc-ids).

What landed (ir/eval.py)

• DiscoveryCase (query + multi-gold artifact_ids; empty gold = abstention) + save_cases/load_cases (JSONL, optional __meta__ header for a corpus-version anchor).
• as_doc_retriever — adapts a corpus to ef's retriever contract.
• retrieval_report — pure-ef path (recall@k / NDCG@k / MRR / MAP). Use to A/B dense vs hybrid.
• evaluate_discovery — single retrieval pass → the same ef metric primitives plus failure-mode taxonomy (hit_rank_1 / surfaced_low_rank / retrieval_miss) and an optional score-threshold abstention proxy. Returns DiscoveryReport.
• distractor_robustness_curve (+ distractor_curve_from_cases) — seeded needle-in-a-haystack accuracy vs catalog size; the one diagnostic ef doesn't provide.
• validate_cases — drift detection (gold ids that left the corpus; corpora are live + machine-specific, so committed cases can fall out of sync).
• CLI: ir eval <corpus> <cases.jsonl> [--mode] [--k] (warns on drift).
• Tests: 16 hermetic tests (light embedder, disjoint-vocab corpus for exact assertions) + a committed demo fixture. Full suite green.

Deliberate scope cuts / fragilities

• Abstention is a selection concern; here it's a retrieval-side score-threshold proxy, clearly labelled. Real abstention belongs with the selection stage.
• Multi-gold is supported (overlapping capabilities) via Qrels grades; single-gold case-gen will need a dedup/cluster step to avoid under-crediting.
• No inspect_ai / deepdiff dependency added (kept core light: numpy/dol/ef/vd/argh). An optional Inspect-AI adapter is a possible later extra.

Refs #1.

[thorwhalen]

Dev journal — eval harness, PR 2 (case generation)

ir.eval_gen adds the build-time generator (back-translation + name-masking + abstention slice + injectable LLM + corpus_signature anchor + ir eval-gen CLI). Generation needs an LLM (oa, lazy-imported); scoring stays offline. PR for the branch is open.

Masking is the load-bearing correctness feature

The name is scrubbed from the description before generation, and an output guard drops any intent that reuses it. Two adversarial-review findings hardened it:

• The line parser stripped meaningful leading characters from real queries (3D→D, -9 degrees→degrees, .env→env) — fixed to strip only genuine list markers.
• The guard required the multi-word name adjacent and in order, so reordered tokens slipped past while BM25 (bag-of-words) still matched. Fixed: whitespace/separator-tolerant masking + a token-set guard aligned to how BM25 actually scores. A single shared content word is not treated as a leak (that's the legitimate semantic signal).

Preliminary finding from a real run (n=8 gold, skills corpus)

Scoring real back-translated, name-masked queries with MiniLM: dense NDCG@10 ≈ 0.85 vs hybrid ≈ 0.69 — dense wins. Masking removes the rare identifiers the BM25 leg of hybrid was designed to exploit, so on paraphrastic discovery queries the lexical leg adds noise and RRF drags the strong dense ranking down. This is corpus/query-dependent: hybrid's advantage is for identifier-bearing queries, which masking deliberately excludes. Needs a full generated set (all artifacts, k=5, judge-filtered) to confirm — but it validates that the harness answers the dense-vs-hybrid question.

Deferred

• Full eval-set generation + freeze (large, machine-specific — generated on demand, not committed).
• Optional LLM-judge filter to drop trivial/parametric-knowledge cases.
• Optional Inspect-AI scorer adapter.

Refs #1.

[thorwhalen]

Eval measurement run — findings (2026-06-06)

First full run of the harness (ir.eval + ir.eval_gen, ir 0.1.3/0.1.4). Generation: gpt-4o-mini back-translation, name-masked, k=5, +~15% abstention. Scoring: offline, embedder all-MiniLM-L6-v2. Full write-up: misc/docs/ir_05 -- Capability-Discovery Eval Run -- Findings (Dense vs Lexical vs Hybrid).md.

Headline: the preliminary "dense ≫ hybrid" (n=8) reverses on full sets, and the winner is corpus-dependent

NDCG@10:

corpus / lens	n_gold	dense	lexical	hybrid	winner
skills — strict	776	0.659	0.641	0.694	hybrid
skills — near-dup expanded	776	0.705	0.679	0.744	hybrid
skills — judge-clean	704	0.728	0.709	0.772	hybrid
skills — judge-graded	719	0.733	0.684	0.765	hybrid
packages — strict	582	0.583	0.305	0.489	dense
reports — strict (n=100)	100	0.431	0.367	0.477	hybrid

Significance (skills, paired Wilcoxon on per-query NDCG@10): hybrid > dense is significant under all four lenses (p = 8.9e-5 … 6.6e-6); hybrid > lexical significant everywhere (down to p=1.5e-20); dense ≈ lexical (n.s.) in 3 of 4 lenses.

Mechanism: lexical signal density in the (name-masked) description. skills/reports descriptions are keyword-rich → BM25 has signal → hybrid fuses to the top. packages descriptions are terse → BM25 is noise → it drags hybrid below pure dense. hybrid is the safest default (wins 2/3 corpora, never catastrophic); pure lexical is never best.

Four scoring lenses (the conclusion survives all)

• near-dup: skills has 27% near-duplicate artifacts (same skill global + package-scoped, e.g. acquire-references / acquire-references@citeget). Expanding gold to the equivalence class lifts NDCG@1 ~0.11 — ~27% of "rank-1 misses" were the sibling at rank 1.
• judge-clean / judge-graded: a 32-agent LLM-judge audit of all 776 gold probes → 704 good / 42 trivial / 15 misrouted / 15 ambiguous (90.7% good — generation quality is high). The judge flagged additional genuinely-relevant skills for 41% of probes (single-gold understates true relevance).

Distractor-robustness (k=1, accuracy as catalog grows 1→157)

MiniLM: dense & hybrid both decay gracefully and stay close; lexical decays fastest. Even the best mode loses ~half its rank-1 accuracy by N≈64 → motivates the selection/filtering stage (#11) as the catalog grows.

Infrastructure findings (surfaced, not fixed)

1. BM25 is rebuilt per query in ir.retrieve (no persisted index) → lexical/hybrid are O(corpus×queries). A full lexical/hybrid pass over the 16,836-record reports corpus didn't finish in 10 min CPU (had to down-sample to 100). Biggest blocker to running ir at scale.
2. Near-dup artifacts (above) — dedupe at index time, or support gold equivalence classes in ir.eval.
3. ef.evaluation._RETRIEVAL_METRICS is private — ir.eval mirrors it; expose a public RETRIEVAL_METRICS.
4. distractor_robustness_curve reloads the embedder per sub-corpus build → impractical with MiniLM unless a pre-built embedder object is threaded through.

Deferred #12 items status

• LLM-judge case filter → done (judge-audit workflow, this run).
• Frozen real eval fixture → case files are machine-specific (corpus is live/local); not committed. Open question whether to commit a curated sample.
• Inspect-AI scorer adapter → still deferred.

[thorwhalen]

Selector tuning landed (PR #24 → ir 0.1.8). Added ir.eval.sweep_selector (+ SelectionSweep/SelectionGridPoint, ir sweep-select CLI), which scores a max_k × rel × min_score grid off one retrieval pass (refactored evaluate_selection into _retrieve_for_cases + _score_selection).

Empirical result over real skills/packages/reports × dense/hybrid (findings: misc/docs/ir_06): the old conservative defaults max_k=5, rel=0.6 were dominated in all 6 runs; tuned to max_k=3, rel=0.9 (~halves commit size, ~doubles precision; mode-agnostic optimum).

Still deferred here: Inspect-AI scorer adapter; committing a frozen real fixture (machine-specific).

[thorwhalen]

min_score calibration shipped → PR #25

The min_score calibration noted here as a next step from the selector-tuning findings is implemented and up for review in #25 (branch feature/min-score-calibration).

What landed

• ir.eval.calibrate_min_score() + MinScoreCalibration + load_calibration() — abstention-floor calibration framed as a 1-D separability problem (in-scope vs out-of-scope top-score distributions), not a selector sweep (which is degenerate for abstention_accuracy). Reuses _retrieve_for_cases (one pass) and the n_gold_retrieved retrieval-miss conditioning from evaluate_selection.
• Persisted per-(corpus, mode) on a new CorpusStore.calibration view (kept apart from build config); discover(min_score="auto") loads it (opt-in; default unchanged), with embedder-staleness + non-finite guards and a vd_degraded flag.
• CLI ir calibrate-min-score, ir discover --min-score auto, floors in ir info. 22 hermetic tests; suite 165 green.

Empirical findings (misc/docs/ir_07, real corpora skills/packages/reports, MiniLM): a calibrated floor is a useful-but-imperfect filter — dense/lexical reach Youden's J ≈ 0.65–0.90; hybrid is worst (≈0.33–0.61) because RRF is rank-based and discards score magnitude. Calibrate abstention on dense/lexical, not hybrid.

Still deferred under #12 (unchanged by this PR):

• Inspect-AI scorer adapter.
• A committed, embedder-pinned frozen real fixture for reproducible calibration/eval in CI (the calibration numbers above are machine-local).

New follow-up surfaced (not filed): ir.retrieve._rrf_fuse returns raw dense scores when the lexical list is empty, and RRF discards magnitude generally — a future enhancement could gate hybrid abstention on the dense sub-score. Noted in ir_07 §6.

[thorwhalen]

Closing — the eval harness is complete and shipped: scoring (ir.eval, #18), LLM case generation (ir.eval_gen, #19), full eval run (ir_05), selector tuning (ir_06/#24), min_score calibration (ir_07/#25), and magnitude-preserving fusion (ir_08/#31). The two items deferred under this issue are resolved: the frozen real fixture is committed (#27, tests/fixtures/mini_corpus.json), and the Inspect-AI scorer adapter is tracked separately as #29 (deferred — no consumer). Roadmap umbrella stays in #1.