Add a standardized, preregistered retrieval benchmark harness#552
Closed
tobocop2 wants to merge 8 commits into
Closed
Add a standardized, preregistered retrieval benchmark harness#552tobocop2 wants to merge 8 commits into
tobocop2 wants to merge 8 commits into
Conversation
evals/benchmark compares lilbee against another RAG system on public labeled datasets with the generator and embedding model held constant, so retrieval is the only variable. Tier 1 scores each system's ranked results against human relevance labels with pytrec_eval (nDCG@10, Recall@20, MRR@10) with no model in the loop; Tier 2 layers RAGAS answer metrics plus the existing blind judge on top. Passage datasets carry native qrels; table and PDF datasets derive retrieval labels from human gold-evidence annotations via one documented function, disclosed in the manifest. A frozen manifest fingerprints the datasets, metrics, and models before the run (and refuses a judge equal to the generator); paired-bootstrap CIs and a randomization p-value gate every reported difference. The heavy scorers and dataset loaders stay in a standalone pod-install requirements file so they never move the shipped lock, and are lazily imported so the suite runs without them.
The repo .gitignore has a blanket *.txt rule, so evals/benchmark/requirements.txt was silently never committed and a fresh clone could not install the benchmark deps. Add a negation for it and commit the file.
ensure_fts_index() runs table.optimize() whenever the FTS index already exists. On a large corpus that call can hit a LanceDB encoding bug and raise (Arrow list offset overflow), and the failure fell through to leave _fts_ready False -- silently dropping every subsequent query to vector-only with no error surfaced. The index itself is complete and still serves queries, so mark hybrid ready before the best-effort optimize and downgrade an optimize() failure to a warning. Found while benchmarking retrieval: every hybrid config collapsed to the vector-only baseline until this was fixed.
tobocop2
force-pushed
the
feat/eval-benchmark
branch
from
July 18, 2026 02:05
1123fc4 to
6aedffb
Compare
The retrieval benchmark showed no single lexical_fusion_weight wins every corpus: SciFact wants a strong BM25 arm, NFCorpus a weak one, FiQA none. Rather than fix one weight, adaptive_fusion (off by default) scales the BM25 arm per query by how peaked the vector ranking is -- a top hit standing clear of the field downweights lexical toward zero, a flat ranking keeps it. adaptive_fusion_ margin sets the vector-similarity margin at which the lexical arm is fully silenced. lexical_fusion_weight becomes the ceiling the rule scales down from, so the default path is unchanged.
Significance-tested nDCG@10/Recall@20/MRR@10 for hybrid fusion vs the vector-only baseline on BEIR SciFact, NFCorpus, and FiQA, with the frozen manifest, gzipped TREC run files, and qrels for independent re-scoring. Hybrid significantly beats dense on NFCorpus and SciFact and significantly regresses on FiQA; the corpus-dependent optimal weight is what motivates adaptive fusion.
Adaptive per-query fusion at confidence margin 0.15, scored against dense on the same three BEIR datasets: significant wins on NFCorpus (+0.0085, p=0.033) and SciFact (+0.0205, p=0.009), and the smallest FiQA regression of any config (-0.0184 vs -0.056 at fixed w=1.0). Best single policy found; does not fully erase FiQA's regression against pure dense.
Owner
Author
|
Superseded by the consolidated retrieval PR #557 — review and merge there. Converted to draft to keep it off the review queue; the branch and history stay intact for reference. |
tobocop2
marked this pull request as draft
July 18, 2026 04:55
Owner
Author
|
Superseded by the consolidated retrieval PR #557, which contains this change. Closing; the branch stays intact for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The retrieval work needs a comparison that survives scrutiny, not a homegrown judge a skeptic can wave away. That means standard metrics, public human-labeled data, and statistics on every claimed difference.
Solution
evals/benchmarkcompares lilbee against another RAG system on public labeled datasets with the generator and embedding model held constant, so retrieval is the only variable. Tier 1 scores each system's ranked results against human relevance labels with pytrec_eval (nDCG@10, Recall@20, MRR@10) with no model in the loop, so the core numbers are exactly reproducible. Tier 2 layers RAGAS answer metrics plus the existing blind judge on top. Passage datasets carry native TREC qrels; the table and PDF datasets derive retrieval labels from human gold-evidence annotations through one documented function, disclosed in the frozen manifest. That manifest fingerprints the datasets, metrics, and models before a run and refuses a judge equal to the generator; paired-bootstrap confidence intervals and a randomization p-value gate every reported difference, and a difference whose interval crosses zero is reported as not significant.The heavy scorers and dataset loaders stay in a standalone pod-install requirements file so installing them never moves the shipped lock, and they are imported lazily so the suite runs without them. The harness lives outside the package, so nothing here ships in the wheel.