Add a standardized, preregistered retrieval benchmark harness

.gitignore

@@ -76,3 +76,6 @@ packaging/snap-compat/payload/
 packaging/snap-cuda/payload/
 # Built engine binaries land here during wheel/executable builds.
 packaging/engine-wheel/lilbee_engine/bin/
+
+# The benchmark deps list is intentionally tracked despite the *.txt rule.
+!evals/benchmark/requirements.txt

docs/usage.md

@@ -751,6 +751,8 @@ something feels off.
 | `LILBEE_TITLE_SEARCH` | `false` | Match queries against document titles as a third hybrid-search arm, so a query naming a document by title surfaces its chunks |
 | `LILBEE_TITLE_SEARCH_WEIGHT` | `0.5` | Title arm weight in rank fusion (1.0 = equal voice with the vector and text arms) |
 | `LILBEE_LEXICAL_FUSION_WEIGHT` | `1.0` | BM25 arm weight in rank fusion (1.0 = equal to the vector arm; lower it when a strong embedder should dominate a corpus where the lexical arm adds noise) |
+| `LILBEE_ADAPTIVE_FUSION` | `false` | Scale the BM25 arm's weight per query by how confident the vector arm is (a peaked dense ranking downweights lexical, a flat one keeps it) instead of using a fixed `LILBEE_LEXICAL_FUSION_WEIGHT`. That weight becomes the ceiling the adaptive rule scales down from |
+| `LILBEE_ADAPTIVE_FUSION_MARGIN` | `0.3` | Vector-similarity margin (top hit minus the field) at which adaptive fusion fully silences the BM25 arm; smaller values downweight lexical more aggressively |
 | `LILBEE_ADAPTIVE_THRESHOLD` | `false` | Widen the distance threshold step by step when too few results pass, on the vector-only fallback path (no effect on the default hybrid search) |
 | `LILBEE_ADAPTIVE_THRESHOLD_STEP` | `0.2` | How much to widen per step when adaptive threshold triggers |
 | `LILBEE_TEMPORAL_FILTERING` | `true` | When the query contains temporal cues ("recent", "last week"), filter results by document date and sort by recency |

evals/benchmark/README.md

@@ -0,0 +1,121 @@
+# Benchmark harness: lilbee vs RAGFlow
+
+A preregistered, reproducible A/B of retrieval quality. Two arms answer the
+same public, human-labeled datasets with the same served model, so retrieval is
+the only variable:
+
+- **Arm A - lilbee**: the `test/retrieval-parity` branch with every parity
+  feature enabled.
+- **Arm B - RAGFlow**: the DeepDoc pipeline at its defaults.
+
+One embedder and one generator are served once and pointed at by both arms, so
+generation is identical and only the retrieval stack changes. Lives outside
+`src/` on purpose: it never ships in the package.
+
+## Two tiers
+
+1. **Tier 1 - retrieval, no model opinion.** Each arm returns a ranked list per
+   query, written as a TREC run file. `pytrec_eval` scores it against the
+   dataset's published relevance labels (qrels): nDCG@10, Recall@20, MRR@10.
+   Nothing is graded by a model, so the numbers are bit-for-bit reproducible by
+   anyone with the same run files and qrels.
+2. **Tier 2 - answer quality.** Both arms answer with the same model; RAGAS
+   scores faithfulness, answer relevancy, and context precision/recall. The
+   blind duplicate-arm judge from `evals/retrieval` is reused as a corroborating
+   signal, with its noise floor measured by grading one arm twice.
+
+Every cross-arm difference is paired per query and gets a bootstrap 95% CI and a
+randomization-test p-value. A difference whose CI crosses zero is reported as
+not significant, not as a win.
+
+## Native vs derived labels
+
+Passage sets (BEIR SciFact/FiQA/NFCorpus, HotpotQA, TREC-COVID) ship native TREC
+qrels, used as published. The document-structured QA sets (TAT-DQA, OTT-QA) have
+no retrieval labels of their own, so their qrels are **derived** from human
+gold-evidence annotations by one documented pure function
+(`datasets.derive_qrels_from_evidence`). Every derived dataset is marked
+`label_kind: derived` in the manifest and flagged in the report.
+
+## Pod workflow
+
+One A100-80GB in a named tmux so a reclaimed pod reattaches. Preregister before
+any data moves, then run each arm, score, and power off.
+
+```bash
+cd ~/lilbee  # the repo checkout; run everything from the repo root
+RUN=/tmp/bench
+
+# 0. Freeze the preregistration. Nothing can be cherry-picked after this.
+uv run python -m evals.benchmark preregister \
+  --manifest evals/benchmark/manifest.example.yaml --out "$RUN/manifest.frozen.json"
+
+# 1. Serve the shared model with lilbee (all parity features on) and stand up
+#    RAGFlow pointed at the same OpenAI-compatible endpoint.
+export LILBEE_TITLE_SEARCH=true LILBEE_NEIGHBOR_EXPANSION=2 \
+  LILBEE_TABLE_EXTRACTION=true LILBEE_LAYOUT_DETECTION=true LILBEE_INTENT_LLM=true
+lilbee serve --port 8080 &
+uv run python -m evals.benchmark.bootstrap_ragflow \
+  --base-url http://127.0.0.1:9380 --api-key "$RAGFLOW_KEY" \
+  --corpus-dir "$RUN/corpus" --llm-model qwen2.5-72b-instruct \
+  --embedding-model qwen3-embedding
+
+# 2. Ingest the same corpus into both systems (lilbee add; bootstrap uploaded
+#    RAGFlow's copy above).
+
+# 3. Collect a TREC run file per arm. Each query is checkpointed, so a killed
+#    run resumes.
+uv run python -m evals.benchmark collect --system lilbee \
+  --queries "$RUN/queries.jsonl" --base-url http://127.0.0.1:8080 \
+  --run-tag lilbee --run "$RUN/run-lilbee.trec" --checkpoint "$RUN/ck-lilbee.jsonl"
+uv run python -m evals.benchmark collect --system ragflow \
+  --queries "$RUN/queries.jsonl" --base-url http://127.0.0.1:9380 \
+  --api-key "$RAGFLOW_KEY" --dataset-id "$RAGFLOW_DATASET" \
+  --run-tag ragflow --run "$RUN/run-ragflow.trec" --checkpoint "$RUN/ck-ragflow.jsonl"
+
+# 4. Tier 1: score each run against the qrels with pytrec_eval.
+uv run python -m evals.benchmark score-ir --qrels "$RUN/qrels.json" \
+  --run "$RUN/run-lilbee.trec" --dataset scifact --run-tag lilbee \
+  --out "$RUN/ir-lilbee.jsonl"
+uv run python -m evals.benchmark score-ir --qrels "$RUN/qrels.json" \
+  --run "$RUN/run-ragflow.trec" --dataset scifact --run-tag ragflow \
+  --out "$RUN/ir-ragflow.jsonl"
+
+# 5. Tier 2: generate answers on each arm, then score with RAGAS.
+uv run python -m evals.benchmark answer --queries "$RUN/queries.jsonl" \
+  --ground-truth "$RUN/references.json" --base-url http://127.0.0.1:8080 \
+  --arm lilbee --out "$RUN/answers-lilbee.jsonl"
+uv run python -m evals.benchmark score-ragas \
+  --samples "$RUN/answers-lilbee.jsonl" --out "$RUN/results.jsonl"
+
+# 6. Paired statistics (CI + p) into the same results file.
+uv run python -m evals.benchmark stats --manifest "$RUN/manifest.frozen.json" \
+  --metrics-a "$RUN/ir-lilbee.jsonl" --metrics-b "$RUN/ir-ragflow.jsonl" \
+  --arm-a-label lilbee --arm-b-label ragflow --out "$RUN/results.jsonl"
+
+# 7. Render the report, then power the pod off.
+uv run python -m evals.benchmark report \
+  --results "$RUN/results.jsonl" --out "$RUN/report.md"
+```
+
+Keep the frozen manifest, run files, and qrels: with those, a third party can
+re-score the whole Tier-1 comparison without the pod.
+
+## Optional dependencies
+
+The heavy scorers and dataset loaders are imported lazily, so the harness (and
+its tests) load without them. They are kept out of the shipped lock on purpose
+(installing them would otherwise drag core dependencies backward), so install
+them from the standalone requirements file on the benchmark pod:
+
+```bash
+uv pip install -r evals/benchmark/requirements.txt   # pytrec_eval, ragas, beir, datasets
+```
+
+## Determinism and resume
+
+- The frozen manifest pins the bootstrap seed, resamples, and alpha, so every
+  CI is reproducible.
+- `collect` and `answer` checkpoint per query. Kill and re-run freely; only
+  unfinished queries repeat, and the run file is rebuilt from the full
+  checkpoint each time.

evals/benchmark/RESULTS.md

@@ -0,0 +1,107 @@
+# Retrieval benchmark results — Qwen3-Embedding-0.6B
+
+A significance-tested measurement of lilbee's hybrid retrieval against its own
+vector-only baseline on three public, human-labeled BEIR datasets. The point of
+this run is to answer two questions with evidence, not opinion: does lilbee's
+rank fusion actually improve retrieval over the raw embedder, and is a single
+fusion weight the right design.
+
+## Method
+
+- **Embedder:** Qwen3-Embedding-0.6B (GGUF, Q8_0), served once; every arm points
+  at the same model, so retrieval is the only variable.
+- **Datasets:** BEIR SciFact, NFCorpus, FiQA, using their published TREC qrels
+  as-is. No labels were derived or altered.
+- **Metrics:** nDCG@10, Recall@20, MRR@10, scored with `pytrec_eval` against the
+  native qrels. Nothing is graded by a model, so the numbers are reproducible by
+  anyone with the run files and qrels.
+- **Arms:** `dense` is vector-only (the lexical and title arms silenced);
+  `w=X` is hybrid fusion with the BM25 arm at `lexical_fusion_weight=X` and the
+  title arm on. This isolates what fusion adds over the embedder alone.
+- **Significance:** every hybrid-vs-dense difference is paired per query and gets
+  a bootstrap 95% CI and a randomization-test p-value. A difference whose CI
+  crosses zero is reported as not significant, not as a win.
+- **Condition:** query expansion was off (no chat model served), so these are
+  pure first-pass retrieval numbers with no LLM query rewriting on either arm.
+
+## Results (nDCG@10)
+
+| Dataset | dense | w=1.0 | w=0.75 | w=0.5 | w=0.25 |
+|---|---|---|---|---|---|
+| NFCorpus (n=322) | 0.3666 | 0.3734 | 0.3743 | 0.3756 | **0.3767** |
+| SciFact (n=300) | 0.6981 | **0.7280** | 0.7226 | 0.7167 | 0.7155 |
+| FiQA (n=648) | **0.4598** | 0.4033 | 0.4289 | 0.4333 | 0.4363 |
+
+## Significance (best hybrid weight vs. dense, nDCG@10)
+
+| Dataset | best hybrid | Δ vs dense | 95% CI | p | verdict |
+|---|---|---|---|---|---|
+| NFCorpus | w=0.25 | **+0.0100** | [+0.0022, +0.0180] | 0.012 | significant win |
+| SciFact | w=1.0 | **+0.0299** | [+0.0073, +0.0528] | 0.011 | significant win |
+| FiQA | w=0.25 | **−0.0235** | [−0.0356, −0.0118] | 0.0004 | significant regression |
+
+On FiQA every hybrid weight loses to dense (w=1.0 is worst at −0.0564, p=0.0001);
+lowering the weight reduces the damage but never recovers it. On NFCorpus the
+win is only significant at lower weights (w=1.0 is +0.0068, p=0.21, not
+significant). On SciFact every weight wins, and higher is better.
+
+## What this establishes
+
+1. **Hybrid fusion adds real, significant value on two of three corpora** — once
+   it actually runs (see the bug below). This is the raw embedder plus lilbee's
+   fusion beating the raw embedder alone, measured, not asserted.
+2. **No single fixed fusion weight wins everywhere.** SciFact wants a strong
+   lexical arm (w=1.0), NFCorpus wants a weak one (w=0.25), and FiQA wants none.
+   The optimal weight is corpus-dependent, which is the empirical case for
+   gating the lexical weight per query rather than fixing it — the adaptive
+   fusion this run motivates.
+
+## The bug this run caught
+
+Before any of these numbers meant anything, every hybrid config scored
+identically to the vector-only baseline. The cause was a production defect, not
+a benchmark artifact: `ensure_fts_index()` runs `table.optimize()` on an
+existing index, which on a real-sized corpus hits a LanceDB encoding bug and
+raises; the failure was swallowed and left hybrid search disabled, so every
+query silently fell back to vector-only. It only surfaced on real data — a
+tiny local corpus never tripped the LanceDB bug. Fixed by marking an existing,
+queryable index ready before the best-effort optimize, so an optimize failure
+degrades to a warning instead of disabling retrieval corpus-wide.
+
+## Reproducing
+
+The frozen manifest, the TREC run files, and the qrels are kept under
+`results/`. With those, anyone can re-score Tier 1 without a GPU:
+
+```bash
+gunzip -k results/scifact/run-w1.0.trec.gz
+python -m evals.benchmark score-ir --qrels results/scifact/qrels.json \
+  --run results/scifact/run-w1.0.trec --dataset scifact --run-tag w1.0 --out /tmp/ir.jsonl
+```
+
+## Adaptive fusion
+
+Because no fixed weight wins everywhere, a follow-up run measured `adaptive_fusion`
+(the BM25 weight scaled per query by how peaked the vector ranking is) at three
+confidence margins, on the same three datasets. Best margin was 0.15.
+
+| Dataset | dense | w=1.0 | adaptive (m=0.15) | adaptive vs dense |
+|---|---|---|---|---|
+| NFCorpus | 0.3666 | 0.3732 | 0.3751 | +0.0085, p=0.033, **sig win** |
+| SciFact | 0.6981 | 0.7280 | 0.7186 | +0.0205, p=0.009, **sig win** |
+| FiQA | 0.4593 | 0.4027 | 0.4410 | −0.0184, p=0.003, sig regression |
+
+Adaptive fusion is the best single policy found. It beats the fixed w=1.0 default
+on all three datasets, keeps the significant NFCorpus and SciFact wins, and gives
+the smallest FiQA regression of any config (−0.018 vs −0.056 at w=1.0 and −0.023
+at the best fixed weight). It does not fully erase FiQA's regression — pure dense
+still wins there — so the vector-margin confidence signal reduces, but does not
+eliminate, the cases where the lexical arm hurts. Run files and stats are under
+`results-adaptive/`.
+
+## Scope
+
+This is the 0.6B-embedder study: it validates the fusion mechanism and the
+FTS-optimize fix, and it sizes the corpus-dependence of the lexical weight. It
+is deliberately not a headline "lilbee vs. everyone" number — a larger embedder
+and the RAGFlow A/B arm are separate runs.

evals/benchmark/__init__.py

@@ -0,0 +1,9 @@
+"""Preregistered, reproducible A/B benchmark: lilbee vs RAGFlow retrieval.
+
+Two arms answer the same public, human-labeled datasets with the same served
+model, so retrieval is the only variable. Tier 1 scores ranked results against
+the datasets' own relevance labels with pytrec_eval (no model opinion); Tier 2
+layers RAGAS answer-quality metrics on top, corroborated by the blind judge
+from ``evals.retrieval``. Lives outside ``src/`` on purpose: it never ships in
+the package.
+"""

evals/benchmark/__main__.py

@@ -0,0 +1,6 @@
+"""Runs the benchmark CLI: python -m evals.benchmark <subcommand>."""
+
+from evals.benchmark.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())