Every number in this document was produced by a command in this repository, on the machine described below, against the corpus and question set committed here. Nothing is estimated, rounded up, or carried over from a previous run. Where a result is unflattering it is reported as measured, because a benchmark that only reports wins is not a benchmark.
Reproduce the whole thing:
python -m evals.fetch # download and verify the corpus
python -m evals.retrieval.run --split test --rebuild
python -m evals.generation.run --split test| Machine | 8 vCPU Intel Xeon cloud VM, no GPU |
| OS / Python | Linux 6.12.94, glibc 2.39, CPython 3.12.3 |
| Commit | dfea94d |
| Date | 2026-08-10 |
| PostgreSQL | 17 with pgvector, HNSW index, m=16, ef_construction=64, ef_search=100 |
| Embedder | static-vectors — SIF-pooled en_core_web_md, one principal component removed, 384-d column |
| Generator | local / extractive-v1 — no network, no API key |
| Load | Machine otherwise idle. Latency figures from a contended run were discarded |
Latency here is one process talking to a local PostgreSQL over loopback. It is a measurement of the pipeline's own cost, not of a deployed system: no network, no concurrency, no cold cache, no other tenants. Treat the numbers as a floor and as a way to compare stages against each other, which is what they are used for below.
Ten machine-learning papers from arXiv — Attention Is All You Need, BERT, RoBERTa, ResNet, U-Net, GAN, Adam, Batch Normalisation, ViT, and the original RAG paper. 140 pages, 587 chunks, 124,211 tokens under the default chunker.
The PDFs are not committed. python -m evals.fetch downloads them from their canonical
arXiv URLs and verifies each against a SHA-256 in evals/datasets/corpus.lock.json. That
lock is the difference between "I re-ran the benchmark" and "I re-ran the benchmark on the
same data": arXiv re-typesets papers, and a silently changed PDF would move every number
here with no visible cause.
277 hand-written questions: 209 answerable, 68 unanswerable. Each answerable question carries a quoted passage of evidence, which is resolved to character offsets at corpus build time. Relevance is defined as the retrieved chunk's character range overlaps a labelled evidence span in the same document — not "the answer string appears in the chunk", which would credit a chunk that happens to repeat the answer text somewhere irrelevant, and which would make the chunker comparison meaningless.
The set is split in half — 138 train, 139 test — stratified by question type, by a deterministic hash of the question. Every number below is on the test half. The fusion weights, the reranker and the abstention gate were all fitted on the train half and have never seen these questions.
277 questions over 10 arXiv papers. Question types: numeric (92), factoid (34), procedural (29), definition (25), comparison (19), multi_hop (10), plus 68 unanswerable.
105 answerable test questions. hit@5 carries a 95% Wilson interval, because these are
proportions over about a hundred items and two arms that differ by four questions do not
differ.
| Arm | hit@1 | 95% CI | hit@5 | 95% CI | MRR | nDCG@5 |
|---|---|---|---|---|---|---|
| Chunker (hybrid, no reranker) | ||||||
| fixed | 0.448 | 0.356–0.543 | 0.724 | 0.631–0.800 | 0.569 | 0.566 |
| recursive | 0.400 | 0.311–0.496 | 0.695 | 0.602–0.775 | 0.531 | 0.514 |
| sentence | 0.391 | 0.303–0.486 | 0.762 | 0.672–0.833 | 0.535 | 0.550 |
| structure | 0.409 | 0.320–0.505 | 0.695 | 0.602–0.775 | 0.538 | 0.508 |
| Strategy (structure chunker) | ||||||
| vector only | 0.114 | 0.067–0.189 | 0.276 | 0.200–0.368 | 0.182 | 0.177 |
| keyword only | 0.400 | 0.311–0.496 | 0.676 | 0.582–0.758 | 0.529 | 0.502 |
| hybrid, equal weights | 0.314 | 0.233–0.408 | 0.505 | 0.411–0.599 | 0.400 | 0.371 |
| hybrid, tuned weights | 0.409 | 0.320–0.505 | 0.695 | 0.602–0.775 | 0.538 | 0.508 |
| Reranker (hybrid, tuned) | ||||||
| lexical (hand-weighted) | 0.457 | 0.365–0.552 | 0.686 | 0.592–0.767 | 0.569 | 0.525 |
| feature (trained) — default | 0.610 | 0.514–0.697 | 0.809 | 0.724–0.873 | 0.692 | 0.644 |
The trained reranker is the one unambiguous win. Its hit@1 interval is the only one in the table that clears the tuned hybrid arm's without touching it. At hit@5 the improvement looks just as large but the intervals do overlap, so this document does not claim it. Overlapping intervals are a conservative screen rather than a test: these are paired observations on the same questions, and a paired test would have more power. Non-overlap is enough to assert a difference; overlap is not enough to deny one.
The vector arm is weak on this corpus, and hybrid retrieval with naive weights is worse
than keyword alone. Equal-weight hybrid sits below the keyword arm it is built from. This
is the opposite of the result hybrid search is supposed to produce, and the reason is the
embedder. SIF-pooled static word vectors have no useful position for ef_construction or
3.5 × 10⁻⁴, and this corpus is full of exactly that — numeric questions are the largest
single type in the set. Blending a weak ranking into a strong one at equal weight drags the
strong one down.
Tuning the fusion weights on the training half recovers keyword-only performance and no more — the sweep chose 0.15 vector, 0.85 keyword, which is a polite way of saying it learned to mostly ignore the vector arm. Hybrid retrieval is carrying its own weight here only after being told not to trust half of itself. With a hosted embedder the balance would very likely change; the honest statement is that it was not measured, because a benchmark that needs an API key is a benchmark a reader cannot reproduce.
The chunkers are indistinguishable at this sample size. All four hit@5 intervals overlap
heavily, and so do all four hit@1 intervals; the ordering between them changes with the
metric, and structure wins neither. Structure-aware chunking is still the default, for
reasons the table cannot measure: it produces the section paths and page numbers that make a
citation legible, and it keeps tables intact. It is chosen on those grounds and not on a
retrieval score it did not win.
Default configuration, per query:
| Stage | p50 | p95 |
|---|---|---|
| embed the question | 0.12 ms | 0.15 ms |
| vector arm | 3.73 ms | 4.01 ms |
| keyword arm | 5.87 ms | 12.64 ms |
| fusion | 0.05 ms | 0.05 ms |
| rerank (20 features) | 4.58 ms | 5.49 ms |
| end to end | 14.5 ms | 22.3 ms |
Reranking costs 5.4 ms at p50 in this run, measured against the same run's unreranked arm (9.2 ms against 14.5 ms).
The keyword arm is the most expensive stage, ahead of both the vector lookup and the
reranker. That is the reverse of the usual intuition, and it is the direct consequence of
PostgreSQL full-text ranking with ts_rank_cd being real work while an HNSW lookup over a
few hundred vectors is nearly free. It also tells you where the first optimisation would go
if this needed to be faster: the two arms run sequentially on one session today and are
perfectly independent. Treat the absolute figures as a shape rather than a measurement —
they move with cache state, which is why only the reranker's marginal cost is quoted as a
number.
hit@5 by question type, default configuration:
| Type | Questions | hit@5 | MRR |
|---|---|---|---|
| definition | 13 | 0.923 | 0.686 |
| comparison | 9 | 0.889 | 0.833 |
| factoid | 17 | 0.882 | 0.718 |
| numeric | 46 | 0.804 | 0.683 |
| multi-hop | 5 | 0.800 | 0.687 |
| procedural | 15 | 0.600 | 0.610 |
Procedural questions ("how do you initialise the weights before training") are the weak spot: the answer is spread across several sentences or a numbered list, so a single chunk rarely contains all of it and the evidence span is only partially covered. Comparison questions do well because they name two things explicitly, which is precisely what a keyword arm is good at. The per-type buckets are small — nine questions for comparison — so read these as directions.
139 test questions (105 answerable, 34 unanswerable), local extractive model.
| Metric | Value |
|---|---|
| Answerable (105) | |
| refusal rate | 0.181 (95% CI 0.119 – 0.265) |
| answer token F1 | 0.167 |
| reference-answer recall | 0.513 |
| quote is labelled evidence | 0.558 |
| cited anything | 1.000 |
| citation precision | 0.509 |
| citation recall | 0.785 |
| groundedness | 0.989 |
| unsupported claim rate | 0.000 |
| invalid citation marker rate | 0.000 |
| Unanswerable (34) | |
| refusal rate | 0.471 (95% CI 0.315 – 0.633) |
| answered anyway | 18 |
| Latency | |
| end to end p50 / p95 | 16.3 ms / 24.5 ms |
| output tokens p50 / p95 | 71 / 337 |
Read the answer-quality numbers as a measurement of the default generator, not of the
system. extractive-v1 selects and stitches sentences from the retrieved passages. It
cannot hallucinate — the near-perfect groundedness and the zero unsupported-claim rate are
consequences of extraction, not achievements — and it cannot write a concise answer either,
which is why token F1 sits far below reference-answer recall: the right content is usually
in there, surrounded by the rest of the sentence it came from. Swapping in
LLM_PROVIDER=openai would move F1 substantially and groundedness downward. Both of those
are predictions, not measurements, and are not in the table.
Citations work. Every answer that was produced cited something and no answer contained an invalid marker. Citation recall runs well ahead of precision, which says the system tends to cite more passages than strictly necessary rather than fewer. Separately, an integration test asserts that every citation quote produced over the whole corpus slices the original document back out exactly, character for character. That is a correctness property rather than a score, and it is the property the reader's highlight depends on.
The abstention gate is the weakest part of the system, and this is the honest headline.
It refuses 16 of 34 unanswerable questions (0.471, 95% CI 0.315–0.633) while wrongly refusing 19 of 105 answerable ones (0.181, 95% CI 0.119–0.265). Balanced accuracy is 0.645, against 0.500 for never refusing at all.
The reason is structural rather than a tuning failure. Every feature the gate sees is a function of retrieval scores, and the retriever gives an unanswerable question's nearest passages roughly the scores it gives an answerable one's — because those passages genuinely are about the right subject, they simply do not contain the answer. "Is this passage about the question" and "does this passage answer the question" are different questions, and only the first is visible in a ranking.
The threshold is chosen against a false-refusal budget of 15% on the training half, which is why the gate is cautious rather than aggressive: sweeping for balanced accuracy alone picks a threshold that catches almost every unanswerable question and refuses well over half of the answerable ones, which would be a far worse product. Note that the budget is a constraint on the training half only — on the held-out half the false-refusal rate comes out slightly above it, which is what a threshold fitted on a hundred-odd questions does. Fixing this properly needs a content-level signal — a small entailment check between the question and the top passage — and that is recorded as future work rather than claimed.
Ten papers, 140 pages, from PDF to searchable, on the same machine:
| Documents | 10 |
| Pages | 140 |
| Chunks produced | 587 |
| Tokens indexed | 124,211 |
| Total re-index time | 1.01 s |
That figure is re-indexing from the already-parsed representation — chunk, embed, write — which is what the eval harness does between arms. It is not full ingestion: it excludes PDF parsing, which dominates a real upload and varies by an order of magnitude across parsers and documents. A full end-to-end ingestion benchmark is not in this document because it has not been run under controlled conditions; the number would be a guess dressed as a measurement.
- Throughput and concurrency. Nothing in this document is a load test. There is no requests-per-second figure because none was measured.
- Time to first token. The default provider composes its answer locally and then emits it word by word with a fixed delay, so its TTFT measures the delay constant. A real figure needs a hosted model, which needs a key. The mechanism is covered by integration tests; the number waits for a deployment that has one.
- Hosted model quality. The OpenAI, Gemini and Ollama providers are implemented and tested, and were not benchmarked, because a benchmark a reader cannot re-run is not evidence.
- Anything about a corpus other than this one. Ten ML papers is a narrow, jargon-heavy, numeric-heavy domain. The result that keyword beats vector retrieval is a result about this corpus and this embedder, and it should not be generalised.
The corpus lock pins the exact PDF bytes, the split is a deterministic hash, and the models are committed, so the quality metrics should reproduce exactly on any machine. Latency will not: it is hardware, and the figures above are from an idle 8-vCPU VM with PostgreSQL on loopback.
If a number here disagrees with a re-run, the re-run is right. Every result file in
evals/results/ records the git commit, the platform and the full settings it was produced
under, for exactly that reason.