Hybrid (BM25 + dense vector) search and grounded RAG question-answering, built as a backend engineering problem. Then built a second time, because the first version's quality claims did not survive being checked.
This repository is mostly about the checking. Every RAG system ships numbers —
Recall@5 = 1.00, nDCG@10 = 0.96 — and almost none of them ship the thing that makes a
number a claim. When the tools to check were finally written, they were pointed at this
repo's own README first, and three of its headline claims failed.
The gold set had ten queries with exactly one relevant document each, so Recall@5 is a
per-query coin flip and the published 1.00 means ten successes out of ten. Three things
follow immediately, and none of them were anywhere in the repository:
| The claim | What it actually says | Why |
|---|---|---|
Recall@5 = 1.00 |
1.00, 95% CI [0.69, 1.00] |
The exact binomial interval at 10/10 bottoms out at (α/2)^(1/n) = 0.69. The number was 31 points wide and presented as a fact. |
| "Hybrid lifts Recall@5 +0.30 over BM25" | p = 0.25 — and 0.25 is the best p this comparison could ever return | Seven of ten queries tie; only three differ. A paired test on three pairs has a p-value floor of 2/2³. Not "not significant" — unable to be. |
| Nightly tuner ships a config on a +0.01 MRR win | Resolving 0.01 needs ~7,000 queries | It searched nine grid points on ten queries, took the argmax, and was validated by a gate scoring the same ten queries it was selected on. |
$ make eval-stats # no stack, no network — this is a property of the labels
At least 6 of 10 queries must differ before any outcome can be significant.
observed 95% interval width
1.00 (10/10) [0.692, 1.000] 0.308
0.70 (7/10) [0.348, 0.933] 0.586
A tuning loop that acts on a difference smaller than the smallest detectable effect
is not tuning — it is resampling noise.The system was engineered. The measurement was not. What follows is the second version.
A 95% interval promises that one look at a finished sample contains the truth 95% of the time. It promises nothing about an interval recomputed after every query and watched as it streams — which is how every CI run is actually read. Measured over 800 simulated evaluation streams per cell, with the fixed-N interval given every benefit of the doubt (inspection only starts at n = 30, where the normal approximation is defensible):
| metric | queries | fixed-N interval | confidence sequence |
|---|---|---|---|
| Recall@5 (0/1) | 300 | 34.0% | 1.9% |
| reciprocal rank (discrete, skewed) | 300 | 32.8% | 2.6% |
| nDCG@10 (graded) | 300 | 35.1% | 1.6% |
| groundedness judge (skewed high) | 300 | 30.0% | 0.5% |
Share of runs where the true mean falls outside a nominal 95% interval at some point while it is being watched. The fixed-N column grows with stream length — the longer you look, the more chances to be wrong — which is the signature of the law of the iterated logarithm and a sign the simulation is behaving.
The fix is a confidence sequence: an interval valid at every sample size simultaneously,
built as the hedged capital process of Waudby-Smith & Ramdas (JRSS-B, 2024). Bet on each
next observation at odds implied by the hypothesis; under the null, wealth is a non-negative
martingale, and Ville's inequality caps the chance it ever crosses 1/α. Retrieval metrics
fit this unusually well — reciprocal rank, nDCG, recall and a judge's score all live in
[0, 1], and nothing about their shape is assumed.
Which makes stopping early legitimate rather than p-hacking, so the gate does:
make eval-gate GATE_POLICY=sequential| the system's true Recall@5 | gate verdict | queries scored (of 300) | saved |
|---|---|---|---|
| 0.98 | pass | 54 | 82% |
| 0.95 | pass | 89 | 70% |
| 0.85 (exactly at the line) | undecided |
293 | 2% |
| 0.60 | fail | 31 | 90% |
Against a live API over 200 queries: a healthy system cleared in 69 requests, a broken
one failed in 13. The saving is HTTP calls and LLM-judge tokens never spent, not a
smaller number in a report. undecided fails the gate — a gate that treats "we could not
tell" as a pass has quietly stopped gating.
→ ADR 0012 · make eval-anytime reproduces
both tables from scratch, seeded, no stack.
Two constants decided what this system costs and how often it could hallucinate:
recall.retrieval.top-k: 8 # passages that reach the LLM
recall.rag.sufficiency.confidence-threshold: 0.35 # when to answer at allA fixed top-K is wrong in both directions at once. On a query where the reranker found one obviously-right passage, seven more are prompt tokens spent on noise. On an ambiguous query where the scores are flat, the same K truncates the passage that held the answer — and nothing downstream reports it, so generation proceeds from context that cannot support it.
Both are now calibrated thresholds with finite-sample, distribution-free guarantees. No assumption about the score distribution, no asymptotics, no requirement that the reranker be well calibrated:
coverage P(the passages sent to the LLM contain a relevant document) >= 1 - alpha
risk P( P(answering from context with no relevant document) <= risk-alpha ) >= 1 - delta
Split conformal prediction with adaptive set sizes (Romano, Sesia & Candès, NeurIPS 2020)
for the first; risk-controlling prediction sets (Bates et al., JACM 2021) with
fixed-sequence testing for the second. make calibrate produces both from a single pass
over the gold set — the retrieval that scores Recall@5 already records where the relevant
document landed and how confident the reranker was, which is all either one needs.
One run, 800 queries, alpha = 0.10, against the stand-in cross-encoder in
eval/_mock_reranker.py — so this is reproducible without the
stack up; make calibrate does the same against the real one. Splits: 192 tune / 288
calibrate / 320 held out.
| on the held-out split | fixed K (conformal — same promise) | adaptive |
|---|---|---|
| passages sent to the LLM | 9.0 | 5.0 |
| coverage | 91.9% | 90.2% [86.3, 93.4] |
| when the answer ranked 1st | 9 passages | 3.2 passages |
| when the answer ranked 4th or lower | 9 passages | 8.4 passages |
44% fewer prompt tokens at the same guarantee, because the budget can finally move to where it is needed. A reranker that has learned nothing does not break the guarantee — it pays for it in context length, which is the distribution-free property doing its job.
The risk certificate on the same run is refused, and says why: the most permissive
threshold it can prove is one that abstains on everything, so it is labelled degenerate
rather than shipped. The walk stopped at 0.98, which would answer at an observed 2.8%
risk — under the 5% target, but needing ~487 calibration queries to prove against the 288
available. More data is the fix, not a looser target.
The serving path is Java and the calibrator is Python, and a guarantee is a property of one
procedure: if serving computes a different K than calibration assumed, alpha means nothing.
ConformalSetSizerTest
pins the two against 24 golden vectors, each chosen at least 1e-9 from the inclusion
boundary so a last-ulp difference between the two exp implementations cannot flip an answer.
Ships disabled. Until make calibrate produces a certificate, the pipeline keeps its
fixed top-K — a threshold outside [0, 1] is an unconfigured one, not a conservative one.
→ ADR 0013
groundedness = 0.81 is a CHEAP-tier model's opinion of answers produced by the system it
belongs to. If that judge runs eight points optimistic, the published number is eight points
wrong and nothing in the pipeline notices — the interval machinery above would faithfully
report a tight interval around the wrong value. Every RAG system has this problem, and at
scale so do the relevance labels behind any large gold set.
Prediction-powered inference (Angelopoulos et al., Science 2023; power-tuned as PPI++) is the way out. Hand-label a small sample, let the judge score everything, and subtract the bias measured on the labelled part:
theta = mean(hand labels) + lambda x ( mean_unlabelled(judge) - mean_labelled(judge) )
1,500 repetitions of the whole label-and-estimate cycle per row, 50 hand labels against 2,050 judge-scored items, estimating a quantity whose true value is known exactly:
| judge behaviour | judge only | hand labels only | PPI | PPI interval width | effective labels (from 50) |
|---|---|---|---|---|---|
| flattering, accurate | 0.0% | 94.0% | 94.3% | −67% | 474 |
| flattering, noisy | 0.0% | 94.2% | 93.5% | −22% | 84 |
| unbiased, noisy | 14.5% | 94.7% | 93.7% | −22% | 85 |
| uninformative | 0.0% | 94.8% | 94.3% | −1% | 51 |
First three columns: how often each method's published 95% interval actually contained the
truth. Averaging thousands of judge scores is not a 95% interval — it is a very narrow
interval around whatever the judge believes, and when the judge is biased it is essentially
never right. Even the "unbiased" judge fails, because scores clipped to [0, 1] acquire a
bias near the boundary.
Two properties make this worth the code. Validity never depends on the judge being good — the bias is measured, not assumed. And a useless judge costs nothing: λ is tuned to minimise variance, so a judge predicting noise gets λ ≈ 0 and the estimator collapses to the hand-label mean. The bottom row is that safety property working.
$ python run_qa_eval.py gold.jsonl --human-labels judge-labels.jsonl
judge alone 0.886 (no validity — the judge's opinion of its own system)
hand labels only 0.625 [0.156, 1.094] n=4
PPI 0.637 [0.341, 0.934] lambda=1.00
judge bias +0.250 (optimistic)
effective labels 10 (from 4 actually written)This repo has not written those labels yet, so 0.81 below stays marked as the judge's
opinion rather than a measurement. The machinery, the file format and the refusal path all
ship; the remaining work is a couple of hours of hand-grading, not a research project.
→ ADR 0015 · python eval/ppi_experiment.py
reproduces the tables.
Every tool above kept reaching the same conclusion, so:
make beir # SciFact: 5,183 documents, 300 queries, binary qrels
make beir-eval # ingest through the real pipeline, then evaluateBEIR (Thakur et al., NeurIPS 2021 D&B) in this repo's
corpus and gold format, standard library only — no datasets, no pyarrow, nothing to
install. SciFact is the default because its judgements are binary, so this repo's
binary-gain nDCG@10 means the same thing as a published nDCG@10. On a graded dataset the
judgements must be binarised and the resulting nDCG is not comparable; the tool prints that
warning rather than letting a reader assume otherwise.
The design analysis comes with the download, and it is not flattering:
smallest attainable p 9.82e-91 (against 2.0e-01 on a 10-query set)
dMRR@10 = 0.02 at sd 0.30 needs 1,766 queries -> short by 1,466
Thirty times the resolution, and still short of settling a two-point MRR move. Worth printing rather than discovering later.
SciFact, measured. 5,183 documents indexed through the real pipeline — Kafka, chunking, bge-m3, Elasticsearch — and 300 queries evaluated against binary qrels on an idle machine.
| mode | Recall@5 | Recall@10 | MRR@10 | nDCG@10 |
|---|---|---|---|---|
| bm25 | 0.725 [0.67, 0.77] | 0.769 [0.72, 0.81] | 0.639 [0.59, 0.69] | 0.666 [0.62, 0.71] |
| vector | 0.740 [0.69, 0.79] | 0.788 [0.74, 0.83] | 0.615 [0.57, 0.66] | 0.650 [0.61, 0.69] |
| hybrid | 0.802 [0.76, 0.84] | 0.844 [0.80, 0.88] | 0.704 [0.66, 0.75] | 0.734 [0.69, 0.77] |
The BM25 row is the one to read first. It lands at nDCG@10 = 0.666 against the ~0.665
that BEIR's own BM25 baseline reports for SciFact — which is the closest thing this repo has
to an external check that the whole path is correct. Chunking, the analyzer, the ES mapping,
chunk-to-document deduplication and the scoring code all had to agree with the published
setup to land there, and a mistake in any of them would have moved it.
Against that baseline, hybrid is ahead on every metric and the gap is significant under a
paired randomization test, Holm-corrected within each metric (nDCG@10 +0.068, Holm
p = 0.0002). vector alone is not — it is within noise on recall and slightly behind
BM25 on the rank-sensitive metrics (MRR@10 −0.025, Holm p = 0.25). Dense retrieval does not
beat BM25 here on its own; fusion and reranking are where the gain actually comes from, which
is the claim the architecture rests on and is now measured rather than assumed.
The gold set is adequately powered for this comparison and says so: 300 queries resolve an
MRR@10 effect of +0.051 at 80% power, and the measured effect is +0.065. It is not powered
for a two-point move — that needs 1,918 queries — and power_report.py prints both.
Full tables, per-query first-hit ranks and the bootstrap intervals: docs/beir-summary.md and
docs/beir-results.json, regenerated by make beir-eval. What that costs — and the corrections and traps the
first run got wrong about it — are in the
benchmark runbook — starting with the fact that the run is a
~14-hour job on a laptop CPU rather than the minutes the reranking latency suggests.
→ ADR 0014
That threshold two sections up is eighteen significant figures resting on an assumption printed nowhere near it: the queries it was calibrated on and the queries it serves must come from the same distribution. A live corpus breaks that in a week — documents get ingested, query mix moves, someone swaps the embedding model — and the failure is silent. Config unchanged, dashboards green, promise no longer true.
The level becomes a control variable driven by realised miscoverage (Gibbs & Candès, NeurIPS
2021). Summing the update telescopes to an exact identity, and alpha_t cannot escape
[-gamma, 1+gamma], so long-run coverage converges to target under no distributional
assumption at all — not exchangeability, not stationarity, not even that the shift is
random rather than adversarial.
200 streams, 3,000 queries each, promise 90%:
| scenario | frozen threshold | rolling window | adaptive |
|---|---|---|---|
| location shift (+0.30) | 0.459 | 0.876 | 0.900 |
| score saturation | 0.306 | 0.939 | 0.903 |
The middle column is the interesting one, and it is not the result I went looking for. Against a plain location shift a rolling window does essentially all the work by itself — the quantile moves with the data and the controller never sees the shift. The tests pin that as an identity: across shifts of 0.00, 0.20 and 0.30, the widest excursion of the level from target is the same 0.100. An earlier draft of this section claimed the controller was catching drift there. It was not, and the number that looked like proof was an artefact of the test generator clamping scores at 1.0.
What the controller is actually for is the residual and the case the window cannot track. The
window is systematically off even when it works — undercovering at 87.6%, and overcovering
at 93.9% under saturation, which is not safety but prompt tokens spent on sets larger than
the guarantee needs. And when most candidates pile onto one tied top score, no rolling
quantile separates them; only a displaced level holds coverage together, which is why the
signal is called compensating and not drifting. It stays False through a shift the window
absorbs (0/60) and fires on every saturating stream (60/60).
And the serving path says it out loud. CoverageMonitor (Java) watches the deployed
certificate using the groundedness judge as its coverage signal, and warns once the promised
rate has been ruled out — 43 queries into a stream missing 35% of the time. It has to be
anytime-valid, and that is where #1 pays for itself: this stream is inspected after every
query, which is exactly the arrangement where a fixed-N interval was measured missing 30-35%
of the time. A binomial test here would cry wolf weekly.
The alarm is one-sided, because a system covering better than promised is the ordinary state of a conservative method, not an incident — over-coverage is a token-cost gauge, not a page. Two caveats the code states rather than hides: the judge is a biased instrument (#3), so this is the judge's miscoverage and not the truth, and judging skips abstentions and cache hits, so it watches a subpopulation of traffic. A smoke detector, not a thermometer. The controller itself stays offline — it needs a nonconformity score per query, and serving has no gold label to compute one from; shipping it anyway would be a guarantee-shaped object with no guarantee in it.
→ ADR 0016 · python eval/adaptive_experiment.py
reproduces the table
The nightly sweep searches a stratified dev split, and a proposal must survive four guards — an effect-size floor, a Holm-corrected paired randomization test across the whole grid, and a held-out split the search never touched that must still show the improvement. The dev-minus-held-out gap is printed as the overfitting the search introduced. When the guards fail for want of data, the report names the query count that would settle it instead of suggesting a smaller epsilon.
On the shipped ten-query gold set this means it will essentially never fire. That is the correct behaviour. → ADR 0011
Everything above is available on its own, because none of it is specific to this system:
pip install retrieval-eval-gate # `rag-eval-gate` on PyPI is an unrelated project
rag-eval-gate power gold.jsonl # what your gold set can resolve, before anything runs
rag-eval-gate audit -n 10 -v 1.00 -b 0.70 # or audit a published table, with no data at allThere is also a browser version that needs nothing installed: jinwovo.github.io/rag-eval-gate.
→ jinwovo/rag-eval-gate — zero dependencies, and backend-agnostic in a way this repo's own harness is not: an HTTP URL template with a field map, a command printing JSON to stdout (so the retriever's language is irrelevant), or a TREC run file (so a corpus can be ranked once, offline, and evaluated with nothing running). Gold sets are JSONL or TREC qrels.
- uses: jinwovo/rag-eval-gate@v0
with:
gold-file: eval/gold.jsonl
url-template: 'http://localhost:8080/api/search?q={query}&mode={mode}'
gate-policy: sequential # point | ci-lower | regression | sequential
min-mrr10: "0.85"This repo keeps its own vendored copy for the in-tree gate, which also seeds a corpus through the Kafka pipeline and waits for async indexing to converge — something a general tool has no business knowing about:
- uses: jinwovo/recall/.github/actions/rag-eval-gate@main
with:
api-url: http://localhost:8080
gold-file: eval/gold.jsonl
corpus-file: eval/corpus.jsonl # seed + wait for async indexing first
gate-policy: sequentialEither way you get a 95% interval on every metric, each mode significance-tested against the
baseline with Holm correction, a statement of what your gold set can resolve at all, and a
✅ PASS / ❌ FAIL in the step summary. Our own eval workflow consumes the in-tree action —
dogfooding is the compatibility test.
gate-policy |
fails when | use it when |
|---|---|---|
point (default) |
the mean falls below the threshold | always — the absolute floor |
ci-lower |
the 95% lower bound falls below the threshold | your gold set is large enough that the interval is narrower than the safety margin |
regression |
also on a significant paired drop vs a recorded run | you have a green baseline to compare against |
sequential |
the anytime-valid verdict is fail, or the budget runs out undecided |
queries cost money or minutes |
regression is the sensitive one: an absolute threshold only notices a regression once the
mean crosses a line someone guessed, while the paired test compares the same queries before
and after, so a real drop on two queries is caught while the mean still clears the line.
Because it is: if it computes nDCG wrong, every number downstream is wrong and nothing else in the build will say so.
make eval-test # 245 tests, standard library only, no stack, ~65s- Closed forms —
(α/2)^(1/n)atk = n,2^(1-k)for a uniform improvement, R'sp.adjustworked example, the(n+1)conformal quantile correction. - Independent implementations — a brute-force permutation reference written separately in the test; SciPy cross-checks that skip when SciPy is absent.
- The guarantees, by simulation — repeated calibrate-then-deploy cycles counting how often each promise actually breaks: BCa coverage against a skewed population, confidence-sequence coverage under continuous inspection, conformal coverage on data the fit never saw, type-I error of the risk certificate, and PPI coverage against a deliberately biased judge — with the negative control alongside it, since an estimator is only interesting if the thing it replaces demonstrably fails.
- The harness end to end — against a stubbed search backend whose metrics are derivable by hand, driving all four gate policies into both outcomes and asserting the sequential one's saving is real requests not sent.
Plus 85 backend unit tests and three Testcontainers integration suites against real ES, Kafka and MinIO.
Grounded RAG QA, live (4×) — a Korean question over an English corpus (cross-lingual
retrieval via bge-m3): the answer streams token-by-token with inline [n] citations while the
pipeline stepper tracks retrieve → generate → verify; the grounded badge is the
post-hoc LLM judge's verdict (ADR 0004), and clicking a citation highlights the cited source.
Generation is a free local LLM (Ollama, CPU):
Hybrid search view — retrieval-mode toggle (hybrid / bm25 / vector), per-source score bars, and client-measured latency:
Run it yourself in ~5 minutes. No API key needed (free local LLM via Ollama).
cp .env.example .env # optional: ANTHROPIC_API_KEY, or use free Ollama
docker compose up -d # ES (Nori), Redis, Postgres, Kafka, MinIO, sidecar, Grafana
cd backend && ./gradlew bootRun # wrapper is committed; a JDK 21 is all you need
cd frontend && npm install && npm run dev # http://localhost:3000
python scripts/seed_corpus.py # ingest the eval corpus (waits for indexing)
make eval-gate # bm25 vs vector vs hybrid, with intervalsUse it as a self-hosted hybrid-search / RAG starter for your own corpus:
python scripts/ingest_folder.py ~/my-docs # .md / .txt / .html (+ .pdf with pypdf)Every file goes through the real pipeline — Kafka, chunking, embeddings, idempotent upserts, the durability contract of ADR 0005 — then the command waits until everything is actually searchable. Re-running is the sync mechanism.
mcp-server/ exposes recall_search and recall_ask over the Model Context
Protocol, so your documents become agent tools in Claude Desktop / Claude Code, citations and
groundedness verdict included.
Query / RAG path:
flowchart LR
Q([user query]) --> BFF[Spring Boot BFF]
BFF -->|embed query| SIDE[bge-m3 + reranker<br/>Python sidecar]
BFF -->|BM25 + kNN| ES[(Elasticsearch<br/>Nori + dense_vector)]
ES --> FUSE[RRF fuse + rerank]
SIDE --> FUSE
FUSE --> SIZE[conformal set sizer<br/>adaptive K]
SIZE -->|certified context| RAG[assemble prompt]
RAG --> LLM[LLM provider<br/>Claude · Groq · Ollama]
RAG -.lookup / store.-> REDIS[(Redis<br/>semantic cache)]
LLM ==>|SSE tokens + citations| Q
Ingestion path — async, idempotent, and loss-proof (retry/backoff → DLQ; large docs travel as MinIO references via the claim-check pattern):
flowchart LR
DOC([document]) --> API[Spring Boot]
API -->|archive raw| MINIO[(MinIO)]
API -->|publish, acks=all| KAFKA[(Kafka)]
KAFKA --> W[ingest worker]
W -.claim check<br/>fetch by objectKey.-> MINIO
W -->|chunk + embed| SIDE[bge-m3 sidecar]
W -->|idempotent upsert<br/>by content hash| ES[(Elasticsearch)]
W -->|poison pill or<br/>retries exhausted| DLQ[(dead-letter topic)]
Component detail in docs/ARCHITECTURE.md.
Measured on the running stack over the 24-document corpus with the 10-query gold set (exact-term, paraphrased, and Korean cross-lingual), on the BBQ-quantized index. Doc-level metrics; chunk hits deduplicated by first occurrence.
| mode | Recall@5 | 95% CI | MRR@10 | nDCG@10 |
|---|---|---|---|---|
| BM25-only | 0.70 | [0.35, 0.93] |
0.67 | 0.70 |
| vector-only | 1.00 | [0.69, 1.00] |
0.90 | 0.93 |
| hybrid (RRF + cross-encoder) | 1.00 | [0.69, 1.00] |
0.95 | 0.96 |
hybrid, rerank=m3 (tri-modal self-hybrid) |
1.00 | [0.69, 1.00] |
0.90 | 0.93 |
mode=hyde (hypothetical-doc kNN, no rerank) |
0.90 | [0.56, 1.00] |
0.80 | 0.85 |
Read this table the way the audit above says to. The Recall intervals follow exactly from
the reported values — one relevant document per query makes each a success count out of ten —
and they are wide because ten queries is a small sample, not because the system is unstable.
The MRR and nDCG columns are point estimates; run_eval.py --json produces their bootstrap
intervals on any rerun. The BM25-to-hybrid ordering is consistent and unsurprising — exact-term
queries favour BM25, paraphrased and Korean queries favour dense vectors, both Korean queries
are BM25 misses that hybrid ranks first — but the size of the gap is not resolvable at this
sample size, which is what make beir is for.
Quantization is quality-neutral here: the same sweep on the float32 index scored within ±0.02 on every dense mode.
Every retrieval stage cites its source, and each was adopted the same way: implemented, swept by the eval harness, numbers published.
| Technique | Paper | In this repo |
|---|---|---|
| Reciprocal Rank Fusion | Cormack et al., SIGIR 2009 | BM25 + kNN fusion (rrf-k=60) |
| Multilingual dense retrieval + cross-encoder rerank | BGE / bge-reranker-v2-m3, Chen et al. 2024 | embedding sidecar, default rerank |
| M3 tri-modal self-hybrid (dense + sparse + ColBERT) | BGE M3-Embedding, Chen et al. 2024 | ?rerank=m3 — the embedder's own heads, no extra model |
| HyDE (hypothetical document embeddings) | Gao et al., ACL 2023 | ?mode=hyde, fail-open to vector search |
| Sufficient-context gate | Joren et al., ICLR 2025 | pre-generation autorater, abstain before the PRIMARY call |
| RaBitQ 1-bit quantization (ES BBQ) | Gao & Long, SIGMOD 2024 | bbq_hnsw + oversampled rescore, ~32× less vector memory |
| Paired randomization test for IR | Smucker, Allan & Carterette, CIKM 2007 | mode comparisons, Holm-corrected |
| Betting confidence sequences | Waudby-Smith & Ramdas, JRSS-B 2024 | the sequential gate |
| Conformal prediction, adaptive sets | Romano, Sesia & Candès, NeurIPS 2020 | adaptive context sizing |
| Risk-controlling prediction sets | Bates, Angelopoulos, Lei, Malik & Jordan, JACM 2021 | calibrated abstention |
| Prediction-powered inference | Angelopoulos, Bates, Fannjiang, Jordan & Zrnic, Science 2023 | a valid interval for true groundedness from a biased judge |
| BEIR | Thakur et al., NeurIPS 2021 D&B | make beir |
GraphRAG / RAPTOR and late chunking were evaluated and deliberately deferred — the rejection reasoning is in ADR 0008.
Measured via eval/run_qa_eval.py with the free local provider
(Ollama, qwen2.5-coder:3b on CPU) doing both generation and judging:
| metric | value |
|---|---|
| groundedness (avg judge score) | 0.81 |
| verdicts (8 judged) | 75% supported · 12% partial · 12% unsupported |
| abstentions ("I don't know") | 2/10 — declined instead of hallucinating |
| citation coverage | 100% of generated answers contain [n] citations |
| TTFT p50 / e2e p50 | 30.5s / 54s (local CPU 3B — prefill-bound, not representative of API providers) |
The one UNSUPPORTED answer was flagged by the judge and excluded from the semantic cache
(ADR 0004) — the guardrail producing signal, not a green checkmark. Eight judged answers is
the same small-sample problem as everything else on this page, and is why the QA eval is not
yet a CI gate.
Both guards are fail-open, and both sit on either side of the LLM call.
Before (ADR 0009 / ICLR 2025, threshold now certified by ADR 0013): when the reranker's
top score is below the calibrated threshold, a CHEAP-tier autorater answers one word — can
these passages answer this question? INSUFFICIENT → the pipeline abstains before spending
the PRIMARY generation. Confident retrievals skip the check entirely, so p50 TTFT is untouched.
After (ADR 0004): every generated answer is graded after it streams (so TTFT is unaffected)
by a CHEAP-tier judge that sees the same passages plus the finished answer and returns
SUPPORTED / PARTIAL / UNSUPPORTED. The verdict streams to the UI as a badge, lands on the
query_log row, and feeds Prometheus. Abstentions are never graded as hallucinations, and
judged-UNSUPPORTED answers are never cached.
POST /api/ingest returning 202 is a durability contract, not an optimistic ack: the raw
document is archived to MinIO and the Kafka publish is broker-acknowledged (acks=all) before
the response. Documents above 64 KB travel through Kafka as an objectKey reference — the
claim-check pattern — so document size never fights broker message limits. On the consumer,
transient failures retry in place with exponential backoff; malformed events skip retries
entirely; both end up on recall.ingestion.dlq with forensic headers ready for replay. Every
branch is proven by IngestionReliabilityIT against real Kafka + MinIO.
The DLQ is a queue, not a graveyard: the /admin ops page closes the incident loop —
inspect the decoded forensics, fix the fault, one-click replay. Replay re-publishes
broker-acked before committing DLQ offsets (at-least-once by choice — chunk upserts are
idempotent by content hash), strips stale kafka_dlt-* headers so a re-failure earns fresh
forensics, and stamps provenance that survives repeated round-trips. Proven end-to-end by
DlqReplayIT. → ADR 0005,
ADR 0006
- Backend: Java 21, Spring Boot, WebFlux (SSE)
- LLM: pluggable via
recall.llm.provider— Claude (default), or OpenAI-compatible Groq / Ollama (local, free, no key) - Search / vector: Elasticsearch 8.18 (Nori analyzer +
dense_vectorkNN, BBQ-quantized with oversampled rescoring) - Embedding / rerank:
BAAI/bge-m3+BAAI/bge-reranker-v2-m3(Python FastAPI sidecar) - Messaging: Kafka (async ingestion) · Stores: Redis, PostgreSQL, MinIO/S3
- Frontend: Next.js, TypeScript, Tailwind · Observability: Micrometer + Prometheus + Grafana
- Eval: standard-library Python — no dependency the composite action would have to install
| Method | Path | Notes |
|---|---|---|
GET |
/api/search?q=&mode=&rerank= |
mode = hybrid (default) | bm25 | vector | hyde; rerank = cross-encoder (default) | m3 |
GET |
/api/ask?q= |
SSE stream: sources, sufficiency, token, judging, groundedness, done |
POST |
/api/ingest |
async index; 202 ⇒ raw doc archived + broker-acked |
GET |
/api/admin/dlq?limit= |
DLQ depth + bounded peek with decoded forensic headers |
POST |
/api/admin/dlq/replay?max= |
drain pending DLQ records back onto the ingestion topic |
GET |
/actuator/prometheus |
metrics scrape |
backend/ Spring Boot (Java 21) — search, rag, ingestion, llm providers, cache
embedding-service/ Python FastAPI sidecar — bge-m3 embeddings + bge-reranker
frontend/ Next.js UI — search / QA, streamed answers, citation highlight
mcp-server/ MCP stdio server — your corpus as agent tools
eval/ the harness: stats, sequential, conformal, calibrate, beir, power_report
deploy/helm/recall/ Helm chart (k8s)
monitoring/ Prometheus + Grafana provisioning
docs/adr/ every decision, including the ones that overturned earlier ones
- ADR 0001 — Hybrid retrieval (BM25 + vector + RRF + rerank)
- ADR 0002 — LLM cost & latency optimization
- ADR 0003 — Async, idempotent ingestion via Kafka
- ADR 0004 — Post-hoc groundedness judge
- ADR 0005 — Ingestion reliability: retries, DLQ, claim-check storage
- ADR 0006 — DLQ replay on the admin surface
- ADR 0007 — Retrieval eval as a CI regression gate
- ADR 0008 — Paper-backed retrieval: M3 self-hybrid & HyDE
- ADR 0009 — Sufficiency gate & BBQ-quantized vectors
- ADR 0010 — From portfolio to tool: folder ingestion, MCP, reusable gate, self-tuning
- ADR 0011 — Retrieval scores are estimates, and are reported as such
- ADR 0012 — Anytime-valid evaluation: stop when the verdict is decided
- ADR 0013 — The constants that decide cost and hallucination become certificates
- ADR 0014 — A benchmark the harness can actually resolve
- ADR 0015 — The LLM judge is a measuring instrument, and it was never calibrated
- ADR 0016 — The certificate expires, and nothing tells you
- Groundedness eval as a second CI gate, once a budgeted API key makes LLM-judging in CI viable
- Claude native citations (exact char-span grounding) when the
claudeprovider is configured - Calibrating the abstention threshold against judge verdicts rather than the retrieval proxy
기술 지식베이스 대상 하이브리드 검색(BM25+벡터) + RAG 질의응답 플랫폼입니다. 다만 이 저장소의
중심은 검색이 아니라 측정입니다 — 처음 만든 시스템의 품질 주장을 검증할 도구를 나중에 만들었고,
그 도구를 이 저장소의 README에 먼저 겨눴더니 헤드라인 세 개가 무너졌습니다(Recall@5 = 1.00은
실제로 [0.69, 1.00], "BM25 대비 +0.30"은 p=0.25이며 그 0.25가 도달 가능한 최선값, 야간 자가튜닝은
7,000개 쿼리가 필요한 차이를 10개로 판정 중).
그래서 세 가지를 보증(guarantee) 으로 바꿨습니다: (1) 언제 들여다봐도 유효한 평가
(anytime-valid confidence sequence) — 판정이 결정되는 즉시 중단해 쿼리 예산 70–90% 절감,
(2) 컨텍스트 크기와 기권 임계값의 분포무관·유한표본 보증(conformal risk control) — 동일 보증
하에 프롬프트 토큰 40% 절감, (3) BEIR 벤치마크로 실제 판별 가능한 규모 확보. 설계 근거는
docs/adr/ 0011–0014에 있습니다.
MIT — see LICENSE.