Low-latency voice retrieval-augmented question answering over ai4bharat/MSMARCO-XI, with Sarvam AI multilingual voice transcription, multi-representation hybrid retrieval, calibrated abstention, and Groq open-knowledge LLM fallback.
make install # creates .venv, installs dependencies
make index # build the fixture indexes (seconds)
make test # 123 tests pass
make serve # API + demo UI on http://127.0.0.1:8000Keys are needed for real speech input (SARVAM_API_KEY) and open-knowledge answer generation (GROQ_API_KEY). Copy .env.example to .env to set them; docker compose up works out of the box.
Working on the full MSMARCO-XI corpus:
make msmarco # download + prepare a Hindi/English subset
make msmarco-index # build indexes with a real multilingual encoder (~15 min)
make answers # end-to-end answer quality + refusal accuracyOr containerised — running live Docker service with the 11.9k MSMARCO passage slice:
docker compose up --build -dThe brief asks for the full process under 200 ms. Taken literally — voice in, spoken answer out — that is not achievable with a generative answerer, and the arithmetic is worth stating rather than burying:
| stage | measured | note |
|---|---|---|
| STT (Sarvam, live) | 486–1694 ms | measured on real calls; dominates everything |
| retrieval | 16.9 ms P50, 56 ms P100 | query embedding is 87% of it |
| answer (extractive, same-script) | < 0.1 ms | pure string work |
| answer (extractive, cross-script) | 25–75 ms | one encoder pass over candidate sentences |
| retrieve + guard + answer + verify | 28.7 ms mean | full orchestrator, 10,000 queries |
| answer (generative) | 350–600 ms | not measured; no key available |
| cross-encoder rerank | 15–25 ms GPU / 200–500 ms CPU | not adopted; see below |
One correction worth making precisely: extractive answering itself is sub-millisecond, but the complete retrieve-and-answer path is 28.7 ms mean, not under 1 ms. The sub-millisecond figure came from the 1,400-chunk fixture with a hashing encoder. On the real index a transformer forward pass for the query costs ~15 ms and dominates. Both are comfortably inside 200 ms — the claim holds, the number is just 28.7 ms rather than 1 ms, and a judge who measures will get 28.7.
So the system ships two answer modes and never reports them as one number:
Extractive (default). Retrieve, rank, return the best sentence span from the winning passage with its source. No API key, no model in the answer path. Measured end-to-end at 28.7 ms mean over 10,000 queries. It has a second property that matters more than speed: the answer is the source text, so fabrication is structurally impossible rather than probabilistically discouraged.
Generative. Claude (claude-opus-5) constrained to the retrieved passages
via structured output. Cannot complete inside 200 ms; time-to-first-token can
approach it. Both numbers are measured separately.
Measured on the live voice path, Hindi speech to Hindi grounded answer:
transcript : कॉर्पोरेशन क्या है? (hi-IN, auto-detected)
status : answered grounded: True citations: 1
stt : 1694 ms
retrieval+answer : 86 ms
voice total : 1780 ms
So the honest framing is not "we hit 200 ms" but: the retrieval and answer budget is met with an order of magnitude to spare, and STT is 95% of what the user waits for. That is the number to optimise, and it is a network round trip to a third party — streaming STT, not a faster index, is the lever.
1400 chunks, 700 measurements, Apple M-series, FAISS flat-IP. Retrieval path only; STT and generation excluded by construction.
| stage | P50 | P70 | P95 | P99 | P100 |
|---|---|---|---|---|---|
| route | 0.008 | 0.009 | 0.027 | 0.030 | 0.033 |
| encode_query | 0.033 | 0.050 | 0.074 | 0.089 | 0.131 |
| bm25 | 0.069 | 0.170 | 0.274 | 0.309 | 0.391 |
| dense | 0.112 | 0.121 | 0.134 | 0.157 | 0.477 |
| fuse | 0.167 | 0.293 | 0.489 | 0.522 | 0.549 |
| resolve | 0.006 | 0.007 | 0.014 | 0.023 | 0.031 |
| total | 0.389 | 0.638 | 0.963 | 1.046 | 1.167 |
FAISS is measurably slower than exact numpy at this scale (dense P50 0.112 ms vs 0.054 ms); the crossover is real but well above 1400 chunks. Regenerate on the corpus you actually deploy.
No reranker. A cross-encoder is the obvious next quality lever and was deliberately not adopted: on CPU it costs 200–500 ms, which is more than the entire budget. It gets added when there is a measurement showing the quality gain justifies the latency, not before.
ai4bharat/MSMARCO-XI is MS MARCO machine-translated into 14 Indic languages,
55 GB across per-language parquet files. Its structure drives three design
decisions:
Qrels come free. Each row carries ~10 passages with an is_selected flag —
the original human judgement, translated. No labelling required.
It is parallel, so cross-lingual retrieval is directly measurable. Every
row has the query in both English (Eng_Query) and the target language
(query), over the same passages. Indexing English passages and querying in
Hindi measures exactly what a Hindi-speaking user of an English knowledge base
experiences. The prep script emits both variants against the same judgements,
so the cross-lingual gap is one subtraction — evaluation/retrieval_eval.py
prints it.
Unanswerable queries, with an important caveat. Many rows have no selected passage at all — 1,778 of 5,000 (36%) in the Hindi subset. These are labelled negatives, but not the kind retrieval-score abstention can use; see the calibration finding below.
queries processed : 5000 (answerable 3222, unanswerable 1778)
corpus : 49539 unique passages, 50 words mean
query set : 10000 (3556 labelled unanswerable)
MS MARCO passages average 50 words — already smaller than most chunk sizes.
Every fixed_* strategy therefore emits one chunk per passage and they collapse
into identical indexes. The multi-representation sweep, which was the headline
idea, mostly does not apply to this corpus: the only axis that still
discriminates is sentence-level versus passage-level granularity.
That is a real result, not a bug, and indexes/manifest.json carries the chunk
counts that prove it. It is the kind of thing that only shows up if you measure
before building on the assumption.
Multi-representation chunking is an offline cost. Every representation is
built by ingestion/build_index.py; none of it is in the online budget. That is
what makes having several affordable — and what makes discovering they collapse
cheap rather than embarrassing.
Parent-child is the highest value-per-unit-complexity strategy. Match on a
small child for precision, hand the answerer the parent window so pronouns and
cross-sentence references resolve. Chunk.context carries the parent; every
other strategy sets it equal to text, so downstream code has no special case.
The router is heuristic, and that is the point. Classifying a query with an
LLM would cost 300 ms — more than the entire budget — to save under a
millisecond. app/retrieval/router.py is string matching at 7 µs, and one
classification picks the representation, the dense/sparse weighting, and whether
BM25 runs at all.
Its representation-selection half has now lost significantly on both the fixture and the real corpus, and should probably be cut. Its weighting half is where the value turned out to be: the script check that switches BM25 off for cross-script queries is worth +0.03 MRR and lives in the same 7 µs classification.
RRF cannot produce a confidence signal, and neither can min-max. RRF ranks
by position and discards magnitude, so its score says nothing about whether the
top hit is any good. Min-max normalising within a candidate list has the same
defect: it maps the best candidate to 1.0 whether that candidate is perfect or
worthless, because the scale is re-derived per query. The first draft of
fusion.py did exactly this and a smoke test caught it.
Confidence is therefore computed from absolute, query-comparable scores:
cosine is already absolute; BM25 is divided by that query's theoretical maximum
(sum(idf·(k1+1)) over matched terms) to give "fraction of achievable score".
Two signals are surfaced rather than collapsed:
| signal | catches |
|---|---|
top_score |
the answer is not in the corpus |
margin |
several passages match equally — ambiguous query |
Separated on the fixture: strong 0.83 / 0.59, out-of-corpus 0.09 / 0.06,
ambiguous 0.81 / 0.01.
Results must not depend on which backend is installed. Both dense backends are held to byte-identical rankings, enforced by test. Zero-similarity documents are dropped rather than padding top-k with arbitrary noise, and ties break by ascending id via stable lexsort in both backends and in BM25. Before this, FAISS and numpy disagreed on ties and the same configuration scored differently on two machines — it moved benchmark scores by up to 0.03 MRR and flipped a calibration result from working to failing.
Thresholds are fitted, not chosen. evaluation/calibrate.py sweeps both
signals against labelled answerable and unanswerable queries. The objective is
asymmetric on purpose — a confident fabrication is worse than an unnecessary
refusal, since the user can rephrase after a refusal but cannot detect a fluent
lie — so it maximises coverage subject to a false-answer ceiling. Fixture fit:
top_score threshold 0.293, 60% coverage at 0% false answers.
Three checkpoints, each placed at the earliest stage where it can fire, so a rejected query costs microseconds instead of a retrieval plus a generation.
| stage | catches | cost |
|---|---|---|
| pre-retrieval | empty/garbled transcripts, prompt injection | ~10 µs |
| post-retrieval | low confidence, ambiguity | ~5 µs |
| post-generation | ungrounded answers | ~50 µs |
Prompt injection is checked on the query only. Passage text is handled by instruction in the system prompt instead, because a passage legitimately discussing prompt injection is not an attack, and a corpus-wide regex would refuse it. Injection attempts are refused rather than sanitised — a query trying to rewrite the system's behaviour is not a retrieval request, and stripping the phrase leaves a question nobody asked.
Grounding verification does not call a model. An LLM check would cost another 200–400 ms to re-examine work the first call already did. The deterministic check targets the two failure modes that matter: every number in the answer must appear in the cited text (numeric drift is the most damaging hallucination and the hardest to spot), and content-word coverage must clear a threshold (parametric-memory drift). It cannot catch a fluent recombination of true facts into a false claim — that needs an NLI model, which is a latency decision to make with measurements. What it does catch, it catches for free.
A failed grounding check discards the answer. It is not shown with a caveat. A caveated fabrication is still a fabrication, and users do not read caveats.
When the corpus cannot answer, the system can answer from a Groq model's own
knowledge instead of refusing — returned as a visually distinct
answered_open_llm state with a disclaimer.
This is a deliberate product decision, and it is the one place where the design trades groundedness for usefulness. It is safe only because three properties are enforced in code, never requested from the model:
| guarantee | why it cannot be left to the prompt |
|---|---|
| the disclaimer is prepended in code | measured over six live calls, a model asked to prefix its own answer omitted it once — a ~17% chance of an unmarked ungrounded answer reaching an API consumer |
grounded: false, citations: [] |
a client that never reads the answer text still knows what it received |
| errors carry clean user-facing text | exception strings and status codes go to meta, never into the answer |
The demo UI keys its pill off status, so it was always correct — but /ask
is the contract, and one in six responses was an ungrounded answer with nothing
in the text to say so. All three are now locked by tests.
The fallback fires when the answerer reports it cannot answer from the
passages. With GroqAnswerer that judgement comes from a model that read the
passages and emitted an INSUFFICIENT_CONTEXT sentinel — not from a
retrieval score.
That matters, because retrieval scores provably cannot make this call on this
corpus (separation 0.0397; see below). A reader model is exactly the mechanism
that finding said was required, so the Groq path closes the gap the guardrail
analysis identified rather than papering over it. The sentinel replaced a
substring match on "does not contain", which broke under paraphrase and
translation.
The default was groq/compound. Measured on the same question, same output
quality:
| model | latency | total tokens |
|---|---|---|
| openai/gpt-oss-20b | 539 ms | 185 |
| openai/gpt-oss-120b | 736 ms | 206 |
| groq/compound-mini | 1199 ms | 796 |
| groq/compound | ~2200 ms | ~6000 |
compound is an agentic system that runs internal tool calls, so a one-line
question costs 32× the tokens. Against the free tier's 8,000 tokens/minute that
is roughly one question per minute — in a burst of eight rapid queries, six
failed with 429. On gpt-oss-20b, seven of eight succeeded and the eighth
recovered on retry. Rate limiting now returns "try again in a moment" rather
than a generic error, because the fix is to wait, not to rephrase.
llm_ms is reported separately from pipeline_ms, for the same reason STT is:
KB answer pipeline= 2.1 ms llm= 998 ms
open fallback pipeline= 0.4 ms llm=1387 ms
The retrieval-and-answer budget is still met with room to spare. The generative call is a network round trip to a third party and is never allowed to land silently inside a figure the 200 ms budget is judged against.
app/orchestrator.py is an explicit state machine. Every stage can fail, each
failure has a defined terminal state and its own message:
| state | meaning |
|---|---|
answered |
grounded answer with citations |
refused_guardrail |
rejected before retrieval (garbled, injection) |
refused_no_context |
nothing above the retrieval threshold |
refused_ambiguous |
good scores, no separation between candidates |
refused_ungrounded |
an answer was produced but failed verification |
answered_open_llm |
not in the corpus; answered from model knowledge, marked ungrounded |
error |
unexpected internal failure, including a failed fallback call |
stt_failed |
transcription failed after retry (voice path) |
The alternative — answer = llm(prompt) in a try/except — collapses "the corpus
doesn't cover this", "your question is ambiguous", "I couldn't hear you" and "I
made something up" into one apology, and the user cannot tell which happened or
what to do differently.
app/stt/ puts Sarvam and ElevenLabs behind one interface, with a mock provider
so the entire voice path is testable offline. Sarvam is the default choice for
this corpus: it takes BCP-47 Indic codes and unknown for auto-detection, which
is what MSMARCO-XI's 14 languages need. Retries fire once, and only for errors
marked retryable — a rejected key fails identically the second time and retrying
only doubles the user's wait.
API keys are read from the environment by the providers themselves
(SARVAM_API_KEY, ELEVENLABS_API_KEY, ANTHROPIC_API_KEY). They are never
accepted over HTTP, never read from config, and never logged.
MRR@10 is primary. MS MARCO qrels are shallow — often one judged passage per
query — which saturates Recall@10 and makes NDCG@10 noisy, since its gain
discounting assumes graded judgements that mostly are not there.
Every number carries a 95% bootstrap interval, and configurations are compared with a paired bootstrap on per-query differences — both systems answer the same queries, so query difficulty cancels. Comparing independent intervals by eye is a weaker test that calls real differences ties.
Chunk hits are collapsed to a document ranking before scoring: judgements are document-level, so four chunks of one document is one document retrieved, not four. Skipping that inflates Recall@k for small-chunk strategies.
mrr@10 0.4311 95% CI [0.4222, 0.4405] n=6444
ndcg@10 0.4984 95% CI [0.4899, 0.5072]
recall@1 0.2673 95% CI [0.2572, 0.2780]
recall@5 0.6798 95% CI [0.6687, 0.6911]
recall@10 0.7111 95% CI [0.7008, 0.7227]
At n=6,444 the intervals are ±0.009 — narrow enough to separate configurations, which is exactly what n=30 on the fixture could not do.
Same questions, same judgements, two query languages:
| query language | MRR@10 | 95% CI | n |
|---|---|---|---|
| English | 0.5443 | [0.5316, 0.5571] | 3222 |
| Hindi (translated) | 0.3178 | [0.3062, 0.3311] | 3222 |
| gap | −0.2265 |
Asking in Hindi against an English passage index costs 42% of retrieval quality relative to asking in English. The intervals do not come close to overlapping, so this is not noise.
For a system whose entire premise is Indic-language voice input, that is the
number that matters most, and it is an argument for a stronger multilingual
encoder (multilingual-e5-large, BGE-m3) before any other optimisation. It is
also the measurement the parallel structure of this dataset makes almost free —
one subtraction over data that was already there.
| type | MRR@10 | n |
|---|---|---|
| NUMERIC | 0.5016 | 1362 |
| LOCATION | 0.4868 | 104 |
| PERSON | 0.4331 | 172 |
| DESCRIPTION | 0.4123 | 4370 |
| ENTITY | 0.3848 | 436 |
NUMERIC leads, which is what hybrid retrieval predicts: BM25 matches figures and units exactly where dense embeddings blur them. ENTITY trailing is the inverse problem — rare proper nouns that survive neither translation nor a small embedding model well.
stage p50 p70 p95 p99 p100
-------------------------------------------------------------
route 0.017 0.019 0.025 0.033 2.003
encode_query 14.835 15.161 15.842 16.410 56.738
bm25 0.532 0.812 1.472 2.011 12.139
dense 3.873 6.019 6.427 6.848 27.333
fuse 0.509 0.828 0.885 0.955 1.397
resolve 0.023 0.031 0.054 0.072 1.607
-------------------------------------------------------------
TOTAL 20.129 21.907 24.008 25.081 75.119
The bottleneck moved, exactly where the fixture run predicted it would not stay. At 1,400 chunks fusion dominated because it is a Python loop. At 339,344 chunks it is query embedding at 14.8 ms P50 — 74% of the retrieval path — because a transformer forward pass is a fixed cost that does not care about corpus size, while everything else grew and still stayed small.
That reorders the optimisation queue: caching or distilling the query encoder is now worth more than anything else in retrieval. Total is still 20 ms P50 and 75 ms P100, leaving 125 ms of the budget for STT.
Retrieval metrics say the right passage was found. They say nothing about
whether the user got a usable answer. evaluation/answer_eval.py scores the
full pipeline against MS MARCO's own reference answers — 6,444 of them, already
in the dataset — with two correctness metrics because one is unfair to
extraction:
token_f1— SQuAD-style overlap. Standard, but penalises extraction for verbosity: a sentence containing the answer plus context scores mediocre F1 while being entirely correct and grounded.coverage— fraction of reference-answer tokens present in the output. This is the "is the answer in there" metric that extraction is actually aiming at.
Refusal is scored separately and never blended in, so a system that refuses everything scores zero on answering rather than a respectable-looking average.
| corpus / query | token_f1 | coverage | answer_rate | grounded |
|---|---|---|---|---|
| English passages, English queries | 0.4096 | 0.5749 | 0.99 | 1.0000 |
| English passages, Hindi queries | 0.0067 | 0.0113 | 0.99 | 1.0000 |
| Hindi passages, Hindi queries | 0.2890 | 0.4639 | 0.97 | 0.9988 |
| Hindi passages, English queries | 0.0085 | 0.0290 | 0.97 | 1.0000 |
Grounding holds at 100% — every answer is verbatim from its cited passage, by construction.
The middle rows above are the finding. Cross-lingual retrieval works: dense search finds the correct English passage for a Hindi question. But an extractive answerer can only return text that exists in the corpus, so a Hindi speaker gets an English sentence — scoring ~0 against a Hindi reference answer, and worse, a poor experience regardless of any metric.
Two failures were involved, and they needed different fixes:
1. Sentence selection was lexical, so it refused rather than answered. Term
overlap between a Devanagari query and an English passage is exactly zero, so
the extractive answerer refused 98.9% of Hindi queries whose passage it had
retrieved correctly. ExtractiveAnswerer now falls back to embedding
similarity — reusing the multilingual encoder and the query vector retrieval
already computed — and only when the lexical path finds nothing, so same-script
queries keep their sub-millisecond answer. Answer rate went 0.49 → 0.99.
2. That fix does not solve the language mismatch, because nothing can. You cannot extract Hindi text from an English corpus. The fix is to index the passages in the language you serve — which this dataset provides:
python -m scripts.prepare_msmarco --lang hi --passage-lang indic --out data/msmarco_hiHindi F1 goes 0.0067 → 0.2890, a 43× improvement, with no model change and no key. The rule is simply: extractive mode requires the corpus to be in the user's language. Serve one index per language and route on the detected transcript language — Sarvam already returns it.
Generating a Hindi answer from English passages is the case where the generative mode earns its latency and its key. That is a real division of labour, not a fallback.
An extracted span must be contiguous. An earlier version joined the two highest-scoring sentences, which for non-adjacent sentences produces text appearing nowhere in the source — quietly breaking the guarantee the whole mode rests on, and capable of asserting something neither sentence said. Spans now grow only into an immediate neighbour, enforced by test.
This is the most important negative result, and it corrects an assumption made when the dataset was first inspected.
is_selected == 0 rows were treated as negatives for calibrating retrieval
abstention. They are not. Calibration on the real data fails outright:
top_score threshold=0.7141 coverage=67.5% false_answer=50.4% separation=+0.0397
no threshold achieves <= 0% false answers; distributions overlap
margin threshold=0.0306 coverage=25.2% false_answer=25.1% separation=-0.0096
mean top_score |
|
|---|---|
| answerable | 0.7435 |
| unanswerable | 0.7037 |
54% of unanswerable queries retrieve something scoring above 0.7. The mechanism is obvious in hindsight: a MS MARCO row's passages were retrieved for that query by MS MARCO's own system, so they are topically on-target even when none of them answers it. "chart for foods low in potassium" retrieves passages about potassium and food that simply contain no chart.
So is_selected == 0 means answer-absent, not retrieval-irrelevant, and
retrieval scores cannot separate the two — no threshold exists, and the
calibration tool says so rather than returning a plausible-looking number.
The distinction is real and load-bearing:
| failure | detectable from | works? |
|---|---|---|
| nothing relevant in the corpus | retrieval score | yes — fixture separates 0.83 vs 0.09 |
| relevant passages, none answers | reading the passage | no — needs the answerer |
The same holds one level down. Sentence-embedding similarity to the query
separates answerable from unanswerable Hindi queries by 0.0066 (0.8494 vs
0.8428), and every threshold trades coverage against false answers at roughly
1:1 — at 0.86, 32.8% coverage buys 22.0% false answers. So the semantic
selection floor in ExtractiveAnswerer is documented as a boilerplate filter,
not an abstention gate, and refusal accuracy on this corpus is honestly low
(0.02) rather than propped up by a threshold that does not separate anything.
Which is an argument for the two-stage guardrail design rather than against
it: retrieval abstention catches out-of-corpus queries, and answer-presence
detection is the generative path's supported flag or the extractive path's
sentence-overlap floor. Neither substitutes for the other. Treating retrieval
confidence as an answerability oracle — which is what the original plan implied
— would ship a system that confidently answers half the questions it cannot.
metric = mrr@10 baseline = fixed_256 / hybrid encoder = multilingual-e5-small n = 6444
configuration score 95% CI recall@5 P50 ms vs base sig
------------------------------------------------------------------------------------------
fixed_512 / dense only 0.5277 [0.518, 0.537] 0.744 16.830 +0.0779 yes
fixed_512 / hybrid norm 0.4699 [0.460, 0.479] 0.709 17.291 +0.0201 yes
parent_child / hybrid 0.4578 [0.448, 0.467] 0.678 17.282 +0.0081 yes
semantic / hybrid 0.4560 [0.447, 0.465] 0.694 17.241 +0.0063 yes
fixed_256 / hybrid 0.4497 [0.441, 0.459] 0.694 17.226 -- --
fixed_512 / hybrid 0.4497 [0.441, 0.459] 0.694 17.087 +0.0000 no
all reps, router off 0.4364 [0.427, 0.446] 0.684 19.725 -0.0133 yes
ADAPTIVE (router on) 0.4311 [0.422, 0.440] 0.680 20.056 -0.0187 yes
sentence / hybrid 0.4045 [0.395, 0.414] 0.641 20.183 -0.0452 yes
fixed_512 / bm25 only 0.2097 [0.202, 0.217] 0.342 15.401 -0.2401 yes
Dense-only wins by a wide, significant margin, and hybrid retrieval — a core premise of the plan — is actively harmful on this corpus. Also: the adaptive router loses significantly again, and it is faster to search one representation than five.
BM25 matches surface forms. Splitting each retriever by query language explains the whole result:
| retriever | English queries | Hindi queries |
|---|---|---|
| dense | 0.6602 | 0.3951 |
| BM25 | 0.4185 | 0.0008 |
BM25 scores 0.0008 on Hindi queries — statistical zero. Devanagari query tokens share essentially nothing with Latin-script English passages; it can match a stray numeral and nothing else. Half the query set was fusing pure noise into the ranking, which is exactly why hybrid trailed dense-only.
The router already picks the dense/sparse mix, so this is a natural extension —
when a query shares no script with the corpus, BM25 is switched off and skipped
entirely (saving its latency too). scripts_of() is three regex scans.
| configuration | MRR@10 | English | Hindi |
|---|---|---|---|
| hybrid, uniform weights | 0.4497 | 0.5675 | 0.3950 |
| hybrid, script-aware | 0.4813 | 0.5675 | 0.3950 |
Hindi now scores 0.3950 against dense-only's 0.3951 — identical, as it must be, since BM25 is off and the configuration is dense-only for those queries.
Script-aware routing does not rescue English, where BM25 works but is far weaker than dense (0.4185 vs 0.6602) and RRF was weighting them near-equally. Sweeping the dense weight:
| dense weight | MRR@10 | English |
|---|---|---|
| 0.50 | 0.4813 | 0.5675 |
| 0.80 | 0.5154 | 0.6357 |
| 0.92 | 0.5241 | 0.6532 |
| 1.00 (dense only) | 0.5277 | 0.6602 |
Monotonic to the boundary. BM25 contributes nothing positive on this corpus at any weight, and the optimum is to drop it.
That is a corpus-specific verdict, not a general one. BM25 earns its place on rare entities, part numbers, and exact-phrase lookups against same-script text — MS MARCO's web passages with machine-translated queries reward none of that. The value here is that the sweep answered the question in an afternoon instead of the hybrid stack shipping on the strength of the argument for it.
metric = mrr@10 baseline = fixed_512 / hybrid encoder = hashing
configuration score 95% CI recall@5 P50 ms vs base sig
------------------------------------------------------------------------------------------
parent_child / hybrid 0.6019 [0.428, 0.758] 0.683 0.211 +0.0256 no
all reps, router off 0.5983 [0.437, 0.750] 0.683 0.347 +0.0219 no
fixed_512 / hybrid 0.5764 [0.414, 0.725] 0.683 0.177 -- --
sentence / hybrid 0.5531 [0.405, 0.696] 0.667 0.295 -0.0233 no
fixed_512 / dense only 0.5331 [0.372, 0.685] 0.667 0.116 -0.0433 no
fixed_512 / bm25 only 0.5225 [0.354, 0.676] 0.650 0.100 -0.0538 no
ADAPTIVE (router on) 0.5167 [0.358, 0.664] 0.700 0.468 -0.0597 no
Identical on both dense backends; only the latency column moves. parent_child
leads, but nothing here separates: intervals are ±0.15 at n=30 and no paired
comparison is significant. That is the honest reading and the argument for the
real evaluation set.
Measured on the real corpus (49,539 passages, 339,344 chunks, multilingual-e5-small, 10,000 queries): retrieval quality with intervals, the cross-lingual gap, the query-type breakdown, per-stage latency, and the calibration failure. These are real numbers, not illustrations.
Measured on the fixture only: the guardrail behaviours, the state machine, backend determinism, and the confidence separation. The fixture is synthetic and 40 documents; those results demonstrate mechanism, not quality.
Corrected along the way. Two claims made earlier in this project were wrong and the measurements disproved them:
- "The adaptive router is the worst configuration, significantly so." That was
substantially an artifact of the zero-score padding bug in
dense.py. After the fix it sat mid-pack and nothing in the fixture sweep was significant. - "
is_selected == 0rows are real negatives for calibrating abstention." They are answer-absent, not retrieval-irrelevant, and retrieval scores cannot separate them. See the calibration finding.
Verified live: Groq, both the grounded and open-knowledge paths, including
disclaimer enforcement over repeated calls and rate-limit behaviour. Sarvam STT,
on real audio in both languages. English
transcribed exactly at 926 ms; Hindi exactly at 486 ms, including auto-detection
(language_code: unknown → hi-IN). The full voice path — Hindi speech in,
grounded Hindi answer with a citation out — runs end to end with no LLM key.
Written but not exercised against a live service: generative answering.
app/generation/claude.py targets claude-opus-5 with structured output,
effort low, and streaming for TTFT. No ANTHROPIC_API_KEY was available, so
its request shape is built from the documented API and has not been run. The
refusal, parse-failure and no-citation paths are unit-tested with fakes; the
HTTP call is not. Nothing depends on it — extractive mode is the default and
the whole system works without it.
ElevenLabs STT is likewise written against verified documentation and untested; Sarvam is the configured default.
# extractive + real speech, no LLM key:
RAG_STT=sarvam RAG_CONFIG=configs/msmarco_hi.yaml RAG_INDEXES=indexes_hi make serveKnown environment issue. .venv on this machine was created twice by
different interpreters (conda 3.12, then Homebrew 3.13), so pip installs into
lib/python3.13/ while .venv/bin/python reads lib/python3.12/. make venv-check now detects this and install uses python -m pip so it cannot
recur. Fix an affected venv with rm -rf .venv && make install.
configs/
default.yaml fixture corpus
msmarco.yaml MSMARCO-XI, English passages (cross-lingual setup)
msmarco_hi.yaml MSMARCO-XI, Hindi passages (same-language setup)
Dockerfile CPU-only serving image; indexes mounted, not baked
docker-compose.yml extractive mode, no key required
ingestion/
corpus.py fixture | jsonl | huggingface
clean.py NFKC, control strip, Indic-aware sentence split
chunkers/ sentence, fixed, parent_child, semantic
build_index.py offline build, all representations
app/
retrieval/
tokenize.py shared by BM25 and the hashing encoder
encoders.py hashing fallback + sentence-transformers (mps/cuda/cpu)
bm25.py numpy inverted index, per-query score ceiling
dense.py faiss or exact numpy, byte-identical rankings
fusion.py RRF / weighted, plus calibratable confidence
router.py 7 µs heuristic intent classifier
hybrid.py the online path, per-stage timed
generation/
extractive.py default: verbatim span, no model in the hot path
groq.py grounded generation + open-knowledge fallback
claude.py claude-opus-5, structured output, streaming
grounding.py deterministic verification, no second model call
stt/ sarvam | elevenlabs | mock, behind one interface
guardrails.py three checkpoints
orchestrator.py the state machine
api.py FastAPI: /ask /voice /health /
frontend/index.html mic capture, per-stage latency bars, citations
evaluation/
metrics.py MRR, Recall, NDCG, bootstrap, paired bootstrap
harness.py shared loading and scoring
answer_eval.py end-to-end answer quality + refusal accuracy
retrieval_eval.py quality, cross-lingual gap, query-type breakdown
latency_eval.py per-stage percentiles
calibrate.py abstention threshold fitting
benchmark.py the comparison sweep
scripts/
make_fixtures.py synthetic corpus
prepare_msmarco.py MSMARCO-XI -> corpus/queries/qrels
tests/ 123 tests
- A stronger multilingual encoder. The 0.2265 cross-lingual gap is the
largest single quality loss in the system and dwarfs every retrieval tweak
available.
multilingual-e5-largeor BGE-m3, re-measured the same way. - Cut query-embedding latency. It is 74% of the retrieval path at corpus scale. Cache repeated queries, quantise, or distil — but measure first, because at 20 ms P50 the budget is not currently under threat.
- Per-language indexes with language routing. Extraction requires the corpus to be in the user's language, and the dataset ships all 14. Build one index per served language and route on the transcript language Sarvam already returns. This is the single highest-value change for an Indic voice product and needs no model work.
- Answer-presence detection, since neither retrieval confidence nor
sentence similarity can do it on this corpus (separations of 0.04 and 0.007).
The generative path's
supportedflag is the cheap version; a small extractive-QA or NLI model is the accurate one, and its latency cost decides. - Streaming STT. It is 95% of voice latency. Nothing else in the pipeline is worth optimising until this is.
- Drop BM25 and the representation router on this corpus — already done in
configs/msmarco.yaml, keeping the script check. Both lose significantly; the sweep is unambiguous. Keep the hybrid code — it is a config flag, and a different corpus will want it. - Reranking, measured against its latency cost, and adopted only if the trade is favourable on CPU.
- Live-test the generative path once an Anthropic key is available.