Repository navigation
[#336] Hybrid lexical/semantic search (PoC) - #338
Youssef-joe wants to merge 10 commits into
Conversation
String::truncate panics when MAX_CHUNK_CHARS falls inside a multi-byte character, which a real embedding run hit on files with non-ASCII comments or string literals. Cut back to the nearest char boundary instead.
…ch indexes Walk the workspace and compute per-file SHA-256 once per search in a shared source module. The embedding index consumes that scan instead of performing its own walk and hash pass; the lexical sidecar will use the same scan.
Index the full extracted Tree-sitter symbol range into a separately versioned lexical.json sidecar next to the embedding cache. Persist postings and symbol metadata only, keep the embedding truncation unchanged, and update incrementally from the shared workspace scan. A file hash is recorded only after that file parses successfully.
Rank the lexical sidecar with a code-aware tokenizer and BM25, expand graph neighbours, and apply the existing token budget. With no embeddings-capable server, /search and semantic_search now return ranked lexical results instead of an error.
…gainst source ranges Intersect posting lists to narrow candidates, group them by file, read each file once, and verify phrase order/adjacency or identifier token boundaries only inside the symbol's line range. Verified matches are marked exact and ranked first. Verification is a reranking step, not a recall guarantee.
Combine the semantic ranking (existing sketch filter and diversity penalty) with the BM25 ranking using plain Reciprocal Rank Fusion (k = 60) before graph expansion and the token budget. Verified exact matches are promoted ahead of fused results; equal fused scores prefer the semantic candidate.
Keep the lexical index independent of the embeddings server: when the embedding build or query embedding fails after startup, fall back to BM25 results instead of returning an error. When lexical retrieval is unavailable but embeddings work, keep semantic-only behavior. Add a regression test for the server disappearing after startup.
fa37b1c to
d01b21d
Compare
The diversity penalty compared every chunk with every seed selected before it, so one query cost O(chunks^2 * dimensions). On this workspace (~21k chunks at 768 dimensions) that is roughly six minutes per query, which makes the benchmark — and `/search` on any repository this size — unusable. Only the head of the ranking can reach the result set: `search` reads at most `top_k * 2` entries, and RRF fuses the head of the semantic ranking. Restrict the pass to the top `DIVERSITY_CANDIDATE_LIMIT` (256) candidates. Query time drops to milliseconds and ordering is unchanged for everything that can actually be returned. Found by the Phase 7 benchmark once it ran against a full-size index.
What this branch changes, measuredAll numbers below are from this repository, on the rebased branch (22,492 extracted symbols, 18 scored fixture queries; 3 standalone-constant queries excluded as extraction coverage gaps). 1. Search no longer depends on an embeddings server
2. Index cost — the reason the fallback matters
A user with no GPU, or a fresh clone, gets usable search in seconds instead of hours. 3. Retrieval quality — lexical-only
These held constant across the rebase (+806 symbols) and across tightening the fixture so an ambiguous name can no longer satisfy a query, so they are not an artifact of easy expectations. 4. Two bugs this work exposedQuadratic diversity pass (fixed in Duplicate ids in warm lexical rebuilds (fixed in Phase 2). Term frequencies were reconstructed keyed by document id, but ids are not unique here:
So a warm rebuild silently changed BM25 weights for 6.4% of the index. Now keyed by document position, with a regression test. 5. Query formatting — observed, not changed here
Incremental rebuild, observed liveThe rebase onto PendingSemantic-only and hybrid MRR / Recall@5 / mean latency, on the same fixture. The index is at ~16,000 / 22,492 symbols and completing now; I will post the fused comparison and the decision-gate outcome as a follow-up rather than claim it early. Note that plain RRF can regress conceptual queries when a document appears in both lists — the early (pre-fix) run showed one case where semantic scored 1.000 and hybrid 0.000 for |
d01b21d to
37e3157
Compare
Follow-up: semantic and hybrid resultsThe full semantic index finished, so the comparison promised above is complete. The
Gate outcome: not passed. Hybrid beats semantic-only overall. It loses to semantic
Question for #336: should I try the refinement for (1)? Among verified exact hits, a Separately, the runtime does not send embeddinggemma's retrieval prompt |
Add a committed fixture and binary that compares lexical-only, semantic-only, and hybrid ranking, reporting MRR, Recall@5, latency, indexing time, and sidecar size. Standalone constants that the current Rust extractor does not emit are tracked as coverage gaps. Expectations name unique ids where a symbol name is ambiguous. --verbose prints the lexical, semantic, and hybrid hits. Record the results in the README.
Hybrid search placed verified exact hits first but then ordered them by their RRF score. RRF is rank-only: lexical mnemosyne-systems#1 and mnemosyne-systems#2 differ by ~0.0003, while appearing anywhere in the semantic list adds >= ~0.003, so a test that calls an identifier outranked its definition. Exact hits now keep the order the lexical ranking gave them (BM25), and RRF orders only the remaining candidates, as agreed in the design. A verified hit also keeps its exact flag when a higher-scoring graph-expanded copy replaces it. Update the benchmark results.
37e3157 to
1d57789
Compare
Exact/phrase regression: cause, fix, and updated numbersThe issueIn the previous run, hybrid was worse than lexical-only on exact identifiers (MRR 0.857 Both symbols verify as exact, because the identifier appears inside both line ranges. b.exact.cmp(&a.exact).then_with(|| b.score.total_cmp(&a.score)) // score = RRFRRF is rank-only. Lexical #1 and #2 differ by The fix (
|
| metric | semantic-only | lexical-only | hybrid |
|---|---|---|---|
| overall MRR | 0.294 | 0.541 | 0.567 |
| overall Recall@5 | 0.444 | 0.667 | 0.667 |
| conceptual MRR (n=10) | 0.278 | 0.173 | 0.220 |
| conceptual Recall@5 (n=10) | 0.500 | 0.400 | 0.400 |
| exact MRR (n=7) | 0.357 | 1.000 | 1.000 |
| phrase rank | miss | 1 | 1 |
| mean latency | 85 ms | 6 ms | 101 ms |
- Hybrid now has the best overall MRR, and matches lexical-only on every exact and phrase
query. - Conceptual queries are unchanged by the fix. Hybrid is still below semantic-only on
MRR and Recall@5. - Lexical-only moved slightly (0.536 → 0.541) only because the corpus changed (this
branch's new code is part of the indexed repo). - Earlier latencies (230 / 276 ms) were measured under concurrent CPU load. These are
from an otherwise idle machine.
Correction: my previous comment and the description reported hybrid conceptual
Recall@5 as 0.500. The run's own log shows 0.400 (4 of 10). It was a transcription
error and is fixed in the description and the bench README.
Where this leaves the gate
Exact and phrase queries now pass within the agreed design. The conceptual gap remains,
and it is inherent to plain RRF rewarding presence in both lists: in "apply the semantic token budget", semantic has budget_cut at rank 1 and hybrid at rank 2.
Closing it would need weighted RRF or query-type routing, both outside what was agreed,
so I'll bring that question back to #336.
Summary
PoC for #336: add an
independently versioned lexical BM25 sidecar and fuse it with the existing semantic
ranking using plain Reciprocal Rank Fusion (
k = 60), before the existing graphexpansion and token-budget cut.
This follows the Updated Design agreed in the discussion. It does not change the
symbol extraction strategy; standalone constants such as
MAX_CHUNK_CHARS,EMBED_BATCH_TOKEN_BUDGETandRRF_Kare tracked as extraction coverage gaps andexcluded from the benchmark metrics.
Agreed design
lexical.json), next tochunks.json.only after that file parses successfully.
truncation (1,600 chars) is unchanged.
::, paths).each candidate file is read at most once per query and only the symbol's line range is
checked. Verification is reranking/validation, not a recall guarantee.
k = 60) fuses the remaining lexical and semantic candidates./searchandsemantic_searchbehave identically on the same workspace.Phases
One commit per phase; each is independently reviewable.
a71aac8avoid UTF-8 truncation panic in embedding chunksa9213acshare workspace scan and hashing between search indexes3de2b94add versioned lexical postings sidecar3e7fd72add BM25 lexical-only retrievalcb67e00verify quoted phrases and exact identifiers against source ranges47674aafuse lexical and semantic rankings with RRFb22f61aharden lexical and semantic fallback paths3bc9dacbound the diversity pass to the result headef17c2dadd hybrid search benchmark1d57789keep lexical order among verified exact hitsBenchmark
Fixture:
src/bin/orangu-search-bench/queries.json, evaluated on this repository at1d57789(22,493 extracted symbols). Expectations name unique ids where a symbol nameis ambiguous, so an unrelated same-named symbol cannot satisfy a query. The phrase query
is
"lexical search index could not be built"— chosen because it only exists inproduction code, not in test code. 18 scored queries; 3 standalone-constant queries are
excluded as extraction coverage gaps.
Semantic and hybrid latency include one embedding request for the query; the ranking
itself is millisecond-scale after
3bc9dac. (Earlier latency figures of 230 / 276 mswere measured under concurrent CPU load.)
Correction: an earlier version of this description reported hybrid conceptual
Recall@5 as 0.500. The run's own log shows 0.400 (4 of 10); it was a transcription error.
What the numbers say
best Recall@5.
query at rank 1) and is strictly better than semantic-only.
lexical 0.173) and is below semantic on Recall@5 (0.400 vs 0.500).
The exact/phrase regression in the first run (hybrid 0.857, phrase rank 2) was an
implementation bug, fixed in
1d57789: verified exact hits were placed first but thenordered among themselves by RRF score. RRF is rank-only, so lexical #1 and #2 differ by
~0.0003, while appearing anywhere in the semantic list adds ≥ ~0.003. For
probe_endpoint, a test that calls it was in the semantic list and outranked thedefinition. Exact hits now keep the lexical (BM25) order, and RRF orders only the
remaining candidates — which is what the agreed design says.
The conceptual gap is inherent to plain RRF, which rewards presence in both lists: e.g.
"apply the semantic token budget"— semantic returnsbudget_cutat rank 1, hybrid atrank 2, because another symbol sits high in lexical and low in semantic.
Decision gate
agreed tolerance).
rather than adding undocumented semantic/lexical weights.
Decision gate — outcome
vs 0.500.
lexical-only).
lexical-only).
Exact and phrase queries now pass within the agreed design. Conceptual queries do not,
and fixing them would need weighted RRF or query-type routing, both outside the agreed
design. Following the gate, that question goes back to #336.
Implementation notes
lexical ranking gave them; RRF orders only the remaining candidates. A verified hit
keeps its exact flag if a higher-scoring graph-expanded copy of the same symbol
replaces it.
document id. This is a deterministic tie-break, not a weight.
penalty cannot be applied after fusion without changing the design; this is the
interpretation of "preserve the existing diversity penalty".
most once, and only the symbol's line range is inspected. An explicit identifier is
matched on token boundaries, so
RRF_Kdoes not verify againstRRF_K_EXTRA.gated on the embeddings server, so a server that disappears after startup degrades to
BM25 instead of failing the tool.
Testing
cargo fmt --check— clean.cargo clippy --all-targets— clean (at1d57789).cargo test --lib(at1d57789) — 903 passed, 24 ignored.cargo test(at the previous head) — 2,303 passed, 149 ignored, 3 failed, all inorangu-server, which this branch does not touch:suggest::tests::format_suggestion_has_no_gpu_section_without_a_gpu— pre-existing,detects the host GPU.
workers::node::tests::a_hybrid_tree_hands_its_states_backanda_large_frame_crosses_tls— timing-sensitive; failed under concurrent CPU load andpass when re-run on their own.
incremental add/change/remove, full symbol range, tokenizer, BM25 ordering,
lexical-only retrieval, phrase adjacency, identifier token boundaries, exact-match
promotion, lexical order among exact hits, RRF tie-break, the bounded diversity pass, and both fallback paths.
Out of scope