Retrieval quality: title-aware hybrid, adaptive fusion, context expansion, and structural filtering#557
Open
tobocop2 wants to merge 12 commits into
Open
Retrieval quality: title-aware hybrid, adaptive fusion, context expansion, and structural filtering#557tobocop2 wants to merge 12 commits into
tobocop2 wants to merge 12 commits into
Conversation
A hit that lands mid-argument loses the text before and after it. With neighbor_expansion set, each selected chunk pulls up to N adjacent chunks per side from its own source and merges them into one contiguous passage, deduplicating the chunker overlap. Expansion runs after the budget fit and spends only leftover tokens, so originals are never dropped for neighbors; citation numbering is untouched and page/line spans are recomputed truthfully. Off by default (neighbor_expansion=0) pending eval evidence. bb-m3mo
The lexical index covered only chunk text, so BM25 could never match a document's title or filename. Every chunk now carries a nullable title (extraction metadata when available, cleaned filename stem otherwise) with its own FTS index, queried as a third RRF arm at a configurable weight when title_search is on (off by default pending eval evidence). Existing chunk-FTS call sites pin fts_columns explicitly: once a second FTS index exists, an unpinned query searches every indexed column, which would silently widen BM25 semantics. Old indexes keep working: schemas evolve in place and the title arm contributes nothing until reindex. Source rows also persist extraction title/authors/created_at for future filtering. bb-t8pr
Setting the data root through the LILBEE_DATA_ROOT environment variable delivered it to the config resolver as a raw string, and deriving the documents/data/lancedb child directories with the / operator then raised TypeError: unsupported operand type(s) for /: 'str' and 'str'. Coerce the value to Path up front so the env var works the same as every other way of setting the data root.
lilbee passes raw user query text to the lexical arm, so a query LanceDB parses as a phrase (e.g. a quoted span) reaches it as a phrase query. The chunk and title FTS indexes were created without positions, so those queries raised and the arm silently returned nothing. Create both indexes with_position=True. Existing indexes need a reindex to gain phrase support; they degrade to vector-only until then rather than erroring. bb-e4qw (cherry picked from commit a2cd684)
Rank fusion has weighed the vector and BM25 arms equally, which dilutes a strong dense embedder on corpora where the lexical arm adds more noise than signal. Add lexical_fusion_weight (default 1.0, so the equal-weight behaviour is byte-for-byte unchanged) to scale the BM25 arm's share of the fused score down toward the dense arm. The right value is corpus-dependent and meant to be set from the retrieval benchmark, not guessed. Wired through the config field, its HTTP/MCP/env surfaces, the TUI settings map, and fuse_arms.
ensure_fts_index() runs table.optimize() whenever the FTS index already exists. On a large corpus that call can hit a LanceDB encoding bug and raise (Arrow list offset overflow), and the failure fell through to leave _fts_ready False -- silently dropping every subsequent query to vector-only with no error surfaced. The index itself is complete and still serves queries, so mark hybrid ready before the best-effort optimize and downgrade an optimize() failure to a warning. Found while benchmarking retrieval: every hybrid config collapsed to the vector-only baseline until this was fixed. (cherry picked from commit 6aedffb)
The retrieval benchmark showed no single lexical_fusion_weight wins every corpus: SciFact wants a strong BM25 arm, NFCorpus a weak one, FiQA none. Rather than fix one weight, adaptive_fusion (off by default) scales the BM25 arm per query by how peaked the vector ranking is -- a top hit standing clear of the field downweights lexical toward zero, a flat ranking keeps it. adaptive_fusion_ margin sets the vector-similarity margin at which the lexical arm is fully silenced. lexical_fusion_weight becomes the ceiling the rule scales down from, so the default path is unchanged. (cherry picked from commit a0c2a96)
The retrieval benchmark validated adaptive per-query fusion against the fixed lexical weight on BEIR SciFact, NFCorpus, and FiQA: at margin 0.15 it beats the fixed w=1.0 default on all three, keeps the significant NFCorpus and SciFact wins, and cuts the FiQA regression from -0.056 to -0.018 (the biggest gain on the one corpus a fixed lexical arm hurt most). Default adaptive_fusion=true, adaptive_fusion_margin=0.15; set adaptive_fusion=false to pin the fixed weight.
This was referenced Jul 18, 2026
…lean-consolidated
…nto tmp-clean-consolidated
…-clean-consolidated
The UFO A/B (bb-5bjp) traced a real context-precision drop to document-structure chunks entering retrieval: the title arm matches cover/title pages (they carry the doc title) and table extraction indexes tables-of-contents, and neither ever answers a question. A conservative detector drops unambiguous TOCs (dot-leaders to page numbers) and cover pages (classification banner + short + shouting-case) from the search results, on the top_k*2 candidate buffer so real passages remain. On by default; filter_structural_chunks=false keeps them. Faithfulness was unaffected in the A/B (the relevant passages were already retrieved), so this recovers precision without touching answer quality. (cherry picked from commit 8ecbbf8)
tobocop2
force-pushed
the
feat/retrieval-consolidated
branch
from
July 18, 2026 06:55
8ecbbf8 to
495c4da
Compare
tobocop2
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This rolls up the retrieval-quality work into one branch so it can be reviewed and merged as a unit. Each section is a self-contained change that also has its own PR for detailed history.
Off
main,make checkgreen (100% coverage) on the combined tree. The eval/benchmark harness tooling is intentionally NOT here — it lives on its own branch (evals/retrieval-harness) so this PR stays focused on lilbee's core.Title-aware hybrid retrieval, plus three FTS fixes (#549)
Document titles become a searchable lexical arm: every chunk carries a nullable title (extraction metadata or cleaned filename) with its own FTS index, fused as a third reciprocal-rank arm at a configurable weight. Three defects in the same hybrid-search surface are fixed alongside it:
lexical_fusion_weightscales the BM25 arm relative to the vector arm.optimize(), so anoptimize()failure on a large corpus (a LanceDB encoding bug) logs a warning instead of silently disabling hybrid search and dropping every query to vector-only.Context-window expansion (#550)
After retrieval, each selected chunk pulls a few adjacent same-source chunks and merges them into one contiguous passage (deduplicating the chunker's overlap), so a hit landing mid-argument keeps the sentences around it. Spends only the token budget left after the originals are fitted, and never widens table chunks.
Adaptive per-query fusion, on by default (#556)
The benchmark showed no single fixed fusion weight wins every corpus. Adaptive fusion scales the BM25 arm per query by how confident the vector arm is: a peaked dense ranking downweights the lexical arm toward zero, a flat one keeps it. On BEIR it beat the fixed equal-weight default on all three datasets, kept the significant NFCorpus and SciFact wins, and cut FiQA's regression from -0.056 to -0.018 nDCG@10. Ships on by default (
adaptive_fusion=true,adaptive_fusion_margin=0.15); setadaptive_fusion=falseto pin the fixed weight.Structural-chunk filter (bb-pkn6)
A cross-corpus A/B traced a real context-precision drop to document-structure chunks entering retrieval: the title arm matches cover/title pages (they carry the doc title) and table extraction indexes tables-of-contents, and neither ever answers a question. A conservative detector drops unambiguous TOCs and cover pages from the search results (on the
top_k*2buffer so real passages remain). On by default (filter_structural_chunks); faithfulness was unaffected in the A/B, so it recovers precision without touching answer quality.Config crash fix (#554)
Setting the data root via
LILBEE_DATA_ROOTdelivered it as a string and crashed path resolution (str / "documents"). The value is coerced toPathup front so the env var works like every other way of setting the root.Not in this PR: the eval + benchmark harness
The blind retrieval eval harness and the preregistered benchmark (which validated all of the above) live on branch
evals/retrieval-harness— tooling that runs against a deployed lilbee, kept out of the core so it doesn't cluttersrc/.