Make document titles searchable in hybrid retrieval#549
Closed
tobocop2 wants to merge 4 commits into
Closed
Conversation
The lexical index covered only chunk text, so BM25 could never match a document's title or filename. Every chunk now carries a nullable title (extraction metadata when available, cleaned filename stem otherwise) with its own FTS index, queried as a third RRF arm at a configurable weight when title_search is on (off by default pending eval evidence). Existing chunk-FTS call sites pin fts_columns explicitly: once a second FTS index exists, an unpinned query searches every indexed column, which would silently widen BM25 semantics. Old indexes keep working: schemas evolve in place and the title arm contributes nothing until reindex. Source rows also persist extraction title/authors/created_at for future filtering. bb-t8pr
lilbee passes raw user query text to the lexical arm, so a query LanceDB parses as a phrase (e.g. a quoted span) reaches it as a phrase query. The chunk and title FTS indexes were created without positions, so those queries raised and the arm silently returned nothing. Create both indexes with_position=True. Existing indexes need a reindex to gain phrase support; they degrade to vector-only until then rather than erroring. bb-e4qw (cherry picked from commit a2cd684)
Rank fusion has weighed the vector and BM25 arms equally, which dilutes a strong dense embedder on corpora where the lexical arm adds more noise than signal. Add lexical_fusion_weight (default 1.0, so the equal-weight behaviour is byte-for-byte unchanged) to scale the BM25 arm's share of the fused score down toward the dense arm. The right value is corpus-dependent and meant to be set from the retrieval benchmark, not guessed. Wired through the config field, its HTTP/MCP/env surfaces, the TUI settings map, and fuse_arms.
ensure_fts_index() runs table.optimize() whenever the FTS index already exists. On a large corpus that call can hit a LanceDB encoding bug and raise (Arrow list offset overflow), and the failure fell through to leave _fts_ready False -- silently dropping every subsequent query to vector-only with no error surfaced. The index itself is complete and still serves queries, so mark hybrid ready before the best-effort optimize and downgrade an optimize() failure to a warning. Found while benchmarking retrieval: every hybrid config collapsed to the vector-only baseline until this was fixed. (cherry picked from commit 6aedffb)
This was referenced Jul 18, 2026
Owner
Author
|
Superseded by the consolidated retrieval PR #557 — review and merge there. Converted to draft to keep it off the review queue; the branch and history stay intact for reference. |
tobocop2
marked this pull request as draft
July 18, 2026 04:55
Owner
Author
|
Superseded by the consolidated retrieval PR #557, which contains this change. Closing; the branch stays intact for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The lexical index covers only chunk text, so BM25 can never match a document's title or filename; a query naming a document finds it only if the words happen to appear in the body.
Three adjacent defects in the same hybrid-search surface come along for the ride, all of which silently degrade retrieval. The FTS indexes were built without token positions, so a phrase query (raw user text reaches LanceDB as a phrase) errored instead of matching. Rank fusion weighed the vector and BM25 arms equally with no way to favor a strong dense embedder. And
ensure_fts_index()runstable.optimize()on an existing index, which on a real-sized corpus can hit a LanceDB encoding bug and raise, leaving the index flag unset so every subsequent query silently fell back to vector-only with no error surfaced.Solution
Every chunk carries a nullable title (extraction metadata when available, cleaned filename stem otherwise) with its own FTS index, fused as a third reciprocal-rank arm at a configurable weight when
title_searchis on (off by default until the eval harness proves it). Existing chunk-FTS call sites now pin their column explicitly, because once a second FTS index exists an unpinned query silently searches every indexed column. Old indexes keep working untouched: schemas evolve in place and the title arm contributes nothing until a reindex populates titles. Source rows also persist extraction title, authors, and creation date.Both FTS indexes are built with token positions so phrase queries match instead of erroring.
lexical_fusion_weight(default 1.0, so equal-weight fusion is byte-for-byte unchanged) scales the BM25 arm down toward the dense arm. And an existing, queryable index is marked hybrid-ready before the best-effortoptimize(), so anoptimize()failure logs a warning instead of disabling hybrid search corpus-wide. A retrieval benchmark on BEIR confirmed the optimize bug: every hybrid config scored identically to the vector-only baseline until it was fixed.