Retrieval quality: title-aware hybrid, adaptive fusion, context expansion, and structural filtering

docs/architecture.md

@@ -641,7 +641,7 @@ flowchart TD
     SM -->|No prefix| PROBE[BM25 Confidence Probe]
 
     PROBE --> CONF{Confident AND separated?}
-    CONF -->|Yes| DUAL[Dual-Arm Retrieval]
+    CONF -->|Yes| DUAL[Hybrid Retrieval]
     CONF -->|No| EXPAND[LLM Query Expansion]
 
     EXPAND --> GEXP[+ Graph Expansion]
@@ -677,19 +677,20 @@ flowchart TD
 #### History Condensation
 **On by default** (`LILBEE_HISTORY_REWRITE`). A follow-up question is condensed into a standalone retrieval query using the recent chat history (one small LLM call, skipped when there is no history). Retrieval sees only the query text: without this, "what about his brother?" is embedded and BM25-matched with its pronouns. The user's original wording still reaches the answering prompt.
 
-#### Hybrid Search (Dual-Arm Reciprocal-Rank Fusion)
-**Always on.** Retrieves the BM25 arm (LanceDB FTS) and the vector arm independently, each fetched exactly `top_k` deep, then fuses by reciprocal rank into one canonical [0, 1] score:
+#### Hybrid Search (Weighted Reciprocal-Rank Fusion)
+**Always on.** Retrieves a vector arm and a BM25 chunk arm (LanceDB FTS) independently, each fetched exactly `top_k` deep, then fuses by weighted reciprocal rank into one [0, 1] score. An optional third BM25 arm over document titles joins when `LILBEE_TITLE_SEARCH` is on:
 
 ```
-score = (rank_weight(vector_rank) + rank_weight(bm25_rank)) / 2,  rank_weight(r) = 61 / (60 + r)
+score = Σ_arm weight_arm × rank_weight(rank_arm) / Σ_arm weight_arm,  rank_weight(r) = 61 / (60 + r)
 ```
 
-A row ranked first by both arms scores 1.0; a row one arm never retrieved contributes 0 for that arm, so an arm's top hit still scores 0.5 and stays visible next to rows deep in the other arm. The fused ordering is final: no diversity selection runs on the hybrid path.
+The vector arm weighs 1.0, the lexical arm `LILBEE_LEXICAL_FUSION_WEIGHT`, the title arm `LILBEE_TITLE_SEARCH_WEIGHT`. A row ranked first by every arm scores 1.0; a row only one arm retrieved scores that arm's share of the total weight, so a peaked single-arm hit stays visible next to rows deep in the other arms. The fused ordering is final: no diversity selection runs on the hybrid path.
 
-- **Why rank fusion and not score fusion**: a convex combination of normalized raw scores (`alpha × vector_similarity + (1 − alpha) × normalized_bm25`) was tried here and regressed graded precision about 20% against RRF, at every blend weight. Cosine similarities sit in a high narrow band, giving every dense neighbor a score floor that crowds out lexically-certain rows; ranks are scale-free, so neither arm's score distribution can drown the other. (On the rank-vs-score question, see Bruch et al. 2024, "[An Analysis of Fusion Functions for Hybrid Retrieval](https://dl.acm.org/doi/10.1145/3596512)".) Arm depth matters as much as the formula: a row one arm is certain about scores a fixed 0.5, while rows both arms rank mid-pool accumulate two contributions, so deep candidate pools flood the fused top-k with both-arm mediocrity; arms therefore stay `top_k` deep.
+- **Adaptive fusion** (`LILBEE_ADAPTIVE_FUSION`, on by default): the lexical arm's weight is scaled per query by how peaked the vector ranking is, so a confident dense arm downweights lexical and a flat one keeps it. `LILBEE_LEXICAL_FUSION_WEIGHT` is the ceiling the rule scales down from. Scores still normalize against that configured budget (a constant), not the per-query adapted weight, so scores from the separate sub-searches expansion merges stay on one comparable scale.
+- **Why rank fusion and not score fusion**: a convex combination of normalized raw scores (`alpha × vector_similarity + (1 − alpha) × normalized_bm25`) was tried here and regressed graded precision about 20% against RRF, at every blend weight. Cosine similarities sit in a high narrow band, giving every dense neighbor a score floor that crowds out lexically-certain rows; ranks are scale-free, so neither arm's score distribution can drown the other. (On the rank-vs-score question, see Bruch et al. 2024, "[An Analysis of Fusion Functions for Hybrid Retrieval](https://dl.acm.org/doi/10.1145/3596512)".) Arm depth matters as much as the formula: rows both arms rank mid-pool accumulate two contributions, so deep candidate pools flood the fused top-k with both-arm mediocrity; arms therefore stay `top_k` deep.
 - **Why no MMR on this path**: for lexical queries the relevant passages are often mutually similar (they quote the same identifiers), which is exactly what diversity selection penalizes; running MMR over the fused pool measurably traded relevant lexical hits for diverse off-topic neighbors. MMR still runs on the vector-only fallback path.
 - **Canonical score**: every search path sets `SearchChunk.score`, and every downstream stage (sorting, filtering, greedy set cover, concept boost, reranker blending) compares only that field. `distance`, `bm25_score`, and the legacy `relevance_score` remain as provenance.
-- **Abstention**: the canonical score is [0, 1] with fixed meaning (0.5 = top of one arm), so `min_relevance_score` is a usable floor: when every retrieved chunk falls below it, ask refuses instead of feeding noise as context. Raw RRF sums (~0.016-0.033 total range) made any threshold meaningless.
+- **Abstention**: `min_relevance_score` gates on the fused [0, 1] score. The score normalizes against the configured weight budget (a constant), so an arm's top hit scores a stable share of it and the threshold is a usable floor: when every retrieved chunk falls below it, ask refuses instead of feeding noise as context. Raw RRF sums (~0.016-0.033 total range) made any threshold meaningless.
 - **max_distance** applies to rows whose *only* signal is a far vector match; a row the BM25 arm also matched keeps its standing regardless of distance, since dropping it would re-bury exactly the identifier hits fusion exists to preserve.
 - **When it helps**: queries with specific terms, function names, error messages, exact phrases, document identifiers.
 

docs/usage.md

@@ -814,10 +814,17 @@ something feels off.
 | `LILBEE_RERANK_BLEND` | `true` | Blend reranker scores with retrieval fusion; off = the reranker's own ordering stands |
 | `LILBEE_HYDE` | `false` | Enable Hypothetical Document Embeddings: an LLM drafts a hypothetical answer, that's embedded, and results are merged with the original query's. Adds ~500 ms per query; helps on vague questions |
 | `LILBEE_HYDE_WEIGHT` | `0.7` | How much to trust HyDE results relative to the direct query (0.0-1.0) |
+| `LILBEE_TITLE_SEARCH` | `false` | Match queries against document titles as a third hybrid-search arm, so a query naming a document by title surfaces its chunks. Documents indexed before this feature existed have no stored title; run `lilbee rebuild` to make them title-searchable |
+| `LILBEE_TITLE_SEARCH_WEIGHT` | `0.5` | Title arm weight in rank fusion (1.0 = equal voice with the vector and text arms) |
+| `LILBEE_LEXICAL_FUSION_WEIGHT` | `1.0` | BM25 arm weight in rank fusion (1.0 = equal to the vector arm; lower it when a strong embedder should dominate a corpus where the lexical arm adds noise) |
+| `LILBEE_ADAPTIVE_FUSION` | `true` | Scale the BM25 arm's weight per query by how confident the vector arm is (a peaked dense ranking downweights lexical, a flat one keeps it) instead of using a fixed `LILBEE_LEXICAL_FUSION_WEIGHT`. That weight becomes the ceiling the adaptive rule scales down from. Set to `false` to pin the fixed weight |
+| `LILBEE_ADAPTIVE_FUSION_MARGIN` | `0.15` | Vector-similarity margin (top hit minus the field) at which adaptive fusion fully silences the BM25 arm; smaller values downweight lexical more aggressively. `0` disables adaptation, leaving the lexical arm at its full fixed weight |
+| `LILBEE_FILTER_STRUCTURAL_CHUNKS` | `false` | Drop tables-of-contents and classification-banner cover/title pages from search results. Off by default: an evaluation A/B found the filter net-negative, since its cover-page heuristic (which fires only on classified-document banners) also caught short banner-carrying body pages. When on, a query-matched or top-ranked page is never dropped. Re-validate per corpus before turning it on |
 | `LILBEE_ADAPTIVE_THRESHOLD` | `false` | Widen the distance threshold step by step when too few results pass, on the vector-only fallback path (no effect on the default hybrid search) |
 | `LILBEE_ADAPTIVE_THRESHOLD_STEP` | `0.2` | How much to widen per step when adaptive threshold triggers |
 | `LILBEE_TEMPORAL_FILTERING` | `true` | When the query contains temporal cues ("recent", "last week"), filter results by document date and sort by recency |
 | `LILBEE_MAX_CONTEXT_SOURCES` | `8` | Max chunks included in the LLM's RAG context. Raise for more coverage, lower for shorter prompts |
+| `LILBEE_NEIGHBOR_EXPANSION` | `0` | Adjacent chunks pulled in on each side of every retrieved passage and merged into one contiguous excerpt, so a hit that lands mid-argument regains its surrounding text. `0` disables |
 | `LILBEE_EXPANSION_SHORT_QUERY_TOKENS` | `2` | Queries this short (in tokens) skip LLM query expansion |
 | `LILBEE_ANN_INDEX_THRESHOLD` | `50000` | Chunk count above which sync builds the ANN vector index |
 | `LILBEE_MAX_REASONING_CHARS` | `64000` | Cap on a reasoning model's thinking output per answer |

src/lilbee/app/settings_map.py

@@ -857,6 +857,42 @@ def get_default(key: str) -> object:
         group=SettingGroup.RETRIEVAL,
         help_text="Candidate-pool multiplier over top_k before reranking",
     ),
+    "title_search": SettingDef(
+        bool,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="Match queries against document titles as a third hybrid-search arm",
+    ),
+    "title_search_weight": SettingDef(
+        float,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="Title arm weight in rank fusion (1.0 = equal voice with the other arms)",
+    ),
+    "lexical_fusion_weight": SettingDef(
+        float,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="BM25 arm weight in fusion (1.0 = equal to vector; lower to favor dense)",
+    ),
+    "adaptive_fusion": SettingDef(
+        bool,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="Scale the BM25 weight per query by vector-arm confidence, not a fixed value",
+    ),
+    "adaptive_fusion_margin": SettingDef(
+        float,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="Vector-similarity margin at which adaptive fusion fully silences the BM25 arm",
+    ),
+    "filter_structural_chunks": SettingDef(
+        bool,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="Drop tables-of-contents and classification-banner cover pages from results",
+    ),
     "history_rewrite": SettingDef(
         bool,
         nullable=False,
@@ -903,6 +939,12 @@ def get_default(key: str) -> object:
         group=SettingGroup.RETRIEVAL,
         help_text="Maximum unique sources contributing chunks to a single answer",
     ),
+    "neighbor_expansion": SettingDef(
+        int,
+        nullable=False,
+        group=SettingGroup.RETRIEVAL,
+        help_text="Adjacent chunks merged into each retrieved passage per side (0 = off)",
+    ),
     "diversity_max_per_source": SettingDef(
         int,
         nullable=False,

src/lilbee/cli/commands/ingest_sync.py

@@ -400,6 +400,7 @@ def index(
     apply_overrides(data_dir=data_dir, use_global=use_global)
     store = get_services().store
     store.ensure_fts_index()
+    store.ensure_scalar_indexes()
     built = store.ensure_vector_index(force=True)
     if cfg.json_mode:
         json_output({"command": "index", "vector_index": built})

src/lilbee/core/config/model.py

@@ -95,10 +95,11 @@ class Config(BaseSettings):
     max_embed_chars: int = Field(default=2000, ge=1)
     top_k: int = ConfigField(default=12, ge=1, writable=True)
     max_distance: float = ConfigField(default=0.75, ge=0.0, writable=True)
-    # Abstention floor against the canonical [0, 1] relevance score
-    # (0.0 = no filtering). When every retrieved chunk falls below it, ask
-    # refuses instead of feeding noise as context. On the fused reciprocal-rank
-    # scale an arm's top hit scores 0.5, so useful floors start around 0.4.
+    # Abstention floor against the [0, 1] fused relevance score (0.0 = no
+    # filtering). When every retrieved chunk falls below it, ask refuses instead
+    # of feeding noise as context. The fused score normalizes against the
+    # configured weight budget (a constant), so an arm's top hit scores a stable
+    # share of it; useful floors start around 0.4. Tune against your own corpus.
     min_relevance_score: float = ConfigField(default=0.0, ge=0.0, writable=True)
     adaptive_threshold: bool = ConfigField(default=False, writable=True)
     rag_system_prompt: str = ConfigField(
@@ -194,6 +195,42 @@ class Config(BaseSettings):
     # fusion arms stay exactly top_k deep.
     candidate_multiplier: int = ConfigField(default=3, ge=1, writable=True)
 
+    # Third lexical arm in hybrid search: BM25 over document titles, fused with
+    # the vector and chunk arms so a query naming a document by title surfaces
+    # its chunks. Off by default until the eval harness measures it.
+    title_search: bool = ConfigField(default=False, writable=True)
+
+    # Title arm weight relative to a full arm in rank fusion (1.0 = equal voice
+    # with the vector and chunk arms).
+    title_search_weight: float = ConfigField(default=0.5, ge=0.0, le=1.0, writable=True)
+
+    # Lexical (BM25) arm weight relative to the vector arm in rank fusion.
+    # 1.0 gives the two arms equal voice; lowering it lets
+    # a strong dense embedder dominate on corpora where the lexical arm adds
+    # noise rather than signal. The right value is corpus-dependent and set by
+    # the retrieval benchmark, not guessed here.
+    lexical_fusion_weight: float = ConfigField(default=1.0, ge=0.0, le=1.0, writable=True)
+
+    # Adaptive fusion: scale the BM25 arm per query by vector-arm confidence
+    # instead of a fixed lexical_fusion_weight (a peaked dense ranking downweights
+    # lexical, a flat one keeps it). On by default. lexical_fusion_weight is the
+    # ceiling the rule scales down from. Set adaptive_fusion=false to pin the
+    # fixed weight.
+    adaptive_fusion: bool = ConfigField(default=True, writable=True)
+
+    # Vector-similarity margin at which the lexical arm is fully silenced; smaller
+    # = more aggressive downweighting. 0 disables adaptation entirely (the lexical
+    # arm keeps its full fixed weight).
+    adaptive_fusion_margin: float = ConfigField(default=0.15, ge=0.0, le=2.0, writable=True)
+
+    # Drop tables-of-contents and classification-banner cover/title pages from
+    # search results. OFF by default: an evaluation A/B on a government-document
+    # corpus found the filter net-negative, because its cover-page heuristic also
+    # fires on short banner-carrying body pages. When on, a query-matched or
+    # top-ranked page is never dropped (searcher.search), so removal is limited to
+    # structural chunks the query did not hit. Re-validate per corpus before use.
+    filter_structural_chunks: bool = ConfigField(default=False, writable=True)
+
     # Chunk count at/above which sync builds an approximate (ANN) vector index
     # so search stays fast at millions of vectors. Below this, search uses exact
     # flat scan (faster and exact for small vaults). 0 disables the ANN index.
@@ -243,6 +280,11 @@ class Config(BaseSettings):
     # Chunks included in LLM context after adaptive selection.
     max_context_sources: int = ConfigField(default=8, ge=1, writable=True)
 
+    # Adjacent chunks pulled from the same source on each side of every
+    # selected chunk and merged into one contiguous passage, so a hit that
+    # lands mid-argument regains the text before and after it. 0 disables.
+    neighbor_expansion: int = ConfigField(default=0, ge=0, writable=True)
+
     # HyDE (Gao et al. 2022): hypothetical-answer embedding search. +~500ms.
     hyde: bool = ConfigField(default=False, writable=True)
 
@@ -978,14 +1020,24 @@ def _resolve_defaults(cls, data: Any) -> Any:
         if not isinstance(data, dict):
             return data
 
+        # An empty LILBEE_DATA_ROOT (delivered as "") must fall through to default
+        # resolution like an unset one, not become Path(".") = the process cwd.
+        if isinstance(data.get("data_root"), str) and not data["data_root"].strip():
+            data["data_root"] = None
         if data.get("data_root") in (None, _UNSET_PATH):
             data_env = os.environ.get("LILBEE_DATA", "").strip()
             if data_env:
                 data["data_root"] = Path(data_env)
             else:
                 local = find_local_root()
                 data["data_root"] = local if local is not None else default_data_dir()
-        root = data["data_root"]
+        # data_root may arrive as a raw string (e.g. from LILBEE_DATA_ROOT); the
+        # child-path derivations below use ``/``, so coerce to Path first.
+        # expanduser() so a "~/lilbee" value from a systemd unit or .env (which
+        # do not expand ~) points at the home dir instead of creating a literal
+        # ./~ tree that a server lock keyed on this path would then diverge on.
+        root = Path(data["data_root"]).expanduser()
+        data["data_root"] = root
         if data.get("documents_dir") in (None, _UNSET_PATH):
             data["documents_dir"] = root / "documents"
         if data.get("data_dir") in (None, _UNSET_PATH):

src/lilbee/data/export.py

@@ -10,7 +10,8 @@
 from typing import TYPE_CHECKING, cast
 
 from lilbee.data.ingest.extract import chunk_and_embed_pages
-from lilbee.data.store import ChunkWrite, PageTextRecord, SourceType
+from lilbee.data.ingest.title import derive_title
+from lilbee.data.store import ChunkWrite, PageTextRecord, SourceMeta, SourceType
 from lilbee.runtime.progress import DetailedProgressCallback, noop_callback
 
 if TYPE_CHECKING:
@@ -225,6 +226,11 @@ async def import_dataset(
         content_type = source_rows[0]["content_type"] or "text"
         page_texts = [(r["page"], r["text"]) for r in source_rows]
         chunks = await chunk_and_embed_pages(page_texts, name, content_type, on_progress)
+        # Imports carry no extraction metadata; the stem-derived title keeps
+        # imported chunks visible to the title search arm.
+        title = derive_title(name)
+        for chunk in chunks:
+            chunk["title"] = title
         # One locked transaction (cleanup + chunks + page texts + source row) so a
         # failure can't leave the source with its old rows deleted and no new ones;
         # the embedding-dim check inside runs before the cleanup delete.
@@ -238,6 +244,7 @@ async def import_dataset(
                     needs_cleanup=True,
                     page_texts=[dict(r) for r in source_rows],
                     source_type=SourceType.IMPORTED,
+                    meta=SourceMeta(title=title),
                 )
             ],
         )