perf: bound BM25 candidates with PostgreSQL FTS

[Ray0907]

Summary

• prefilter the content and path BM25 channels with the existing GIN-backed PostgreSQL TSV fields
• use OR semantics across query tokens and apply document, section, chunk-type, and signal-path filters before the candidate limit
• keep Python BM25 as the final channel reranker while bounding both normal and fallback candidate loading
• add configurable candidate limits, per-channel metrics, environment examples, and focused unit/HTTP contract coverage

Root cause

The content and path channels loaded every active scoped chunk into Python before checking token overlap and building the BM25 corpus. Large namespaces therefore paid corpus-sized memory and CPU costs for every retrieval step.

The SQL prefilter now uses content_search_tsv or path_search_tsv with websearch_to_tsquery('simple', ...), preserving any-token behavior. Section exclusions are also evaluated inside the scoped query before LIMIT, including a regression guard for sectionless chunks.

Validation

• uv run pytest packages/shared-python/shared/tests/test_retrieval_search_channels.py apps/api/tests/contract/test_retrieval_contract.py -q
  • 24 passed
• make lint
  • all checks passed
• make typecheck
  • 0 errors, 0 warnings

Closes #195

[EricNGOntos]

Thanks for this — classic BM25 candidate loading is a real issue we want to keep fixing (classic mode is a long-lived path even though map-nav is now the default).

We're landing a combined fix in #252 that keeps your strongest pieces:

• apply exclude_sections in SQL before the candidate LIMIT (so exclusions don't eat the budget)
• .env.example + channel unit coverage for the bounded candidate path

It also takes the safer server-side OR tsquery + full-scan fallback from #251, and credits both of you with Co-authored-by.

Closing this PR as superseded by #252 so we don't merge two conflicting approaches for #195. Please keep contributing — follow-ups welcome (classic large-namespace soak tests, metrics, etc.).

[EricNGOntos]

Superseded by #252 (combined landing of #244 + #251 for classic BM25 FTS prefilter).