Make search scale: bounded memory, fast default path - #962
Open
rikvanriel wants to merge 7 commits into
Open
rikvanriel wants to merge 7 commits into
rikvanriel wants to merge 7 commits into
Conversation
When a collection has no changes, reindexing still reads every file to hash it. With thousands of files this costs minutes of IO and hashing even when nothing changed. Add file_sync_state table tracking mtime_ms, size, and content hash per (collection, path). On reindex, stat the file first and skip the read entirely when mtime and size match the cache. If mtime changed but hash is identical only the cached mtime is updated. Skip files over 10MB and empty files, and remove the cache entry on orphan deletion. Second reindex with no changes performs no file reads, taking update from minutes to seconds when the collection is unchanged.
Previously, searchFTS and searchVec loaded the whole document body as content.doc as body. On collections with large transcripts this caused 20 candidates times 6 expansions times 100KB = 12MB of string copies through better-sqlite3 into V8, observed as 4.2GB heap. Now the bodies are bounded at the SQL level using substr(content.doc, 1, 262144) as body. This limits each body to 256KB. With 40 winners the max is 10MB vs unbounded before. Rerank input stays 900 tokens, about 3.6KB, per chunk. The long term fix fetches only the winning chunk via substr(doc, pos, len) for the 40 RRF winners, but this commit prevents the OOM first.
Large result sets previously used all() which materialized full arrays in V8 heap. On an index with 13k files and 89k chunk vectors this means tens of thousands of rows (active paths, sync-state entries, pending-embedding docs, candidate chunk vectors) in a single array, which caused OOM. Now they stream via iterate(): sync-state reads, pending-embedding scans, active-path listings, glob/fuzzy scans, hash_seq eligibility (with early exit past the cap, falling back to ANN), and embedding-body fetches (substr-bound, with a bytes-only path). Small queries remain with all() as they are bounded.
Previously, each winning file loaded its whole document body into JS. On large transcripts this used gigabytes (observed 4.2GB). Now only the winning 3.6KB chunk at its ranked position is fetched via substr(doc, pos+1, len) at the SQL level. 40 winners stay under 500KB total. FTS winners carry no position, so they fall back to the doc head slice via pos 0. Callers pass chunkPos from SearchResult when available. Chunk-only rerank approach copied from qmd-py's bounded hydration code and guard test.
Previously a file scored from its single best chunk hit. Now a file aggregates its top 3 non-overlapping chunks: sort the file's chunk hits by vector distance to the query embedding, best first, then greedily keep a hit unless it overlaps an already-kept chunk (adjacent sequence numbers with character positions within one chunk length). Adjacent chunks share 15 percent overlap, so counting both would double count the same passage. File score is 1-d0 plus 0.25 times 1-d1 plus 0.1 times 1-d2. This approximates reranking the whole file at chunk-vector cost. Chunk distance is the cosine distance from the vector index, capped into a similarity in 0 to 1.
When the top keyword (FTS) score clears 0.70, the keyword ranking is trusted on its own: query expansion and the vector phase are skipped and BM25 plus RRF results are returned directly. This keeps a typical search fast on CPU-only hardware. Lower the strong-signal bar from score 0.85 and gap 0.15 to score 0.70 and gap 0.08, and add rerank-skip thresholds at 0.70 and 0.15 so reranking stops early when chunk vectors already agree with the fused ranking. Skipping the vector phase also skips loading the embedding model, so the fast path avoids a 13 second model load. Make no-rerank the default in CLI, MCP, and SDK: CLI no-rerank defaults to true, MCP rerank defaults to false, SDK skips rerank unless rerank is true. Full rerank stays available with no-rerank false or rerank true. Measured on 5 sample queries over a ~3k-file collection: the slow queries previously took 54 and 71 seconds, the fast path takes 0.5 to 1.5 seconds, about 35 to 60 times faster. Top-5 overlap is 90 percent or more, 4 to 5 files stay the same. Memory on the heaviest query drops from 1.44GB to 164MB, about 9 times less. Multi-vector file scoring from the previous commit keeps recall without a cross-encoder.
Rocchio pseudo-relevance feedback: top keyword matches may share
query vocabulary but not the underlying theme. Their collective
embedding points toward the thematic cluster. Example query "async
task cancellation": top-3 are about future/cancel tokens, their mean
pulls docs about cooperative cancellation that don't mention query
terms.
Flow: keyword plus vector search fused into one ranking (RRF), take
the top 3 files as theme representatives, embed their winning chunk
texts, average into one centroid vector, vector-search the centroid
for 20 nearby files, fuse the centroid hits back into the ranking
as one more list at weight 1.0.
Stored-vector path: reuse the chunk embeddings already stored in the
vector index for the winning files. If all are present, build the
centroid directly from them, avoiding re-embedding and a model load.
Fall back to embedding only when stored vectors are missing or were
made by a different model version.
Auto-enable on weak keyword results: when the fused ranking holds 6
or fewer files, or the top keyword score is below 0.35 (scores run 0
to 1, higher is better), centroid expansion switches on even if
disabled in config, because the fast path returned keyword-only
results with too few matches to trust. Otherwise it stays off by
default until a faster vector index lands.
Eval on 10 queries: centroid changes top5 in 80 percent (overlap 3.0/5
avg 60 percent). Half pull new docs beyond the base top-10 into the
new top-5, 3 of 10 pull in docs ranked outside the base top-100 on
queries where very few documents matched ("async cancellation", "RRF
fusion", "amdgpu freeze"). Cost: the second vector search takes 13s
(flat scan over 88k vectors without a fast index). Fast path stays
0.5-1.5s without centroid.
Author
|
This should fix #952 and related issues. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make search scale: bounded memory, fast default path
Motivation
Searching a multi-thousand-file collection with large transcripts could
use gigabytes of heap (observed 4.2GB) and take over a minute per query
on CPU-only hardware. Three root causes: full document bodies were
loaded into JS for every candidate, every unbounded query used
all()which materialized full arrays in the V8 heap, and every search always
ran query expansion, vector search, embedding-model load, and LLM
reranking even when the keyword ranking was already decisive.
Reindexing had the same shape of problem: with no changes it still read
and hashed every file, costing minutes of IO on large collections.
Results
On 5 sample queries over a ~13k-file collection, the slow queries went
from 54s/71s to 0.5-1.5s (35-60x faster), peak memory on the heaviest
query from 1.44GB to 164MB (~9x less), with 90%+ top-5 overlap against
the old full pipeline (4-5 of the top-5 files unchanged). An unchanged
collection now reindexes in seconds instead of minutes. A 10-query eval
of the optional centroid expansion changes the top-5 in 80% of queries
and pulls genuinely new documents (outside the old top-100) into the
top-5 on half of them.
Patches
feat(index): add mtime+size fast-path via file_sync_state-reindex stats each file first and skips the read entirely when mtime
and size match the cache, so an unchanged collection updates in
seconds; files over 10MB and empty files are skipped.
fix(search): bound hydration at DB level via substr, not JS-caps each fetched body at 256KB in SQL as a first stop-gap against
the heap blowup (40 winners max ~10MB instead of unbounded).
fix(search,embed,index): stream large queries with iterate()-converts every unbounded
all()to streamingiterate(); on anindex with 13k files and 89k chunk vectors this keeps tens of
thousands of rows out of a single V8 array. Small bounded queries
keep
all().feat(search): bound hydration and use chunk-level slices-fetches only each winner's ranked 3.6KB chunk via
substr(doc, pos+1, len), so 40 winners stay under ~500KB total;chunk-only rerank approach copied from qmd-py's bounded hydration
code and guard test.
feat(search): score files from top non-overlapping chunks-scores a file from its top-3 non-overlapping chunk hits (best vector
distance to the query embedding first, adjacent overlapping chunks
skipped so shared 15% overlap is not double-counted), approximating
whole-file rerank at chunk cost.
feat(cli, mcp, sdk): make fast search the default- trusts thekeyword ranking alone when the top FTS score clears 0.70, skipping
expansion, vector search, model load, and rerank; full rerank stays
available via
no-rerank false/rerank true.feat(search): add centroid expansion- optional Rocchio step thataverages the top-3 winners' embeddings and searches the centroid for
thematically related files the keywords missed, reusing stored
vectors when available; auto-enables only when keyword recall looks
thin (off by default otherwise).