perf(search): whole-query result LRU (10x on repeated searches)#612
Conversation
…earch Agents re-issue identical searches constantly. Two sibling caches now serve repeats: SearchResultCache (searchContent - a hit dupes the cached results into the caller's allocator, same ownership contract as a fresh search) and PlainRenderCache (renderPlainSearch - the MCP fast path renders straight to text and never reaches searchContent). 64 entries / 4 MB each, LRU. An entry is served only when BOTH its generation and env fingerprint still match. Explorer.search_gen bumps (atomically - searches hold the SHARED lock) on every mutation that can change results: commitParsedFileOwnedOutline, removeFile, rebuildWordIndex, and the one-shot lazy ranking builds (ensureSymbolIndex, call-graph, co-change). The fingerprint hashes the nine ranking kill-switch env vars, so tests that toggle CODEDB_LEX_FREQ_PENALTY et al mid-process can never be served results computed under the other setting. The generation is read BEFORE a search runs, so a concurrent mutation makes the stored entry stale immediately. CODEDB_NO_SEARCH_CACHE=1 disables both caches. The repo benchmark sets it for its per-query rows (numbers stay comparable across versions) and adds one explicit "cached" row: error 20.7us uncached -> 2.0us hit (10x). 8 new tests: hit identity + caller ownership, indexFile/removeFile invalidation, env-fingerprint staleness, kill-switch bypass, LRU bound, and the renderPlainSearch pair. 822/822 total, e2e MCP 20/20. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c60f7d981
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| pub fn commitParsedFileOwnedOutline(self: *Explorer, path: []const u8, content: []const u8, outline: FileOutline, full_index: bool, skip_trigram: bool) !void { | ||
| self.bumpSearchGen(); |
There was a problem hiding this comment.
Because this bump happens before the exclusive lock and before the indexed state is updated, a concurrent search can read the new generation at searchContent, acquire the shared lock first, compute results from the old index, and then store those old results under the new generation. After this file commit completes, later identical searches will see the same generation and can serve that stale cache entry indefinitely; the same ordering also affects removals/rebuilds that bump before taking the write lock. Bump the generation while holding the write lock after the mutation, or otherwise make the generation read/state read atomic with respect to the mutation.
Useful? React with 👍 / 👎.
Benchmark Regression ReportThresholds: 10.00% and 50,000 ns absolute delta
|
1 participant
Summary
Whole-query LRU caches for repeated searches - agents re-issue identical queries constantly, and until now every repeat re-ran the full tier pipeline.
SearchResultCache(searchContent): a hit dupes the cached results into the caller's allocator - identical ownership contract to a fresh search.PlainRenderCache(renderPlainSearch): the MCPcodedb_searchfast path renders straight to text and never reaches searchContent, so it gets its own rendered-bytes cache.Correctness model
An entry is served only when BOTH validators still match:
Explorer.search_gen, atomic): bumped by every mutation that can change results -commitParsedFileOwnedOutline(all indexing funnels through it),removeFile,rebuildWordIndex, and the one-shot lazy ranking builds (ensureSymbolIndex, call-graph, co-change - these flip ranking gates mid-flight on first use).CODEDB_NO_COCHANGE,CODEDB_LEX_FREQ_PENALTY, ...). The existing test suite toggles these mid-process between identical searches on one explorer; the fingerprint guarantees those can never see stale results.The generation is read BEFORE a search runs, so a concurrent mutation makes the stored entry stale immediately (conservative direction).
CODEDB_NO_SEARCH_CACHE=1disables both caches entirely.Benchmark honesty
The repo benchmark sets
CODEDB_NO_SEARCH_CACHE=1for its per-query rows so numbers stay comparable across versions, and adds one explicitcachedrow:Test plan
zig build test --summary all- 822/822python3 scripts/e2e_mcp_test.py- 20/20Generated with Devin