fix(search): reconcile skip_trigram_files with disk-loaded trigram indexes#615
Merged
Merged
Conversation
…ndexes Snapshot restore parks EVERY file in skip_trigram_files (it cannot know what a disk trigram index covers), and nothing ever removed entries when the index was later mmap-loaded. Worse, loadTrigramFromDiskIfPresent early-returned whenever trigram fileCount() > 0 — and the snapshot freshness pass reindexes changed files into the heap trigram BEFORE that check runs, so one dirty file blocked the disk load entirely. Net effect on the dominant serve/mcp/cli-daemon startup path: tier 3 content-scanned the ENTIRE project on every fall-through query, with recall_complete=false. Measured live on this repo: 613/616 files in the scan set, 9.2ms negative searches -> 0 files, 0.5-0.9ms, recall_complete =true. This matches the production search tail (p90 30ms). - adoptTrigramIndex: single funnel for trigram replacement — swaps, bumps the search generation, and prunes covered files from the skip set - adoptTrigramBase: mmap disk load keeps freshness-reindexed files as a masking overlay (their newer content wins; stale base entries masked) - rebuildTrigrams: prunes what it covers + bumps the generation (it did neither) - both disk-load gates now skip only when already disk-backed or the heap covers the whole project - watcher: trigram cap env override CODEDB_TRIGRAM_CAP (measured on a 20k-file corpus: uncapped = zero-hit queries 7.1->1.4ms for +110MB peak RSS); provenance meta reports the effective cap Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Benchmark Regression ReportThresholds: 10.00% and 50,000 ns absolute delta
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces #614 (auto-closed when its stacked base branch was deleted). Follow-up to #613, now merged into
release/0.2.5825.Fixes the dominant production search-tail bug: on the standard serve/mcp/cli-daemon startup path, tier 3 content-scanned the entire project on every fall-through query with
recall_complete=false.Two compounding failures:
skip_trigram_files(it cannot know what a disk trigram index covers), and nothing reconciled the set when the disk index was later mmap-loaded.loadTrigramFromDiskIfPresentearly-returned whenever trigramfileCount() > 0— and the snapshot freshness pass reindexes changed files into the heap trigram before that check runs, so a single dirty file blocked the disk trigram load entirely.Measured live on this repo: 613/616 files in the scan set, 9.2ms negative searches -> 0 files, 0.5–0.9ms,
recall_complete=true. This matches the production search p90 (30ms).Fix:
adoptTrigramIndex: single funnel for trigram replacement — swaps, bumps the search generation, prunes covered files from the skip setadoptTrigramBase: mmap disk load keeps freshness-reindexed files as a masking overlay, so their newer content wins over stale base entriesrebuildTrigrams: now prunes what it covers + bumps the generation (it did neither)CODEDB_TRIGRAM_CAPenv override (measured on a 20k-file corpus: uncapped = zero-hit queries 7.1ms -> 1.4ms for +110MB peak RSS, +300ms index time); provenance meta reports the effective cap. Default unchanged.Test plan
zig build test— 835/835 (includes new overlay test: new content wins / stale masked / base-only files resolve)python3 scripts/e2e_mcp_test.py— 20/20Generated with Devin