0.2.6

CHANGELOG.md

@@ -1,5 +1,42 @@
 # Changelog
 
+## v0.2.6 (2026-08-12)
+
+Improves full-index (non-fast) performance on large projects, fixes a resolver JOIN defect that both slowed indexing and silently over-matched cross-file references, and moves fuzzy symbol search fully in-memory — resolver 10.2x faster on CodeScope's own index with zero precision loss, plus completed FAST-mode pruning rules and quantified discovery timing.
+
+### 🚀 Performance
+
+- **Dropped dead `semantic_records` rows at the DB write path** (`insertFileResultBatch`): `Literal` records (emitted for every numeric/string literal token, consumed by no downstream stage) and `Variable` records (only the in-memory GraphBuilder uses them; the full-index SQL `buildGraph` and all queries ignore them) are skipped when writing the DB, while `FileResult.records` stays intact so single-file symbol-graph queries keep Variable nodes. Measured on CodeScope's own `engine/src` (171 files): `semantic_records` 95,944 → 25,146 rows, SQLite flush 1,135ms → ~310ms, full-index `index_time` 2.18s → ~1.36s (**-38%**). (`store_batch.cpp`)
+
+- **Fixed a resolver self-join missing the `file_path` term**: the `global_var_types_` / `global_struct_fields_` preloads joined `semantic_records` on `parent_id = original_id AND project_id = project_id` but not `file_path`. Because `original_id` is **per-file** (it restarts each file, so different files reuse the same ids), one TypeRef row matched several files' same-`original_id` parents, inflating the join ~35x (variable-type query: 705k rows) and slowing the preloads. Adding `AND p.file_path = t.file_path` cut resolver time 11.2s → 5.4s (**-52%**) and the goagent full index 20.6s → 14.6s (**-29%**), while **resolving more refs correctly** (8,034 → 8,043) — the accuracy gate stays P/R/F1 = 1.0 (0 FP / 0 FN). (`resolver/pipeline.cpp`)
+
+- Added opt-in per-stage resolver timing (`CODESCOPE_PROFILE_RESOLVER=1`) that prints each load/resolve phase's wall time to stderr (a no-op when unset). (`resolver/pipeline.cpp`)
+
+- **Split `StateBuilder::buildModuleSummaries` into two CTEs**: the previous single 6-table `INSERT...SELECT` combined three `relation` LEFT JOINs with `graph_nodes`; SQLite then built four `COUNT(DISTINCT)` temp B-trees over a blown-up intermediate result (~29s on a 26k-node Go tree, the dominant `enhance_project` cost). Isolating the `graph_nodes` scan into its own `entry` CTE (joined only on `module_id` afterwards) plus `INDEXED BY` on each relation join cuts `buildModuleSummaries` from 17.1s → 174ms and `enhance_project` on goagent from 18.6s → 1.15s (**-94%**), with identical `module_summary` rows (184 modules). (`model/state_builder.cpp`)
+- **Added `idx_sr_proj_file_oid(project_id, file_path, original_id)` for the resolver's variable/field type self-joins** (`loadVarTypesStruct`): those JOIN `semantic_records t ON t.parent_id = p.original_id AND p.file_path = t.file_path` (kind=17 TypeRef → parent entity). The old `idx_sr_oid(project_id, original_id)` covers only `(project_id, original_id)`; because `original_id` is **per-file**, it collides across files, so SQLite pulled every same-numbered row from every file and filtered `file_path` by rowid lookup. On rustc (1M `semantic_records`) that made this one phase ~3.3s and the whole resolver ~7.2s (75% of `buildGraph`). The composite index keys on `(project_id, file_path, original_id)` so the join becomes an index seek (**~63ms, ~52x faster**); SQLite picks it automatically (no `INDEXED BY`). Measured on rustc full index: resolver 7192ms → 3103ms, `buildGraph` 9541ms → 5270ms, wall clock 40.9s → 35.7s (**below the 40s target**), with byte-identical `entity`/`scope`/`import`/`module_summary` counts and the accuracy gate still P/R/F1 = 1.0 (0 FP / 0 FN). Index is added to the membulk drop/create lists too. (`store_schema.cpp`, `store_membulk.cpp`)
+- **Raised the default parse-worker count from 4 → 8** (`indexProject`): the parse phase (tree-sitter + visitor + metrics) is pure CPU and doesn't touch SQLite (a single writer thread batches inserts), so on a 14-core machine the old 4-worker default wasted ~70% of cores. Measured against the full rustc index: 4 → 8 workers drops wall-clock from ~42s → ~34.5s (**-18%**); 8 → 12 yields no further gain because the SQLite writer becomes the throughput ceiling, so 8 is the sweet spot (also leaves cores for other processes). `CODESCOPE_WORKERS` still overrides. (`engine_index_project.cpp`)
+- **Eliminated the per-reference candidate deep-copy in `resolve_loop`** (`ResolverPipeline::run`): the exact-match path did `candidates = *cands` every reference, re-allocating up to `kMaxCandidatesToScore`(50) `Candidate` objects (each with 6 `std::string`s). The vector is now hoisted out of the loop and reserved, the exact path uses `resize()` + element-wise assignment to reuse retained element/string storage (the fuzzy path `clear()`s first), so the ~450k-reference loop reuses one buffer instead of allocating ~450k×50 Candidate copies. `resolve_loop` 1997ms → 1679ms (**-16%**) on rustc, accuracy gate still 0 FP / 0 FN. (`resolver/pipeline.cpp`)
+- **Fuzzy symbol search moved fully in-memory, eliminating per-entity SQL** (`resolver/fuzzy_resolver.{h,cpp}`, `resolver/pipeline.cpp`, `resolver/factors.h`): the resolver previously issued up to 3 SQL queries per unresolved ref (case-insensitive + prefix + suffix, each `prepare`+`step`) and hydrated every fuzzy hit with a per-id `SELECT ... FROM entity WHERE id=?` lookup. `FuzzyResolver` now loads all entities into memory once (ASCII-fold exact index + `sqliteLikeMatch` linear scan, byte-identical to SQLite LIKE semantics, so results match the old SQL path exactly), and `ResolverPipeline::run` builds an `entity_by_id` in-memory map right after Step 0 so fuzzy hits copy full candidates without any SQL round-trip in the hot loop. Measured on CodeScope's own index (215 files, serial full index): `resolver::run` 298ms → 30ms (**10.2x**), `buildGraph` 332ms → 62ms (5.4x), `engine_index_post_parse` 335ms → 66ms, and index-parallel total 686ms → **517ms (-25%)**. Zero precision loss: same-input A/B against the old SQL-fuzzy binary shows identical resolved refs / edges (`test_fuzzy_resolver` 9 assertions pass). On goagent (1,374 files) the in-memory path is fast enough that the old 500ms fuzzy budget never trips — it restores the 5,115 fuzzy queries the budget had silently dropped, resolution 8,201 → 8,287 refs (+86), +64 call edges. On rustc (6,029 files): fuzzy hits 32 → 918, +283 resolved refs, +223 call edges, at lower wall time in every case.
+- **Fuzzy prefix/suffix lookup upgraded from O(N) linear scan to O(log N) binary search** (`resolver/fuzzy_resolver.{h,cpp}`): `loadEntities` now builds two sorted indexes once — folded-name (for prefix) and reversed-folded-name (for suffix) — and wildcard-free queries use `std::lower_bound` instead of walking the whole entity array; `%`/`_` wildcard queries still take the exact `sqliteLikeMatch` linear path so results stay byte-identical to SQLite LIKE.
+- **`StateBuilder::buildModuleSummaries` merged its two `relation` LEFT JOINs into one** (`model/state_builder.cpp`): the previous `r_in`/`r_tgt` pair both scanned `idx_relation_target` and duplicated the target-side work (the self-loop exclusion moved into the incoming/dead `CASE` expressions). On rustc (117k relations × 129k entities) this phase drops from ~5.95s → **~0.25s (23.8x)**, result-identical (EXCEPT-diff both directions = 0).
+- **`idx_scope_kind_name(project_id, kind, name)` index + checked `import.source_scope_id` UPDATE** (`store/store_schema.cpp`, `store/store_graph.cpp`): buildGraph's function-scope INSERT and the import correlated subquery both filter `scope` on `kind`+`name`; without the index they scanned the whole table per entity/import row (rustc: 129,893 entities × 26,975 scopes), dominating the "scope" phase. The index turns it into an index seek, and the UPDATE's return value is now checked so a failure can no longer be silently swallowed (previously it left every `source_scope_id` at its 0 default).
+- **FAST-mode pruning rules completed + discovery timing instrumented** (`filter_policy.{h,cpp}`, `engine_index_project.cpp`): `fast_extra_skip_dirs_` (previously an empty "reserved for future" set, so FAST mode was effectively identical to NORMAL) now skips 11 build/test-artifact dirs (`.output`, `storybook-static`, `__generated__`, `playwright-report`, `test-results`, `allure-results`, `allure-report`, `.sass-cache`, `.scss-cache`, `logs`, `.logs`) plus 4 exact filenames via the new `fast_extra_filenames_` (`.eslintcache`, `.stylelintcache`, `.prettiercache`, `tsconfig.tsbuildinfo`). Discovery gets its own wall-clock log (`discovery=<ms>` with corrected `seen_dirs` counting directories only) — first-time quantification: rustc 143ms / 4,650 dirs, goagent 25ms / 845 dirs. A synthetic A/B project with logs/test-results/.output/.eslintcache drops candidates 4 → 1.
+
+### 🐛 Bug Fixes
+
+- **`get_graph` (and `getProjectOverview`'s stats) still read the deprecated, empty `graph_nodes` / `graph_edges` tables**: `getGraph` counted and paged nodes from `graph_nodes` (no longer written since the v0.2.5 canonical-schema migration, so `nodes` always returned 0) and edges from `graph_edges` (which accumulates stale legacy rows — 16,089 rows vs. 4,415 canonical `relation` rows). Now `getGraph` pages `entity` (nodes) + `relation` (edges) with the columns aliased back to `node_type` / `source_node_id` / `target_node_id` / `edge_type` so the public JSON schema is unchanged; `node_type_filter` maps to `entity.kind` and `edge_type_filter` to `relation.type`. Verified: `get_graph` returns `{total_nodes:39686, total_edges:4415}` matching `get_graph_stats`. (`query_analysis.cpp`)
+- **`FilterPolicy::setMode()` never rebuilt the active skip sets**: `buildActiveSets()` ran only in the constructor, so setting `CODESCOPE_INDEX_MODE=fast` after construction left `fast_extra_skip_dirs_` permanently inert (the completed FAST-mode rules above never applied in that path). `setMode()` now calls `buildActiveSets()`. (`filter_policy.cpp`)
+- **Discovery `seen_dirs` counted every directory-iterator entry including files**: `std::filesystem::recursive_directory_iterator` yields files too, inflating the metric (~44k for a 215-file project). It now counts only `entry.is_directory()`, and the JSON `discovery.seen_dirs` and the new `discovery=<ms>` log share the corrected counter. (`engine_index_project.cpp`)
+- **`import.source_scope_id` UPDATE failures were silently ignored**: `buildGraph` ran the correlated-subquery UPDATE without checking the return value, so on failure every `source_scope_id` stayed at its 0 default (imports lost their owning scope). The result is now checked and errors logged. (`store_graph.cpp`)
+- **Fuzzy prefix/suffix binary-search results are now truncated in rowid order** (`resolver/fuzzy_resolver.cpp`): the O(log N) `lower_bound` paths enumerate matches in folded-name order, whereas the old SQL `name LIKE ? || '%' LIMIT ?` returned rows in rowid scan order. When a query matched more than `kFuzzyCandidateLimit` (5) entities, the retained subset differed, so the winning candidate — and the resolved CALLS edge — could diverge from the old binary on large projects (rustc's 129k entities routinely exceed 5 matches for 3+ char prefixes), contradicting the in-memory rewrite's byte-identical contract. Both paths now collect all matches, sort by id (= rowid = load order), then truncate — restoring the old LIMIT semantics. A/B on the CodeScope self index: edges 1,249 → 1,189, nodes/files unchanged, accuracy gate still P/R/F1 = 1.0. (Follow-up to the in-memory fuzzy rewrite.)
+- **`StateBuilder::buildModuleSummaries` bound 9 parameters but the SQL has only 8 `?` placeholders** (`model/state_builder.cpp`): after the single-JOIN rewrite the bind loop still ran `i <= 9` with a "agg (5) + entry (3) + SELECT (1) = 9" comment; the 9th `sqlite3_bind_int64` hit a non-existent parameter (SQLITE_RANGE, silently ignored). The loop now binds exactly 8 (agg 4 + entry 3 + SELECT 1) and the comment is corrected.
+- **Skills scripts indexed and queried different databases when `CODESCOPE_DB_PATH` was unset** (`skills/analyze.sh`, `skills/index.sh`): `codescope worker` takes the DB as a positional argument (default `/tmp/codescope_index.db`) while `codescope cli` resolves it from the `CODESCOPE_DB_PATH` env var (default `.codescope/codescope.db`). With the default environment the scripts silently reported stats from an empty/stale DB after indexing. Both scripts now `export CODESCOPE_DB_PATH="$DB"` so worker and cli share one database.
+
+### 📚 Docs & Skills
+
+- **Skills scripts updated to the current MCP tool set**: `index_project` and `get_hotspots` are no longer in `TOOL_HANDLERS`, so `skills/index.sh` and `skills/hotspots.sh` now use `codescope worker <db> <dir> <lang> <name> <pid>` (serial) / `codescope index-parallel` (parallel) for indexing and `get_knowledge_graph` for hotspot-style density queries; `analyze.sh` and `skills/skills.md` (plus `docs/{en,zh}/skills.md`) were updated to match, with `CODESCOPE_DB_PATH` honored for the DB location. (`skills/*.sh`, `skills/skills.md`, `docs/en/skills.md`, `docs/zh/skills.md`)
+- **Benchmark section re-measured after the fuzzy ordering fix** (`README.md`, `README.zh.md`): §7 now reflects `target/release/codescope` (v0.2.6) in `CODESCOPE_INDEX_MODE=normal` on the 2026-08-14 run — CodeScope self 0.95 s / 1,189 edges (pre-fix: 1,249 edges; nodes/files unchanged), tinygo 1.77 s / 4,485 edges, rustc 38.94 s / 117,284 edges; per-query MCP latency (median of 7), micro benchmarks, cross-file CALLS ratios, and the corrected `graph_query` figure (88.8 ms with LIMIT 100 — the previously published 0.03 ms was the error-path response of an invalid DSL, not a real query).
+
 ## v0.2.5 (2026-08-05)
 
 Removes the LadybugDB (Kuzu) dependency entirely — SQLite is now the **sole graph store** on all platforms (CSR `adjacency`/`adjacency_rev` tables + C++ BFS). Graph queries, verifiers, and self-check tools behave identically to the LadybugDB-backed build, with faster indexing (rust: 52.6s → 31.6s, -40%; no graph-rebuild pass) and zero external runtime dependencies. Also fixes three self-check defects found while dogfooding the new backend. Restores the three capabilities that the Step 10 sprint had formally sunset (complexity metrics, n-gram semantic vector search, and metrics-driven readiness), hardens FunctionImplements verification with real call-chain + signature evidence, and fixes Go interface embedding (composition) dispatch.

Cargo.lock

@@ -4,7 +4,7 @@ version = 4
 
 [[package]]
 name = "codescope"
-version = "0.2.5"
+version = "0.2.6"
 dependencies = [
  "libc",
  "once_cell",