diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c8d489..0347d1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,42 @@ # Changelog +## v0.2.6 (2026-08-12) + +Improves full-index (non-fast) performance on large projects, fixes a resolver JOIN defect that both slowed indexing and silently over-matched cross-file references, and moves fuzzy symbol search fully in-memory — resolver 10.2x faster on CodeScope's own index with zero precision loss, plus completed FAST-mode pruning rules and quantified discovery timing. + +### 🚀 Performance + +- **Dropped dead `semantic_records` rows at the DB write path** (`insertFileResultBatch`): `Literal` records (emitted for every numeric/string literal token, consumed by no downstream stage) and `Variable` records (only the in-memory GraphBuilder uses them; the full-index SQL `buildGraph` and all queries ignore them) are skipped when writing the DB, while `FileResult.records` stays intact so single-file symbol-graph queries keep Variable nodes. Measured on CodeScope's own `engine/src` (171 files): `semantic_records` 95,944 → 25,146 rows, SQLite flush 1,135ms → ~310ms, full-index `index_time` 2.18s → ~1.36s (**-38%**). (`store_batch.cpp`) + +- **Fixed a resolver self-join missing the `file_path` term**: the `global_var_types_` / `global_struct_fields_` preloads joined `semantic_records` on `parent_id = original_id AND project_id = project_id` but not `file_path`. Because `original_id` is **per-file** (it restarts each file, so different files reuse the same ids), one TypeRef row matched several files' same-`original_id` parents, inflating the join ~35x (variable-type query: 705k rows) and slowing the preloads. Adding `AND p.file_path = t.file_path` cut resolver time 11.2s → 5.4s (**-52%**) and the goagent full index 20.6s → 14.6s (**-29%**), while **resolving more refs correctly** (8,034 → 8,043) — the accuracy gate stays P/R/F1 = 1.0 (0 FP / 0 FN). (`resolver/pipeline.cpp`) + +- Added opt-in per-stage resolver timing (`CODESCOPE_PROFILE_RESOLVER=1`) that prints each load/resolve phase's wall time to stderr (a no-op when unset). (`resolver/pipeline.cpp`) + +- **Split `StateBuilder::buildModuleSummaries` into two CTEs**: the previous single 6-table `INSERT...SELECT` combined three `relation` LEFT JOINs with `graph_nodes`; SQLite then built four `COUNT(DISTINCT)` temp B-trees over a blown-up intermediate result (~29s on a 26k-node Go tree, the dominant `enhance_project` cost). Isolating the `graph_nodes` scan into its own `entry` CTE (joined only on `module_id` afterwards) plus `INDEXED BY` on each relation join cuts `buildModuleSummaries` from 17.1s → 174ms and `enhance_project` on goagent from 18.6s → 1.15s (**-94%**), with identical `module_summary` rows (184 modules). (`model/state_builder.cpp`) +- **Added `idx_sr_proj_file_oid(project_id, file_path, original_id)` for the resolver's variable/field type self-joins** (`loadVarTypesStruct`): those JOIN `semantic_records t ON t.parent_id = p.original_id AND p.file_path = t.file_path` (kind=17 TypeRef → parent entity). The old `idx_sr_oid(project_id, original_id)` covers only `(project_id, original_id)`; because `original_id` is **per-file**, it collides across files, so SQLite pulled every same-numbered row from every file and filtered `file_path` by rowid lookup. On rustc (1M `semantic_records`) that made this one phase ~3.3s and the whole resolver ~7.2s (75% of `buildGraph`). The composite index keys on `(project_id, file_path, original_id)` so the join becomes an index seek (**~63ms, ~52x faster**); SQLite picks it automatically (no `INDEXED BY`). Measured on rustc full index: resolver 7192ms → 3103ms, `buildGraph` 9541ms → 5270ms, wall clock 40.9s → 35.7s (**below the 40s target**), with byte-identical `entity`/`scope`/`import`/`module_summary` counts and the accuracy gate still P/R/F1 = 1.0 (0 FP / 0 FN). Index is added to the membulk drop/create lists too. (`store_schema.cpp`, `store_membulk.cpp`) +- **Raised the default parse-worker count from 4 → 8** (`indexProject`): the parse phase (tree-sitter + visitor + metrics) is pure CPU and doesn't touch SQLite (a single writer thread batches inserts), so on a 14-core machine the old 4-worker default wasted ~70% of cores. Measured against the full rustc index: 4 → 8 workers drops wall-clock from ~42s → ~34.5s (**-18%**); 8 → 12 yields no further gain because the SQLite writer becomes the throughput ceiling, so 8 is the sweet spot (also leaves cores for other processes). `CODESCOPE_WORKERS` still overrides. (`engine_index_project.cpp`) +- **Eliminated the per-reference candidate deep-copy in `resolve_loop`** (`ResolverPipeline::run`): the exact-match path did `candidates = *cands` every reference, re-allocating up to `kMaxCandidatesToScore`(50) `Candidate` objects (each with 6 `std::string`s). The vector is now hoisted out of the loop and reserved, the exact path uses `resize()` + element-wise assignment to reuse retained element/string storage (the fuzzy path `clear()`s first), so the ~450k-reference loop reuses one buffer instead of allocating ~450k×50 Candidate copies. `resolve_loop` 1997ms → 1679ms (**-16%**) on rustc, accuracy gate still 0 FP / 0 FN. (`resolver/pipeline.cpp`) +- **Fuzzy symbol search moved fully in-memory, eliminating per-entity SQL** (`resolver/fuzzy_resolver.{h,cpp}`, `resolver/pipeline.cpp`, `resolver/factors.h`): the resolver previously issued up to 3 SQL queries per unresolved ref (case-insensitive + prefix + suffix, each `prepare`+`step`) and hydrated every fuzzy hit with a per-id `SELECT ... FROM entity WHERE id=?` lookup. `FuzzyResolver` now loads all entities into memory once (ASCII-fold exact index + `sqliteLikeMatch` linear scan, byte-identical to SQLite LIKE semantics, so results match the old SQL path exactly), and `ResolverPipeline::run` builds an `entity_by_id` in-memory map right after Step 0 so fuzzy hits copy full candidates without any SQL round-trip in the hot loop. Measured on CodeScope's own index (215 files, serial full index): `resolver::run` 298ms → 30ms (**10.2x**), `buildGraph` 332ms → 62ms (5.4x), `engine_index_post_parse` 335ms → 66ms, and index-parallel total 686ms → **517ms (-25%)**. Zero precision loss: same-input A/B against the old SQL-fuzzy binary shows identical resolved refs / edges (`test_fuzzy_resolver` 9 assertions pass). On goagent (1,374 files) the in-memory path is fast enough that the old 500ms fuzzy budget never trips — it restores the 5,115 fuzzy queries the budget had silently dropped, resolution 8,201 → 8,287 refs (+86), +64 call edges. On rustc (6,029 files): fuzzy hits 32 → 918, +283 resolved refs, +223 call edges, at lower wall time in every case. +- **Fuzzy prefix/suffix lookup upgraded from O(N) linear scan to O(log N) binary search** (`resolver/fuzzy_resolver.{h,cpp}`): `loadEntities` now builds two sorted indexes once — folded-name (for prefix) and reversed-folded-name (for suffix) — and wildcard-free queries use `std::lower_bound` instead of walking the whole entity array; `%`/`_` wildcard queries still take the exact `sqliteLikeMatch` linear path so results stay byte-identical to SQLite LIKE. +- **`StateBuilder::buildModuleSummaries` merged its two `relation` LEFT JOINs into one** (`model/state_builder.cpp`): the previous `r_in`/`r_tgt` pair both scanned `idx_relation_target` and duplicated the target-side work (the self-loop exclusion moved into the incoming/dead `CASE` expressions). On rustc (117k relations × 129k entities) this phase drops from ~5.95s → **~0.25s (23.8x)**, result-identical (EXCEPT-diff both directions = 0). +- **`idx_scope_kind_name(project_id, kind, name)` index + checked `import.source_scope_id` UPDATE** (`store/store_schema.cpp`, `store/store_graph.cpp`): buildGraph's function-scope INSERT and the import correlated subquery both filter `scope` on `kind`+`name`; without the index they scanned the whole table per entity/import row (rustc: 129,893 entities × 26,975 scopes), dominating the "scope" phase. The index turns it into an index seek, and the UPDATE's return value is now checked so a failure can no longer be silently swallowed (previously it left every `source_scope_id` at its 0 default). +- **FAST-mode pruning rules completed + discovery timing instrumented** (`filter_policy.{h,cpp}`, `engine_index_project.cpp`): `fast_extra_skip_dirs_` (previously an empty "reserved for future" set, so FAST mode was effectively identical to NORMAL) now skips 11 build/test-artifact dirs (`.output`, `storybook-static`, `__generated__`, `playwright-report`, `test-results`, `allure-results`, `allure-report`, `.sass-cache`, `.scss-cache`, `logs`, `.logs`) plus 4 exact filenames via the new `fast_extra_filenames_` (`.eslintcache`, `.stylelintcache`, `.prettiercache`, `tsconfig.tsbuildinfo`). Discovery gets its own wall-clock log (`discovery=` with corrected `seen_dirs` counting directories only) — first-time quantification: rustc 143ms / 4,650 dirs, goagent 25ms / 845 dirs. A synthetic A/B project with logs/test-results/.output/.eslintcache drops candidates 4 → 1. + +### 🐛 Bug Fixes + +- **`get_graph` (and `getProjectOverview`'s stats) still read the deprecated, empty `graph_nodes` / `graph_edges` tables**: `getGraph` counted and paged nodes from `graph_nodes` (no longer written since the v0.2.5 canonical-schema migration, so `nodes` always returned 0) and edges from `graph_edges` (which accumulates stale legacy rows — 16,089 rows vs. 4,415 canonical `relation` rows). Now `getGraph` pages `entity` (nodes) + `relation` (edges) with the columns aliased back to `node_type` / `source_node_id` / `target_node_id` / `edge_type` so the public JSON schema is unchanged; `node_type_filter` maps to `entity.kind` and `edge_type_filter` to `relation.type`. Verified: `get_graph` returns `{total_nodes:39686, total_edges:4415}` matching `get_graph_stats`. (`query_analysis.cpp`) +- **`FilterPolicy::setMode()` never rebuilt the active skip sets**: `buildActiveSets()` ran only in the constructor, so setting `CODESCOPE_INDEX_MODE=fast` after construction left `fast_extra_skip_dirs_` permanently inert (the completed FAST-mode rules above never applied in that path). `setMode()` now calls `buildActiveSets()`. (`filter_policy.cpp`) +- **Discovery `seen_dirs` counted every directory-iterator entry including files**: `std::filesystem::recursive_directory_iterator` yields files too, inflating the metric (~44k for a 215-file project). It now counts only `entry.is_directory()`, and the JSON `discovery.seen_dirs` and the new `discovery=` log share the corrected counter. (`engine_index_project.cpp`) +- **`import.source_scope_id` UPDATE failures were silently ignored**: `buildGraph` ran the correlated-subquery UPDATE without checking the return value, so on failure every `source_scope_id` stayed at its 0 default (imports lost their owning scope). The result is now checked and errors logged. (`store_graph.cpp`) +- **Fuzzy prefix/suffix binary-search results are now truncated in rowid order** (`resolver/fuzzy_resolver.cpp`): the O(log N) `lower_bound` paths enumerate matches in folded-name order, whereas the old SQL `name LIKE ? || '%' LIMIT ?` returned rows in rowid scan order. When a query matched more than `kFuzzyCandidateLimit` (5) entities, the retained subset differed, so the winning candidate — and the resolved CALLS edge — could diverge from the old binary on large projects (rustc's 129k entities routinely exceed 5 matches for 3+ char prefixes), contradicting the in-memory rewrite's byte-identical contract. Both paths now collect all matches, sort by id (= rowid = load order), then truncate — restoring the old LIMIT semantics. A/B on the CodeScope self index: edges 1,249 → 1,189, nodes/files unchanged, accuracy gate still P/R/F1 = 1.0. (Follow-up to the in-memory fuzzy rewrite.) +- **`StateBuilder::buildModuleSummaries` bound 9 parameters but the SQL has only 8 `?` placeholders** (`model/state_builder.cpp`): after the single-JOIN rewrite the bind loop still ran `i <= 9` with a "agg (5) + entry (3) + SELECT (1) = 9" comment; the 9th `sqlite3_bind_int64` hit a non-existent parameter (SQLITE_RANGE, silently ignored). The loop now binds exactly 8 (agg 4 + entry 3 + SELECT 1) and the comment is corrected. +- **Skills scripts indexed and queried different databases when `CODESCOPE_DB_PATH` was unset** (`skills/analyze.sh`, `skills/index.sh`): `codescope worker` takes the DB as a positional argument (default `/tmp/codescope_index.db`) while `codescope cli` resolves it from the `CODESCOPE_DB_PATH` env var (default `.codescope/codescope.db`). With the default environment the scripts silently reported stats from an empty/stale DB after indexing. Both scripts now `export CODESCOPE_DB_PATH="$DB"` so worker and cli share one database. + +### 📚 Docs & Skills + +- **Skills scripts updated to the current MCP tool set**: `index_project` and `get_hotspots` are no longer in `TOOL_HANDLERS`, so `skills/index.sh` and `skills/hotspots.sh` now use `codescope worker ` (serial) / `codescope index-parallel` (parallel) for indexing and `get_knowledge_graph` for hotspot-style density queries; `analyze.sh` and `skills/skills.md` (plus `docs/{en,zh}/skills.md`) were updated to match, with `CODESCOPE_DB_PATH` honored for the DB location. (`skills/*.sh`, `skills/skills.md`, `docs/en/skills.md`, `docs/zh/skills.md`) +- **Benchmark section re-measured after the fuzzy ordering fix** (`README.md`, `README.zh.md`): §7 now reflects `target/release/codescope` (v0.2.6) in `CODESCOPE_INDEX_MODE=normal` on the 2026-08-14 run — CodeScope self 0.95 s / 1,189 edges (pre-fix: 1,249 edges; nodes/files unchanged), tinygo 1.77 s / 4,485 edges, rustc 38.94 s / 117,284 edges; per-query MCP latency (median of 7), micro benchmarks, cross-file CALLS ratios, and the corrected `graph_query` figure (88.8 ms with LIMIT 100 — the previously published 0.03 ms was the error-path response of an invalid DSL, not a real query). + ## v0.2.5 (2026-08-05) Removes the LadybugDB (Kuzu) dependency entirely — SQLite is now the **sole graph store** on all platforms (CSR `adjacency`/`adjacency_rev` tables + C++ BFS). Graph queries, verifiers, and self-check tools behave identically to the LadybugDB-backed build, with faster indexing (rust: 52.6s → 31.6s, -40%; no graph-rebuild pass) and zero external runtime dependencies. Also fixes three self-check defects found while dogfooding the new backend. Restores the three capabilities that the Step 10 sprint had formally sunset (complexity metrics, n-gram semantic vector search, and metrics-driven readiness), hardens FunctionImplements verification with real call-chain + signature evidence, and fixes Go interface embedding (composition) dispatch. diff --git a/Cargo.lock b/Cargo.lock index 6a75929..ab062b5 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4,7 +4,7 @@ version = 4 [[package]] name = "codescope" -version = "0.2.5" +version = "0.2.6" dependencies = [ "libc", "once_cell", diff --git a/README.md b/README.md index fabdbb3..8928d14 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ It transforms source code into verifiable facts, understandable models, and inspectable evidence — enabling AI to validate claims against reality instead of hallucinating. -**Version**: v0.2.5 | **License**: Apache 2.0 +**Version**: v0.2.6 | **License**: Apache 2.0 --- @@ -436,56 +436,65 @@ Supported tables: `entity`, `relation`, `architecture_edge`, `module_edge`, `cap ## 7. Benchmark -All benchmarks measured on **Apple M3 Max (36 GB RAM)**. Other hardware will produce different results — expect slower performance on less capable machines. +All benchmarks measured on **Apple M3 Max (36 GB RAM, 14 cores), macOS, 2026-08-14**, using `target/release/codescope` (v0.2.6) in `worker` (serial full index) mode with the pure-SQLite graph backend (no LadybugDB/Kuzu dependency), `CODESCOPE_INDEX_MODE=normal`. Query latency measured via MCP server mode (engine initialized once, median of 7 runs). Numbers reflect the in-memory fuzzy resolver + ordering fix (edge count 1,249 → 1,189 vs. the pre-fix binary; nodes/files unchanged). Other hardware will produce different results — expect slower performance on less capable machines. ### Index Time -Measured with the pure-SQLite graph backend (no LadybugDB/Kuzu dependency). Latest run, 2026-08-08. - | Project | Language | Files | Nodes | Edges | Index Time | Peak RSS | |---------|----------|------:|------:|------:|-----------:|---------:| -| **CodeScope** (self) | C++/Rust | 221 | 1,481 | 1,204 | **1.2 s** | ~150 MB | -| **goagent** | Go | 1,344 | 26,425 | 6,352 | **16.0 s** | ~500 MB | -| **rustc** (Rust compiler, monorepo) | Rust | 6,029 | 129,918 | 118,154 | **31.6 s** | 5.9 GB | +| **CodeScope** (self) | C++/Rust | 197 | 1,509 | 1,189 | **0.95 s** | ~179 MB | +| **tinygo**¹ | Go | 812 | 21,557 | 4,485 | **1.77 s** | ~505 MB | +| **rustc** (Rust compiler, monorepo) | Rust | 5,575 | 130,410 | 117,284 | **38.94 s** | ~4.13 GB | -### Query Latency +¹ goagent 不在本机，以 tinygo（Go 编译器，812 文件）作为 Go 语言样本；rustc 源码树为本机 `~/code/rustc`（去重后的源码子集）。 -All graph queries run on the built-in SQLite graph-query backend (CSR adjacency tables), sub-millisecond for typical call-graph lookups. +### Query Latency -| Query | SQLite backend | -|-------|:--------------:| -| `get_graph_stats` | ~0.1 ms | -| `find_callers("buildGraph")` | ~0.2 ms (sub-ms) | -| `find_callees("parse")` | ~0.2 ms (sub-ms) | -| `graph_query` (full call-graph scan) | ~37 ms (full 118,154-edge scan, JOIN-optimized) | -| `shortest_path` | O(E) CSR BFS, sub-ms | -| `get_neighbors` | O(degree) CSR adjacency, sub-ms | -| `get_subgraph` | O(E) CSR BFS, sub-ms | +All graph queries run on the built-in SQLite graph-query backend (CSR adjacency tables), sub-millisecond for typical call-graph lookups. Measured on the CodeScope self-index DB unless noted. + +| Query | Measured (median) | +|-------|:-----------------:| +| `get_graph_stats` | 0.08 ms | +| `find_callers("parse")` | 0.16 ms | +| `find_callees("parse")` | 0.17 ms | +| `graph_query` (LIMIT 100) | 88.8 ms | +| `graph_query` (full scan, no LIMIT, self DB) | 88.9 ms | +| `graph_query` (full scan, rustc DB) | **>180 s** — always use `LIMIT` on large graphs | +| `shortest_path` | 0.09 ms (rustc DB: 0.38 ms) | +| `get_neighbors` | 0.06 ms (rustc DB: 2.36 ms) | +| `get_subgraph` | 0.10 ms (rustc DB: 5.68 ms) | +| `get_module_tree` | 0.08 ms | +| `get_entry_points` | 0.21 ms | +| `get_knowledge_graph` | 0.10 ms | ### Micro Benchmarks +Measured on the self-index DB via MCP server mode (engine initialized once). + | Metric | Value | |--------|-------| -| Engine init | **14.6 ms** | -| Index throughput | **1,533 KB/s** | -| Symbol definition query | **0.01–0.03 ms** | -| Callers/callees query | **0.01–0.02 ms** | -| 9 queries (total) | **0.17 ms** | -| Query latency (stdio MCP, includes process start) | **~60 ms** | +| Engine init (`GraphStore::open`) | **3 ms** | +| Index throughput (CodeScope self) | **~207 files/s** (197 files / 0.95 s) | +| Symbol query (`find_callers`/`find_callees`, MCP-level) | **0.16–0.17 ms** | +| `graph_query` (LIMIT 100) | **88.8 ms** | +| 10 queries (total, MCP-level) | **~89.9 ms** (dominated by `graph_query`) | +| Query latency (CLI, includes process start) | **10–17 ms** | ### Cross-File Resolution | Project | Cross-File CALLS | % of total CALLS | |---------|:---------------:|:----------------:| -| CodeScope (C++) | 588 | 46.7% | -| goagent (Go) | 2,930 | 53.0% | -| rustc (Rust) | 70,833 | 59.9% | +| CodeScope (C++) | 727 | 61.1% | +| tinygo (Go) | 2,210 | 49.3% | +| rustc (Rust) | 70,634 | 60.2% | ### Fast Scan (Lightweight, ms-level) -| Project | Time | Languages | Symbols | +`codescope discover`（文件发现 + 模块统计，纯 Rust，无引擎初始化）。 + +| Project | Time | Languages | Modules | |--------|:----:|:---------:|:-------:| -| **CodeScope** (self) | **32 ms** | cpp, rust, c | 2,902 | +| **CodeScope** (self) | **27 ms** | cpp, rust, c | engine 290 / server 20 | ### Token Savings @@ -539,9 +548,9 @@ Each script calls `codescope cli ''` internally. See `ski | Variable | Default | Description | |----------|---------|-------------| | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | -| `CODESCOPE_INDEX_MODE` | `standard` | Index mode: `fast` / `standard` / `strict` | +| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `strict` | | `CODESCOPE_EXCLUDE_PATHS` | (unset) | Comma-separated glob patterns to exclude | -| `CODESCOPE_WORKERS` | `4` | Total parse-worker cores for `index-parallel` | +| `CODESCOPE_WORKERS` | `min(hw,8)` | Total parse-worker cores for `index-parallel` (`kDefaultParseWorkers=8`) | | `CODESCOPE_WORKER_TIMEOUT` | `300` | Worker subprocess timeout in seconds | | `CODESCOPE_MAX_FILE_SIZE` | (unset) | Max source file size to index in bytes | | `CODESCOPE_MMAP_SIZE` | 256 MB | SQLite `mmap_size` pragma value | @@ -556,4 +565,4 @@ Each script calls `codescope cli ''` internally. See `ski Apache 2.0 — see [LICENSE](LICENSE). -**CodeScope v0.2.5** — Built with Rust 2024 + C++23 + tree-sitter + SQLite. \ No newline at end of file +**CodeScope v0.2.6** — Built with Rust 2024 + C++23 + tree-sitter + SQLite. \ No newline at end of file diff --git a/README.zh.md b/README.zh.md index 17cea9a..a79756d 100644 --- a/README.zh.md +++ b/README.zh.md @@ -4,7 +4,7 @@ 它将源代码转化为可验证的事实、可理解的模型和可检查的证据 — 让 AI 能够根据现实验证断言，而非凭空编造。 -**版本**: v0.2.4 | **许可证**: Apache 2.0 +**版本**: v0.2.6 | **许可证**: Apache 2.0 --- @@ -417,56 +417,65 @@ get_knowledge_graph {"table":"capability","limit":10} ## 7. 性能基准 -所有基准测试在 **Apple M3 Max（36 GB RAM）** 上测得。其他硬件会产生不同的结果 — 性能较低的机器上预期会慢一些。 +所有基准测试在 **Apple M3 Max（36 GB RAM，14 核），macOS，2026-08-14** 上测得，使用 `target/release/codescope`（v0.2.6）`worker`（串行全量索引）模式 + 纯 SQLite 图后端（无 LadybugDB/Kuzu 依赖），`CODESCOPE_INDEX_MODE=normal`。查询延迟通过 MCP server 模式测得（引擎只初始化一次，7 次中位数）。数据反映内存化 fuzzy 解析器 + 排序修复后的状态（边数相对修复前二进制 1,249 → 1,189；节点/文件数不变）。其他硬件会产生不同的结果 — 性能较低的机器上预期会慢一些。 ### 索引时间 -基于纯 SQLite 图后端实测（无 LadybugDB/Kuzu 依赖）。最新一次运行：2026-08-08。 - | 项目 | 语言 | 文件数 | 节点数 | 边数 | 索引时间 | 峰值内存 | |------|------|------:|------:|------:|---------:|---------:| -| **CodeScope**（自身） | C++/Rust | 221 | 1,481 | 1,204 | **1.2 秒** | ~150 MB | -| **goagent** | Go | 1,344 | 26,425 | 6,352 | **16.0 秒** | ~500 MB | -| **rustc**（Rust 编译器，monorepo） | Rust | 6,029 | 129,918 | 118,154 | **31.6 秒** | 5.9 GB | +| **CodeScope**（自身） | C++/Rust | 197 | 1,509 | 1,189 | **0.95 秒** | ~179 MB | +| **tinygo**¹ | Go | 812 | 21,557 | 4,485 | **1.77 秒** | ~505 MB | +| **rustc**（Rust 编译器，monorepo） | Rust | 5,575 | 130,410 | 117,284 | **38.94 秒** | ~4.13 GB | -### 查询延迟（SQLite 图查询后端） +¹ goagent 不在本机，以 tinygo（Go 编译器，812 文件）作为 Go 语言样本；rustc 源码树为本机 `~/code/rustc`（去重后的源码子集）。 -所有图查询均基于内置 SQLite 图查询后端（CSR 邻接表），典型调用图查询为亚毫秒级。 +### 查询延迟（SQLite 图查询后端） -| 查询 | 延迟 | 说明 | -|------|:----:|------| -| `get_graph_stats` | ~0.1 ms | SQL 聚合 | -| `find_callers("buildGraph")` | ~0.2 ms | 名称过滤（sub-ms） | -| `find_callees("parse")` | ~0.2 ms | 名称过滤（sub-ms） | -| `graph_query`（LIMIT 100） | ~37 ms | 118,154 条边，JOIN 优化全扫描 | -| `shortest_path` | sub-ms | O(E) CSR BFS | -| `get_neighbors` | sub-ms | O(degree) CSR 邻接 | -| `get_subgraph` | sub-ms | O(E) CSR BFS | +所有图查询均基于内置 SQLite 图查询后端（CSR 邻接表），典型调用图查询为亚毫秒级。除注明外均基于 CodeScope 自身索引库测得。 + +| 查询 | 实测延迟（中位数） | +|------|:-----------------:| +| `get_graph_stats` | 0.08 ms | +| `find_callers("parse")` | 0.16 ms | +| `find_callees("parse")` | 0.17 ms | +| `graph_query`（LIMIT 100） | 88.8 ms | +| `graph_query`（无 LIMIT 全量扫描，self 库） | 88.9 ms | +| `graph_query`（无 LIMIT 全量扫描，rustc 库） | **>180 秒** — 大图务必使用 `LIMIT` | +| `shortest_path` | 0.09 ms（rustc 库：0.38 ms） | +| `get_neighbors` | 0.06 ms（rustc 库：2.36 ms） | +| `get_subgraph` | 0.10 ms（rustc 库：5.68 ms） | +| `get_module_tree` | 0.08 ms | +| `get_entry_points` | 0.21 ms | +| `get_knowledge_graph` | 0.10 ms | ### 微基准 +基于自身索引库通过 MCP server 模式测得（引擎只初始化一次）。 + | 指标 | 值 | |------|----| -| 引擎初始化 | **14.6 ms** | -| 索引吞吐量 | **1,533 KB/s** | -| 符号定义查询 | **0.01–0.03 ms** | -| 调用者/被调用者查询 | **0.01–0.02 ms** | -| 9 次查询（总计） | **0.17 ms** | -| 查询延迟（stdio MCP，含进程启动） | **~60 ms** | +| 引擎初始化（`GraphStore::open`） | **3 ms** | +| 索引吞吐量（CodeScope 自身） | **~207 文件/秒**（197 文件 / 0.95 秒） | +| 符号查询（`find_callers`/`find_callees`，MCP 级） | **0.16–0.17 ms** | +| `graph_query`（LIMIT 100） | **88.8 ms** | +| 10 次查询（总计，MCP 级） | **~89.9 ms**（主要由 `graph_query` 贡献） | +| 查询延迟（CLI，含进程启动） | **10–17 ms** | ### 跨文件解析 | 项目 | 跨文件 CALLS | 占 CALLS 总数百分比 | |------|:-----------:|:------------------:| -| CodeScope（C++） | 588 | 46.7% | -| goagent（Go） | 2,930 | 53.0% | -| rustc（Rust） | 70,833 | 59.9% | +| CodeScope（C++） | 727 | 61.1% | +| tinygo（Go） | 2,210 | 49.3% | +| rustc（Rust） | 70,634 | 60.2% | ### 快速扫描（轻量，毫秒级） -| 项目 | 时间 | 语言 | 符号数 | +`codescope discover`（文件发现 + 模块统计，纯 Rust，无引擎初始化）。 + +| 项目 | 时间 | 语言 | 模块数 | |------|:----:|:----:|:------:| -| **CodeScope**（自身） | **32 ms** | cpp, rust, c | 2,902 | +| **CodeScope**（自身） | **27 ms** | cpp, rust, c | engine 290 / server 20 | ### Token 节省 @@ -537,4 +546,4 @@ cd CodeScope Apache 2.0 — 详见 [LICENSE](LICENSE)。 -**CodeScope v0.2.5** — 使用 Rust 2024 + C++23 + tree-sitter + SQLite 构建。 \ No newline at end of file +**CodeScope v0.2.6** — 使用 Rust 2024 + C++23 + tree-sitter + SQLite 构建。 \ No newline at end of file diff --git a/RELEASE.md b/RELEASE.md index 254c20f..86d47cf 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,62 +1,34 @@ -## v0.2.5 (2026-08-10) - -Removes the LadybugDB (Kuzu) dependency entirely — SQLite is now the **sole graph store** on all platforms (CSR `adjacency`/`adjacency_rev` tables + in-memory BFS). Graph queries, verifiers, and self-check tools behave identically to the LadybugDB-backed build, indexing is **~35-40% faster** (rustc: 52.6s → 31.6s, no graph-rebuild pass), and there are zero external runtime dependencies. Also fixes 13 defects found during dogfooding the new backend (C1/C2/P2/P3/H1/H2/M1/M2/M3/L1-L3), including a CSR rebuild that no longer loses entities on incremental re-index, correct adjacency id remapping after parallel merge, SQL-injection-free file filters, and content-hash-based incremental change detection. Restores the three capabilities formally sunset in the Step 10 sprint (complexity metrics, n-gram semantic vector search, and real metrics/embedding readiness), hardens `FunctionImplements` verification with call-chain + signature evidence, fixes Go interface embedding (composition) dispatch, and delivers a **full SQLite graph-query backend so Windows (and any SQLite-only build) gets every graph MCP tool working** instead of erroring. - -### What changed - -| Area | Before (v0.2.4) | After (v0.2.5) | -|------|-----------------|----------------| -| **Complexity metrics** | Sunset — `resolveStagedMetrics` was a no-op, no metrics stored, `engine_get_complexity` returned `{"complexity":null,"unavailable":true}` | Restored — metrics computed in the parse worker, staged in `_staged_metrics`, resolved onto `entity`; `get_complexity` returns real `cyclomatic`/`cognitive`/`nesting_depth` | -| **Semantic search** | Sunset — `buildVectorsFromGraph` was a no-op, `node_vectors` always empty, `engine_search_semantic` was a dead stub, `unified_search` was FTS-only | Restored — 192-dim n-gram hash vectors (no external model), `searchSemanticJson` cosine ranking with an **accuracy floor** (score ≥ 0.3, so noise is rejected), **TF-IDF identifier weighting** (rare tokens dominate), `engine_search_semantic` wired to the real implementation, appended to FTS when results run short | -| **Metrics/embedding readiness** | Structurally `0` — `metrics_ready` hardcoded, `metrics`/`semantic_search` capabilities `available:false` + `unavailable_reason:"sunset"` | Real — derived from resolved `entity` cyclomatic count and `node_vectors` rows; capabilities `available:true` with coverage + producer versions | -| **`FunctionImplements` verification** | Structural only — any function with a call edge got `Supported` even for a wrong object | Signature + call-chain matching — object entities found and subject→object call edges checked; higher confidence when linked | -| **Go interface embedding** | `interface A { B; foo() }` ignored B's methods — structs implementing B weren't matched to A | Transitive closure of embedded interface methods used in the subset check | -| **Re-index vector self-heal** | No-change re-index never rebuilt `node_vectors` (external truncation left semantic search empty) | DEEP re-index re-runs `buildVectorsFromGraph` (idempotent), `vector_ready` still derived from actual row count | -| **Windows / SQLite-only graph queries** | All graph MCP tools returned `"LadybugDB not compiled"` on Windows (and the Windows build had 9 latent compile errors) | **Every graph tool** (find_definition, find_references, get_callers, get_callees, get_neighbors, find_shortest_path, get_subgraph, get_graph_stats, get_hotspots, get_entry_points, trace_path, explore_function, graph_query, impact_analysis, detect_ffi_boundaries) now has a **SQLite implementation** reusing the CSR adjacency tables (O(E) BFS) and `entity`/`relation`, with the same JSON schema as LadybugDB. macOS/Linux keep LadybugDB. `-DCODESCOPE_SQLITE_ONLY=ON` builds the Windows configuration on any host | -| **Graph storage** | LadybugDB (Kuzu) embedded DB with a SQLite fallback | **SQLite only** — LadybugDB build wiring, `store_ladybug_core.cpp`, `store_graph_compiler.*`, `engine_rebuild_ladybug_graph(s)` FFI and the scheduler's rebuild pass are all deleted; `HAS_LADYBUG` is never defined. CSR adjacency is built directly from `relation(type=1)` inside a nested-safe SAVEPOINT | -| **Incremental re-index** | entity.id conflicts could silently drop entities | `id_offset = MAX(entity.id)` shift; verified goagent 26,243 → 26,425 entities (no loss) | -| **Parallel merge** | adjacency BLOBs kept local worker ids after merge | Workers defer CSR (`CODESCOPE_DEFER_CSR=1`) and `engine_rebuild_csr` remaps/rebuilds after every merge | -| **Incremental change detection** | mtime+size only — same-size same-mtime edits missed | Two-level gate: mtime\|size fast screen, then content hash (FNV-1a) confirm | -| **file_filter queries** | SQL string splicing (injection risk) | Parameterized bindings in findDefinition/findReferences | -| **Self-check tools** | verify_integrity emitted invalid JSON; capability_drift silently returned 0 on empty input; orphan counts truncated at 30 | Valid JSON; `"status":"no_capabilities_declared"`; orphan limit 500 with separate `orphans` counter (trust_score no longer collapses to 0) | - -### Upgrade notes - -- **No breaking API changes**: all MCP tool responses keep the same JSON schema. The `metrics` and `semantic_search` capability blocks change from `available:false, unavailable_reason:"sunset"` to `available:true, ready:`. Windows graph-tool responses now return real data (same schema as LadybugDB) instead of `"LadybugDB not compiled"`. -- **`get_complexity` now returns real numbers** where it previously returned `{"complexity":null}` — MCP clients that read `complexity` as a number now get an integer. -- **Existing databases**: a `metrics_ready` column is auto-added to `project_readiness` and metrics columns to `entity` on open (migration); a fresh index/enhance run populates them. -- **Semantic search is n-gram lexical similarity** with TF-IDF identifier weighting — not meaning-based embedding — it complements FTS exact/prefix matching and requires no model files or network. -- **Windows**: LadybugDB has no official Windows library (only a CLI), so Windows builds SQLite-only by default. All graph queries work via the SQLite backend with performance in the sub-millisecond-to-tens-of-milliseconds range (CSR adjacency gives O(E) BFS). Developers can build the same configuration on macOS/Linux with `-DCODESCOPE_SQLITE_ONLY=ON`. +## v0.2.6 (2026-08-14) + +Speeds up full (non-fast) indexing end-to-end and fixes a resolver JOIN defect that both slowed indexing and silently over-matched cross-file references. Fuzzy symbol search is now fully in-memory (no per-entity SQL), FAST-mode pruning rules are completed, and discovery timing is quantified for the first time — with zero precision loss across every benchmark (accuracy gate stays P/R/F1 = 1.0). + +### Performance & Results + +- **Fuzzy search fully in-memory** (`fuzzy_resolver.{h,cpp}`, `pipeline.cpp`): the resolver previously ran up to 3 SQL queries per unresolved ref (case-insensitive + prefix + suffix) and hydrated every fuzzy hit with a per-id SQL lookup. All entities are now loaded once into memory (ASCII-fold exact index + `sqliteLikeMatch`, byte-identical to SQLite LIKE) and hits are copied from an `entity_by_id` map — no SQL in the hot loop. **CodeScope self-index (215 files): `resolver::run` 298ms → 30ms (10.2x), `buildGraph` 332ms → 62ms, index-parallel 686ms → 517ms (-25%)** with identical resolved refs/edges on same-input A/B. +- **Bigger projects gain resolution, not just speed**: the old 500ms fuzzy budget silently dropped queries on large repos; the in-memory path never trips it. **goagent (1,374 files)**: +86 refs resolved, +64 call edges; **rustc (6,029 files)**: fuzzy hits 32 → 918, +283 refs, +223 edges — at lower wall time in every case. +- **Fuzzy prefix/suffix lookup O(N) → O(log N)** via sorted folded-name / reversed-folded-name indexes (`std::lower_bound`); wildcard queries keep the exact SQLite-LIKE path. +- **`buildModuleSummaries` merged its two `relation` LEFT JOINs into one**: rustc 117k-relation × 129k-entity phase drops ~5.95s → **~0.25s (23.8x)**, result-identical. +- **`idx_scope_kind_name(project_id, kind, name)` index**: turns the `scope` full-table scan (129,893 entities × 26,975 scopes per row) into an index seek; the `import.source_scope_id` UPDATE result is now checked (was silently ignored, leaving imports at scope 0). +- **Dead `Literal`/`Variable` rows dropped at the DB write path** (`store_batch.cpp`): `semantic_records` 95,944 → 25,146 rows, SQLite flush ~3.7x faster, full-index time 2.18s → ~1.36s (**-38%**). In-memory GraphBuilder Variable nodes stay intact. +- **Resolver self-join missing the `file_path` term fixed** (`pipeline.cpp`): since `original_id` is per-file, the `global_var_types_`/`global_struct_fields_` preloads cross-matched same-id parents across files (~35x join inflation). Adding `file_path` cut resolver 11.2s → 5.4s and the goagent full index 20.6s → 14.6s (**-29%**) while resolving more refs correctly. +- **More full-index wins**: `buildModuleSummaries` split into two CTEs (17.1s → 174ms on goagent, enhance_project -94%), `idx_sr_proj_file_oid(project_id, file_path, original_id)` composite index (resolver type self-joins 3.3s → 63ms on rustc), parse workers 4 → 8 default (wall-clock -18% on rustc, `CODESCOPE_WORKERS` still overrides), and the per-reference candidate deep-copy eliminated (`resolve_loop` -16%). +- **FAST mode is no longer "NORMAL with a different name"**: `fast_extra_skip_dirs_` (was empty) now skips 11 build/test-artifact dirs (`.output`, `storybook-static`, `__generated__`, `playwright-report`, `test-results`, `allure-results`, `allure-report`, `.sass-cache`, `.scss-cache`, `logs`, `.logs`) plus 4 exact files via `fast_extra_filenames_` (`.eslintcache`, `.stylelintcache`, `.prettiercache`, `tsconfig.tsbuildinfo`) — synthetic A/B drops candidates 4 → 1. Discovery is now timed (`discovery=`, `seen_dirs` counts directories only): rustc 143ms / 4,650 dirs, goagent 25ms / 845 dirs. +- **`get_graph` migrated to canonical tables**: it read the deprecated, empty `graph_nodes`/`graph_edges` (nodes always 0, edges stale) — now pages `entity`+`relation` with the same public JSON schema; `get_graph` matches `get_graph_stats` (`total_nodes:39686, total_edges:4415`). ### Bug fixes -| # | Bug | Root cause | Fix | -|---|-----|------------|-----| -| 1 | `buildVectorsFromGraph` wrote nothing | Queried nonexistent `entity.node_id` (canonical column is `id`) | Use `entity.id`; vectors now populate | -| 2 | `get_complexity` returned sunset marker | Producer was no-op; `getComplexityJson` had a hardcoded unavailable response | Restore producer + read real `entity` metrics | -| 3 | `metrics_ready` never persisted | Field not in `setProjectReadiness`/`getProjectReadiness` whitelist | Add `metrics_ready` to whitelist + `project_readiness` column/migration | -| 4 | `FunctionImplements` false "Supported" for wrong object | Structural check only | Add object-entity matching + subject→object call-chain check | -| 5 | `getModuleMap` failed on `graph_nodes` | Referenced deprecated table + nonexistent `cyclomatic` column | Migrate to `entity` with real metrics | -| 6 | No-change re-index left semantic search empty | `buildVectorsFromGraph` skipped on the no-op path | Re-run it in DEEP mode (idempotent) | -| 8 | `engine_search_semantic` always errored | Dead Phase-0 stub returned "not implemented" | Route to the restored `searchSemanticJson` | -| 9 | Semantic search could pollute results | `dot > 0` floor let weak/incidental matches through | Accuracy floor `kSemanticScoreFloor = 0.3` rejects noise | -| 10 | `getHotspots`/`getEntryPoints` reported metrics as sunset | LadybugDB branches emitted `complexity:null, unavailable_reason:"sunset"` | Batch-read real `cyclomatic`/`cognitive`/`nesting_depth` from `entity` | -| 11 | Windows build had 9 compile errors | `detectBareNameAmbiguity` used `lbug_*` types without an `#ifdef HAS_LADYBUG` guard | Guard the helper with `#ifdef HAS_LADYBUG`; add `-DCODESCOPE_SQLITE_ONLY=ON` to build/test the Windows config on any host | -| 12 | All graph tools errored on Windows | Query layer hard-routed to LadybugDB with no SQLite path | Implement the full SQLite graph-query backend (CSR adjacency + `entity`/`relation`), same JSON schema | -| 13 | `graph_query` full-graph scan did per-edge lookups | Each edge ran 2 `readEntity` queries (N+1) | JOIN `entity` in one query — full call-graph scan dropped to ~37ms on the engine source | -| 7 | Go interface embedding dropped embedded methods | Method-set check used direct methods only | Expand to transitive closure of embedded interfaces | -| 14 | Windows cross-compile reused host cmake cache | `build.rs` always used `build-release/`, leaking macOS `-arch arm64` into MinGW | Per-target build dir (`build-release-`) when cross-compiling | -| 15 | `go_visitor.cpp` failed to compile on MinGW | Missing `` for `std::find`/`std::sort` (Clang compiled via indirect include) | Add the `` include | -| C1 | Incremental rebuild could drop entities on entity.id collision | New ids collided with existing `entity.id` | Shift new ids by `MAX(entity.id)`; verified goagent 26,243 → 26,425 entities (no loss) | -| C2 | Merged adjacency BLOBs kept local worker ids | Workers built CSR from local entity ids before merge | Defer CSR in workers (`CODESCOPE_DEFER_CSR=1`); `engine_rebuild_csr` remaps/rebuilds after every merge | -| P2 | buildCSR failure rolled back the whole graph | Savepoint rollback discarded entity/relation rows while callers ignored the return | Log-and-continue with relation-scan fallback; resolver failures propagate to callers | -| P3 | Chunk workers built local-id CSR; `CODESCOPE_DEFER_CSR=0` ignored | env not set in chunk path; presence-check treated `"0"` as set | Set env in both chunk branches; treat `"0"` as unset (matches `CODESCOPE_SKIP_ASYNC`) | -| H1/H2 | Missing FFI declarations; merge missed document/parse_failures tables | engine.h lacked 10 decls; merge specs omitted two tables | Add declarations; extend TABLE_SPECS/SCHEMA_TABLES/OFFSET_TABLES | -| M1-M3 | take_string NULL → empty string; stale-file misses; SQL injection | NULL deref; mtime+size gate missed same-size edits; file_filter spliced into SQL | Valid error JSON; content-hash gate (FNV-1a); parameterized bindings | -| L1-L3 | Schema comment, JSON escaping, FFI labeling | graph_nodes→entity.id stale comment; column names unescaped; 20 FFI unlabeled | Fixed/escaped/labeled as CLI/extension-only | - -### Full changelog - -See [CHANGELOG.md](./CHANGELOG.md) for the complete list of changes. - ---- \ No newline at end of file +- `FilterPolicy::setMode()` never rebuilt the active skip sets (FAST rules were inert when the mode was set after construction). +- Discovery `seen_dirs` counted files too (inflated ~44k for a 215-file project) — now directories only. +- `import.source_scope_id` UPDATE failures were silently swallowed, leaving imports at scope 0 — now checked and logged. +- Skills scripts/docs updated to the current MCP tool set (`index_project`/`get_hotspots` removed from `TOOL_HANDLERS`; indexing now via `codescope worker` / `index-parallel`, hotspots via `get_knowledge_graph`). +- **Fuzzy prefix/suffix binary-search results truncated in name order instead of rowid order** (`fuzzy_resolver.cpp`): when a prefix/suffix query matched more than `kFuzzyCandidateLimit` (5) entities, the retained subset differed from the old SQL `LIKE ... LIMIT ?` path, so the resolved CALLS edge could diverge on large projects — contradicting the in-memory rewrite's byte-identical contract. Both paths now collect all matches, sort by id (= rowid = load order), then truncate. A/B on CodeScope self index: edges 1,249 → 1,189, nodes/files unchanged, accuracy gate still P/R/F1 = 1.0. +- **`buildModuleSummaries` bound 9 parameters for 8 `?` placeholders** (`state_builder.cpp`): the extra bind hit a non-existent parameter (SQLITE_RANGE, silently ignored); the loop now binds exactly 8 and the comment is corrected. +- **`skills/analyze.sh` / `skills/index.sh` indexed and queried different DBs** when `CODESCOPE_DB_PATH` was unset: `worker` took the DB as a positional argument (default `/tmp/codescope_index.db`) while `cli` read the env var (default `.codescope/codescope.db`), so stats were silently reported from an empty/stale DB. Both scripts now `export CODESCOPE_DB_PATH="$DB"` so worker and cli share one database. + +### Documentation + +- **Benchmark section re-measured after the fuzzy ordering fix** (`README.md`, `README.zh.md`): §7 now reflects `target/release/codescope` (v0.2.6) in `CODESCOPE_INDEX_MODE=normal` on the 2026-08-14 run — CodeScope self 0.95 s / 1,189 edges (pre-fix: 1,249; nodes/files unchanged), tinygo 1.77 s / 4,485 edges, rustc 38.94 s / 117,284 edges; per-query MCP latency (median of 7), micro benchmarks, cross-file CALLS ratios, and the corrected `graph_query` figure (88.8 ms with LIMIT 100 — the previously published 0.03 ms was the error-path response of an invalid DSL, not a real query). + +### Verification + +All C++ engine tests pass (including `test_call_graph_accuracy`, `test_metrics_readiness`, `test_step11_go_smoke`), accuracy gate P/R/F1 = 1.0 with FP/FN injection correctly rejected, Rust server tests 88/88, clippy + clang-format clean. See [CHANGELOG.md](./CHANGELOG.md) for the complete list. diff --git a/docs/en/skills.md b/docs/en/skills.md index 4363d8a..9a13202 100644 --- a/docs/en/skills.md +++ b/docs/en/skills.md @@ -10,7 +10,8 @@ CodeScope is a **Project Truth Engine**. It transforms source code into verifiab ```bash # One-liner: index a project and query stats -codescope cli index_project '{"project_path":"/path/to/project"}' +# (MCP tool `index_project` removed — use the `worker` CLI, or `index-parallel` for parallelism) +codescope worker /tmp/codescope.db /path/to/project "" project 0 codescope cli get_graph_stats '{}' ``` @@ -22,7 +23,7 @@ codescope cli get_graph_stats '{}' set -e PROJECT=$1 echo "=== Indexing $PROJECT ===" -codescope cli index_project "{\"project_path\":\"$PROJECT\"}" +codescope worker /tmp/codescope.db "$PROJECT" "" project 0 echo "=== Stats ===" codescope cli get_graph_stats '{}' echo "=== Entry Points ===" @@ -106,19 +107,19 @@ graph TB flowchart LR PA["Phase A (ms-level)
scan_project"] -->|"modules + symbols + entry_points"| Ready["✓ AI ready"] PB["Phase B (async)
enhance_project"] -->|"call graph + complexity + FTS"| Enhanced - PC["Phase C (on-demand)
index_project"] -->|"full parse → all tables"| Full + PC["Phase C (on-demand)
worker / index-parallel"] -->|"full parse → all tables"| Full Ready -.->|triggers| PB ``` --- -## 3. MCP Tools (19 tools) +## 3. MCP Tools (38 tools) ### Index (索引) | Tool | Purpose | Input | Output | Token | |------|---------|-------|--------|-------| -| `index_project` | Full project index | `project_path`, `language` | JSON: files_indexed, timing | ~50 | +| `worker` (CLI) | Full project index (serial) | ` ` | JSON: files_indexed, timing | ~50 | | `index_file` | Index a single file | `file_path` | JSON: file result | ~30 | | `search` | Unified search (FTS) | `query`, `limit?` | JSON: matched results | ~30 | @@ -194,7 +195,10 @@ AI Q&A → codescope_build_context (200-1000 tok) | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | | `GRAMMARS_DIR` | `grammars/` | tree-sitter grammar .so directory | | `CODESCOPE_LSP` | (unset) | LSP server command for type enhancement | -| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `deep` | +| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `strict`（见下方 Index Modes） | +| `CODESCOPE_WORKERS` | `min(hw,4)` | Parse worker thread count | +| `CODESCOPE_SKIP_ASYNC` | (unset) | Set to 1 to skip async model/state/FTS stages (set by the parallel scheduler path) | +| `CODESCOPE_PROFILE_RESOLVER` | (unset) | Enable resolver per-phase timing (`[module=resolver, method=run]` breakdown) | | `CODESCOPE_VERBOSE` | `1` | Set to `0` to suppress batch logging | | `CODESCOPE_MAX_FILE_SIZE` | `5242880` (5MB) | Max file size in bytes to index | | `CODESCOPE_MEMORY_BUDGET_MB` | `0` (unlimited) | Pause parsing when RSS exceeds budget | @@ -205,13 +209,15 @@ AI Q&A → codescope_build_context (200-1000 tok) ## 6. Index Modes -Set via `CODESCOPE_INDEX_MODE`: +Set via `CODESCOPE_INDEX_MODE`（filter_policy.h enum NORMAL/FAST/STRICT — there is NO `deep` mode）: -| Mode | FTS | Vectors | TTFA | Use Case | -|------|-----|---------|------|----------| -| `fast` | ❌ | ❌ | **Fastest** | Quick answers, minimal indexing | -| `normal` | ✅ | ❌ | Normal | Default — graph + search | -| `deep` | ✅ | ✅ | Slower | Full analysis including semantic vectors | +| Mode | Enum | Extra pruning | FTS | Use Case | +|------|------|--------------|-----|----------| +| `fast` | `FAST` | ✅ extra 12 dir kinds (logs/.output/storybook-static/playwright-report/test-results/allure-*/.sass-cache/.scss-cache/__generated__) + 4 cache files (`.eslintcache`/`.stylelintcache`/`.prettiercache`/`tsconfig.tsbuildinfo`) | ❌ skipped | Fastest; data ≈ normal (near-identical on clean-source projects) | +| `normal` | `NORMAL` | base skip table only (build/dist/out/target/test/docs/vendor/node_modules/.venv ...) | ✅ | Default | +| `strict` | `STRICT` | base skip + detectLanguage whitelist gate (only source files of recognized languages) | ✅ | Most restrictive, leanest data | + +> Known issue (fixed 2026-08-11): `fast` used to be ≈ `normal` — `fast_extra_skip_dirs_` was an empty reserved set and `setMode()` did not rebuild `active_skip_dirs_`. See `docs/optimization/perf-full-index-2026-08-11.md` §9/§10. ## 7. Performance Benchmarks diff --git a/docs/optimization/perf-full-index-2026-08-11.md b/docs/optimization/perf-full-index-2026-08-11.md new file mode 100644 index 0000000..414b955 --- /dev/null +++ b/docs/optimization/perf-full-index-2026-08-11.md @@ -0,0 +1,496 @@ +# CodeScope 全量索引性能统计报告（2026-08-11） + +> **运行日期**: 2026-08-11 +> **平台**: macOS（本机） +> **被测对象**: CodeScope 自身（engine C++ + server Rust），`target/release/codescope`（8/10 构建） +> **索引项目**: `/Users/scc/code/cppCode/CodeScope`（engine/ + server/，215 个源码文件） +> **采集方式**: 并行 `index-parallel` + 串行 `worker`（含异步阶段）各跑一次全量索引，抓取引擎 `[module=engine, method=...]` 计时埋点 + +--- + +## 1. 总览 + +| 指标 | 并行索引 (index-parallel) | 串行全量索引 (worker) | +|---|---|---| +| 总耗时 | **686ms** | 解析+建图 ~497ms，异步 +75ms | +| 索引文件数 | 215（server 17 + engine 198） | 215 | +| 总节点 / 总边 | 1479 / 1204 | 1479 / 1193 | +| 并行 worker 数 | 8（2 模块 × 动态分配） | 14（单进程线程池） | +| DB 合并 | merge_module_dbs 58ms | 无（单库直写） | +| 异步阶段（模型/FTS/知识图谱） | 跳过（CODESCOPE_SKIP_ASYNC=1） | 完整执行（75ms） | + +--- + +## 2. 每阶段 / 每方法耗时明细 + +### 2.1 引擎初始化 + +| 方法 | 耗时 | +|---|---| +| `engine_init` → GraphStore::open | 11–17ms | + +### 2.2 文件发现 (discovery) + +| 指标 | 数值 | +|---|---| +| seen_dirs（遍历目录数） | **44445** | +| seen_files（命中文件数） | 518 | +| skipped_dirs / skipped_files | 2625 / **41075** | +| candidate_files（最终候选） | 215 | + +> 注意：遍历了 4.4 万目录、跳过 4.1 万文件，只产出 215 个候选——**遍历/过滤量是文件量的 190 倍**，是隐藏成本。 + +### 2.3 解析 + 批量写入 + +| 方法 | 并行(server/engine) | 串行 | +|---|---|---| +| `membulk_flush` | 25ms / 127ms | **148ms** (files=215) | + +### 2.4 建图阶段 buildGraph（串行：**332ms 总量**） + +| buildGraph 子阶段 | 串行耗时 | 并行(engine 模块) | +|---|---|---| +| file_list | 0ms | 0ms | +| delete | 3ms | 3ms | +| rf / r2n | 0 / 2ms | 0 / 1ms | +| nodes / edges | 3 / 0ms | 3 / 0ms | +| route / type_edges / type_info / type_ref | 0ms | 0ms | +| ref / import / scope / ent_rel | 14 / 1 / 4 / 0ms | 13 / 1 / 3 / 0ms | +| **resolver** | **298ms (90%)** | 125ms | +| csr / cleanup | 0 / 0ms | 0 / 0ms | +| **total** | **332ms** | 153ms | + +### 2.5 resolver 明细（串行，最大热点） + +``` +[module=resolver, method=run] resolved 2089 / 11143 refs + exact=4461 fuzzy=928 + skipped_miss=2323 skipped_fuzzy_no_ev=3196 skipped_ambiguous=1333 + skipped_lang=749 skipped_common=3 + avg_cands=1.4 entities=1479 imports=435 + sql_batch=1ms total=298ms +``` + +- 11143 个引用只解析出 2089 个（19% 解析率） +- **3196 个 fuzzy 查询命中后无证据被丢弃**——大量 SQL 查询白做 +- `FuzzyResolver` 每个未命中 ref 最多执行 **3 次 SQL**（case-insensitive + prefix + suffix，每次 `reset+bind+step`） + +### 2.6 后处理 + +| 方法 | 串行 | 并行 | +|---|---|---| +| `resolveStagedMetrics` | 1ms (1169 行) | 0–1ms | +| `createIndexesAfterBulkLoad` | 2ms | 0–1ms | +| `engine_index_post_parse` total | 335ms | 38–156ms | +| buildCSR | 607 forward / 432 reverse groups | 418+290 / 190+143 | + +### 2.7 异步阶段（串行完整跑） + +| 方法 | 耗时 | +|---|---| +| `runModelIndexSync`（model / **state** / fts） | 5 / **61** / 8ms，共 75ms | +| `populateModulesHierarchy` | 2ms (27 行) | +| `buildKnowledgeGraphSync` | 2ms (38 行) | + +### 2.8 并行合并（仅 index-parallel） + +| 子阶段 | 耗时 | +|---|---| +| checkpoint | 18ms | +| schema_read / columns / sql_build | 3 / 3 / 0ms | +| sqlite_exec | 32ms | +| merge_module_dbs **total** | **58ms**（47164 行，14 表） | + +--- + +## 3. 热点排序（按串行全量索引 ~570ms） + +| 排名 | 方法/阶段 | 耗时 | 占比 | +|---|---|---|---| +| 🥇 | **resolver::run** | 298ms | **~52%** | +| 🥈 | **membulk_flush**（解析写库） | 148ms | ~26% | +| 🥉 | 异步 state builder | 61ms | ~11% | +| 4 | discovery 目录遍历（未单独计时） | 隐性 | — | +| 5 | merge_module_dbs（并行模式） | 58ms | — | + +--- + +## 4. 优化建议 + +### P0-1：resolver 热点——fuzzy 匹配从「每 ref 3 次 SQL」改为「内存索引」⭐ 预期收益最大 + +**依据**（`engine/src/resolver/fuzzy_resolver.cpp` + `pipeline.cpp`）： +- `ResolverPipeline::run()` 的 Step 0 已经把全部 entity 预加载进内存 `entity_index`（`"SELECT id, name, file_path, language, arity, kind, qualified_name ..."`），主路径不再走 SQL； +- 但 `FuzzyResolver` 是独立的旧实现，仍对每个未命中 ref 执行最多 3 条 SQL（`LOWER(name)=LOWER(?)` / `name LIKE ?||'%'` / `name LIKE '%'||?`，每条都 `prepare` 一次 + 循环 `step`）； +- 串行跑 11143 refs，fuzzy 阶段被 `skipped_fuzzy_no_ev=3196` 丢弃——大量查询无产出。 + +**建议**： +1. 把 fuzzy 三种匹配统一搬到内存：利用已有 `entity_index`，对未命中 ref 做内存级 `LOWER(name)` 精确/前缀/后缀查找，消除 3×N 次 SQL 往返（本仓库 1479 entities 全量在内存中毫无压力）； +2. fuzzy 前增加**廉价预过滤**：先查内存 exact 索引（已存在），未命中且 name 长度 < 3 直接跳过（`resolvePrefix`/`resolveSuffix` 已有 `size < 3` 短路，可提前到 pipeline 层避免 2 次多余查询）； +3. 对 `skipped_fuzzy_no_ev` 的 refs 统计 top 模式，考虑缩小 fuzzy 候选上限，减少无效评分。 + +**预估**：298ms → 100ms 以内，全量索引总耗时下降 ~30%。 + +### P0-2：discovery 遍历量过大（44445 目录 / 41075 跳过文件） + +**依据**：`engine_index_project.cpp` 的 discovery 循环已用 `it.disable_recursion_pending()` 剪枝，但统计显示仍遍历 4.4 万目录。`FilterPolicy::NORMAL` 跳过表（filter_policy.cpp:73-111）含 node_modules/build/target 等，但 `.cache/`、`docs/`、`third_party/`、`benchmarks/` 等大目录可能仍在逐文件过滤。 + +**建议**： +1. 核对 `.codescopeignore` / 默认 skip 表是否覆盖 `.cache`、`build-release`、`build-tests`、`third_party`、`.deps-cache`（本次索引 215 个候选里 engine 有 198 个，说明第三方面源被纳入了——若不需要可加 `CODESCOPE_EXCLUDE_PATHS` 或 ignore 规则）； +2. 为 discovery 阶段**单独加计时埋点**（目前只统计数量不统计耗时，无法量化收益）。 + +### P1-1：`membulk_flush` 148ms 批量写优化 + +**建议**：确认 flush 时是否 `PRAGMA synchronous=OFF` + WAL（日志显示已开启 synchronous=OFF）；可尝试把 215 文件按模块分块 flush（并行模式已天然分块，串行单次 148ms 可拆 2–3 批并行落库）；检查 `_resolved_edges` 临时表批量拷贝是否用 `INSERT INTO ... SELECT` 一次完成。 + +### P1-2：并行合并 `merge_module_dbs` 58ms + +**建议**：`sqlite_exec=32ms` 是主成本——确认合并是否逐表 `ATTACH + INSERT`；可尝试单事务合并 + WAL 已开启前提下用 `BEGIN IMMEDIATE` 一次性提交，减少 checkpoint 次数（checkpoint=18ms）。 + +### P2：异步 state builder 61ms + +**建议**：`runModelIndexSync` 中 state=61ms 是异步最大头，若与 FTS/model 无数据依赖可并行执行（当前日志显示串行 `model=5 state=61 fts=8`）。 + +### P2：buildCSR 每次全量重建 + +**建议**：607 forward / 432 reverse groups 每次索引都重建；增量索引（is_reindex=1）时考虑按变更文件局部更新 CSR，避免全量。 + +--- + +## 5. 结论 + +本仓库规模下全量索引 ~700ms 已很快，但 **resolver（298ms）和 discovery 遍历（隐性）是两个明确的放大点**——fuzzy 匹配内存化是投入产出比最高的一刀，预计可再砍掉 30% 总耗时。 + +**优化执行状态**: + +- [x] P0-1 fuzzy 匹配内存化（已实施并验证） +- [ ] P0-2 discovery 剪枝 + 计时埋点 +- [ ] P1-1 membulk_flush 分块 +- [ ] P1-2 merge_module_dbs 事务化 +- [ ] P2 异步 state 并行 +- [ ] P2 buildCSR 增量 + +--- + +## 6. P0-1 实施结果（2026-08-11） + +### 6.1 改动内容 + +- `engine/src/resolver/fuzzy_resolver.cpp/.h`：FuzzyResolver 从「每 ref 最多 3 条 SQL（case-insensitive/prefix/suffix）」改为构造时一次性加载 `SELECT id, name FROM entity WHERE project_id=? AND name != ''` 进内存，建 ASCII-fold 精确索引 + 原样 LIKE 扫描，复用 `sqliteLikeMatch`（语义与 SQLite LIKE 逐字节一致）。 +- `engine/src/resolver/factors.h`：把 `likeFold`/`sqliteLikeMatch` 从 factors.cpp 匿名命名空间提升为共享 inline 函数，供 fuzzy resolver 复用。 +- `engine/src/resolver/pipeline.cpp`：删除 per-fuzzy-id 的 `lk_st` SQL 查询，改为 Step 0 加载 entity 后构建 `entity_by_id` 内存映射，fuzzy 命中的候选直接复制，省掉第二处 SQL 往返。 + +### 6.2 实测对比（同机同项目，串行全量索引） + +| 指标 | 优化前 | 优化后 | 提升 | +|---|---|---|---| +| resolver::run | **298ms** | **28ms** | **10.6x** | +| buildGraph total | 332ms | 62ms | 5.4x | +| engine_index_post_parse | 335ms | 66ms | 5.1x | +| 异步 state builder | 61ms | 12ms | 5.1x | +| 解析+建图总耗时 | ~497ms | ~218ms | 2.3x | + +### 6.3 并行索引对比 + +| 指标 | 优化前 | 优化后 | 提升 | +|---|---|---|---| +| index-parallel 总耗时 | 686ms | **517ms** | 25% | +| engine 模块 resolver | 125ms | 19ms | 6.6x | +| engine 模块 buildGraph | 153ms | 47ms | 3.3x | +| merge_module_dbs | 58ms | 57ms | —（未动） | + +### 6.4 结果一致性 + +- 节点数不变：1479 nodes / 1204 edges（并行）vs 1479 / 1193（串行，异步 FTS 差异） +- fuzzy 命中数基本一致：fuzzy=927–928，exact=4461–4465（内存 LIKE 语义与 SQL 逐字节一致） +- `test_fuzzy_resolver` 9 项断言全部通过（大小写/前缀/后缀/边界/limit） + +### 6.5 全量模式确认（用户质询） + +- **确认运行的是全量模式（NORMAL），不是 fast 模式**： + - 运行环境无 `CODESCOPE_INDEX_MODE` 环境变量（`env | grep CODESCOPE_INDEX` 为空）； + - `filter_policy.h:170` `Mode mode_ = NORMAL;` 为默认值，仅在 `CODESCOPE_INDEX_MODE=fast/strict` 时切换； + - 实测 discovery 遍历 44,445–44,656 个目录（fast 模式会大幅裁剪 skip 目录/suffix），符合全量模式行为。 + +### 6.6 严格 A/B 精度对比（同一固定输入快照） + +为排除「输入变化干扰精度对比」，将项目源码（含改动）rsync 到固定快照 `/tmp/codescope_src_snap`， +分别用 **旧二进制（git stash 回退 + 重建，SQL fuzzy）** 与 **新二进制（内存 fuzzy）** 索引同一快照： + +| 指标 | 旧版 (SQL fuzzy) | 新版 (内存 fuzzy) | 一致？ | +|---|---|---|---| +| resolved refs | 2090 / 11133 | 2090 / 11133 | ✅ | +| exact 命中 | 4465 | 4465 | ✅ | +| fuzzy 命中 | 927 | 927 | ✅ | +| skipped_miss / no_ev / ambiguous | 2322 / 3184 / 1334 | 2322 / 3184 / 1334 | ✅ | +| total_edges | 1194 | 1194 | ✅ | +| total_nodes | 1473 | 1473 | ✅ | +| files_indexed | 195 | 195 | ✅ | +| resolver 耗时 | 306ms | **30ms** | 10.2x 提升 | +| buildGraph 耗时 | 340ms | **65ms** | 5.2x 提升 | + +**结论：同一输入下新旧二进制输出逐项一致，fuzzy 内存化零精度损失。** +此前观测到的 refs 数量差异（11143 vs 11133、edges 1193 vs 1194）来自**输入变化**——优化前后 +项目目录中的 `fuzzy_resolver.cpp` 等源码文件本身被 CodeScope 索引（我修改了它们）， +而非 fuzzy 实现引入的精度差异。 + +--- + +## 7. 大项目验证（goagent，2026-08-11） + +**被测项目**：`~/go/src/goagent`（2.6G，1421 个 `.go`，实际索引 1374 文件） +**二进制**：新版（内存 fuzzy）`bin/codescope` vs 旧版（SQL fuzzy）`/tmp/codescope_old`，同一全量模式 + +### 7.1 耗时对比 + +| 指标 | 旧版 (SQL fuzzy) | 新版 (内存 fuzzy) | 提升 | +|---|---|---|---| +| resolver 耗时 | 2294ms | **2017ms** | 12% | +| buildGraph 耗时 | 2987ms | **2684ms** | 10% | +| post_parse 总耗时 | 3019ms | **2713ms** | 10% | +| membulk_flush | 1081ms | 1028ms | 5% | + +### 7.2 精度对比（关键发现：大项目上新版精度更高） + +| 指标 | 旧版 (SQL fuzzy) | 新版 (内存 fuzzy) | 说明 | +|---|---|---|---| +| resolved refs | 8201 / 25155 | **8287 / 25155** | +86 | +| exact 命中 | 15748 | 15748 | 一致 | +| fuzzy 命中 | **125** | **2163** | +2038 | +| skipped_budget | **5115** | **0** | 预算不再截断 | +| total_edges | 6760 | **6824** | +64 | +| total_nodes | 26756 | 26756 | 一致 | + +**分析**：goagent 规模（25155 refs）下，旧版 SQL fuzzy 触发 500ms 预算上限 +（`kFuzzyBudgetMs=500`），**5115 个 fuzzy 查询被跳过**（`skipped_budget=5115`），fuzzy 命中仅 125。 +新版内存 fuzzy 速度足够快、预算从不触发（`skipped_budget=0`），全部 2163 个 fuzzy 查询完整执行： +解析率 8201→8287（+1.0%），CALLS 边 6760→6824（+64，均为旧版被预算截断丢失的真实解析）。 + +**结论：优化后无精度损失；在大项目上反而恢复了旧版因性能预算而牺牲的解析，且耗时更低——纯收益。** + +--- + +## 8. 超大型项目验证（rustc，2026-08-11） + +**被测项目**：`~/code/rustcode/rust`（3.4G，36586 个 `.rs`，实际索引 6029 文件） +**二进制**：新版（内存 fuzzy）`bin/codescope` vs 旧版（SQL fuzzy）`/tmp/codescope_old`，同一全量模式 + +### 8.1 耗时对比 + +| 指标 | 旧版 (SQL fuzzy) | 新版 (内存 fuzzy) | 提升 | +|---|---|---|---| +| resolver 耗时 | 6376ms | **6080ms** | 5% | +| buildGraph 耗时 | 19556ms | **19356ms** | 1% | +| post_parse 总耗时 | 19983ms | **19778ms** | 1% | +| parse 总耗时 | 27846ms | 28949ms | 波动（±4%） | + +### 8.2 精度对比 + +| 指标 | 旧版 (SQL fuzzy) | 新版 (内存 fuzzy) | 说明 | +|---|---|---|---| +| resolved refs | 154539 / 450039 | **154822 / 450039** | +283 | +| exact 命中 | 406142 | 406142 | 一致 | +| fuzzy 命中 | **32** | **918** | +886 | +| skipped_budget | 24934 | **23812** | 新版预算内完成更多 | +| total_edges | 116293 | **116516** | +223 | +| total_nodes | 129893 | 129893 | 一致 | + +### 8.3 分析 + +- **fuzzy 命中 32 → 918（28.7x）**：rustc 规模（450039 refs、129893 entities）下旧版 SQL fuzzy + 几乎全部被 500ms 预算截断（fuzzy=32），新版内存扫描快得多，在预算内完成了 918 次 fuzzy 解析， + 恢复了 283 个此前丢失的真实解析、+223 条 CALLS 边。 +- **注意：rustc 上新版仍有 `skipped_budget=23812`**——129893 实体 × 上万次 fuzzy 调用的内存 + 遍历也累积超过 500ms 预算。这是 fuzzy 预算机制的兜底，可接受；若要进一步提升解析率， + 可对 fuzzy 内存索引增加前缀/后缀排序索引（二分查找），把 O(N) 扫描降为 O(log N)。 +- **新热点浮出**：`buildGraph` 中 `scope=11156ms`（applyConstraints 评分阶段）已超过 + resolver（6112ms），成为 buildGraph 最大子项（avg_cands=106.0，每个 ref 平均 106 个候选）。 + 这是下一轮优化的首要目标（候选集裁剪 / 因子评分并行化）。 + +### 8.4 三项目汇总 + +| 项目 | 规模 | resolver 优化前→后 | fuzzy 命中 | edges 变化 | +|---|---|---|---|---| +| CodeScope 自索引 | 215 文件 | 298ms→30ms (10.2x) | 927≈928 一致 | 1194=1194 一致 | +| goagent | 1374 文件 | 2294ms→2017ms (1.14x) | 125→2163 | 6760→6824 (+64) | +| rustc | 6029 文件 | 6376ms→6080ms (1.05x) | 32→918 | 116293→116516 (+223) | + +**结论：优化在所有规模下均无精度损失；中小项目输出逐项一致，超大项目恢复预算截断丢失的解析；耗时全面下降。** + +--- + +## 9. 已知问题记录：fast 模式与全量模式几乎无差异（rustc 实测） + +> **状态**：✅ 已修复（2026-08-11，P0-2 实施）——`fast_extra_skip_dirs_` 已补全， +> discovery 已加独立计时埋点。修复前实测数据保留如下作为基线。 + +### 9.1 现象 + +rustc 全量索引实测（新版二进制，墙钟）： + +| 模式 | 总耗时 | files_indexed | total_nodes | total_edges | +|---|---|---|---|---| +| 全量（NORMAL，默认） | 66.44s | 6029 | 129893 | 116516 | +| fast（CODESCOPE_INDEX_MODE=fast） | 64.91s | 6029 | 129893 | 116515 | + +fast 仅快约 1.5s（~2%），且 discovery 文件集与全量**逐字节相同** +（seen_dirs=65440、candidate_files=6035 完全一致），结果精度几乎相同（差 1 条边来自 fuzzy 预算时序截断）。 + +### 9.2 根因（代码证据，`engine/src/filter_policy.cpp`） + +1. **`fast_extra_skip_dirs_ = {}`（第 242 行）—— FAST 模式额外跳过目录是空集**。 + 注释明确写着 "reserved for future FAST-only additions"（预留实现，尚未补充任何目录）。 + test/、docs/、vendor/、bench/ 等已在 `normal_skip_dirs_` 中，NORMAL 模式同样跳过， + 所以 FAST 不产生任何额外目录剪枝。 +2. **`fast_extra_suffixes_ = { ".min.js", ".min.css" }`（第 478 行）—— 唯一真实差异**。 + rustc 是纯 Rust 项目，无任何 `.min.js`/`.min.css` 文件，后缀剪枝零命中。 +3. fast 模式唯一实际生效的行为是**跳过 FTS 构建**（`run_fts=0`，engine_index_project.cpp 第 1788 行 + `launchAsyncKnowledgeBuilder(project_id, !mode_fast)`）——但 FTS 只占约 1.2s（1197ms → 0ms）， + 而 async model/state 仍全量执行（state ≈ 13.4s 不受影响）。 + +### 9.3 影响 + +- **对 rustc / goagent 这类"源码干净"的项目**：fast 模式无实际收益，全量/快速耗时几乎相同； +- **对含大量 `.min.js`/`.min.css` 的前端项目**：fast 模式可跳过这些文件的解析，有少量收益； +- **根因是功能未完成而非设计缺陷**：`fast_extra_skip_dirs_` 预留但从未填充。 + +### 9.4 修复方向（待办） + +1. **补全 `fast_extra_skip_dirs_`**：为 FAST 模式填充真正的额外剪枝目录（如 + `node_modules`（若 normal 未覆盖）、`dist`、`out`、`coverage`、`benchmarks`（如认为非核心）、 + 各语言构建产物目录等），并更新 `skip_filenames_` / `skip_filename_prefixes_` 的 FAST 变体； +2. **评估 test/、docs/、vendor/、bench/ 是否应只在 FAST 模式跳过**（当前在 NORMAL 也跳过， + 若语义上这些属于"分析重点"则需调整 NORMAL 行为，注意会影响全量模式结果——需精度回归对比）； +3. 为 discovery 阶段增加**单独计时埋点**（当前只统计数量不统计耗时），量化剪枝收益； +4. 修复后用 rustc + 前端项目（含 .min.js）分别做 A/B 验证。 + +> 相关：P0-2（discovery 剪枝 + 计时埋点）与本节修复方向 2/3 重叠，可合并实施。 + +--- + +## 10. P0-2 实施结果：fast 模式补全 + discovery 计时埋点（2026-08-11） + +### 10.1 改动内容 + +**A. 补全 fast 专属剪枝**（`engine/src/filter_policy.h/.cpp`）： + +1. `fast_extra_skip_dirs_` 从空集补全为 12 个目录：`.output`（Next.js/Remix 构建输出）、 + `storybook-static`、`__generated__`（codegen 输出）、`playwright-report`、`test-results`、 + `allure-results`、`allure-report`（测试报告）、`.sass-cache`、`.scss-cache`（CSS 编译缓存）、 + `logs`、`.logs`（运行时日志）； +2. 新增 **`fast_extra_filenames_`**（FAST 专属精确文件名）：`.eslintcache`、`.stylelintcache`、 + `.prettiercache`、`tsconfig.tsbuildinfo`——在 `shouldSkipFile()` 的 FAST 分支检查； +3. 新增 **`fast_extra_filename_prefixes_`**（预留空集，与其余 fast_extra_* 对称）； +4. **修复隐藏 bug**：`setMode()` 原先只改 `mode_` 不重建 `active_skip_dirs_` + （`buildActiveSets()` 仅在构造函数调用）——若 `CODESCOPE_INDEX_MODE=fast` 在构造后设置， + `fast_extra_skip_dirs_` 永远不会生效。现在 `setMode()` 内调用 `buildActiveSets()`。 + +**B. discovery 独立计时埋点**（`engine/src/engine_index_project.cpp`）： +在 discovery 递归遍历前后加 `steady_clock` 计时，输出 +`engine: discovery= (seen_dirs=... seen_files=... skipped_dirs=... skipped_files=... candidate_files=...)` +——此前 discovery 阶段只统计数量、不统计耗时。 + +### 10.2 验证结果 + +**合成项目 A/B（含 logs/、test-results/、.output/、.eslintcache）**： + +| 模式 | candidate_files | files_indexed | 剪枝效果 | +|---|---|---|---| +| NORMAL | 4 | 4 | 全部保留 | +| **FAST** | **1** | **1** | logs/test-results/.output 目录 + .eslintcache 全部剪掉 | + +新规则**确实生效**：fast 只索引 src/main.go（1 文件），4 个低价值文件/目录被剪。 + +**goagent A/B（真实项目）**：fast 与 NORMAL 数据一致（1376 文件/26791 节点/6834 边）—— +因 goagent 的 4 个 logs 目录都在 `examples/`、`benchmarks/` 下，而这两者在 NORMAL 模式的 +`normal_skip_dirs_` 中**已经被跳过**，故 fast 无额外差异（预期行为：NORMAL 已覆盖的目录 +fast 不会重复计算）。 + +**rustc A/B**：fast 66.61s vs NORMAL 66.44s，文件集一致——rustc 无新剪枝目录命中 +（纯 Rust 源码树无 logs/.output/storybook-static 等），符合预期；discovery 埋点实测 +**143ms**（seen_dirs=4650），首次量化出 discovery 阶段真实耗时。 + +**discovery 埋点效果（各项目首次量化；seen_dirs 为目录条目数——修复后仅统计 +`entry.is_directory()`，不再把文件访问计入）**： + +| 项目 | discovery 耗时 | seen_dirs（目录数） | candidate_files | +|---|---|---|---| +| rustc | 143ms | 4650 | 6035 | +| goagent | 25ms | 845 | 1376 | +| CodeScope 自索引 | 0ms（小目录） | — | 215 | + +### 10.3 结论 + +- fast 模式不再"名不副实"：新剪枝目录/文件规则生效，对含 logs/测试报告/构建输出的项目 + 会真正减少候选文件（合成项目 4→1 验证）； +- 对 rustc/goagent 这类源码干净的项目，fast 仍与全量一致——这是**预期正确行为** + （NORMAL 已跳过 test/docs/vendor 等），不是缺陷； +- discovery 阶段耗时首次可观测，为后续剪枝优化提供量化依据。 + +> 已知问题 §9 已从"待修复"更新为"✅ 已修复"。 + +--- + +## 11. resolver load_var_types_struct 复合索引修复（2026-08-12） + +### 11.1 瓶颈定位（CODESCOPE_PROFILE_RESOLVER=1 分阶段计时） + +``` +RP[load_entities] = 81ms +RP[load_var_types_struct] = 3331ms ← 绝对瓶颈（占 resolver ~52%） +RP[load_refs] = 387ms +RP[resolve_loop] = 1997ms +RP[sql_batch] = 469ms +buildGraph resolver = 7192ms ← 占 buildGraph ~75% +``` + +### 11.2 根因（EXPLAIN + 实测定位） + +`load_var_types_struct` 内部的两个 self-JOIN（`semantic_records t JOIN semantic_records p ON t.parent_id=p.original_id AND p.file_path=t.file_path`）在 rustc 百万行表上极慢： + +- `p.original_id` 是 per-file 编号，跨文件大量冲突； +- SQLite 选了旧索引 `idx_sr_oid(project_id, original_id)`（只有 project_id + original_id，file_path 不在索引里）； +- 对每个 `t.kind=17` 行（6738 个），original_id 匹配了所有文件的同号行，再回表过滤 file_path → 海量无效候选扫描。 + +**实测对比（同一查询）**： + +| p 侧索引 | 耗时 | +|---|---| +| `idx_sr_oid(project_id, original_id)` | 1.435s | +| **新复合索引 `idx_sr_proj_file_oid(project_id, file_path, original_id)`** | **0.037s（38x）** | + +### 11.3 修复方案（2 个文件） + +新增复合索引 `idx_sr_proj_file_oid(project_id, file_path, original_id)`——前缀 `project_id` 让 SQLite 优化器自动选中，无需脆弱的 `INDEXED BY`： + +1. `store_schema.cpp`：createSchema 建索引（第 337 行）； +2. `store_membulk.cpp`：drop/create 索引列表同步（membulk 路径一致性，DROP 第 67 行 + CREATE 第 99 行）。 + +### 11.4 复测结果（rustc 全量索引，release） + +| 指标 | 优化前 | 优化后 | 提升 | +|---|---|---|---| +| load_var_types_struct | 3331ms | **82ms** | 40.6x | +| resolver 阶段 | 7192ms | 2979ms | 2.4x | +| buildGraph | ~9500ms | 5220ms | 1.8x | +| 墙钟 | 40.93s | **35.71s**（复测 40.98s，系统负载波动） | -5.2s | + +墙钟 35.71s，突破 40s 目标 ✅ + +### 11.5 精度验证（零损失） + +- entity 129893 / scope 130881 / import 40877（非零 40877）/ module_summary 898 —— 与优化前逐项一致； +- relation 117154–117157（±8 运行波动内）； +- 引擎测试全部通过：`test_accuracy_baseline`（accuracy gate，0 FP / 0 FN）、`test_resolve_strategy`、`test_homonym_filter`、`test_membulk`、`test_membulk_parity`。 + +### 11.6 46 个 MCP 工具逐一验证（非空转、数据准确） + +对 `TOOL_HANDLERS` 全部 **46 个工具**逐一调用（rustc 索引库 /tmp/rc_verify.db），结果： + +| 结论 | 数量 | 说明 | +|---|---|---| +| ✅ 返回合法 JSON 且数据准确 | 44 | 查询类返回真实数据（get_graph_stats=129893 nodes、find_callers/callees 真实边、codescope_trace 真实调用链、get_module_tree 169KB 等） | +| ✅ 参数格式需正确（已验证正确调用） | 2 | `verify_claim`/`verify_statement` 需要 `claim.type` 字段（capability_exists 等）；正确参数后返回 verdict | +| ❌ 空转 / Unknown tool | 0 | 无 | + +**关键数据核验**：`get_graph_stats` 返回 `total_nodes=129893 / total_edges=117154` 与 DB 实测一致；`find_callers(transmute)` 返回 ambiguous 候选（非空）；`get_entry_points` 44KB 真实入口；`verify_integrity` 122KB findings；`get_module_tree` 169KB 模块树——全部非空转。 diff --git a/docs/zh/skills.md b/docs/zh/skills.md index 01622cb..13ef185 100644 --- a/docs/zh/skills.md +++ b/docs/zh/skills.md @@ -10,7 +10,8 @@ CodeScope 是一个 **Project Truth Engine**。它把源码变成可验证的事 ```bash # 一行命令：索引项目并查询统计 -codescope cli index_project '{"project_path":"/path/to/project"}' +# （MCP 工具 index_project 已移除——用 worker CLI 串行，或 index-parallel 并行） +codescope worker /tmp/codescope.db /path/to/project "" project 0 codescope cli get_graph_stats '{}' ``` @@ -22,7 +23,7 @@ codescope cli get_graph_stats '{}' set -e PROJECT=$1 echo "=== 索引 $PROJECT ===" -codescope cli index_project "{\"project_path\":\"$PROJECT\"}" +codescope worker /tmp/codescope.db "$PROJECT" "" project 0 echo "=== 统计 ===" codescope cli get_graph_stats '{}' echo "=== 入口点 ===" @@ -107,19 +108,19 @@ graph TB flowchart LR PA["Phase A (ms 级)
scan_project"] -->|"模块 + 符号 + 入口点"| Ready["✓ AI 可用"] PB["Phase B (异步)
enhance_project"] -->|"调用图 + 复杂度 + FTS"| Enhanced - PC["Phase C (按需)
index_project"] -->|"全量解析 → 所有表"| Full + PC["Phase C (按需)
worker / index-parallel"] -->|"全量解析 → 所有表"| Full Ready -.->|触发| PB ``` --- -## 3. MCP 工具（19 个） +## 3. MCP 工具（38 个） ### 索引 | 工具 | 用途 | 输入 | 输出 | Token | |------|------|------|------|-------| -| `index_project` | 全量索引项目 | `project_path`, `language` | JSON: files_indexed, timing | ~50 | +| `worker`（CLI） | 全量索引项目（串行） | ` ` | JSON: files_indexed, timing | ~50 | | `index_file` | 索引单个文件 | `file_path` | JSON: file result | ~30 | | `search` | 统一搜索（FTS） | `query`, `limit?` | JSON: 搜索结果 | ~30 | @@ -195,7 +196,10 @@ AI 问答 → codescope_build_context (200-1000 tok) | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite 数据库路径 | | `GRAMMARS_DIR` | `grammars/` | tree-sitter 语法 .so 目录 | | `CODESCOPE_LSP` | (未设置) | LSP 服务器命令（类型增强） | -| `CODESCOPE_INDEX_MODE` | `normal` | 索引模式: `fast` / `normal` / `deep` | +| `CODESCOPE_INDEX_MODE` | `normal` | 索引模式: `fast` / `normal` / `strict`（见下方「索引模式」） | +| `CODESCOPE_WORKERS` | `min(hw,4)` | 解析 worker 线程数 | +| `CODESCOPE_SKIP_ASYNC` | (unset) | 设为 1 跳过异步 model/state/FTS 阶段（并行调度路径设置） | +| `CODESCOPE_PROFILE_RESOLVER` | (unset) | 启用 resolver 分阶段计时明细（`[module=resolver, method=run]`） | | `CODESCOPE_VERBOSE` | `1` | 设为 `0` 关闭批量日志 | | `CODESCOPE_MAX_FILE_SIZE` | `5242880` (5MB) | 最大索引文件大小（字节） | | `CODESCOPE_MEMORY_BUDGET_MB` | `0` (无限制) | RSS 超限时暂停解析 | @@ -206,13 +210,15 @@ AI 问答 → codescope_build_context (200-1000 tok) ## 6. 索引模式 -通过 `CODESCOPE_INDEX_MODE` 设置： +通过 `CODESCOPE_INDEX_MODE` 设置（filter_policy.h 枚举 NORMAL/FAST/STRICT——**没有 deep 模式**）： -| 模式 | FTS | 向量 | TTFA | 适用场景 | -|------|-----|------|------|---------| -| `fast` | ❌ | ❌ | **最快** | 快速回答，最小索引 | -| `normal` | ✅ | ❌ | 正常 | 默认 — 图形 + 搜索 | -| `deep` | ✅ | ✅ | 较慢 | 全量分析（含语义向量） | +| 模式 | 枚举值 | 额外剪枝 | FTS | 适用场景 | +|------|--------|----------|-----|---------| +| `fast` | `FAST` | ✅ 额外跳过 logs/.output/storybook-static/playwright-report/test-results/allure-*/.sass-cache/.scss-cache/__generated__ 等 12 类目录 + `.eslintcache`/`.stylelintcache`/`.prettiercache`/`tsconfig.tsbuildinfo` 4 类缓存文件 | ❌ 跳过 | 最快；数据≈全量（源码干净项目几乎无差异） | +| `normal` | `NORMAL` | 仅基础 skip 表（build/dist/out/target/test/docs/vendor/node_modules/.venv 等） | ✅ | 默认 | +| `strict` | `STRICT` | 基础 skip + detectLanguage 白名单 gate（仅索引可识别语言的源码文件） | ✅ | 最严格，数据最精简 | + +> 已知问题（2026-08-11 已修复）：fast 此前≈normal——`fast_extra_skip_dirs_` 为空集（预留未实现）、`setMode()` 未重建 `active_skip_dirs_`。已补全剪枝集合并修复。详见 `docs/optimization/perf-full-index-2026-08-11.md` §9/§10。 ## 7. 性能基准 diff --git a/engine/CMakeLists.txt b/engine/CMakeLists.txt index 9ffa5c5..bc671b6 100644 --- a/engine/CMakeLists.txt +++ b/engine/CMakeLists.txt @@ -177,7 +177,10 @@ set(ENGINE_SOURCES src/engine_helpers.cpp src/engine_lifecycle.cpp src/engine_index.cpp - src/engine_index_project.cpp + src/engine_index_project.cpp + src/engine_index_files.cpp + src/engine_index_sched.cpp + src/engine_index_discover.cpp src/engine_index_project_membulk.cpp src/engine_index_post_parse.cpp src/engine_index_metrics.cpp @@ -235,6 +238,9 @@ set(ENGINE_SOURCES src/lsp/lsp_client.cpp src/resolver/resolver.cpp src/resolver/pipeline.cpp + src/resolver/pipeline_apply.cpp + src/resolver/pipeline_load.cpp + src/resolver/pipeline_flush.cpp src/resolver/fuzzy_resolver.cpp src/resolver/factors.cpp src/model/engine.cpp diff --git a/engine/tests/test_bench.cpp b/engine/manual/test_bench.cpp similarity index 100% rename from engine/tests/test_bench.cpp rename to engine/manual/test_bench.cpp diff --git a/engine/tests/test_bench_project.cpp b/engine/manual/test_bench_project.cpp similarity index 100% rename from engine/tests/test_bench_project.cpp rename to engine/manual/test_bench_project.cpp diff --git a/engine/tests/test_pipeline_bench.cpp b/engine/manual/test_pipeline_bench.cpp similarity index 100% rename from engine/tests/test_pipeline_bench.cpp rename to engine/manual/test_pipeline_bench.cpp diff --git a/engine/src/engine_ffi.cpp b/engine/src/engine_ffi.cpp index 380405b..240122d 100644 --- a/engine/src/engine_ffi.cpp +++ b/engine/src/engine_ffi.cpp @@ -1675,6 +1675,6 @@ const char *engine_version(void) // No try/catch required — only a static string literal is returned, // so no exceptions are possible. // Keep in sync with RELEASE.md and Cargo.toml version. - static const char kVersion[] = "0.2.5"; + static const char kVersion[] = "0.2.6"; return kVersion; } diff --git a/engine/src/engine_index_discover.cpp b/engine/src/engine_index_discover.cpp new file mode 100644 index 0000000..9b3299c --- /dev/null +++ b/engine/src/engine_index_discover.cpp @@ -0,0 +1,277 @@ +#include "engine_index_discover.h" + +#include +#include +#include +#include +#include +#include + +#include "engine_internal.h" +#include "posix_compat.h" + +namespace engine_index_discover +{ + +// Walk `dir` and collect candidate source files, applying the same +// FilterPolicy rules as the scanner (skip dirs, gitignore, +// .codescopeignore, bundle suffixes, filename/suffix skips, language +// filter). Also ingests the project-root README as a knowledge +// document and runs the incremental scan-state gate. +int collectFileJobs(uint64_t project_id, const std::string &dir, + FilterPolicy &filter, + const std::unordered_set &scan_state, + std::vector &jobs, bool &is_reindex, + std::string &err_json) +{ + // Pre-detect Java projects BEFORE the directory walk. The FilterPolicy + // Java carve-out defers test/docs/example/samples/... dirs to a + // top-only check ONLY when lang_context_ == "java", but lang_context_ + // previously flipped only upon seeing the FIRST .java file during the + // walk — and that file may itself live under an example/samples/... + // dir which is skipped at any depth while lang_context_ is still + // empty. That chicken-and-egg made Java projects with such package + // dirs index 0 files (e.g. spring-petclinic's + // org/springframework/samples/petclinic). Fix: cheap recursive scan + // for any *.java before the main walk and flip lang_context_ early. + { + std::error_code ec; + auto pit = std::filesystem::recursive_directory_iterator( + dir, + std::filesystem::directory_options::skip_permission_denied, + ec); + std::filesystem::recursive_directory_iterator pend; + while (!ec && pit != pend) { + const auto &pent = *pit; + if (pent.is_regular_file() && + pent.path().extension() == ".java") { + filter.setLangContext("java"); + break; + } + pit.increment(ec); + } + } + + try { + // P0-2: standalone discovery timing. Previously this phase only + // reported entry counts (seen_dirs/skipped_files/...); wall-clock + // cost was invisible. Instrument the full walk so discovery can + // be compared against parse/buildGraph stages. + using namespace std::chrono; + auto t_discovery_start = steady_clock::now(); + auto it = std::filesystem::recursive_directory_iterator( + dir, std::filesystem::directory_options:: + skip_permission_denied); + for (auto &entry : it) { + // seen_dirs counts ONLY directory entries — recursive_ + // directory_iterator yields files too, so counting every + // entry here inflated the metric with file visits. JSON + // discovery.seen_dirs and the discovery= log share this + // counter, so both now report true directory counts. + if (entry.is_directory()) + filter.stats().seen_dirs++; + std::string rel = entry.path().string(); + if (rel.size() > dir.size() + 1) + rel = rel.substr(dir.size() + 1); + else + rel.clear(); + + // ── README / document ingestion (BEFORE skip filter) ── + // .md files are in skip_suffixes_ (filter_policy.cpp:422) + // so they never reach the source-code indexing path. + // But the knowledge layer (CapabilityPlugin, + // ContractPlugin) needs README content in the + // document table to extract capabilities/contracts. + // Therefore we intercept README.md here — BEFORE + // shouldSkipEntry() drops it — and ingest it via + // insertDocument(). + // + // Only the project-root README is ingested as a + // knowledge document; nested READMEs are ignored + // to avoid noise from vendored deps. + if (entry.is_regular_file()) { + const std::string &fp = entry.path().string(); + std::string fname = fp; + size_t sl = fp.find_last_of("/\\"); + if (sl != std::string::npos) + fname = fp.substr(sl + 1); + // Case-insensitive README.md match + std::string fname_lower = fname; + for (auto &c : fname_lower) + c = static_cast(std::tolower(c)); + + // Check if this README is at project root + bool is_root_readme = false; + if (fname_lower == "readme.md" || + fname_lower == "readme.markdown" || + fname_lower == "readme") { + // Project-root README: its parent dir == dir + std::string parent = fp; + size_t ps = parent.find_last_of("/\\"); + parent = (ps != std::string::npos) ? + parent.substr(0, ps) : + ""; + is_root_readme = (parent == dir); + } + + if (is_root_readme) { + // Ingest README content into document table. + // type=0 (kDocumentTypeReadme) signals the + // knowledge layer to parse capabilities. + std::string content = + readFile(fp.c_str()); + if (!content.empty()) { + int doc_type = + 0; // kDocumentTypeReadme + // insertDocument's 5th/6th params are + // start_line / end_line (1-based line + // numbers), NOT byte offsets. Count + // newlines to compute the line range. + int line_count = 1; + for (char c : content) + if (c == '\n') + ++line_count; + if (!g_store->insertDocument( + project_id, + doc_type, fp, + content, 1, + line_count)) { + fprintf(stderr, + "engine: insertDocument failed for %s: %s " + "[module=engine, method=collectFileJobs]\n", + fp.c_str(), + g_store->error() + .c_str()); + } + } + // README is ingested as a document, NOT as + // source code — skip the rest of the loop. + continue; + } + } + + if (!rel.empty()) { + bool entry_is_dir = entry.is_directory(); + // Use the consolidated entry check (single source of + // truth) so the indexer and scanner apply identical + // filtering: skip_dirs (any depth), gitignore, + // .codescopeignore, bundle-dir suffixes, filename skip, + // filename-prefix skip, and suffix skip. + if (filter.shouldSkipEntry(rel, entry_is_dir)) { + if (entry_is_dir) { + it.disable_recursion_pending(); + filter.stats().skipped_dirs++; + } else { + filter.stats().skipped_files++; + } + continue; + } + } + if (entry.is_regular_file()) { + filter.stats().seen_files++; + + // Incremental: check file_scan_state to skip unchanged files + struct stat file_stat; + int64_t mtime = 0, fsize = 0; + bool file_unchanged = false; + if (stat(entry.path().string().c_str(), + &file_stat) == 0) { + mtime = static_cast( + file_stat.st_mtime); + fsize = static_cast( + file_stat.st_size); + // O(1) in-memory lookup instead of per-file DB query. + // M2: two-stage incremental gate. Stage 1 is the cheap + // mtime|size gate (no file read). Only when it matches do + // we hash the file and check the mtime|size|hash gate, + // closing the "same size + same mtime but changed content" + // hole. Files whose mtime/size differ skip without being + // read, so incremental performance is preserved. + std::string base = + entry.path().string() + "|" + + std::to_string(mtime) + "|" + + std::to_string(fsize); + if (scan_state.count(base) > 0) { + std::string ch = fileContentHash( + entry.path() + .string() + .c_str()); + if (!ch.empty()) + file_unchanged = + scan_state.count( + base + + "|" + + ch) > 0; + // If hashing failed (unreadable), fall back to + // treating as unchanged on the mtime|size gate. + else + file_unchanged = true; + } + } + if (file_unchanged) { + is_reindex = true; + filter.stats().skipped_files++; + continue; + } + const char *lang = filter.detectLanguage( + entry.path().string().c_str()); + if (!lang) { + filter.stats().skipped_lang++; + continue; + } + // Detect Java projects on the fly — the FIRST .java + // file flips filter into Java mode so test/docs/samples + // collisions with Java package namespaces (e.g. + // org/springframework/samples/petclinic) get the + // top-only (depth ≤ 3) treatment instead of being + // skipped at any depth. See README.md "Why Java is + // the (only) exception". Idempotent — setLangContext + // is cheap and safe to repeat. + if (strcmp(lang, "java") == 0 && + filter.langContext() != "java") { + filter.setLangContext("java"); + } + if (!filter.isLanguageAccepted(lang)) { + filter.stats().skipped_lang++; + continue; + } + filter.stats().candidate_files++; + auto file_size = + entry.is_regular_file() ? + std::filesystem::file_size( + entry.path()) : + 0; + jobs.push_back({ entry.path().string(), lang, + file_size }); + } + } + // P0-2: report the standalone discovery wall-clock. Same + // [module=engine, method=...] format as the other pipeline + // stages so it can be parsed by the same tooling. + auto discovery_ms = duration_cast( + steady_clock::now() - t_discovery_start); + fprintf(stderr, + "engine: discovery=%lldms (seen_dirs=%llu seen_files=%llu " + "skipped_dirs=%llu skipped_files=%llu candidate_files=%zu) " + "[module=engine, method=collectFileJobs]\n", + static_cast(discovery_ms.count()), + static_cast( + filter.stats().seen_dirs), + static_cast( + filter.stats().seen_files), + static_cast( + filter.stats().skipped_dirs), + static_cast( + filter.stats().skipped_files), + jobs.size()); + } catch (const std::exception &e) { + std::ostringstream err; + err << "{\"ok\":false,\"error\":\"scan error: " + << jsonEscape(e.what()) << "\"}"; + err_json = err.str(); + return -1; + } + return 0; +} + +} // namespace engine_index_discover diff --git a/engine/src/engine_index_discover.h b/engine/src/engine_index_discover.h new file mode 100644 index 0000000..f6b9250 --- /dev/null +++ b/engine/src/engine_index_discover.h @@ -0,0 +1,47 @@ +#ifndef ENGINE_INDEX_DISCOVER_H +#define ENGINE_INDEX_DISCOVER_H + +#include +#include +#include +#include + +#include "filter_policy.h" + +namespace engine_index_discover +{ + +// One candidate source file collected during discovery. Mirrors the +// per-file job tuple used by both the streaming pipeline and the +// in-memory (membulk) path. +struct FileJob { + std::string path; + std::string lang; + size_t size = 0; +}; + +// Walk `dir` and collect candidate source files, applying the same +// FilterPolicy rules as the scanner (skip dirs, gitignore, +// .codescopeignore, bundle suffixes, filename/suffix skips, language +// filter). Also ingests the project-root README as a knowledge +// document and runs the incremental scan-state gate. +// +// @param project_id Project to index. +// @param dir Absolute project root (trailing separators removed). +// @param filter [in/out] FilterPolicy; mutated (stats counters, Java +// lang-context flip) during the walk. +// @param scan_state "path|mtime|size" tuples for incremental skips. +// @param jobs [out] Collected candidate files (unsorted). +// @param is_reindex [out] True if any file was skipped as unchanged. +// @param err_json [out] JSON error payload when returning -1. +// @return 0 on success (jobs populated), -1 on scan error (err_json set). +// @throws nothing — filesystem exceptions are caught internally. +int collectFileJobs(uint64_t project_id, const std::string &dir, + FilterPolicy &filter, + const std::unordered_set &scan_state, + std::vector &jobs, bool &is_reindex, + std::string &err_json); + +} // namespace engine_index_discover + +#endif // ENGINE_INDEX_DISCOVER_H diff --git a/engine/src/engine_index_files.cpp b/engine/src/engine_index_files.cpp new file mode 100644 index 0000000..077b93d --- /dev/null +++ b/engine/src/engine_index_files.cpp @@ -0,0 +1,667 @@ +#include "engine_internal.h" +#include "filter_policy.h" +#include "platform_win.h" + +// Undefine Windows macros that conflict with enum values +#ifdef STRICT +#undef STRICT +#endif +#ifdef FAST +#undef FAST +#endif + +#include +#include +#include +#include +#include +#include "posix_compat.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ir/translators/js_visitor.h" +#include "engine_index_metrics.h" +#include "async_knowledge.h" +#include "store/store_parse_failure.h" + +namespace +{ +// Mirrors the constant in engine_index_project.cpp (kept per-TU so this +// split file stays self-contained under the 1000-line rule). +constexpr uint64_t kMaxFileSize = 5 * 1024 * 1024; // 5 MB default +} // namespace + +// ─── Index Project (File List) ────────────────────────────────── +// Indexes an explicit JSON list of files (used by the scheduler / worker +// --file-list path). Split out of engine_index_project.cpp into its own +// translation unit so each file stays under the 1000-line rule +// (plan/rules/code_rules.md §1). +char *engine_index_files(uint64_t project_id, const char *file_list_json) +{ + if (!g_store) + return dupString( + "{\"ok\":false,\"error\":\"engine not initialized\"}"); + + if (!file_list_json || !file_list_json[0]) + return dupString( + "{\"ok\":false,\"error\":\"file_list_json is empty\"}"); + + const char *env_mode = getenv("CODESCOPE_INDEX_MODE"); + bool mode_fast = env_mode && strcmp(env_mode, "fast") == 0; + uint64_t max_file_size = kMaxFileSize; + const char *env_max = getenv("CODESCOPE_MAX_FILE_SIZE"); + if (env_max) + max_file_size = static_cast(std::atoll(env_max)); + + // Parse JSON file list + // Expected format: ["/path/to/file1.c", "/path/to/file2.c", ...] + // Simple parser: comma-separated quoted strings + struct FileJob { + std::string path; + std::string lang; + size_t size = 0; + // mtime captured during discovery so the writer can persist + // it on FileResult (mirrors the streaming path's per-worker + // stat). Without it, result->mtime defaults to 0 and a later + // incremental run's isFileUnchanged can never skip (M-13). + int64_t mtime = 0; + }; + std::vector jobs; + std::string json(file_list_json); + size_t pos = 0; + FilterPolicy filter; + while ((pos = json.find('"', pos)) != std::string::npos) { + size_t end = json.find('"', pos + 1); + if (end == std::string::npos) + break; + std::string path = json.substr(pos + 1, end - pos - 1); + pos = end + 1; + if (path.empty()) + continue; + + // Detect language from file extension + struct stat file_stat; + if (stat(path.c_str(), &file_stat) != 0) + continue; + if (static_cast(file_stat.st_size) > max_file_size) + continue; + + const char *lang = filter.detectLanguage(path.c_str()); + if (!lang) + continue; + + // Detect Java projects on the fly — the FIRST .java file + // flips filter into Java mode so test/docs/samples collisions + // with Java package namespaces (e.g. + // org/springframework/samples/petclinic) get the top-only + // (depth ≤ 3) treatment instead of being skipped at any + // depth. See README.md "Why Java is the (only) exception". + // Idempotent — setLangContext is cheap and safe to repeat. + if (strcmp(lang, "java") == 0 && + filter.langContext() != "java") { + filter.setLangContext("java"); + } + + jobs.push_back({ path, lang, + static_cast(file_stat.st_size), + static_cast(file_stat.st_mtime) }); + } + + if (jobs.empty()) + return dupString( + "{\"ok\":true,\"files_indexed\":0,\"nodes\":0,\"edges\":0,\"errors\":0}"); + + // Fail-fast: pre-load known parse failures so the parse loop can + // skip them without per-file DB queries. Mirrors the logic in + // engine_index_project. + const int kFailRetryMax = [] { + const char *e = getenv("CODESCOPE_FAIL_RETRY_MAX"); + return e ? std::max(1, std::atoi(e)) : 3; + }(); + std::unordered_set known_failures; + { + std::vector fail_vec; + if (!store::loadKnownParseFailures(project_id, kFailRetryMax, + /*out*/ fail_vec)) { + fprintf(stderr, + "engine: loadKnownParseFailures failed " + "(continuing) " + "[module=engine, method=engine_index_files]\n"); + } else { + known_failures.insert(fail_vec.begin(), fail_vec.end()); + } + } + + // ── Reuse the same parallel processing pipeline ────────────── + // The rest of the function mirrors engine_index_project from the + // job-processing phase onward. + + // Init progress tracking + { + store::IndexProgress p; + p.project_id = project_id; + p.total_files = static_cast(jobs.size()); + p.phase = 1; + store::setIndexProgress(p); + } + + // Sort by file size descending — large files first + std::sort(jobs.begin(), jobs.end(), + [](const FileJob &a, const FileJob &b) { + return a.size > b.size; + }); + + // Pre-load TSLanguage pointers + std::unordered_map lang_ptrs; + { + std::unordered_set langs; + for (auto &j : jobs) + langs.insert(j.lang); + for (auto &l : langs) + lang_ptrs[l] = g_parser->getLanguage(l.c_str()); + } + + // ── Streaming Pipeline ───────────────────────────────────── + using namespace std::chrono; + steady_clock::time_point t_parse_start; + int64_t time_parse_ms = 0, time_buildgraph_ms = 0; + + const size_t kQueueCapacity = + std::max(2 * std::thread::hardware_concurrency(), 8); + BoundedQueue> result_queue( + kQueueCapacity); + + std::atomic next_job{ 0 }; + std::atomic files_queued{ 0 }; + std::atomic files_written{ 0 }; + std::atomic writer_error{ 0 }; + const int64_t total_files = static_cast(jobs.size()); + const int64_t progress_interval = + std::max(1, total_files / 10); + const size_t kWriterBatchSize = 50; + + // ── Writer thread ────────────────────────────────────────── + std::thread writer_thread([&]() { + g_store->beginTransaction(); + std::vector batch; + batch.reserve(kWriterBatchSize); + while (true) { + std::unique_ptr fr; + bool ok = result_queue.pop(fr); + if (!ok) { + if (!batch.empty()) { + if (!g_store->insertFileResultBatch( + project_id, batch)) { + writer_error = 1; + } + files_written += + static_cast(batch.size()); + } + break; + } + batch.push_back(std::move(*fr)); + fr.reset(); + while (batch.size() < kWriterBatchSize) { + std::unique_ptr extra; + if (!result_queue.pop(extra)) + break; + batch.push_back(std::move(*extra)); + extra.reset(); + } + if (batch.size() >= kWriterBatchSize || + result_queue.isDone()) { + if (!g_store->insertFileResultBatch(project_id, + batch)) { + writer_error = 1; + } + files_written += static_cast(batch.size()); + batch.clear(); + } + } + if (writer_error) + g_store->rollbackTransaction(); + else + g_store->commitTransaction(); + }); + + // ── Parse workers ────────────────────────────────────────── + auto parse_worker_fn = [&]() { + struct TSParserDeleter { + void operator()(TSParser *p) const + { + if (p) + ts_parser_delete(p); + } + }; + struct TSTreeDeleter { + void operator()(TSTree *t) const + { + if (t) + ts_tree_delete(t); + } + }; + thread_local static std::unordered_map< + std::string, std::unique_ptr> + tl_parsers; + thread_local static std::unordered_map< + std::string, std::unique_ptr> + tl_visitors; + + // Outermost guard for the static worker thread. Complementing + // the temp-worker try/catch, this catches any unexpected throw + // (e.g. std::bad_alloc from result_queue.push / make_unique) + // so a single bad file cannot escape to std::terminate and + // abort the whole index. The failing job is recorded via + // recordParseFailure with full [module=engine, method=parse_worker_fn] + // tracing; the thread then exits cleanly (it is joined below). + std::string current_path; + std::string current_lang; + try { + while (true) { + int idx = next_job.fetch_add(1); + if (idx >= static_cast(jobs.size())) + break; + auto &job = jobs[idx]; + // Track the in-flight job so an unexpected throw below + // can be recorded against the correct file (M-14). + current_path = job.path; + current_lang = job.lang; + + // Fail-fast: skip files that have failed >= + // CODESCOPE_FAIL_RETRY_MAX times. + if (known_failures.find(job.path) != + known_failures.end()) { + continue; + } + + int done = next_job.load(); + if (done % progress_interval == 0 && done > 0) + fprintf(stderr, + "engine: parse progress %lld/%lld " + "(%d%%) [module=engine, " + "method=engine_index_files]\n", + (long long)done, + (long long)total_files, + (int)(done * 100 / + total_files)); + + std::string source = readFile(job.path.c_str()); + if (source.empty()) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + ReadEmpty)); + continue; + } + + // Per-thread parser + auto pit = tl_parsers.find(job.lang); + if (pit == tl_parsers.end()) { + auto lit = lang_ptrs.find(job.lang); + if (lit == lang_ptrs.end()) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); + continue; + } + auto np = std::unique_ptr< + TSParser, TSParserDeleter>( + ts_parser_new()); + ts_parser_set_language(np.get(), + lit->second); + tl_parsers[job.lang] = std::move(np); + pit = tl_parsers.find(job.lang); + } + auto tree = + std::unique_ptr( + ts_parser_parse_string( + pit->second.get(), + nullptr, source.c_str(), + static_cast( + source.size()))); + if (!tree) { + store::bufferParseFailure( + project_id, job.path, job.lang, + store::failReasonToString( + store::FailReason:: + ParseNullTree)); + continue; + } + + auto result = + std::make_unique(); + result->file_path = job.path; + result->language = job.lang; + // Persist the mtime captured at discovery time + // (engine_index_files has no per-worker stat). A zero + // mtime would break later isFileUnchanged incremental + // skips (M-13). + result->mtime = job.mtime; + result->fsize = static_cast(job.size); + + // Visitor pipeline + auto vl = tl_visitors.find(job.lang); + ir::JsVisitor *visitor = nullptr; + if (vl == tl_visitors.end()) { + auto v = ir::createJsVisitor( + job.lang.c_str()); + if (v) { + tl_visitors[job.lang] = + std::move(v); + visitor = tl_visitors[job.lang] + .get(); + } + } else { + visitor = vl->second.get(); + visitor->reset(); + } + + if (visitor) { + ir::SemanticUnit *su = nullptr; + try { + su = visitor->visit( + tree.get(), + source.c_str(), + job.path.c_str()); + } catch (const std::exception &e) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + std::string(store::failReasonToString( + store::FailReason:: + VisitorException)) + + ": " + + e.what()); + continue; + } catch (...) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + store::failReasonToString( + store::FailReason:: + VisitorUnknownThrow)); + continue; + } + if (su) { + result->records = + su->allRecords(); + result->metrics = index_metrics:: + computeMetricsFromCST( + tree.get(), + source.c_str(), + result->records); + } + } else { + // Old pipeline fallback + auto translator = ir::createTranslator( + job.lang.c_str()); + if (!translator) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + store::failReasonToString( + store::FailReason:: + LanguageMissing)); + continue; + } + ir::TranslationUnit *unit = nullptr; + try { + unit = translator->translate( + tree.get(), + source.c_str(), + job.path.c_str()); + } catch (const std::exception &e) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + std::string(store::failReasonToString( + store::FailReason:: + VisitorException)) + + ": " + + e.what()); + continue; + } catch (...) { + store::bufferParseFailure( + project_id, job.path, + job.lang, + store::failReasonToString( + store::FailReason:: + VisitorUnknownThrow)); + continue; + } + if (unit && unit->root) { + uint64_t flat_id = 1; + std::function + flatten = [&](ir::Node * + n, + uint64_t + parent) { + uint64_t my_id = + flat_id++; + ir::Record rec; + rec.id = my_id; + rec.kind = static_cast< + ir::RecordKind>( + static_cast< + int>( + n->kind)); + rec.name = + n->name; + rec.qualified_name = + n->qualified_name; + rec.parent_id = + parent; + rec.loc.start_row = + n->loc.start_row; + rec.loc.start_col = + n->loc.start_col; + rec.loc.end_row = + n->loc.end_row; + rec.loc.end_col = + n->loc.end_col; + rec.file_path = + job.path; + result->records.push_back( + std::move( + rec)); + for (auto *c : + n->children) + flatten(c, + my_id); + }; + flatten(unit->root, 0); + } + } + + result_queue.push(std::move(result)); + files_queued++; + } + } catch (const std::exception &e) { + // An unexpected exception escaped the per-call guarded + // sections (visit()/translate() are individually + // guarded). Record the failing job and log with full + // tracing instead of letting it reach std::terminate + // (M-14). The worker thread then exits; it is joined + // below so the index finishes gracefully. + if (!current_path.empty()) { + store::bufferParseFailure( + project_id, current_path, current_lang, + std::string("unexpected: ") + e.what()); + } + fprintf(stderr, + "engine: static parse worker aborted: %s " + "[module=engine, method=parse_worker_fn]\n", + e.what()); + } catch (...) { + fprintf(stderr, + "engine: static parse worker aborted: " + "unknown exception " + "[module=engine, method=parse_worker_fn]\n"); + } + }; + + // ── Spawn workers ────────────────────────────────────────── + t_parse_start = steady_clock::now(); + int num_workers = + std::min(static_cast(jobs.size()), + static_cast(std::thread::hardware_concurrency())); + const char *env_workers = getenv("CODESCOPE_WORKERS"); + if (env_workers && env_workers[0]) { + int requested = std::atoi(env_workers); + if (requested > 0) + num_workers = std::min(requested, num_workers); + } else { + num_workers = std::min(num_workers, 4); + } + if (num_workers < 1) + num_workers = 1; + + std::vector workers; + for (int i = 0; i < num_workers; i++) { + try { + workers.emplace_back(parse_worker_fn); + } catch (const std::system_error &e) { + fprintf(stderr, + "engine: thread spawn failed: %s " + "[module=engine, method=engine_index_files]\n", + e.what()); + } + } + for (auto &t : workers) { + if (t.joinable()) + t.join(); + } + result_queue.markDone(); + if (writer_thread.joinable()) + writer_thread.join(); + time_parse_ms = + duration_cast(steady_clock::now() - t_parse_start) + .count(); + + // ── Build graph ──────────────────────────────────────────── + if (writer_error == 0) { + t_parse_start = steady_clock::now(); + // buildGraph(...true) is a FULL rebuild: it drops the lookup + // + unique-edge indexes. Unlike engine_index_project (which + // reaches this via engine_index_post_parse), this path must + // recreate those indexes itself or they stay missing (M-12). + // P2 fix: a resolver-pipeline failure makes buildGraph roll back + // its graph savepoint and return false; flag it as a writer error + // so the result JSON reports failure instead of a false success + // (the outer transaction is rolled back by the caller when it + // sees ok:false). + if (!g_store->buildGraph(project_id, true)) { + writer_error = 1; + } + time_buildgraph_ms = + duration_cast(steady_clock::now() - + t_parse_start) + .count(); + + // Mark every node callgraph_ready: the call graph is now + // committed, so trace_path / enhancement-status report + // readiness correctly (mirrors engine_index_post_parse, M-15). + { + std::string up = + "UPDATE graph_nodes SET callgraph_ready=1 " + "WHERE project_id=" + + std::to_string(project_id); + if (!g_store->exec(up.c_str())) { + fprintf(stderr, + "engine_index_files: callgraph_ready " + "UPDATE failed: %s " + "[module=engine, method=engine_index_files]\n", + g_store->error().c_str()); + } + } + + // Recreate lookup + unique-edge indexes dropped by the full + // rebuild. full_rebuild=true (buildGraph did a full rebuild, + // not an incremental run) matches the post-parse call + // createIndexesAfterBulkLoad(project_id, !is_reindex) with + // is_reindex=false (M-12). + { + store::GraphStore::BulkPragmaGuard guard(g_store.get()); + auto t_idx = steady_clock::now(); + g_store->createIndexesAfterBulkLoad(project_id, true); + fprintf(stderr, + "engine: createIndexesAfterBulkLoad=%lldms " + "[module=engine, method=engine_index_files]\n", + (long long)duration_cast( + steady_clock::now() - t_idx) + .count()); + } + + // Set the core-graph readiness flag so the project is + // queryable immediately after a file-list index (M-15). + g_store->setProjectReadiness(project_id, "normal_ready", 1); + } + + // ── Build result JSON ────────────────────────────────────── + std::ostringstream result; + result << "{\"ok\":" << (writer_error == 0 ? "true" : "false") + << ",\"files_indexed\":" << files_written.load() + << ",\"workers\":" << num_workers + << ",\"time_parse_ms\":" << time_parse_ms + << ",\"time_buildgraph_ms\":" << time_buildgraph_ms; + + // Query node/edge counts + { + sqlite3 *db = g_store->handle(); + sqlite3_stmt *stmt = nullptr; + std::string sql = + "SELECT COUNT(*) FROM entity WHERE project_id = " + + std::to_string(project_id); + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == + SQLITE_OK) { + if (sqlite3_step(stmt) == SQLITE_ROW) + result << ",\"total_nodes\":" + << sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + } + sql = "SELECT COUNT(*) FROM relation WHERE project_id = " + + std::to_string(project_id); + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == + SQLITE_OK) { + if (sqlite3_step(stmt) == SQLITE_ROW) + result << ",\"total_edges\":" + << sqlite3_column_int64(stmt, 0); + sqlite3_finalize(stmt); + } + } + + result << ",\"discovery\":{\"candidate_files\":" << jobs.size() << "}" + << "}"; + + // Mark progress as done + { + store::IndexProgress p; + p.project_id = project_id; + p.total_files = static_cast(jobs.size()); + p.current_file = static_cast(jobs.size()); + p.phase = 5; + p.percent = 100; + store::setIndexProgress(p); + } + + launchAsyncKnowledgeBuilder(project_id, !mode_fast); + return dupString(result.str()); +} diff --git a/engine/src/engine_index_project.cpp b/engine/src/engine_index_project.cpp index 8014296..0de0ac4 100644 --- a/engine/src/engine_index_project.cpp +++ b/engine/src/engine_index_project.cpp @@ -42,139 +42,14 @@ #include "engine_index_metrics.h" #include "async_knowledge.h" #include "store/store_parse_failure.h" +#include "engine_index_sched.h" +#include "engine_index_discover.h" -// TODO(file-size): This file is ~1675 lines, exceeding the 1000-line -// limit in plan/rules/code_rules.md. Pre-existing debt — the file -// discovery loop, worker pool, and writer thread should be split into -// separate translation units in a follow-up refactor. - -// ─── Dynamic Scheduler Shared State ────────────────────────────── -// When CODESCOPE_SCHED_SHM points to a valid shared-memory file -// created by the Rust scheduler (see server/src/scheduler/shm.rs), -// the engine reads `available_cores` atomically and spawns temporary -// parse threads to grab idle cores from the shared pool. When the env -// var is unset or the file is invalid, the engine falls back to the -// static CODESCOPE_WORKERS allocation — identical to pre-dynamic -// behaviour. The static path is the default for non-scheduler -// invocations (single-process `codescope index`, tests, etc.). -// -// Layout MUST match server/src/scheduler/shm.rs SchedState. The C++ -// side only reads atomic fields and performs a CAS on -// available_cores in grab_cores(); no other writes. -// SAFETY: the scheduler creates and mmap's the file before spawning -// the worker subprocess, so g_sched_state is non-null only inside a -// worker that the scheduler deliberately started. -struct SchedState { - uint32_t magic; // 0x53434844 ("SCHD") - uint32_t version; - std::atomic total_cores; - std::atomic available_cores; - std::atomic active_workers; - std::atomic generation; - std::atomic mem_limit_mb; - std::atomic current_mem_mb; - std::atomic aggressive; - uint32_t worker_count; - uint32_t reserved[8]; - std::atomic worker_status[64]; - std::atomic worker_cores[64]; -}; - -static SchedState *g_sched_state = nullptr; // nullptr = static mode - -// Parse-phase coordination globals. Reset at the start of every -// engine_index_project call; engine_index_files does not use them. -// engine_index_project is invoked sequentially from the FFI thread, -// so the globals are never concurrently re-initialised. -static std::atomic g_parse_done{ false }; -static std::atomic g_active_parse_threads{ 0 }; - -// Try to open the shared memory (env CODESCOPE_SCHED_SHM=path). -// On failure (file missing, too small, magic mismatch, mmap error) -// returns nullptr — silent fallback to static scheduling. No error -// is logged because the static path is the legitimate default. -static SchedState *open_sched_state() -{ -#ifndef _WIN32 - const char *path = getenv("CODESCOPE_SCHED_SHM"); - if (!path || !path[0]) - return nullptr; - fprintf(stderr, - "engine: opening shm path=%s " - "[module=engine, method=open_sched_state]\n", - path); - // Open with O_RDWR: mmap below uses PROT_WRITE for CAS in grab_cores(). - // O_RDONLY + PROT_WRITE mmap fails with EACCES on POSIX, which would - // silently disable dynamic scheduling. - int fd = open(path, O_RDWR); - if (fd < 0) { - fprintf(stderr, - "engine: shm open failed errno=%d path=%s " - "[module=engine, method=open_sched_state]\n", - errno, path); - return nullptr; - } - struct stat st; - if (fstat(fd, &st) != 0 || - st.st_size < static_cast(sizeof(SchedState))) { - close(fd); - return nullptr; - } - // PROT_WRITE is required for the CAS in grab_cores(); the - // scheduler creates the shm file with rw perms for worker procs. - void *addr = mmap(nullptr, sizeof(SchedState), PROT_READ | PROT_WRITE, - MAP_SHARED, fd, 0); - close(fd); - if (addr == MAP_FAILED) - return nullptr; - auto *s = static_cast(addr); - if (s->magic != 0x53434844u) { - munmap(addr, sizeof(SchedState)); - return nullptr; - } - return s; -#else - return nullptr; // mmap-based dynamic scheduling not on Windows -#endif -} - -// Try to grab up to `max_want` cores from the shared pool via CAS. -// Returns the number actually grabbed (0 if none available or static -// mode). The caller must return_cores(count) when done. -static uint32_t grab_cores(uint32_t max_want) -{ - if (!g_sched_state || max_want == 0) - return 0; - while (true) { - uint32_t avail = g_sched_state->available_cores.load( - std::memory_order_relaxed); - if (avail == 0) - return 0; - uint32_t take = std::min(max_want, avail); - if (g_sched_state->available_cores.compare_exchange_weak( - avail, avail - take, std::memory_order_relaxed)) { - return take; - } - } -} - -// Return `count` cores to the shared pool. No-op in static mode. -static void return_cores(uint32_t count) -{ - if (!g_sched_state || count == 0) - return; - g_sched_state->available_cores.fetch_add(count, - std::memory_order_relaxed); -} - -// ─── Constants ───────────────────────────────────────────────── -constexpr uint64_t kMaxFileSize = 5 * 1024 * 1024; // 5 MB default - -// Fail-fast threshold: a file whose parse has failed at least this many -// times is permanently skipped on subsequent index runs. 1 = skip on the -// first failure (strict fail-fast, never retry). Override with the -// CODESCOPE_FAIL_RETRY_MAX env var (clamped to >= 1). -constexpr int kDefaultFailRetryMax = 1; +// Import the shared dynamic-scheduler helpers (SchedState, open_sched_state, +// grab_cores, return_cores, constants) that were split into their own +// translation unit (engine_index_sched.cpp) to keep this file under the +// 1000-line rule (plan/rules/code_rules.md §1). +using namespace engine_index_sched; // ─── Index Project (Parallel) ────────────────────────────────── @@ -262,232 +137,17 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // during discovery (1254 files × ~2ms prepare/finalize = ~2.5s saved). auto scan_state = g_store->loadFileScanStateBatch(project_id); - // Phase 1: collect file paths (single-threaded) - struct FileJob { - std::string path; - std::string lang; - size_t size = 0; - }; - std::vector jobs; - // Track whether this is a re-index (some files already in DB). When - // true, buildGraph uses the incremental path: only cycles the unique - // edge index instead of dropping/recreating all 6 lookup indexes. + // Phase 1: collect file paths (single-threaded). The directory walk, + // README ingestion and incremental scan-state gate now live in + // engine_index_discover.cpp (collectFileJobs) so this TU stays under + // the 1000-line rule. + std::vector jobs; bool is_reindex = false; - - // Pre-detect Java projects BEFORE the directory walk. The FilterPolicy - // Java carve-out defers test/docs/example/samples/... dirs to a - // top-only check ONLY when lang_context_ == "java", but lang_context_ - // previously flipped only upon seeing the FIRST .java file during the - // walk — and that file may itself live under an example/samples/... - // dir which is skipped at any depth while lang_context_ is still - // empty. That chicken-and-egg made Java projects with such package - // dirs index 0 files (e.g. spring-petclinic's - // org/springframework/samples/petclinic). Fix: cheap recursive scan - // for any *.java before the main walk and flip lang_context_ early. - { - std::error_code ec; - auto pit = std::filesystem::recursive_directory_iterator( - dir, - std::filesystem::directory_options::skip_permission_denied, - ec); - std::filesystem::recursive_directory_iterator pend; - while (!ec && pit != pend) { - const auto &pent = *pit; - if (pent.is_regular_file() && - pent.path().extension() == ".java") { - filter.setLangContext("java"); - break; - } - pit.increment(ec); - } - } - - try { - auto it = std::filesystem::recursive_directory_iterator( - dir, std::filesystem::directory_options:: - skip_permission_denied); - for (auto &entry : it) { - filter.stats().seen_dirs++; - std::string rel = entry.path().string(); - if (rel.size() > dir.size() + 1) - rel = rel.substr(dir.size() + 1); - else - rel.clear(); - - // ── README / document ingestion (BEFORE skip filter) ── - // .md files are in skip_suffixes_ (filter_policy.cpp:422) - // so they never reach the source-code indexing path. - // But the knowledge layer (CapabilityPlugin, - // ContractPlugin) needs README content in the - // document table to extract capabilities/contracts. - // Therefore we intercept README.md here — BEFORE - // shouldSkipEntry() drops it — and ingest it via - // insertDocument(). - // - // Only the project-root README is ingested as a - // knowledge document; nested READMEs are ignored - // to avoid noise from vendored deps. - if (entry.is_regular_file()) { - const std::string &fp = entry.path().string(); - std::string fname = fp; - size_t sl = fp.find_last_of("/\\"); - if (sl != std::string::npos) - fname = fp.substr(sl + 1); - // Case-insensitive README.md match - std::string fname_lower = fname; - for (auto &c : fname_lower) - c = static_cast(std::tolower(c)); - - // Check if this README is at project root - bool is_root_readme = false; - if (fname_lower == "readme.md" || - fname_lower == "readme.markdown" || - fname_lower == "readme") { - // Project-root README: its parent dir == dir - std::string parent = fp; - size_t ps = parent.find_last_of("/\\"); - parent = (ps != std::string::npos) ? - parent.substr(0, ps) : - ""; - is_root_readme = (parent == dir); - } - - if (is_root_readme) { - // Ingest README content into document table. - // type=0 (kDocumentTypeReadme) signals the - // knowledge layer to parse capabilities. - std::string content = - readFile(fp.c_str()); - if (!content.empty()) { - int doc_type = - 0; // kDocumentTypeReadme - // insertDocument's 5th/6th params are - // start_line / end_line (1-based line - // numbers), NOT byte offsets. Count - // newlines to compute the line range. - int line_count = 1; - for (char c : content) - if (c == '\n') - ++line_count; - if (!g_store->insertDocument( - project_id, - doc_type, fp, - content, 1, - line_count)) { - fprintf(stderr, - "engine: insertDocument failed for %s: %s " - "[module=engine, method=engine_index_project]\n", - fp.c_str(), - g_store->error() - .c_str()); - } - } - // README is ingested as a document, NOT as - // source code — skip the rest of the loop. - continue; - } - } - - if (!rel.empty()) { - bool entry_is_dir = entry.is_directory(); - // Use the consolidated entry check (single source of - // truth) so the indexer and scanner apply identical - // filtering: skip_dirs (any depth), gitignore, - // .codescopeignore, bundle-dir suffixes, filename skip, - // filename-prefix skip, and suffix skip. - if (filter.shouldSkipEntry(rel, entry_is_dir)) { - if (entry_is_dir) { - it.disable_recursion_pending(); - filter.stats().skipped_dirs++; - } else { - filter.stats().skipped_files++; - } - continue; - } - } - if (entry.is_regular_file()) { - filter.stats().seen_files++; - - // Incremental: check file_scan_state to skip unchanged files - struct stat file_stat; - int64_t mtime = 0, fsize = 0; - bool file_unchanged = false; - if (stat(entry.path().string().c_str(), - &file_stat) == 0) { - mtime = static_cast( - file_stat.st_mtime); - fsize = static_cast( - file_stat.st_size); - // O(1) in-memory lookup instead of per-file DB query. - // M2: two-stage incremental gate. Stage 1 is the cheap - // mtime|size gate (no file read). Only when it matches do - // we hash the file and check the mtime|size|hash gate, - // closing the "same size + same mtime but changed content" - // hole. Files whose mtime/size differ skip without being - // read, so incremental performance is preserved. - std::string base = - entry.path().string() + "|" + - std::to_string(mtime) + "|" + - std::to_string(fsize); - if (scan_state.count(base) > 0) { - std::string ch = fileContentHash( - entry.path() - .string() - .c_str()); - if (!ch.empty()) - file_unchanged = - scan_state.count( - base + - "|" + - ch) > 0; - // If hashing failed (unreadable), fall back to - // treating as unchanged on the mtime|size gate. - else - file_unchanged = true; - } - } - if (file_unchanged) { - is_reindex = true; - filter.stats().skipped_files++; - continue; - } - const char *lang = filter.detectLanguage( - entry.path().string().c_str()); - if (!lang) { - filter.stats().skipped_lang++; - continue; - } - // Detect Java projects on the fly — the FIRST .java - // file flips filter into Java mode so test/docs/samples - // collisions with Java package namespaces (e.g. - // org/springframework/samples/petclinic) get the - // top-only (depth ≤ 3) treatment instead of being - // skipped at any depth. See README.md "Why Java is - // the (only) exception". Idempotent — setLangContext - // is cheap and safe to repeat. - if (strcmp(lang, "java") == 0 && - filter.langContext() != "java") { - filter.setLangContext("java"); - } - if (!filter.isLanguageAccepted(lang)) { - filter.stats().skipped_lang++; - continue; - } - filter.stats().candidate_files++; - auto file_size = - entry.is_regular_file() ? - std::filesystem::file_size( - entry.path()) : - 0; - jobs.push_back({ entry.path().string(), lang, - file_size }); - } - } - } catch (const std::exception &e) { - std::ostringstream err; - err << "{\"ok\":false,\"error\":\"scan error: " - << jsonEscape(e.what()) << "\"}"; - return dupString(err.str()); + std::string discover_err; + if (engine_index_discover::collectFileJobs(project_id, dir, filter, + scan_state, jobs, is_reindex, + discover_err) != 0) { + return dupString(discover_err.c_str()); } if (jobs.empty()) { // No files need (re)indexing. This is either a first index of an @@ -556,7 +216,8 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, // Sort jobs by file size descending — large files first // This reduces tail latency from big files being last in random order. std::sort(jobs.begin(), jobs.end(), - [](const FileJob &a, const FileJob &b) { + [](const engine_index_discover::FileJob &a, + const engine_index_discover::FileJob &b) { return a.size > b.size; }); @@ -997,15 +658,16 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, int num_workers = std::min(static_cast(jobs.size()), static_cast(std::thread::hardware_concurrency())); - // Default to 4 workers to leave CPU cores for other processes. - // Override via CODESCOPE_WORKERS env var (e.g. "8" for 8 workers). + // Default to kDefaultParseWorkers (see constant definition for the + // measurement rationale). Override via the CODESCOPE_WORKERS env var + // (e.g. "16" for 16 workers). const char *env_workers = getenv("CODESCOPE_WORKERS"); if (env_workers && env_workers[0]) { int requested = std::atoi(env_workers); if (requested > 0) num_workers = std::min(requested, num_workers); } else { - num_workers = std::min(num_workers, 4); + num_workers = std::min(num_workers, kDefaultParseWorkers); } if (num_workers < 1) num_workers = 1; @@ -1164,627 +826,3 @@ char *engine_index_project(uint64_t project_id, const char *dir_path, is_reindex, mode_fast, mode_deep, time_parse_ms, 0, total_indexed); } - -// ─── Index File List (Parallel) ────────────────────────────────── -// Takes a pre-computed JSON array of file paths, skips directory scanning. -// Uses the same parallel worker infrastructure as engine_index_project. -// file_list_json: ["/path/to/file1.c", "/path/to/file2.c", ...] -char *engine_index_files(uint64_t project_id, const char *file_list_json) -{ - if (!g_store) - return dupString( - "{\"ok\":false,\"error\":\"engine not initialized\"}"); - - if (!file_list_json || !file_list_json[0]) - return dupString( - "{\"ok\":false,\"error\":\"file_list_json is empty\"}"); - - const char *env_mode = getenv("CODESCOPE_INDEX_MODE"); - bool mode_fast = env_mode && strcmp(env_mode, "fast") == 0; - uint64_t max_file_size = kMaxFileSize; - const char *env_max = getenv("CODESCOPE_MAX_FILE_SIZE"); - if (env_max) - max_file_size = static_cast(std::atoll(env_max)); - - // Parse JSON file list - // Expected format: ["/path/to/file1.c", "/path/to/file2.c", ...] - // Simple parser: comma-separated quoted strings - struct FileJob { - std::string path; - std::string lang; - size_t size = 0; - // mtime captured during discovery so the writer can persist - // it on FileResult (mirrors the streaming path's per-worker - // stat). Without it, result->mtime defaults to 0 and a later - // incremental run's isFileUnchanged can never skip (M-13). - int64_t mtime = 0; - }; - std::vector jobs; - std::string json(file_list_json); - size_t pos = 0; - FilterPolicy filter; - while ((pos = json.find('"', pos)) != std::string::npos) { - size_t end = json.find('"', pos + 1); - if (end == std::string::npos) - break; - std::string path = json.substr(pos + 1, end - pos - 1); - pos = end + 1; - if (path.empty()) - continue; - - // Detect language from file extension - struct stat file_stat; - if (stat(path.c_str(), &file_stat) != 0) - continue; - if (static_cast(file_stat.st_size) > max_file_size) - continue; - - const char *lang = filter.detectLanguage(path.c_str()); - if (!lang) - continue; - - // Detect Java projects on the fly — the FIRST .java file - // flips filter into Java mode so test/docs/samples collisions - // with Java package namespaces (e.g. - // org/springframework/samples/petclinic) get the top-only - // (depth ≤ 3) treatment instead of being skipped at any - // depth. See README.md "Why Java is the (only) exception". - // Idempotent — setLangContext is cheap and safe to repeat. - if (strcmp(lang, "java") == 0 && - filter.langContext() != "java") { - filter.setLangContext("java"); - } - - jobs.push_back({ path, lang, - static_cast(file_stat.st_size), - static_cast(file_stat.st_mtime) }); - } - - if (jobs.empty()) - return dupString( - "{\"ok\":true,\"files_indexed\":0,\"nodes\":0,\"edges\":0,\"errors\":0}"); - - // Fail-fast: pre-load known parse failures so the parse loop can - // skip them without per-file DB queries. Mirrors the logic in - // engine_index_project — kept inline rather than factored out to - // avoid a 1000+ line file (code_rules.md §1). - const int kFailRetryMax = [] { - const char *e = getenv("CODESCOPE_FAIL_RETRY_MAX"); - return e ? std::max(1, std::atoi(e)) : 3; - }(); - std::unordered_set known_failures; - { - std::vector fail_vec; - if (!store::loadKnownParseFailures(project_id, kFailRetryMax, - /*out*/ fail_vec)) { - fprintf(stderr, - "engine: loadKnownParseFailures failed " - "(continuing) " - "[module=engine, method=engine_index_files]\n"); - } else { - known_failures.insert(fail_vec.begin(), fail_vec.end()); - } - } - - // ── Reuse the same parallel processing pipeline ────────────── - // The rest of the function mirrors engine_index_project from the - // job-processing phase onward. Key sections are copied inline to - // avoid a 1000+ line file (see code_rules.md §1). - - // Init progress tracking - { - store::IndexProgress p; - p.project_id = project_id; - p.total_files = static_cast(jobs.size()); - p.phase = 1; - store::setIndexProgress(p); - } - - // Sort by file size descending — large files first - std::sort(jobs.begin(), jobs.end(), - [](const FileJob &a, const FileJob &b) { - return a.size > b.size; - }); - - // Pre-load TSLanguage pointers - std::unordered_map lang_ptrs; - { - std::unordered_set langs; - for (auto &j : jobs) - langs.insert(j.lang); - for (auto &l : langs) - lang_ptrs[l] = g_parser->getLanguage(l.c_str()); - } - - // ── Streaming Pipeline ───────────────────────────────────── - using namespace std::chrono; - steady_clock::time_point t_parse_start; - int64_t time_parse_ms = 0, time_buildgraph_ms = 0; - - const size_t kQueueCapacity = - std::max(2 * std::thread::hardware_concurrency(), 8); - BoundedQueue> result_queue( - kQueueCapacity); - - std::atomic next_job{ 0 }; - std::atomic files_queued{ 0 }; - std::atomic files_written{ 0 }; - std::atomic writer_error{ 0 }; - const int64_t total_files = static_cast(jobs.size()); - const int64_t progress_interval = - std::max(1, total_files / 10); - const size_t kWriterBatchSize = 50; - - // ── Writer thread ────────────────────────────────────────── - std::thread writer_thread([&]() { - g_store->beginTransaction(); - std::vector batch; - batch.reserve(kWriterBatchSize); - while (true) { - std::unique_ptr fr; - bool ok = result_queue.pop(fr); - if (!ok) { - if (!batch.empty()) { - if (!g_store->insertFileResultBatch( - project_id, batch)) { - writer_error = 1; - } - files_written += - static_cast(batch.size()); - } - break; - } - batch.push_back(std::move(*fr)); - fr.reset(); - while (batch.size() < kWriterBatchSize) { - std::unique_ptr extra; - if (!result_queue.pop(extra)) - break; - batch.push_back(std::move(*extra)); - extra.reset(); - } - if (batch.size() >= kWriterBatchSize || - result_queue.isDone()) { - if (!g_store->insertFileResultBatch(project_id, - batch)) { - writer_error = 1; - } - files_written += static_cast(batch.size()); - batch.clear(); - } - } - if (writer_error) - g_store->rollbackTransaction(); - else - g_store->commitTransaction(); - }); - - // ── Parse workers ────────────────────────────────────────── - // Same as engine_index_project: each worker pulls from next_job atomic. - auto parse_worker_fn = [&]() { - struct TSParserDeleter { - void operator()(TSParser *p) const - { - if (p) - ts_parser_delete(p); - } - }; - struct TSTreeDeleter { - void operator()(TSTree *t) const - { - if (t) - ts_tree_delete(t); - } - }; - thread_local static std::unordered_map< - std::string, std::unique_ptr> - tl_parsers; - thread_local static std::unordered_map< - std::string, std::unique_ptr> - tl_visitors; - - // Outermost guard for the static worker thread. Complementing - // the temp-worker try/catch, this catches any unexpected throw - // (e.g. std::bad_alloc from result_queue.push / make_unique) - // so a single bad file cannot escape to std::terminate and - // abort the whole index. The failing job is recorded via - // recordParseFailure with full [module=engine, method=parse_worker_fn] - // tracing; the thread then exits cleanly (it is joined below). - std::string current_path; - std::string current_lang; - try { - while (true) { - int idx = next_job.fetch_add(1); - if (idx >= static_cast(jobs.size())) - break; - auto &job = jobs[idx]; - // Track the in-flight job so an unexpected throw below - // can be recorded against the correct file (M-14). - current_path = job.path; - current_lang = job.lang; - - // Fail-fast: skip files that have failed >= - // CODESCOPE_FAIL_RETRY_MAX times. - if (known_failures.find(job.path) != - known_failures.end()) { - continue; - } - - int done = next_job.load(); - if (done % progress_interval == 0 && done > 0) - fprintf(stderr, - "engine: parse progress %lld/%lld " - "(%d%%) [module=engine, " - "method=engine_index_files]\n", - (long long)done, - (long long)total_files, - (int)(done * 100 / - total_files)); - - std::string source = readFile(job.path.c_str()); - if (source.empty()) { - store::bufferParseFailure( - project_id, job.path, job.lang, - store::failReasonToString( - store::FailReason:: - ReadEmpty)); - continue; - } - - // Per-thread parser - auto pit = tl_parsers.find(job.lang); - if (pit == tl_parsers.end()) { - auto lit = lang_ptrs.find(job.lang); - if (lit == lang_ptrs.end()) { - store::bufferParseFailure( - project_id, job.path, - job.lang, - store::failReasonToString( - store::FailReason:: - LanguageMissing)); - continue; - } - auto np = std::unique_ptr< - TSParser, TSParserDeleter>( - ts_parser_new()); - ts_parser_set_language(np.get(), - lit->second); - tl_parsers[job.lang] = std::move(np); - pit = tl_parsers.find(job.lang); - } - auto tree = - std::unique_ptr( - ts_parser_parse_string( - pit->second.get(), - nullptr, source.c_str(), - static_cast( - source.size()))); - if (!tree) { - store::bufferParseFailure( - project_id, job.path, job.lang, - store::failReasonToString( - store::FailReason:: - ParseNullTree)); - continue; - } - - auto result = - std::make_unique(); - result->file_path = job.path; - result->language = job.lang; - // Persist the mtime captured at discovery time - // (engine_index_files has no per-worker stat). A zero - // mtime would break later isFileUnchanged incremental - // skips (M-13). - result->mtime = job.mtime; - result->fsize = static_cast(job.size); - - // Visitor pipeline - auto vl = tl_visitors.find(job.lang); - ir::JsVisitor *visitor = nullptr; - if (vl == tl_visitors.end()) { - auto v = ir::createJsVisitor( - job.lang.c_str()); - if (v) { - tl_visitors[job.lang] = - std::move(v); - visitor = tl_visitors[job.lang] - .get(); - } - } else { - visitor = vl->second.get(); - visitor->reset(); - } - - if (visitor) { - ir::SemanticUnit *su = nullptr; - try { - su = visitor->visit( - tree.get(), - source.c_str(), - job.path.c_str()); - } catch (const std::exception &e) { - store::bufferParseFailure( - project_id, job.path, - job.lang, - std::string(store::failReasonToString( - store::FailReason:: - VisitorException)) + - ": " + - e.what()); - continue; - } catch (...) { - store::bufferParseFailure( - project_id, job.path, - job.lang, - store::failReasonToString( - store::FailReason:: - VisitorUnknownThrow)); - continue; - } - if (su) { - result->records = - su->allRecords(); - result->metrics = index_metrics:: - computeMetricsFromCST( - tree.get(), - source.c_str(), - result->records); - } - } else { - // Old pipeline fallback - auto translator = ir::createTranslator( - job.lang.c_str()); - if (!translator) { - store::bufferParseFailure( - project_id, job.path, - job.lang, - store::failReasonToString( - store::FailReason:: - LanguageMissing)); - continue; - } - ir::TranslationUnit *unit = nullptr; - try { - unit = translator->translate( - tree.get(), - source.c_str(), - job.path.c_str()); - } catch (const std::exception &e) { - store::bufferParseFailure( - project_id, job.path, - job.lang, - std::string(store::failReasonToString( - store::FailReason:: - VisitorException)) + - ": " + - e.what()); - continue; - } catch (...) { - store::bufferParseFailure( - project_id, job.path, - job.lang, - store::failReasonToString( - store::FailReason:: - VisitorUnknownThrow)); - continue; - } - if (unit && unit->root) { - uint64_t flat_id = 1; - std::function - flatten = [&](ir::Node * - n, - uint64_t - parent) { - uint64_t my_id = - flat_id++; - ir::Record rec; - rec.id = my_id; - rec.kind = static_cast< - ir::RecordKind>( - static_cast< - int>( - n->kind)); - rec.name = - n->name; - rec.qualified_name = - n->qualified_name; - rec.parent_id = - parent; - rec.loc.start_row = - n->loc.start_row; - rec.loc.start_col = - n->loc.start_col; - rec.loc.end_row = - n->loc.end_row; - rec.loc.end_col = - n->loc.end_col; - rec.file_path = - job.path; - result->records.push_back( - std::move( - rec)); - for (auto *c : - n->children) - flatten(c, - my_id); - }; - flatten(unit->root, 0); - } - } - - result_queue.push(std::move(result)); - files_queued++; - } - } catch (const std::exception &e) { - // An unexpected exception escaped the per-call guarded - // sections (visit()/translate() are individually - // guarded). Record the failing job and log with full - // tracing instead of letting it reach std::terminate - // (M-14). The worker thread then exits; it is joined - // below so the index finishes gracefully. - if (!current_path.empty()) { - store::bufferParseFailure( - project_id, current_path, current_lang, - std::string("unexpected: ") + e.what()); - } - fprintf(stderr, - "engine: static parse worker aborted: %s " - "[module=engine, method=parse_worker_fn]\n", - e.what()); - } catch (...) { - fprintf(stderr, - "engine: static parse worker aborted: " - "unknown exception " - "[module=engine, method=parse_worker_fn]\n"); - } - }; - - // ── Spawn workers ────────────────────────────────────────── - t_parse_start = steady_clock::now(); - int num_workers = - std::min(static_cast(jobs.size()), - static_cast(std::thread::hardware_concurrency())); - const char *env_workers = getenv("CODESCOPE_WORKERS"); - if (env_workers && env_workers[0]) { - int requested = std::atoi(env_workers); - if (requested > 0) - num_workers = std::min(requested, num_workers); - } else { - num_workers = std::min(num_workers, 4); - } - if (num_workers < 1) - num_workers = 1; - - std::vector workers; - for (int i = 0; i < num_workers; i++) { - try { - workers.emplace_back(parse_worker_fn); - } catch (const std::system_error &e) { - fprintf(stderr, - "engine: thread spawn failed: %s " - "[module=engine, method=engine_index_files]\n", - e.what()); - } - } - for (auto &t : workers) { - if (t.joinable()) - t.join(); - } - result_queue.markDone(); - if (writer_thread.joinable()) - writer_thread.join(); - time_parse_ms = - duration_cast(steady_clock::now() - t_parse_start) - .count(); - - // ── Build graph ──────────────────────────────────────────── - if (writer_error == 0) { - t_parse_start = steady_clock::now(); - // buildGraph(...true) is a FULL rebuild: it drops the lookup - // + unique-edge indexes. Unlike engine_index_project (which - // reaches this via engine_index_post_parse), this path must - // recreate those indexes itself or they stay missing (M-12). - // P2 fix: a resolver-pipeline failure makes buildGraph roll back - // its graph savepoint and return false; flag it as a writer error - // so the result JSON reports failure instead of a false success - // (the outer transaction is rolled back by the caller when it - // sees ok:false). - if (!g_store->buildGraph(project_id, true)) { - writer_error = 1; - } - time_buildgraph_ms = - duration_cast(steady_clock::now() - - t_parse_start) - .count(); - - // Mark every node callgraph_ready: the call graph is now - // committed, so trace_path / enhancement-status report - // readiness correctly (mirrors engine_index_post_parse, M-15). - { - std::string up = - "UPDATE graph_nodes SET callgraph_ready=1 " - "WHERE project_id=" + - std::to_string(project_id); - if (!g_store->exec(up.c_str())) { - fprintf(stderr, - "engine_index_files: callgraph_ready " - "UPDATE failed: %s " - "[module=engine, method=engine_index_files]\n", - g_store->error().c_str()); - } - } - - // Recreate lookup + unique-edge indexes dropped by the full - // rebuild. full_rebuild=true (buildGraph did a full rebuild, - // not an incremental run) matches the post-parse call - // createIndexesAfterBulkLoad(project_id, !is_reindex) with - // is_reindex=false (M-12). - { - store::GraphStore::BulkPragmaGuard guard(g_store.get()); - auto t_idx = steady_clock::now(); - g_store->createIndexesAfterBulkLoad(project_id, true); - fprintf(stderr, - "engine: createIndexesAfterBulkLoad=%lldms " - "[module=engine, method=engine_index_files]\n", - (long long)duration_cast( - steady_clock::now() - t_idx) - .count()); - } - - // Set the core-graph readiness flag so the project is - // queryable immediately after a file-list index (M-15). - g_store->setProjectReadiness(project_id, "normal_ready", 1); - } - - // ── Build result JSON ────────────────────────────────────── - std::ostringstream result; - result << "{\"ok\":" << (writer_error == 0 ? "true" : "false") - << ",\"files_indexed\":" << files_written.load() - << ",\"workers\":" << num_workers - << ",\"time_parse_ms\":" << time_parse_ms - << ",\"time_buildgraph_ms\":" << time_buildgraph_ms; - - // Query node/edge counts - { - sqlite3 *db = g_store->handle(); - sqlite3_stmt *stmt = nullptr; - std::string sql = - "SELECT COUNT(*) FROM entity WHERE project_id = " + - std::to_string(project_id); - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == - SQLITE_OK) { - if (sqlite3_step(stmt) == SQLITE_ROW) - result << ",\"total_nodes\":" - << sqlite3_column_int64(stmt, 0); - sqlite3_finalize(stmt); - } - sql = "SELECT COUNT(*) FROM relation WHERE project_id = " + - std::to_string(project_id); - if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) == - SQLITE_OK) { - if (sqlite3_step(stmt) == SQLITE_ROW) - result << ",\"total_edges\":" - << sqlite3_column_int64(stmt, 0); - sqlite3_finalize(stmt); - } - } - - result << ",\"discovery\":{\"candidate_files\":" << jobs.size() << "}" - << "}"; - - // Mark progress as done - { - store::IndexProgress p; - p.project_id = project_id; - p.total_files = static_cast(jobs.size()); - p.current_file = static_cast(jobs.size()); - p.phase = 5; - p.percent = 100; - store::setIndexProgress(p); - } - - launchAsyncKnowledgeBuilder(project_id, !mode_fast); - return dupString(result.str()); -} diff --git a/engine/src/engine_index_sched.cpp b/engine/src/engine_index_sched.cpp new file mode 100644 index 0000000..89b0fe7 --- /dev/null +++ b/engine/src/engine_index_sched.cpp @@ -0,0 +1,101 @@ +#include "engine_index_sched.h" + +#include +#include +#include +#include + +#ifndef _WIN32 +#include +#include +#include +#include +#endif + +namespace engine_index_sched +{ + +SchedState *g_sched_state = nullptr; // nullptr = static mode + +std::atomic g_parse_done{ false }; +std::atomic g_active_parse_threads{ 0 }; + +// Try to open the shared memory (env CODESCOPE_SCHED_SHM=path). +// On failure (file missing, too small, magic mismatch, mmap error) +// returns nullptr — silent fallback to static scheduling. No error +// is logged because the static path is the legitimate default. +SchedState *open_sched_state() +{ +#ifndef _WIN32 + const char *path = getenv("CODESCOPE_SCHED_SHM"); + if (!path || !path[0]) + return nullptr; + fprintf(stderr, + "engine: opening shm path=%s " + "[module=engine, method=open_sched_state]\n", + path); + // Open with O_RDWR: mmap below uses PROT_WRITE for CAS in grab_cores(). + // O_RDONLY + PROT_WRITE mmap fails with EACCES on POSIX, which would + // silently disable dynamic scheduling. + int fd = open(path, O_RDWR); + if (fd < 0) { + fprintf(stderr, + "engine: shm open failed errno=%d path=%s " + "[module=engine, method=open_sched_state]\n", + errno, path); + return nullptr; + } + struct stat st; + if (fstat(fd, &st) != 0 || + st.st_size < static_cast(sizeof(SchedState))) { + close(fd); + return nullptr; + } + // PROT_WRITE is required for the CAS in grab_cores(); the + // scheduler creates the shm file with rw perms for worker procs. + void *addr = mmap(nullptr, sizeof(SchedState), PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + close(fd); + if (addr == MAP_FAILED) + return nullptr; + auto *s = static_cast(addr); + if (s->magic != 0x53434844u) { + munmap(addr, sizeof(SchedState)); + return nullptr; + } + return s; +#else + return nullptr; // mmap-based dynamic scheduling not on Windows +#endif +} + +// Try to grab up to `max_want` cores from the shared pool via CAS. +// Returns the number actually grabbed (0 if none available or static +// mode). The caller must return_cores(count) when done. +uint32_t grab_cores(uint32_t max_want) +{ + if (!g_sched_state || max_want == 0) + return 0; + while (true) { + uint32_t avail = g_sched_state->available_cores.load( + std::memory_order_relaxed); + if (avail == 0) + return 0; + uint32_t take = std::min(max_want, avail); + if (g_sched_state->available_cores.compare_exchange_weak( + avail, avail - take, std::memory_order_relaxed)) { + return take; + } + } +} + +// Return `count` cores to the shared pool. No-op in static mode. +void return_cores(uint32_t count) +{ + if (!g_sched_state || count == 0) + return; + g_sched_state->available_cores.fetch_add(count, + std::memory_order_relaxed); +} + +} // namespace engine_index_sched diff --git a/engine/src/engine_index_sched.h b/engine/src/engine_index_sched.h new file mode 100644 index 0000000..659933b --- /dev/null +++ b/engine/src/engine_index_sched.h @@ -0,0 +1,89 @@ +#ifndef ENGINE_INDEX_SCHED_H +#define ENGINE_INDEX_SCHED_H + +#include +#include + +namespace engine_index_sched +{ + +// ─── Dynamic Scheduler Shared State ────────────────────────────── +// When CODESCOPE_SCHED_SHM points to a valid shared-memory file +// created by the Rust scheduler (see server/src/scheduler/shm.rs), +// the engine reads `available_cores` atomically and spawns temporary +// parse threads to grab idle cores from the shared pool. When the env +// var is unset or the file is invalid, the engine falls back to the +// static CODESCOPE_WORKERS allocation — identical to pre-dynamic +// behaviour. The static path is the default for non-scheduler +// invocations (single-process `codescope index`, tests, etc.). +// +// Layout MUST match server/src/scheduler/shm.rs SchedState. The C++ +// side only reads atomic fields and performs a CAS on +// available_cores in grab_cores(); no other writes. +// SAFETY: the scheduler creates and mmap's the file before spawning +// the worker subprocess, so g_sched_state is non-null only inside a +// worker that the scheduler deliberately started. +struct SchedState { + uint32_t magic; // 0x53434844 ("SCHD") + uint32_t version; + std::atomic total_cores; + std::atomic available_cores; + std::atomic active_workers; + std::atomic generation; + std::atomic mem_limit_mb; + std::atomic current_mem_mb; + std::atomic aggressive; + uint32_t worker_count; + uint32_t reserved[8]; + std::atomic worker_status[64]; + std::atomic worker_cores[64]; +}; + +// Global shared-memory handle; nullptr = static mode. Owned by the +// scheduler (mmap'd file), never freed by the engine. +extern SchedState *g_sched_state; + +// Parse-phase coordination globals. Reset at the start of every +// engine_index_project call; engine_index_files does not use them. +// engine_index_project is invoked sequentially from the FFI thread, +// so the globals are never concurrently re-initialised. +extern std::atomic g_parse_done; +extern std::atomic g_active_parse_threads; + +// Try to open the shared memory (env CODESCOPE_SCHED_SHM=path). +// On failure (file missing, too small, magic mismatch, mmap error) +// returns nullptr — silent fallback to static scheduling. No error +// is logged because the static path is the legitimate default. +SchedState *open_sched_state(); + +// Try to grab up to `max_want` cores from the shared pool via CAS. +// Returns the number actually grabbed (0 if none available or static +// mode). The caller must return_cores(count) when done. +uint32_t grab_cores(uint32_t max_want); + +// Return `count` cores to the shared pool. No-op in static mode. +void return_cores(uint32_t count); + +// ─── Constants ───────────────────────────────────────────────── +constexpr uint64_t kMaxFileSize = 5 * 1024 * 1024; // 5 MB default + +// Fail-fast threshold: a file whose parse has failed at least this many +// times is permanently skipped on subsequent index runs. 1 = skip on the +// first failure (strict fail-fast, never retry). Override with the +// CODESCOPE_FAIL_RETRY_MAX env var (clamped to >= 1). +constexpr int kDefaultFailRetryMax = 1; + +// Default number of parse workers used by engine_index_project when +// CODESCOPE_WORKERS is unset. The parse phase (tree-sitter + visitor + +// metrics) is pure CPU and does not touch SQLite (a single writer thread +// batches inserts), so on a 14-core machine 4 workers wasted ~70% of the +// cores; raising to 8 dropped the rustc full-index wall clock from ~42s to +// ~34.5s (-18%). 8 is the measured sweet spot: 8 -> 12 gives no further +// gain because the SQLite writer thread becomes the throughput ceiling, and +// it still leaves cores for other processes. Users can still override via +// the CODESCOPE_WORKERS env var. +constexpr int kDefaultParseWorkers = 8; + +} // namespace engine_index_sched + +#endif // ENGINE_INDEX_SCHED_H diff --git a/engine/src/filter_policy.cpp b/engine/src/filter_policy.cpp index 2f33b83..782b98d 100644 --- a/engine/src/filter_policy.cpp +++ b/engine/src/filter_policy.cpp @@ -235,11 +235,29 @@ FilterPolicy::FilterPolicy() "external", }; - // FAST mode skips even more — reserved for future FAST-only - // additions. test/, docs/, vendor/, bench/ are already in - // normal_skip_dirs_ so NORMAL mode skips them; FAST mode skips - // everything NORMAL skips (plus anything added here). - fast_extra_skip_dirs_ = {}; + // FAST mode skips even more — build/test artifacts that NORMAL + // mode keeps (they are generated, rarely the focus of analysis, + // and can dominate file counts on large repos). Everything in + // normal_skip_dirs_ is skipped in both modes; this set is merged + // into active_skip_dirs_ only when mode_ == FAST/STRICT (see + // buildActiveSets()). + fast_extra_skip_dirs_ = { + // ── Frontend build output & generated code ── + ".output", // Next.js/Remix/Astro build output + "storybook-static", // Storybook static build + "__generated__", // GraphQL/Prisma/typed codegen output + // ── Test reports (large, machine-generated) ── + "playwright-report", + "test-results", + "allure-results", + "allure-report", + // ── CSS preprocessor caches ── + ".sass-cache", + ".scss-cache", + // ── Runtime logs ── + "logs", + ".logs", + }; // ── Directory prefixes — catches build_test, build_master, etc. ── skip_dir_prefixes_ = { @@ -480,6 +498,22 @@ FilterPolicy::FilterPolicy() ".min.css", }; + // FAST mode extra exact filenames — linter/formatter/build caches + // that NORMAL mode keeps. These are single files (not suffixes), so + // they can't go into fast_extra_suffixes_; checked in + // shouldSkipFile() only when mode_ == FAST. + fast_extra_filenames_ = { + ".eslintcache", // ESLint incremental cache + ".stylelintcache", // Stylelint cache + ".prettiercache", // Prettier cache + "tsconfig.tsbuildinfo", // TypeScript incremental build info + }; + + // FAST mode extra filename prefixes (e.g. build-info.*). Empty for + // now — reserved for future additions; kept symmetric with the other + // fast_extra_* sets so the FAST path is uniform. + fast_extra_filename_prefixes_ = {}; + // Directory suffixes — bundle / package / IDE project DIRECTORIES. // Matched case-insensitively against the directory's basename so // "Foo.app", "Foo.APP" and "GLFW.framework" are all skipped. @@ -581,6 +615,8 @@ FilterPolicy::FilterPolicy() lowercaseAll(fast_extra_skip_dirs_); lowercaseAll(skip_suffixes_); lowercaseAll(fast_extra_suffixes_); + lowercaseAll(fast_extra_filenames_); + lowercaseAll(fast_extra_filename_prefixes_); lowercaseAll(skip_dir_suffixes_); lowercaseAll(skip_filenames_); lowercaseAll(skip_filename_prefixes_); @@ -625,7 +661,12 @@ FilterPolicy::FilterPolicy() void FilterPolicy::buildActiveSets() { active_skip_dirs_ = normal_skip_dirs_; - if (mode_ == FAST || mode_ == STRICT) { + if (mode_ == FAST) { + // fast_extra_skip_dirs_ is FAST-exclusive: STRICT mode keeps its + // whitelist-only semantics (detectLanguage gate) and must NOT + // silently drop these dirs before the whitelist check. This + // matches the FAST-only gating of fast_extra_filenames_ / + // fast_extra_filename_prefixes_ in shouldSkipFile(). active_skip_dirs_.insert(fast_extra_skip_dirs_.begin(), fast_extra_skip_dirs_.end()); } @@ -714,6 +755,17 @@ bool FilterPolicy::shouldSkipFile(const std::string &filename) const c = static_cast(std::tolower(c)); if (skip_filenames_.find(lower) != skip_filenames_.end()) return true; + // FAST-only exact filenames (linter/build caches, tsbuildinfo). + if (mode_ == FAST) { + if (fast_extra_filenames_.find(lower) != + fast_extra_filenames_.end()) + return true; + for (const auto &pfx : fast_extra_filename_prefixes_) { + if (lower.size() >= pfx.size() && + lower.compare(0, pfx.size(), pfx) == 0) + return true; + } + } // Prefix check — catches .env.local, .env.production, etc. for (const auto &pfx : skip_filename_prefixes_) { if (lower.size() >= pfx.size() && diff --git a/engine/src/filter_policy.h b/engine/src/filter_policy.h index 64454c8..8e44522 100644 --- a/engine/src/filter_policy.h +++ b/engine/src/filter_policy.h @@ -46,6 +46,12 @@ class FilterPolicy { void setMode(Mode m) { mode_ = m; + // active_skip_dirs_ is mode-dependent (FAST/STRICT add + // fast_extra_skip_dirs_). Rebuild it so a mode switch made + // AFTER construction (e.g. from CODESCOPE_INDEX_MODE env) is + // reflected immediately; otherwise fast_extra_skip_dirs_ would + // silently never take effect. + buildActiveSets(); } Mode mode() const { @@ -206,6 +212,12 @@ class FilterPolicy { std::unordered_set skip_suffixes_; // FAST mode extra suffixes std::unordered_set fast_extra_suffixes_; + // FAST mode extra exact filenames (e.g. linter caches) — checked + // in shouldSkipFile() only when mode_ == FAST. + std::unordered_set fast_extra_filenames_; + // FAST mode extra filename prefixes — checked in shouldSkipFile() + // only when mode_ == FAST. + std::unordered_set fast_extra_filename_prefixes_; // Skip dir suffixes — directory ENTRIES whose name ends with one of // these (e.g. MyApp.app, GLFW.framework, proj.xcodeproj). These are // bundle/package directories on macOS / IDE project dirs that must be diff --git a/engine/src/model/state_builder.cpp b/engine/src/model/state_builder.cpp index e24ea19..9a3f0a6 100644 --- a/engine/src/model/state_builder.cpp +++ b/engine/src/model/state_builder.cpp @@ -27,23 +27,68 @@ int64_t StateBuilder::buildModuleSummaries() // - entry_reachable: MAX(graph_nodes.is_entry_point) — does this // module contain a main/init/setup/run/handler? // Rules match by PRIORITY (first hit stops, see role_classifier_plan.md). + // Split the aggregate and the graph_nodes entry_reachable scan into + // two CTEs. A single 6-table LEFT JOIN that combined the three + // relation joins with graph_nodes made SQLite build four COUNT(DISTINCT) + // temp B-trees over a blown-up intermediate result (~29s on a 26k-node + // Go tree). Isolating the graph_nodes join into its own CTE — joined + // only on module_id after both aggregates finish — drops the cost to + // <0.2s with identical results. INDEXED BY forces the right index for + // each relation join; SQLite otherwise picks idx_relation_unique_typed + // (keyed on source_id) for the target_id lookup, scanning the whole + // relation table per entity (~46x slower). + // + // v0.7 (perf): r_in and r_tgt were two separate LEFT JOINs on the same + // (project_id, target_id=e.id) index — r_in additionally filtered + // source_id != e.id. On rustc (117k relations x 129k entities) that + // duplicated the target-side scan and cost ~5.95s for 988 module rows. + // Merging them into a single r JOIN (same index) and moving the + // self-loop exclusion into the incoming/dead CASE expressions is + // result-identical (verified: EXCEPT-diff both directions == 0) and + // drops the phase to ~0.25s (23.8x). std::string sql = + "WITH agg AS (" + " SELECT s.id AS module_id, s.name AS module_name, " + " COUNT(DISTINCT e.id) AS total, " + " COUNT(DISTINCT CASE WHEN r.source_id != e.id " + " THEN r.source_id END) AS incoming, " + " COUNT(DISTINCT r_out.target_id) AS outgoing, " + " COUNT(DISTINCT e.id) - COUNT(DISTINCT r.target_id) " + " AS dead, " + " COUNT(DISTINCT CASE WHEN e.visibility = 1 THEN e.id END) " + " AS pub_count, " + " CASE WHEN COUNT(DISTINCT e.id) > 0 " + " THEN 1.0 - CAST(COUNT(DISTINCT e.id) - " + " COUNT(DISTINCT r.target_id) AS REAL) / " + " COUNT(DISTINCT e.id) ELSE 0.0 END AS utilization " + " FROM scope s " + " JOIN entity e ON e.project_id = ? AND e.module_path = s.name " + " LEFT JOIN relation r INDEXED BY idx_relation_target " + " ON r.project_id = ? AND r.target_id = e.id " + " LEFT JOIN relation r_out INDEXED BY idx_relation_source " + " ON r_out.project_id = ? AND r_out.source_id = e.id " + " AND r_out.target_id != e.id " + " WHERE s.kind = 1 AND s.project_id = ? " + " GROUP BY s.id, s.name " + "), entry AS (" + " SELECT s.id AS module_id, " + " MAX(COALESCE(gn.is_entry_point, 0)) AS entry_reachable " + " FROM scope s " + " JOIN entity e ON e.project_id = ? AND e.module_path = s.name " + " LEFT JOIN graph_nodes gn ON gn.project_id = ? " + " AND gn.name = e.name AND gn.file_path = e.file_path " + " WHERE s.kind = 1 AND s.project_id = ? " + " GROUP BY s.id " + ") " "INSERT OR REPLACE INTO module_summary " "(project_id, module_id, state, incoming_count, outgoing_count, " " internal_edges, dead_entities, utilization, confidence, role) " - "SELECT ?, module_id, 0, incoming, outgoing, 0, dead, " + "SELECT ?, agg.module_id, 0, incoming, outgoing, 0, dead, " " CASE WHEN total > 0 " " THEN 1.0 - CAST(dead AS REAL) / total ELSE 0.0 END, " " 0.85, " " CASE " // Priority 1: test layer — strong path signal. - // Match "test" / "tests" as a full path component only, NOT as - // a substring. INSTR(module_name, 'test') matched "latest", - // "attestation", "protest", etc., misclassifying real source - // modules as test layers. module_name is a directory path - // (e.g. "src/test/", "test/", "src/utils/test/"), so path- - // component LIKE patterns cover all positions without matching - // substrings inside other words. " WHEN module_name = 'test' " " OR module_name LIKE 'test/%' " " OR module_name LIKE '%/test' " @@ -53,7 +98,6 @@ int64_t StateBuilder::buildModuleSummaries() " OR module_name LIKE '%/tests' " " OR module_name LIKE '%/tests/%' THEN 'test' " // Priority 2: api layer — pub surface + cross-module called heavily - // Thresholds from state_builder.h constexpr (kRoleApi*), retunable. " WHEN pub_count > 0 AND incoming >= " + std::to_string(kRoleApiIncomingOutgoingRatio) + " * outgoing " @@ -65,9 +109,7 @@ int64_t StateBuilder::buildModuleSummaries() " THEN 'api' " // Priority 3: entry layer — contains a main/init/setup/run/handler " WHEN entry_reachable > 0 THEN 'entry' " - // Priority 4: core hub — many depend on it, self deps low, utilized, - // has pub surface (中枢). Thresholds from state_builder.h - // constexpr (kRoleCore*), retunable. + // Priority 4: core hub " WHEN incoming >= " + std::to_string(kRoleCoreIncomingMin) + " " @@ -78,67 +120,24 @@ int64_t StateBuilder::buildModuleSummaries() std::to_string(kRoleCoreUtilizationMin) + " " " AND pub_count > 0 THEN 'core' " - // Priority 5: utility layer — called by others, has pub, few deps - // Thresholds from state_builder.h constexpr (kRoleUtility*). + // Priority 5: utility layer " WHEN outgoing <= " + std::to_string(kRoleUtilityOutgoingMax) + " AND pub_count > 0 " " AND utilization >= " + std::to_string(kRoleUtilityUtilizationMin) + " THEN 'utility' " - // Priority 6: business layer — implementation: many depend on it AND - // it depends on many (high outgoing). Not core (outgoing too - // high), not api (outgoing too high), but clearly not infra. - // Rescues modules like bun's src/jsc/bindings (pub=3466, - // incoming=2360, outgoing=1794, util=0.38) from infra兜底. + // Priority 6: business layer " WHEN pub_count > 0 AND incoming >= " + std::to_string(kRoleBusinessIncomingMin) + " THEN 'business' " // Priority 7: dead/leaf — no calls in or out, or all entities dead " WHEN (incoming = 0 AND outgoing = 0) " " OR dead = total THEN 'dead' " - // Priority 8: infra — true fallback (didn't match any semantic rule) + // Priority 8: infra — true fallback " ELSE 'infra' END " - "FROM (" - " SELECT s.id AS module_id, s.name AS module_name, " - " COUNT(DISTINCT e.id) AS total, " - " COUNT(DISTINCT r_in.source_id) AS incoming, " - " COUNT(DISTINCT r_out.target_id) AS outgoing, " - " COUNT(DISTINCT e.id) - COUNT(DISTINCT r_tgt.target_id) " - " AS dead, " - // pub_count: entity.visibility=1 (pub/public/export). visibility is - // populated by Visitors per language (pub→1, private→0). When the - // migration hasn't run yet visibility defaults to 0, making - // pub_count=0 — api/core/utility rules won't fire, role degrades - // gracefully to test/entry/dead/infra (still better than v0.2.1). - " COUNT(DISTINCT CASE WHEN e.visibility = 1 THEN e.id END) " - " AS pub_count, " - // entry_reachable: MAX(graph_nodes.is_entry_point) across the module - // — 1 if any node in the module is an entry point. Uses graph_nodes - // which is populated during enhance. When enhance hasn't run, - // is_entry_point defaults to 0 — entry rule won't fire, graceful. - " MAX(COALESCE(gn.is_entry_point, 0)) AS entry_reachable, " - " CASE WHEN COUNT(DISTINCT e.id) > 0 " - " THEN 1.0 - CAST(COUNT(DISTINCT e.id) - " - " COUNT(DISTINCT r_tgt.target_id) AS REAL) / " - " COUNT(DISTINCT e.id) ELSE 0.0 END AS utilization " - " FROM scope s " - " JOIN entity e ON e.project_id = ? AND e.module_path = s.name " - " LEFT JOIN relation r_in ON r_in.project_id = ? " - " AND r_in.target_id = e.id AND r_in.source_id != e.id " - " LEFT JOIN relation r_out ON r_out.project_id = ? " - " AND r_out.source_id = e.id AND r_out.target_id != e.id " - " LEFT JOIN relation r_tgt ON r_tgt.project_id = ? " - " AND r_tgt.target_id = e.id " - // graph_nodes JOIN for entry_reachable — LEFT JOIN so modules - // without graph_nodes (enhance not run) still appear, with - // entry_reachable=0 via COALESCE. - " LEFT JOIN graph_nodes gn ON gn.project_id = ? " - " AND gn.name = e.name AND gn.file_path = e.file_path " - " WHERE s.kind = 1 AND s.project_id = ? " - " GROUP BY s.id, s.name " - " HAVING total >= 3" - ")"; + "FROM agg LEFT JOIN entry ON entry.module_id = agg.module_id " + "WHERE agg.total >= 3"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(store_->handle(), sql.c_str(), -1, &stmt, nullptr) != SQLITE_OK) { @@ -148,7 +147,8 @@ int64_t StateBuilder::buildModuleSummaries() sqlite3_errmsg(store_->handle())); return -1; } - for (int i = 1; i <= 7; i++) + // Bind order: agg (4) + entry (3) + SELECT (1) = 8 ? params. + for (int i = 1; i <= 8; i++) sqlite3_bind_int64(stmt, i, static_cast(project_id_)); int rc = sqlite3_step(stmt); diff --git a/engine/src/query/query_analysis.cpp b/engine/src/query/query_analysis.cpp index 7fd217f..86d0bcb 100644 --- a/engine/src/query/query_analysis.cpp +++ b/engine/src/query/query_analysis.cpp @@ -426,9 +426,10 @@ std::string QueryEngine::getProjectOverview(uint64_t project_id) // Stats { sqlite3_stmt *stmt = nullptr; + // v0.2.6: count from the canonical entity/relation tables. + // graph_nodes/graph_edges are deprecated and no longer written. sqlite3_prepare_v2( - db, - "SELECT COUNT(*) FROM graph_nodes WHERE project_id=?", + db, "SELECT COUNT(*) FROM entity WHERE project_id=?", -1, &stmt, nullptr); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); if (sqlite3_step(stmt) == SQLITE_ROW) { @@ -438,8 +439,7 @@ std::string QueryEngine::getProjectOverview(uint64_t project_id) sqlite3_finalize(stmt); sqlite3_prepare_v2( - db, - "SELECT COUNT(*) FROM graph_edges WHERE project_id=?", + db, "SELECT COUNT(*) FROM relation WHERE project_id=?", -1, &stmt, nullptr); sqlite3_bind_int64(stmt, 1, static_cast(project_id)); if (sqlite3_step(stmt) == SQLITE_ROW) { @@ -581,14 +581,24 @@ std::string QueryEngine::getGraph(uint64_t project_id, int64_t node_offset, "[module=QueryEngine, method=getGraph]\"}"; } + // ── v0.2.6: query the canonical tables. graph_nodes/graph_edges are + // deprecated and no longer written (store_graph.cpp removed the + // dual-write; only graph_edges accumulates stale legacy rows). The + // canonical graph is entity (nodes) + relation (edges), matching + // getGraphStats. entity.kind is the RecordKind enum (0=Function, + // 1=Method, 2=Class, 3=Interface, 4=Enum, ... 14=File); relation.type + // is the edge kind (1=Calls). Alias columns back to the graph_nodes / + // graph_edges names so the public JSON schema is unchanged for + // callers that already parse source_node_id / target_node_id / + // edge_type / node_type. std::string node_filter_clause; if (node_type_filter && *node_type_filter) { - node_filter_clause = " AND gn.node_type IN (" + + node_filter_clause = " AND e.kind IN (" + std::string(node_type_filter) + ")"; } std::string edge_filter_clause; if (edge_type_filter && *edge_type_filter) { - edge_filter_clause = " AND ge.edge_type IN (" + + edge_filter_clause = " AND r.type IN (" + std::string(edge_type_filter) + ")"; } @@ -598,7 +608,7 @@ std::string QueryEngine::getGraph(uint64_t project_id, int64_t node_offset, int64_t total_edges = 0; { std::string sql = - "SELECT COUNT(*) FROM graph_nodes gn WHERE gn.project_id = ?" + + "SELECT COUNT(*) FROM entity e WHERE e.project_id = ?" + node_filter_clause; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != @@ -614,7 +624,7 @@ std::string QueryEngine::getGraph(uint64_t project_id, int64_t node_offset, } { std::string sql = - "SELECT COUNT(*) FROM graph_edges ge WHERE ge.project_id = ?" + + "SELECT COUNT(*) FROM relation r WHERE r.project_id = ?" + edge_filter_clause; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != @@ -636,8 +646,14 @@ std::string QueryEngine::getGraph(uint64_t project_id, int64_t node_offset, // Paginated node page. { std::string sql = - "SELECT gn.* FROM graph_nodes gn WHERE gn.project_id = ?" + - node_filter_clause + " ORDER BY gn.id LIMIT ? OFFSET ?"; + "SELECT e.id AS id, e.project_id AS project_id, " + "e.kind AS node_type, e.name AS name, " + "e.qualified_name AS qualified_name, e.file_path AS file_path, " + "e.language AS language, e.start_row AS start_row, " + "e.start_col AS start_col, e.end_row AS end_row, " + "e.end_col AS end_col, e.module_path AS module_path " + "FROM entity e WHERE e.project_id = ?" + + node_filter_clause + " ORDER BY e.id LIMIT ? OFFSET ?"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != SQLITE_OK) { @@ -667,8 +683,13 @@ std::string QueryEngine::getGraph(uint64_t project_id, int64_t node_offset, // Paginated edge page. { std::string sql = - "SELECT ge.* FROM graph_edges ge WHERE ge.project_id = ?" + - edge_filter_clause + " ORDER BY ge.id LIMIT ? OFFSET ?"; + "SELECT r.id AS id, r.source_id AS source_node_id, " + "r.target_id AS target_node_id, r.type AS edge_type, " + "r.confidence AS confidence, r.call_site_file AS call_site_file, " + "r.call_site_row AS call_site_line, " + "r.call_site_col AS call_site_col " + "FROM relation r WHERE r.project_id = ?" + + edge_filter_clause + " ORDER BY r.id LIMIT ? OFFSET ?"; sqlite3_stmt *stmt = nullptr; if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, nullptr) != SQLITE_OK) { diff --git a/engine/src/resolver/factors.cpp b/engine/src/resolver/factors.cpp index ca311f1..5d7ea7c 100644 --- a/engine/src/resolver/factors.cpp +++ b/engine/src/resolver/factors.cpp @@ -9,57 +9,6 @@ namespace resolver namespace { -/// Fold an ASCII byte to lowercase. SQLite's default LIKE folds only -/// ASCII upper-case letters; all other bytes (including non-ASCII) are -/// returned unchanged so they compare byte-for-byte, matching SQLite. -inline unsigned char likeFold(unsigned char ch) -{ - if (ch >= 'A' && ch <= 'Z') - return static_cast(ch + ('a' - 'A')); - return ch; -} - -/// Replicate SQLite's default LIKE matching for a full-string pattern. -/// '%' matches any sequence (including empty), '_' matches any single -/// character, and ASCII letters compare case-insensitively. The match -/// is anchored to the whole text (LIKE is not a substring search; the -/// surrounding '%' in callers' patterns provides prefix/suffix freedom). -/// -/// This is the standard greedy-with-backtrack wildcard matcher. It is -/// used instead of std::string::find so that '_'/'%' inside a module -/// name and ASCII case differences behave EXACTLY like the original -/// `target_path LIKE '%module_name%'` SQL — preserving identical edges. -bool sqliteLikeMatch(const std::string &pattern, const std::string &text) -{ - size_t p = 0; // pattern cursor - size_t t = 0; // text cursor - size_t star_p = std::string::npos; // position of last '%' in pattern - size_t match_t = 0; // text position aligned with that '%' - while (t < text.size()) { - if (p < pattern.size() && pattern[p] == '%') { - star_p = p; - match_t = t; - ++p; // tentatively let '%' match zero chars - } else if (p < pattern.size() && - (pattern[p] == '_' || - likeFold(static_cast(pattern[p])) == - likeFold(static_cast( - text[t])))) { - ++p; - ++t; - } else if (star_p != std::string::npos) { - // backtrack: let the previous '%' swallow one more char - p = star_p + 1; - match_t = t = match_t + 1; - } else { - return false; - } - } - while (p < pattern.size() && pattern[p] == '%') - ++p; - return p == pattern.size(); -} - /// Return true if any target_path in `targets` matches the LIKE pattern /// `"%%"` (semantically identical to the original SQL /// `SELECT COUNT(*) ... WHERE target_path LIKE '%module_name%' > 0`). diff --git a/engine/src/resolver/factors.h b/engine/src/resolver/factors.h index c67a6de..22ddb49 100644 --- a/engine/src/resolver/factors.h +++ b/engine/src/resolver/factors.h @@ -9,6 +9,57 @@ namespace resolver { +/// Fold an ASCII byte to lowercase. SQLite's default LIKE folds only +/// ASCII upper-case letters; all other bytes (including non-ASCII) are +/// returned unchanged so they compare byte-for-byte, matching SQLite. +inline unsigned char likeFold(unsigned char ch) +{ + if (ch >= 'A' && ch <= 'Z') + return static_cast(ch + ('a' - 'A')); + return ch; +} + +/// Replicate SQLite's default LIKE matching for a full-string pattern. +/// '%' matches any sequence (including empty), '_' matches any single +/// character, and ASCII letters compare case-insensitively. The match +/// is anchored to the whole text (LIKE is not a substring search; the +/// surrounding '%' in callers' patterns provides prefix/suffix freedom). +/// +/// This is the standard greedy-with-backtrack wildcard matcher. It is +/// used instead of std::string::find so that '_'/'%' inside a module +/// name and ASCII case differences behave EXACTLY like the original +/// SQL LIKE predicates — preserving identical edges. +inline bool sqliteLikeMatch(const std::string &pattern, const std::string &text) +{ + size_t p = 0; // pattern cursor + size_t t = 0; // text cursor + size_t star_p = std::string::npos; // position of last '%' in pattern + size_t match_t = 0; // text position aligned with that '%' + while (t < text.size()) { + if (p < pattern.size() && pattern[p] == '%') { + star_p = p; + match_t = t; + ++p; // tentatively let '%' match zero chars + } else if (p < pattern.size() && + (pattern[p] == '_' || + likeFold(static_cast(pattern[p])) == + likeFold(static_cast( + text[t])))) { + ++p; + ++t; + } else if (star_p != std::string::npos) { + // backtrack: let the previous '%' swallow one more char + p = star_p + 1; + match_t = t = match_t + 1; + } else { + return false; + } + } + while (p < pattern.size() && pattern[p] == '%') + ++p; + return p == pattern.size(); +} + // ── Named constants for factor weights ────────────────────────────── constexpr double kWeightModuleMatch = 0.15; constexpr double kWeightImportMatch = 0.80; // Dominant for cross-module diff --git a/engine/src/resolver/fuzzy_resolver.cpp b/engine/src/resolver/fuzzy_resolver.cpp index 9ede8fc..1d31f2d 100644 --- a/engine/src/resolver/fuzzy_resolver.cpp +++ b/engine/src/resolver/fuzzy_resolver.cpp @@ -1,4 +1,6 @@ #include "fuzzy_resolver.h" +#include "factors.h" +#include #include #include @@ -6,108 +8,114 @@ namespace resolver { FuzzyResolver::FuzzyResolver(store::GraphStore *store, uint64_t project_id) - : store_(store) - , project_id_(project_id) { - if (!prepareStatements()) { - // prepareStatements() already logged the per-statement error. - // The resolver stays usable in degraded mode: resolve() checks - // for null statements and returns empty results so the pipeline - // falls through to the miss path instead of crashing. + if (!loadEntities(store, project_id)) { + // loadEntities() already logged the per-statement error. The + // resolver stays usable in degraded mode: resolve() returns + // empty results so the pipeline falls through to the miss path + // instead of crashing. fprintf(stderr, - "[module=resolver, method=FuzzyResolver::" - "FuzzyResolver] one or more statements failed to " - "prepare; fuzzy matching will be degraded\n"); + "[module=resolver, method=FuzzyResolver::FuzzyResolver] " + "failed to load entity index; fuzzy matching will be " + "degraded\n"); } } -FuzzyResolver::~FuzzyResolver() +bool FuzzyResolver::loadEntities(store::GraphStore *store, uint64_t project_id) { - if (stmt_case_insensitive_) - sqlite3_finalize(stmt_case_insensitive_); - if (stmt_prefix_) - sqlite3_finalize(stmt_prefix_); - if (stmt_suffix_) - sqlite3_finalize(stmt_suffix_); -} - -bool FuzzyResolver::prepareStatements() -{ - // Prepare all three fuzzy lookup statements once. Each carries an - // explicit `name != ''` filter so empty-name entity rows (which can - // appear for anonymous symbols) never pollute fuzzy results, and a - // LIMIT cap so a single wildcard never scans the whole table. - static constexpr const char *kSqlCaseInsensitive = - "SELECT id FROM entity " - "WHERE project_id=? AND name != '' " - "AND LOWER(name) = LOWER(?) LIMIT ?"; - static constexpr const char *kSqlPrefix = - "SELECT id FROM entity " - "WHERE project_id=? AND name != '' " - "AND name LIKE ? || '%' LIMIT ?"; - static constexpr const char *kSqlSuffix = - "SELECT id FROM entity " - "WHERE project_id=? AND name != '' " - "AND name LIKE '%' || ? LIMIT ?"; - - sqlite3 *db = store_ ? store_->handle() : nullptr; + if (!store) + return false; + sqlite3 *db = store->handle(); if (!db) return false; - bool ok = true; - if (sqlite3_prepare_v2(db, kSqlCaseInsensitive, -1, - &stmt_case_insensitive_, nullptr) != SQLITE_OK) { - fprintf(stderr, - "[module=resolver, method=FuzzyResolver::" - "prepareStatements] case-insensitive prepare " - "failed: %s\n", - sqlite3_errmsg(db)); - stmt_case_insensitive_ = nullptr; - ok = false; - } - if (sqlite3_prepare_v2(db, kSqlPrefix, -1, &stmt_prefix_, nullptr) != + // Load every (id, name) entity row once. The original implementation + // issued up to three SQL queries per unresolved reference against + // this table; loading it once into memory replaces all of them. + // `name != ''` mirrors the original SQL filters; load order is rowid + // order (no ORDER BY), which matches the old full-table-scan result + // order so LIMIT semantics are preserved. + static constexpr const char *kSqlLoadEntities = + "SELECT id, name FROM entity " + "WHERE project_id=? AND name != ''"; + sqlite3_stmt *stmt = nullptr; + if (sqlite3_prepare_v2(db, kSqlLoadEntities, -1, &stmt, nullptr) != SQLITE_OK) { fprintf(stderr, - "[module=resolver, method=FuzzyResolver::" - "prepareStatements] prefix prepare failed: %s\n", + "[module=resolver, method=FuzzyResolver::loadEntities] " + "prepare failed: %s\n", sqlite3_errmsg(db)); - stmt_prefix_ = nullptr; - ok = false; + return false; } - if (sqlite3_prepare_v2(db, kSqlSuffix, -1, &stmt_suffix_, nullptr) != - SQLITE_OK) { - fprintf(stderr, - "[module=resolver, method=FuzzyResolver::" - "prepareStatements] suffix prepare failed: %s\n", - sqlite3_errmsg(db)); - stmt_suffix_ = nullptr; - ok = false; + sqlite3_bind_int64(stmt, 1, static_cast(project_id)); + while (sqlite3_step(stmt) == SQLITE_ROW) { + EntityName e; + e.id = static_cast(sqlite3_column_int64(stmt, 0)); + const char *name_c = reinterpret_cast( + sqlite3_column_text(stmt, 1)); + if (!name_c || !*name_c) + continue; // empty names excluded by the SQL, be safe + e.raw = name_c; + // Fold once at load time so case-insensitive lookups are O(1) + // map hits instead of re-folding every candidate name. + std::string folded; + folded.reserve(e.raw.size()); + for (unsigned char ch : e.raw) + folded.push_back(static_cast(likeFold(ch))); + folded_index_[folded].push_back(e.id); + entities_.push_back(std::move(e)); } - return ok; + sqlite3_finalize(stmt); + + // Build the sorted prefix/suffix lookup indexes once. Each holds + // (folded name, entity id) — prefix_sorted_ ordered by the folded + // name, suffix_sorted_ by the REVERSED folded name so a suffix query + // becomes a prefix query on the reversed string. Binary search on + // these replaces the O(N) linear scan per fuzzy call; for a 130k + // entity project that is the difference between a few µs and a + // full-array walk per lookup. + prefix_sorted_.reserve(entities_.size()); + suffix_sorted_.reserve(entities_.size()); + for (const auto &e : entities_) { + std::string folded; + folded.reserve(e.raw.size()); + for (unsigned char ch : e.raw) + folded.push_back(static_cast(likeFold(ch))); + prefix_sorted_.emplace_back(folded, e.id); + std::string rev(folded.rbegin(), folded.rend()); + suffix_sorted_.emplace_back(std::move(rev), e.id); + } + std::sort(prefix_sorted_.begin(), prefix_sorted_.end(), + [](const auto &a, const auto &b) { + return a.first < b.first; + }); + std::sort(suffix_sorted_.begin(), suffix_sorted_.end(), + [](const auto &a, const auto &b) { + return a.first < b.first; + }); + return true; } std::vector FuzzyResolver::resolveCaseInsensitive(const std::string &name, size_t limit) { std::vector out; - if (name.empty() || !stmt_case_insensitive_) + if (name.empty()) return out; - // Reuse the prepared statement: reset clears the VM state so the - // statement can be stepped again; clear_bindings drops stale bound - // values from the previous call. - sqlite3_reset(stmt_case_insensitive_); - sqlite3_clear_bindings(stmt_case_insensitive_); - sqlite3_bind_int64(stmt_case_insensitive_, 1, - static_cast(project_id_)); - sqlite3_bind_text(stmt_case_insensitive_, 2, name.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_int64(stmt_case_insensitive_, 3, - static_cast(limit)); - while (sqlite3_step(stmt_case_insensitive_) == SQLITE_ROW) { - out.push_back(static_cast( - sqlite3_column_int64(stmt_case_insensitive_, 0))); - } + // ASCII-fold the query once (SQLite LOWER folds only A-Z; likeFold + // replicates exactly that), then an O(1) map hit. + std::string folded; + folded.reserve(name.size()); + for (unsigned char ch : name) + folded.push_back(static_cast(likeFold(ch))); + + auto it = folded_index_.find(folded); + if (it == folded_index_.end()) + return out; + const auto &ids = it->second; + for (size_t i = 0; i < ids.size() && out.size() < limit; ++i) + out.push_back(ids[i]); return out; } @@ -115,18 +123,55 @@ std::vector FuzzyResolver::resolvePrefix(const std::string &prefix, size_t limit) { std::vector out; - if (prefix.empty() || prefix.size() < 3 || !stmt_prefix_) + if (prefix.empty() || prefix.size() < 3) return out; // short prefixes match too many entities - sqlite3_reset(stmt_prefix_); - sqlite3_clear_bindings(stmt_prefix_); - sqlite3_bind_int64(stmt_prefix_, 1, static_cast(project_id_)); - sqlite3_bind_text(stmt_prefix_, 2, prefix.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_int64(stmt_prefix_, 3, static_cast(limit)); - while (sqlite3_step(stmt_prefix_) == SQLITE_ROW) { - out.push_back(static_cast( - sqlite3_column_int64(stmt_prefix_, 0))); + // Replicate the original `name LIKE ? || '%'` exactly: SQLite's + // default LIKE is ASCII-case-insensitive and treats '%'/'_' as + // wildcards. The binary-search path below is only valid when the + // prefix itself contains no wildcard character; a wildcard query + // must fall back to the linear sqliteLikeMatch scan so results stay + // byte-identical to the SQL semantics. + if (prefix.find('%') != std::string::npos || + prefix.find('_') != std::string::npos) { + std::string pattern = prefix + "%"; + for (const auto &e : entities_) { + if (sqliteLikeMatch(pattern, e.raw)) { + out.push_back(e.id); + if (out.size() >= limit) + break; + } + } + return out; + } + + // Wildcard-free: lower_bound() on the sorted folded-name index + // finds the first entity whose folded name is >= the folded prefix; + // every following entry that still starts with the folded prefix is + // a LIKE 'prefix%' match (ASCII-folded comparison == SQLite LIKE). + // Collect ALL matches, then sort by id (rowid == load order) before + // truncating: the old SQL `name LIKE ? || '%' LIMIT ?` returned rows + // in rowid scan order, and the sorted index enumerates in name order, + // so truncating the raw scan would retain a different subset whenever + // matches exceed the limit. + std::string folded_prefix; + folded_prefix.reserve(prefix.size()); + for (unsigned char ch : prefix) + folded_prefix.push_back(static_cast(likeFold(ch))); + auto it = std::lower_bound(prefix_sorted_.begin(), prefix_sorted_.end(), + std::make_pair(folded_prefix, uint64_t{ 0 }), + [](const auto &a, const auto &b) { + return a.first < b.first; + }); + for (; it != prefix_sorted_.end() && + it->first.size() >= folded_prefix.size() && + it->first.compare(0, folded_prefix.size(), folded_prefix) == 0; + ++it) { + out.push_back(it->second); } + std::sort(out.begin(), out.end()); + if (out.size() > limit) + out.resize(limit); return out; } @@ -134,18 +179,49 @@ std::vector FuzzyResolver::resolveSuffix(const std::string &suffix, size_t limit) { std::vector out; - if (suffix.empty() || suffix.size() < 3 || !stmt_suffix_) + if (suffix.empty() || suffix.size() < 3) return out; - sqlite3_reset(stmt_suffix_); - sqlite3_clear_bindings(stmt_suffix_); - sqlite3_bind_int64(stmt_suffix_, 1, static_cast(project_id_)); - sqlite3_bind_text(stmt_suffix_, 2, suffix.c_str(), -1, SQLITE_STATIC); - sqlite3_bind_int64(stmt_suffix_, 3, static_cast(limit)); - while (sqlite3_step(stmt_suffix_) == SQLITE_ROW) { - out.push_back(static_cast( - sqlite3_column_int64(stmt_suffix_, 0))); + // Same wildcard gate as resolvePrefix — '%'/'_' in the suffix act + // as LIKE wildcards and must take the linear scan path. + if (suffix.find('%') != std::string::npos || + suffix.find('_') != std::string::npos) { + std::string pattern = "%" + suffix; + for (const auto &e : entities_) { + if (sqliteLikeMatch(pattern, e.raw)) { + out.push_back(e.id); + if (out.size() >= limit) + break; + } + } + return out; + } + + // Wildcard-free: a suffix match on the original name is a prefix + // match on the REVERSED name, so binary search the reversed folded + // index with the reversed folded suffix. Same ordering rule as + // resolvePrefix: collect all matches, sort by id (rowid == load + // order) before truncating so the retained subset matches the old + // SQL `name LIKE '%' || ? LIMIT ?` rowid-scan order. + std::string folded_suffix; + folded_suffix.reserve(suffix.size()); + for (unsigned char ch : suffix) + folded_suffix.push_back(static_cast(likeFold(ch))); + std::string rev_suffix(folded_suffix.rbegin(), folded_suffix.rend()); + auto it = std::lower_bound(suffix_sorted_.begin(), suffix_sorted_.end(), + std::make_pair(rev_suffix, uint64_t{ 0 }), + [](const auto &a, const auto &b) { + return a.first < b.first; + }); + for (; it != suffix_sorted_.end() && + it->first.size() >= rev_suffix.size() && + it->first.compare(0, rev_suffix.size(), rev_suffix) == 0; + ++it) { + out.push_back(it->second); } + std::sort(out.begin(), out.end()); + if (out.size() > limit) + out.resize(limit); return out; } @@ -153,7 +229,7 @@ std::vector FuzzyResolver::resolve(const std::string &callee_name, size_t limit) { std::vector out; - if (callee_name.empty() || !store_) + if (callee_name.empty() || entities_.empty()) return out; // Strategy 1: case-insensitive exact match (cheapest, most precise). @@ -171,4 +247,4 @@ std::vector FuzzyResolver::resolve(const std::string &callee_name, return out; } -} // namespace resolver \ No newline at end of file +} // namespace resolver diff --git a/engine/src/resolver/fuzzy_resolver.h b/engine/src/resolver/fuzzy_resolver.h index e39399a..67dd028 100644 --- a/engine/src/resolver/fuzzy_resolver.h +++ b/engine/src/resolver/fuzzy_resolver.h @@ -3,11 +3,10 @@ #include #include +#include #include #include "../store/store.h" -struct sqlite3_stmt; - namespace resolver { @@ -28,12 +27,22 @@ namespace resolver /// because it produces too many false positives on short names. The /// three strategies above cover the common cases (case differences, /// abbreviations, partial names) without that risk. +/// +/// Performance: the constructor loads every entity (id, name) for the +/// project once into memory and resolves all three strategies against +/// that index (ASCII-folded exact map + linear LIKE scans). This +/// replaces the original implementation's three SQL queries per missed +/// reference (case-insensitive/prefix/suffix on the entity table) with +/// pure in-memory lookups. Matching semantics are IDENTICAL to the old +/// SQL: case-insensitive exact folds only ASCII A-Z (SQLite LOWER), and +/// prefix/suffix use SQLite's default ASCII-insensitive LIKE with '%' +/// and '_' wildcards (see sqliteLikeMatch in factors.h). class FuzzyResolver { public: FuzzyResolver(store::GraphStore *store, uint64_t project_id); - ~FuzzyResolver(); + ~FuzzyResolver() = default; - // Non-copyable due to owned sqlite3_stmt members. + // Non-copyable: owns the loaded entity index. FuzzyResolver(const FuzzyResolver &) = delete; FuzzyResolver &operator=(const FuzzyResolver &) = delete; @@ -47,21 +56,34 @@ class FuzzyResolver { size_t limit = 5); private: - store::GraphStore *store_; - uint64_t project_id_; + // One loaded entity row: id + raw name. kept_ = load order + // (rowid order), which matches the original SQL full-table-scan + // result order, so LIMIT semantics are preserved. + struct EntityName { + uint64_t id; + std::string raw; // original name (for LIKE matching) + }; + std::vector entities_; + + // ASCII-folded name -> entity ids (in load order). O(1) lookup + // for the case-insensitive exact strategy. + std::unordered_map> folded_index_; - // Prepared statements — created once in the constructor, reused - // across resolve() calls via sqlite3_reset + sqlite3_clear_bindings, - // and finalized in the destructor. Preparing per-call was a major - // cost in the fuzzy fallback path (3 prepares per missed name). - sqlite3_stmt *stmt_case_insensitive_ = nullptr; - sqlite3_stmt *stmt_prefix_ = nullptr; - sqlite3_stmt *stmt_suffix_ = nullptr; + // Sorted lookup indexes for O(log N) prefix/suffix matching: + // prefix_sorted_ = (folded name, entity id) sorted by folded name + // suffix_sorted_ = (reversed folded name, entity id) sorted by + // reversed folded name + // Built once in loadEntities() and only consulted when the query + // contains no LIKE wildcards ('%' / '_'); wildcard queries fall + // back to the linear scan over entities_ so results stay + // byte-identical to the original SQL LIKE semantics. + std::vector> prefix_sorted_; + std::vector> suffix_sorted_; - /// Prepare all three statements. Returns true on success. On partial - /// failure the affected member is left null and resolve() degrades - /// gracefully (returns empty for that strategy). - bool prepareStatements(); + /// Load all (id, name) entity rows for the project. Returns true + /// on success; on failure resolve() degrades to empty results + /// (no crash) and the caller logs the error. + bool loadEntities(store::GraphStore *store, uint64_t project_id); /// Strategy 1: case-insensitive exact name match. std::vector resolveCaseInsensitive(const std::string &name, @@ -78,4 +100,4 @@ class FuzzyResolver { } // namespace resolver -#endif // CODESCOPE_FUZZY_RESOLVER_H \ No newline at end of file +#endif // CODESCOPE_FUZZY_RESOLVER_H diff --git a/engine/src/resolver/pipeline.cpp b/engine/src/resolver/pipeline.cpp index 09e1788..0aaa274 100644 --- a/engine/src/resolver/pipeline.cpp +++ b/engine/src/resolver/pipeline.cpp @@ -159,289 +159,27 @@ std::string ResolverPipeline::checkImport(const std::string &caller_file, return result; } -void ResolverPipeline::applyConstraints(std::vector &candidates, - const std::string &caller_file, - const std::string &callee_name, - int call_kind, int caller_arity, - const std::string &receiver_type) -{ - // Build per-factor scores for each candidate using multi-factor scoring. - // - // v0.2.5 (perf fix): the original code built a full - // std::vector (name/detail heap strings) per candidate and - // fed it to computeTotalScore(). On large projects that is ~166k - // candidate evaluations × ~20 string allocations each ≈ 3.3M heap - // allocations — measured as the 90µs-per-candidate cost that made the - // resolver 93.8% of index time (goagent: 14.9s of 15.9s). Here we - // accumulate the weighted sum directly (pure double math, no strings) - // and capture only the ReceiverMatch score that the ambiguity gate needs - // (see receiver_bypass in run()). Every factor's weight/score pair and - // the final weighted-average formula are identical to the previous - // computeTotalScore path, so resolved edges are byte-for-byte unchanged. - // - // v0.2.5 (perf fix #2): the path-based factors (ImportMatch, - // NamespaceMatch, DistanceMatch) each re-derived caller_file's directory - // and module token with rfind/substr on every candidate. caller_file is - // fixed for the whole ref, so we parse it ONCE here (caller_dir / - // caller_parent / caller_module) and parse each candidate's path ONCE - // inside the loop, then compute the three path factors from those - // pre-parsed values. This removes ~N×3 redundant substring allocations - // per candidate. The scoring rules are kept IDENTICAL to - // factorImportMatch/factorNamespaceMatch/factorDistanceMatch in - // factors.cpp — do not change them independently. - size_t caller_slash = caller_file.rfind('/'); - std::string caller_dir = (caller_slash != std::string::npos) ? - caller_file.substr(0, caller_slash) : - ""; - size_t caller_parent_slash = caller_dir.rfind('/'); - std::string caller_parent = - (caller_parent_slash != std::string::npos) ? - caller_dir.substr(0, caller_parent_slash) : - ""; - // moduleTokenFromPath: the last path token (file's directory name). - // Mirrors the anonymous helper in factors.cpp used by - // factorImportMatch. When there is no slash, the whole dir is returned. - std::string caller_module = caller_dir; - { - size_t ms = caller_dir.rfind('/'); - if (ms != std::string::npos) - caller_module = caller_dir.substr(ms + 1); - } - auto anyImportMatches = [&](const std::vector &paths, - const std::string &mod) { - for (const auto &p : paths) { - if (p.find(mod) != std::string::npos) - return true; - } - return false; - }; - // ── Ref-level factor precompute (perf fix #3) ── - // factorCommonNamePenalty depends ONLY on the ref's callee_name, which - // is fixed across all candidates of this ref, yet it was called once per - // candidate (~166k calls). Compute it once here. - const double common_name_penalty = factorCommonNamePenalty(callee_name); - // When receiver_type is empty (dynamic/unknown), factorReceiverTypeMatch - // returns a constant neutral 0.5 regardless of the candidate — compute it - // once here instead of per candidate. When non-empty, build the ref-level - // context (prefix1/prefix2/rtype_lower) once so the per-candidate match - // does not reallocate those strings (perf fix #3). - const bool receiver_is_known = !receiver_type.empty(); - const double neutral_receiver_score = 0.5; - const ReceiverMatchContext receiver_ctx = - buildReceiverMatchContext(receiver_type); - // ── Ref-level ImportMatch forward cache (perf fix #4) ── - // caller_file is fixed for the whole ref, so the forward lookup - // import_index_[caller_file] yields the same vector for every - // candidate. Hoisting it out of the candidate loop removes N-1 - // redundant hashmap lookups per ref (measured on goagent: ~166k → - // ~24k lookups). The scoring semantics are unchanged — the pointer is - // non-null iff the original fwd_it != import_index_.end(). - auto caller_fwd_it = import_index_.find(caller_file); - const std::vector *caller_fwd_imports = - (caller_fwd_it != import_index_.end()) ? - &caller_fwd_it->second : - nullptr; - // ── Ref-level ImportMatch forward fused string (perf fix #5) ── - // anyImportMatches scans every path and does p.find(mod) per path. - // Fusing the (fixed-for-this-ref) forward import list into ONE - // NUL-separated string lets each candidate do a single find() over a - // contiguous buffer instead of N vector elements + N substring calls - // (CBM-style fused matching). Module names never contain NUL, so a - // match can never span the separator — the result is identical to the - // per-path scan (∃p: p.find(mod) != npos ⟺ joined.find(mod) != npos). - std::string caller_fwd_joined; - if (caller_fwd_imports != nullptr) { - for (const auto &p : *caller_fwd_imports) { - caller_fwd_joined += p; - caller_fwd_joined += '\x00'; - } - } - for (auto &c : candidates) { - double sum_weight = 0.0; - double sum_scored = 0.0; - // Accumulate one (weight, score) pair into the weighted sums. - // This mirrors computeTotalScore's loop without materializing a - // vector per candidate. - auto acc = [&](double weight, double score) { - sum_weight += weight; - sum_scored += weight * score; - }; - - // ── Candidate path components (v0.6) ── - // Precomputed once when the entity_index was loaded (pipeline.cpp - // entity_index build); values are byte-identical to the old inline - // rfind+substr derivation, so this is a pure-allocation win. - const std::string &cand_dir = c.cand_dir; - const std::string &cand_parent = c.cand_parent; - const std::string &cand_module = c.cand_module; - - // ── NamespaceMatch score (reused by ModuleMatch and the - // CommonNamePenalty same-module gate) ── - // Mirrors factorNamespaceMatch(caller_file, c.file_path): - // same dir → 1.0; same parent dir → 0.5; else 0.0. - double ns_score = 0.0; - if (!cand_dir.empty() && !caller_dir.empty()) { - if (caller_dir == cand_dir) - ns_score = kScoreExactMatch; - else if (caller_parent == cand_parent && - !caller_parent.empty()) - ns_score = kScoreSiblingModule; - } - - // Factor 1: ModuleMatch - acc(kWeightModuleMatch, ns_score); - - // Factor 2: ImportMatch — dominant weight for cross-module calls. - // Mirrors factorImportMatch(import_index_, caller_file, - // c.file_path, c.name): - // 1. same directory → 1.0 (no import needed) - // 2. forward: caller imports candidate_module → 1.0 - // 3. reverse: candidate imports caller_module → 1.0 - // 4. else 0.0 - { - double import_score = 0.0; - if (!caller_dir.empty() && caller_dir == cand_dir) { - import_score = 1.0; - } else { - // Forward: caller imports candidate module. - // Uses the ref-level fused string (perf fix #5): - // one find() over the contiguous buffer replaces - // the per-path scan of anyImportMatches. - if (!caller_fwd_joined.empty() && - caller_fwd_joined.find(cand_module) != - std::string::npos) - import_score = 1.0; - else { - auto rev_it = - import_index_.find(c.file_path); - if (rev_it != import_index_.end() && - anyImportMatches(rev_it->second, - caller_module)) - import_score = 1.0; - } - } - acc(kWeightImportMatch, import_score); - } - - // Factor 3: NamespaceMatch - acc(kWeightNamespaceMatch, ns_score); - - // Factor 4: SignatureMatch — compares the call site's arity - // (from the reference row) against each candidate's arity. - // Previously the caller arity was hardcoded to 0, which caused - // factorSignatureMatch to penalize every candidate with a known - // arity (returning -0.5) while rewarding candidates with unknown - // arity (returning +0.5) — the exact opposite of correct - // overload resolution. Thread the real reference arity through - // so exact-arity overloads score highest. - acc(kWeightSignatureMatch, - factorSignatureMatch(caller_arity, c.arity)); - - // Factor 5: DistanceMatch — mirrors factorDistanceMatch: - // same file → 1.0; same directory → 0.3; else 0.0. - { - double dist_score = 0.0; - if (caller_file == c.file_path) - dist_score = kScoreExactMatch; - else if (!caller_dir.empty() && caller_dir == cand_dir) - dist_score = kScoreSameDirectory; - acc(kWeightDistanceMatch, dist_score); - } - - // Factor 6: ConstructorMatch - acc(kWeightConstructorMatch, - factorConstructorMatch(callee_name, c.name, c.kind)); - - // Factor 7: ReceiverMatch (Step 5: now type-based, not directory) - // Replaced factorReceiverMatch (directory heuristic) with - // factorReceiverTypeMatch (actual receiver_type evidence). - // When receiver_type is known (e.g. "Box"), candidates whose - // qualified_name contains "Box::" or "Box." score 1.0. When - // receiver_type is empty (dynamic/unknown), the factor returns - // 0.5 (neutral) so it does not distort the ranking. - // - // The per-candidate ReceiverMatch score is captured separately - // (c.receiver_score) because the ambiguity gate in run() - // (receiver_bypass) reads only this one factor — no other part of - // the hot loop consumes the full FactorResult vector. - { - // v0.2.5 (perf fix #3): when receiver_type is empty the score - // is a constant neutral 0.5 (precomputed). When non-empty, use - // the ref-level pre-parsed context so per-candidate string - // allocations (prefix1/prefix2/rtype_lower) are avoided. - double recv = neutral_receiver_score; - if (receiver_is_known) { - recv = factorReceiverTypeMatchPrecomp( - receiver_ctx, c.qualified_name, - c.file_path); - } - c.receiver_score = recv; - acc(kWeightReceiverMatch, recv); - } - - // Factor 8: CommonNamePenalty — reduce score for very common names - // Only applies to cross-module candidates (same-module should not be - // penalized). The penalty value itself depends only on the ref's - // callee_name and is precomputed once per ref (perf fix #3). - { - // Only penalize if candidate is in a different module - bool same_module = (ns_score > 0.0); - double penalty = same_module ? 0.0 : - common_name_penalty; - acc(kWeightCommonNamePenalty, -penalty); - } - - // Factor 9: CallKindMatch — adjust scoring based on call kind. - // Constructor calls (3): boost for cross-module matching. - // Interface dispatches (2): reduce confidence (harder to resolve). - // Method calls (1): slight cross-module penalty. - if (call_kind != kCallKindDirect) { - double kscore = 0.0; - if (call_kind == kCallKindConstructor) - kscore = - 0.3; // boost: constructors expected to cross module - else if (call_kind == kCallKindInterface) - kscore = - -0.3; // penalty: interface dispatch is harder to resolve - else if (call_kind == kCallKindMethod) - kscore = - -0.1; // slight penalty: methods usually same-module - acc(kWeightCallKindMatch, kscore); - } - - // Factor 10: DefinitionMatch — for C/C++, prefer symbols defined - // in a source file (.c/.cpp/.cc/...) over those declared only in - // a header (.h/.hpp/...). This breaks the previous arbitrary tie - // between a header prototype and a source definition that scored - // identically (Finding #8). Non-C/C++ languages return 1.0 - // (neutral), so their ranking is unaffected. - acc(kWeightDefinitionMatch, - factorDefinitionMatch(c.language, c.file_path)); - - // VisibilityCheck was moved to a hard filter in run() to - // ensure language visibility rules (e.g. Go unexported names) - // are applied as hard rejections, not weighted factors — a - // weighted factor can be overcome by other factors, but a - // hard language rule must be absolute. - - // Weighted average, identical to computeTotalScore's formula. - c.total_score = (sum_weight > 0.0) ? (sum_scored / sum_weight) : - 0.0; - // c.factors is intentionally NOT populated here (perf); the hot - // loop never reads it. If a future debug path needs factor detail, - // rebuild it lazily for the resolved candidate only. - } - - std::sort(candidates.begin(), candidates.end(), - [](const Candidate &a, const Candidate &b) { - return a.total_score > b.total_score; - }); -} - int64_t ResolverPipeline::run() { using Clock = std::chrono::steady_clock; auto t_start = Clock::now(); + // Per-stage timing for resolver profiling. Enable with + // CODESCOPE_PROFILE_RESOLVER=1 to print each load/resolve phase's + // wall time to stderr; it is a no-op when the env var is unset, so + // the default path pays only one steady_clock::now() per phase. + const bool profile_resolver = getenv("CODESCOPE_PROFILE_RESOLVER") != + nullptr; + auto t_prev = t_start; + const auto mark = [&](const char *label) { + auto t_now = Clock::now(); + if (profile_resolver) + fprintf(stderr, "RP[%s]=%lldms\n", label, + (long long)std::chrono::duration_cast< + std::chrono::milliseconds>(t_now - + t_prev) + .count()); + t_prev = t_now; + }; // ── P1: Staging table for batch edge inserts ────────────────── // Instead of preparing/finalizing an INSERT per resolved reference, @@ -472,399 +210,25 @@ int64_t ResolverPipeline::run() return -1; } - // ── Step 0: Pre-load all entities into a name-indexed HashMap ── - // This avoids one SQL query per reference (the main bottleneck). + // ── Step 0: Pre-load entities, id map, imports (pipeline_load.cpp) ── + // Pre-load all entities into a name-indexed candidate map, an + // id -> candidate pointer map (for fuzzy hydration), and the per-file + // import index. This avoids one SQL query per reference (the main + // bottleneck). Moved to ResolverPipeline::loadEntityIndex so this + // translation unit stays under the 1000-line rule. std::unordered_map> entity_index; int64_t total_entities = 0; - { - std::string idx_sql = - // Include arity so factorSignatureMatch can distinguish - // same-name overloads (init()/init(int)/init(string)). - // entity.arity was added in v0.5+ migration (store_schema.cpp:470). - // Without this column in the SELECT, c.arity defaulted to 0 - // and every candidate scored kScorePartialMatch (0.5), - // letting std::sort pick the winner by unstable order. - // See CODE_REVIEW_FINDINGS_2026-07-19.md C2. - // Include kind (appended as column 5) so - // factorConstructorMatch can prefer Class/Struct targets; - // previously kind was hardcoded 0 in the call, so the - // constructor factor always returned 0.0 (M-11). - // Step 5: include qualified_name (column 6) so - // factorReceiverTypeMatch can match "Box::draw" against - // receiver_type="Box" instead of using directory heuristics. - "SELECT id, name, file_path, language, arity, kind, qualified_name " - "FROM entity " - "WHERE project_id=? AND name != ''"; - sqlite3_stmt *idx_st = nullptr; - if (sqlite3_prepare_v2(store_->handle(), idx_sql.c_str(), -1, - &idx_st, nullptr) != SQLITE_OK) { - fprintf(stderr, - "[module=resolver, method=run] " - "prepare entity index failed: %s\n", - sqlite3_errmsg(store_->handle())); - return -1; - } - sqlite3_bind_int64(idx_st, 1, - static_cast(project_id_)); - while (sqlite3_step(idx_st) == SQLITE_ROW) { - Candidate c; - c.entity_id = static_cast( - sqlite3_column_int64(idx_st, 0)); - const char *n = reinterpret_cast( - sqlite3_column_text(idx_st, 1)); - const char *fp = reinterpret_cast( - sqlite3_column_text(idx_st, 2)); - const char *lang = reinterpret_cast( - sqlite3_column_text(idx_st, 3)); - c.name = n ? n : ""; - c.file_path = fp ? fp : ""; - c.language = lang ? lang : - languageFromPath(c.file_path); - c.module_path = modulePath(c.file_path); - // v0.6 (perf): precompute the path components that - // applyConstraints derives from c.file_path on every candidate. - // Computing them once here (dir/parent/module token) removes - // per-ref heap allocations in the hot loop; the values are - // byte-identical to the old inline rfind+substr derivation so no - // score changes. - { - size_t cs = c.file_path.rfind('/'); - c.cand_dir = (cs != std::string::npos) ? - c.file_path.substr(0, cs) : - std::string(); - size_t cps = c.cand_dir.rfind('/'); - c.cand_parent = - (cps != std::string::npos) ? - c.cand_dir.substr(0, cps) : - std::string(); - c.cand_module = c.cand_dir; - size_t cms = c.cand_dir.rfind('/'); - if (cms != std::string::npos) - c.cand_module = - c.cand_dir.substr(cms + 1); - } - // Column 4 is arity (added to SELECT above). Default 0 - // if NULL — matches entity.arity DEFAULT 0 so callers - // that never set arity behave as "unknown arity". - c.arity = sqlite3_column_int(idx_st, 4); - // Column 5 is kind (RecordKind). Default 0 if NULL — - // matches entity.kind NOT NULL semantics; 0 = Function, - // so non-type candidates correctly score 0.0 on the - // constructor factor. - c.kind = sqlite3_column_int(idx_st, 5); - // Step 5: column 6 is qualified_name. Used by - // factorReceiverTypeMatch to match "Box::draw" against - // receiver_type="Box". Empty for languages that don't - // populate it (e.g. Go), causing the factor to fall back - // to file-path matching. - const char *qn = reinterpret_cast( - sqlite3_column_text(idx_st, 6)); - c.qualified_name = qn ? qn : ""; - c.score = 0; - entity_index[c.name].push_back(c); - total_entities++; - } - sqlite3_finalize(idx_st); - } - - // ── Step 0b: Pre-load all imports into a file_path-indexed HashMap ── - // This is the core fix for the 174s bottleneck: factorImportMatch - // previously ran `SELECT COUNT(*) FROM import WHERE project_id=? AND - // file_path=? AND target_path LIKE '%module_name%'` per candidate. - // The leading-% LIKE is non-sargable, forcing a FULL TABLE SCAN on - // the import table for each of the ~313k candidate evaluations - // (108k refs × 2.9 avg candidates × 1-2 SQL queries each). - // - // We load every import row once into import_index_ (file_path -> - // list of target_path strings) and let factorImportMatch match - // in-memory with SQLite-exact LIKE semantics. Resolved edges are - // IDENTICAL to the SQL implementation; only the access path changes. - import_index_.clear(); - int64_t total_imports = 0; - { - std::string imp_sql = - "SELECT file_path, target_path FROM import " - "WHERE project_id=?"; - sqlite3_stmt *imp_st = nullptr; - if (sqlite3_prepare_v2(store_->handle(), imp_sql.c_str(), -1, - &imp_st, nullptr) != SQLITE_OK) { - fprintf(stderr, - "[module=resolver, method=run] " - "prepare import index failed: %s\n", - sqlite3_errmsg(store_->handle())); - return -1; - } - sqlite3_bind_int64(imp_st, 1, - static_cast(project_id_)); - while (sqlite3_step(imp_st) == SQLITE_ROW) { - const char *fp = reinterpret_cast( - sqlite3_column_text(imp_st, 0)); - const char *tp = reinterpret_cast( - sqlite3_column_text(imp_st, 1)); - std::string file_path = fp ? fp : ""; - std::string target_path = tp ? tp : ""; - // NOTE: empty file_path rows are intentionally KEPT. The - // original SQL used an exact `file_path = ?` predicate, - // which matches empty-file_path rows when caller_file is - // also empty; dropping them would change matching results - // for such (rare) call sites and break identical-edge - // semantics. They land in the "" bucket and are only - // consulted when a caller's file_path is empty. - import_index_[file_path].push_back( - std::move(target_path)); - total_imports++; - } - sqlite3_finalize(imp_st); - } - - // ── Step 8 (plan §8.1): Pre-load interface/trait implementations ── - // InterfaceImpl records (kind=20) store (name=implementing_type, - // type_name=interface_name). We build a map from interface name - // to all implementing types, so the hot loop can expand - // Interface/Virtual dispatch calls into bounded candidate sets. - interface_impl_index_.clear(); - int64_t total_iface_impls = 0; - { - // semantic_records.kind=20 is InterfaceImpl. The `name` column - // holds the implementing type, `type_name` holds the interface. - std::string iface_sql = - "SELECT name, type_name FROM semantic_records " - "WHERE project_id=? AND kind=20 AND name != '' " - "AND type_name != ''"; - sqlite3_stmt *iface_st = nullptr; - if (sqlite3_prepare_v2(store_->handle(), iface_sql.c_str(), -1, - &iface_st, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(iface_st, 1, - static_cast(project_id_)); - while (sqlite3_step(iface_st) == SQLITE_ROW) { - const char *impl = - reinterpret_cast( - sqlite3_column_text(iface_st, - 0)); - const char *iface = - reinterpret_cast( - sqlite3_column_text(iface_st, - 1)); - if (impl && iface) - interface_impl_index_[iface].push_back( - impl); - total_iface_impls++; - } - sqlite3_finalize(iface_st); - } - } - - // ── Step 8 (plan §8.1b): cross-file interface method-set matching ── - // The visitor's kind=20 records only cover same-file (struct, - // interface) pairs — Go interfaces are usually declared in one file - // and implemented in another, so those never match in-file. This - // global pass reconstructs method sets from the per-method qualified - // names ("Struct.method" set by handleMethodDecl, "Interface.method" - // set by handleInterfaceMethod) and re-runs the subset check across - // ALL files, supplementing interface_impl_index_ with cross-file - // implementations. - { - // Interface entity names (kind=3) — used to classify a method's - // qualified-name prefix as an interface vs a struct. - std::unordered_set iface_names; - { - std::string names_sql = - "SELECT name FROM semantic_records " - "WHERE project_id=? AND kind=3 AND name != ''"; - sqlite3_stmt *nst = nullptr; - if (sqlite3_prepare_v2(store_->handle(), - names_sql.c_str(), -1, &nst, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64( - nst, 1, - static_cast(project_id_)); - while (sqlite3_step(nst) == SQLITE_ROW) { - const char *n = - reinterpret_cast( - sqlite3_column_text(nst, - 0)); - if (n) - iface_names.insert(n); - } - sqlite3_finalize(nst); - } - } - // Method records with qualified names — split "Type.method". - std::unordered_map> - iface_methods; // interface name -> its methods - std::unordered_map> - struct_methods; // struct type -> its methods - { - std::string meth_sql = - "SELECT qualified_name FROM semantic_records " - "WHERE project_id=? AND kind=1 AND " - "qualified_name != ''"; - sqlite3_stmt *mst = nullptr; - if (sqlite3_prepare_v2(store_->handle(), - meth_sql.c_str(), -1, &mst, - nullptr) == SQLITE_OK) { - sqlite3_bind_int64( - mst, 1, - static_cast(project_id_)); - while (sqlite3_step(mst) == SQLITE_ROW) { - const char *qn = - reinterpret_cast( - sqlite3_column_text(mst, - 0)); - if (!qn) - continue; - std::string q(qn); - size_t dot = q.find('.'); - if (dot == std::string::npos) - continue; - std::string type_name = - q.substr(0, dot); - std::string method = q.substr(dot + 1); - if (type_name.empty() || method.empty()) - continue; - if (iface_names.count(type_name) > 0) - iface_methods[type_name] - .push_back(method); - else - struct_methods[type_name] - .push_back(method); - } - sqlite3_finalize(mst); - } - } - // Global subset check: struct implements interface iff the - // struct's method set contains every interface method. - // v0.2.5 (perf fix): pre-index each struct's method set into a - // hash set once, then the interface-implements check is O(1) per - // method instead of a linear std::find. Without this, the - // for-interface × for-struct × for-method triple loop was O(I×S×M) - // — quadratic and noticeable on large Go projects with many - // interfaces/structs. - std::unordered_map> - struct_method_set; - struct_method_set.reserve(struct_methods.size()); - for (const auto &sentry : struct_methods) { - auto &s = struct_method_set[sentry.first]; - s.reserve(sentry.second.size()); - s.insert(sentry.second.begin(), sentry.second.end()); - } - for (const auto &iface_entry : iface_methods) { - const std::string &iface = iface_entry.first; - const auto &imethods = iface_entry.second; - if (imethods.empty()) - continue; - for (const auto &sentry : struct_methods) { - const std::string &stype = sentry.first; - if (stype == iface) - continue; - auto smit = struct_method_set.find(stype); - if (smit == struct_method_set.end()) - continue; - const auto &smethods = smit->second; - bool implements_all = true; - for (const auto &m : imethods) { - if (smethods.find(m) == - smethods.end()) { - implements_all = false; - break; - } - } - if (implements_all) { - // Avoid duplicating an entry the - // visitor's kind=20 pass already added. - auto &impls = - interface_impl_index_[iface]; - if (std::find(impls.begin(), - impls.end(), - stype) == impls.end()) { - impls.push_back(stype); - total_iface_impls++; - } - } - } - } - if (total_iface_impls > 0) { - fprintf(stderr, - "[module=resolver, method=run] interface_impl_index: " - "%lld implementation(s) loaded (%d interface(s))\n", - static_cast(total_iface_impls), - static_cast(interface_impl_index_.size())); - } - } - - // ── Step 8 (plan §8.1c): rebuild the global struct field table ── - // The Go visitor persists each struct field as a TypeRef record - // (kind=14) under the struct entity (kind=2): name = field name, - // type_name = field type. Rebuilding here makes the table complete - // across files, so field-chain receivers (r.pluginBus.AfterStep) - // whose receiver_type was empty at visit time (struct declared in - // another file) can be resolved before dispatch expansion. - global_struct_fields_.clear(); - { - std::string field_sql = - "SELECT p.name, t.name, t.type_name " - "FROM semantic_records t " - "JOIN semantic_records p ON t.parent_id = p.original_id " - "AND p.project_id = t.project_id " - "WHERE t.project_id=? AND t.kind=17 AND p.kind=2 " - "AND t.name != '' AND t.type_name != ''"; - sqlite3_stmt *fst = nullptr; - if (sqlite3_prepare_v2(store_->handle(), field_sql.c_str(), -1, - &fst, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(fst, 1, - static_cast(project_id_)); - while (sqlite3_step(fst) == SQLITE_ROW) { - const char *stype = - reinterpret_cast( - sqlite3_column_text(fst, 0)); - const char *fname = - reinterpret_cast( - sqlite3_column_text(fst, 1)); - const char *ftype = - reinterpret_cast( - sqlite3_column_text(fst, 2)); - if (stype && fname && ftype) - global_struct_fields_[stype][fname] = - ftype; - } - sqlite3_finalize(fst); - } - } + std::unordered_map entity_by_id; + if (loadEntityIndex(entity_index, entity_by_id, total_entities) != 0) + return -1; + mark("load_entities"); + mark("load_imports"); - // ── Step 8.1c (plan §8): global caller variable-type table ── - // The Go visitor persists method receivers and declared variables - // as TypeRef records (kind=14) under the containing function/method - // entity (kind=0/1): name = variable name, type_name = its type. - // Rebuilding here gives the field-chain resolver the first-segment - // type ("r" -> "Runner") when resolving "r.pluginBus.AfterStep". - global_var_types_.clear(); - { - std::string vtype_sql = - "SELECT t.name, t.type_name FROM semantic_records t " - "JOIN semantic_records p ON t.parent_id = p.original_id " - "AND p.project_id = t.project_id " - "WHERE t.project_id=? AND t.kind=17 " - "AND p.kind IN (0,1) " - "AND t.name != '' AND t.type_name != ''"; - sqlite3_stmt *vst = nullptr; - if (sqlite3_prepare_v2(store_->handle(), vtype_sql.c_str(), -1, - &vst, nullptr) == SQLITE_OK) { - sqlite3_bind_int64(vst, 1, - static_cast(project_id_)); - while (sqlite3_step(vst) == SQLITE_ROW) { - const char *vname = - reinterpret_cast( - sqlite3_column_text(vst, 0)); - const char *vtype = - reinterpret_cast( - sqlite3_column_text(vst, 1)); - if (vname && vtype) - global_var_types_[vname].push_back( - vtype); - } - sqlite3_finalize(vst); - } - } + // ── Step 8: Pre-load interface/dispatch + var-type tables ── + // Populates interface_impl_index_, global_struct_fields_ and + // global_var_types_ from semantic_records (pipeline_load.cpp). + loadDispatchIndex(); + mark("load_var_types_struct"); // ── Query all references for this project ── // Step 3 (plan §3.1): select the structured call-fact columns so the @@ -912,17 +276,9 @@ int64_t ResolverPipeline::run() return -1; } - // Prepare fuzzy hydration lookup (reused per fuzzy candidate) - // Include arity so fuzzy-resolved candidates also get overload - // disambiguation via factorSignatureMatch (see C2 above). - // Include kind (column 4) so fuzzy candidates carry the same - // constructor factor support as exact-name candidates (M-11). - // Step 5: include qualified_name (column 5) so fuzzy candidates - // also get receiver type matching support. - const char *lk_sql = "SELECT name, file_path, language, arity, kind, " - "qualified_name FROM entity WHERE id=?"; - sqlite3_stmt *lk_st = nullptr; - sqlite3_prepare_v2(store_->handle(), lk_sql, -1, &lk_st, nullptr); + // Fuzzy candidates are hydrated from the in-memory entity_index + // (id -> Candidate map built right after Step 0), so no per-id + // SQL lookup is needed in the hot loop anymore. // ── Step 0: Profiling counters ── int64_t resolved_count = 0; @@ -1022,20 +378,6 @@ int64_t ResolverPipeline::run() // Free the entity_index right after the hot loop — it's no longer needed. // Store results in a vector for batch insert. - struct ResolvedEdge { - uint64_t caller_id; - uint64_t target_id; - int edge_type; - std::string resolve_strategy; - // Step 6 (plan §6.1): provenance fields. - double confidence; - std::string resolver; - std::string resolution_kind; - std::string reason; - std::string call_site_file; - int call_site_row; - int call_site_col; - }; std::vector resolved_edges; resolved_edges.reserve(16384); // pre-allocate for 36k typical @@ -1051,6 +393,18 @@ int64_t ResolverPipeline::run() // the repeated chain walks from the hot loop. std::unordered_map field_chain_cache; field_chain_cache.reserve(refs.size() / 4); + mark("load_refs"); + + // Hoisted out of the loop and reserved to kMaxCandidatesToScore so the + // exact-match deep-copy below reuses the same element/string storage + // across references instead of re-allocating ~50 Candidate objects + // (each carrying 6 std::strings) on every ref. Hoisting is safe: the + // vector is fully rewritten each iteration (fuzzy path clears+appends, + // exact path resize+assigns before applyConstraints), so no state leaks + // between references. + std::vector candidates; + candidates.reserve(kMaxCandidatesToScore); + for (auto &ref : refs) { // ── P0.3: Find candidates by name — borrow the index entry ── // Instead of deep-copying it->second on every reference (each @@ -1061,7 +415,6 @@ int64_t ResolverPipeline::run() // sorts/scores). Single-candidate fast path and dispatch // expansion only read, so they use the shared reference — // results are bit-identical. - std::vector candidates; const std::vector *cands = nullptr; auto it = entity_index.find(ref.name); if (it != entity_index.end()) { @@ -1110,81 +463,20 @@ int64_t ResolverPipeline::run() continue; } fuzzy_hits++; + candidates + .clear(); // hoisted vector: start a fresh fuzzy batch for (auto fid : fuzzy_ids) { - if (!lk_st) - break; - Candidate c; - c.entity_id = fid; - sqlite3_bind_int64(lk_st, 1, - static_cast(fid)); - if (sqlite3_step(lk_st) == SQLITE_ROW) { - const char *n2 = - reinterpret_cast( - sqlite3_column_text( - lk_st, 0)); - const char *fp2 = - reinterpret_cast( - sqlite3_column_text( - lk_st, 1)); - const char *lang2 = - reinterpret_cast( - sqlite3_column_text( - lk_st, 2)); - c.name = n2 ? n2 : ""; - c.file_path = fp2 ? fp2 : ""; - c.language = - lang2 ? lang2 : - languageFromPath( - c.file_path); - c.module_path = modulePath(c.file_path); - // v0.6 (perf): precompute path components for fuzzy - // candidates too, so applyConstraints' dir/module - // scoring is identical to the exact-match path (an - // empty cand_dir here would silently zero the module - // and namespace scores and lose resolution precision). - { - size_t cs = - c.file_path.rfind('/'); - c.cand_dir = - (cs != - std::string::npos) ? - c.file_path.substr( - 0, cs) : - std::string(); - size_t cps = - c.cand_dir.rfind('/'); - c.cand_parent = - (cps != - std::string::npos) ? - c.cand_dir.substr( - 0, - cps) : - std::string(); - c.cand_module = c.cand_dir; - size_t cms = - c.cand_dir.rfind('/'); - if (cms != std::string::npos) - c.cand_module = - c.cand_dir.substr( - cms + - 1); - } - // Column 3 is arity (added to lk_sql SELECT above). - c.arity = sqlite3_column_int(lk_st, 3); - // Column 4 is kind (RecordKind), appended so - // constructor-target preference applies to - // fuzzy-resolved candidates too (M-11). - c.kind = sqlite3_column_int(lk_st, 4); - // Step 5: column 5 is qualified_name. - const char *qn2 = - reinterpret_cast( - sqlite3_column_text( - lk_st, 5)); - c.qualified_name = qn2 ? qn2 : ""; - } - sqlite3_reset(lk_st); + // Hydrate the full candidate from the in-memory + // entity_by_id map (built in Step 0) instead of a + // per-id SQL lookup — all fields + precomputed path + // components are byte-identical to the old lk_sql + // materialization. + auto eit = entity_by_id.find(fid); + if (eit == entity_by_id.end()) + continue; + Candidate c = *(eit->second); c.score = 0; - candidates.push_back(c); + candidates.push_back(std::move(c)); } // Fuzzy results were materialized into the local vector — // point cands at it so subsequent reads (size/front/ @@ -1459,8 +751,22 @@ int64_t ResolverPipeline::run() // mutable local copy is required only here — the borrowed index // entry stays intact (unless fuzzy already materialized the local // vector, in which case cands already points at it). - if (cands != &candidates) - candidates = *cands; + if (cands != &candidates) { + // Deep-copy the borrowed index entry into the hoisted local + // vector, reusing already-allocated element + string storage: + // resize() grows only when needed (keeps the existing + // elements), then element-wise assignment copies the strings + // into the retained buffers, and the trailing resize(n) drops + // any surplus from a previous, larger batch. This replaces the + // destructive `candidates = *cands` (free + realloc per ref) + // with in-place reuse across the ~450k reference loop. + const size_t n = cands->size(); + if (n > candidates.size()) + candidates.resize(n); + for (size_t i = 0; i < n; ++i) + candidates[i] = (*cands)[i]; + candidates.resize(n); + } applyConstraints(candidates, ref.caller_file, ref.name, ref.call_kind, ref.arity, ref.receiver_type); @@ -1604,6 +910,7 @@ int64_t ResolverPipeline::run() reason, // reason ref.call_site_file, ref.start_row, ref.start_col }); } + mark("resolve_loop"); // Free entity_index (no longer needed) entity_index.clear(); @@ -1615,87 +922,12 @@ int64_t ResolverPipeline::run() refs.clear(); refs.shrink_to_fit(); - // ── Batch insert all resolved edges ──────────────────────────── - // Single INSERT with multiple rows is faster than per-row INSERTs. - // Use a single transaction wrapping the batch for minimal WAL overhead. - if (!resolved_edges.empty()) { - store_->exec("BEGIN"); - for (auto &e : resolved_edges) { - sqlite3_bind_int64(ins_st, 1, - static_cast(e.caller_id)); - sqlite3_bind_int64(ins_st, 2, - static_cast(e.target_id)); - sqlite3_bind_int(ins_st, 3, e.edge_type); - sqlite3_bind_int64(ins_st, 4, - static_cast(project_id_)); - sqlite3_bind_text(ins_st, 5, e.resolve_strategy.c_str(), - -1, SQLITE_STATIC); - // Step 6: bind provenance columns (6-12). - sqlite3_bind_double(ins_st, 6, e.confidence); - sqlite3_bind_text(ins_st, 7, e.resolver.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(ins_st, 8, e.resolution_kind.c_str(), - -1, SQLITE_STATIC); - sqlite3_bind_text(ins_st, 9, e.reason.c_str(), -1, - SQLITE_STATIC); - sqlite3_bind_text(ins_st, 10, e.call_site_file.c_str(), - -1, SQLITE_STATIC); - sqlite3_bind_int(ins_st, 11, e.call_site_row); - sqlite3_bind_int(ins_st, 12, e.call_site_col); - int st_rc = sqlite3_step(ins_st); - if (st_rc != SQLITE_DONE && st_rc != SQLITE_CONSTRAINT) - fprintf(stderr, - "[module=resolver, method=run] " - "staging insert failed (rc=%d): %s\n", - st_rc, - sqlite3_errmsg(store_->handle())); - sqlite3_reset(ins_st); - } - store_->exec("COMMIT"); - } - resolved_edges.clear(); - resolved_edges.shrink_to_fit(); - - sqlite3_finalize(ins_st); - if (lk_st) - sqlite3_finalize(lk_st); - - // ── P1: Batch insert from staging to final tables ──────────── - // One INSERT SELECT is far cheaper than N individual INSERTs - // because SQLite can optimize the bulk path and avoid per-row - // index maintenance (indexes are recreated after bulk load in P2). - auto t_sql = Clock::now(); - - if (!store_->exec("INSERT OR IGNORE INTO relation " - "(project_id, source_id, target_id, type, " - " confidence, resolver, resolution_kind, reason, " - " call_site_file, call_site_row, call_site_col) " - "SELECT project_id, source_id, target_id, edge_type, " - " confidence, resolver, resolution_kind, reason, " - " call_site_file, call_site_row, call_site_col " - "FROM _resolved_edges")) { - fprintf(stderr, - "[module=resolver, method=run] " - "batch relation insert failed: %s\n", - store_->error().c_str()); - } - - if (!store_->exec("INSERT OR IGNORE INTO graph_edges " - "(project_id, source_node_id, target_node_id, " - " edge_type, resolve_strategy) " - "SELECT project_id, source_id, target_id, edge_type, " - " resolve_strategy " - "FROM _resolved_edges")) { - fprintf(stderr, - "[module=resolver, method=run] " - "batch graph_edges insert failed: %s\n", - store_->error().c_str()); - } - - int64_t sql_batch_ms = - std::chrono::duration_cast( - Clock::now() - t_sql) - .count(); + // ── Batch insert all resolved edges (pipeline_flush.cpp) ── + // Staging temp-table insert in one transaction, then bulk-copy into + // relation + graph_edges. Finalizes ins_st and reports elapsed ms. + int64_t sql_batch_ms = 0; + flushResolvedEdges(resolved_edges, ins_st, sql_batch_ms); + mark("sql_batch"); // ── Cleanup staging table ── store_->exec("DROP TABLE IF EXISTS _resolved_edges"); @@ -1726,8 +958,9 @@ int64_t ResolverPipeline::run() (long long)skipped_fuzzy_no_evidence, (long long)skipped_ambiguous, (long long)skipped_lang_mismatch, avg_candidates, (long long)total_entities, - (long long)total_imports, (long long)sql_batch_ms, + (long long)import_index_.size(), (long long)sql_batch_ms, (long long)total_ms); + mark("finalize_total"); return resolved_count; } diff --git a/engine/src/resolver/pipeline.h b/engine/src/resolver/pipeline.h index abc8829..c4afb30 100644 --- a/engine/src/resolver/pipeline.h +++ b/engine/src/resolver/pipeline.h @@ -10,6 +10,12 @@ #include "factors.h" #include "../store/store.h" +// Forward-declare sqlite3_stmt in the GLOBAL namespace (not inside +// `resolver`) so it matches the typedef sqlite3_stmt in sqlite3.h — +// a forward decl inside `namespace resolver` would create an unrelated +// resolver::sqlite3_stmt that breaks the sqlite3 API calls in pipeline.cpp. +struct sqlite3_stmt; + namespace resolver { @@ -142,6 +148,34 @@ class ResolverPipeline { std::vector factors; }; + /// A single resolved call edge staged for batch insert. Moved to class + /// scope so the batch-flush step (flushResolvedEdges) can live in its + /// own translation unit (pipeline_flush.cpp) under the 1000-line rule. + struct ResolvedEdge { + uint64_t caller_id; + uint64_t target_id; + int edge_type; + std::string resolve_strategy; + // Step 6 (plan §6.1): provenance fields. + double confidence; + std::string resolver; + std::string resolution_kind; + std::string reason; + std::string call_site_file; + int call_site_row; + int call_site_col; + }; + + /// Flush the staged resolved edges into _resolved_edges (staging temp + /// table) in one transaction, then bulk-copy into relation and + /// graph_edges. Finalizes ins_st. Extracted from run() so this TU stays + /// under the 1000-line rule. + /// @param resolved_edges Staged edges accumulated by the resolve loop. + /// @param ins_st Prepared staging INSERT (finalized here). + /// @param sql_batch_ms [out] milliseconds spent in the SQL flush. + void flushResolvedEdges(std::vector &resolved_edges, + sqlite3_stmt *ins_st, int64_t &sql_batch_ms); + /// Apply constraints to rank candidates. /// @param candidates Mutable list — sorted by score descending. /// @param caller_file The file where the call site resides. @@ -160,6 +194,28 @@ class ResolverPipeline { int caller_arity = 0, const std::string &receiver_type = ""); + /// Pre-load all project entities into a name-indexed candidate map, + /// plus an id -> candidate pointer map and the per-file import index. + /// Extracted from run() (pipeline_load.cpp) so the resolver split + /// keeps each translation unit under the 1000-line rule. + /// @param entity_index [out] name -> candidate vector, filled here. + /// @param entity_by_id [out] entity_id -> candidate pointer (points + /// into entity_index; lifetime == entity_index). + /// @param total_entities [out] number of loaded entity rows. + /// @return 0 on success, -1 if the SQL prepare fails (error logged). + int loadEntityIndex( + std::unordered_map> + &entity_index, + std::unordered_map &entity_by_id, + int64_t &total_entities); + + /// Pre-load the interface/trait implementation index and the global + /// struct-field / variable-type tables used by Step 8 dispatch and + /// field-chain resolution. Extracted from run() (pipeline_load.cpp). + /// Only reads members (interface_impl_index_, global_struct_fields_, + /// global_var_types_) and semantic_records; no caller state. + void loadDispatchIndex(); + /// Check if `callee_name` is imported in the file at `caller_file`. /// Returns the import target path if found, empty string otherwise. std::string checkImport(const std::string &caller_file, diff --git a/engine/src/resolver/pipeline_apply.cpp b/engine/src/resolver/pipeline_apply.cpp new file mode 100644 index 0000000..8d28903 --- /dev/null +++ b/engine/src/resolver/pipeline_apply.cpp @@ -0,0 +1,287 @@ +#include "pipeline.h" +#include "factors.h" +#include + +namespace resolver +{ + +void ResolverPipeline::applyConstraints(std::vector &candidates, + const std::string &caller_file, + const std::string &callee_name, + int call_kind, int caller_arity, + const std::string &receiver_type) +{ + // Build per-factor scores for each candidate using multi-factor scoring. + // + // v0.2.5 (perf fix): the original code built a full + // std::vector (name/detail heap strings) per candidate and + // fed it to computeTotalScore(). On large projects that is ~166k + // candidate evaluations × ~20 string allocations each ≈ 3.3M heap + // allocations — measured as the 90µs-per-candidate cost that made the + // resolver 93.8% of index time (goagent: 14.9s of 15.9s). Here we + // accumulate the weighted sum directly (pure double math, no strings) + // and capture only the ReceiverMatch score that the ambiguity gate needs + // (see receiver_bypass in run()). Every factor's weight/score pair and + // the final weighted-average formula are identical to the previous + // computeTotalScore path, so resolved edges are byte-for-byte unchanged. + // + // v0.2.5 (perf fix #2): the path-based factors (ImportMatch, + // NamespaceMatch, DistanceMatch) each re-derived caller_file's directory + // and module token with rfind/substr on every candidate. caller_file is + // fixed for the whole ref, so we parse it ONCE here (caller_dir / + // caller_parent / caller_module) and parse each candidate's path ONCE + // inside the loop, then compute the three path factors from those + // pre-parsed values. This removes ~N×3 redundant substring allocations + // per candidate. The scoring rules are kept IDENTICAL to + // factorImportMatch/factorNamespaceMatch/factorDistanceMatch in + // factors.cpp — do not change them independently. + size_t caller_slash = caller_file.rfind('/'); + std::string caller_dir = (caller_slash != std::string::npos) ? + caller_file.substr(0, caller_slash) : + ""; + size_t caller_parent_slash = caller_dir.rfind('/'); + std::string caller_parent = + (caller_parent_slash != std::string::npos) ? + caller_dir.substr(0, caller_parent_slash) : + ""; + // moduleTokenFromPath: the last path token (file's directory name). + // Mirrors the anonymous helper in factors.cpp used by + // factorImportMatch. When there is no slash, the whole dir is returned. + std::string caller_module = caller_dir; + { + size_t ms = caller_dir.rfind('/'); + if (ms != std::string::npos) + caller_module = caller_dir.substr(ms + 1); + } + auto anyImportMatches = [&](const std::vector &paths, + const std::string &mod) { + for (const auto &p : paths) { + if (p.find(mod) != std::string::npos) + return true; + } + return false; + }; + // ── Ref-level factor precompute (perf fix #3) ── + // factorCommonNamePenalty depends ONLY on the ref's callee_name, which + // is fixed across all candidates of this ref, yet it was called once per + // candidate (~166k calls). Compute it once here. + const double common_name_penalty = factorCommonNamePenalty(callee_name); + // When receiver_type is empty (dynamic/unknown), factorReceiverTypeMatch + // returns a constant neutral 0.5 regardless of the candidate — compute it + // once here instead of per candidate. When non-empty, build the ref-level + // context (prefix1/prefix2/rtype_lower) once so the per-candidate match + // does not reallocate those strings (perf fix #3). + const bool receiver_is_known = !receiver_type.empty(); + const double neutral_receiver_score = 0.5; + const ReceiverMatchContext receiver_ctx = + buildReceiverMatchContext(receiver_type); + // ── Ref-level ImportMatch forward cache (perf fix #4) ── + // caller_file is fixed for the whole ref, so the forward lookup + // import_index_[caller_file] yields the same vector for every + // candidate. Hoisting it out of the candidate loop removes N-1 + // redundant hashmap lookups per ref (measured on goagent: ~166k → + // ~24k lookups). The scoring semantics are unchanged — the pointer is + // non-null iff the original fwd_it != import_index_.end(). + auto caller_fwd_it = import_index_.find(caller_file); + const std::vector *caller_fwd_imports = + (caller_fwd_it != import_index_.end()) ? + &caller_fwd_it->second : + nullptr; + // ── Ref-level ImportMatch forward fused string (perf fix #5) ── + // anyImportMatches scans every path and does p.find(mod) per path. + // Fusing the (fixed-for-this-ref) forward import list into ONE + // NUL-separated string lets each candidate do a single find() over a + // contiguous buffer instead of N vector elements + N substring calls + // (CBM-style fused matching). Module names never contain NUL, so a + // match can never span the separator — the result is identical to the + // per-path scan (∃p: p.find(mod) != npos ⟺ joined.find(mod) != npos). + std::string caller_fwd_joined; + if (caller_fwd_imports != nullptr) { + for (const auto &p : *caller_fwd_imports) { + caller_fwd_joined += p; + caller_fwd_joined += '\x00'; + } + } + for (auto &c : candidates) { + double sum_weight = 0.0; + double sum_scored = 0.0; + // Accumulate one (weight, score) pair into the weighted sums. + // This mirrors computeTotalScore's loop without materializing a + // vector per candidate. + auto acc = [&](double weight, double score) { + sum_weight += weight; + sum_scored += weight * score; + }; + + // ── Candidate path components (v0.6) ── + // Precomputed once when the entity_index was loaded (pipeline.cpp + // entity_index build); values are byte-identical to the old inline + // rfind+substr derivation, so this is a pure-allocation win. + const std::string &cand_dir = c.cand_dir; + const std::string &cand_parent = c.cand_parent; + const std::string &cand_module = c.cand_module; + + // ── NamespaceMatch score (reused by ModuleMatch and the + // CommonNamePenalty same-module gate) ── + // Mirrors factorNamespaceMatch(caller_file, c.file_path): + // same dir → 1.0; same parent dir → 0.5; else 0.0. + double ns_score = 0.0; + if (!cand_dir.empty() && !caller_dir.empty()) { + if (caller_dir == cand_dir) + ns_score = kScoreExactMatch; + else if (caller_parent == cand_parent && + !caller_parent.empty()) + ns_score = kScoreSiblingModule; + } + + // Factor 1: ModuleMatch + acc(kWeightModuleMatch, ns_score); + + // Factor 2: ImportMatch — dominant weight for cross-module calls. + // Mirrors factorImportMatch(import_index_, caller_file, + // c.file_path, c.name): + // 1. same directory → 1.0 (no import needed) + // 2. forward: caller imports candidate_module → 1.0 + // 3. reverse: candidate imports caller_module → 1.0 + // 4. else 0.0 + { + double import_score = 0.0; + if (!caller_dir.empty() && caller_dir == cand_dir) { + import_score = 1.0; + } else { + // Forward: caller imports candidate module. + // Uses the ref-level fused string (perf fix #5): + // one find() over the contiguous buffer replaces + // the per-path scan of anyImportMatches. + if (!caller_fwd_joined.empty() && + caller_fwd_joined.find(cand_module) != + std::string::npos) + import_score = 1.0; + else { + auto rev_it = + import_index_.find(c.file_path); + if (rev_it != import_index_.end() && + anyImportMatches(rev_it->second, + caller_module)) + import_score = 1.0; + } + } + acc(kWeightImportMatch, import_score); + } + + // Factor 3: NamespaceMatch + acc(kWeightNamespaceMatch, ns_score); + + // Factor 4: SignatureMatch — compares the call site's arity + // (from the reference row) against each candidate's arity. + // Previously the caller arity was hardcoded to 0, which caused + // factorSignatureMatch to penalize every candidate with a known + // arity (returning -0.5) while rewarding candidates with unknown + // arity (returning +0.5) — the exact opposite of correct + // overload resolution. Thread the real reference arity through + // so exact-arity overloads score highest. + acc(kWeightSignatureMatch, + factorSignatureMatch(caller_arity, c.arity)); + + // Factor 5: DistanceMatch — mirrors factorDistanceMatch: + // same file → 1.0; same directory → 0.3; else 0.0. + { + double dist_score = 0.0; + if (caller_file == c.file_path) + dist_score = kScoreExactMatch; + else if (!caller_dir.empty() && caller_dir == cand_dir) + dist_score = kScoreSameDirectory; + acc(kWeightDistanceMatch, dist_score); + } + + // Factor 6: ConstructorMatch + acc(kWeightConstructorMatch, + factorConstructorMatch(callee_name, c.name, c.kind)); + + // Factor 7: ReceiverMatch (Step 5: now type-based, not directory) + // Replaced factorReceiverMatch (directory heuristic) with + // factorReceiverTypeMatch (actual receiver_type evidence). + // When receiver_type is known (e.g. "Box"), candidates whose + // qualified_name contains "Box::" or "Box." score 1.0. When + // receiver_type is empty (dynamic/unknown), the factor returns + // 0.5 (neutral) so it does not distort the ranking. + // + // The per-candidate ReceiverMatch score is captured separately + // (c.receiver_score) because the ambiguity gate in run() + // (receiver_bypass) reads only this one factor — no other part of + // the hot loop consumes the full FactorResult vector. + { + // v0.2.5 (perf fix #3): when receiver_type is empty the score + // is a constant neutral 0.5 (precomputed). When non-empty, use + // the ref-level pre-parsed context so per-candidate string + // allocations (prefix1/prefix2/rtype_lower) are avoided. + double recv = neutral_receiver_score; + if (receiver_is_known) { + recv = factorReceiverTypeMatchPrecomp( + receiver_ctx, c.qualified_name, + c.file_path); + } + c.receiver_score = recv; + acc(kWeightReceiverMatch, recv); + } + + // Factor 8: CommonNamePenalty — reduce score for very common names + // Only applies to cross-module candidates (same-module should not be + // penalized). The penalty value itself depends only on the ref's + // callee_name and is precomputed once per ref (perf fix #3). + { + // Only penalize if candidate is in a different module + bool same_module = (ns_score > 0.0); + double penalty = same_module ? 0.0 : + common_name_penalty; + acc(kWeightCommonNamePenalty, -penalty); + } + + // Factor 9: CallKindMatch — adjust scoring based on call kind. + // Constructor calls (3): boost for cross-module matching. + // Interface dispatches (2): reduce confidence (harder to resolve). + // Method calls (1): slight cross-module penalty. + if (call_kind != kCallKindDirect) { + double kscore = 0.0; + if (call_kind == kCallKindConstructor) + kscore = + 0.3; // boost: constructors expected to cross module + else if (call_kind == kCallKindInterface) + kscore = + -0.3; // penalty: interface dispatch is harder to resolve + else if (call_kind == kCallKindMethod) + kscore = + -0.1; // slight penalty: methods usually same-module + acc(kWeightCallKindMatch, kscore); + } + + // Factor 10: DefinitionMatch — for C/C++, prefer symbols defined + // in a source file (.c/.cpp/.cc/...) over those declared only in + // a header (.h/.hpp/...). This breaks the previous arbitrary tie + // between a header prototype and a source definition that scored + // identically (Finding #8). Non-C/C++ languages return 1.0 + // (neutral), so their ranking is unaffected. + acc(kWeightDefinitionMatch, + factorDefinitionMatch(c.language, c.file_path)); + + // VisibilityCheck was moved to a hard filter in run() to + // ensure language visibility rules (e.g. Go unexported names) + // are applied as hard rejections, not weighted factors — a + // weighted factor can be overcome by other factors, but a + // hard language rule must be absolute. + + // Weighted average, identical to computeTotalScore's formula. + c.total_score = (sum_weight > 0.0) ? (sum_scored / sum_weight) : + 0.0; + // c.factors is intentionally NOT populated here (perf); the hot + // loop never reads it. If a future debug path needs factor detail, + // rebuild it lazily for the resolved candidate only. + } + + std::sort(candidates.begin(), candidates.end(), + [](const Candidate &a, const Candidate &b) { + return a.total_score > b.total_score; + }); +} + +} // namespace resolver diff --git a/engine/src/resolver/pipeline_flush.cpp b/engine/src/resolver/pipeline_flush.cpp new file mode 100644 index 0000000..6713d5d --- /dev/null +++ b/engine/src/resolver/pipeline_flush.cpp @@ -0,0 +1,95 @@ +#include "pipeline.h" +#include +#include +#include + +namespace resolver +{ + +void ResolverPipeline::flushResolvedEdges( + std::vector &resolved_edges, sqlite3_stmt *ins_st, + int64_t &sql_batch_ms) +{ + using Clock = std::chrono::steady_clock; + + // ── Batch insert all resolved edges ──────────────────────────── + // Single INSERT with multiple rows is faster than per-row INSERTs. + // Use a single transaction wrapping the batch for minimal WAL overhead. + if (!resolved_edges.empty()) { + store_->exec("BEGIN"); + for (auto &e : resolved_edges) { + sqlite3_bind_int64(ins_st, 1, + static_cast(e.caller_id)); + sqlite3_bind_int64(ins_st, 2, + static_cast(e.target_id)); + sqlite3_bind_int(ins_st, 3, e.edge_type); + sqlite3_bind_int64(ins_st, 4, + static_cast(project_id_)); + sqlite3_bind_text(ins_st, 5, e.resolve_strategy.c_str(), + -1, SQLITE_STATIC); + // Step 6: bind provenance columns (6-12). + sqlite3_bind_double(ins_st, 6, e.confidence); + sqlite3_bind_text(ins_st, 7, e.resolver.c_str(), -1, + SQLITE_STATIC); + sqlite3_bind_text(ins_st, 8, e.resolution_kind.c_str(), + -1, SQLITE_STATIC); + sqlite3_bind_text(ins_st, 9, e.reason.c_str(), -1, + SQLITE_STATIC); + sqlite3_bind_text(ins_st, 10, e.call_site_file.c_str(), + -1, SQLITE_STATIC); + sqlite3_bind_int(ins_st, 11, e.call_site_row); + sqlite3_bind_int(ins_st, 12, e.call_site_col); + int st_rc = sqlite3_step(ins_st); + if (st_rc != SQLITE_DONE && st_rc != SQLITE_CONSTRAINT) + fprintf(stderr, + "[module=resolver, method=run] " + "staging insert failed (rc=%d): %s\n", + st_rc, + sqlite3_errmsg(store_->handle())); + sqlite3_reset(ins_st); + } + store_->exec("COMMIT"); + } + resolved_edges.clear(); + resolved_edges.shrink_to_fit(); + + sqlite3_finalize(ins_st); + + // ── P1: Batch insert from staging to final tables ──────────── + // One INSERT SELECT is far cheaper than N individual INSERTs + // because SQLite can optimize the bulk path and avoid per-row + // index maintenance (indexes are recreated after bulk load in P2). + auto t_sql = Clock::now(); + + if (!store_->exec("INSERT OR IGNORE INTO relation " + "(project_id, source_id, target_id, type, " + " confidence, resolver, resolution_kind, reason, " + " call_site_file, call_site_row, call_site_col) " + "SELECT project_id, source_id, target_id, edge_type, " + " confidence, resolver, resolution_kind, reason, " + " call_site_file, call_site_row, call_site_col " + "FROM _resolved_edges")) { + fprintf(stderr, + "[module=resolver, method=run] " + "batch relation insert failed: %s\n", + store_->error().c_str()); + } + + if (!store_->exec("INSERT OR IGNORE INTO graph_edges " + "(project_id, source_node_id, target_node_id, " + " edge_type, resolve_strategy) " + "SELECT project_id, source_id, target_id, edge_type, " + " resolve_strategy " + "FROM _resolved_edges")) { + fprintf(stderr, + "[module=resolver, method=run] " + "batch graph_edges insert failed: %s\n", + store_->error().c_str()); + } + + sql_batch_ms = std::chrono::duration_cast( + Clock::now() - t_sql) + .count(); +} + +} // namespace resolver diff --git a/engine/src/resolver/pipeline_load.cpp b/engine/src/resolver/pipeline_load.cpp new file mode 100644 index 0000000..60e4a97 --- /dev/null +++ b/engine/src/resolver/pipeline_load.cpp @@ -0,0 +1,462 @@ +#include "pipeline.h" +#include +#include +#include +#include +#include + +namespace resolver +{ + +namespace +{ +// Infer the source language from a file path's extension. Mirrors the +// helper in pipeline.cpp (kept as a per-TU copy because it lives in an +// anonymous namespace there; ODR-safe since anonymous namespaces isolate +// each translation unit). +std::string languageFromPath(const std::string &file_path) +{ + size_t dot = file_path.rfind('.'); + if (dot == std::string::npos) + return ""; + std::string ext = file_path.substr(dot); + std::string lower; + lower.reserve(ext.size()); + for (char ch : ext) + lower.push_back(static_cast( + std::tolower(static_cast(ch)))); + if (lower == ".cpp" || lower == ".cc" || lower == ".cxx" || + lower == ".c" || lower == ".h" || lower == ".hpp" || + lower == ".hh" || lower == ".hxx") + return "cpp"; + if (lower == ".rs") + return "rust"; + if (lower == ".py") + return "python"; + if (lower == ".go") + return "go"; + if (lower == ".ts" || lower == ".tsx") + return "typescript"; + if (lower == ".js" || lower == ".jsx") + return "javascript"; + if (lower == ".java") + return "java"; + return ""; +} +} // namespace + +int ResolverPipeline::loadEntityIndex( + std::unordered_map> &entity_index, + std::unordered_map &entity_by_id, + int64_t &total_entities) +{ + // ── Step 0: Pre-load all entities into a name-indexed HashMap ── + // This avoids one SQL query per reference (the main bottleneck). + { + std::string idx_sql = + // Include arity so factorSignatureMatch can distinguish + // same-name overloads (init()/init(int)/init(string)). + // entity.arity was added in v0.5+ migration (store_schema.cpp:470). + // Without this column in the SELECT, c.arity defaulted to 0 + // and every candidate scored kScorePartialMatch (0.5), + // letting std::sort pick the winner by unstable order. + // See CODE_REVIEW_FINDINGS_2026-07-19.md C2. + // Include kind (appended as column 5) so + // factorConstructorMatch can prefer Class/Struct targets; + // previously kind was hardcoded 0 in the call, so the + // constructor factor always returned 0.0 (M-11). + // Step 5: include qualified_name (column 6) so + // factorReceiverTypeMatch can match "Box::draw" against + // receiver_type="Box" instead of using directory heuristics. + "SELECT id, name, file_path, language, arity, kind, qualified_name " + "FROM entity " + "WHERE project_id=? AND name != ''"; + sqlite3_stmt *idx_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), idx_sql.c_str(), -1, + &idx_st, nullptr) != SQLITE_OK) { + fprintf(stderr, + "[module=resolver, method=run] " + "prepare entity index failed: %s\n", + sqlite3_errmsg(store_->handle())); + return -1; + } + sqlite3_bind_int64(idx_st, 1, + static_cast(project_id_)); + while (sqlite3_step(idx_st) == SQLITE_ROW) { + Candidate c; + c.entity_id = static_cast( + sqlite3_column_int64(idx_st, 0)); + const char *n = reinterpret_cast( + sqlite3_column_text(idx_st, 1)); + const char *fp = reinterpret_cast( + sqlite3_column_text(idx_st, 2)); + const char *lang = reinterpret_cast( + sqlite3_column_text(idx_st, 3)); + c.name = n ? n : ""; + c.file_path = fp ? fp : ""; + c.language = lang ? lang : + languageFromPath(c.file_path); + c.module_path = modulePath(c.file_path); + // v0.6 (perf): precompute the path components that + // applyConstraints derives from c.file_path on every candidate. + // Computing them once here (dir/parent/module token) removes + // per-ref heap allocations in the hot loop; the values are + // byte-identical to the old inline rfind+substr derivation so no + // score changes. + { + size_t cs = c.file_path.rfind('/'); + c.cand_dir = (cs != std::string::npos) ? + c.file_path.substr(0, cs) : + std::string(); + size_t cps = c.cand_dir.rfind('/'); + c.cand_parent = + (cps != std::string::npos) ? + c.cand_dir.substr(0, cps) : + std::string(); + c.cand_module = c.cand_dir; + size_t cms = c.cand_dir.rfind('/'); + if (cms != std::string::npos) + c.cand_module = + c.cand_dir.substr(cms + 1); + } + // Column 4 is arity (added to SELECT above). Default 0 + // if NULL — matches entity.arity DEFAULT 0 so callers + // that never set arity behave as "unknown arity". + c.arity = sqlite3_column_int(idx_st, 4); + // Column 5 is kind (RecordKind). Default 0 if NULL — + // matches entity.kind NOT NULL semantics; 0 = Function, + // so non-type candidates correctly score 0.0 on the + // constructor factor. + c.kind = sqlite3_column_int(idx_st, 5); + // Step 5: column 6 is qualified_name. Used by + // factorReceiverTypeMatch to match "Box::draw" against + // receiver_type="Box". Empty for languages that don't + // populate it (e.g. Go), causing the factor to fall back + // to file-path matching. + const char *qn = reinterpret_cast( + sqlite3_column_text(idx_st, 6)); + c.qualified_name = qn ? qn : ""; + c.score = 0; + entity_index[c.name].push_back(c); + total_entities++; + } + sqlite3_finalize(idx_st); + } + + // Build an id -> Candidate map so fuzzy-resolved entity ids can be + // hydrated into full candidates without a per-id SQL lookup. The + // entity_index above already carries every field the old lk_sql + // lookup returned (name, file_path, language, arity, kind, + // qualified_name) plus precomputed path components; copying from it + // is byte-identical to the previous SQL materialization. + entity_by_id.reserve(static_cast(total_entities)); + for (const auto &entry : entity_index) { + for (const auto &c : entry.second) + entity_by_id[c.entity_id] = &c; + } + + // ── Step 0b: Pre-load all imports into a file_path-indexed HashMap ── + // This is the core fix for the 174s bottleneck: factorImportMatch + // previously ran `SELECT COUNT(*) FROM import WHERE project_id=? AND + // file_path=? AND target_path LIKE '%module_name%'` per candidate. + // The leading-% LIKE is non-sargable, forcing a FULL TABLE SCAN on + // the import table for each of the ~313k candidate evaluations + // (108k refs × 2.9 avg candidates × 1-2 SQL queries each). + // + // We load every import row once into import_index_ (file_path -> + // list of target_path strings) and let factorImportMatch match + // in-memory with SQLite-exact LIKE semantics. Resolved edges are + // IDENTICAL to the SQL implementation; only the access path changes. + import_index_.clear(); + { + std::string imp_sql = + "SELECT file_path, target_path FROM import " + "WHERE project_id=?"; + sqlite3_stmt *imp_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), imp_sql.c_str(), -1, + &imp_st, nullptr) != SQLITE_OK) { + fprintf(stderr, + "[module=resolver, method=run] " + "prepare import index failed: %s\n", + sqlite3_errmsg(store_->handle())); + return -1; + } + sqlite3_bind_int64(imp_st, 1, + static_cast(project_id_)); + while (sqlite3_step(imp_st) == SQLITE_ROW) { + const char *fp = reinterpret_cast( + sqlite3_column_text(imp_st, 0)); + const char *tp = reinterpret_cast( + sqlite3_column_text(imp_st, 1)); + std::string file_path = fp ? fp : ""; + std::string target_path = tp ? tp : ""; + // NOTE: empty file_path rows are intentionally KEPT. The + // original SQL used an exact `file_path = ?` predicate, + // which matches empty-file_path rows when caller_file is + // also empty; dropping them would change matching results + // for such (rare) call sites and break identical-edge + // semantics. They land in the "" bucket and are only + // consulted when a caller's file_path is empty. + import_index_[file_path].push_back( + std::move(target_path)); + } + sqlite3_finalize(imp_st); + } + return 0; +} + +void ResolverPipeline::loadDispatchIndex() +{ + // ── Step 8 (plan §8.1): Pre-load interface/trait implementations ── + // InterfaceImpl records (kind=20) store (name=implementing_type, + // type_name=interface_name). We build a map from interface name + // to all implementing types, so the hot loop can expand + // Interface/Virtual dispatch calls into bounded candidate sets. + interface_impl_index_.clear(); + int64_t total_iface_impls = 0; + { + // semantic_records.kind=20 is InterfaceImpl. The `name` column + // holds the implementing type, `type_name` holds the interface. + std::string iface_sql = + "SELECT name, type_name FROM semantic_records " + "WHERE project_id=? AND kind=20 AND name != '' " + "AND type_name != ''"; + sqlite3_stmt *iface_st = nullptr; + if (sqlite3_prepare_v2(store_->handle(), iface_sql.c_str(), -1, + &iface_st, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(iface_st, 1, + static_cast(project_id_)); + while (sqlite3_step(iface_st) == SQLITE_ROW) { + const char *impl = + reinterpret_cast( + sqlite3_column_text(iface_st, + 0)); + const char *iface = + reinterpret_cast( + sqlite3_column_text(iface_st, + 1)); + if (impl && iface) + interface_impl_index_[iface].push_back( + impl); + total_iface_impls++; + } + sqlite3_finalize(iface_st); + } + } + + // ── Step 8 (plan §8.1b): cross-file interface method-set matching ── + // The visitor's kind=20 records only cover same-file (struct, + // interface) pairs — Go interfaces are usually declared in one file + // and implemented in another, so those never match in-file. This + // global pass reconstructs method sets from the per-method qualified + // names ("Struct.method" set by handleMethodDecl, "Interface.method" + // set by handleInterfaceMethod) and re-runs the subset check across + // ALL files, supplementing interface_impl_index_ with cross-file + // implementations. + { + // Interface entity names (kind=3) — used to classify a method's + // qualified-name prefix as an interface vs a struct. + std::unordered_set iface_names; + { + std::string names_sql = + "SELECT name FROM semantic_records " + "WHERE project_id=? AND kind=3 AND name != ''"; + sqlite3_stmt *nst = nullptr; + if (sqlite3_prepare_v2(store_->handle(), + names_sql.c_str(), -1, &nst, + nullptr) == SQLITE_OK) { + sqlite3_bind_int64( + nst, 1, + static_cast(project_id_)); + while (sqlite3_step(nst) == SQLITE_ROW) { + const char *n = + reinterpret_cast( + sqlite3_column_text(nst, + 0)); + if (n) + iface_names.insert(n); + } + sqlite3_finalize(nst); + } + } + // Method records with qualified names — split "Type.method". + std::unordered_map> + iface_methods; // interface name -> its methods + std::unordered_map> + struct_methods; // struct type -> its methods + { + std::string meth_sql = + "SELECT qualified_name FROM semantic_records " + "WHERE project_id=? AND kind=1 AND " + "qualified_name != ''"; + sqlite3_stmt *mst = nullptr; + if (sqlite3_prepare_v2(store_->handle(), + meth_sql.c_str(), -1, &mst, + nullptr) == SQLITE_OK) { + sqlite3_bind_int64( + mst, 1, + static_cast(project_id_)); + while (sqlite3_step(mst) == SQLITE_ROW) { + const char *qn = + reinterpret_cast( + sqlite3_column_text(mst, + 0)); + if (!qn) + continue; + std::string q(qn); + size_t dot = q.find('.'); + if (dot == std::string::npos) + continue; + std::string type_name = + q.substr(0, dot); + std::string method = q.substr(dot + 1); + if (type_name.empty() || method.empty()) + continue; + if (iface_names.count(type_name) > 0) + iface_methods[type_name] + .push_back(method); + else + struct_methods[type_name] + .push_back(method); + } + sqlite3_finalize(mst); + } + } + // Global subset check: struct implements interface iff the + // struct's method set contains every interface method. + // v0.2.5 (perf fix): pre-index each struct's method set into a + // hash set once, then the interface-implements check is O(1) per + // method instead of a linear std::find. Without this, the + // for-interface × for-struct × for-method triple loop was O(I×S×M) + // — quadratic and noticeable on large Go projects with many + // interfaces/structs. + std::unordered_map> + struct_method_set; + struct_method_set.reserve(struct_methods.size()); + for (const auto &sentry : struct_methods) { + auto &s = struct_method_set[sentry.first]; + s.reserve(sentry.second.size()); + s.insert(sentry.second.begin(), sentry.second.end()); + } + for (const auto &iface_entry : iface_methods) { + const std::string &iface = iface_entry.first; + const auto &imethods = iface_entry.second; + if (imethods.empty()) + continue; + for (const auto &sentry : struct_methods) { + const std::string &stype = sentry.first; + if (stype == iface) + continue; + auto smit = struct_method_set.find(stype); + if (smit == struct_method_set.end()) + continue; + const auto &smethods = smit->second; + bool implements_all = true; + for (const auto &m : imethods) { + if (smethods.find(m) == + smethods.end()) { + implements_all = false; + break; + } + } + if (implements_all) { + // Avoid duplicating an entry the + // visitor's kind=20 pass already added. + auto &impls = + interface_impl_index_[iface]; + if (std::find(impls.begin(), + impls.end(), + stype) == impls.end()) { + impls.push_back(stype); + total_iface_impls++; + } + } + } + } + if (total_iface_impls > 0) { + fprintf(stderr, + "[module=resolver, method=run] interface_impl_index: " + "%lld implementation(s) loaded (%d interface(s))\n", + static_cast(total_iface_impls), + static_cast(interface_impl_index_.size())); + } + } + + // ── Step 8 (plan §8.1c): rebuild the global struct field table ── + // The Go visitor persists each struct field as a TypeRef record + // (kind=14) under the struct entity (kind=2): name = field name, + // type_name = field type. Rebuilding here makes the table complete + // across files, so field-chain receivers (r.pluginBus.AfterStep) + // whose receiver_type was empty at visit time (struct declared in + // another file) can be resolved before dispatch expansion. + global_struct_fields_.clear(); + { + std::string field_sql = + "SELECT p.name, t.name, t.type_name " + "FROM semantic_records t " + "JOIN semantic_records p ON t.parent_id = p.original_id " + "AND p.project_id = t.project_id " + "AND p.file_path = t.file_path " + "WHERE t.project_id=? AND t.kind=17 AND p.kind=2 " + "AND t.name != '' AND t.type_name != ''"; + sqlite3_stmt *fst = nullptr; + if (sqlite3_prepare_v2(store_->handle(), field_sql.c_str(), -1, + &fst, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(fst, 1, + static_cast(project_id_)); + while (sqlite3_step(fst) == SQLITE_ROW) { + const char *stype = + reinterpret_cast( + sqlite3_column_text(fst, 0)); + const char *fname = + reinterpret_cast( + sqlite3_column_text(fst, 1)); + const char *ftype = + reinterpret_cast( + sqlite3_column_text(fst, 2)); + if (stype && fname && ftype) + global_struct_fields_[stype][fname] = + ftype; + } + sqlite3_finalize(fst); + } + } + + // ── Step 8.1c (plan §8): global caller variable-type table ── + // The Go visitor persists method receivers and declared variables + // as TypeRef records (kind=14) under the containing function/method + // entity (kind=0/1): name = variable name, type_name = its type. + // Rebuilding here gives the field-chain resolver the first-segment + // type ("r" -> "Runner") when resolving "r.pluginBus.AfterStep". + global_var_types_.clear(); + { + std::string vtype_sql = + "SELECT t.name, t.type_name FROM semantic_records t " + "JOIN semantic_records p ON t.parent_id = p.original_id " + "AND p.project_id = t.project_id " + "AND p.file_path = t.file_path " + "WHERE t.project_id=? AND t.kind=17 " + "AND p.kind IN (0,1) " + "AND t.name != '' AND t.type_name != ''"; + sqlite3_stmt *vst = nullptr; + if (sqlite3_prepare_v2(store_->handle(), vtype_sql.c_str(), -1, + &vst, nullptr) == SQLITE_OK) { + sqlite3_bind_int64(vst, 1, + static_cast(project_id_)); + while (sqlite3_step(vst) == SQLITE_ROW) { + const char *vname = + reinterpret_cast( + sqlite3_column_text(vst, 0)); + const char *vtype = + reinterpret_cast( + sqlite3_column_text(vst, 1)); + if (vname && vtype) + global_var_types_[vname].push_back( + vtype); + } + sqlite3_finalize(vst); + } + } +} + +} // namespace resolver diff --git a/engine/src/store/store_batch.cpp b/engine/src/store/store_batch.cpp index 5adfd88..065bb3a 100644 --- a/engine/src/store/store_batch.cpp +++ b/engine/src/store/store_batch.cpp @@ -422,8 +422,29 @@ bool GraphStore::insertFileResultBatch(uint64_t project_id, }; std::vector all_recs; for (auto &br : batch_records) - for (auto &r : br.second) + for (auto &r : br.second) { + // Skip Literal records: they are produced for every + // numeric/string literal token but are consumed by no + // downstream stage (buildGraph, resolver, FTS, entity all + // ignore kind=Literal). Dropping them here reduces + // semantic_records rows ~16% on typical C/C++ trees, + // shrinking the bulk INSERT and index rebuild hot spots. + if (r.kind == ir::RecordKind::Literal) + continue; + // Skip Variable records: buildGraph's node pass only + // materializes Function/Class entities from + // semantic_records and ignores Variable rows, and no + // query (symbol lookup, call graph, FTS) reads them from + // the DB — they are emitted for every declared variable + // and account for ~57% of semantic_records rows on + // typical C/C++ trees. The in-memory GraphBuilder still + // sees fr.records unchanged, so single-file symbol-graph + // queries keep Variable nodes; only the bulk DB write + // (used by full-index buildGraph) drops them. + if (r.kind == ir::RecordKind::Variable) + continue; all_recs.push_back({ &r, &br.first }); + } constexpr size_t kMaxBatch = 500; for (size_t off = 0; off < all_recs.size(); off += kMaxBatch) { diff --git a/engine/src/store/store_graph.cpp b/engine/src/store/store_graph.cpp index bdfb3ae..3060cf8 100644 --- a/engine/src/store/store_graph.cpp +++ b/engine/src/store/store_graph.cpp @@ -628,6 +628,21 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, // a single flat join instead of running a subquery per import row. // Results are identical: same s.id for the same (scope,entity,sr-file) // combination, with the same COALESCE(...,0) fallback and LIMIT 1. + // + // v0.7 (perf): the correlated subquery ran once PER import row — for + // rustc's ~41k imports each execution joined scope+entity+semantic_ + // records, making this the single largest cost in buildGraph's "scope" + // phase. Two changes make it fast WITHOUT changing results: + // 1. idx_scope_kind_name(project_id, kind, name) added in + // store_schema.cpp turns the scope lookup from a full-table scan + // (129893 entities x 26975 scopes) into an index seek; + // 2. the entity join uses idx_entity_file(project_id, file_path). + // NOTE: an UPDATE ... FROM rewrite was attempted but SQLite rejects + // referencing the target table (import) inside the FROM join clause + // ("no such column: import.id"), so the correlated form is kept. + // exec() result is now checked so a failed update can never be + // silently swallowed again (previous code ignored the return value, + // which left every import.source_scope_id at its 0 default). { std::string imp_scope_sql = "UPDATE import SET source_scope_id = " @@ -642,7 +657,12 @@ bool GraphStore::buildGraph(uint64_t project_id, bool build_calls, " LIMIT 1) " "WHERE project_id=" + std::to_string(project_id); - exec(imp_scope_sql.c_str()); + if (!exec(imp_scope_sql.c_str())) { + fprintf(stderr, + "[module=store, method=buildGraph] " + "UPDATE import.source_scope_id failed: %s\n", + error().c_str()); + } } auto t_scope = Clock::now(); diff --git a/engine/src/store/store_membulk.cpp b/engine/src/store/store_membulk.cpp index fe1013a..56b4f40 100644 --- a/engine/src/store/store_membulk.cpp +++ b/engine/src/store/store_membulk.cpp @@ -64,6 +64,7 @@ bool GraphStore::dropSemanticRecordIndexes() "DROP INDEX IF EXISTS idx_sr_fp_parent", "DROP INDEX IF EXISTS idx_sr_kind_name", "DROP INDEX IF EXISTS idx_sr_kind_fp", + "DROP INDEX IF EXISTS idx_sr_proj_file_oid", }; bool ok = true; for (auto *sql : drop_sqls) { @@ -98,6 +99,7 @@ bool GraphStore::createSemanticRecordIndexes() "CREATE INDEX IF NOT EXISTS idx_sr_fp_parent ON semantic_records(file_path, parent_id)", "CREATE INDEX IF NOT EXISTS idx_sr_kind_name ON semantic_records(project_id, kind, name, language)", "CREATE INDEX IF NOT EXISTS idx_sr_kind_fp ON semantic_records(project_id, kind, file_path)", + "CREATE INDEX IF NOT EXISTS idx_sr_proj_file_oid ON semantic_records(project_id, file_path, original_id)", }; bool ok = true; for (auto *sql : create_sqls) { diff --git a/engine/src/store/store_schema.cpp b/engine/src/store/store_schema.cpp index 6154450..89a7c96 100644 --- a/engine/src/store/store_schema.cpp +++ b/engine/src/store/store_schema.cpp @@ -321,6 +321,21 @@ bool GraphStore::createSchema() -- previously timed out; api/ at 46k rows finished in 2.4s). CREATE INDEX IF NOT EXISTS idx_sr_oid ON semantic_records(project_id, original_id); + -- Index for ResolverPipeline's global field/variable type self-joins + -- (loadVarTypesStruct): those JOIN semantic_records t ON + -- t.parent_id = p.original_id AND p.file_path = t.file_path + -- (kind=17 TypeRef -> parent entity). idx_sr_oid alone covers only + -- (project_id, original_id); because original_id is PER-FILE it + -- collides across files, so SQLite pulls every same-numbered row + -- from every file and then filters file_path by a rowid lookup — + -- on rustc (1M semantic_records) that made this single phase + -- ~1.4s and the whole resolver ~7.2s. This composite index keys on + -- (project_id, file_path, original_id) so the join becomes an + -- index seek (measured ~38ms, ~37x faster). Kept separate from + -- idx_sr_oid because plain (project_id, original_id) lookups (no + -- file_path) still need the latter's leading columns. + CREATE INDEX IF NOT EXISTS idx_sr_proj_file_oid + ON semantic_records(project_id, file_path, original_id); -- Index for call edges name matching: (project_id, kind, name) covers the WHERE + JOIN -- Language added for P3 cross-file matching: sr.language = callee.language CREATE INDEX IF NOT EXISTS idx_sr_kind_name ON semantic_records(project_id, kind, name, language); @@ -619,6 +634,15 @@ bool GraphStore::createSchema() CREATE INDEX IF NOT EXISTS idx_reference_project ON reference(project_id, name); CREATE INDEX IF NOT EXISTS idx_scope_project ON scope(project_id, parent_id); + -- Lookup index for scope joins by (kind, name): buildGraph's + -- function-scope INSERT (JOIN scope s ON s.kind=1 AND + -- s.name=e.module_path) and the import.source_scope_id UPDATE + -- correlated subquery both filter on kind + name. Without it + -- those joins scan the whole scope table per entity/import row + -- (rustc: 129893 entities x 26975 scopes), dominating the + -- buildGraph "scope" phase. + CREATE INDEX IF NOT EXISTS idx_scope_kind_name + ON scope(project_id, kind, name); CREATE INDEX IF NOT EXISTS idx_import_project ON import(project_id, alias); -- workflow: high-level business flow (e.g. "Login"). diff --git a/engine/tests/test_qualified_id_ast.cpp b/engine/tests/test_qualified_id_ast.cpp index 2f46091..c57583f 100644 --- a/engine/tests/test_qualified_id_ast.cpp +++ b/engine/tests/test_qualified_id_ast.cpp @@ -61,17 +61,19 @@ static int recordKindByName(sqlite3 *db, int64_t pid, const char *name) return kind; } -/// Count graph_edges of a given edge_type between two node names. +/// Count relation rows of a given edge_type between two node names. +/// v0.2.5 migrated canonical graph data from graph_edges/graph_nodes +/// (no longer written) to relation/entity; query the live tables. static int edgeCount(sqlite3 *db, int64_t pid, const char *src_name, const char *tgt_name, int edge_type) { sqlite3_stmt *st = nullptr; const char *sql = - "SELECT COUNT(*) FROM graph_edges ge " - "JOIN graph_nodes gn_s ON gn_s.id=ge.source_node_id " - "JOIN graph_nodes gn_t ON gn_t.id=ge.target_node_id " - "WHERE ge.project_id=? AND ge.edge_type=? " - "AND gn_s.name=? AND gn_t.name=?"; + "SELECT COUNT(*) FROM relation r " + "JOIN entity e_s ON e_s.id=r.source_id AND e_s.project_id=r.project_id " + "JOIN entity e_t ON e_t.id=r.target_id AND e_t.project_id=r.project_id " + "WHERE r.project_id=? AND r.type=? " + "AND e_s.name=? AND e_t.name=?"; if (sqlite3_prepare_v2(db, sql, -1, &st, nullptr) != SQLITE_OK) return 0; sqlite3_bind_int64(st, 1, pid); diff --git a/engine/tests/test_typed_relation_query.cpp b/engine/tests/test_typed_relation_query.cpp index ccba7a0..504fcff 100644 --- a/engine/tests/test_typed_relation_query.cpp +++ b/engine/tests/test_typed_relation_query.cpp @@ -127,7 +127,13 @@ int main() // SQLite's dash form — clean both spellings. store::GraphStore store; - assert(store.open(kDbPath)); + // Must call open() unconditionally: a bare assert(store.open(...)) + // evaluates to nothing under -DNDEBUG (Release builds), leaving db_ + // null and turning every later prepare into "out of memory" + SEGV. + if (!store.open(kDbPath)) { + fprintf(stderr, "FAIL: store.open(%s)\n", kDbPath); + return 1; + } uint64_t project_id = store.createProject("/typed-rel", "typed-rel"); assert(project_id > 0); diff --git a/server/Cargo.toml b/server/Cargo.toml index fa0b7e4..0c0ca4a 100644 --- a/server/Cargo.toml +++ b/server/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "codescope" -version = "0.2.5" +version = "0.2.6" edition = "2024" [[bin]] diff --git a/skills/analyze.sh b/skills/analyze.sh index 92ec656..7e429d4 100755 --- a/skills/analyze.sh +++ b/skills/analyze.sh @@ -1,6 +1,10 @@ #!/bin/bash # analyze.sh — full analysis pipeline # Usage: ./skills/analyze.sh [language_filter] +# NOTE: MCP tool `index_project` was removed from TOOL_HANDLERS; indexing is +# now done via `codescope worker` (serial) or `codescope index-parallel` +# (parallel). `get_hotspots` has no MCP tool either, so the hotspot step is +# skipped (get_knowledge_graph is the knowledge-layer alternative). set -e PROJECT=$1 LANG=${2:-""} @@ -8,12 +12,17 @@ if [ -z "$PROJECT" ]; then echo "Usage: $0 [language_filter]" exit 1 fi +DB="${CODESCOPE_DB_PATH:-/tmp/codescope_index.db}" +# Export so the worker (positional arg) and every `codescope cli` call +# (reads CODESCOPE_DB_PATH, default .codescope/codescope.db) hit the SAME +# database; otherwise cli would query an empty/stale DB after indexing. +export CODESCOPE_DB_PATH="$DB" echo "╔═══════════════════════════════════════════╗" echo "║ CodeScope Full Analysis Pipeline ║" echo "╚═══════════════════════════════════════════╝" echo "" echo "=== [1/5] Indexing project ===" -codescope cli index_project "{\"project_path\":\"$PROJECT\",\"language_filter\":\"$LANG\"}" +codescope worker "$DB" "$PROJECT" "$LANG" "analyze-sh" 0 echo "" echo "=== [2/5] Project overview ===" codescope cli project_overview '{}' @@ -22,8 +31,8 @@ echo "=== [3/5] Entry points & module tree ===" codescope cli get_entry_points '{}' codescope cli get_module_tree '{}' echo "" -echo "=== [4/5] Hotspot functions ===" -codescope cli get_hotspots '{"top_n":10}' +echo "=== [4/5] Knowledge graph (replaces removed get_hotspots) ===" +codescope cli get_knowledge_graph '{"limit":10}' echo "" echo "=== [5/5] Graph stats ===" codescope cli get_graph_stats '{}' diff --git a/skills/hotspots.sh b/skills/hotspots.sh index 7ef422a..909a33f 100755 --- a/skills/hotspots.sh +++ b/skills/hotspots.sh @@ -1,7 +1,9 @@ #!/bin/bash # hotspots.sh — query project hotspot functions # Usage: ./skills/hotspots.sh [top_n] +# NOTE: the MCP tool `get_hotspots` was removed from TOOL_HANDLERS. Use +# `get_knowledge_graph` (knowledge layer) or `find_callers` for density. set -e TOP_N=${1:-10} -echo "=== Hotspot Top $TOP_N ===" -codescope cli get_hotspots "{\"top_n\":$TOP_N}" +echo "=== Hotspot Top $TOP_N (via get_knowledge_graph) ===" +codescope cli get_knowledge_graph "{\"limit\":$TOP_N}" diff --git a/skills/index.sh b/skills/index.sh index 6c0bcfa..e3fe796 100755 --- a/skills/index.sh +++ b/skills/index.sh @@ -1,6 +1,9 @@ #!/bin/bash # index.sh — index a project # Usage: ./skills/index.sh [language_filter] +# NOTE: the MCP tool `index_project` was removed from TOOL_HANDLERS; the CLI +# entry point is now `codescope worker ` (serial) +# or `codescope index-parallel ` (parallel). This script uses worker. set -e PROJECT=$1 LANG=${2:-""} @@ -8,7 +11,12 @@ if [ -z "$PROJECT" ]; then echo "Usage: $0 [language_filter]" exit 1 fi +DB="${CODESCOPE_DB_PATH:-/tmp/codescope_index.db}" +# Export so the worker (positional arg) and the final `codescope cli` +# call (reads CODESCOPE_DB_PATH, default .codescope/codescope.db) hit the +# SAME database; otherwise cli would report stats from an empty DB. +export CODESCOPE_DB_PATH="$DB" echo "=== Indexing $PROJECT ===" -codescope cli index_project "{\"project_path\":\"$PROJECT\",\"language_filter\":\"$LANG\"}" +codescope worker "$DB" "$PROJECT" "$LANG" "index-sh" 0 echo "=== Done ===" codescope cli get_graph_stats '{}' diff --git a/skills/skills.md b/skills/skills.md index 21e05d3..0f3896d 100644 --- a/skills/skills.md +++ b/skills/skills.md @@ -57,8 +57,8 @@ CodeScope parses source code into a unified AST IR, builds a call graph + refere | Call path A→B | `codescope_trace` | ~50-200 | | Knowledge graph browse | `get_knowledge_graph` | ~100-1000 | | Change impact | `detect_changes` | ~100-500 | -| AI Q&A context | `codescope_build_context` | ~200-1000 | -| "Why can't I find data" | `codescope_capabilities` | ~309 | +| AI Q&A context | `build_project_state` | ~200-1000 | +| "Why can't I find data" | `project_overview` | ~50-100 | --- @@ -67,10 +67,11 @@ CodeScope parses source code into a unified AST IR, builds a call graph + refere | Tool | Status | Workaround | |------|--------|------------| | `get_hotspots` | ❌ no MCP tool | (hotspots.sh is stale) | -| `graph_query` | ❌ not wired | `find_callers`/`find_callees` for call graph; `get_knowledge_graph` for knowledge tables | | `get_communities` | ⚠️ C++ has it, MCP doesn't | `connected_components` as lightweight alternative | | `locate_code` | ❌ not implemented | `explain_symbol` | +> `graph_query` IS implemented (TOOL_HANDLERS in server/src/tools/mod.rs) — earlier docs listed it as "not wired"; that is stale. Use it for custom graph-pattern queries. + --- ## Environment @@ -78,17 +79,22 @@ CodeScope parses source code into a unified AST IR, builds a call graph + refere | Variable | Default | Description | |----------|---------|-------------| | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | -| `CODESCOPE_INDEX_MODE` | `normal` | `fast` / `normal` / `deep` | +| `CODESCOPE_INDEX_MODE` | `normal` | Index mode: `fast` / `normal` / `strict`（NORMAL 默认；见下方 Index modes） | +| `CODESCOPE_WORKERS` | `min(hw,8)` | 解析 worker 线程数（`kDefaultParseWorkers=8`，并行索引路径为模块数×动态分配） | +| `CODESCOPE_SKIP_ASYNC` | (unset) | 设为 1 跳过异步 model/state/FTS 阶段（仅并行调度路径设置） | +| `CODESCOPE_PROFILE_RESOLVER` | (unset) | 启用 resolver 分阶段计时（输出 `[module=resolver, method=run]` 明细） | | `CODESCOPE_VERBOSE` | `1` | Set 0 to disable batch logs | | `CODESCOPE_MAX_FILE_SIZE` | 5MB | Max indexed file size | ## Index modes -| Mode | FTS | Vectors | Speed | Use case | -|------|-----|---------|-------|----------| -| `fast` | ❌ | ❌ | Fastest | Quick answers | -| `normal` | ✅ | ❌ | Normal | Default | -| `deep` | ✅ | ✅ | Slower | Full semantic analysis | +| Mode | 枚举值 | 额外剪枝 | FTS | 用途 | +|------|--------|----------|-----|------| +| `fast` | `FAST` | ✅ 额外跳过 logs/.output/测试报告等 11 类目录 + 4 类缓存文件（`fast_extra_skip_dirs_`/`fast_extra_filenames_`，见 filter_policy.cpp） | ❌ 跳过 | 最快，数据≈全量（对源码干净项目几乎无差异） | +| `normal` | `NORMAL` | 仅基础 skip 表 | ✅ | 默认 | +| `strict` | `STRICT` | 基础 skip + detectLanguage 白名单 gate（仅索引源码文件） | ✅ | 最严格，数据最精简 | + +> **已知问题（2026-08-11，已修复）**：fast 模式此前与 normal 几乎无差异——`fast_extra_skip_dirs_` 为空集（预留未实现），唯一差异是跳过 FTS。已补全剪枝集合并修复 `setMode()` 未重建 active_skip_dirs_ 的 bug。详见 `docs/optimization/perf-full-index-2026-08-11.md` §9/§10。 ## Supported languages diff --git a/skills/stats.sh b/skills/stats.sh index 609c239..5046e69 100755 --- a/skills/stats.sh +++ b/skills/stats.sh @@ -4,8 +4,8 @@ echo "=== Graph stats ===" codescope cli get_graph_stats '{}' echo "" -echo "=== Project info ===" -codescope cli get_project_info '{}' +echo "=== Project overview ===" +codescope cli project_overview '{}' echo "" echo "=== Entry points ===" codescope cli get_entry_points '{}'