Repository navigation
Outline: count in-scope edges and imports in SQL instead of walking every edge in the store #994
Description
Why
explore operation=outline (get_repo_outline) gets slower as the store grows, even when the caller only sees one repository. In the tools sweep (each tool called 15 times against a store holding only the gortex repository, then against a store holding six repositories), outline p50 was 4.3 s on the one-repo store and 12.6-13.7 s for every repository on the six-repo store, small repositories included. The cost follows the size of the whole store, not the size of the repository the session is bound to.
Outline is a slow tool. Two concurrent outline calls hold dispatch slots for about 13 s, and other agents' reads queue behind them. This issue is the narrow, urgent part of the fix: no schema change and no new per-repo counters.
Current state
On main, internal/mcp/tools_outline.go:
handleGetRepoOutline, bound-session path: callss.scopedNodesLight(ctx)(internal/mcp/server.go). That isgraph.AllNodesLight(reader)over the whole store, filtered in Go byQueryOptions.ScopeAllows(internal/query/subgraph.go). The handler builds aninScope map[string]boolfrom the result and counts languages in Go.- Edge count: for a bound session it runs
for _, e := range reader.AllEdges()(around line 110). That reads every edge in the store just to count the ones whose two endpoints are in scope. Unbound sessions already use the O(1)reader.EdgeCount()andreader.Stats(). mostImportedFiles(around line 216): on the SQLite backend it callsFileImportCounts(scope)(internal/graph/store_sqlite/store_aggregators.go) with every in-scope node ID, about 190k for the gortex repository, which runs as chunkede.to_id IN (...)queries. Readers without the aggregator, such as overlay views, fall back to anAllEdges()walk (around line 240) with oneGetNodecall per edge.- Communities and hotspots are filtered after the fact by membership in
inScope.entryPointsfiltersFindNodesByName("main")byinScope. - The
edgestable has norepo_prefixcolumn, so scoping an edge means joining both endpoints tonodes. Thenodes_by_repoindex on(repo_prefix, view_gen)exists.graph.ReadRepoNodesLight/RepoNodesLight(internal/graph/repo_light_nodes.go) already read light nodes for a set of repo prefixes, but they do not include unowned or global-external nodes, so they do not matchScopeAllowson their own.
What to do
- Add a store capability,
graph.ScopedOutlineAggregator(name open), implemented by the SQLite store and by the in-memoryGraphas the reference. It takes a scope: workspace id, project id and repo allow-set, i.e. the same inputsScopeAllowsreads. It returns:- the in-scope node count and node counts by language, from a
GROUP BY languageovernodesrestricted by the scope predicate; - the in-scope edge count, from
COUNT(*)overedgesjoined tonodeson bothfrom_idandto_id(sameview_gen), with both endpoints passing the scope predicate; - per-file import counts, from the existing
FileImportCountsquery with the scope predicate applied to the target node through the join, instead of an ID list.
- the in-scope node count and node counts by language, from a
- Write the scope predicate in SQL once, as a shared helper that mirrors
ScopeAllowsexactly: the node's workspace falls back torepo_prefix, the project falls back torepo_prefix(and only applies when a workspace or repo narrow is also set), a node with an emptyrepo_prefixpasses the repo narrow, and synthetic global externals are always visible. Make the queries usenodes_by_repo; check withEXPLAIN QUERY PLANthat neither table is fully scanned when a repo allow-set is present. - In
handleGetRepoOutline, when the reader implements the capability, use it for the summary and import sections. Do not callscopedNodesLightorAllEdgeson that path. - For communities, hotspots and entry points, decide membership without the whole-store ID set: check scope only for the candidate IDs (hotspot list,
mainfunctions, community members until one matches) through a batched lookup (GetNodesByIDs) plusScopeAllows, never throughAllNodesLightover the whole store. - Keep today's code as the fallback for readers that do not implement the capability (overlay views), so overlay-active sessions still see their own buffers.
- Unbound sessions keep their current path (
Stats(),EdgeCount()).
Out of scope
- Per-repo row counters maintained on write, and caching the outline per repository and graph revision (the later ≤300 ms outline target, planned for v0.64.12).
- Repo-scoping
analyze kind=dead_code, and theAllEdges()sweep of the other tools. - Any schema or store-format change.
- Hotspots returning empty results before the first analysis pass.
Done when
- Outline p50 on the six-repo store is ≤1 s for every tracked repository (n=15 calls per repository, steady state, same host as the v0.64.10 baseline). Today it is 12.6-13.7 s.
- Outline p50 on the one-repo store is within ±15% of v0.64.10 (4.3 s, so ≤4.9 s) on the same host.
- A test with a counting reader (e.g.
TestOutlineBoundSessionNeverScansAllEdges) shows that a bound-session outline on the SQLite backend makes zeroAllEdges()calls and zeroAllNodes/AllNodesLightcalls. - A golden test (e.g.
TestOutlineGolden) shows the outline output decodes to the same JSON as the current implementation for the same data and scope, in each of: bound to one repository, bound to a multi-repo workspace, a project narrow, unbound, and a scope that contains global externals and nodes with an empty repo prefix. - A shared store test checks that the SQLite and in-memory implementations return the same node, language, edge and import counts for the same scopes.
- Overlay-active outline still reflects unsaved buffers (existing overlay tests pass unchanged).
- No schema version bump: a store written by v0.64.10 opens with this build, and a store written by this build opens with v0.64.10.
-
go test -race ./internal/mcp/... ./internal/graph/...passes.
How to verify
go test -race ./internal/mcp/ -run 'Outline'andgo test -race ./internal/graph/... -run 'Outline|FileImport'.- Run
EXPLAIN QUERY PLANon the new queries against a multi-repo store. They must show index use (nodes_by_repo,edges_by_from/edges_by_to) and noSCAN edgesorSCAN nodes. - Benchmark: run the tools sweep (each tool 15 times per repository) against the one-repo and six-repo stores with v0.64.10 and with this build on the same host. Post outline p50/p95 per repository on this issue.
Dependencies
Blocks:
Related issues
Planning
- Milestone: v0.64.11
- Priority: latency (first priority)
- Estimate: ~1 engineer-day
Activity
zzet commentedon Oct 25, 2026
Validation is still in progress; the performance gate has not passed.
A diagnostic single-repository run on the supplied local host used independent store/source copies and back-to-back requests in rotating build order, after enrichment completed. Across 9 fully matched outline requests, median latency was 9.356 s for the combined candidate, 4.895 s for pinned main, and 2.934 s for v0.64.10. Median per-request candidate/main ratio was 1.882. I stopped this diagnostic before the final 15-call acceptance sample because the regression is clear.
The actual bound session supplies a workspace scope without a repository allow-set. Its query plan uses the generation index and residual workspace filtering; the production contract filter also repeats the endpoint joins for metadata corrections. I am profiling these statements and fixing the measured bottleneck before accepting the PR. Existing targeted race tests and golden/parity tests pass, but those do not establish the latency target.
Builds: candidate
348b046028edd1adfd2ca5193b3cf5f4ee40b00b(includes PR #1097 at5dc6a543), maincec9df693abdd59ac99d2906cc252f2b7eca4d5b, baselinedefa9f8e7a670792b5b693744d88488f741a8160. Benchmark materials remain local and uncommitted.zzet commentedon Oct 25, 2026
The revised implementation and combined candidate pass targeted race tests (scope/output parity, metadata encodings, generation isolation, capability fences), pinned lint, and native builds. The full refreshed PR CI run is still in progress.
Completed a 15-request paired single-repository outline run on the supplied Apple M5 Pro host:
7890aeb6defa9f8e8a1b0aa8(outline head05e87b67)All 15 logical requests succeeded on every build: zero missing samples, busy replies, request errors, or cleanup failures. Each build had a separate source/store clone, all daemons stayed alive simultaneously, and identical requests were sent back to back in rotating order. Analysis publication was verified through the supported API on each build before measurement.
This is within the same-host tag comparison's ±15% band. The apparent improvement is near the host's reported 10–15% noise range, so no strong speedup claim is made.
Comparability limits: this is a consistent fully analyzed workload. The historical tool sweep called outline before triggering analysis for its first repository, and supplied a path selector that the current common schemas reject. These measurements use the supported pathless, session-bound aggregate operation on all builds; they are not an exact replay of the historical wire requests. Benchmark material and raw paths remain local and uncommitted.
The six-repository run (15 calls per repository) is now underway. The ≤1 s target has not been accepted; this issue remains In Progress.
zzet commentedon Oct 25, 2026
The latest outline implementation is at
7af70a19; its focused race tests and pinned lint pass. Combined candidateb22afdaaalso passes the expanded targeted races and native build. The build records both safe C/C++ flags and a clean VCS revision.A short same-host diagnostic on 2026-10-25 used independent stores, simultaneous daemons, fully published analysis, and rotating back-to-back requests. Baselines remain main
7890aeb6and tagv0.64.10(defa9f8e). This is one measured call per completed case, not an n=15/p50 acceptance run.The main failure is an in-band server deadline, not the client's 300-second wait. It is a censored failure, so there is no successful three-build completion-time ratio for that request.
The six-repository result still misses the 1-second target substantially. I stopped the diagnostic to profile the remaining cost, including actual hotspot retention and scoped-query stages. A partial second-repository observation is retained privately but is not an acceptance result. All owned daemons stopped and cleanup reported no failures. No benchmark material is committed, and the issue remains In Progress.
zzet commentedon Oct 25, 2026
Acceptance decision (2026-10-25): the current improvement is accepted, and the unmet six-repository <=1-second outline target is deferred to #1002. The tracked carry-forward retains #1002's stronger 300ms uncached / 50ms repeat-call goals for the next milestone. This is a performance deferral, not a claim that the target passed. No outline cache/counter work from #1002 is included in PR #1097.
PR #1097 at
7af70a19preserves exact scope/output semantics, uses scoped SQL counts and generation community queries, and retains only a bounded complete global hotspot ranking. The final combinedb22afdaadiagnostics recorded:Each entry represents one measured request, not n=15 or p50 acceptance. A separate isolated-daemon profile confirmed the optimized paths and retained hotspot cache (307 rows, 68.7 KB); SQL remained the main cost at roughly 3.90-5.96s. Completed query-only prototypes preserved exact counts but showed no reliable warm total improvement, so they remain scratch-only and are not part of the PR.
Local focused race tests and pinned lint pass at
7af70a19; the combined candidate also passes expanded targeted races and its native build. Required CI and cross-version store compatibility remain merge gates and are not waived. This decision starts no #1002 implementation or milestone/board transition.zzet commentedon Oct 25, 2026
Cross-version store checks completed using the actual Store API on fresh, exclusive scratch clones. Each writer persisted and read a marker before closing; the reader then opened the same clone and checked the marker, a sampled source node, node count, schema version and file identity.
b22afdaa/ v0.64.107890aeb6b22afdaaThe candidate and main already use schema 39; the older release uses schema 21. Backward payload preservation did not pass, and the main control demonstrates the same pre-existing behavior. #994 adds no schema version or store-format change. The successful older-reader opens used the normal cache-rebuild policy and must not be described as format-preserving compatibility.
All original store/WAL/SHM identity manifests remained unchanged. This is a sampled Store API smoke check, not exhaustive graph parity or a daemon-readiness test. The targeted outline/overlay parity race tests provide separate behavioral coverage; full PR CI remains the merge gate.
zzet commentedon Oct 25, 2026
The latest full CI run is blocked by an unchanged Perl grammar dependency compiling on the Ubuntu 26.04 runner. Both lint and the Linux rest partition fail at
bsearch.h:72with a_Genericmacro expansion error. The failure is tracked separately in #1114; the compiler flags and test coverage will be preserved.The outline PR remains In Progress until that fix is integrated and the affected checks pass. The approved outline performance deferral to #1002 remains unchanged.
zzet commentedon Oct 25, 2026
Accepted and completed in #1097, merged as 86b4d29 after full cross-platform CI passed at 2e69f0d. Security and skill-drift checks also passed. The final combined local build, full SQLite race suite and seven-package targeted race pass remain recorded in the PR.
This closure follows the explicitly approved decision to accept the structural improvement and defer the unmet one-second target to #1002. The recorded 3.730s single-repository and 30.769s six-repository diagnostics are n=1 observations, not n=15/p50 acceptance results; no unmet latency gate is represented as passed. The pre-existing backward cache-reset behavior is documented in the compatibility evidence, and this PR makes no schema or store-format change.
#1002 retains the outline performance follow-up and its stronger future goals. This issue is closed and its project-board item is Done.
perf(outline): aggregate scoped counts without whole-store scans (zze…