Skip to content

graph refresh re-stages the entire corpus when one file changes — a one-line edit costs a full rebuild #209

Description

@theyashasvipandey

graph refresh is engine.sync() — there is one incremental path under two
names. It stages the whole corpus whenever any work is needed, regardless of
how little changed.

Measured on honojs/hono @ 4.12.32 (381 indexed files, 13,240 nodes), in two
independent runs on different machines:

run A run B files indexed
clean build 71 s 65 s 381
graph refresh, nothing changed 14 s 14 s 0
graph refresh, one file touched 69 s 71 s 381
graph refresh, one more file touched 68 s 70 s 381

Appending one comment line to one file re-indexes all 381. The file count is
identical in both runs and comes from the command's own output, so it is the
reliable figure here; the wall-clock ratio against a clean build lands anywhere
from ~97% to ~109% depending on which machine's baseline you divide by, and is
not worth treating as a constant.

The no-op short-circuit does work, and its 14 s was identical in both runs —
that path does fixed work, walking and hashing the corpus to prove nothing
changed.

Cause

In src/graph/engine-impl.ts, sync(changedFiles) computes changedSources
(line 389), uses it for the short-circuit at 395–405, then calls
stageCorpus(this.rootDir, …) at 409 — the root, not the changed files.
publish() rewrites the derived graph from that full staging.

The code states the reason plainly, at line 407:

Re-stage the whole semantic corpus. This makes an arbitrary sync sequence
converge to the same graph as a clean build and re-resolves cross-file refs.

Two details that shape any fix

changedFiles is already empty in production. Both call sites pass []:
refreshGraphWithLease (src/graph/maintenance.ts:628) and the isolated
candidate path (src/graph/candidate-entry.ts:82). Every non-empty
sync([...]) in the tree is a test. So changedSources.length === 0 always
holds, and the decision is made entirely by the snapshot/corpus comparison at
396–404 — the parameter contributes nothing to the live path.

The comparison already knows exactly which files changed, then discards it.
sourceCorpusMatchesFileRecords (line 1520) builds a path→contentHash map of
the indexed corpus and compares it against the current one per file — then
returns a single boolean. The per-file delta a targeted re-stage would need is
computed on every refresh and thrown away. That is the cheapest place to start,
whatever is decided about the rest.

This is not a tuning fix

  • Cross-file references. Editing one file can resolve or break references in
    others. Knowing which requires a reverse-dependency map that does not exist.
  • The TypeScript program. Semantic extraction forms a compiler program over
    the project, not over a file.
  • Node identity. Ids and fingerprint/LSH continuity are computed against the
    whole fresh set. Partial updates risk the identity drift the reconciler exists
    to prevent, and a wrong MOVED is worse than a slow build.

Direction to discuss

A reverse-dependency index — which files reference symbols defined in the
changed file — so a one-file edit re-stages that file plus its dependents, and
publication touches only their rows. Convergence with a clean build has to stay
provable, which is the hard part and the reason this is discuss-first.

Framing

docs/design/graph-freshness-recovery.md:13 already says "'Refresh' is not a
claim of incremental speed", so the behaviour is documented and intentional.
This issue is not reporting a surprise — it is asking whether the correctness
argument still requires full re-staging, and recording what the current
guarantee costs on a mid-size repository.

Related to #140, where this cost lands on a ~1,600-file repository, and to the
grounding-pass rebuild described in that thread. Not a duplicate: #140 tracks
rebuild variance and memory.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    code-graphWork related to the AST and Tree-sitter code graphcore / discuss-firstCore behavior that requires maintainer discussion before implementationneeds-researchRequires investigation before implementationperformancePerformance, memory usage, or scalability improvements

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions