Skip to content

perf(core): stop rebuilding the whole messages_fts index in every incremental finalize - #77

Open
Cyb3rN8 wants to merge 1 commit into
tommy0103:mainfrom
Cyb3rN8:feat/incremental-fts-skip-rebuild
Open

perf(core): stop rebuilding the whole messages_fts index in every incremental finalize#77
Cyb3rN8 wants to merge 1 commit into
tommy0103:mainfrom
Cyb3rN8:feat/incremental-fts-skip-rebuild

Conversation

@Cyb3rN8

@Cyb3rN8 Cyb3rN8 commented Aug 19, 2026

Copy link
Copy Markdown

Fixes #75.

Root cause and reproduction

Every finalize runs messages_fts('rebuild'), regenerating the FTS index from the whole messages table. Cost is proportional to index size, not change size: measured on a 1.8 GB / 688k-message index, ~10.3 s under unicode61 and ~30.2 s under trigram (#14), out of a ~50 s incremental build. File stat-comparison (8,669 jsonl, 0.07 s) is not the bottleneck — the rebuild is.

The rebuild is redundant on the incremental path: messages_fts is external-content maintained by the ai/au/ad triggers, persist upserts messages via ON CONFLICT DO UPDATE (fires AU) and deletes via plain DELETE (fires AD). No current write shape bypasses the triggers; FTS5 integrity-check (rank=1) passes on the live index.

Change

  • Incremental finalize performs the rebuild exactly once per index (heals rows that predate trigger-only maintenance, e.g. written by an INSERT OR REPLACE-era version) and records __messages_fts_synced__ in index_state in the same transaction — an interrupted build redoes the heal; the marker never runs ahead of the work (converges when re-run, per the contributing guide's destructive-work rule).
  • The force path keeps its wholesale rebuild as the last line of defence and re-records the marker it wipes with DELETE FROM index_state.
  • rebuildMemoryFts is deliberately untouched: memories is tiny, and --attune writes it outside build transactions, so the cheap belt-and-suspenders refresh stays.

Verification

  • New tests/incremental-fts-sync-marker.test.mjs:
    • poison-posting probe (an FTS row with no messages counterpart): survives an incremental build (rebuild skipped), cleared by a force build, cleared exactly once for a marker-less index (the heal), with integrity-check after each path;
    • pins the trigger semantics for persist's exact upsert/delete shapes — the writes the wholesale rebuild used to bail out.
  • Full suite on this branch head: 469/469 pass, typecheck clean, lint 0 errors (4 pre-existing warnings in unrelated app tests, present on main).
  • Real-machine effect (daemon-style refresh loop): incremental build 40–56 s → 10.6–11.4 s with real changes, 2.4 s floor with none; search behavior verified unchanged against LIKE ground truth.

🤖 Generated with Claude Code

…remental finalize

The finalize step ran INSERT INTO messages_fts(messages_fts) VALUES('rebuild')
on every build, regenerating the full-text index from scratch no matter how
little changed. On a real-world 1.8 GB index (~688k messages) that rebuild
alone is ~10 s under unicode61 and ~30 s under a trigram tokenizer — the
dominant fixed cost of an otherwise cheap incremental build, paid on every
daemon refresh and every query-path freshness build.

The rebuild is redundant on the incremental path: messages_fts is an
external-content table maintained by the ai/au/ad triggers, and persist writes
messages with INSERT ... ON CONFLICT DO UPDATE (fires AU) and deletes them with
plain DELETE (fires AD), so no write shape bypasses the triggers. The historic
reason for the wholesale rebuild — INSERT OR REPLACE writes, whose implicit
deletes fire no DELETE trigger — no longer exists in the messages write path.

Incremental finalize now performs the rebuild exactly once per index, to heal
databases whose rows predate trigger-only maintenance, and records the
__messages_fts_synced__ marker in index_state within the same transaction (an
interrupted build redoes the heal; the marker never runs ahead of the work).
The force path keeps its wholesale rebuild as the last line of defence and
re-records the marker it wipes with DELETE FROM index_state.

Verified with a poison-posting probe (a messages_fts row with no messages
counterpart): it survives an incremental build (rebuild skipped), is cleared by
a force build, and is cleared by the one-time heal on a marker-less index.
FTS5 integrity-check (rank=1) passes after each path, and an upsert/delete
round-trip pins the trigger semantics the finalize used to bail out.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incremental builds rebuild the whole messages_fts index every finalize — ~30s fixed cost independent of change size

1 participant