[Core] Batch canonical replay SQLite probes

[tommy0103]

Summary

• Batch existing canonical-row reads so large snapshot replay no longer executes one SQLite UPSERT/probe per unchanged message, tool call, or tool result.
• Preserve the existing conditional UPSERT, FTS, stable-rowid, duration, retraction, and transaction contracts through the shared app/Core persistence seam.
• Keep record streams below 250 entries on the direct write path so small delta providers do not pay an extra database read.

Scope

• In scope:
  • Bounded replay batches and binding-independent existing-row prefetch
  • Ordered in-memory comparison of all authoritative message/tool fields
  • Duration comparison against the same evolving message state
  • Cross-binding statement-count, FTS, NULL-transition, rowid, and ordering coverage
• Out of scope:
  • Provider parsing/checkpoint strategy, including the independent Codex append work in PR perf(core): resume verified Codex appends #148
  • Project-path history scans tracked by [Core] Project-path refresh remains O(history) for one affected session #105
  • Write policies for sessions, summaries, subagents, workflows, workflow agents, or index_state

Key Changes

• packages/core/src/persist.ts activates bounded replay filtering once a stream reaches 250 records, prefetches canonical rows in key-safe batches, and sends only new or changed exact-value records through the existing UPSERTs.
• The replay cache advances in record order and treats delete-session as a hard boundary, preserving duplicate-key, duration, retraction, and delete/reinsert semantics.
• tests/persist-replay-probes.test.mjs makes SQLite statement executions observable on both real bindings and proves a 4,004-record replay needs only 51 bulk reads plus 8 genuine writes.
• Direct and replay-filter paths independently run the complete authoritative-field matrix. Repeated-key and delete/reinsert tests explicitly cross the 250-record threshold and assert that prefetch ran.
• Shared persistence fixtures now live in tests/persist-test-fixtures.mjs, avoiding duplicate binding and canonical-record setup.
• The persistence ADR and three-platform CI contract pin the batching boundary and regression suite.

Validation

• npm test: pass, 667/667 tests under normal filesystem-event permissions
• npm run typecheck: pass for root and app tsconfigs
• node --experimental-strip-types --experimental-sqlite --experimental-test-module-mocks --test tests/persist-replay-probes.test.mjs tests/persist-idempotency.test.mjs tests/persist.test.mjs tests/indexer-upsert-drift.test.mjs: pass, 20/20 tests
• npx eslint packages/core/src/persist.ts tests/persist-test-fixtures.mjs tests/persist-idempotency.test.mjs tests/persist-replay-probes.test.mjs: pass, 0 errors
• git diff --check: pass
• Repository-wide npm run lint: locally blocked by 94 pre-existing errors in user-owned untracked generated artifacts under docs/book/site/dist, resume_rebuild, tmp, and output; no errors are reported for this PR's files

Statement-count regression benchmark

Both node:sqlite and better-sqlite3 run the same controlled replay:

Classification	Records
Emitted	4,004
Existing	4,000
New	4
Modified	4
Unchanged	3,996

Canonical SQLite executions fall from 4,004 per-record UPSERT/probes to 59 total: 51 bounded reads and 8 new/modified writes. The same suite verifies that a one-record delta performs zero prefetch reads and one direct write.

The regression benchmark records persistence and finalize in separate timer regions rather than treating their sum as persistence. One representative local run reported:

Binding	Persistence	Finalize
node:sqlite	10.1 ms	0.95 ms
better-sqlite3	5.1 ms	0.79 ms

These synthetic timings are diagnostics, not pass/fail thresholds; statement counts are the regression assertion.

Controlled real-corpus phase measurements

The benchmark replayed the same 24,197,667-byte Codex session against the same 1,098,883,072-byte local index inside rollback-only transactions.

Phase	Elapsed
Provider parse	71.6 ms
Persistence (origin/main)	63.5 ms
Persistence (this PR)	14.3 ms
Finalize, independently measured	862.8 ms

The independent finalize measurement separates its own work further: project-path refresh 862.1 ms, unresolved backfill 0.46 ms, workflow healing 0.20 ms, and FTS readiness 0.04 ms. The dominant finalize cost is the separate #105 project-path scan and is not attributed to persistence.

Kind	Emitted	Existing	New	Modified	Unchanged
Messages	1,999	1,999	0	0	1,999
Tool calls	1,094	1,094	0	0	1,094
Tool results	1,094	1,094	0	0	1,094
Durations	59	59	0	1	58

Persistence	origin/main	This PR
Canonical statement executions	4,246	53
Elapsed	63.5 ms	14.3 ms

The residual 53 executions are 51 batch reads plus 2 ordered duration updates present in the replay. Statement executions fell 98.8%; persistence latency fell 77.6%.

Controlled complete incremental-build A/B

A copy-on-write clone of the same 1.10 GB index received the same appended real transcript record for each trial. Each arm ran complete buildIndex work, including discovery, parse, persistence, project-path refresh, and finalize; order alternated across three restored trials.

Complete build	Trial 1	Trial 2	Trial 3	Median
origin/main	292.1 ms	1,283.5 ms	1,437.9 ms	1,283.5 ms
This PR	283.4 ms	251.9 ms	915.8 ms	283.4 ms

Cold-cache variance is visible and #105 finalize work remains in both arms, but the new implementation was faster in every paired trial; median complete-build latency fell 77.9%.

Risks and Rollback

• The comparison field lists mirror the existing conditional UPSERT field sets; regression tests independently change every authoritative field on both the direct and replay-filter paths and cover NULL transitions.
• Batches retain only 250 records and split key lookups below SQLite's portable binding limit. Transcript values stay bound parameters.
• Rollback is a single implementation commit revert. There is no schema migration, provider cursor change, or canonical marker bump.

Related Issues

• Related to [Codex] Append replay rewrites unchanged canonical rows #130
• Related to [Core] Project-path refresh remains O(history) for one affected session #105
• Related to [Core] Eliminate corpus-wide work from ordinary CLI incremental finalize #125

Closes #132