Sync read amplification: remaining work after PR #299 (cold-sweep 7× transcript reads, Linux re-baseline, CI thresholds)

[willwashburn]

Status against main @ 5f9aeca (2026-10-01): the evaluation this issue asked for is done and documented in docs/benchmarks.md ("2026-09-28 sweep profile and the tokscale techniques" and "2026-09-29 read amplification"). PR #299 (merged 2026-09-30, "Part of #215", closes #42) fixed the incremental-sweep amplification. This issue is narrowed to what that PR listed as remaining.

Decisions recorded (docs/benchmarks.md, "Decisions")

tokscale technique	Decision	Why
rayon parallel parse	Reject for now	parsing is <1% of a cold sweep, <0.5% of an incremental one; writes are ~90% of cold cost and serialized behind one connection
walkdir with trusted file_type()	Already adopted	collect_matching_files_inner trusts DirEntry::file_type(); the walk is ~2%
Typed / SIMD JSON	Reject for now	serde_json::Value construction is <1% of every phase measured
Sampled-content fingerprints	Adopted for transcripts, deferred for flat logs	transcript cursors hash a head-and-tail window; the three flat logs keep the whole-prefix SHA-256 at <1%

What PR #299 fixed

Incremental sweep after a 1 KiB append on the 100 MB store: 838 MiB read (7.7× the store) → 114 MiB (1.05×); 1.77 s → 0.39 s on later sweeps. Causes removed: two session_events scans per sweep in the identity refresh (now driven from sessions via idx_session_events_project), an O(files × sessions) discovery locator lookup (ORDER BY +session_id), two window digests plus two whole-prefix hashes per unchanged transcript (now a ctime-settled stamp on filesystems with a real ctime, 6 h expiry, digest kept on FAT/exFAT/unknown/Windows), redundant .sync-state.json merges and marker recounts.

Remaining

1. Cold-sweep transcript amplification. A 53 KB Claude transcript is read seven times on first ingest (metadata fold, record walk, window digest, re-hash at commit, discovery head read): Claude 6.8×, Codex 4.4×, Cursor 4.3×, Grok 1.7× of on-disk bytes. Documented, not fixed. Target: one read per file per cold pass, with the commit-time digest computed from the bytes already in hand.
2. Cold sweep is write-bound (sqlite3_step 89%, fsync 35%, per-row autocommit at synchronous = FULL). Two measured prototypes, neither shipped: PRAGMA synchronous = NORMAL for the sweep (−21%) and one transaction per Claude transcript (−6%). The first is a durability decision (last commits can vanish on power loss while .sync-state.json stamps survive; the destination marker is meant to detect that) and wants its own review.
3. Linux numbers. All of the above was measured on macOS with an uncommitted DYLD_INSERT_LIBRARIES byte-count shim. Re-run the full profile on Linux so the harness's /proc/self/io Bytes read column is populated, and replace the stale 2a1e81a rows in the published table (which still show 47.1 GiB for a cold sync and 1.7 GiB for a no-op tick).
4. CI gate thresholds (scripts/benchmark-thresholds.json) are unchanged since [G1] Full-sync and hydration throughput baseline, memory ceiling, and a CI regression gate #176; re-baseline on ubuntu-latest after (3) so the improvement cannot regress silently.
5. Optional: commit the byte-count shim (or a Linux-only /proc path) so per-reader attribution is reproducible.

Acceptance

• docs/benchmarks.md carries a Linux-measured table for current main with Bytes read populated, and the cold-sweep per-reader table shows Claude transcripts at ≤ 2× on-disk bytes.
• scripts/benchmark-thresholds.json baselines are re-measured and justified in the PR that moves them.
• A decision (adopt / reject, with the durability argument) is recorded for synchronous = NORMAL on the sweep.

Related

• perf(ai-hist): Reflex sync repeatedly re-scans and backfills large local histories #42 (closed via PR perf(sync): stop re-reading and re-querying unchanged files on every sweep (#42, #215) #299), [G1] Full-sync and hydration throughput baseline, memory ceiling, and a CI regression gate #176 (closed), [G1] Incremental transcript hydration: byte-offset cursors, prefix validation, in-progress message handling, bounded memory #173 (closed), [G1] Live capture: fs-event watch with debounce + polling fallback, stat-only change fingerprint, and a Claude hook fast path (--hook claude) #174 (watch tick cost), PR perf(sync): stop re-reading and re-querying unchanged files on every sweep (#42, #215) #299.