Repository navigation
[G1] Incremental transcript hydration: byte-offset cursors, prefix validation, in-progress message handling, bounded memory #173
Copy link
Copy link
Closed
Description
Part of #160 (group 1 — independent). Required for relayhistory to be a viable live source for burn's watch loop (1 s polling / fs-event debounced ticks on files that grow by a few KB at a time).
Problem
Targeted hydration and transcript sync re-parse whole files on any change:
ingest_claude_transcript_as(crates/ai-hist/src/lib.rs:3004) doesfs::read_to_string(path)— the entire transcript into memory — then re-upserts every row.- The unchanged short-circuit (
hydrate.rs:187-201) is stamp-based ((len, mtime_ns, inode)viastamp_of,lib.rs:3665); any byte change re-parses everything. - Global sync transcript walks are also stamp-based (
claude_sessions_v3lib.rs:2644-2700,codex_rollouts_v5:1549-1721). - Byte-offset incremental reading exists only for the two flat prompt logs
~/.claude/history.jsonl/~/.codex/history.jsonl:sync_jsonl_incremental(lib.rs:1298),CompleteJsonlReader/FileCursor(:980-1268) storing{offset, mtime_ns, dev/ino, prefix sha256}with prefix re-validation andJSONL_CHUNK_LINES = 2000commits. That machinery is the right shape; it is not applied to transcripts. - A Claude assistant message is written as several JSONL rows over time (one per content block; only one carries
usage), and a message with nostop_reasonis still streaming. Re-parsing whole files hides this; incremental parsing must handle it explicitly. burn's rule (reader/claude/incremental.rs:646-664): the committed offset backs up to the byte offset of the earliest in-progress assistant messageId, so the next pass re-reads it; partial trailing lines are dropped. Codex: burn keeps aCommittedSnapshotthat advances only attask_complete(codex/incremental.rs:790-843), discarding a half-written turn. - Multi-hundred-MB transcripts exist on fleet machines (
docs/sqlite-contention-incident-2026-08-06.md,#42"repeatedly re-scans and backfills large local histories",#533 GB DB).
Scope
- Transcript cursors in
session_hydration_checkpoints/observation_hydration_checkpoints: addcommitted_offset INTEGER,prefix_hash TEXT(sha256 of bytes[0, committed_offset)or of the first N KiB + the last line before the offset — pick and document),dev_ino TEXT,parser_state_json TEXT(opaque per-source state: lastevent_index, lastcall_indexmap, Codex cumulative token baseline + open turn, Claude in-progress message ids). Rotation rule identical to burn's:inode changed || mtime < cursor.mtime || size < committed_offset || prefix_hash mismatch→ full re-parse from 0. - Streaming readers: replace
read_to_stringwith aBufReadloop fromcommitted_offset; drop a trailing partial line; commit everyJSONL_CHUNK_LINESin oneImmediatetransaction as the flat-log path does. - In-progress semantics (Claude): rows of an assistant message with no
stop_reason(or whoseusagecarrier has not arrived) are held inparser_state_jsonand not written assession_eventsuntil complete;committed_offsetis set to the start of the earliest held message. Expose the held count inHydrateSessionResult.diagnostics(HYDRATION_IN_PROGRESS_MESSAGES). Codex: hold everything after the lasttask_complete. - Sidecars: subagent transcripts and
agent-*.meta.json(hydrate.rs:1016source_snapshotstamps them) get their own cursor rows keyed by locator, so a growing sidecar does not re-parse the parent. - Global sync uses the same cursors for transcripts (retire the
path -> stampmaps behind a one-time migration). - Memory ceiling: a test hydrates a synthetic 200 MB Claude transcript (generated in the test, not committed) and asserts peak RSS stays under a documented bound (e.g. < 64 MiB above baseline) and that appending 1 KiB then re-hydrating reads ≈1 KiB (
bytes_readcounter on the result). HYDRATION_PARSER_VERSIONbump;docs/session-catalog.md"Rescans & source stamps" section rewritten;docs/benchmarks.mdgains the incremental numbers (coordinate with the throughput-baseline issue).
Acceptance
- Corpus
incomplete-then-complete: hydrating the truncated prefix yields no assistant event andHYDRATION_IN_PROGRESS_MESSAGES = 1; hydrating the full file yields the message exactly once with usage. interleaved-turns/parent-chain-out-of-orderproduce identical snapshots whether hydrated in one pass or in 3 appends.- A file truncated below
committed_offsettriggers a full re-parse and aHYDRATION_SOURCE_ROTATEDdiagnostic. - The 200 MB memory test passes in CI (or is
#[ignore]with a documented local command, if CI runners cannot host it — say which).
Out of scope
fs-event watching (live-capture issue); the change-feed consumer API (group 2).
Reactions are currently unavailable
Activity