Skip to content

[G1] Incremental transcript hydration: byte-offset cursors, prefix validation, in-progress message handling, bounded memory #173

Closed

Description

@willwashburn

Part of #160 (group 1 — independent). Required for relayhistory to be a viable live source for burn's watch loop (1 s polling / fs-event debounced ticks on files that grow by a few KB at a time).

Problem

Targeted hydration and transcript sync re-parse whole files on any change:

  • ingest_claude_transcript_as (crates/ai-hist/src/lib.rs:3004) does fs::read_to_string(path) — the entire transcript into memory — then re-upserts every row.
  • The unchanged short-circuit (hydrate.rs:187-201) is stamp-based ((len, mtime_ns, inode) via stamp_of, lib.rs:3665); any byte change re-parses everything.
  • Global sync transcript walks are also stamp-based (claude_sessions_v3 lib.rs:2644-2700, codex_rollouts_v5 :1549-1721).
  • Byte-offset incremental reading exists only for the two flat prompt logs ~/.claude/history.jsonl / ~/.codex/history.jsonl: sync_jsonl_incremental (lib.rs:1298), CompleteJsonlReader/FileCursor (:980-1268) storing {offset, mtime_ns, dev/ino, prefix sha256} with prefix re-validation and JSONL_CHUNK_LINES = 2000 commits. That machinery is the right shape; it is not applied to transcripts.
  • A Claude assistant message is written as several JSONL rows over time (one per content block; only one carries usage), and a message with no stop_reason is still streaming. Re-parsing whole files hides this; incremental parsing must handle it explicitly. burn's rule (reader/claude/incremental.rs:646-664): the committed offset backs up to the byte offset of the earliest in-progress assistant messageId, so the next pass re-reads it; partial trailing lines are dropped. Codex: burn keeps a CommittedSnapshot that advances only at task_complete (codex/incremental.rs:790-843), discarding a half-written turn.
  • Multi-hundred-MB transcripts exist on fleet machines (docs/sqlite-contention-incident-2026-08-06.md, #42 "repeatedly re-scans and backfills large local histories", #53 3 GB DB).

Scope

  1. Transcript cursors in session_hydration_checkpoints / observation_hydration_checkpoints: add committed_offset INTEGER, prefix_hash TEXT (sha256 of bytes [0, committed_offset) or of the first N KiB + the last line before the offset — pick and document), dev_ino TEXT, parser_state_json TEXT (opaque per-source state: last event_index, last call_index map, Codex cumulative token baseline + open turn, Claude in-progress message ids). Rotation rule identical to burn's: inode changed || mtime < cursor.mtime || size < committed_offset || prefix_hash mismatch → full re-parse from 0.
  2. Streaming readers: replace read_to_string with a BufRead loop from committed_offset; drop a trailing partial line; commit every JSONL_CHUNK_LINES in one Immediate transaction as the flat-log path does.
  3. In-progress semantics (Claude): rows of an assistant message with no stop_reason (or whose usage carrier has not arrived) are held in parser_state_json and not written as session_events until complete; committed_offset is set to the start of the earliest held message. Expose the held count in HydrateSessionResult.diagnostics (HYDRATION_IN_PROGRESS_MESSAGES). Codex: hold everything after the last task_complete.
  4. Sidecars: subagent transcripts and agent-*.meta.json (hydrate.rs:1016 source_snapshot stamps them) get their own cursor rows keyed by locator, so a growing sidecar does not re-parse the parent.
  5. Global sync uses the same cursors for transcripts (retire the path -> stamp maps behind a one-time migration).
  6. Memory ceiling: a test hydrates a synthetic 200 MB Claude transcript (generated in the test, not committed) and asserts peak RSS stays under a documented bound (e.g. < 64 MiB above baseline) and that appending 1 KiB then re-hydrating reads ≈1 KiB (bytes_read counter on the result).
  7. HYDRATION_PARSER_VERSION bump; docs/session-catalog.md "Rescans & source stamps" section rewritten; docs/benchmarks.md gains the incremental numbers (coordinate with the throughput-baseline issue).

Acceptance

  • Corpus incomplete-then-complete: hydrating the truncated prefix yields no assistant event and HYDRATION_IN_PROGRESS_MESSAGES = 1; hydrating the full file yields the message exactly once with usage.
  • interleaved-turns / parent-chain-out-of-order produce identical snapshots whether hydrated in one pass or in 3 appends.
  • A file truncated below committed_offset triggers a full re-parse and a HYDRATION_SOURCE_ROTATED diagnostic.
  • The 200 MB memory test passes in CI (or is #[ignore] with a documented local command, if CI runners cannot host it — say which).

Out of scope

fs-event watching (live-capture issue); the change-feed consumer API (group 2).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions