Repository navigation
[G1] Live capture: fs-event watch with debounce + polling fallback, stat-only change fingerprint, and a Claude hook fast path (--hook claude) #174
Copy link
Copy link
Closed
Description
Part of #160 (group 1 — independent). docs/substrate-improvements-proposal.md P0.1 already calls live capture "the single most urgent item" because Claude JSONL transcripts get cleaned up; once burn depends on relayhistory, burn's cost history disappears with them too.
Current state
ai-hist watch/import --watchexist (crates/ai-hist-cli/src/lib.rs:238, 559-563→watch_loop_with_connectors(db, interval, scope, connectors)): a fixed-interval polling loop running a fullsynceach tick. No filesystem events, no debounce, no stat-only short-circuit, so each tick pays the full transcript-walk cost (stamp comparison per file across~/.claude/projects/**,~/.codex/sessions/**,~/.codex/archived_sessions/**,~/.cursor/projects/**,~/.grok/sessions/**, plus the OpenCode DB).- No hook ingestion:
grep -rln "transcript_path\|PostToolUse\|SessionStart" crates sdk-ts/src docs examplesmatches only the proposal doc.
burn has the mature version of all three in crates/relayburn-sdk/src/ingest/ and crates/relayburn-cli/src/commands/ingest.rs, which will be deleted at cutover — port them here:
- fs events (
ingest/fs_events.rs:108-129,watch_loop.rs):notify::recommended_watcherrecursive over the harness roots,Create|Modify|Removeonly, bursts collapsed into a single-permittokio::sync::Notify, 200 ms debounce, 30 s slow-poll backstop, polling fallback at 1000 ms when notify cannot attach /--no-fsevents/ network FS. fs-event ticks passforce=truebecause notify can fire before the write flushes (ingest.rs:100-109); a periodic tick that finds a run in flight skips, a manualtick()joins it (watch_loop.rs:200-251). - stat-only fingerprint (
ingest.rs:218-268):"count:totalBytes:hash"fold over every candidate file's(path, len, mtime)— no file opened; equal to the stored value → return early. Written after cursors and only when no file was skipped for a transient error, so a failure can never become sticky (ingest.rs:386-404). - hook fast path (
commands/ingest.rs:290-340,ingest_claude_transcript_pathingest.rs:502-582): reads one Claude Code hook JSON payload from stdin (session_id,transcript_path,hook_event_name,cwd), hydrates exactly that transcript, resolves stamps, re-stats after parsing so the stored cursor is at EOF (pre-parse size would cause duplicate rows on the next sweep), and never fails the parent tool call: missing/rotated transcript → empty report, exit 0. Withouttranscript_pathit falls back to a full sweep.
Scope
ai-hist-engine:watch::WatchLoop { debounce_ms, poll_interval_ms, slow_poll_ms, use_fs_events, roots: from providers' watch_roots() }withrun(),tick(),stop(); addfn watch_roots(&self) -> Vec<PathBuf>toShallowSessionProvider(OpenCode returns the DB file's directory; relay returns none). Dependency:notify = { version = "8", default-features = false, features = ["macos_fsevent"] }as burn does.source_fingerprint()over the same candidate enumeration discovery uses (Candidatelist,discover.rs:181), stored in.sync-state.json;syncshort-circuits when equal unlessforce.ai-hist ingest --hook claude [--quiet](name TBD; keepimportalias): stdin payload →hydrate_session_atby transcript path (needs aSessionLocator::Pathentry point that bypasses catalog lookup but still runsvalidate_provider_path,hydrate.rs:1159), then a bounded related-sidecar hydrate. Exit 0 always; diagnostics to stderr only when not--quiet. Document thesettings.jsonhook wiring (SessionStart,PostToolUse,Stop,PreCompact) indocs/agent-integration.mdwith the exact JSON, and add aPreCompact-specific note: hydrate before the compaction rewrites the transcript.- Codex/OpenCode hook equivalents: characterize whether either exposes lifecycle hooks; if so add
--hook codex; if not, document and rely on watch. - CLI
ai-hist watchgains--no-fsevents,--debounce-ms,--interval; TS SDKwatch()async iterator over tick reports (rides the group-2 exposure issue if the napi shape is contentious; the CLI must land here). - Tests: port burn's
watch_loop_tests.rsandorchestration_tests.rscases (debounce collapse, in-flight join, force on fs tick, fingerprint-equal skip, transient error not sticky, hook with missing transcript exits 0).
Acceptance
- Appending one line to a watched Claude transcript triggers exactly one hydration within
debounce + 100 ms(test with a tempdir HOME). - With nothing changed, a
watchtick opens zero files (assert viaDiscoveryCounters.files_opened == 0and the fingerprint short-circuit path). echo '{"session_id":"x","transcript_path":"/nonexistent"}' | ai-hist ingest --hook claude --quiet; echo $?prints0.- A hook-ingested transcript followed by a full
syncproduces no duplicate events (unique-key upsert + cursor at EOF).
Out of scope
The change-feed API consumers use to learn what the tick ingested (group 2); remote connectors in watch mode.
Reactions are currently unavailable
Activity
fix(watch): put a lost watch back inside the debounce window, and kee…
fix(watch): keep a change whose sweep never ran, and retry it
fix(watch): keep a change whose sweep failed, too