Skip to content

[G1] Live capture: fs-event watch with debounce + polling fallback, stat-only change fingerprint, and a Claude hook fast path (--hook claude) #174

Closed

Description

@willwashburn

Part of #160 (group 1 — independent). docs/substrate-improvements-proposal.md P0.1 already calls live capture "the single most urgent item" because Claude JSONL transcripts get cleaned up; once burn depends on relayhistory, burn's cost history disappears with them too.

Current state

  • ai-hist watch / import --watch exist (crates/ai-hist-cli/src/lib.rs:238, 559-563 → watch_loop_with_connectors(db, interval, scope, connectors)): a fixed-interval polling loop running a full sync each tick. No filesystem events, no debounce, no stat-only short-circuit, so each tick pays the full transcript-walk cost (stamp comparison per file across ~/.claude/projects/**, ~/.codex/sessions/**, ~/.codex/archived_sessions/**, ~/.cursor/projects/**, ~/.grok/sessions/**, plus the OpenCode DB).
  • No hook ingestion: grep -rln "transcript_path\|PostToolUse\|SessionStart" crates sdk-ts/src docs examples matches only the proposal doc.

burn has the mature version of all three in crates/relayburn-sdk/src/ingest/ and crates/relayburn-cli/src/commands/ingest.rs, which will be deleted at cutover — port them here:

  • fs events (ingest/fs_events.rs:108-129, watch_loop.rs): notify::recommended_watcher recursive over the harness roots, Create|Modify|Remove only, bursts collapsed into a single-permit tokio::sync::Notify, 200 ms debounce, 30 s slow-poll backstop, polling fallback at 1000 ms when notify cannot attach / --no-fsevents / network FS. fs-event ticks pass force=true because notify can fire before the write flushes (ingest.rs:100-109); a periodic tick that finds a run in flight skips, a manual tick() joins it (watch_loop.rs:200-251).
  • stat-only fingerprint (ingest.rs:218-268): "count:totalBytes:hash" fold over every candidate file's (path, len, mtime) — no file opened; equal to the stored value → return early. Written after cursors and only when no file was skipped for a transient error, so a failure can never become sticky (ingest.rs:386-404).
  • hook fast path (commands/ingest.rs:290-340, ingest_claude_transcript_path ingest.rs:502-582): reads one Claude Code hook JSON payload from stdin (session_id, transcript_path, hook_event_name, cwd), hydrates exactly that transcript, resolves stamps, re-stats after parsing so the stored cursor is at EOF (pre-parse size would cause duplicate rows on the next sweep), and never fails the parent tool call: missing/rotated transcript → empty report, exit 0. Without transcript_path it falls back to a full sweep.

Scope

  1. ai-hist-engine: watch::WatchLoop { debounce_ms, poll_interval_ms, slow_poll_ms, use_fs_events, roots: from providers' watch_roots() } with run(), tick(), stop(); add fn watch_roots(&self) -> Vec<PathBuf> to ShallowSessionProvider (OpenCode returns the DB file's directory; relay returns none). Dependency: notify = { version = "8", default-features = false, features = ["macos_fsevent"] } as burn does.
  2. source_fingerprint() over the same candidate enumeration discovery uses (Candidate list, discover.rs:181), stored in .sync-state.json; sync short-circuits when equal unless force.
  3. ai-hist ingest --hook claude [--quiet] (name TBD; keep import alias): stdin payload → hydrate_session_at by transcript path (needs a SessionLocator::Path entry point that bypasses catalog lookup but still runs validate_provider_path, hydrate.rs:1159), then a bounded related-sidecar hydrate. Exit 0 always; diagnostics to stderr only when not --quiet. Document the settings.json hook wiring (SessionStart, PostToolUse, Stop, PreCompact) in docs/agent-integration.md with the exact JSON, and add a PreCompact-specific note: hydrate before the compaction rewrites the transcript.
  4. Codex/OpenCode hook equivalents: characterize whether either exposes lifecycle hooks; if so add --hook codex; if not, document and rely on watch.
  5. CLI ai-hist watch gains --no-fsevents, --debounce-ms, --interval; TS SDK watch() async iterator over tick reports (rides the group-2 exposure issue if the napi shape is contentious; the CLI must land here).
  6. Tests: port burn's watch_loop_tests.rs and orchestration_tests.rs cases (debounce collapse, in-flight join, force on fs tick, fingerprint-equal skip, transient error not sticky, hook with missing transcript exits 0).

Acceptance

  • Appending one line to a watched Claude transcript triggers exactly one hydration within debounce + 100 ms (test with a tempdir HOME).
  • With nothing changed, a watch tick opens zero files (assert via DiscoveryCounters.files_opened == 0 and the fingerprint short-circuit path).
  • echo '{"session_id":"x","transcript_path":"/nonexistent"}' | ai-hist ingest --hook claude --quiet; echo $? prints 0.
  • A hook-ingested transcript followed by a full sync produces no duplicate events (unique-key upsert + cursor at EOF).

Out of scope

The change-feed API consumers use to learn what the tick ingested (group 2); remote connectors in watch mode.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions