Skip to content

[G1] Model currently-dropped record types: compaction/summary boundaries, system rows, non-text content blocks, Codex lifecycle events #165

Closed

Description

@willwashburn

Part of #160 (group 1 — independent).

Problem

Both parsers silently drop record types that a "complete and thorough" ledger must keep. Locations (all in crates/ai-hist/src/lib.rs):

Claude

  • :3073 — let Some(content) = message.and_then(|m| m.get("content")) else { continue; } drops every record without message.content. That includes type:"summary" rows, type:"system" rows with subtype:"compact_boundary" (burn emits a CompactionEvent from these — reader/claude/incremental.rs:482-510), and system rows carrying parent_tool_use_id + agent_id/subagent_session_id (burn turns these into subagent-notification tool-result events — reader/claude/tool_results.rs:178-240).
  • :3297 — _ => {} drops every content block that is not text|thinking|tool_use|tool_result: image, document, redacted_thinking, server_tool_use, web_search_tool_result, mcp_tool_use/mcp_tool_result, and thinking signatures.
  • _meta.replaces / _meta.collapsedCalls on tool results (tool-replacement metadata; burn reader/claude/tool_results.rs:295-340) are not read.
  • toolUseResult.agentId is read for file edits and relationships but is not stored as a fact on the tool-result event itself.

Codex

  • :2502 and :2504 — _ => {} drops compacted, turn_diff, stream_error, exec_command_begin, patch_apply_begin, mcp_tool_call_begin, entered_review_mode, task_started, and any subagent_* lifecycle event (subagent_*_{complete,done,finished,terminated}; burn reader/codex.rs:534-542).
  • :2495-2497 — response_item/reasoning (encrypted) only flips saw_model_output; the presence of an encrypted reasoning block (and its byte length) is not recorded.

The session_events.kind CHECK constraint (crates/ai-hist-core/src/lib.rs:149) allows only text|thinking|tool_use|tool_result, so none of these can be stored without a schema change.

Design

Add a session_markers table rather than overloading session_events:

CREATE TABLE session_markers (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  source TEXT NOT NULL, session_id TEXT NOT NULL,
  marker_uid TEXT NOT NULL,              -- deterministic, mirrors event_uid
  ts_ms INTEGER, message_id TEXT, parent_id TEXT, turn_id TEXT,
  kind TEXT NOT NULL,                    -- see list below
  subkind TEXT,                          -- provider-native subtype, verbatim
  payload_json TEXT,                     -- allowlisted provider fields, never raw content bytes
  UNIQUE(source, session_id, marker_uid)
);

kind values (first pass): compaction_boundary, summary, subagent_notification, task_started, task_complete, turn_diff, stream_error, tool_begin, review_mode, unsupported_block (subkind = the block type, payload_json = {bytes, has_signature} only — never image/document bytes), encrypted_reasoning (payload_json = {bytes}), tool_replacement (payload_json = {replaces:[…], collapsedCalls:n}), and unknown for any provider type not yet classified (stored, not dropped).

Also add session_events.raw_kind TEXT so a tool_result that came from a system subagent notification vs a tool_result block is distinguishable, and populate tokens_before_compact for compaction markers from the preceding assistant message's cache_read_input_tokens (burn claude/relationships.rs:384-399).

Scope

  1. Schema + migration marker; bump HYDRATION_PARSER_VERSION.
  2. Claude: handle the three dropped families above; add tests on the corpus fixtures compact-boundary, system-subagent-notification, replacement-meta, task-notification, and a new fixture containing a summary row, an image block, a redacted_thinking block and a thinking signature.
  3. Codex: handle every event_msg/response_item type currently in the _ => {} arms; the marker's subkind is the verbatim payload type so a future unknown type is still recorded as kind = "unknown" rather than dropped (the stored-unknown row is the loud version of "prefer a loud failure to a confident default").
  4. Read API: session_markers_page(conn, source, session_id, cursor, limit) in core with the same (ts_ms IS NULL, ts_ms, id) keyset shape as tool calls; napi + TS + MCP exposure rides the group-2 SDK exposure issue but the core function lands here.
  5. docs/session-catalog.md matrix rows for every marker kind × source; docs/architecture.md operation table row.

Acceptance

  • Snapshot for compact-boundary shows one compaction_boundary marker with tokens_before_compact.
  • Snapshot for system-subagent-notification shows a subagent_notification marker linked by message_id/parent_id to the spawning tool_use.
  • A Codex rollout containing compacted, turn_diff, stream_error yields three markers; an invented event_msg type zzz_future yields kind = "unknown", subkind = "zzz_future".
  • No image/document bytes are ever stored (test asserts payload_json length < 256 for unsupported_block).

Out of scope

Classification of slash-command triads / control prompts (group-2 "control events" issue); fork/resume relationships (separate group-1 issue).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions