Repository navigation
[G1] Model currently-dropped record types: compaction/summary boundaries, system rows, non-text content blocks, Codex lifecycle events #165
Description
Part of #160 (group 1 — independent).
Problem
Both parsers silently drop record types that a "complete and thorough" ledger must keep. Locations (all in crates/ai-hist/src/lib.rs):
Claude
:3073—let Some(content) = message.and_then(|m| m.get("content")) else { continue; }drops every record withoutmessage.content. That includestype:"summary"rows,type:"system"rows withsubtype:"compact_boundary"(burn emits aCompactionEventfrom these —reader/claude/incremental.rs:482-510), and system rows carryingparent_tool_use_id+agent_id/subagent_session_id(burn turns these into subagent-notification tool-result events —reader/claude/tool_results.rs:178-240).:3297—_ => {}drops every content block that is nottext|thinking|tool_use|tool_result:image,document,redacted_thinking,server_tool_use,web_search_tool_result,mcp_tool_use/mcp_tool_result, and thinkingsignatures._meta.replaces/_meta.collapsedCallson tool results (tool-replacement metadata; burnreader/claude/tool_results.rs:295-340) are not read.toolUseResult.agentIdis read for file edits and relationships but is not stored as a fact on the tool-result event itself.
Codex
:2502and:2504—_ => {}dropscompacted,turn_diff,stream_error,exec_command_begin,patch_apply_begin,mcp_tool_call_begin,entered_review_mode,task_started, and anysubagent_*lifecycle event (subagent_*_{complete,done,finished,terminated}; burnreader/codex.rs:534-542).:2495-2497—response_item/reasoning(encrypted) only flipssaw_model_output; the presence of an encrypted reasoning block (and its byte length) is not recorded.
The session_events.kind CHECK constraint (crates/ai-hist-core/src/lib.rs:149) allows only text|thinking|tool_use|tool_result, so none of these can be stored without a schema change.
Design
Add a session_markers table rather than overloading session_events:
CREATE TABLE session_markers (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source TEXT NOT NULL, session_id TEXT NOT NULL,
marker_uid TEXT NOT NULL, -- deterministic, mirrors event_uid
ts_ms INTEGER, message_id TEXT, parent_id TEXT, turn_id TEXT,
kind TEXT NOT NULL, -- see list below
subkind TEXT, -- provider-native subtype, verbatim
payload_json TEXT, -- allowlisted provider fields, never raw content bytes
UNIQUE(source, session_id, marker_uid)
);kind values (first pass): compaction_boundary, summary, subagent_notification, task_started, task_complete, turn_diff, stream_error, tool_begin, review_mode, unsupported_block (subkind = the block type, payload_json = {bytes, has_signature} only — never image/document bytes), encrypted_reasoning (payload_json = {bytes}), tool_replacement (payload_json = {replaces:[…], collapsedCalls:n}), and unknown for any provider type not yet classified (stored, not dropped).
Also add session_events.raw_kind TEXT so a tool_result that came from a system subagent notification vs a tool_result block is distinguishable, and populate tokens_before_compact for compaction markers from the preceding assistant message's cache_read_input_tokens (burn claude/relationships.rs:384-399).
Scope
- Schema + migration marker; bump
HYDRATION_PARSER_VERSION. - Claude: handle the three dropped families above; add tests on the corpus fixtures
compact-boundary,system-subagent-notification,replacement-meta,task-notification, and a new fixture containing asummaryrow, animageblock, aredacted_thinkingblock and a thinking signature. - Codex: handle every
event_msg/response_itemtype currently in the_ => {}arms; the marker'ssubkindis the verbatim payload type so a future unknown type is still recorded askind = "unknown"rather than dropped (the stored-unknownrow is the loud version of "prefer a loud failure to a confident default"). - Read API:
session_markers_page(conn, source, session_id, cursor, limit)in core with the same(ts_ms IS NULL, ts_ms, id)keyset shape as tool calls; napi + TS + MCP exposure rides the group-2 SDK exposure issue but the core function lands here. docs/session-catalog.mdmatrix rows for every marker kind × source;docs/architecture.mdoperation table row.
Acceptance
- Snapshot for
compact-boundaryshows onecompaction_boundarymarker withtokens_before_compact. - Snapshot for
system-subagent-notificationshows asubagent_notificationmarker linked bymessage_id/parent_idto the spawningtool_use. - A Codex rollout containing
compacted,turn_diff,stream_erroryields three markers; an inventedevent_msgtypezzz_futureyieldskind = "unknown", subkind = "zzz_future". - No image/document bytes are ever stored (test asserts
payload_jsonlength < 256 forunsupported_block).
Out of scope
Classification of slash-command triads / control prompts (group-2 "control events" issue); fork/resume relationships (separate group-1 issue).
Activity
Merge origin/main into claude/g1-165-session-markers