Claude Code: merge streaming duplicate assistant records by message.id + requestId (per-field max usage) and drop <synthetic> placeholder rows

[willwashburn]

Status against main @ 338ce5f (2026-09-21): still open, and sharper. #164 (PR #190) now stores request_id (crates/ai-hist/src/ingest.rs:6826-6828) and #172 (PR #194) groups usage per request in the session_requests view (crates/ai-hist/src/session_usage.rs:60-105). That view takes MIN(token_json) and counts usage_variants = COUNT(DISTINCT token_json); when the copies disagree the request "reports no usage and carries the ambiguous-usage-copies diagnostic" (docs/usage-accounting.md:205-213). The design assumes Claude copies one identical blob onto every record of a request. tokscale's finding is that streaming writes the same message.id:requestId several times with growing usage, so on main every streamed turn would be refused as ambiguous rather than reported at its final value. <synthetic> is still unhandled (git grep '<synthetic>' crates/ai-hist/src is empty).

Summary

Two Claude Code transcript facts tokscale (commit d8fd670, crates/tokscale-core/src/sessions/claudecode.rs) handles:

1. Streaming duplicates. The same API response is written to the JSONL more than once while it streams. Each line has its own uuid but the same message.id and requestId; later lines carry more complete usage and content. tokscale dedups on messageId:requestId and merges token fields with a per-field max.
2. <synthetic> placeholder model. Local API-error and auth notices are written as type: "assistant" records with message.model = "<synthetic>". They are not model output.

1. Streaming duplicates

claudecode.rs:487-492:

CC's streaming API writes the same messageId:requestId multiple times as the response streams in; later entries often carry more complete token counts. We merge duplicates using per-field max to always keep the highest value seen for each token type.

Key (claudecode.rs:622-660): "{message.id}:{requestId}", else "message:{message.id}". Merge (claudecode.rs:886-914):

t.input       = t.input.max(usage.input_tokens.unwrap_or(0).max(0));
t.output      = t.output.max(usage.output_tokens.unwrap_or(0).max(0));
t.cache_read  = t.cache_read.max(usage.cache_read_input_tokens.unwrap_or(0).max(0));
t.cache_write = t.cache_write.max(usage.cache_creation_input_tokens.unwrap_or(0).max(0));
// duration: never shrunk by an out-of-order duplicate

Assistant keys are globally stable (they come from the API), so a turn replayed into a forked transcript collapses across files; tool_result keys are session-scoped because tool_use_id is per conversation (claudecode.rs:998-1007).

What main does with such a transcript

• ingest_claude_transcript_as (ingest.rs:6601) keys events on the record uuid, so each streamed copy becomes its own session_events rows with that copy's token_json and partial text. ai-hist events and getSessionEventsPage show the turn N times.
• session_requests groups those rows by request-id:<requestId>; distinct token_json blobs make usage_variants > 1, so session_usage_summary reports the request as ambiguous-usage-copies with no usage. The raw_row_sum test at session_usage.rs:870-888 proves the identical-copy case, not the growing-copy case.
• Needs confirmation on a real transcript ([G1] Checked-in harness fixture corpus and parser characterization tests (port burn's 40 fixtures) #163 corpus): whether a current Claude Code version still writes multiple lines per requestId with differing usage. tokscale's parser and merge logic are the only public evidence; if it no longer happens, close this half.

Proposed change

• Parser-side merge, so the view's invariant (one blob per request) stays true: when a record's (request_id, message.id) matches rows already stored for the session, compute the per-field max across the stored token_json and the new one, and write that blob onto all rows of the request (upsert by event_uid). Never shrink a stored value: duplicates arrive out of order (tokscale's duration rule, claudecode.rs:903-912).
• Content: prefer the last copy's text. Either upsert the canonical rows in place and remove earlier partial copies through a named heal (the delete_claude_record_rows pattern, ingest.rs:6278, respecting the retention invariant from Regression test: events already ingested from a Claude transcript must survive an in-place rewrite on resume/compact #213 / docs/architecture.md "Evidence retention": only rows the parser can name), or keep them with raw_kind = "stream_partial" so readers can filter. Decide and document in docs/usage-accounting.md "The dedup rule".
• Add the case to docs/usage-accounting.md: growing copies are one measurement, contradictory copies (a field that decreases) stay ambiguous-usage-copies.

2. <synthetic> placeholder rows

claudecode.rs:470-475, :1292-1306:

Claude Code writes local API-error and auth notices as assistant messages with <synthetic> rather than an API model. A following tool result has no model of its own, so it must not inherit that placeholder ...

fn is_claude_synthetic_placeholder_model(model: &str) -> bool { model.trim().eq_ignore_ascii_case("<synthetic>") }

What main does

The record has message.role = "assistant" and text content, so it is stored as a normal assistant/text event with model = "<synthetic>", it becomes a session_requests row (it has a message.id), it can become the session's last_assistant_text in sessions list, and model inheritance for a following tool result can pick <synthetic>.

Proposed change

• Store it as a session_markers row (kind = "local_notice", subkind = "synthetic", bounded payload_json with the notice text) instead of an assistant event; exclude from last_assistant_text, session_requests and usage; never inherit <synthetic> as a model.

Acceptance

• Fixture streamed-turn.jsonl: 3 records, same message.id + requestId, usage.output_tokens = 5, 40, 120, last has the full text. After hydration: one logical assistant turn with the full text, token_json.output_tokens = 120 on every row of the request, session_requests.usage_variants = 1, session_usage_summary reports 120 (not 165, not refused).
• Same records in the order 5, 120, 40: still 120.
• <synthetic> assistant record followed by a tool_result: no assistant event with model = "<synthetic>", one local_notice marker, last_assistant_text is the previous real assistant text, the tool result's model is NULL.

Related

• [G1] Capture per-message raw facts burn needs: requestId, stop_reason, agent version, sidechain/meta flags, Codex turn ids #164 (closed), [G1] Usage normalization in core: per-request usage records, session summary, and moving attribute_session_usage out of the cloud plugin (supersedes #99 scope) #172 (closed), [G1] Model currently-dropped record types: compaction/summary boundaries, system rows, non-text content blocks, Codex lifecycle events #165 (closed, markers), [G1] Incremental transcript hydration: byte-offset cursors, prefix validation, in-progress message handling, bounded memory #173 (closed, incremental hydration: the in-progress tail is the same duplicate shape), Regression test: events already ingested from a Claude transcript must survive an in-place rewrite on resume/compact #213 (closed, retention invariant), [G1] Checked-in harness fixture corpus and parser characterization tests (port burn's 40 fixtures) #163.