[G1] Grok CLI: event-level parity (chat_history + updates.jsonl join, tools, subagents, compaction, real timestamps) #167
Copy link
Copy link
Closed
Description
Part of #160 (group 1 — independent). Absorbs the sourcing half of burn #489 ("Add Grok CLI harness support"), which will no longer be implemented in burn.
Problem
Grok hydration stores prompts only:
ingest_grok_session(crates/ai-hist/src/lib.rs:4006) +grok_chat_text(:4180): readssummary.json(info.id,info.cwd/git_root_dir,head_branch,created_at,updated_at,info.model) andchat_history*.jsonluser records (skippingsynthetic_reason), producinghistoryrows andlast_assistant_text.- Timestamps are synthesized:
timestamp_ms = first_ts + index(lib.rs:4030) — one millisecond per prompt, not real times. That is a fabricated fact in a ledger whose whole value is truthfulness. - No
session_events,tool_calls,file_edits,token_json; relationship capability"never".
burn #489 already did the format research. Grok session layout:
~/.grok/sessions/<url-encoded-cwd>/<session-id>/
chat_history.jsonl ← record types: system, user, reasoning, assistant, tool_result
updates.jsonl ← turnStartMs, totalTokens, timestamps (ACP/UI stream)
summary.json ← session id, cwd, model, git metadata
prompt_context.json ← AGENTS.md snapshot
signals.json ← session aggregates (contextTokensUsed, turnCount, compactionCount)
subagents/ ← child session metadata
compaction_checkpoints/
Key facts from #489: Grok does not log per-turn input_tokens/output_tokens; updates.jsonl carries a totalTokens context proxy that can decrease on compaction; models observed grok-composer-2.5-fast, grok-build; tool names Shell, Read, Write, StrReplace/Edit, Grep, Glob, Task, WebSearch/WebFetch, CallMcpTool.
Scope
- Fixture: a real Grok session dir in
tests/fixtures/grok/includingupdates.jsonl,signals.json,subagents/,compaction_checkpoints/. - Parser
ingest_grok_sessionrewrite:chat_history.jsonl:user→session_events(user,text)+history(existing filter forsynthetic_reasonstays);reasoning→session_events(assistant,thinking)usingsummarytext only (encrypted blobs → anencrypted_reasoningmarker, see the dropped-record-types issue);assistant→session_events(assistant,text)+ onesession_events(assistant,tool_use)+tool_callsrow pertool_calls[]entry (args_json= arguments;targetpicked per tool like Claude'scodex_pick_tool_targetpattern);tool_result→session_events(tool_result)paired by call id,is_errorwhen the result says so;system→ marker.updates.jsonljoin: real timestamps per turn (turnStartMsand per-update timestamps) replace the synthesizedfirst_ts + index;totalTokenssnapshots stored astoken_json = {"context_total_tokens": n, "source": "updates.jsonl"}on the assistant event so consumers can see the proxy and that it is not billing usage. Do not estimate input/output tokens here (that is burn's estimator; relayhistory records only what the harness wrote).signals.json→sessionsrow extras (models_json, turn count) and asession_markerscompaction_boundarypercompaction_checkpoints/entry.prompt_context.json→ store the path/hash on the session (burn's overhead attribution reads it later); do not copy the AGENTS.md body into the DB.subagents/→record_relationshiprows withevidence_kind = "grok_subagent_dir"; updaterelationship_capabilitiesfrom"never"to what the format supports.
- Discovery (
discover.rs:1126-1156): useupdates.jsonlforfirst_activity_ms/last_activity_mswhen present. - Capability matrix + README +
docs/session-catalog.md: state "usage: context-total proxy only, no per-turn billing tokens".
Acceptance
- No Grok prompt timestamp is synthesized; a test asserts two prompts from the fixture have timestamps that match
updates.jsonl, and thefirst_ts + indexcode path is deleted. - Snapshot shows
tool_callsforShell/Editentries, atool_resultevent paired to each, ≥1file_editsrow for aWrite/StrReplacecall, and acompaction_boundarymarker. hydrateSessionreportspartialfor Grok (no billing usage) once the honesty issue lands, with the reason indiagnostics.- burn Add host-composed source connector intake to the SessionStore facade #489 is updated with a comment pointing here for the sourcing half; burn keeps only pricing (
xaioverrides),TOOL_ALIASES,AGENTS.mdoverhead mapping, and the usage estimator.
Out of scope
Token estimation, pricing, burn overhead for prompt_context.json.
Reactions are currently unavailable
Activity
feat(hydrate): Grok is `partial`, because Grok records no billing tokens
Merge origin/main into claude/g1-165-session-markers