Skip to content

[G1] Grok CLI: event-level parity (chat_history + updates.jsonl join, tools, subagents, compaction, real timestamps) #167

Closed

Description

@willwashburn

Part of #160 (group 1 — independent). Absorbs the sourcing half of burn #489 ("Add Grok CLI harness support"), which will no longer be implemented in burn.

Problem

Grok hydration stores prompts only:

  • ingest_grok_session (crates/ai-hist/src/lib.rs:4006) + grok_chat_text (:4180): reads summary.json (info.id, info.cwd/git_root_dir, head_branch, created_at, updated_at, info.model) and chat_history*.jsonl user records (skipping synthetic_reason), producing history rows and last_assistant_text.
  • Timestamps are synthesized: timestamp_ms = first_ts + index (lib.rs:4030) — one millisecond per prompt, not real times. That is a fabricated fact in a ledger whose whole value is truthfulness.
  • No session_events, tool_calls, file_edits, token_json; relationship capability "never".

burn #489 already did the format research. Grok session layout:

~/.grok/sessions/<url-encoded-cwd>/<session-id>/
  chat_history.jsonl   ← record types: system, user, reasoning, assistant, tool_result
  updates.jsonl        ← turnStartMs, totalTokens, timestamps (ACP/UI stream)
  summary.json         ← session id, cwd, model, git metadata
  prompt_context.json  ← AGENTS.md snapshot
  signals.json         ← session aggregates (contextTokensUsed, turnCount, compactionCount)
  subagents/           ← child session metadata
  compaction_checkpoints/

Key facts from #489: Grok does not log per-turn input_tokens/output_tokens; updates.jsonl carries a totalTokens context proxy that can decrease on compaction; models observed grok-composer-2.5-fast, grok-build; tool names Shell, Read, Write, StrReplace/Edit, Grep, Glob, Task, WebSearch/WebFetch, CallMcpTool.

Scope

  1. Fixture: a real Grok session dir in tests/fixtures/grok/ including updates.jsonl, signals.json, subagents/, compaction_checkpoints/.
  2. Parser ingest_grok_session rewrite:
    • chat_history.jsonl: user → session_events(user,text) + history (existing filter for synthetic_reason stays); reasoning → session_events(assistant,thinking) using summary text only (encrypted blobs → an encrypted_reasoning marker, see the dropped-record-types issue); assistant → session_events(assistant,text) + one session_events(assistant,tool_use) + tool_calls row per tool_calls[] entry (args_json = arguments; target picked per tool like Claude's codex_pick_tool_target pattern); tool_result → session_events(tool_result) paired by call id, is_error when the result says so; system → marker.
    • updates.jsonl join: real timestamps per turn (turnStartMs and per-update timestamps) replace the synthesized first_ts + index; totalTokens snapshots stored as token_json = {"context_total_tokens": n, "source": "updates.jsonl"} on the assistant event so consumers can see the proxy and that it is not billing usage. Do not estimate input/output tokens here (that is burn's estimator; relayhistory records only what the harness wrote).
    • signals.json → sessions row extras (models_json, turn count) and a session_markers compaction_boundary per compaction_checkpoints/ entry.
    • prompt_context.json → store the path/hash on the session (burn's overhead attribution reads it later); do not copy the AGENTS.md body into the DB.
    • subagents/ → record_relationship rows with evidence_kind = "grok_subagent_dir"; update relationship_capabilities from "never" to what the format supports.
  3. Discovery (discover.rs:1126-1156): use updates.jsonl for first_activity_ms/last_activity_ms when present.
  4. Capability matrix + README + docs/session-catalog.md: state "usage: context-total proxy only, no per-turn billing tokens".

Acceptance

  • No Grok prompt timestamp is synthesized; a test asserts two prompts from the fixture have timestamps that match updates.jsonl, and the first_ts + index code path is deleted.
  • Snapshot shows tool_calls for Shell/Edit entries, a tool_result event paired to each, ≥1 file_edits row for a Write/StrReplace call, and a compaction_boundary marker.
  • hydrateSession reports partial for Grok (no billing usage) once the honesty issue lands, with the reason in diagnostics.
  • burn Add host-composed source connector intake to the SessionStore facade #489 is updated with a comment pointing here for the sourcing half; burn keeps only pricing (xai overrides), TOOL_ALIASES, AGENTS.md overhead mapping, and the usage estimator.

Out of scope

Token estimation, pricing, burn overhead for prompt_context.json.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions