feat(usage): make provider token accounting complete and auditable

[Sinity]

Summary

Polylogue needs a comprehensive provider-usage accounting revamp. The archive currently has three different things that are easy to confuse:

• provider-reported usage telemetry, partially materialized for Claude Code;
• visible transcript text counts, including huge tool/result payloads;
• provider/account UI summaries such as Codex /usage and Claude Code usage stats.

Those are not interchangeable. Today Polylogue can answer text-volume questions, and it can price some Claude Code rows, but it cannot yet produce an honest cross-provider usage ledger with coverage, caveats, cached-vs-uncached splits, reasoning tokens, current-window usage, cumulative session usage, and provider/account reconciliation.

This should be treated as a product feature and data-quality fix, not as a narrow Codex parser patch.

Billing mode: subscription vs API — do not conflate, and do not imply spend

Token counts and dollars are different axes, and a per-token dollar figure is actual
spend only under pay-per-token API billing. Three distinct things must never collapse
into one "cost" number:

1. Token counts (input / output / cache / reasoning) — real, provider-reported,
   auditable. The ground truth.
2. API-equivalent dollar cost = tokens × catalog rate. This equals real spend only
   when the usage was billed per-token via an API key. For usage under a subscription
   (Claude Max/Pro, ChatGPT Plus/Pro, Codex via a ChatGPT plan) the marginal token cost is
   ~$0 — the real cost is the flat subscription fee, and the per-token figure is a
   notional "what this would have cost at API rates," not money spent. The archive's
   current estimated USD where priced ≈ $89,003 is precisely this trap: it is an
   API-equivalent notional, not $89k of spend, when the underlying sessions ran on a
   subscription.
3. Subscription quota / consumption — the provider-UI view (Codex /usage
   lifetime/peak/weekly; Claude Code all-time / 30d / 7d token stats). This measures how
   much of your plan you have consumed, a third axis distinct from both token counts and
   API-equivalent dollars.

Requirements this imposes:

• Billing mode is a first-class, captured dimension, not an inference. It is an
  account/plan fact absent from the transcript, so it must be operator-declared per
  origin/account/model via config (e.g. claude-code-session ⇒ subscription:max, an
  API-keyed origin ⇒ api). Default to unknown.
• Every dollar figure carries its billing-mode label — api_spend (real) vs
  api_equivalent_notional (subscription/unknown) — never a bare "cost." Never present a
  dollar figure as spend when billing mode is subscription or unknown. A subscription
  rollup may show the flat fee and the notional API-equivalent side by side, clearly
  distinguished.
• Provider-UI totals are the quota axis, reconciled and labeled as plan consumption,
  not equated with either token sums or dollars.

Evidence from the live archive

Local archive inspected on 2026-06-22:

• index.db: 16,331 sessions, 4,167,733 messages, 795,847,071 stored words.
• Stored words are dominated by tool/result text, not billable conversational usage:
  • tool: 2,114,848 messages, 626,139,954 words.
  • assistant: 1,680,184 messages, 104,399,334 words.
  • user: 212,765 messages, 62,828,718 words.
  • system: 27,869 messages, 2,478,897 words.
• Origin totals from sessions:
  • codex-session: 2,424 sessions, 1,818,098 messages, 515,077,798 words, 1,430,485 tool messages.
  • claude-code-session: 10,092 sessions, 2,257,847 messages, 242,816,789 words, 668,873 tool messages.
  • chatgpt-export: 2,613 sessions, 69,712 messages, 26,465,627 words.
  • claude-ai-export: 1,012 sessions, 12,476 messages, 2,852,707 words.
  • aistudio-drive: 190 sessions, 9,600 messages, 8,634,150 words.

Existing token tables are incomplete by provider:

• messages already has input_tokens, output_tokens, cache_read_tokens, cache_write_tokens, and duration_ms.
• session_model_usage already has per-session/per-model rollups, pricing fields, and cache token columns.
• session_model_usage currently contains 15,614 rows across 14,066 sessions with:
  • input tokens: 756,944,355
  • output tokens: 327,793,687
  • cache-read tokens: 161,586,382,700
  • cache-write tokens: 4,762,317,995
  • estimated USD where priced: 89,003.40796069
• Those populated values are overwhelmingly Claude Code. Codex model names exist in session_model_usage but with all token columns zero.
• codex-session has 2,424 indexed sessions and 1,818,098 messages, but 0 messages with token fields populated. All Codex input/output/cache token sums in messages are zero.

Raw acquisition coverage also has usage-relevant holes:

• source.db.raw_sessions has no parse errors for the inspected origins:
  • claude-code-session: 10,094 raw rows, 0 parse errors.
  • codex-session: 2,426 raw rows, 0 parse errors.
  • chatgpt-export: 2,617 raw rows, 0 parse errors.
  • claude-ai-export: 1,012 raw rows, 0 parse errors.
  • aistudio-drive: 238 raw rows, 0 parse errors.
• Acquired-but-not-materialized rows exist despite no parse errors:
  • codex-session: 2 raw sessions missing from sessions.
  • claude-code-session: 2 raw sessions missing from sessions.
  • chatgpt-export: 4 raw sessions missing from sessions.
  • aistudio-drive: 238 raw rows, 0 materialized sessions in this archive snapshot.
• The two Codex acquired-not-materialized rows are huge and usage-significant:
  • 019e309f-7614-7381-a8ac-f9080f304ee6, 278,927,296 bytes, 47,149 token_count events, max cumulative total_token_usage.total_tokens 3,980,610,780, with 3,913,381,120 cached input tokens.
  • 019e1a5c-0abe-71e1-8adf-8ae8d4cc71a6, 174,043,306 bytes, 23,697 token_count events, max cumulative total_token_usage.total_tokens 2,395,913,107, with 2,352,167,808 cached input tokens.

Provider/account UI comparison shows why semantics must be explicit:

• Codex CLI /usage reports lifetime 92B tokens, peak 3.06B, and daily/weekly/cumulative views over the last 12 months.
• Claude Code all-time UI reports 972.8m total tokens, 2.5k sessions, 222 active days / 399; last 30 days reports 231.1m tokens; last 7 days reports 37.6k tokens.
• Polylogue currently has Claude Code non-cache message usage around 1.085B tokens and cache-inclusive usage around 167.433B tokens. That is not necessarily contradictory, but it proves the UI must label provider semantics, cache inclusion, pricing semantics, and coverage instead of showing one undifferentiated “tokens” number.

Code findings

• polylogue/sources/parsers/claude/code_parser.py extracts Claude Code per-message message.usage into ParsedMessage token fields, including cache_read_input_tokens and cache_creation_input_tokens.
• polylogue/storage/sqlite/archive_tiers/write.py aggregates message token fields into session_model_usage and prices known models.
• polylogue/sources/parsers/codex.py only extracts token fields from message-like records or direct usage/tokens dictionaries.
• Codex token_count events are currently preserved as session_events. tests/unit/sources/test_codex_event_stream_contract.py explicitly asserts that a trailing token_count event does not mutate the prior message, which is correct for cumulative event streams but leaves no separate usage ledger.
• Representative current Codex raw rows contain event_msg records with payload.type = token_count, payload.info.total_token_usage, payload.info.last_token_usage, and model_context_window. External Codex issue reports document the same shape and the distinction between cumulative totals and per-request last usage:
  • Add reasoning token usage on json output openai/codex#5276
  • Include token_count events in non-interactive session files openai/codex#9660
  • Include per-API-call token usage (last) in turn.completed JSONL events openai/codex#17539

Target product model

Polylogue should expose an honest usage model with at least these categories:

• transcript_text_estimate: derived from stored visible text; useful for corpus size and approximate context packaging; never presented as provider-reported usage.
• provider_reported_usage: exact counters emitted by the source/provider when available.
• usage_event: raw per-turn/per-request/per-session usage observations with source refs, timestamps, event ids/positions, and shape version.
• usage_rollup: derived rollups by session, logical session, day/week/month, origin, model, account/provider, cwd/project, repo, branch, tool/action class, and active/inactive lineage where available.
• coverage: explicit accounting of known/missing/unsupported provider usage data so zero does not mean “no usage” when it really means “not extracted”.

Usage dimensions should include, when the source supports them:

• input tokens;
• uncached input tokens or effective input tokens;
• cached input tokens / cache read tokens;
• cache write / cache creation tokens;
• output tokens;
• reasoning output tokens;
• total tokens as provider reports it;
• model context window;
• current/request usage (last_token_usage) vs cumulative session/thread usage (total_token_usage);
• model name and effort/reasoning level;
• duration/latency;
• cost in USD/credits, with pricing catalog/provenance;
• billing/effective-cost semantics, including whether cache reads are billed at discounted rates or only used for volume accounting;
• provider/account UI reconciliation fields where possible.

Acceptance criteria

1. Usage vocabulary and docs

   • Define the difference between transcript text estimates, provider-reported usage, provider UI totals, rollups, cost estimates, and context-window utilization.
   • Define billing mode (api vs subscription vs unknown) as a captured dimension; require every dollar figure to be labeled api_spend vs api_equivalent_notional, and never present subscription/unknown usage as spend.
   • Document cache semantics: cache read, cache write/cache creation, uncached input, output, reasoning output, total, billable/effective tokens.
   • Document last/request usage versus cumulative session usage, especially for Codex.
   • Document which providers/origins currently have exact usage, partial usage, text-only estimates, or no reliable usage telemetry.

2. Storage/read model

   • Add or revise a usage-event ledger rather than overloading messages with cumulative provider counters.
   • Preserve source evidence refs: origin, native session id, raw id/artifact id where available, source path/blob hash, event position/timestamp, provider payload shape/version, and extraction confidence.
   • Keep session_model_usage or its successor as a rollup driven from the ledger, not as a decorative table that silently contains skeleton rows with zeros.
   • Add rollups for session, logical session, day/week/month, origin, provider account if available, model, cwd/project/repo, and active lineage where available.
   • Represent missing/unsupported usage as explicit coverage states, not zeros.

3. Codex extraction

   • Parse current Codex event_msg payload.type = token_count events with nested info.total_token_usage, info.last_token_usage, and model_context_window.
   • Preserve both last_token_usage and total_token_usage without double-counting cumulative totals.
   • Capture input_tokens, cached_input_tokens, output_tokens, reasoning_output_tokens, and total_tokens.
   • Support older/simple token_count payloads already represented in tests.
   • Add fixtures for current real-world Codex JSONL shapes, including interactive and codex exec differences where possible.
   • Existing behavior that token_count does not mutate the prior message should remain true; usage belongs in the usage ledger and derived rollups.

4. Claude Code extraction and reconciliation

   • Keep existing per-message Claude Code usage extraction, but route it through the same ledger/rollup vocabulary as Codex.
   • Preserve cache_read_input_tokens and cache_creation_input_tokens as first-class fields.
   • Reconcile Polylogue’s exact counters with Claude Code UI/all-time windows by labeling cache-inclusive, cache-exclusive, priced, and provider-UI-comparable totals.
   • Report model-level totals for models currently visible in the archive, including Claude, DeepSeek, and any future Claude Code model names.

5. ChatGPT, Claude.ai export, AI Studio/Gemini, and future providers

   • Audit every supported origin for provider-reported token/cost telemetry in raw exports or sidecars.
   • Where exports have no token telemetry, expose text-estimate-only coverage explicitly.
   • For ChatGPT exports, keep reported costs or model metadata distinct from exact token usage when exact counters are absent.
   • For AI Studio/Gemini, resolve why 238 validation-passed raw rows are not materialized in the inspected archive before claiming usage coverage.
   • Add provider-shape fixtures and negative tests for “no telemetry available” so docs/surfaces do not imply exact usage.

6. CLI/API/MCP/product surfaces

   • Add a coherent usage surface, e.g. polylogue analyze usage or an ops/analyze usage command, with --json output.
   • Include coverage summaries by origin/model/time window: exact, partial, estimate-only, missing, unsupported, and acquired-not-materialized.
   • Include cached-vs-uncached splits, reasoning tokens, output tokens, context-window utilization, and cost estimates where supported.
   • Provide daily/weekly/monthly/cumulative views comparable to provider UIs, while clearly labeling differences.
   • Expose the same data through Python API and MCP for agent workflows.
   • Make ops status / diagnostics surface suspicious gaps: raw rows acquired but not materialized, sessions with model rows but zero usage, and providers with raw token events not projected.

7. Diagnostics and debt accounting

   • Add an archive diagnostic that distinguishes:
     • raw rows acquired but not materialized;
     • raw rows with parse errors;
     • materialized sessions without messages;
     • messages with model names but no usage;
     • raw provider usage events not projected into the usage ledger;
     • rollup rows whose inputs are stale.
   • These diagnostics should be cheap enough for normal status/workload use and should support exact/deep modes for large archives.

8. Tests and verification

   • Add unit fixtures for Codex nested token_count events, Claude Code message usage, ChatGPT cost-without-token telemetry, and text-only provider exports.
   • Add rollup tests proving cumulative Codex totals are not double-counted and last_token_usage can be summed by turn/request.
   • Add cross-surface tests for CLI JSON, API, and MCP usage payloads.
   • Add regression coverage for acquired-not-materialized rows with no parse error showing up as coverage/debt.
   • Keep generated docs/schemas in sync if new JSON output surfaces are added.

9. Migration/rebuild plan

   • Follow Polylogue’s fresh-first schema policy: if storage shape changes, document the re-ingest/rebuild path rather than adding an in-place upgrade chain.
   • Rebuild usage ledger/rollups from existing raw/source evidence where possible so the operator does not have to reacquire provider data unnecessarily.
   • Make the expected runtime and disk impact visible before the rebuild on large archives.

Non-goals

• Do not pretend visible text word counts are exact provider tokens.
• Do not attach cumulative Codex totals to every message or prior assistant message.
• Do not preserve old misleading zero-token behavior for compatibility.
• Do not add a registry that is not consumed by extraction, rollups, docs, and tests.

Suggested first implementation slice

A good first slice is Codex + coverage plumbing because the raw data is present and the current zero-token behavior is plainly wrong:

1. Introduce the usage-event model/table or equivalent internal primitive.
2. Extract Codex token_count events into that primitive.
3. Roll up per-session and per-day Codex usage with last-vs-total semantics.
4. Add CLI JSON output plus coverage diagnostics showing the previous zero-token gap.
5. Then fold Claude Code through the same primitive so cross-provider views have one shape.

After that, audit ChatGPT/Claude.ai/AI Studio/Gemini telemetry and complete the provider coverage matrix.

---

Current state (audited 2026-06-27)

Readiness: ~52% already in code · scope L · conflict risk medium · depends on: none

Assessment & remaining work: The core ledger infrastructure is in place: session_provider_usage_events DDL (index schema v6), Codex token_count extraction into that table, Claude Code routing through the same vocabulary, polylogue analyze usage CLI command, provider_usage MCP tool, and API facade provider_usage_report all exist. The remaining scope is: (1) add daily/weekly/monthly/cumulative time-window rollups and cwd/project/repo rollup dimensions to storage/usage.py and surface them in CLI/API/MCP (AC 2, AC 6); (2) document the usage vocabulary standalone (AC 1) and write an explicit rebuild plan for the usage ledger (AC 9); (3) audit and add provider-shape fixtures for ChatGPT/Claude.ai/AI Studio "no telemetry" negative tests, investigate the AI Studio 238-row materialization gap, and add cross-surface CLI/API/MCP usage payload tests (AC 5, AC 8); (4) add provider UI reconciliation labeling for Claude Code cache-inclusive vs cache-exclusive vs priced totals (AC 4). These remaining ACs span multiple non-trivial files and warrant multiple PRs stacked against master.

Primary files: polylogue/storage/usage.py, polylogue/storage/sqlite/archive_tiers/index.py, polylogue/storage/sqlite/archive_tiers/write.py, polylogue/sources/parsers/codex.py, polylogue/cli/commands/diagnostics.py, polylogue/mcp/server_tools.py, polylogue/api/archive.py, docs/cost-model.md

Grounded re-assessment from the 2026-06-27 backlog triage; verify against source before acting.