Adopt OTel GenAI usage schema and fix Claude token accounting - #26
Merged
Conversation
Replace the flat UsageRow with an OTel-GenAI-shaped row: gen_ai.* dotted attribute keys plus a thirdeye envelope. Key changes: - Inclusive input_tokens; optional cache/reasoning fields are int | None, serialized only when present (absent-vs-zero distinction). - total_tokens is now a derived property, never stored or serialized. - attributes() emits only gen_ai.* keys (incl. gen_ai.conversation.id) as the future OTLP exporter surface; to_dict() merges the envelope. - ATTRIBUTE_KEYS maps fields to the exact spec names, no abbreviations. Tests cover round-trip, None-omission, zero-preservation, conformance to the nine literal spec names, and envelope/attribute separation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Their payload formats were never verified against real data and are
replaced in a later plan by Antigravity and a rebuilt Cursor.
- Delete src/thirdeye/platforms/{gemini,cursor}/ and their test files
- Remove the orphaned gemini_model_response.json fixture
- Reduce PLATFORMS registry in commands/add.py to {claude, codex}
- Drop the thirdeye-gemini-* and thirdeye-cursor-hook console scripts
- Fix ingest --platform help text (cursor -> codex)
- Add find_orphaned_hooks() + `thirdeye add --list` so users are warned
about now-orphaned hook entries left in ~/.gemini and ~/.cursor config
- Rewrite tests/test_add_command.py for the two-platform registry and
cover find_orphaned_hooks with tmp_path fixtures
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior claude_transcript.jsonl was 4 hand-authored lines with 1 distinct message.id — too small to exercise per-call deduplication, which is why the duplicate-row defect passed a green suite. Replace it with a curated 15-line subset of a real Claude Code 2.1.214 transcript (543 assistant frames in the source), selected to exercise the extractor's edge cases while staying at 17 KB: - one message.id on 6 identical-usage frames (the dedup case) - one <synthetic> frame (also the only requestId==null frame) - inclusive-input arithmetic: sample call input_tokens=2 with cache_read=254643 + cache_creation=504 -> 255149 - 8 distinct message.ids -> 7 expected de-duplicated calls All prose scrubbed (text/thinking/tool input+result, cwd, gitBranch, sessionId, absolute paths). Ships machine-readable claude_transcript.expected.json and a provenance README with pasted jq verification. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce usage/read.py: iter_calls collapses raw sidecar rows into one row per distinct (session_id, call_id), last-wins, in first-appearance order; call_totals sums input/output tokens over rows. Dedup lives here alone — UsageStore.iter_rows stays a faithful raw mirror, documented as such and guarded by a test asserting duplicates survive. Update test_usage_store.py make_row to the OTel GenAI UsageRow shape. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude Code 2.1.195 exports 16 hook events; thirdeye only listened for 10. Add PostToolUseFailure, SubagentStart, UserPromptExpansion, PreCompact, PostCompact, and PermissionDenied. - constants.py: six new HOOK_EVENTS entries with thirdeye-claude-* names - hooks.py: six one-line _emit handlers. PostToolUseFailure emits tool_result (not error) so failed calls pair with their tool_call in the web view; SubagentStart emits subagent_start while SubagentStop keeps subagent_message to avoid orphaning recorded sessions. Both asymmetries are commented as intentional. - pyproject.toml: six matching [project.scripts] entry points - tests: constants (16 entries, script names, tomllib entry-point cross-check) and hooks (mapped type, key stripping, session_id no-op, trigger preservation) for the new events test_claude_install.py::test_all_ten_events_registered hardcoded a count of 10, which is unavoidably stale once HOOK_EVENTS grows to 16 (install() enumerates HOOK_EVENTS). The install path itself is unchanged; updated the single assertion to derive the count from HOOK_EVENTS and renamed it test_all_events_registered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Normalize cache-inclusive input tokens (input_tokens + cache_read + cache_creation), emit one row per source frame carrying a dedup key (message.id → requestId → uuid), and drop synthetic placeholder frames. - _extract_row now requires type=="assistant" with a non-empty message.usage; the stale flat frame["usage"] fallback is removed. - "<synthetic>" model frames (zero-token injected messages) are dropped. - Cache fields pass through as int|None — absent stays absent, never coerced to 0. - reasoning_output_tokens is always None (Anthropic reports no thinking token breakout). - Deleted the "transcript frame shape is unverified" first-capture warning; the shape is now verified against a real transcript. Deduplication of the repeated per-block frames stays out of the writer (usage/read.py owns it). Tests drive from the scrubbed real-data fixture and its expected.json companion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cover the spec-mandated rejection branches (empty usage, both token fields absent, non-assistant/non-dict frames, synthetic model) and the corrupt-line skip, raising _extract_row coverage to 98%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace UsageStore(sd).iter_rows() with iter_calls(sd) in aggregate_by_day so duplicated per-frame rows collapse to one call before bucketing. TimeBucket keeps its four fields plus derived total_tokens. Rewrite tests against the new UsageRow signature, guarding dedup with the six-duplicate case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump SCHEMA_VERSION to 2 and replace the usage table: - PRIMARY KEY (session_id, call_id) instead of (session_id, seq), so a turn's shared Stop-hook seq no longer collapses every call into one row. - gen_ai_* column names; nullable cache/reasoning columns preserve the absent-vs-zero distinction. - seq demoted to an indexed plain column; indexes on response_model, ts, platform, seq. connect() drops and rebuilds when user_version < SCHEMA_VERSION (the DB is purely derived from sidecars, so no data migration is warranted). _refresh_one reads through iter_calls (the single dedup path) and upserts with ON CONFLICT DO UPDATE so a later append that corrects a call wins. Byte-offset bookmark still gates re-reads; shrunk-sidecar and malformed-line logging are preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Session usage now reads through usage.read.iter_calls so repeated per-frame rows collapse to one logical call. The session table renders the new per-call fields: response_model as a model column, plus cache read / cache creation / reasoning columns that show '-' for an absent (None) value and '0' for a reported zero. The global platform filter drops the removed gemini platform, leaving claude and codex. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route `thirdeye usage` show and rollup views through the canonical `usage/read.py::iter_calls` dedup path instead of raw SQL, so the command honors the single-dedup-point invariant, renders the new `gen_ai.response.model` plus cache columns (absent -> "-", distinct from a reported 0), and emits dotted `gen_ai.*` keys in `--json` via `UsageRow.to_dict()` rather than underscored SQL names. Totals are the derived `input + output` sum for display and sorting. Add `thirdeye usage reset`: reports the sidecar/session/DB-row counts it will destroy, refuses without `--yes` (non-zero, names the flag, deletes nothing), then removes every usage.jsonl / usage.state.json and usage.db while leaving events.alog, events.idx, tags.jsonl, meta.yaml and upstream transcripts untouched. Afterward it scans for orphaned thirdeye-gemini-*/thirdeye-cursor-hook commands (detection only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cover exact reset counts, refusal-still-reports-counts, tags.jsonl and upstream-transcript preservation, orphaned-hook detection (present/absent), per-session --sort ts, --model filter, --since/--until window, empty-result message, and rollup TOTAL = input + output with dedup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…& ts parsing - Surface the broken `usage reindex`: restore the three reindex tests, marking the two schema-dependent ones xfail with a reason pointing at the schema-v1 index.py gap (the schema-v2 migration is the usage-index task). They fail today (0 rows / no call_id column), flagging the gap instead of hiding it. - Restore the 6 errors tests deleted without cause (format-independent), adapting the lone gemini entry to codex. - Normalize `_row_ts` to UTC-aware and guard TypeError so `usage <id> --since` never crashes on a naive timestamp. - Move `session_dir` to the top-level paths import; drop the local imports. - Rename the rollup `--json` key `turns` -> `calls` to match the CALLS header. - Close the reset sqlite connection in a finally and remove usage.db-wal/-shm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Finding 1: Codex usage capture was silently broken by the shared UsageRow rewrite. Update _extract_usage_row to the new OTel GenAI schema: derive a stable call_id from the frame's byte offset, set provider_name="openai", response_model, operation_name="chat", pass cache/reasoning fields as None, and drop the stored total_tokens. Align test_usage_codex assertions with the dotted gen_ai.* serialized keys. Finding 2: Migrate tests/test_usage_index.py to the new UsageRow (call_id, provider_name, response_model; drop model/total_tokens), swap the dropped gemini seed for codex, and update the incremental-refresh assertion to the whole-file iter_calls upsert semantics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-expresses the per-turn usage store in OpenTelemetry GenAI semantic
conventions and fixes the token-accounting defects found by auditing the
extractors against real transcripts: duplicated Claude rows, dropped cache
tokens, non-comparable input-token columns, and an index that collapsed a
turn's calls into one. Also brings Claude Code capture up to date (16 hook
events instead of 10, rewritten usage extractor) and removes the unverified
Gemini and Cursor platforms.
Changes
UsageRowingen_ai.*terms with inclusive input tokens andabsent-vs-zero optional fields (
usage/types.py).usage/read.py) and route the dailyaggregation through it (
usage/aggregate.py); keep the sidecar a faithfulraw mirror (
usage/store.py).call_idso a turn'scalls are no longer collapsed (
usage/index.py).cache-read and cache-creation tokens (
platforms/claude/usage.py); wire thesix missing hook events, including
PostToolUseFailure(
platforms/claude/hooks.py,constants.py).usage reset(
commands/usage.py); update the web session and global usage views(
web/routes/usage.py,web/templates/usage/*.html).(
platforms/gemini/*,platforms/cursor/*).document fixture provenance (
tests/fixtures/usage/*); adjust the Codexextractor to the new
UsageRowcontract without changing Codex behavior(
platforms/codex/usage.py).Test plan
uv run pytest tests/test_usage_types.py tests/test_usage_read.py -vconfirms the OTel
UsageRowschema and the call-level dedup read path.uv run pytest tests/test_usage_claude.py -vconfirms the Claudeextractor emits one row per call with inclusive input/cache tokens.
uv run pytest tests/test_claude_hooks.py tests/test_claude_constants.py -vconfirms all 16 hook events, including
PostToolUseFailure, are wired.uv run pytest tests/test_usage_index.py tests/test_usage_aggregate.py -vconfirms the index keys by
call_idand aggregation reads the dedup path.uv run pytest tests/test_usage_command.py tests/web/test_routes_usage.py -vconfirms the CLI (including
usage reset) and web views render the new columns.uv run ruff format src tests && uv run ruff check src testspasses.🤖 Generated by workbench pr_writer