Skip to content

feat(telemetry): measure codex from its own session files - #1057

Merged
blafourcade merged 16 commits into
nextfrom
feat/telemetry-codex
Oct 19, 2026
Merged

blafourcade merged 16 commits into
nextfrom
feat/telemetry-codex

Conversation

@blafourcade

@blafourcade blafourcade commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

🎯 What & why

A person who works with Codex in a clone that opted in now sees that work in aidd telemetry report. Until now only Claude Code was read. The decision, the mapping and the properties are in contract.md, and the measurements are in evidence.md.

🛠️ How it works

  • One more reader in the ingest list. Resolving, consent, storing, attribution and the report are unchanged.
  • The format (codex-transcript-usage.ts):
    • Only a token_usage_record line is a call. Its usage is read, never the running totals beside it.
    • Model and directory come from the latest turn_context. Version and branch come from the file's first session_meta.
    • It is made once per file read, so it can hold that context.
  • The source (codex-transcript-source-adapter.ts) reads CODEX_HOME, else ~/.codex, under sessions/ and archived_sessions/.
    • A .jsonl that moved is read from byte 0. Lines before the saved offset are handed on as not counted.
    • read-usage-use-case.ts feeds those lines to the format for context and drops what they give. P1 holds because of this.
    • A .jsonl.zst is decompressed as a stream, once. Its decompressed bytes are counted against the size its frame header declares, because Node 22's zlib ends a cut frame quietly, short (zstd-frame.ts). One that does not decompress, or decompresses to another size, is one unrecognised shape naming why: it keeps its position, is tried again next run, and stops no other measurement. An I/O error still fails the read.
  • Shared reading. The Claude Code adapter's block reader, listing and open-stat-read skeleton moved into transcript-files.ts, so both sources share one splitter (byte 0x0a only). Claude Code reads exactly as before, and its adapter tests are unchanged.
  • Sandboxes. Every sandbox that runs the CLI now relocates CODEX_HOME: tests/e2e/helpers.ts, the telemetry sandbox, and both Claude end-to-end scripts.

🧪 How to verify

  • cd cli && pnpm test
  • npx vitest run tests/contexts/telemetry/application/codex-transcript-ingest.property.integration.test.ts: P1 (any split of a file's growth stores the same ledger, and the runs' counts add up) and P3 (plain, then compressed, each call stored once).
  • Real corpus, read-only: the CLI reader and an independent probe agree on 75,584 calls and 9,313,648,283 tokens. 617 of 618 threads match Codex's own cumulative totals. See evidence.md, "Verification".
  • pnpm test passes on Node 22.18.0 and 24.20.0 (6851 tests each).
  • Mutation: changed lines score 96.8, and the telemetry scope scores 96.7 against a floor of 96. Each surviving mutant is justified in evidence.md.
  • The review's case against the built binary: 100 random bytes as a .jsonl.zst in a clone that opted in, then report --json twice. Both exit 0 with unrecognised_shapes 1.

⚠️ Heads-up

  • A .jsonl.zst that does not decompress is tried again every run: calls decoded before the bad frame are stored once, but that run's figures count them again each time. Written as a limit.
  • For a .jsonl.zst, the saved offset is the file's compressed size. The file is read once, and its size and identity keep it read. offsets.json keeps its format; the contract and the usage contract now say what the offset means for these files. An unterminated last line in one is never read.
  • The plan said the format receives the "past the offset" flag. Instead, the use case drops what uncounted lines give, so the format never sees the flag. The behaviour is the same and simpler, and P1 plus a mutant prove it.
  • run.oldest_transcript_at is the oldest file across both tools. Codex's archive can be older than Claude Code's retention, and that is what the field states.
  • Tests that write a .zst import zlib as a namespace and skip when zstd is absent, so they load on Node 22.12 to 22.14.
  • New residual limits in the usage contract:
    • task attribution comes from the branch only
    • git_branch is the branch the session started on
    • Codex-only users are measured when a report runs
    • cache writes are part of input_tokens, billed only on GPT-5.6 and later; a negative input was never seen
    • the calls Codex writes no record for, and the cost fields its file lacks
    • a fork's copied records share their response_id, so they are stored once
    • per-call records ship from Codex 0.153, so an earlier session counts nothing; 0.149 pre-releases to 0.163 were observed
    • a thread resumed from a history with no per-call records counts only its recorded calls
    • a .zst whose frame declares no size is read unchecked, and of a file of several frames zlib reads the first alone
    • a .zst that does not decompress is retried, its decoded calls counted again in each run's figures
    • a .jsonl turning into a .jsonl.zst counts that run's unrecognised shapes and not-stored calls again
  • mutation-covers-source.arch.test.ts once timed out at 6 s against the 5 s default, during a pre-commit while the scripts suite ran beside it. It passed alone and on the retry.

🔗 Linked issue

None. Found by the release 2 scope: Codex is measured.

✅ I certify

  • I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw

blafourcade and others added 10 commits October 19, 2026 06:18
The block reader, the listing and the open-stat-read skeleton move out of
the Claude Code source into a module every source shares. A source may
hand lines before the saved offset as not counted, and the read use case
makes its format once per file read, so a format can hold what earlier
lines said. Claude Code reads exactly as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
A reader of Codex's rollout files joins the ingest list. Only a
token_usage_record line is a call; its model and directory come from the
latest turn_context before it, its version and branch from the file's
first session_meta, so a file that moved is read from its start and the
lines before the saved offset only build that context. Compressed files
are read whole, once, as a stream; a Node without zstd names each one.
Every sandbox that runs the CLI now relocates CODEX_HOME.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Kills the mutants the changed-line run left alive in the Codex reader:
the rollout name pattern's anchors and wildcard, the oldest-written time
of its files, and a session_meta read after a turn_context.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
…mits

The usage contract gains Codex's files, its line-to-record table and its
residual limits, among them a thread resumed from a history with no
per-call records. The plugin README, the envelope reference and both
memory banks stop saying Claude Code only. The task folder holds the
contract, the plan and the evidence, the real-corpus reconciliation
included.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
The telemetry command's description said it measures Claude Code
sessions only. The plugin README now says why a Codex call takes its task
from its branch alone: the plugin's hooks run in Claude Code.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
A .jsonl.zst whose stream zlib cannot decompress failed every ingest and
report, for both tools, and the session-start hook hid it. It is now one
unrecognised shape naming zlib's code; the file keeps its position and
is tried again by the next run. An I/O error still fails the read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
The memory bank said the sandbox guard holds the CODEX_HOME relocation;
it did not. It now does, even when the runner sets its own CODEX_HOME.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
A truncated frame is named Z_BUF_ERROR, not ZSTD_error_*: the test holds
both prefixes of what is passed over. The code is read with String()
rather than a fallback no error reaches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
The consent explanation shown before opting in, the stop and answer
steps, the plugin and marketplace descriptions, the README, the
architecture, FAQ and catalog pages said Claude Code only. The retention
hint now says a tool may have removed older session files. The usage
contract states what a .jsonl.zst's offset is and what a file turning
compressed counts again, and the evidence records review round 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
blafourcade and others added 6 commits October 19, 2026 09:39
Node 22's zlib ends a cut zstd frame with no error and short output, so
a truncated .jsonl.zst was read as complete, its position saved and its
missing calls lost. The reader now counts the decompressed bytes against
the content size the frame header declares; another size is one
unrecognised shape, and the file keeps its position to be read again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Kills the mutants the changed-line run left in the frame header reader:
a single segment's width read from the flag, a frame declaring no size,
the shortest header holding one, and a stream left unchecked when its
frame declares nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Per-call records ship from Codex 0.153, earlier sessions count nothing;
the calls Codex writes no record for, the cost fields its file lacks,
cache writes as OpenAI bills them, frames without a size before 0.152,
coexisting plain and compressed copies, and forks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TnCgcujM2xi6T6dkmVogVw
@blafourcade
blafourcade marked this pull request as ready for review October 19, 2026 09:29
@blafourcade
blafourcade requested a review from a team as a code owner October 19, 2026 09:29
@blafourcade
blafourcade merged commit e8aabe0 into next Oct 19, 2026
61 checks passed
@blafourcade
blafourcade deleted the feat/telemetry-codex branch October 19, 2026 09:29
@blafourcade blafourcade added this to the This week milestone Oct 22, 2026 — with Claude
@aidd-bot aidd-bot Bot mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant