Skip to content

[bug]: Codex usage before mid-October missing: archived rollouts (.jsonl.zst) are skipped by the scanner #817

Closed

Description

@tyb1117

Preflight checklist

What happened?

All Codex usage stored in compressed archives is missing from the dashboard, even though the raw rollout files exist locally and contain full usage data.

Newer Codex versions compress older local sessions into rollout-*.jsonl.zst. On this Windows machine:

  • ~/.codex/archived_sessions/ -> 41 files, rollout-*.jsonl.zst (2026-06-09 -> 2026-09-14)
  • ~/.codex/sessions/ -> 2 more rollout-*.jsonl.zst, alongside the regular uncompressed .jsonl files

Effect: the earliest usage the dashboard reports on this machine is 2026-08-25 (that date belongs to another provider); Codex itself only appears in mid-October, once fresh uncompressed sessions show up. For reference, replaying the whole local corpus (archived files decompressed + uncompressed sessions) gives 18,433 event_msg/token_count events (~2.26B tokens); TokenTracker currently sees none of the compressed-archive portion.

Steps to reproduce

  1. Have Codex sessions stored compressed (Codex compresses older sessions automatically): ~/.codex/archived_sessions/ (and sometimes sessions/) contains rollout-.jsonl.zst next to the regular rollout-.jsonl files.
  2. On a fresh TokenTracker state, run npx tokentracker-cli sync (or just open the dashboard for the first time).
  3. Look at usage for the covered dates -> all Codex usage in that range is absent; the daily chart shows 0 for those days.

Expected behavior

Codex scanning should also pick up rollout-*.jsonl.zst -- both under sessions/** and in the flat archived_sessions/ layout -- decompressing before parsing, the same way the DSH provider already reads session.jsonl[.zstd]. Dedup stays safe: Codex event keys are sessionUUID + timestamp, so an archived copy of an already-counted session re-reads as a no-op.

Is this a regression?

None

TokenTracker version

1.4.3

Node.js version

v22.18.0

How did you install it?

npm / npx (CLI)

OS / version

Windows 11

Output of tokentracker doctor (if applicable)

Doctor report

- [OK] runtime.base_url - base_url set
- [WARN] runtime.device_token - device token missing
- [OK] runtime.dashboard_url - dashboard_url set
- [OK] runtime.http_timeout_ms - http timeout resolved
- [OK] runtime.debug - debug disabled
- [OK] runtime.auto_retry_no_spawn - auto retry spawn enabled
- [OK] fs.tracker_dir - tracker dir readable
- [OK] fs.config_json - config.json readable
- [OK] cli.entrypoint - cli entrypoint readable
- [OK] network.base_url - HTTP 404 (reachable)
- [OK] notify.configured - notify configured
- [OK] upload.last_error - no upload errors
- [OK] kiro.installs - native ide=no cli=no; wsl not detected (mode wsl-first)

Summary: ok 12 | warn 1 | fail 0 | critical 0

Anything else?

Root cause pointers (function names + lines; verified identical in 1.4.2 and in the 1.4.3 npm tarball):

  • src/lib/rollout.js -- isRolloutFileName() L179-184: name.startsWith("rollout-") && name.endsWith(".jsonl")
  • src/lib/rollout.js -- listRolloutFilesDeep() L201-217 (the recursive variant added to handle flat-archived sessions; see code comment at L195-200): same .jsonl filter at L209 -> the compressed files sitting in exactly that directory are silently skipped.
  • src/lib/codex-rollout-parser.js -- listJsonlFiles() L146-171 (filter at L163) and the deep walk at L334: same .jsonl-only filter.

The zstd decoding machinery already exists in the codebase (DSH provider / PR #463; Zed thread decoding in rollout.js uses zlib.zstdDecompressSync with an @mongodb-js/zstd fallback, including the multi-frame caveat), so this looks like a discovery-layer addition rather than new machinery.

Happy to help: I can test a patched build on the affected machine and attempt a PR if that's welcome.

Activity

  1. linear commented on Oct 16, 2026

    @linear
  2. xiufengsun commented on Oct 17, 2026

    @xiufengsun
    Owner

    Confirmed as a real discovery-layer gap on current main (v1.4.3).

    isRolloutFileName / listRolloutFilesDeep and the Codex parser’s .jsonl-only walks skip rollout-*.jsonl.zst under both sessions/ and archived_sessions/. Zstd decode helpers already exist for DeepSeek Harness, so this is mostly discovery + a whole-file decompress path before the existing JSONL parser (compressed archives are effectively immutable, so byte-offset resume into the compressed stream is the hard part to get wrong).

    Not shipping a half-fix in today’s maintainer pass — the incremental offset / resume machinery in parseRolloutIncremental needs an explicit compressed-file branch and tests against a real .jsonl.zst fixture. Tracking for a dedicated fix PR next; happy to review a contributor PR if you want to take a first pass.

  3. added 2 commits that reference this issue on Oct 18, 2026
    750809b
    32596d2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions