Repository navigation
[bug]: Codex usage before mid-October missing: archived rollouts (.jsonl.zst) are skipped by the scanner #817
Description
Preflight checklist
- I searched existing issues and didn't find a duplicate.
- I'm on the latest version.
What happened?
All Codex usage stored in compressed archives is missing from the dashboard, even though the raw rollout files exist locally and contain full usage data.
Newer Codex versions compress older local sessions into rollout-*.jsonl.zst. On this Windows machine:
- ~/.codex/archived_sessions/ -> 41 files, rollout-*.jsonl.zst (2026-06-09 -> 2026-09-14)
- ~/.codex/sessions/ -> 2 more rollout-*.jsonl.zst, alongside the regular uncompressed .jsonl files
Effect: the earliest usage the dashboard reports on this machine is 2026-08-25 (that date belongs to another provider); Codex itself only appears in mid-October, once fresh uncompressed sessions show up. For reference, replaying the whole local corpus (archived files decompressed + uncompressed sessions) gives 18,433 event_msg/token_count events (~2.26B tokens); TokenTracker currently sees none of the compressed-archive portion.
Steps to reproduce
- Have Codex sessions stored compressed (Codex compresses older sessions automatically): ~/.codex/archived_sessions/ (and sometimes sessions/) contains rollout-.jsonl.zst next to the regular rollout-.jsonl files.
- On a fresh TokenTracker state, run
npx tokentracker-cli sync(or just open the dashboard for the first time). - Look at usage for the covered dates -> all Codex usage in that range is absent; the daily chart shows 0 for those days.
Expected behavior
Codex scanning should also pick up rollout-*.jsonl.zst -- both under sessions/** and in the flat archived_sessions/ layout -- decompressing before parsing, the same way the DSH provider already reads session.jsonl[.zstd]. Dedup stays safe: Codex event keys are sessionUUID + timestamp, so an archived copy of an already-counted session re-reads as a no-op.
Is this a regression?
None
TokenTracker version
1.4.3
Node.js version
v22.18.0
How did you install it?
npm / npx (CLI)
OS / version
Windows 11
Output of tokentracker doctor (if applicable)
Doctor report
- [OK] runtime.base_url - base_url set
- [WARN] runtime.device_token - device token missing
- [OK] runtime.dashboard_url - dashboard_url set
- [OK] runtime.http_timeout_ms - http timeout resolved
- [OK] runtime.debug - debug disabled
- [OK] runtime.auto_retry_no_spawn - auto retry spawn enabled
- [OK] fs.tracker_dir - tracker dir readable
- [OK] fs.config_json - config.json readable
- [OK] cli.entrypoint - cli entrypoint readable
- [OK] network.base_url - HTTP 404 (reachable)
- [OK] notify.configured - notify configured
- [OK] upload.last_error - no upload errors
- [OK] kiro.installs - native ide=no cli=no; wsl not detected (mode wsl-first)
Summary: ok 12 | warn 1 | fail 0 | critical 0Anything else?
Root cause pointers (function names + lines; verified identical in 1.4.2 and in the 1.4.3 npm tarball):
- src/lib/rollout.js -- isRolloutFileName() L179-184:
name.startsWith("rollout-") && name.endsWith(".jsonl") - src/lib/rollout.js -- listRolloutFilesDeep() L201-217 (the recursive variant added to handle flat-archived sessions; see code comment at L195-200): same
.jsonlfilter at L209 -> the compressed files sitting in exactly that directory are silently skipped. - src/lib/codex-rollout-parser.js -- listJsonlFiles() L146-171 (filter at L163) and the deep walk at L334: same
.jsonl-only filter.
The zstd decoding machinery already exists in the codebase (DSH provider / PR #463; Zed thread decoding in rollout.js uses zlib.zstdDecompressSync with an @mongodb-js/zstd fallback, including the multi-frame caveat), so this looks like a discovery-layer addition rather than new machinery.
Happy to help: I can test a patched build on the affected machine and attempt a PR if that's welcome.
Activity
linear commentedon Oct 16, 2026
001-407
xiufengsun commentedon Oct 17, 2026
Confirmed as a real discovery-layer gap on current main (v1.4.3).
isRolloutFileName/listRolloutFilesDeepand the Codex parser’s.jsonl-only walks skiprollout-*.jsonl.zstunder bothsessions/andarchived_sessions/. Zstd decode helpers already exist for DeepSeek Harness, so this is mostly discovery + a whole-file decompress path before the existing JSONL parser (compressed archives are effectively immutable, so byte-offset resume into the compressed stream is the hard part to get wrong).Not shipping a half-fix in today’s maintainer pass — the incremental offset / resume machinery in
parseRolloutIncrementalneeds an explicit compressed-file branch and tests against a real.jsonl.zstfixture. Tracking for a dedicated fix PR next; happy to review a contributor PR if you want to take a first pass.fix(codex): harden zst archive decode and Node 20 tests
fix(codex): scan compressed rollout archives (.jsonl.zst/.zstd) (#829)
chore(release): v1.4.5 (#831)