Skip to content

fix: apply streaming deduplication to project cost, prediction, and heatmap - #37

Open
long-910 wants to merge 1 commit into
mainfrom
claude/fix-dedupe-readers-y9ahrw
Open

long-910 wants to merge 1 commit into
mainfrom
claude/fix-dedupe-readers-y9ahrw

Conversation

@long-910

@long-910 long-910 commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

The requestId deduplication introduced in 0.5.1 (#32) only covered the global status-bar cost path (readJsonlFile / readAllUsage). Three other features each had their own ad-hoc JSONL parsing loop with no deduplication, so they still counted every streamed content-block line and inflated their figures 2–4× in agentic sessions:

  • Project-level cost (src/data/projectCost.ts) — status bar suffix and dashboard Project card
  • Burn rate / prediction (src/data/prediction.ts) — also skews rate-limit and budget exhaustion estimates
  • Usage heatmap (src/webview/heatmap.ts) — daily and hourly aggregates

Changes

  • Added a shared deduplicated reader to jsonlReader.ts:
    export interface UsageEntry { timestamp: number; usage: TokenUsage; cwd?: string }
    export async function readUsageEntries(filePath: string): Promise<UsageEntry[]>
    It builds on the existing readJsonlFile dedupe (by requestId, falling back to message.id) and additionally validates timestamps and normalizes all four usage fields.
  • Rewired readAllUsage, projectCost.ts, prediction.ts, and webview/heatmap.ts onto it, deleting the three duplicated parsing loops.
  • docs/DATA.md: new "Streaming Deduplication (applies to ALL readers)" section so future features go through the shared reader.
  • CHANGELOG.md: Unreleased → Fixed entry.

Tests

  • New readUsageEntries suite: dedupe by shared requestId, field normalization + ms timestamp parsing, skipping invalid entries, missing-file graceful degradation.
  • Fixed a time-of-day flake in the heatmap suite: aggregates costs for active days used "N hours ago" offsets that cross midnight when CI runs before 03:00 local (it was failing in this session's container at 02:15 UTC). Timestamps are now anchored to local start-of-today.
  • npm run lint ✅, npm run compile ✅, unit suites (jsonlReader / prediction / heatmap / cache, 34 tests) ✅ via mocha. The @vscode/test-electron runner cannot download VS Code in this sandbox; please rely on CI for the extension-host suites.

🤖 Generated with Claude Code

https://claude.ai/code/session_012ZLCLQeudf1LQBnBSyyr4f


Generated by Claude Code

…eatmap

The requestId deduplication added in 0.5.1 (#32) only covered the global
status-bar cost path (readJsonlFile/readAllUsage). Project-level costs,
the burn-rate prediction, and the usage heatmap each had their own JSONL
parsing loop with no deduplication, so those figures were still inflated
2-4x in agentic sessions.

- Add shared readUsageEntries() to jsonlReader.ts: deduplicated entries
  with validated ms timestamps and normalized usage fields
- Rewire projectCost.ts, prediction.ts, and webview/heatmap.ts onto it
- Add readUsageEntries unit tests
- Make the "aggregates costs for active days" heatmap test deterministic
  (hours-ago offsets crossed midnight when run before 03:00 local)
- Document the shared reader requirement in docs/DATA.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZLCLQeudf1LQBnBSyyr4f
long-910 pushed a commit that referenced this pull request Jul 7, 2026
The ubuntu job failed on the pre-existing heatmap test flake that only
reproduces between 02:00 and 03:00 local time; the fix ships in #37.
Empty commit to re-run CI outside that window.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZLCLQeudf1LQBnBSyyr4f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants