diff --git a/README.de.md b/README.de.md index 32fbdb67f..e336bf639 100644 --- a/README.de.md +++ b/README.de.md @@ -6,7 +6,7 @@ ### Sieh genau, was du für KI ausgibst – über jedes CLI hinweg -Sammle automatisch Token-Zahlen von **32 KI-Coding-Tools**, aggregiere sie lokal und sieh echte Kostentrends in einem schönen Dashboard. Kein Cloud-Konto, keine API-Keys, kein Setup – nur ein Befehl. +Sammle automatisch Token-Zahlen von **33 KI-Coding-Tools**, aggregiere sie lokal und sieh echte Kostentrends in einem schönen Dashboard. Kein Cloud-Konto, keine API-Keys, kein Setup – nur ein Befehl. [![npm version](https://img.shields.io/npm/v/tokentracker-cli.svg?color=blue)](https://www.npmjs.com/package/tokentracker-cli) [![npm downloads](https://img.shields.io/npm/dm/tokentracker-cli.svg?color=brightgreen)](https://www.npmjs.com/package/tokentracker-cli) @@ -88,7 +88,7 @@ Aktualisieren mit `brew upgrade --cask xiufengsun/tokentracker/tokentracker`. De ## ✨ Features -- 🔌 **32 KI-Tools out of the box** — Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness +- 🔌 **33 KI-Tools out of the box** — Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness,TRAE Work CN - 🏠 **100 % lokal** — Token-Daten verlassen nie deinen Rechner. Kein Konto, keine API-Keys. - 🚀 **Zero Config** — Hooks installieren sich beim ersten Start automatisch. Von null zum Dashboard in 30 Sekunden. - 📊 **Schönes Dashboard** — Nutzungstrends, Kostenaufschlüsselung nach Modell, GitHub-ähnliche Aktivitäts-Heatmap, Projektzuordnung @@ -195,6 +195,7 @@ Aktualisieren mit `brew upgrade --cask xiufengsun/tokentracker/tokentracker`. De | **Qoder** | ✅ Auto | Passiver SQLite-Reader (`Qoder/SharedClientCache/cache/db/local.db`; liest nur assistant-`token_info`, trennt Cache-Eingaben und liest keine Prompts oder Antworten) sowie Plan Credits und Ultimate-Gratisaufrufe aus der lokalen Qoder-Sitzung | | **AnythingLLM Desktop** | ✅ Auto | Passiver SQLite-Reader (`anythingllm-desktop/storage/anythingllm.db`, nur Token-Metriken pro Nachricht) | | **Claude Science** | ✅ Auto | Passiver SQLite-Reader (`~/.claude-science/operon-cli.db`, nur die Token-Zähler der `frames`-Tabelle; keine Prompts, Artefakte oder Forschungsinhalte). Kein natives Windows-Build — unter Windows läuft die App in WSL und wird von dort gelesen. | +| **TRAE Work CN** | ✅ Auto | Liest bei vorhandener lokaler TRAE Work CN-Anmeldung während eines zulässigen Syncs außerhalb des Hintergrundbetriebs unter macOS die Session-Token-Nutzung der angemeldeten App; die interne API kann sich ändern | > **Muss ich Plugins oder Hooks manuell installieren?** Nein. `tokentracker` (oder `tokentracker init`) erledigt alles beim ersten Start: > - **Hook-basiert** (Claude Code, Codex, Gemini, Every Code, CodeBuddy, WorkBuddy, Grok Build) — wir schreiben einen SessionEnd-Hook oder TOML-Notify-Eintrag in die Konfiguration des Tools. @@ -211,11 +212,11 @@ Fehlt dein Tool? [Erstelle ein Issue](https://github.com/xiufengsun/TokenTracker ## 🆚 Warum TokenTracker? -> **Suchst du eine ccusage-Alternative mit GUI?** TokenTracker unterstützt 32 Tools (nicht nur Claude Code), bietet native macOS- und Windows-Apps + Desktop-Widgets und dedupliziert Token-Datensätze korrekt über alle Provider hinweg – damit deine Zahlen mit dem Billing der Provider übereinstimmen. +> **Suchst du eine ccusage-Alternative mit GUI?** TokenTracker unterstützt 33 Tools (nicht nur Claude Code), bietet native macOS- und Windows-Apps + Desktop-Widgets und dedupliziert Token-Datensätze korrekt über alle Provider hinweg – damit deine Zahlen mit dem Billing der Provider übereinstimmen. | | **TokenTracker** | ccusage | Cursor Stats | |---|---|---|---| -| **Unterstützte KI-Tools** | **32** | 1 (Claude) | 1 (Cursor) | +| **Unterstützte KI-Tools** | **33** | 1 (Claude) | 1 (Cursor) | | **Lokal, kein Konto** | ✅ | ✅ | ❌ | | **Native Desktop-App** | ✅ macOS + Windows | ❌ | ❌ | | **Desktop-Widgets** | ✅ 4 Widgets | ❌ | ❌ | @@ -230,7 +231,7 @@ Fehlt dein Tool? [Erstelle ein Issue](https://github.com/xiufengsun/TokenTracker ```mermaid flowchart LR - A["KI-Coding-Tools
Claude · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM"] + A["KI-Coding-Tools
Claude · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM · Claude Science · DeepSeek Harness · TRAE Work CN"] A -->|Hooks lösen aus| B[Token Tracker] B -->|Logs parsen
30-Min-UTC-Buckets| C[(Lokales SQLite)] C --> D[Web-Dashboard] diff --git a/README.ja.md b/README.ja.md index 92ac4f44f..f134dfa0f 100644 --- a/README.ja.md +++ b/README.ja.md @@ -6,7 +6,7 @@ ### AI に使ったコストを正確に把握 — すべての CLI を横断して -**32 種類の AI コーディングツール**からトークン数を自動収集し、ローカルで集計、美しいダッシュボードで本当のコスト推移を可視化します。クラウドアカウント不要、API キー不要、セットアップ不要 — コマンド 1 つで完了です。 +**33 種類の AI コーディングツール**からトークン数を自動収集し、ローカルで集計、美しいダッシュボードで本当のコスト推移を可視化します。クラウドアカウント不要、API キー不要、セットアップ不要 — コマンド 1 つで完了です。 [![npm version](https://img.shields.io/npm/v/tokentracker-cli.svg?color=blue)](https://www.npmjs.com/package/tokentracker-cli) [![npm downloads](https://img.shields.io/npm/dm/tokentracker-cli.svg?color=brightgreen)](https://www.npmjs.com/package/tokentracker-cli) @@ -88,7 +88,7 @@ brew install xiufengsun/tokentracker/tokentracker ## ✨ 機能 -- 🔌 **32 種類の AI ツールを標準対応** — Claude Code、Codex CLI、Cursor、Gemini CLI、Antigravity、Kiro、OpenCode、OpenClaw、Every Code、Hermes Agent、GitHub Copilot、Kimi Code、CodeBuddy、WorkBuddy、Grok Build、oh-my-pi、pi、Prime Agent、Craft Agents、Reasonix、Kilo CLI、Kilo Code、Roo Code、Zed Agent、Goose、Droid、Mimo Code、ZCode、Qoder、AnythingLLM Desktop、Claude Science、DeepSeek Harness +- 🔌 **33 種類の AI ツールを標準対応** — Claude Code、Codex CLI、Cursor、Gemini CLI、Antigravity、Kiro、OpenCode、OpenClaw、Every Code、Hermes Agent、GitHub Copilot、Kimi Code、CodeBuddy、WorkBuddy、Grok Build、oh-my-pi、pi、Prime Agent、Craft Agents、Reasonix、Kilo CLI、Kilo Code、Roo Code、Zed Agent、Goose、Droid、Mimo Code、ZCode、Qoder、AnythingLLM Desktop、Claude Science、DeepSeek Harness、TRAE Work CN - 🏠 **100% ローカル** — トークンデータがマシンから外に出ることはありません。アカウント不要、API キー不要。 - 🚀 **ゼロコンフィグ** — Hook は初回実行で自動インストール。0 からダッシュボードまで 30 秒。 - 📊 **美しいダッシュボード** — 使用トレンド、モデル別コスト内訳、GitHub スタイルのアクティビティヒートマップ、プロジェクト別の帰属表示 @@ -187,6 +187,7 @@ brew install xiufengsun/tokentracker/tokentracker | **pi** (`@mariozechner/pi-coding-agent`) | ✅ 自動 | パッシブリーダー (`~/.pi/agent/sessions/**/*.jsonl`) | | **Prime Agent** | ✅ 自動 | メタデータのみのパッシブ使用量リーダー (`~/.prime/agent/sessions/*.jsonl`) | | **Craft Agents** | ✅ 自動 | パッシブなセッションリーダー (`~/.craft-agent` + workspace session logs) | +| **Reasonix** | ✅ 自動 | パッシブなテレメトリリーダー (`~/.reasonix/**/*.jsonl.telemetry.json`) | | **Roo Code** (VS Code 拡張) | ✅ 自動 | パッシブな `ui_messages.json` リーダー (`rooveterinaryinc.roo-cline`) | | **Zed Agent** | ✅ 自動 | パッシブな SQLite リーダー (`threads.db`、hosted `zed.dev` models only) | | **Goose** (Block) | ✅ 自動 | パッシブな SQLite リーダー (`sessions.db`、cumulative deltas) | @@ -196,11 +197,13 @@ brew install xiufengsun/tokentracker/tokentracker | **Qoder** | ✅ 自動 | パッシブ SQLite リーダー (`Qoder/SharedClientCache/cache/db/local.db`; assistant の `token_info` のみを読み、キャッシュ入力を分離。prompt/response は読みません) と、Qoder ローカルセッションからの Plan Credits / Ultimate 無料呼び出し上限 | | **AnythingLLM Desktop** | ✅ 自動 | パッシブな SQLite リーダー (`anythingllm-desktop/storage/anythingllm.db`、メッセージごとの token 指標のみ) | | **Claude Science** | ✅ 自動 | パッシブな SQLite リーダー (`~/.claude-science/operon-cli.db`、`frames` テーブルの token カウンタのみ。prompt・成果物・研究内容は読みません)。ネイティブ Windows 版はなく、Windows では WSL 内で動作するアプリを読み取ります。 | +| **DeepSeek Harness** | ✅ 自動 | パッシブなセッションリーダー (`~/.dsh/sessions/**/session.jsonl[.zstd]`、セッションヘッダーと assistant イベントを解析し、複数フレームの zstd 展開に対応) | +| **TRAE Work CN** | ✅ 自動 | ローカル TRAE Work CN のサインイン認証がある場合、実行可能な非バックグラウンド同期中に macOS のサインイン済みアプリから session-token 使用量を読み取ります。内部 API は変更される可能性があります | > **プラグインや hook を手動でインストールする必要はありますか?** いいえ。`tokentracker`（または `tokentracker init`）が初回実行ですべて処理します: > - **Hook ベース**のツール (Claude Code、Codex、Gemini、Every Code、**CodeBuddy**、**WorkBuddy**、**Grok Build**) — ツール自身の設定に SessionEnd hook または TOML notify エントリーを書き込みます。 > - **プラグインベース**のツール (OpenCode、**OpenClaw**) — プラグインは npm パッケージ内に同梱されています。OpenClaw のセッションプラグインは `~/.tokentracker/tracker/openclaw-plugin/openclaw-session-sync/` にあり、OpenClaw 自身の CLI でリンクして有効化したうえで、同期を起動するセッション終了イベントを許可するために `hooks.allowConversationAccess=true` を設定します。ダウンロードもドラッグ＆ドロップも不要です。 -> - **パッシブリーダー** (Cursor、Kiro、Hermes、Kimi Code、Copilot、**Grok Build**、**oh-my-pi**、**pi**、**Craft Agents**、**Kilo CLI**、**Kilo Code**、**Roo Code**、**Antigravity**、**Zed Agent**、**Goose**、**Droid**、**Mimo Code**、**ZCode**、**AnythingLLM Desktop**、**Claude Science**) — これらのツールには何もインストールしません。ツールがすでに出力しているファイル (SQLite DB、JSONL、OTEL エクスポート、session logs) を読むだけです。Copilot App / CLI の使用量は `~/.copilot/session-store.db` からリクエスト単位で読み取ります。`data.db` は旧データ移行時のベースラインとして一度だけ使い、store が正規ソースになった後は監視専用です。Chat 拡張と旧 CLI は引き続き OTEL を使用し、重複するリクエストは TokenTracker が一度だけ集計します。移行前の混在 App/CLI 履歴でモデルを安全に分離できない残量は、推測したリクエストモデルではなく `github-copilot-legacy` の集計値として保持します。 +> - **パッシブリーダー** (Cursor、Kiro、Hermes、Kimi Code、Copilot、**Grok Build**、**oh-my-pi**、**pi**、**Craft Agents**、**Reasonix**、**Kilo CLI**、**Kilo Code**、**Roo Code**、**Antigravity**、**Zed Agent**、**Goose**、**Droid**、**Mimo Code**、**ZCode**、**AnythingLLM Desktop**、**Claude Science**、**DeepSeek Harness**) — これらのツールには何もインストールしません。ツールがすでに出力しているファイル (SQLite DB、JSONL、OTEL エクスポート、session logs) を読むだけです。Copilot App / CLI の使用量は `~/.copilot/session-store.db` からリクエスト単位で読み取ります。`data.db` は旧データ移行時のベースラインとして一度だけ使い、store が正規ソースになった後は監視専用です。Chat 拡張と旧 CLI は引き続き OTEL を使用し、重複するリクエストは TokenTracker が一度だけ集計します。移行前の混在 App/CLI 履歴でモデルを安全に分離できない残量は、推測したリクエストモデルではなく `github-copilot-legacy` の集計値として保持します。 > - **Grok Build の推定** — 現在のローカルテレメトリは `updates.jsonl` の累積 `totalTokens` を公開していますが、安定したプロンプト/出力/キャッシュの内訳はありません。`signals.json` は `contextTokensUsed` のスナップショットを使ったフォールバックとして残っています。コールごとの利用詳細が利用可能になるまで、TokenTracker は Grok のコストを推定します。 > > いつでも `tokentracker status` を実行すれば、各統合の状態を確認できます。`skipped` と表示されている場合、`detail` 列にその理由が示されます（例: ツール CLI が `PATH` にない、設定が読めない）。 @@ -215,7 +218,7 @@ brew install xiufengsun/tokentracker/tokentracker | | **TokenTracker** | ccusage | Cursor stats | |--------------------------|:---:|:---:|:---:| -| **対応 AI ツール数** | **32** | 1 (Claude) | 1 (Cursor) | +| **対応 AI ツール数** | **33** | 1 (Claude) | 1 (Cursor) | | **ローカルファースト、アカウント不要** | ✅ | ✅ | ❌ | | **ネイティブデスクトップアプリ** | ✅ macOS + Windows | ❌ | ❌ | | **デスクトップウィジェット** | ✅ 4 種類 | ❌ | ❌ | @@ -227,7 +230,7 @@ brew install xiufengsun/tokentracker/tokentracker ```mermaid flowchart LR - A["AI coding tools
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM"] + A["AI coding tools
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM · Claude Science · DeepSeek Harness · TRAE Work CN"] A -->|hooks trigger| B[Token Tracker] B -->|parse logs
30-min UTC buckets| C[(Local SQLite)] C --> D[Web Dashboard] diff --git a/README.ko.md b/README.ko.md index c275bcb44..9494e63c5 100644 --- a/README.ko.md +++ b/README.ko.md @@ -6,7 +6,7 @@ ### 모든 CLI에서 AI에 쓰는 비용을 정확히 파악 -**32개의 AI 코딩 도구**에서 토큰 수치를 자동으로 수집하고 로컬에서 집계해, 실제 비용 추세를 아름다운 대시보드에서 확인. 클라우드 계정 불필요, API Key 불필요, 셋업 불필요 — 명령 한 줄이면 끝. +**33개의 AI 코딩 도구**에서 토큰 수치를 자동으로 수집하고 로컬에서 집계해, 실제 비용 추세를 아름다운 대시보드에서 확인. 클라우드 계정 불필요, API Key 불필요, 셋업 불필요 — 명령 한 줄이면 끝. [![npm version](https://img.shields.io/npm/v/tokentracker-cli.svg?color=blue)](https://www.npmjs.com/package/tokentracker-cli) [![npm downloads](https://img.shields.io/npm/dm/tokentracker-cli.svg?color=brightgreen)](https://www.npmjs.com/package/tokentracker-cli) @@ -88,7 +88,7 @@ brew install xiufengsun/tokentracker/tokentracker ## ✨ 기능 -- 🔌 **32개의 AI 도구 기본 지원** — Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness +- 🔌 **33개의 AI 도구 기본 지원** — Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness,TRAE Work CN - 🏠 **100% 로컬** — 토큰 데이터가 기기를 떠나지 않습니다. 계정 없음, API Key 없음. - 🚀 **제로 설정** — 첫 실행 시 Hook 자동 설치. 0에서 대시보드까지 30초. - 📊 **아름다운 대시보드** — 사용 추세, 모델별 비용 분석, GitHub 스타일 활동 히트맵, 프로젝트 귀속 정보 @@ -187,6 +187,7 @@ brew install xiufengsun/tokentracker/tokentracker | **pi** (`@mariozechner/pi-coding-agent`) | ✅ 자동 | 패시브 리더 (`~/.pi/agent/sessions/**/*.jsonl`) | | **Prime Agent** | ✅ 자동 | 메타데이터 전용 패시브 사용량 리더 (`~/.prime/agent/sessions/*.jsonl`) | | **Craft Agents** | ✅ 자동 | 패시브 세션 리더 (`~/.craft-agent` + workspace session logs) | +| **Reasonix** | ✅ 자동 | 패시브 텔레메트리 리더 (`~/.reasonix/**/*.jsonl.telemetry.json`) | | **Roo Code** (VS Code 확장) | ✅ 자동 | 패시브 `ui_messages.json` 리더 (`rooveterinaryinc.roo-cline`) | | **Zed Agent** | ✅ 자동 | 패시브 SQLite 리더 (`threads.db`, hosted `zed.dev` models only) | | **Goose** (Block) | ✅ 자동 | 패시브 SQLite 리더 (`sessions.db`, cumulative deltas) | @@ -196,11 +197,13 @@ brew install xiufengsun/tokentracker/tokentracker | **Qoder** | ✅ 자동 | 패시브 SQLite 리더 (`Qoder/SharedClientCache/cache/db/local.db`; assistant `token_info`만 읽고 캐시 입력을 분리하며 prompt/response는 읽지 않음) 및 Qoder 로컬 세션의 Plan Credits / Ultimate 무료 호출 한도 | | **AnythingLLM Desktop** | ✅ 자동 | 패시브 SQLite 리더 (`anythingllm-desktop/storage/anythingllm.db`, 메시지별 token 지표만 읽음) | | **Claude Science** | ✅ 자동 | 패시브 SQLite 리더 (`~/.claude-science/operon-cli.db`, `frames` 테이블의 token 카운터만 읽으며 prompt·산출물·연구 내용은 읽지 않음). 네이티브 Windows 빌드가 없어 Windows에서는 WSL 안에서 실행되는 앱을 읽습니다. | +| **DeepSeek Harness** | ✅ 자동 | 패시브 세션 리더 (`~/.dsh/sessions/**/session.jsonl[.zstd]`, 세션 헤더와 assistant 이벤트를 파싱하고 멀티 프레임 zstd 압축 해제를 지원) | +| **TRAE Work CN** | ✅ 자동 | 로컬 TRAE Work CN 인증이 있을 때 실행 가능한 비백그라운드 동기화 중에 macOS의 로그인된 앱에서 session-token 사용량을 읽습니다. 내부 API는 변경될 수 있습니다 | > **플러그인이나 hook을 수동으로 설치해야 하나요?** 아니요. `tokentracker` (또는 `tokentracker init`)가 첫 실행에서 모든 것을 처리합니다: > - **Hook 기반** 도구 (Claude Code, Codex, Gemini, Every Code, **CodeBuddy**, **WorkBuddy**, **Grok Build**) — 도구 자체의 설정에 SessionEnd hook 또는 TOML notify 엔트리를 작성합니다. > - **플러그인 기반** 도구 (OpenCode, **OpenClaw**) — 플러그인은 npm 패키지 안에 포함되어 있습니다. OpenClaw 세션 플러그인은 `~/.tokentracker/tracker/openclaw-plugin/openclaw-session-sync/`에 있으며, OpenClaw 자체 CLI로 링크하고 활성화한 뒤 동기화를 트리거하는 세션 종료 이벤트를 허용하도록 `hooks.allowConversationAccess=true`를 설정합니다. 다운로드, 드래그 앤 드롭 불필요. -> - **패시브 리더** (Cursor, Kiro, Hermes, Kimi Code, Copilot, **Grok Build**, **oh-my-pi**, **pi**, **Craft Agents**, **Kilo CLI**, **Kilo Code**, **Roo Code**, **Antigravity**, **Zed Agent**, **Goose**, **Droid**, **Mimo Code**, **ZCode**, **AnythingLLM Desktop**, **Claude Science**) — 이들 도구에는 아무것도 설치하지 않습니다. 도구가 이미 생성하는 파일 (SQLite DB, JSONL, OTEL export, session logs)만 읽습니다. Copilot App / CLI 사용량은 `~/.copilot/session-store.db`에서 요청별로 읽습니다. `data.db`는 레거시 마이그레이션 기준선으로 한 번만 사용하며 store가 정식 소스가 된 뒤에는 관찰 전용으로 유지됩니다. Chat 확장과 이전 CLI는 계속 OTEL을 사용하며, TokenTracker가 겹치는 요청을 한 번만 집계합니다. 마이그레이션 전 혼합 App/CLI 기록에서 모델을 안전하게 분리할 수 없는 잔여분은 추정 모델 대신 `github-copilot-legacy` 집계로 보존합니다. +> - **패시브 리더** (Cursor, Kiro, Hermes, Kimi Code, Copilot, **Grok Build**, **oh-my-pi**, **pi**, **Craft Agents**, **Reasonix**, **Kilo CLI**, **Kilo Code**, **Roo Code**, **Antigravity**, **Zed Agent**, **Goose**, **Droid**, **Mimo Code**, **ZCode**, **AnythingLLM Desktop**, **Claude Science**, **DeepSeek Harness**) — 이들 도구에는 아무것도 설치하지 않습니다. 도구가 이미 생성하는 파일 (SQLite DB, JSONL, OTEL export, session logs)만 읽습니다. Copilot App / CLI 사용량은 `~/.copilot/session-store.db`에서 요청별로 읽습니다. `data.db`는 레거시 마이그레이션 기준선으로 한 번만 사용하며 store가 정식 소스가 된 뒤에는 관찰 전용으로 유지됩니다. Chat 확장과 이전 CLI는 계속 OTEL을 사용하며, TokenTracker가 겹치는 요청을 한 번만 집계합니다. 마이그레이션 전 혼합 App/CLI 기록에서 모델을 안전하게 분리할 수 없는 잔여분은 추정 모델 대신 `github-copilot-legacy` 집계로 보존합니다. > - **Grok Build 추정** — 현재 로컬 텔레메트리는 `updates.jsonl`의 누적 `totalTokens`를 노출하지만, 안정적인 프롬프트/출력/캐시 분할은 제공하지 않습니다; `signals.json`은 `contextTokensUsed` 스냅샷을 사용한 폴백으로 남아 있습니다. 호출별 사용 상세 정보가 제공될 때까지 TokenTracker는 Grok 비용을 추정합니다. > > 언제든 `tokentracker status`로 각 통합의 상태를 확인할 수 있습니다. `skipped`로 표시되면 `detail` 컬럼이 이유를 설명합니다 (예: 도구 CLI가 `PATH`에 없음, 설정 읽기 불가). @@ -215,7 +218,7 @@ brew install xiufengsun/tokentracker/tokentracker | | **TokenTracker** | ccusage | Cursor stats | |--------------------------|:---:|:---:|:---:| -| **지원하는 AI 도구 수** | **32** | 1 (Claude) | 1 (Cursor) | +| **지원하는 AI 도구 수** | **33** | 1 (Claude) | 1 (Cursor) | | **로컬 우선, 계정 불필요** | ✅ | ✅ | ❌ | | **네이티브 데스크톱 앱** | ✅ macOS + Windows | ❌ | ❌ | | **데스크톱 위젯** | ✅ 4종 | ❌ | ❌ | @@ -227,7 +230,7 @@ brew install xiufengsun/tokentracker/tokentracker ```mermaid flowchart LR - A["AI coding tools
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM"] + A["AI coding tools
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM · Claude Science · DeepSeek Harness · TRAE Work CN"] A -->|hooks trigger| B[Token Tracker] B -->|parse logs
30-min UTC buckets| C[(Local SQLite)] C --> D[Web Dashboard] diff --git a/README.md b/README.md index bf3255054..9006330b0 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ ### Track every AI token — then bring your usage to life -An accurate, local-first token usage and cost dashboard for **32 AI coding tools** — plus a desktop pet, **4 native widgets**, and **15 achievement tracks**. No cloud account, no API keys, no setup. +An accurate, local-first token usage and cost dashboard for **33 AI coding tools** — plus a desktop pet, **4 native widgets**, and **15 achievement tracks**. No cloud account, no API keys, no setup. [![npm version](https://img.shields.io/npm/v/tokentracker-cli.svg?color=blue)](https://www.npmjs.com/package/tokentracker-cli) [![npm downloads](https://img.shields.io/npm/dm/tokentracker-cli.svg?color=brightgreen)](https://www.npmjs.com/package/tokentracker-cli) @@ -138,7 +138,7 @@ An Arch `PKGBUILD` for a local pacman install lives in `TokenTrackerLinux/packag ## ✨ Features -- 🔌 **32 AI tools out of the box** — Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness +- 🔌 **33 AI tools out of the box** — Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness,TRAE Work CN - 🏠 **100% local** — Token data never leaves your machine. No account, no API keys. - 🚀 **Zero config** — Hooks auto-install on first run. From zero to dashboard in 30 seconds. - 📊 **Beautiful dashboard** — Usage trends, cost breakdowns by model, GitHub-style activity heatmap, project attribution @@ -259,6 +259,7 @@ An Arch `PKGBUILD` for a local pacman install lives in `TokenTrackerLinux/packag | **AnythingLLM Desktop** | ✅ Auto | Passive SQLite reader (`anythingllm-desktop/storage/anythingllm.db`; reads per-message token metrics only, never prompts or responses) | | **Claude Science** | ✅ Auto | Passive SQLite reader (`~/.claude-science/operon-cli.db`; reads the `frames` table's token counters only, never prompts, artifacts or research content). No native Windows build — on Windows the app runs inside WSL and is read from there. | | **DeepSeek Harness** | ✅ Auto | Passive session reader (`~/.dsh/sessions/**/session.jsonl[.zstd]`; parses session header and assistant events, handles multi-frame zstd decompression) | +| **TRAE Work CN** | ✅ Auto | During eligible non-background sync when local TRAE Work CN auth exists, reads session-token usage from the locally signed-in macOS app; its internal API may change | > **Do I need to install any plugin or hook manually?** No. `tokentracker` (or `tokentracker init`) handles everything on first run: > - **Hook-based** tools (Claude Code, Codex, Gemini, Every Code, **CodeBuddy**, **WorkBuddy**, **Grok Build**) — we write a SessionEnd hook or TOML notify entry into the tool's own config. @@ -277,11 +278,11 @@ Missing your tool? [Open an issue](https://github.com/xiufengsun/TokenTracker/is ## 🆚 Why TokenTracker? -> **Looking for a ccusage alternative with a GUI?** TokenTracker covers 32 tools (not just Claude Code), adds native macOS and Windows apps + desktop widgets, and de-duplicates token records correctly across providers — so your numbers match the providers' own billing. +> **Looking for a ccusage alternative with a GUI?** TokenTracker covers 33 tools (not just Claude Code), adds native macOS and Windows apps + desktop widgets, and de-duplicates token records correctly across providers — so your numbers match the providers' own billing. | | **TokenTracker** | ccusage | Cursor stats | |--------------------------|:---:|:---:|:---:| -| **AI tools supported** | **32** | 1 (Claude) | 1 (Cursor) | +| **AI tools supported** | **33** | 1 (Claude) | 1 (Cursor) | | **Local-first, no account** | ✅ | ✅ | ❌ | | **Native desktop app** | ✅ macOS + Windows | ❌ | ❌ | | **Desktop widgets** | ✅ 4 widgets | ❌ | ❌ | @@ -296,7 +297,7 @@ Missing your tool? [Open an issue](https://github.com/xiufengsun/TokenTracker/is ```mermaid flowchart LR - A["AI coding tools
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi Code · CodeBuddy · WorkBuddy · Grok Build · Kilo CLI · Kilo Code
Antigravity · oh-my-pi · pi · Craft · Roo · Zed · Goose · Droid · Mimo · ZCode · Qoder · AnythingLLM · Claude Science"] + A["AI coding tools
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi Code · CodeBuddy · WorkBuddy · Grok Build · Kilo CLI · Kilo Code
Antigravity · oh-my-pi · pi · Craft · Roo · Zed · Goose · Droid · Mimo · ZCode · Qoder · AnythingLLM · Claude Science · DeepSeek Harness · TRAE Work CN"] A -->|hooks trigger| B[Token Tracker] B -->|parse logs
30-min UTC buckets| C[(Local SQLite)] C --> D[Web Dashboard] diff --git a/README.zh-CN.md b/README.zh-CN.md index 777bc8566..6d6f97b6c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,7 +6,7 @@ ### 跨所有 CLI，看清你到底在 AI 上花了多少钱 -自动采集 **32 款 AI 编码工具** 的 token 用量，全程本地聚合，用一套漂亮的 Dashboard 看真实成本与趋势。不需要云账号、不需要 API Key、不需要任何配置 —— 一条命令搞定。 +自动采集 **33 款 AI 编码工具** 的 token 用量，全程本地聚合，用一套漂亮的 Dashboard 看真实成本与趋势。不需要云账号、不需要 API Key、不需要任何配置 —— 一条命令搞定。 [![npm version](https://img.shields.io/npm/v/tokentracker-cli.svg?color=blue)](https://www.npmjs.com/package/tokentracker-cli) [![npm downloads](https://img.shields.io/npm/dm/tokentracker-cli.svg?color=brightgreen)](https://www.npmjs.com/package/tokentracker-cli) @@ -90,7 +90,7 @@ brew install xiufengsun/tokentracker/tokentracker ## ✨ 特性 -- 🔌 **开箱即用支持 32 款 AI 工具** —— Claude Code、Codex CLI、Cursor、Gemini CLI、Kiro、OpenCode、OpenClaw、Every Code、Hermes Agent、GitHub Copilot、Kimi Code、CodeBuddy、WorkBuddy、Grok Build、oh-my-pi、pi、Prime Agent、Craft Agents、Reasonix、Kilo CLI、Kilo Code、Roo Code、Antigravity、Zed Agent、Goose、Droid、Mimo Code、ZCode、Qoder、AnythingLLM Desktop、Claude Science、DeepSeek Harness +- 🔌 **开箱即用支持 33 款 AI 工具** —— Claude Code、Codex CLI、Cursor、Gemini CLI、Kiro、OpenCode、OpenClaw、Every Code、Hermes Agent、GitHub Copilot、Kimi Code、CodeBuddy、WorkBuddy、Grok Build、oh-my-pi、pi、Prime Agent、Craft Agents、Reasonix、Kilo CLI、Kilo Code、Roo Code、Antigravity、Zed Agent、Goose、Droid、Mimo Code、ZCode、Qoder、AnythingLLM Desktop、Claude Science、DeepSeek Harness、TRAE Work CN - 🏠 **100% 本地** —— Token 数据绝不离开你的机器。无账号、无 API Key - 🚀 **零配置** —— 首次运行自动安装所有 hook。30 秒从零到 Dashboard - 📊 **漂亮的 Dashboard** —— 用量趋势、按模型的成本分解、GitHub 风格活跃度热力图、按项目归因 @@ -200,6 +200,7 @@ brew install xiufengsun/tokentracker/tokentracker | **AnythingLLM Desktop** | ✅ 自动 | 被动 SQLite 读取（`anythingllm-desktop/storage/anythingllm.db`；只读取每条消息的 token 指标，不读取 prompt 或回复） | | **Claude Science** | ✅ 自动 | 被动 SQLite 读取（`~/.claude-science/operon-cli.db`；只读取 `frames` 表的 token 计数，不读取 prompt、产物或研究内容）。没有原生 Windows 版——Windows 上该应用运行在 WSL 内，从 WSL 中读取。 | | **DeepSeek Harness** | ✅ 自动 | 被动读取会话日志（`~/.dsh/sessions/**/session.jsonl[.zstd]`；解析会话头部与 assistant 事件，支持多帧 zstd 解压） | +| **TRAE Work CN** | ✅ 自动 | 仅在存在本地 TRAE Work CN 登录授权的可执行非后台同步期间，读取 macOS 本地登录应用的 session-token 用量；内部 API 可能变化 | > **需要手动装什么插件 / hook 吗？** 不需要。`tokentracker`（或 `tokentracker init`）第一次跑的时候会全部搞定： > - **基于 hook 的工具**（Claude Code、Codex、Gemini、Every Code、**CodeBuddy**、**WorkBuddy**、**Grok Build**）—— 我们把 SessionEnd hook 或 TOML notify 条目写入它们自己的配置文件 @@ -220,7 +221,7 @@ brew install xiufengsun/tokentracker/tokentracker | | **TokenTracker** | ccusage | Cursor 自带统计 | |--------------------------|:---:|:---:|:---:| -| **支持的 AI 工具数** | **32** | 1（Claude） | 1（Cursor） | +| **支持的 AI 工具数** | **33** | 1（Claude） | 1（Cursor） | | **本地优先，无需账号** | ✅ | ✅ | ❌ | | **原生桌面 App** | ✅ macOS + Windows | ❌ | ❌ | | **桌面小组件** | ✅ 4 个小组件 | ❌ | ❌ | @@ -232,7 +233,7 @@ brew install xiufengsun/tokentracker/tokentracker ```mermaid flowchart LR - A["AI 编码工具
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM"] + A["AI 编码工具
Claude Code · Codex · Cursor · Gemini · Kiro
OpenCode · OpenClaw · Every Code · Hermes · Copilot
Kimi · CodeBuddy · WorkBuddy · Grok · Kilo · Roo · Zed · Goose
Antigravity · oh-my-pi · pi · Craft · Droid · Mimo · ZCode · Qoder · AnythingLLM · Claude Science · DeepSeek Harness · TRAE Work CN"] A -->|hook 触发| B[Token Tracker] B -->|解析日志
30 分钟 UTC 桶| C[(本地 SQLite)] C --> D[Web Dashboard] diff --git a/dashboard/edge-patches/tokentracker-account-devices.ts b/dashboard/edge-patches/tokentracker-account-devices.ts index 9e7178104..b68a579fb 100644 --- a/dashboard/edge-patches/tokentracker-account-devices.ts +++ b/dashboard/edge-patches/tokentracker-account-devices.ts @@ -85,7 +85,7 @@ interface GroupedRow { // branch intentionally ignores p_device_ids, so a per-device query would add // the user's ENTIRE account-level total to EVERY device (the "N identical // devices" skew). A device breakdown must therefore exclude them. -const ACCOUNT_LEVEL_SOURCES = new Set(["cursor"]); +const ACCOUNT_LEVEL_SOURCES = new Set(["cursor", "trae-cn"]); async function sumDeviceTokens( client: ReturnType, diff --git a/dashboard/edge-patches/tokentracker-ingest.ts b/dashboard/edge-patches/tokentracker-ingest.ts index bd88c1c18..71db980f4 100644 --- a/dashboard/edge-patches/tokentracker-ingest.ts +++ b/dashboard/edge-patches/tokentracker-ingest.ts @@ -105,7 +105,75 @@ export default async function (req: Request): Promise { : Array.isArray(body.hourly) ? body.hourly : []; - if (!Array.isArray(buckets) || buckets.length === 0) { + + // Account session states (kind: "account_session_state" queue records) + // are canonical observations of one provider-side session (trae-cn): + // identity is (user, source, session_id) - device_id is NOT identity, so + // every device of the account converges onto the same row via the LWW + // upsert rpc. Downward / model / bucket corrections are one whole-row + // replace; absence never deletes (contract NOT PROVEN). A states-only + // batch is a meaningful upload and must not fall into the no-buckets + // reject. + const rawStates = Array.isArray(body.account_session_states) ? body.account_session_states : []; + if (rawStates.length > 500) { + return json({ error: "Too many account session states (max 500)" }, 400); + } + const TOKEN_KEYS = [ + "input_tokens", + "output_tokens", + "cached_input_tokens", + "cache_creation_input_tokens", + "reasoning_output_tokens", + "total_tokens", + ] as const; + const states: Record[] = []; + for (const r of rawStates) { + const source = typeof r?.source === "string" ? r.source.trim().toLowerCase() : ""; + const sessionId = typeof r?.session_id === "string" ? r.session_id.trim() : ""; + const model = typeof r?.model === "string" ? r.model.trim() : ""; + const bucketStart = Date.parse(String(r?.bucket_start ?? "")); + const verifiedAt = Date.parse(String(r?.snapshot_verified_at ?? "")); + const tokens: Record = {}; + let tokensOk = true; + for (const k of TOKEN_KEYS) { + const v = Number(r?.[k]); + if (!Number.isSafeInteger(v) || v < 0) tokensOk = false; + else tokens[k] = v; + } + // Fail closed on a malformed state record: a bogus canonical session row + // would corrupt the account's cross-device truth. The parser guarantees + // total = input + cached + cacheCreation + output; enforce it here too. + if ( + source !== "trae-cn" || + !sessionId || + sessionId.length > 256 || + !model || + model.length > 200 || + !Number.isFinite(bucketStart) || + !Number.isFinite(verifiedAt) || + !tokensOk || + tokens.total_tokens !== + tokens.input_tokens + tokens.cached_input_tokens + tokens.cache_creation_input_tokens + tokens.output_tokens + ) { + return json({ error: "Invalid account session state" }, 400); + } + states.push({ + source, + session_id: sessionId, + model, + bucket_start: new Date(bucketStart).toISOString(), + snapshot_verified_at: new Date(verifiedAt).toISOString(), + ...tokens, + }); + } + // Same-session collisions inside one batch would make the LWW upsert's + // ON CONFLICT affect a row twice (Postgres error). Keep the LAST record: + // the append-only CLI queue writes later records with newer stamps. + const stateMap = new Map>(); + for (const r of states) stateMap.set(r.source + "|" + r.session_id, r); + const stateRows = Array.from(stateMap.values()); + + if ((!Array.isArray(buckets) || buckets.length === 0) && stateRows.length === 0) { return json({ error: "No usage buckets provided" }, 400); } if (buckets.length > 500) { @@ -157,5 +225,20 @@ export default async function (req: Request): Promise { if (upsertErr) return json({ error: upsertErr.message }, 500); + if (stateRows.length > 0) { + // Write AFTER the bucket rows of the same upload landed, so canonical + // session state never advances for an upload whose device-level rows + // failed. The rpc applies a STRICTLY-newer LWW guard + // (EXCLUDED.snapshot_verified_at > stored): replays are idempotent and + // a transport retry of an older observation can never displace a newer + // one. Equal-stamp conflicts keep the first-applied row (stable under + // retries). + const { error: stateErr } = await client.database.rpc( + "tokentracker_upsert_account_session_states", + { p_user_id: userId, p_states: stateRows }, + ); + if (stateErr) return json({ error: stateErr.message }, 500); + } + return json({ ok: true, inserted: rows.length, skipped: 0 }); } diff --git a/dashboard/edge-patches/tokentracker-leaderboard-profile.ts b/dashboard/edge-patches/tokentracker-leaderboard-profile.ts index 1fd8d9a1a..b2f5bba61 100644 --- a/dashboard/edge-patches/tokentracker-leaderboard-profile.ts +++ b/dashboard/edge-patches/tokentracker-leaderboard-profile.ts @@ -418,7 +418,7 @@ function canonicalSource(s: string) { // RPC, and tokentracker-leaderboard-refresh.ts // (parity: test/account-source-parity.test.js). // deno-lint-ignore no-unused-vars -const ACCOUNT_LEVEL_SOURCES = new Set(["cursor"]); +const ACCOUNT_LEVEL_SOURCES = new Set(["cursor", "trae-cn"]); // ─────────────────────────── Window bounds ────────────────────────── function normalizeTimeZone(value: string | null): string | null { @@ -532,8 +532,8 @@ async function fetchDailyGroupedRows( ): Promise { // The two-class cross-device aggregation runs in Postgres // (account_usage_grouped), matching tokentracker-leaderboard-refresh.ts: - // * ACCOUNT-LEVEL sources (cursor): ONE canonical whole row per - // (bucket, source, model), across ALL devices (device-independent). + // * ACCOUNT-LEVEL sources (cursor, trae-cn): ONE canonical whole row + // per (bucket, source, model), across ALL devices (device-independent). // * MACHINE-LEVEL sources: summed across the user's ACTIVE devices // (revoked_at IS NULL), dropping historic device churn. // Day grain is enough — the modal only aggregates by day. The cached diff --git a/dashboard/index.html b/dashboard/index.html index 3b51c707c..f9f436718 100644 --- a/dashboard/index.html +++ b/dashboard/index.html @@ -123,7 +123,7 @@ "price": "0", "priceCurrency": "USD" }, - "description": "Local-first AI coding usage tracker for 32 tools, with a desktop pet, four widgets, achievements, native macOS and Windows apps, accurate cross-provider deduplication, and no prompt collection." + "description": "Local-first AI coding usage tracker for 33 tools, with a desktop pet, four widgets, achievements, native macOS and Windows apps, accurate cross-provider deduplication, and no prompt collection." }, { "@type": "WebSite", @@ -196,6 +196,7 @@ { "@type": "ListItem", "position": 30, "name": "AnythingLLM Desktop" }, { "@type": "ListItem", "position": 31, "name": "Claude Science" }, { "@type": "ListItem", "position": 32, "name": "DeepSeek Harness" } + { "@type": "ListItem", "position": 33, "name": "TRAE Work CN" } ] }, { @@ -213,7 +214,7 @@ "@type": "Quest "name": "Which AI coding CLIs does Token Tracker support?", "acceptedAnswer": { "@type": "Answer", - "text": "Token Tracker supports 32 AI coding tools, including Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, and DeepSeek Harness." + "text": "Token Tracker supports 33 AI coding tools, including Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness, and TRAE Work CN." } }, { @@ -245,7 +246,7 @@ "name": "How do I monitor AI token usage and cost across tools?", "acceptedAnswer": { "@type": "Answer", - "text": "Install Token Tracker with npx tokentracker-cli. It is an AI token usage and cost tracker that monitors token consumption and spend across all 32 supported tools in one dashboard, with per-model pricing across 2,200+ models so you can see cost by model, project, and time window without manual tracking." + "text": "Install Token Tracker with npx tokentracker-cli. It is an AI token usage and cost tracker that monitors token consumption and spend across all 33 supported tools in one dashboard, with per-model pricing across 2,200+ models so you can see cost by model, project, and time window without manual tracking." } }, { @@ -267,7 +268,7 @@

Token Tracker: monitor AI token usage and cost with a local dashboard and desktop companion

Token Tracker is a free, open-source, local-first dashboard that automatically tracks and monitors token usage - and cost across 32 AI coding tools — including Claude Code, OpenAI Codex, Cursor, Gemini CLI, and DeepSeek Harness. A desktop pet + and cost across 33 AI coding tools — including Claude Code, OpenAI Codex, Cursor, Gemini CLI, DeepSeek Harness, and TRAE Work CN. A desktop pet reacts to real token activity, four desktop widgets keep usage visible, and achievements turn coding history into progress. Native macOS and Windows apps sit on top of accurate cross-provider deduplication. Token counts only — never prompts or conversations. @@ -292,7 +293,7 @@

What you get after installing

  • Desktop pet — a pixel companion that reacts to real token burn, streaks, and rest.
  • Four desktop widgets — usage, activity heatmap, top models, and limits at a glance.
  • Achievements — unlock milestones from local and optional account-wide activity.
    • -
  • Unified usage dashboard — input, output, cached, and cache-creation tokens across 32 AI coding tools.
    • +
  • Unified usage dashboard — input, output, cached, and cache-creation tokens across 33 AI coding tools.
  • Cost analysis — per-model pricing across 2,200+ models with an offline snapshot.
  • Usage limits view — live quota and subscription status for 13 providers, with last-good caching for local providers such as Antigravity and Qoder.
  • Service Status page — live operational and incident status from eight official provider status pages.
    • @@ -303,11 +304,11 @@

    What you get after installing

    Privacy by design: token counts only, never prompts or conversation content. Local-first; cloud sync is optional and powers only the public leaderboard.

    -

    Supported AI coding tools (32)

    +

    Supported AI coding tools (33)

    Use Token Tracker as a Claude Code token tracker, Codex token tracker, Cursor token tracker, Gemini token tracker, OpenCode token tracker, GitHub Copilot token tracker, or DeepSeek Harness token tracker — one dashboard - covers all 32 tools below. + covers all 33 tools below.

    • Claude Code — Anthropic's official coding agent CLI; integrated via SessionEnd hook.
      • @@ -342,6 +343,7 @@

      Supported AI coding tools (32)

    • AnythingLLM Desktop — passive per-message token metrics reader.
    • Claude Science — passive reader for the frames table of operon-cli.db.
    • DeepSeek Harness — passive session reader (~/.dsh/sessions/**/session.jsonl[.zstd]).
      • +
    • TRAE Work CN — macOS signed-in app session-token reader; its internal API may change.

    Who should use Token Tracker

    @@ -372,7 +374,7 @@

    How do I install Token Tracker?

    Run npx tokentracker-cli. The CLI auto-detects your AI coding tools, installs hooks, and opens the dashboard.

    Which AI coding tools does Token Tracker support?

    -

    Token Tracker supports 32 tools: Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, and DeepSeek Harness.

    +

    Token Tracker supports 33 tools: Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro, OpenCode, OpenClaw, Every Code, Hermes Agent, GitHub Copilot, Kimi Code, CodeBuddy, WorkBuddy, Grok Build, oh-my-pi, pi, Prime Agent, Craft Agents, Reasonix, Kilo CLI, Kilo Code, Roo Code, Zed Agent, Goose, Droid, Mimo Code, ZCode, Qoder, AnythingLLM Desktop, Claude Science, DeepSeek Harness, and TRAE Work CN.

    Is Token Tracker free and open source?

    Yes. Source on GitHub at github.com/xiufengsun/TokenTracker; npm package tokentracker-cli.

    @@ -384,7 +386,7 @@

    What does Token Tracker track?

    Input, output, cached, and cache-creation tokens per model, project, and time window. It also surfaces usage limits for 13 providers, including Claude, Codex, Cursor, Gemini, Antigravity, Qoder, and Qoder CN.

    How do I monitor AI token usage and cost across tools?

    -

    Token Tracker is an AI token usage and cost tracker: install it with npx tokentracker-cli, and it monitors token consumption and spend across all 32 tools in one dashboard, with per-model pricing across 2,200+ models so you can see cost by model, project, and time window without manual tracking.

    00+ models so you can see cost by model, project, and time window without manual tracking.

    +

    Token Tracker is an AI token usage and cost tracker: install it with npx tokentracker-cli, and it monitors token consumption and spend across all 33 tools in one dashboard, with per-model pricing across 2,200+ models so you can see cost by model, project, and time window without manual tracking.

    Does Token Tracker work offline?

    Yes. Token Tracker is local-first. All parsing, aggregation, and dashboard rendering happen on your machine. Cloud sync is optional and powers only the public leaderboard.

    diff --git a/dashboard/public/llms.txt b/dashboard/public/llms.txt index b699bdc06..1512691a3 100644 --- a/dashboard/public/llms.txt +++ b/dashboard/public/llms.txt @@ -23,7 +23,7 @@ timestamps, never prompts, responses, or source code. - [npm package](https://www.npmjs.com/package/tokentracker-cli) - [Latest release](https://github.com/xiufengsun/TokenTracker/releases/latest) -## Supported AI coding tools (32) +## Supported AI coding tools (33) - Claude Code - Codex CLI - Cursor @@ -56,6 +56,7 @@ timestamps, never prompts, responses, or source code. - AnythingLLM Desktop - Claude Science - DeepSeek Harness +- TRAE Work CN ## Capabilities - Desktop pet that reacts to real token burn, streaks, and rest diff --git a/dashboard/src/content/copy.csv b/dashboard/src/content/copy.csv index c81eb924d..553564283 100644 --- a/dashboard/src/content/copy.csv +++ b/dashboard/src/content/copy.csv @@ -767,6 +767,7 @@ limits.provider.antigravity,ui,LimitsPage,UsageLimitsPanel,provider_antigravity, limits.provider.zcode,ui,LimitsPage,UsageLimitsPanel,provider_zcode,"ZCode",,active provider.display.omp,ui,*,ProviderDisplay,omp,"oh-my-pi",,active provider.display.deepseek_harness,ui,*,ProviderDisplay,deepseek_harness,"DeepSeek Harness",,active +provider.display.trae_work_cn,ui,*,ProviderDisplay,trae_work_cn,"TRAE Work CN",,active limits.label.zcode_glm52,ui,LimitsPage,UsageLimitsPanel,zcode_glm52,"GLM-5.2",,active limits.label.zcode_glm5t,ui,LimitsPage,UsageLimitsPanel,zcode_glm5t,"GLM-5-Turbo",,active limits.label.zcode_5h,ui,LimitsPage,UsageLimitsPanel,zcode_5h,"5h",,active diff --git a/dashboard/src/content/i18n/zh-TW/core.json b/dashboard/src/content/i18n/zh-TW/core.json index d124cd94e..421749270 100644 --- a/dashboard/src/content/i18n/zh-TW/core.json +++ b/dashboard/src/content/i18n/zh-TW/core.json @@ -154,6 +154,7 @@ "limits.provider.copilot": "GitHub Copilot", "provider.display.omp": "oh-my-pi", "provider.display.deepseek_harness": "DeepSeek Harness", + "provider.display.trae_work_cn": "TRAE Work 中國版", "limits.provider.antigravity": "Antigravity", "limits.provider.zcode": "ZCode", "limits.provider.opencode_go": "OpenCode Go", diff --git a/dashboard/src/content/i18n/zh/core.json b/dashboard/src/content/i18n/zh/core.json index 5288d4c0f..ad5a2124e 100644 --- a/dashboard/src/content/i18n/zh/core.json +++ b/dashboard/src/content/i18n/zh/core.json @@ -154,6 +154,7 @@ "limits.provider.copilot": "GitHub Copilot", "provider.display.omp": "oh-my-pi", "provider.display.deepseek_harness": "DeepSeek Harness", + "provider.display.trae_work_cn": "TRAE Work 中国版", "limits.provider.antigravity": "Antigravity", "limits.provider.zcode": "ZCode", "limits.provider.opencode_go": "OpenCode Go", diff --git a/dashboard/src/lib/provider-display.js b/dashboard/src/lib/provider-display.js index cf574ffdf..f6fae3db1 100644 --- a/dashboard/src/lib/provider-display.js +++ b/dashboard/src/lib/provider-display.js @@ -12,6 +12,8 @@ const SPECIAL_PROVIDER_COPY_KEYS = { deepseek: "provider.display.deepseek_harness", dsh: "provider.display.deepseek_harness", omp: "provider.display.omp", + traecn: "provider.display.trae_work_cn", + traeworkcn: "provider.display.trae_work_cn", }; function normalizedProviderKey(value) { diff --git a/dashboard/src/lib/provider-display.test.js b/dashboard/src/lib/provider-display.test.js index b57250dc6..b5e3ab43e 100644 --- a/dashboard/src/lib/provider-display.test.js +++ b/dashboard/src/lib/provider-display.test.js @@ -35,4 +35,11 @@ describe("formatProviderDisplayName", () => { expect(formatProviderDisplayName("dsh")).toBe("DeepSeek Harness"); expect(formatProviderDisplayName("deepseek")).toBe("DeepSeek Harness"); }); + + it.each(["trae-cn", "TRAE-CN", "Trae_Cn", "TRAE Work CN"])( + "uses the catalog-backed TRAE Work CN name for %s", + (value) => { + expect(formatProviderDisplayName(value)).toBe("TRAE Work CN"); + }, + ); }); diff --git a/dashboard/src/ui/dashboard/components/ProviderIcon.jsx b/dashboard/src/ui/dashboard/components/ProviderIcon.jsx index 2af28c221..3d335ad92 100644 --- a/dashboard/src/ui/dashboard/components/ProviderIcon.jsx +++ b/dashboard/src/ui/dashboard/components/ProviderIcon.jsx @@ -117,6 +117,63 @@ function CodeBuddyIcon({ size = 16, className = "" }) { ); } +// TRAE Work CN — official app-logo mark reverse-engineered from +// https://work.trae.cn/ favicon (48×48) and PWA manifest icon-512.png +// (pixel-accurate geometry, 512×512 canvas). +// +// Structure (confirmed against 512px source RGBA raster): +// • White rounded square (#FFFFFF, corners transparent so the circular +// alpha mask reads correctly on any browser tab chrome). +// • Pure black (#000000) bracket-frame glyph over it: +// – a full-width top crossbar, +// – two 36px-thick vertical arms (left arm floats; right arm anchors +// the bottom crossbar which therefore sits 36px inset on the left), +// – two spindle accents midway inside the arms (rhombic ellipses, +// rx≈24 ry≈22). +// +// The white plate + solid black ink matches the favicon pixel-for-pixel and +// therefore stays consistent with the brand assets served from work.trae.cn. +function TraeCnIcon({ size = 16, className = "" }) { + return ( + + ); +} + function HermesIcon({ size = 16, className = "" }) { return ( @@ -313,6 +370,7 @@ const PROVIDER_ICON_MAP = { "PI-GITHUB-COPILOT": PiIcon, "PI-COPILOT": PiIcon, ROOCODE: RoocodeIcon, + "TRAE-CN": TraeCnIcon, ZED: ZedIcon, }; diff --git a/dashboard/src/ui/dashboard/components/ProviderIcon.test.jsx b/dashboard/src/ui/dashboard/components/ProviderIcon.test.jsx index 22adc68ab..f2a493316 100644 --- a/dashboard/src/ui/dashboard/components/ProviderIcon.test.jsx +++ b/dashboard/src/ui/dashboard/components/ProviderIcon.test.jsx @@ -77,6 +77,22 @@ describe("ProviderIcon", () => { expect(icon?.querySelector('path[fill="currentColor"]')).not.toBeNull(); } }); + + it("renders the compact TRAE CN mark instead of the unknown-provider placeholder", () => { + const { container } = render(); + const icon = container.querySelector('svg[data-brand="trae-cn"]'); + + expect(icon).not.toBeNull(); + expect(icon).toHaveAttribute("width", "20"); + expect(icon).toHaveAttribute("height", "20"); + expect(icon).toHaveAttribute("aria-hidden", "true"); + // Official app-icon palette: white rounded square + black mask, fixed + // colors instead of currentColor (same policy as the Craft mark). + expect(icon?.querySelector("rect")).not.toBeNull(); + expect(icon?.querySelector("path")).not.toBeNull(); + expect(icon?.querySelector("circle")).toBeNull(); + expect(container.querySelector(".text-oai-gray-400")).toBeNull(); + }); }); it("renders the Qoder CN green-crescent mark from its own asset", () => { diff --git a/dashboard/src/ui/marketing/LogoCarousel.jsx b/dashboard/src/ui/marketing/LogoCarousel.jsx index 5f50e46a1..a902867c6 100644 --- a/dashboard/src/ui/marketing/LogoCarousel.jsx +++ b/dashboard/src/ui/marketing/LogoCarousel.jsx @@ -1,5 +1,7 @@ import React, { useCallback, useEffect, useMemo, useState } from "react"; import { AnimatePresence, motion } from "motion/react"; +import { copy } from "../../lib/copy"; +import { useLocale } from "../../hooks/useLocale.js"; import { ProviderIcon } from "../dashboard/components/ProviderIcon.jsx"; // Animated logo wall, adapted from cult-ui's LogoCarousel @@ -39,6 +41,7 @@ function distributeLogos(allLogos, columnCount) { } const LogoColumn = React.memo(function LogoColumn({ logos, index, currentTime, onHoverChange }) { + useLocale(); const [hovered, setHovered] = useState(false); const cycleInterval = 4000; // ms each logo stays visible const columnDelay = index * 400; // stagger columns so they flip out of sync @@ -102,7 +105,7 @@ const LogoColumn = React.memo(function LogoColumn({ logos, index, currentTime, o exit={{ opacity: 0, y: -4 }} transition={{ duration: 0.15 }} > - {logo.name} + {logo.nameKey ? copy(logo.nameKey) : logo.name} )} diff --git a/dashboard/src/ui/marketing/LogoCarousel.test.jsx b/dashboard/src/ui/marketing/LogoCarousel.test.jsx new file mode 100644 index 000000000..e92d73101 --- /dev/null +++ b/dashboard/src/ui/marketing/LogoCarousel.test.jsx @@ -0,0 +1,54 @@ +import { fireEvent, render, screen, waitFor } from "@testing-library/react"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { useLocale } from "../../hooks/useLocale.js"; +import { EN_LOCALE, LOCALE_STORAGE_KEY, ZH_CN_LOCALE } from "../../lib/locale"; +import { LocaleProvider } from "../foundation/LocaleProvider.jsx"; +import { LogoCarousel } from "./LogoCarousel.jsx"; + +const TRAE_LOGO = { + id: 32, + name: "TRAE Work CN", + nameKey: "provider.display.trae_work_cn", + provider: "trae-cn", +}; + +function LocaleProbe() { + const { setLocale } = useLocale(); + + return ( + + ); +} + +beforeEach(() => { + window.localStorage.setItem(LOCALE_STORAGE_KEY, EN_LOCALE); + vi.spyOn(globalThis, "setInterval").mockImplementation(() => 0); +}); + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe("LogoCarousel localization", () => { + it("resolves keyed logo names using the current locale at render time", async () => { + const { container } = render( + + + + , + ); + + let icon; + await waitFor(() => { + icon = container.querySelector('svg[data-brand="trae-cn"]'); + expect(icon).not.toBeNull(); + }); + fireEvent.mouseEnter(icon); + expect(await screen.findByText("TRAE Work CN")).toBeInTheDocument(); + + fireEvent.click(screen.getByRole("button", { name: "switch-to-zh" })); + expect(await screen.findByText("TRAE Work 中国版")).toBeInTheDocument(); + }); +}); diff --git a/dashboard/src/ui/marketing/agent-logos.js b/dashboard/src/ui/marketing/agent-logos.js index f6f734c7c..e6eefcc5a 100644 --- a/dashboard/src/ui/marketing/agent-logos.js +++ b/dashboard/src/ui/marketing/agent-logos.js @@ -1,6 +1,7 @@ // Canonical list of supported coding agents, in display order. Shared by the // marketing landing carousel and the dashboard auth/expired gate so the two // surfaces can't drift. Names are tooltip/a11y metadata only. + export const AGENT_LOGOS = [ { id: 1, name: "Claude Code", provider: "claude" }, { id: 2, name: "Codex", provider: "codex" }, @@ -34,4 +35,12 @@ export const AGENT_LOGOS = [ { id: 30, name: "AnythingLLM", provider: "anythingllm" }, { id: 31, name: "Claude Science", provider: "claude-science" }, { id: 32, name: "DeepSeek Harness", provider: "dsh" }, + { + id: 33, + // No hardcoded English fallback name: every consumer renders through + // copy() (LogoCarousel prefers nameKey; see LogoCarousel.test.jsx), so a + // parallel "name" string would just duplicate the copy.csv entry. + nameKey: "provider.display.trae_work_cn", + provider: "trae-cn", + }, ]; diff --git a/docs/PRIVACY.md b/docs/PRIVACY.md index 01a372298..ecc068c10 100644 --- a/docs/PRIVACY.md +++ b/docs/PRIVACY.md @@ -10,14 +10,14 @@ This document lists **every** network request the software can make, what each o ## 1. What never leaves your machine -TokenTracker's parsers extract numbers and timestamps only. +TokenTracker's parsers extract numbers and timestamps only. TRAE Work CN is a narrow exception to the local-only source model: during an eligible non-background sync, when local TRAE Work CN auth exists, TokenTracker transmits the existing sign-in authorization from the locally signed-in app to TRAE's internal API for a read-only usage request. It is not an unconditional or default generic network request. That authorization is never persisted or logged. -**Never read, never recorded anywhere:** +**Apart from the disclosed TRAE Work CN authorization, never read or recorded anywhere:** - **Prompts, responses, and conversation bodies** - **File contents** from your projects - **Commit messages and diffs** — Git attribution runs `git log` locally, uses the subject line only to detect reverts, and keeps nothing -- **API keys, cookies, and session tokens** belonging to your AI providers +- **API keys, cookies, and session tokens** belonging to your AI providers are never persisted or logged by TokenTracker **Recorded locally, never uploaded:** @@ -54,6 +54,7 @@ To erase everything TokenTracker knows about you, delete that directory. `tokent | **Anonymous heartbeat** | `srctyff5.us-east.insforge.app` | A one-way hash of the machine id, app version, OS platform, and shell (`cli` / `mac` / `win`). Nothing else. | At most once per day | | **Dashboard analytics** | `us.i.posthog.com` (PostHog) | Pageviews and explicitly instrumented feature events, plus which shell you use. Autocapture and session recording are **off**; browser Do-Not-Track is respected. | While the dashboard is open | | **Provider quota reads** | The provider's own API (`api.anthropic.com`, `chatgpt.com`, `cursor.com`, `api.github.com`, `api.kimi.com`, `api.z.ai`, `qoder.com`, `qoder.com.cn`, `openapi.qoder.sh`, `openapi.qoder.com.cn`, `cloudcode-pa.googleapis.com`, …) | Whatever that provider's own endpoint requires, authenticated with the credentials **that provider already stored on your machine**. These requests go directly from your machine to the provider — they never pass through our servers, and we never see the response. | While quota bars are visible | +| **TRAE Work CN usage read** | TRAE's internal API | Transmits the existing sign-in authorization from the locally signed-in TRAE Work CN app to TRAE; reads usage metadata only. TokenTracker never persists or logs the auth token or prompt/response content. | During eligible non-background sync when local TRAE Work CN auth exists | | **GitHub star count** | `api.github.com` | Nothing but the request itself (public repo metadata) | On dashboard load | | **Update check** | `api.github.com` | Nothing but the request itself | Windows: once at launch. macOS: only when you click "Check for Updates" | diff --git a/docs/screenshots/trae-cn-sync-verification.png b/docs/screenshots/trae-cn-sync-verification.png new file mode 100644 index 000000000..7ddd55e27 Binary files /dev/null and b/docs/screenshots/trae-cn-sync-verification.png differ diff --git a/migrations/20260817120000_account-session-states.sql b/migrations/20260817120000_account-session-states.sql new file mode 100644 index 000000000..fed4aed69 --- /dev/null +++ b/migrations/20260817120000_account-session-states.sql @@ -0,0 +1,409 @@ +-- Account-level sources serve CORRECTABLE account snapshots, not append-only +-- machine logs: TRAE CN (trae-cn) can revise a session's totals downward, +-- move it to another model, or shift its half-hour bucket. Deduplicating +-- those corrections at the HOUR level cannot work: a fresh device's snapshot +-- starts at its first observed session, so an hour-level owner can never +-- safely displace a stale tuple for an hour the fresh snapshot has no data +-- for (that would require an unproven "absent = deleted" contract), and a +-- bucket migration (10:00 -> 10:30) leaves the old hour stranded. +-- +-- Fix: canonical truth at the SESSION level. TRAE's usage API carries a +-- session_id whose observed stability splits into (evidence 2026-08-17, +-- one account, three real fetches 137 -> 141 -> 164 sessions): +-- repeated-fetch stability VERIFIED (137/137 persisted across two +-- independent fetches, 0 disappearances; +-- one session revised upward mid-window +-- KEPT its session_id) +-- cross-window stability VERIFIED (window-subset queries return +-- exact id subsets) +-- no duplicate ids OBSERVED (all fetched responses) +-- cross-device same-account NOT DIRECTLY VERIFIED: the request body +-- carries no device discriminator, but that +-- is necessary, not sufficient - a device +-- or login context could ride inside the +-- JWT / server auth context. No second +-- independent device/auth experiment was +-- run. If it were ever DISPROVEN (same +-- logical session, different ids per +-- device), this PK would split one logical +-- session into competing rows and the +-- identity must be re-evaluated. +-- The CLI parser (src/lib/rollout.js) emits one queue record per CHANGED +-- session (kind: "account_session_state"); the ingest edge upserts them +-- here via tokentracker_upsert_account_session_states(). Identity is +-- (user_id, source, session_id) - device_id is NOT part of the identity +-- (the API request carries no device discriminator; see the stability +-- split above for what that does and does not prove). +-- +-- Three corrections become ONE whole-row replace: +-- downward S tokens 100 -> 60 row replaced, total 60 +-- model S model A -> B row replaced, only B=100 remains +-- bucket S bucket 10:00 -> 10:30 row replaced, only 10:30=100 +-- A fresh device with no cursor history simply uploads the sessions it sees; +-- the upsert's LWW guard reconciles versions across devices. +-- +-- Absence is NOT PROVEN to mean deletion (see the parser's empty-payload +-- note): nothing here ever deletes a session row. Corrections are explicit +-- observations only. +-- +-- Freshness: the API exposes NO provider-side ordering signal (headers carry +-- only CDN trace ids; rows have no revision/updated_at - probed 2026-08-17). +-- snapshot_verified_at is the CLIENT fetch stamp, set once per real fetch +-- and replayed verbatim by the append-only queue. The upsert applies an +-- observation only when its stamp is STRICTLY NEWER (>) than the stored +-- one, so a transport retry of an old observation is a no-op, and the same +-- observation replayed is idempotent. This is best-effort cross-device +-- ordering, NOT strict correctness: client clock skew can mis-order two +-- conflicting observations of one session, and a sufficiently +-- future-skewed client timestamp can delay later corrections (residual +-- risk; there is no mechanism bounding the skew). +-- +-- Semantics mirrored by src/lib/account-usage-dedup.js and pinned by +-- test/account-usage-dedup.test.js. Apply BEFORE re-deploying the updated +-- scripts/ops/account-usage-grouped-rpc.sql and the ingest edge (both read / +-- write this table). +-- Rollback: DROP TABLE tokentracker_account_session_states; DROP FUNCTION +-- tokentracker_upsert_account_session_states(uuid, jsonb); and re-apply the +-- previous leaderboard_hourly_dedup_v2 definition. + +CREATE TABLE IF NOT EXISTS public.tokentracker_account_session_states ( + user_id uuid NOT NULL, + source text NOT NULL, + -- Provider-side session identity (TRAE session_id): stable across + -- repeated fetches and window changes (VERIFIED), account-scoped rather + -- than device-scoped in the request (cross-device id stability itself is + -- NOT DIRECTLY VERIFIED - see the header's evidence split). + session_id text NOT NULL, + model text NOT NULL, + -- Canonical UTC half-hour bucket of this session's CURRENT placement. + bucket_start timestamptz NOT NULL, + input_tokens bigint NOT NULL, + output_tokens bigint NOT NULL, + cached_input_tokens bigint NOT NULL, + cache_creation_input_tokens bigint NOT NULL, + reasoning_output_tokens bigint NOT NULL, + total_tokens bigint NOT NULL, + -- Client logical fetch stamp (see the freshness note above). Ops metadata + -- only - updated_at (first server write time) never participates in + -- correctness. + snapshot_verified_at timestamptz NOT NULL, + updated_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (user_id, source, session_id), + CHECK ( + total_tokens = + input_tokens + cached_input_tokens + cache_creation_input_tokens + output_tokens + ), + CHECK ( + input_tokens >= 0 AND output_tokens >= 0 AND cached_input_tokens >= 0 + AND cache_creation_input_tokens >= 0 AND reasoning_output_tokens >= 0 + AND total_tokens >= 0 + ) +); + +ALTER TABLE public.tokentracker_account_session_states ENABLE ROW LEVEL SECURITY; +REVOKE ALL ON public.tokentracker_account_session_states FROM PUBLIC, anon, authenticated; +GRANT ALL ON public.tokentracker_account_session_states TO project_admin; + +-- Aggregation reads group by (user, source, bucket). +CREATE INDEX IF NOT EXISTS tokentracker_account_session_states_bucket_idx + ON public.tokentracker_account_session_states (user_id, source, bucket_start); + +-- Batch LWW upsert used by the ingest edge (the SDK's upsert cannot express +-- a conditional DO UPDATE). SECURITY DEFINER with a locked search_path; the +-- edge calls it with the service-role token after authenticating the device. +-- A strict ">" guard makes replay idempotent and prevents a transport retry +-- of an older observation from displacing a newer one; conflicting +-- observations with EQUAL stamps keep the first-applied row (stable under +-- retries). +CREATE OR REPLACE FUNCTION public.tokentracker_upsert_account_session_states( + p_user_id uuid, + p_states jsonb +) RETURNS integer +LANGUAGE plpgsql +SECURITY DEFINER SET search_path = public +AS $fn$ +DECLARE + v_applied integer; +BEGIN + INSERT INTO tokentracker_account_session_states AS t + (user_id, source, session_id, model, bucket_start, + input_tokens, output_tokens, cached_input_tokens, + cache_creation_input_tokens, reasoning_output_tokens, total_tokens, + snapshot_verified_at) + SELECT + p_user_id, x.source, x.session_id, x.model, x.bucket_start, + x.input_tokens, x.output_tokens, x.cached_input_tokens, + x.cache_creation_input_tokens, x.reasoning_output_tokens, x.total_tokens, + x.snapshot_verified_at + FROM jsonb_to_recordset(p_states) AS x( + source text, session_id text, model text, bucket_start timestamptz, + input_tokens bigint, output_tokens bigint, cached_input_tokens bigint, + cache_creation_input_tokens bigint, reasoning_output_tokens bigint, + total_tokens bigint, snapshot_verified_at timestamptz) + ON CONFLICT (user_id, source, session_id) DO UPDATE SET + model = EXCLUDED.model, + bucket_start = EXCLUDED.bucket_start, + input_tokens = EXCLUDED.input_tokens, + output_tokens = EXCLUDED.output_tokens, + cached_input_tokens = EXCLUDED.cached_input_tokens, + cache_creation_input_tokens = EXCLUDED.cache_creation_input_tokens, + reasoning_output_tokens = EXCLUDED.reasoning_output_tokens, + total_tokens = EXCLUDED.total_tokens, + snapshot_verified_at = EXCLUDED.snapshot_verified_at, + updated_at = now() + WHERE EXCLUDED.snapshot_verified_at > t.snapshot_verified_at; + + GET DIAGNOSTICS v_applied = ROW_COUNT; + RETURN v_applied; +END +$fn$; + +REVOKE ALL ON FUNCTION public.tokentracker_upsert_account_session_states(uuid, jsonb) + FROM PUBLIC, anon, authenticated; +GRANT EXECUTE ON FUNCTION public.tokentracker_upsert_account_session_states(uuid, jsonb) + TO project_admin; + +-- Leaderboard: account-level sources now aggregate from SESSION STATES (the +-- canonical cross-device truth), machine-level sources keep the cluster +-- dedup. trae-cn hourly rows (per-device local reconciliations) never enter +-- the leaderboard - the session-state branch replaces both the previous +-- watermark-owner branch and the legacy MAX fallback for trae-cn. 'cursor' +-- (identical rows across devices, no session identity) keeps the legacy +-- whole-row MAX dedup. Historical rollup days self-heal through the cyclic +-- repair in leaderboard_rollup_daily_advance_v2 (superseded below): 7 days +-- per scheduled run (total refresh, every ~6h) from the OLDEST history day, +-- wrapping. FIRST SEEDS are prioritized: when a closed day has trae-cn +-- session states but no trae-cn rollup row yet, the repair window jumps +-- there, so a new account's first ~30-day seed heals in +-- ceil(seed_span / 7) scheduled runs (~30h at the 6h cadence) instead of a +-- full cycle. Corrections to already-covered days (stale values, row exists) +-- keep the ordinary cyclic schedule - their lag scales with TOTAL history +-- length, not the correction's age. Live paths (account_usage_grouped, +-- bounded leaderboard windows) reflect everything immediately. An immediate +-- full rebuild is optional for instant parity (call +-- leaderboard_rollup_daily_replace_v2 for the affected range). +CREATE OR REPLACE FUNCTION public.leaderboard_hourly_dedup_v2( + p_from timestamptz, + p_to timestamptz +) RETURNS TABLE ( + user_id uuid, + source text, + model text, + hour_start timestamptz, + total_tokens bigint, + input_tokens bigint, + output_tokens bigint, + cached_input_tokens bigint, + cache_creation_input_tokens bigint, + reasoning_output_tokens bigint +) +LANGUAGE sql STABLE +AS $func$ + WITH cfg AS ( + SELECT ARRAY['cursor', 'trae-cn']::text[] AS account_sources + ) + -- Deduplicate device-id drift/replays inside one physical machine cluster, + -- then add genuinely distinct machines for the same user/hour/model. + SELECT mac.user_id, mac.source, mac.model, mac.hour_start, + SUM(mac.total_tokens)::bigint AS total_tokens, + SUM(mac.input_tokens)::bigint AS input_tokens, + SUM(mac.output_tokens)::bigint AS output_tokens, + SUM(mac.cached_input_tokens)::bigint AS cached_input_tokens, + SUM(mac.cache_creation_input_tokens)::bigint AS cache_creation_input_tokens, + SUM(mac.reasoning_output_tokens)::bigint AS reasoning_output_tokens + FROM ( + SELECT DISTINCT ON ( + h.user_id, + COALESCE(dm.machine_cluster_id, h.device_id::text), + h.source, + h.model, + h.hour_start + ) + h.user_id, + COALESCE(dm.machine_cluster_id, h.device_id::text) AS machine_cluster_id, + h.source, h.model, h.hour_start, + h.total_tokens::bigint AS total_tokens, + h.input_tokens::bigint AS input_tokens, + h.output_tokens::bigint AS output_tokens, + h.cached_input_tokens::bigint AS cached_input_tokens, + h.cache_creation_input_tokens::bigint AS cache_creation_input_tokens, + h.reasoning_output_tokens::bigint AS reasoning_output_tokens + FROM tokentracker_hourly h + CROSS JOIN cfg + JOIN tokentracker_devices d + ON d.id = h.device_id AND d.revoked_at IS NULL + LEFT JOIN tokentracker_device_machine dm + ON dm.device_id = h.device_id + WHERE h.hour_start >= p_from AND h.hour_start < p_to + AND NOT (h.source = ANY(cfg.account_sources)) + ORDER BY + h.user_id, + COALESCE(dm.machine_cluster_id, h.device_id::text), + h.source, + h.model, + h.hour_start, + h.total_tokens DESC, + h.updated_at DESC + ) mac + GROUP BY mac.user_id, mac.source, mac.model, mac.hour_start + + UNION ALL + + -- 'cursor' (account-level but with NO stable session identity): rows are + -- identical across devices, so the legacy whole-row MAX pick per + -- (user, hour, source, model) dedups them. + SELECT acct.user_id, acct.source, acct.model, acct.hour_start, + acct.total_tokens, acct.input_tokens, acct.output_tokens, + acct.cached_input_tokens, acct.cache_creation_input_tokens, + acct.reasoning_output_tokens + FROM ( + SELECT DISTINCT ON (h.user_id, h.source, h.model, h.hour_start) + h.user_id, h.source, h.model, h.hour_start, + h.total_tokens::bigint AS total_tokens, + h.input_tokens::bigint AS input_tokens, + h.output_tokens::bigint AS output_tokens, + h.cached_input_tokens::bigint AS cached_input_tokens, + h.cache_creation_input_tokens::bigint AS cache_creation_input_tokens, + h.reasoning_output_tokens::bigint AS reasoning_output_tokens + FROM tokentracker_hourly h + WHERE h.hour_start >= p_from AND h.hour_start < p_to + AND h.source = 'cursor' + ORDER BY h.user_id, h.source, h.model, h.hour_start, h.total_tokens DESC, h.updated_at DESC + ) acct + + UNION ALL + + -- trae-cn: canonical account truth aggregated from session states. Every + -- device's observations of the same session collapsed to ONE row by the + -- LWW upsert; corrections (downward / model / bucket) are already + -- reflected because each session exists exactly once. + SELECT s.user_id, s.source, s.model, s.bucket_start AS hour_start, + SUM(s.total_tokens)::bigint AS total_tokens, + SUM(s.input_tokens)::bigint AS input_tokens, + SUM(s.output_tokens)::bigint AS output_tokens, + SUM(s.cached_input_tokens)::bigint AS cached_input_tokens, + SUM(s.cache_creation_input_tokens)::bigint AS cache_creation_input_tokens, + SUM(s.reasoning_output_tokens)::bigint AS reasoning_output_tokens + FROM tokentracker_account_session_states s + WHERE s.bucket_start >= p_from AND s.bucket_start < p_to + AND s.source = 'trae-cn' + GROUP BY s.user_id, s.source, s.model, s.bucket_start +$func$; + +-- --------------------------------------------------------------------------- +-- leaderboard_rollup_daily_advance_v2: supersede the 20260804043427 definition +-- with FIRST-SEED PRIORITIZATION for trae-cn. Everything else (7-day catch-up +-- bootstrap, cyclic 7-day repair from the oldest history day) is unchanged. +-- +-- Why: a new account's first TRAE sync seeds ~30 CLOSED days of session +-- states in one upload. Live paths (account_usage_grouped, bounded +-- leaderboard windows) read session states directly and see the seed +-- immediately, but the leaderboard TOTAL reads the MATERIALIZED rollup for +-- closed days - which has never covered them. The plain cyclic repair would +-- reach the seed only after up to a FULL cycle (history_days / 7 scheduled +-- runs), leaving the TOTAL undercounted versus account/profile for days. +-- The fix is a repositioning only: when the earliest closed day that HAS +-- trae-cn session states but NO trae-cn rollup row exists, the repair window +-- JUMPS there. The seed range then rebuilds deterministically at 7 days per +-- scheduled run (ceil(seed_span / 7) runs; a 30-day seed = 5 runs ~= 30h at +-- the ~6h total-refresh cadence). Corrections to already-covered days are +-- NOT gaps (their rollup row exists, merely stale) and keep the ordinary +-- cyclic schedule; per-run work stays bounded by the same 7-day chunk, so +-- the memory/statement-time budget that motivated the cyclic design is +-- unchanged. Mirrored by the createRollupSim harness and pinned by scenario +-- D in test/leaderboard-rollup-correction.test.js. +CREATE OR REPLACE FUNCTION public.leaderboard_rollup_daily_advance_v2() +RETURNS void +LANGUAGE plpgsql +SET work_mem TO '16MB' +SET hash_mem_multiplier TO '2' +SET statement_timeout TO '25s' +AS $func$ +DECLARE + v_from timestamptz; + v_target timestamptz := date_trunc('day', now() AT TIME ZONE 'UTC') AT TIME ZONE 'UTC'; + v_until timestamptz; + v_min_day date; + v_repair_from date; + v_seed_gap_day date; +BEGIN + SELECT + (date_trunc('day', MIN(h.hour_start) AT TIME ZONE 'UTC') AT TIME ZONE 'UTC')::date + INTO v_min_day + FROM public.tokentracker_hourly h; + v_min_day := COALESCE(v_min_day, (v_target AT TIME ZONE 'UTC')::date); + + SELECT m.through, m.repair_from INTO v_from, v_repair_from + FROM public.tokentracker_leaderboard_rollup_meta_v2 m + WHERE m.id = 1 + FOR UPDATE; + + IF v_from IS NULL THEN + v_from := v_min_day::timestamp AT TIME ZONE 'UTC'; + v_repair_from := v_min_day; + INSERT INTO public.tokentracker_leaderboard_rollup_meta_v2 ( + id, through, repair_from, rebuilt_at + ) VALUES (1, v_from, v_repair_from, now()); + END IF; + + IF v_from < v_target THEN + -- Bootstrap/catch-up: advance at most seven closed days per request. + v_until := LEAST(v_target, v_from + interval '7 days'); + PERFORM public.leaderboard_rollup_daily_replace_v2(v_from, v_until); + UPDATE public.tokentracker_leaderboard_rollup_meta_v2 + SET through = v_until, + rebuilt_at = now() + WHERE id = 1; + RETURN; + END IF; + + -- Once caught up, continuously repair seven historical days per scheduled + -- total refresh. Late history uploads, device revocations, and cluster-map + -- changes therefore self-heal without another whole-history memory spike. + IF v_repair_from >= (v_target AT TIME ZONE 'UTC')::date THEN + v_repair_from := v_min_day; + END IF; + + -- First-seed / uncovered-day prioritization (trae-cn): jump the repair + -- window to the earliest CLOSED day that has trae-cn session states but + -- no trae-cn rollup row yet (see the function header comment). Coverage + -- is PER USER (the rollup PK starts at user_id): another user's same-day + -- row does NOT cover this user's seed. A day whose rollup row exists for + -- that user (even with stale values) is NOT a gap - corrections keep the + -- ordinary cyclic schedule. + SELECT MIN((s.bucket_start AT TIME ZONE 'UTC')::date) INTO v_seed_gap_day + FROM public.tokentracker_account_session_states s + WHERE s.source = 'trae-cn' + AND (s.bucket_start AT TIME ZONE 'UTC')::date < (v_target AT TIME ZONE 'UTC')::date + AND NOT EXISTS ( + SELECT 1 FROM public.tokentracker_leaderboard_rollup_daily_v2 r + WHERE r.user_id = s.user_id + AND r.source = 'trae-cn' + AND r.day = (s.bucket_start AT TIME ZONE 'UTC')::date + ); + IF v_seed_gap_day IS NOT NULL THEN + v_repair_from := v_seed_gap_day; + END IF; + + v_until := LEAST( + v_target, + (v_repair_from::timestamp AT TIME ZONE 'UTC') + interval '7 days' + ); + PERFORM public.leaderboard_rollup_daily_replace_v2( + v_repair_from::timestamp AT TIME ZONE 'UTC', + v_until + ); + UPDATE public.tokentracker_leaderboard_rollup_meta_v2 + SET repair_from = CASE + WHEN v_until >= v_target THEN v_min_day + ELSE (v_until AT TIME ZONE 'UTC')::date + END, + rebuilt_at = now() + WHERE id = 1; +END +$func$; + +REVOKE ALL ON FUNCTION public.leaderboard_rollup_daily_advance_v2() + FROM PUBLIC, anon, authenticated; +GRANT EXECUTE ON FUNCTION public.leaderboard_rollup_daily_advance_v2() + TO project_admin; diff --git a/package.json b/package.json index 3a4dc2649..d2b89f1f3 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "tokentracker-cli", "version": "0.91.1", - "description": "Local-first AI coding token usage and cost tracker for 32 tools, with a dashboard, desktop pet, widgets, achievements, and native macOS/Windows apps.", + "description": "Local-first AI coding token usage and cost tracker for 33 tools, with a dashboard, desktop pet, widgets, achievements, and native macOS/Windows apps.", "main": "src/cli.js", "bin": { "tokentracker-cli": "bin/tracker.js", diff --git a/scripts/ops/account-usage-grouped-rpc.sql b/scripts/ops/account-usage-grouped-rpc.sql index 7cc741e97..42d6453f9 100644 --- a/scripts/ops/account-usage-grouped-rpc.sql +++ b/scripts/ops/account-usage-grouped-rpc.sql @@ -25,12 +25,26 @@ -- is immune to that. The cost: two genuinely distinct machines running the -- SAME model in the SAME half-hour count once, not summed — rare, and far -- better than the systemic ~2x that fold-then-SUM produced. --- * ACCOUNT-LEVEL sources (cursor) come from a per-ACCOUNT cloud API, NOT --- machine logs. Every device that syncs them stores an IDENTICAL copy, so --- SUMming across devices multiplies one account's usage by its device --- count (the v0.42.0 bug: a 2-machine user's Cursor total was double). For --- these, pick ONE canonical row per (hour, source, model) across ALL the --- user's devices — dedup, do not add. +-- * ACCOUNT-LEVEL sources (cursor, trae-cn) come from a per-ACCOUNT cloud +-- API, NOT machine logs, so SUMming across devices multiplies one +-- account's usage by its device count (the v0.42.0 double-count bug). +-- Dedup, do not add. HOW to dedup depends on the source's identity: +-- - cursor rows are identical across devices (no session identity): +-- the legacy whole-row per-(hour, model) MAX pick below. +-- - trae-cn is a CORRECTABLE snapshot: totals can be revised down, a +-- session can move to another model or another half-hour. Hour-level +-- dedup cannot express that (a fresh device's first data bucket +-- cannot safely displace earlier hours), so trae-cn aggregates from +-- tokentracker_account_session_states - ONE canonical row per +-- (user, source, session_id), whole-row-replaced on every change by +-- the LWW upsert (tokentracker_upsert_account_session_states). +-- Downward / model / bucket corrections all collapse into that one +-- replace; cross-device duplication collapses because device_id is +-- NOT part of the identity. Absence never deletes (contract NOT +-- PROVEN). Mirrored by src/lib/account-usage-dedup.js (executable +-- spec; semantics pinned by test/account-usage-dedup.test.js). +-- DEPLOYMENT: this function reads tokentracker_account_session_states — +-- apply the account-session-states migration BEFORE this updated RPC. -- The account-level source list MUST stay in sync with ACCOUNT_LEVEL_SOURCES in -- src/lib/source-metadata.js (parity asserted by test/account-source-parity.test.js). -- @@ -90,7 +104,7 @@ AS $func$ ), -- Account-level source list — keep in sync with src/lib/source-metadata.js. cfg AS ( - SELECT ARRAY['cursor']::text[] AS account_sources + SELECT ARRAY['cursor', 'trae-cn']::text[] AS account_sources ), -- Stage 1: canonicalize to the raw hour grain. hourly AS ( @@ -161,13 +175,14 @@ AS $func$ UNION ALL - -- Account-level: ONE canonical whole row per (hour, source, model) across - -- ALL devices (NOT active-filtered — the data is device-independent and an - -- active-only filter would drop it if last synced by a since-revoked one). - SELECT acct.hour_start, acct.source, acct.model, - acct.total_tokens, acct.input_tokens, acct.output_tokens, - acct.cached_input_tokens, acct.cache_creation_input_tokens, - acct.reasoning_output_tokens, acct.conversations + -- 'cursor' (account-level, no session identity): rows are identical + -- across devices, so the legacy whole-row MAX pick per + -- (hour, source, model) dedups them. NOT active-device-filtered: + -- account truth follows the account, not which device last synced. + SELECT d.hour_start, d.source, d.model, + d.total_tokens, d.input_tokens, d.output_tokens, + d.cached_input_tokens, d.cache_creation_input_tokens, + d.reasoning_output_tokens, d.conversations FROM ( SELECT DISTINCT ON (h.hour_start, h.source, h.model) h.hour_start, h.source, h.model, @@ -178,13 +193,32 @@ AS $func$ h.cache_creation_input_tokens::bigint AS cache_creation_input_tokens, h.reasoning_output_tokens::bigint AS reasoning_output_tokens, h.conversations::bigint AS conversations - FROM tokentracker_hourly h CROSS JOIN cfg + FROM tokentracker_hourly h WHERE h.user_id = p_user_id AND h.hour_start >= p_from AND h.hour_start < p_to - AND h.source = ANY(cfg.account_sources) + AND h.source = 'cursor' ORDER BY h.hour_start, h.source, h.model, h.total_tokens DESC, h.updated_at DESC - ) acct + ) d + + UNION ALL + + -- trae-cn: canonical account truth from session states (one row per + -- session after the LWW upsert). NOT active-device-filtered. + SELECT s.bucket_start AS hour_start, s.source, s.model, + SUM(s.total_tokens)::bigint AS total_tokens, + SUM(s.input_tokens)::bigint AS input_tokens, + SUM(s.output_tokens)::bigint AS output_tokens, + SUM(s.cached_input_tokens)::bigint AS cached_input_tokens, + SUM(s.cache_creation_input_tokens)::bigint AS cache_creation_input_tokens, + SUM(s.reasoning_output_tokens)::bigint AS reasoning_output_tokens, + COUNT(*)::bigint AS conversations + FROM tokentracker_account_session_states s + WHERE s.user_id = p_user_id + AND s.bucket_start >= p_from + AND s.bucket_start < p_to + AND s.source = 'trae-cn' + GROUP BY s.bucket_start, s.source, s.model ), -- Stage 2: bucket the canonical hour rows to tz-local trunc, then aggregate. loc AS ( diff --git a/scripts/ops/ui-hardcode-baseline.json b/scripts/ops/ui-hardcode-baseline.json index 7c9cc42e2..73288eaca 100644 --- a/scripts/ops/ui-hardcode-baseline.json +++ b/scripts/ops/ui-hardcode-baseline.json @@ -1,12 +1,12 @@ { - "generatedAt": "2026-08-11T09:06:06.707Z", + "generatedAt": "2026-08-18T02:42:39.974Z", "root": "dashboard/src", "rules": { "colors": "hex/rgb(a) literals", "rawText": "JSX text nodes containing letters or digits" }, "totals": { - "colors": 377, + "colors": 384, "rawText": 161 }, "files": { @@ -370,7 +370,7 @@ ] }, "dashboard/src/ui/dashboard/components/ProviderIcon.jsx": { - "colors": 2, + "colors": 9, "rawText": 0, "rawTextTokens": [] }, diff --git a/src/commands/init.js b/src/commands/init.js index 1ef3a6391..451b23021 100644 --- a/src/commands/init.js +++ b/src/commands/init.js @@ -103,7 +103,7 @@ const DIVIDER = "----------------------------------------------"; const DEFAULT_DASHBOARD_URL = "https://www.tokentracker.cc"; // Single source of truth for the welcome screen's provider count + sample list. -// test/discovery-metadata.test.js keeps this aligned with public 27-tool copy. +// test/discovery-metadata.test.js keeps this aligned with public 32-tool copy. const SUPPORTED_PROVIDERS = [ "Claude Code", "Codex CLI", @@ -137,6 +137,7 @@ const SUPPORTED_PROVIDERS = [ "AnythingLLM Desktop", "Claude Science", "DeepSeek Harness", + "TRAE Work CN", ]; async function cmdInit(argv) { diff --git a/src/commands/status.js b/src/commands/status.js index 9488490ca..cc5b248ee 100644 --- a/src/commands/status.js +++ b/src/commands/status.js @@ -89,6 +89,13 @@ const { resolveCopilotAppDbPaths, probeWslDistros, } = require("../lib/rollout"); +const { + TRAE_CN_USAGE_ENV, + isTraeCnUsageEnabled, + resolveTraeCnStoragePath, + readTraeCnAuthFromStorage, + extractTraeCnToken, +} = require("../lib/trae-cn-config"); const wsl = require("../lib/wsl-probe"); const { getWslMode, isInvalidWslMode, shouldProbeWsl, discoverWslHome } = wsl; const { resolveInstallPaths, resolveZcodeNativeDbPath } = require("../lib/install-resolver"); @@ -642,6 +649,34 @@ async function cmdStatus(argv = []) { ? readTraeEntitlementFromStorage(traeStoragePath) : null; + // Trae SOLO CN — opt-in usage-API reader. Unlike the passive entitlement + // reader above, this source sends the locally stored sign-in JWT to TRAE's + // official API, so status surfaces the opt-in flag, the resolved storage + // path, and whether the auth blob decrypts — the three inputs a "why is my + // TRAE CN data missing" diagnosis needs. + // A resolvable default path does NOT mean the app is installed - the macOS + // resolver always derives one. Match the sync path semantics exactly: + // installed iff the storage file actually exists on disk. + const traeCnStoragePath = resolveTraeCnStoragePath({ env: process.env, home }); + const traeCnInstalled = Boolean( + traeCnStoragePath && fssync.existsSync(traeCnStoragePath), + ); + let traeCnAuthState = "not-signed-in"; + if (traeCnInstalled) { + try { + const traeCnAuth = readTraeCnAuthFromStorage({ env: process.env, home, platform: process.platform }); + if (traeCnAuth) { + extractTraeCnToken(traeCnAuth); + traeCnAuthState = "readable"; + } + } catch (error) { + // Distinguish a real IO failure (unreadable) from present-but-bad + // data (malformed); neither ever carries storage contents. + traeCnAuthState = error?.code === "TRAE_CN_STORAGE_UNREADABLE" ? "unreadable" : "malformed"; + } + } + const traeCnUsageOptIn = isTraeCnUsageEnabled(process.env); + // Grok Build (xAI TUI) const grokHookState = await probeGrokHookState({ home, trackerDir, env: process.env }); const grokSessions = grokHookState.hasGrokInstall || grokHookState.sessionsDir @@ -933,6 +968,14 @@ async function cmdStatus(argv = []) { ...(traeEntitlement ? { entitlement: traeEntitlement } : {}), } : { installed: false }, + "trae-cn": traeCnInstalled + ? { + installed: true, + detail: traeCnStoragePath, + auth: traeCnAuthState, + usage_opt_in: traeCnUsageOptIn, + } + : { installed: false }, grok_build: grokInstalled ? { installed: true, @@ -1105,6 +1148,9 @@ async function cmdStatus(argv = []) { traeEntitlement ? `- Trae SOLO plan: ${formatTraeEntitlementLine(traeEntitlement)}` : null, + traeCnInstalled + ? `- Trae SOLO CN: usage sync ${traeCnUsageOptIn ? "opted in" : `off (set ${TRAE_CN_USAGE_ENV}=1 to enable)`}, auth ${traeCnAuthState} (${traeCnStoragePath})` + : null, ...(() => { if (!hermesInstalled) return []; return [`- Hermes Agent: state.db found (${hermesPath})`]; diff --git a/src/commands/sync.js b/src/commands/sync.js index f41933645..3e43a4798 100644 --- a/src/commands/sync.js +++ b/src/commands/sync.js @@ -112,6 +112,7 @@ const { resolveDroidModel, resolveDshSessionFiles, parseDshIncremental, + parseTraeCnApiIncremental, bucketKey, toUtcHalfHourStart, totalsKey, @@ -134,6 +135,11 @@ const { parseCursorCsv, } = require("../lib/cursor-config"); const { purgeProjectUsage } = require("../lib/project-usage-purge"); +const { + fetchTraeCnUsageWithAuth, + resolveTraeCnStoragePath, + isTraeCnUsageEnabled, +} = require("../lib/trae-cn-config"); const { isCodexSessionCursorPath, isCursorStoreRetry, @@ -291,6 +297,7 @@ const AUTO_SYNC_SOURCES = new Set([ "qoder-cn", "reasonix", "roocode", + "trae-cn", "workbuddy", "zcode", "zed", @@ -452,6 +459,15 @@ async function cmdSync(argv, context = {}) { const lockWaitOptions = context && typeof context === "object" ? context.lockWaitOptions : undefined; + // Narrow test-only seam for TRAE Work CN: deterministic injected fetch + a + // fixed "now" for the rolling range. Production keeps the global fetch and + // current time; no public CLI flag/config is introduced. + const traeCnFetchImpl = context && typeof context === "object" + ? context.traeCnFetchImpl + : undefined; + const traeCnNowMs = context && typeof context === "object" + ? context.traeCnNowMs + : undefined; const syncDiagnostics = diagnostics && typeof diagnostics === "object" ? diagnostics : null; const home = os.homedir(); const { trackerDir } = await resolveTrackerPaths({ home }); @@ -1663,6 +1679,93 @@ async function cmdSync(argv, context = {}) { } } + // ── Trae Work CN (国内版) — account-level usage API ── + // A simple explicit rolling 30-day window captured once per sync (no + // cursor checkpoints / full-history import / page retries / background + // requests). Runs only when the user has opted in via + // TOKENTRACKER_TRAE_CN_USAGE=1 (the read transmits the locally stored + // sign-in JWT to TRAE's official endpoint, so it is never default-on), + // the sync is non-lightweight, the source is allowed, and the CN storage + // file exists — which excludes both ordinary background and `--auto + // --background --all-local-sources`. Fetching goes through the + // storage-backed helper so its single 401/403 reread/retry is used; an + // empty response is a successful no-op; each window is bounded to 100 + // pages/2,000 rows and an over-capacity window is split into staggered + // sub-windows (still over capacity at the finest allowed granularity, or + // any other fetch/page/schema/auth failure) skips the parser and leaves + // prior data untouched while unrelated providers continue. + let traeCnResult = { recordsProcessed: 0, eventsAggregated: 0, bucketsQueued: 0 }; + if ( + !isBackgroundLightweightSync && + isTraeCnUsageEnabled(process.env) && + sourceAllowed("trae-cn") + ) { + const nowMs = Number.isFinite(traeCnNowMs) ? traeCnNowMs : Date.now(); + const fetchImpl = typeof traeCnFetchImpl === "function" ? traeCnFetchImpl : fetch; + const endTime = Math.floor(nowMs / 1000); + // Align the REAL fetch start down to a half-hour boundary so the API + // range fully contains every bucket it can touch (a raw 08:37 start + // would only partially cover the 08:30 bucket) - session bucket floors + // never straddle the queried range. + const HALF_HOUR_SEC = 30 * 60; + const startTime = Math.max( + 1, + Math.floor((endTime - 30 * 24 * 60 * 60) / HALF_HOUR_SEC) * HALF_HOUR_SEC, + ); + try { + // Absent storage means "not signed in" — a silent skip, not an error. + const traeCnStoragePath = resolveTraeCnStoragePath({ env: process.env, home }); + if (traeCnStoragePath && fssync.existsSync(traeCnStoragePath)) { + if (progress?.enabled) { + progress.start(`Fetching TRAE Work CN usage...`); + } + const traeCnUsage = await fetchTraeCnUsageWithAuth({ + start_time: startTime, + end_time: endTime, + fetchImpl, + env: process.env, + home, + }); + const traeCnSessions = Array.isArray(traeCnUsage?.sessions) ? traeCnUsage.sessions : []; + // An empty payload parses as a pure no-op: the TRAE absence + // contract is NOT PROVEN (no evidence that a missing session means + // deleted/zero), so an empty response asserts nothing - no usage + // mutation and no session states. Non-empty snapshots append one + // canonical session-state observation per CHANGED session (the + // cloud LWW upsert reconciles cross-device versions). + { + if (progress?.enabled && traeCnSessions.length > 0) { + progress.start( + `Parsing TRAE Work CN ${renderBar(0)} 0/${formatNumber( + traeCnSessions.length, + )} records | buckets 0`, + ); + } + traeCnResult = await parseTraeCnApiIncremental({ + sessions: traeCnSessions, + cursors, + queuePath, + onProgress: makeProviderProgress("TRAE Work CN"), + windowStartMs: startTime * 1000, + windowEndMs: endTime * 1000, + // The LOGICAL fetch stamp carried into every session state's + // snapshot_verified_at: the same clock that produced the query + // range, not the later enqueue moment (append-only queue + // replays it verbatim, so transport retries never fake + // freshness). + snapshotVerifiedAtMs: nowMs, + }); + // A partially malformed snapshot throws inside the parser (fail + // closed - it must not become the authoritative window state); + // warnProviderParseFailure below reports it without sensitive + // data while unrelated providers continue. + } + } + } catch (err) { + warnProviderParseFailure("TRAE Work CN", err, opts); + } + } + // ── Kiro (SQLite-based, with JSONL fallback; dual-install aware) ── let kiroResult = { recordsProcessed: 0, eventsAggregated: 0, bucketsQueued: 0 }; if (sourceAllowed("kiro")) { @@ -2584,6 +2687,7 @@ async function cmdSync(argv, context = {}) { qoderCnResult.recordsProcessed + claudeScienceResult.recordsProcessed + cursorResult.recordsProcessed + + traeCnResult.recordsProcessed + kiroResult.recordsProcessed + kiroCliResult.recordsProcessed + hermesResult.recordsProcessed + @@ -2619,6 +2723,7 @@ async function cmdSync(argv, context = {}) { qoderCnResult.bucketsQueued + claudeScienceResult.bucketsQueued + cursorResult.bucketsQueued + + traeCnResult.bucketsQueued + kiroResult.bucketsQueued + kiroCliResult.bucketsQueued + hermesResult.bucketsQueued + @@ -2972,6 +3077,8 @@ function recordCodexColdScanAudit( module.exports = { cmdSync, acquireSyncLock, + readQueueBatch, + drainQueueToCloud, migrateCursorUnknownBuckets, migrateRolloutCumulativeDeltaBuckets, repairCodebuddyLogJsonlOverlap, @@ -3472,7 +3579,10 @@ async function drainQueueToCloud({ baseUrl, deviceToken, queuePath, queueStatePa for (let batch = 0; batch < maxBatches; batch++) { if (offset >= queueSize) break; const result = await readQueueBatch(queuePath, offset, limit); - if (result.buckets.length === 0) break; + // A states-only tail (e.g. a retry batch whose bucket rows already + // uploaded) is still a meaningful upload: it advances the canonical + // session corrections cloud-side. + if (result.buckets.length === 0 && result.sessionStates.length === 0) break; const root = baseUrl.replace(/\/$/, ""); const anonKey = process.env.TOKENTRACKER_INSFORGE_ANON_KEY || ""; @@ -3485,7 +3595,12 @@ async function drainQueueToCloud({ baseUrl, deviceToken, queuePath, queueStatePa const res = await fetch(`${root}/functions/${INGEST_SLUG}`, { method: "POST", headers, - body: JSON.stringify({ hourly: result.buckets }), + body: JSON.stringify({ + hourly: result.buckets, + ...(result.sessionStates.length > 0 + ? { account_session_states: result.sessionStates } + : {}), + }), }); const rawText = await res.text().catch(() => ""); @@ -3522,6 +3637,26 @@ async function readQueueBatch(queuePath, startOffset, maxBuckets) { const rl = readline.createInterface({ input: stream, crlfDelay: Infinity }); const bucketMap = new Map(); + // Account session states (kind: "account_session_state") are queue + // control records, not usage rows: collect them separately so they ride + // the same upload (after the bucket rows in file order) without entering + // the bucket stream. Keyed by (source, session_id) LAST-WINS: the + // append-only queue writes a session's newer observation after its older + // one, and only the latest per session may reach the ingest batch (a + // duplicate would make the cloud ON CONFLICT affect one row twice). + // snapshot_verified_at rides along verbatim from the ORIGINAL fetch, so a + // transport retry never fakes logical freshness. + // + // Batching: session-state records count toward the SAME per-batch record + // cap (maxBuckets) as bucket rows. The ingest edge rejects a request with + // more than 500 states (HTTP 400); before they counted, a states-heavy + // queue (a fresh device's first 30-day TRAE sync appends one record per + // observed session) was read in ONE uncapped batch - the 400 left the + // queue offset untouched and every retry re-read the identical oversized + // request, a permanent upload failure. With the default batchSize of 200 + // (and the MAX_INGEST_BUCKETS=500 clamp above), every request stays under + // the edge cap while batching follows the existing design. + const sessionStateMap = new Map(); let offset = startOffset; let linesRead = 0; for await (const line of rl) { @@ -3534,6 +3669,37 @@ async function readQueueBatch(queuePath, startOffset, maxBuckets) { } catch (_e) { continue; } + if (bucket?.kind === "account_session_state") { + if (typeof bucket.source === "string" && typeof bucket.session_id === "string" && bucket.session_id.trim()) { + const ssSource = bucket.source.trim().toLowerCase(); + sessionStateMap.set(ssSource + "|" + bucket.session_id.trim(), { + source: ssSource, + session_id: bucket.session_id.trim(), + model: typeof bucket.model === "string" ? bucket.model : "", + bucket_start: typeof bucket.bucket_start === "string" ? bucket.bucket_start : "", + input_tokens: Number(bucket.input_tokens) || 0, + output_tokens: Number(bucket.output_tokens) || 0, + cached_input_tokens: Number(bucket.cached_input_tokens) || 0, + cache_creation_input_tokens: Number(bucket.cache_creation_input_tokens) || 0, + reasoning_output_tokens: Number(bucket.reasoning_output_tokens) || 0, + total_tokens: Number(bucket.total_tokens) || 0, + ...(typeof bucket.snapshot_verified_at === "string" && bucket.snapshot_verified_at + ? { snapshot_verified_at: bucket.snapshot_verified_at } + : {}), + }); + // Counted like a bucket row: the record rides THIS batch and its + // bytes are covered by nextOffset, so the next batch resumes right + // after it. + linesRead += 1; + if (linesRead >= maxBuckets) break; + } + continue; + } + // Pre-release watermark records from this feature branch's dev queues + // are obsolete control records: drop them instead of letting them enter + // the bucket stream (they carry no hour_start and would be skipped + // downstream anyway, but be explicit). + if (bucket?.kind === "account_sync_watermark") continue; const hourStart = typeof bucket?.hour_start === "string" ? bucket.hour_start : null; if (!hourStart) continue; const source = (typeof bucket?.source === "string" ? bucket.source.trim().toLowerCase() : "") || "codex"; @@ -3553,7 +3719,11 @@ async function readQueueBatch(queuePath, startOffset, maxBuckets) { rl.close(); stream.close?.(); - return { buckets: Array.from(bucketMap.values()), nextOffset: offset }; + return { + buckets: Array.from(bucketMap.values()), + sessionStates: Array.from(sessionStateMap.values()), + nextOffset: offset, + }; } function normalizeGrokRepairSource(value) { diff --git a/src/lib/account-usage-dedup.js b/src/lib/account-usage-dedup.js new file mode 100644 index 000000000..35e4d6c2d --- /dev/null +++ b/src/lib/account-usage-dedup.js @@ -0,0 +1,267 @@ +"use strict"; + +/** + * Account-level cross-device truth - JS reference implementation. + * + * Account-level sources (ACCOUNT_LEVEL_SOURCES in src/lib/source-metadata.js) + * come from a per-ACCOUNT cloud API, so every device that syncs them stores + * its own copy of the SAME account data. How to dedup depends on the source: + * + * - 'cursor' has NO stable session identity: its rows are identical across + * devices, so the legacy whole-row MAX pick per (hour, source, model) + * dedups them (dedupeAccountLevelRows below). + * - 'trae-cn' is a CORRECTABLE snapshot: totals can be revised down, a + * session can move to another model or another half-hour. Hour-level + * dedup CANNOT express that (a fresh device's first data bucket cannot + * safely displace earlier hours - that would need an unproven "absent = + * deleted" contract - and a 10:00 -> 10:30 bucket migration strands the + * old hour). Canonical truth for trae-cn therefore lives at the SESSION + * level (tokentracker_account_session_states; migrations/ + * 20260817120000_account-session-states.sql): identity is + * (user_id, source, session_id) - device_id is NOT identity (the usage + * API request carries no device discriminator). Session-id evidence + * (2026-08-17): repeated-fetch stability VERIFIED, cross-window + * stability VERIFIED; cross-device same-account stability NOT DIRECTLY + * VERIFIED (no second independent device/auth experiment). Every + * observation of one + * session whole-row-replaces the previous state under a strict LWW + * guard, so the three correction classes collapse into ONE operation: + * + * downward S tokens 100 -> 60 replaced, total 60 + * model S model A -> B replaced, only B remains + * bucket S bucket 10:00 -> 10:30 replaced, only 10:30 remains + * + * ABSENCE is NOT PROVEN to mean deletion, so nothing ever deletes a session + * row: a session missing from a non-empty snapshot asserts nothing, and an + * empty snapshot asserts nothing at all. + * + * Freshness: the TRAE API exposes no provider-side ordering signal (probed + * 2026-08-17: headers carry only CDN trace ids, rows carry no revision), so + * snapshot_verified_at is the CLIENT logical fetch stamp (best-effort + * cross-device ordering under clock skew - documented residual risk, NOT + * strict correctness). It is stamped once per real fetch and replayed + * verbatim by the append-only queue; the LWW guard applies strictly newer + * (>) stamps only, so retries are idempotent and a re-delivered older + * observation cannot displace a newer one. Equal stamps keep the + * first-applied row (stable under retries). + * + * This module is the executable specification of that semantics; the + * deployed SQL (migrations/20260817120000_account-session-states.sql upsert, + * scripts/ops/account-usage-grouped-rpc.sql + leaderboard_hourly_dedup_v2 + * aggregation branches) implements the same algorithm and MUST stay in sync + * - test/account-usage-dedup.test.js pins both. + */ + +// Queue record kind the TRAE CN parser emits for one changed session's +// canonical observation (src/lib/rollout.js appendTraeCnSessionStates). +const SESSION_STATE_KIND = "account_session_state"; + +function parseMs(value) { + const ms = Date.parse(value); + return Number.isFinite(ms) ? ms : null; +} + +// Token columns that make up a canonical session state row (the SQL table's +// non-key, non-metadata columns). +const TOKEN_COLUMNS = [ + "input_tokens", + "output_tokens", + "cached_input_tokens", + "cache_creation_input_tokens", + "reasoning_output_tokens", + "total_tokens", +]; + +function normalizeStateRow(observation) { + const source = String(observation?.source ?? "").trim().toLowerCase(); + const sessionId = String(observation?.session_id ?? "").trim(); + if (!source || !sessionId) return null; + const verifiedMs = parseMs(observation?.snapshot_verified_at); + if (verifiedMs === null) return null; + const bucketStart = String(observation?.bucket_start ?? ""); + if (parseMs(bucketStart) === null) return null; + const model = String(observation?.model ?? ""); + if (!model) return null; + const row = { + source, + session_id: sessionId, + model, + bucket_start: bucketStart, + snapshot_verified_at: observation.snapshot_verified_at, + }; + for (const column of TOKEN_COLUMNS) { + const value = Number(observation?.[column]); + row[column] = Number.isFinite(value) ? value : 0; + } + return row; +} + +/** + * Whole-row LWW replace of canonical session states - mirrors + * tokentracker_upsert_account_session_states() (migrations/ + * 20260817120000_account-session-states.sql): a row applies only when its + * snapshot_verified_at is STRICTLY newer (>) than the stored one. Replays of + * the same observation are no-ops; equal stamps keep the first-applied row. + * + * @param {Map} states - canonical table keyed by + * `${source}|${session_id}` (mirrors the (user_id, source, session_id) PK + * for one account; mutated in place). + * @param {Array} observations - ingest batch (already deduped to one + * row per session by the edge, like the SQL batch upsert requires). + * @returns {number} applied row count (mirrors the SQL ROW_COUNT return). + */ +function upsertAccountSessionStates(states, observations) { + const table = states instanceof Map ? states : new Map(); + const batch = Array.isArray(observations) ? observations : []; + let applied = 0; + for (const observation of batch) { + const row = normalizeStateRow(observation); + if (!row) continue; + const key = `${row.source}|${row.session_id}`; + const stored = table.get(key); + if (!stored) { + table.set(key, row); + applied += 1; + continue; + } + const storedMs = parseMs(stored.snapshot_verified_at) ?? -Infinity; + const rowMs = parseMs(row.snapshot_verified_at) ?? -Infinity; + if (rowMs > storedMs) { + table.set(key, row); + applied += 1; + } + } + return applied; +} + +/** + * Aggregate canonical session states into hourly rows - mirrors the trae-cn + * branch of account_usage_grouped (scripts/ops/account-usage-grouped-rpc.sql) + * and leaderboard_hourly_dedup_v2: SUM per (bucket_start, source, model), + * conversations = session count. + * + * @param {Map|Array} states + * @param {{from?: string, to?: string}} [opts] - optional [from, to) window. + * @returns {Array} rows shaped like tokentracker_hourly aggregates. + */ +function aggregateSessionStates(states, opts = {}) { + const values = states instanceof Map ? [...states.values()] : Array.isArray(states) ? states : []; + const fromMs = parseMs(opts.from ?? ""); + const toMs = parseMs(opts.to ?? ""); + const grouped = new Map(); + for (const state of values) { + const row = normalizeStateRow(state); + if (!row) continue; + const bucketMs = parseMs(row.bucket_start); + if (bucketMs === null) continue; + if (fromMs !== null && bucketMs < fromMs) continue; + if (toMs !== null && bucketMs >= toMs) continue; + const key = `${row.bucket_start}||${row.source}||${row.model}`; + const acc = grouped.get(key) || { + hour_start: row.bucket_start, + source: row.source, + model: row.model, + input_tokens: 0, + output_tokens: 0, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: 0, + conversations: 0, + }; + for (const column of TOKEN_COLUMNS) acc[column] += row[column]; + acc.conversations += 1; + grouped.set(key, acc); + } + return [...grouped.values()].sort( + (a, b) => + String(a.hour_start).localeCompare(String(b.hour_start)) || + String(a.source).localeCompare(String(b.source)) || + String(a.model).localeCompare(String(b.model)), + ); +} + +// Mirrors the legacy SQL DISTINCT ON (hour, source, model) +// ORDER BY total_tokens DESC, updated_at DESC (device_id asc as the final +// deterministic tiebreak). Used ONLY for account-level sources without a +// stable session identity ('cursor') and watermark-less history. +function pickLegacyRow(rows) { + let best = null; + for (const row of rows) { + if (!best) { + best = row; + continue; + } + const rowTotal = Number(row.total_tokens) || 0; + const bestTotal = Number(best.total_tokens) || 0; + if (rowTotal !== bestTotal) { + if (rowTotal > bestTotal) best = row; + continue; + } + const rowUpdated = parseMs(row.updated_at ?? "") ?? -Infinity; + const bestUpdated = parseMs(best.updated_at ?? "") ?? -Infinity; + if (rowUpdated !== bestUpdated) { + if (rowUpdated > bestUpdated) best = row; + continue; + } + if (String(row.device_id) < String(best.device_id)) best = row; + } + return best; +} + +/** + * Legacy whole-row MAX dedup for account-level hourly rows WITHOUT a stable + * session identity ('cursor'): rows are identical across devices, so one + * canonical row per (hour, source, model) suffices. Sources with a session + * identity ('trae-cn') must instead aggregate from canonical session states + * (aggregateSessionStates) - never from per-device hourly rows. + * + * @param {Array<{device_id: string, hour_start: string, source: string, + * model: string, updated_at?: string, total_tokens: number}>} rows - + * every device's tokentracker_hourly-shaped rows. + * @returns {Array} canonical rows - one consistent row per (hour, model). + */ +function dedupeAccountLevelRows(rows) { + if (!Array.isArray(rows)) return []; + // Group rows by (source, hour_start); keep original row objects untouched. + const byHour = new Map(); + for (const row of rows) { + if (!row || typeof row.hour_start !== "string" || !row.hour_start) continue; + const key = `${row.source}||${row.hour_start}`; + const list = byHour.get(key) || []; + list.push(row); + byHour.set(key, list); + } + + const out = []; + for (const [, hourRows] of byHour) { + const byModel = new Map(); + for (const row of hourRows) { + const modelKey = row.model || "unknown"; + const list = byModel.get(modelKey) || []; + list.push(row); + byModel.set(modelKey, list); + } + for (const list of byModel.values()) { + out.push(pickLegacyRow(list)); + } + } + return out; +} + +/** Total tokens of a deduped row set - convenience for tests/diagnostics. */ +function sumTotalTokens(rows) { + return (Array.isArray(rows) ? rows : []).reduce( + (acc, row) => acc + (Number(row?.total_tokens) || 0), + 0, + ); +} + +module.exports = { + SESSION_STATE_KIND, + TOKEN_COLUMNS, + upsertAccountSessionStates, + aggregateSessionStates, + dedupeAccountLevelRows, + sumTotalTokens, +}; diff --git a/src/lib/local-api.js b/src/lib/local-api.js index d3da49494..8fc65777c 100644 --- a/src/lib/local-api.js +++ b/src/lib/local-api.js @@ -338,6 +338,10 @@ function readQueueData(queuePath) { if (!line.trim()) continue; try { const row = JSON.parse(line); + // Account session states (and the pre-release watermark records + // this branch may still hold in a dev queue) are cloud-side control + // records, not usage rows - never surface them locally. + if (row?.kind === "account_session_state" || row?.kind === "account_sync_watermark") continue; // Deduplicate: each sync appends cumulative totals per bucket, so for // each (source, model, hour_start) keep only the latest (last) entry. const key = `${row.source || ""}|${row.model || ""}|${row.hour_start || ""}`; diff --git a/src/lib/rollout.js b/src/lib/rollout.js index 5db1e6a2d..aee500667 100644 --- a/src/lib/rollout.js +++ b/src/lib/rollout.js @@ -16483,6 +16483,489 @@ function readTraeEntitlementFromStorage(storagePath) { }; } +// ───────────────────────────────────────────────────────────────────────────── +// Trae Work CN (国内版) — usage API incremental parser. +// +// TRAE Work CN's usage API reports per-session rows that are NOT append-only: +// a session can be re-reported with corrected token totals, a different model, +// or a shifted time bucket. This parser keeps the last normalized contribution +// per session_id under `cursors.traeCn` and reconciles by subtracting the +// previous contribution from its old bucket before adding the new one, so +// corrections retract the stale tuple instead of stacking. Only the normalized +// contribution is persisted — no raw API rows, cost/credits, auth, refresh +// tokens, or prompt previews. Source is always `trae-cn`. +const TRAE_CN_SOURCE = "trae-cn"; +const TRAE_CN_STATE_VERSION = 1; +const TRAE_CN_UNKNOWN_MODEL = "trae-cn-unknown"; + +function normalizeTraeCnState(raw) { + if (raw === undefined || raw === null) { + return { + version: TRAE_CN_STATE_VERSION, + sessions: {}, + prunedBeforeMs: 0, + updatedAt: null, + }; + } + if (typeof raw !== "object" || Array.isArray(raw)) { + throw new Error("Trae CN cursor state is malformed."); + } + if (raw.version !== TRAE_CN_STATE_VERSION) { + throw new Error(`Trae CN cursor state version ${raw.version} is not supported.`); + } + if (!raw.sessions || typeof raw.sessions !== "object" || Array.isArray(raw.sessions)) { + throw new Error("Trae CN cursor sessions are malformed."); + } + return { + version: TRAE_CN_STATE_VERSION, + sessions: { ...raw.sessions }, + prunedBeforeMs: + Number.isFinite(raw.prunedBeforeMs) && raw.prunedBeforeMs > 0 ? raw.prunedBeforeMs : 0, + updatedAt: typeof raw.updatedAt === "string" ? raw.updatedAt : null, + }; +} + +function isValidTraeCnTotals(totals) { + if (!totals || typeof totals !== "object") return false; + for (const key of [ + "input_tokens", + "cached_input_tokens", + "cache_creation_input_tokens", + "output_tokens", + "reasoning_output_tokens", + "total_tokens", + "billable_total_tokens", + "conversation_count", + ]) { + if (!Number.isSafeInteger(totals[key]) || totals[key] < 0) return false; + } + return true; +} + +// Canonical UTC half-hour bucketStart produced by toUtcHalfHourStart: +// "YYYY-MM-DDTHH:00:00.000Z" or "YYYY-MM-DDTHH:30:00.000Z". +function isValidTraeCnBucketStart(value) { + if (typeof value !== "string") return false; + if (!/^\d{4}-\d{2}-\d{2}T\d{2}:(00|30):00\.000Z$/.test(value)) return false; + const dt = new Date(value); + return Number.isFinite(dt.getTime()) && dt.toISOString() === value; +} + +// Validate a stored prior contribution before it is ever subtracted, so a +// tampered or corrupt cursor fails closed instead of silently rewinding buckets. +// Enforces this provider's fixed canonical invariant (all totals safe +// nonnegative integers, reasoning=0, conversation_count=1, +// total=input+cached+cacheCreation+output, billable=total) plus a canonical +// half-hour UTC bucketStart. +function validateTraeCnStoredContribution(entry) { + if (!entry || typeof entry !== "object") { + throw new Error("Trae CN stored session contribution is malformed."); + } + if ( + typeof entry.model !== "string" || + !entry.model.trim() || + entry.model.includes("|") + ) { + throw new Error("Trae CN stored session model is malformed."); + } + if (!isValidTraeCnBucketStart(entry.bucketStart)) { + throw new Error("Trae CN stored session bucket is malformed."); + } + if (!isValidTraeCnTotals(entry.totals)) { + throw new Error("Trae CN stored session totals are malformed."); + } + const { totals } = entry; + if (totals.reasoning_output_tokens !== 0 || totals.conversation_count !== 1) { + throw new Error("Trae CN stored session totals are malformed."); + } + const sum = + totals.input_tokens + + totals.cached_input_tokens + + totals.cache_creation_input_tokens + + totals.output_tokens; + if (totals.total_tokens !== sum || totals.billable_total_tokens !== totals.total_tokens) { + throw new Error("Trae CN stored session totals are malformed."); + } +} + +// Token fields may live in `row.extra_info` (object or JSON string) or at the +// top level of the row. Missing values are NOT coerced to zero. +function traeCnExtraInfo(row) { + const extra = row?.extra_info; + if (extra && typeof extra === "object" && !Array.isArray(extra)) return extra; + if (typeof extra === "string") { + try { + const parsed = JSON.parse(extra); + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) return parsed; + } catch (_e) {} + } + return null; +} + +function traeCnTokenField(row, extra, key) { + if (extra && extra[key] !== undefined && extra[key] !== null) return extra[key]; + if (row && row[key] !== undefined && row[key] !== null) return row[key]; + return undefined; +} + +// Normalize one raw session row into the canonical contribution, or throw +// before any cursor/bucket mutation happens. Returns +// { sessionId, model, bucketStart, totals }. +function normalizeTraeCnSession(row) { + const sessionId = typeof row?.session_id === "string" ? row.session_id.trim() : ""; + if (!sessionId) { + throw new Error("Trae CN session row is missing session_id."); + } + let model; + if (row?.model_name === undefined || row?.model_name === null) { + model = TRAE_CN_UNKNOWN_MODEL; + } else if (typeof row.model_name === "string") { + model = row.model_name.trim() || TRAE_CN_UNKNOWN_MODEL; + } else { + throw new Error("Trae CN session row has an invalid model_name."); + } + if (model.includes("|")) { + throw new Error("Trae CN session row has an unsupported model name."); + } + if (!Number.isSafeInteger(row?.usage_time) || row.usage_time <= 0) { + throw new Error("Trae CN session row has an invalid usage_time."); + } + const bucketStart = toUtcHalfHourStart(row.usage_time * 1000); + if (!bucketStart) { + throw new Error("Trae CN session row has an invalid usage_time."); + } + const extra = traeCnExtraInfo(row); + // Cache fields may be absent for models without a prompt-cache concept + // (Doubao / DeepSeek on TRAE CN) — an absent cache field means "no cache + // activity", i.e. 0, NOT a malformed row. input/output stay mandatory. + // + // `input_token` is cache-INCLUSIVE: verifying real rows against TRAE CN's + // own credit billing, credits = fresh_input*p + cache_read*(p/4) + output*q + // fits every row with zero residual only under this reading (same + // convention as Codex / Qoder). Peel the cache subsets off into their own + // columns and report only the remainder as ordinary input, otherwise cached + // context is double-counted in dashboards and cost calculations. cache_write + // was 0 across all observed rows; it is treated as a further subset of the + // cache-inclusive input (clamped so the buckets always sum back to it). + const cacheRead = traeCnTokenField(row, extra, "cache_read_token"); + const cacheWrite = traeCnTokenField(row, extra, "cache_write_token"); + const fields = [ + ["input_token", traeCnTokenField(row, extra, "input_token")], + ["output_token", traeCnTokenField(row, extra, "output_token")], + ["cache_read_token", cacheRead === undefined || cacheRead === null ? 0 : cacheRead], + ["cache_write_token", cacheWrite === undefined || cacheWrite === null ? 0 : cacheWrite], + ]; + for (const [label, value] of fields) { + if (!Number.isSafeInteger(value) || value < 0) { + throw new Error(`Trae CN session row has an invalid ${label}.`); + } + } + const rawInput = fields[0][1]; + const outputTokens = fields[1][1]; + const cachedInput = Math.min(rawInput, fields[2][1]); + const cacheCreation = Math.min(rawInput - cachedInput, fields[3][1]); + const total = rawInput + outputTokens; + if (!Number.isSafeInteger(total)) { + throw new Error("Trae CN session row token totals overflow."); + } + return { + sessionId, + model, + bucketStart, + totals: { + input_tokens: rawInput - cachedInput - cacheCreation, + output_tokens: outputTokens, + cached_input_tokens: cachedInput, + cache_creation_input_tokens: cacheCreation, + reasoning_output_tokens: 0, + total_tokens: total, + billable_total_tokens: total, + conversation_count: 1, + }, + }; +} + +// Fail closed instead of letting subtractTotals clamp a stored prior +// contribution that exceeds its source bucket (a corruption signal). +function assertTraeCnBucketCovers(bucketTotals, previousTotals) { + for (const key of [ + "input_tokens", + "cached_input_tokens", + "cache_creation_input_tokens", + "output_tokens", + "reasoning_output_tokens", + "total_tokens", + "billable_total_tokens", + "conversation_count", + ]) { + if ((bucketTotals[key] || 0) < (previousTotals[key] || 0)) { + throw new Error("Trae CN state corruption: stored contribution exceeds its bucket totals."); + } + } +} + +// Account session state: a queue record (kind: "account_session_state") +// carrying the CANONICAL observation of ONE provider-side TRAE CN session. +// Cloud truth for trae-cn lives at the session level +// (tokentracker_account_session_states; ingest edge upserts via +// tokentracker_upsert_account_session_states): identity is +// (user_id, source, session_id) - device_id is NOT identity (the usage API +// request carries no device discriminator). Evidence split (2026-08-17, one +// account, three real fetches 137 -> 141 -> 164): repeated-fetch id +// stability VERIFIED (137/137 persisted, corrections KEPT ids), cross-window +// stability VERIFIED (exact subsets), no duplicate ids OBSERVED. Cross-device +// same-account id stability is NOT DIRECTLY VERIFIED - no device +// discriminator in the request body is necessary but not sufficient (a +// device/login context could ride inside the JWT / server auth context), and +// no second independent device/auth experiment was run. +// +// The three correction classes collapse into ONE whole-row replace: +// downward S tokens 100 -> 60 +// model S model A -> B +// bucket S bucket 10:00 -> 10:30 +// A fresh device with no cursor history uploads every session it observes; +// the cloud LWW guard reconciles versions. ABSENCE is NOT PROVEN to mean +// deletion, so nothing is ever emitted for sessions missing from a +// non-empty snapshot, and an empty payload emits nothing at all. +// +// snapshot_verified_at is the CLIENT logical fetch stamp (the API exposes no +// provider-side ordering signal - headers carry only CDN trace ids, rows +// carry no revision). It is stamped once per real fetch and replayed +// verbatim by this append-only queue; the cloud upsert applies strictly +// newer (>) stamps only, so replays are idempotent and a transport retry of +// an older observation cannot displace a newer one. Cross-device ordering +// under clock skew is a documented residual risk, NOT strict correctness. +// +// Appended AFTER the bucket rows of the same sync (queue order guarantees a +// device's rows land before the states describing them). Bucket-row readers +// skip this record via its kind field (it carries no hour_start and is not +// a usage row). +async function appendTraeCnSessionStates({ queuePath, observations, verifiedAtMs }) { + if (!Array.isArray(observations) || observations.length === 0) return 0; + if (!Number.isFinite(verifiedAtMs)) { + throw new Error("Trae CN session states require a finite verification stamp."); + } + const verifiedAt = new Date(verifiedAtMs).toISOString(); + const lines = []; + for (const obs of observations) { + const t = obs.totals; + lines.push( + JSON.stringify({ + kind: "account_session_state", + source: TRAE_CN_SOURCE, + session_id: obs.sessionId, + model: obs.model, + bucket_start: obs.bucketStart, + input_tokens: t.input_tokens, + output_tokens: t.output_tokens, + cached_input_tokens: t.cached_input_tokens, + cache_creation_input_tokens: t.cache_creation_input_tokens, + reasoning_output_tokens: t.reasoning_output_tokens, + total_tokens: t.total_tokens, + snapshot_verified_at: verifiedAt, + }), + ); + } + await fs.appendFile(queuePath, lines.join("\n") + "\n", "utf8"); + return lines.length; +} + +async function parseTraeCnApiIncremental({ + sessions, + cursors, + queuePath, + onProgress, + windowStartMs, + windowEndMs, + snapshotVerifiedAtMs, +} = {}) { + if (!Array.isArray(sessions)) { + throw new Error("Trae CN sessions must be an array."); + } + if (!cursors || typeof cursors !== "object" || Array.isArray(cursors)) { + throw new Error("Trae CN cursors must be a writable object."); + } + await ensureDir(path.dirname(queuePath)); + + // Validate persisted state up front (fresh absent state initializes version + // 1; malformed state / unexpected version fails closed instead of resetting). + // Deep-clone the normalized working states so reconciliation and enqueue + // mutations never touch `cursors` before the final assignment succeeds. + const hourlyState = structuredClone(normalizeHourlyState(cursors?.hourly)); + const traeCnState = structuredClone(normalizeTraeCnState(cursors?.traeCn)); + + if (sessions.length === 0) { + // Empty payload is a successful no-op: no usage mutation, no session + // state records. + // + // Evidence check (2026-08-16, two real fetches 17min apart over the same + // account: 137 -> 141 sessions, 0 disappeared, 0 rows changed) shows the + // session set is stable, but NOTHING proves 'absent from the response' + // means 'authoritatively deleted / zero' — so the absence contract is + // NOT PROVEN and must stay symmetric everywhere: a session missing from + // a non-empty snapshot never emits a retraction, and an empty response + // asserts nothing at all. Canonical corrections ride on explicit + // observations only. + return { recordsProcessed: 0, eventsAggregated: 0, bucketsQueued: 0 }; + } + + // Full prevalidation + in-payload dedupe BEFORE any cursor/bucket mutation. + // + // ANY malformed row fails the whole snapshot closed. Canonical session + // states assert this device actually understood the snapshot, so a + // partially understood snapshot (99 valid + 1 uninterpretable row) must + // never be enqueued: it would become the canonical cloud state. The + // provider-level try/catch in cmdSync keeps the failure + // isolated (other providers sync on) and the error message carries only + // the count + reason - no session ids, tokens, or credentials. Confirmed + // LEGAL variations must not land here: absent cache fields are already + // coerced to 0 inside normalizeTraeCnSession (models without a prompt + // cache concept); only truly uninterpretable rows (bad session_id / + // usage_time / token numbers / model, impossible schema) throw. + const bySession = new Map(); + let skippedRows = 0; + let firstSkipReason = ""; + for (const row of sessions) { + let normalized; + try { + normalized = normalizeTraeCnSession(row); + } catch (error) { + skippedRows += 1; + if (!firstSkipReason) firstSkipReason = error?.message || "unknown"; + continue; + } + const existing = bySession.get(normalized.sessionId); + if (existing) { + const identical = + existing.model === normalized.model && + existing.bucketStart === normalized.bucketStart && + totalsKey(existing.totals) === totalsKey(normalized.totals); + if (!identical) { + throw new Error("Trae CN session rows contain conflicting contributions."); + } + continue; // exact duplicate within one payload is accepted once + } + bySession.set(normalized.sessionId, normalized); + } + if (skippedRows > 0) { + // Partial snapshots are not authoritative: one uninterpretable row means + // the window was NOT fully verified, so fail closed - no bucket rows, no + // session states, no cursor commit. The next sync replays from the same + // cursor state (idempotent). + throw new Error( + `Trae CN snapshot is not authoritative: ${skippedRows} malformed row${skippedRows !== 1 ? "s" : ""} (first: ${firstSkipReason}).`, + ); + } + const touchedBuckets = new Set(); + const cb = typeof onProgress === "function" ? onProgress : null; + const total = sessions.length; + let index = 0; + let eventsAggregated = 0; + + // Validate EVERY existing stored contribution up front — not just sessions + // present in the incoming payload — so an unrelated malformed entry fails + // closed before any reconciliation mutation. + for (const sessionId of Object.keys(traeCnState.sessions)) { + validateTraeCnStoredContribution(traeCnState.sessions[sessionId]); + } + + // Monotonic prune (same watermark pattern as the kiroCli cap above): the + // sync caller fetches a fixed trailing window, so a session whose entire + // half-hour bucket sits before windowStartMs can never reappear in a later + // payload — its stored contribution is dead reconciliation weight. Prune + // only when the window start moved FORWARD past the persisted watermark: + // an entry evicted under window W1 and re-fetched under an earlier window + // would find previousEntry undefined and count twice. The +30min margin + // matches bucketStart's half-hour floor, which can trail the row's + // usage_time by up to 30 minutes. + if (Number.isFinite(windowStartMs) && windowStartMs > traeCnState.prunedBeforeMs) { + const pruneCutoffMs = windowStartMs - 30 * 60 * 1000; + for (const sessionId of Object.keys(traeCnState.sessions)) { + const bucketMs = Date.parse(traeCnState.sessions[sessionId].bucketStart); + if (Number.isFinite(bucketMs) && bucketMs <= pruneCutoffMs) { + delete traeCnState.sessions[sessionId]; + } + } + traeCnState.prunedBeforeMs = windowStartMs; + } + + // Deterministic: process unique session ids in sorted order. Sessions + // whose canonical observation changed (or that were never seen) are also + // collected for the account_session_state queue records. + const sessionIds = [...bySession.keys()].sort(); + const changedObservations = []; + for (const sessionId of sessionIds) { + const current = bySession.get(sessionId); + const previousEntry = traeCnState.sessions[sessionId]; + + const unchanged = + previousEntry && + totalsKey(previousEntry.totals) === totalsKey(current.totals) && + previousEntry.bucketStart === current.bucketStart && + previousEntry.model === current.model; + if (unchanged) { + index += 1; + if (cb) cb({ index, total, eventsAggregated, bucketsQueued: touchedBuckets.size }); + continue; + } + + if (previousEntry) { + const oldBucket = getHourlyBucket( + hourlyState, + TRAE_CN_SOURCE, + previousEntry.model, + previousEntry.bucketStart, + ); + assertTraeCnBucketCovers(oldBucket.totals, previousEntry.totals); + subtractTotals(oldBucket.totals, previousEntry.totals); + touchedBuckets.add(bucketKey(TRAE_CN_SOURCE, previousEntry.model, previousEntry.bucketStart)); + } + + const bucket = getHourlyBucket(hourlyState, TRAE_CN_SOURCE, current.model, current.bucketStart); + addTotals(bucket.totals, current.totals); + touchedBuckets.add(bucketKey(TRAE_CN_SOURCE, current.model, current.bucketStart)); + + traeCnState.sessions[sessionId] = { + model: current.model, + bucketStart: current.bucketStart, + totals: { ...current.totals }, + updatedAt: new Date().toISOString(), + }; + changedObservations.push({ + sessionId, + model: current.model, + bucketStart: current.bucketStart, + totals: current.totals, + }); + eventsAggregated += 1; + index += 1; + if (cb) cb({ index, total, eventsAggregated, bucketsQueued: touchedBuckets.size }); + } + + const bucketsQueued = await enqueueTouchedBuckets({ queuePath, hourlyState, touchedBuckets }); + // Emit canonical session observations only after the reconciled bucket + // rows are durably queued; a failure here aborts before cursor commit, so + // the next sync replays the (idempotent) reconciliation and re-appends + // both. snapshot_verified_at is stamped once per real fetch; the + // append-only queue replays it verbatim, so transport retries never fake + // freshness. Unchanged sessions emit nothing - a fixed-now no-change sync + // stays byte-identical. + const verifiedAtMs = Number.isFinite(snapshotVerifiedAtMs) ? snapshotVerifiedAtMs : Date.now(); + await appendTraeCnSessionStates({ + queuePath, + observations: changedObservations, + verifiedAtMs, + }); + const updatedAt = new Date().toISOString(); + hourlyState.updatedAt = updatedAt; + traeCnState.updatedAt = updatedAt; + // Assign cursor state only after enqueue succeeds. + cursors.hourly = hourlyState; + cursors.traeCn = traeCnState; + + return { recordsProcessed: total, eventsAggregated, bucketsQueued, skippedRows }; +} + // ───────────────────────────────────────────────────────────────────────────── // DeepSeek Harness (dsh) — passive reader of the harness's session logs. // @@ -17336,6 +17819,7 @@ module.exports = { resolveTraePath, resolveTraeStoragePath, readTraeEntitlementFromStorage, + parseTraeCnApiIncremental, // DeepSeek Harness (dsh) — passive session-log reader resolveDshHome, resolveDshHomes, diff --git a/src/lib/source-metadata.js b/src/lib/source-metadata.js index bc659b630..06d8bf182 100644 --- a/src/lib/source-metadata.js +++ b/src/lib/source-metadata.js @@ -1,4 +1,4 @@ -const ACCOUNT_LEVEL_SOURCES = new Set(["cursor"]); +const ACCOUNT_LEVEL_SOURCES = new Set(["cursor", "trae-cn"]); function normalizeSource(value) { return String(value || "").trim().toLowerCase(); diff --git a/src/lib/trae-cn-config.js b/src/lib/trae-cn-config.js new file mode 100644 index 000000000..f9bf1a483 --- /dev/null +++ b/src/lib/trae-cn-config.js @@ -0,0 +1,539 @@ +"use strict"; + +/** + * TRAE Work CN (国内版) config + usage fetch. Phase A: macOS + test-injected + * paths only. Resolves the CN home, reads the `iCubeAuthInfo://icube.cloudide` + * blob from User/globalStorage/storage.json (plaintext JSON or Base64 tc v5), + * and talks to the TRAE CN usage API. JWTs / refresh tokens are never + * persisted, never logged, and never put into error messages; a + * present-but-unreadable blob fails closed with a generic error. + * + * tc v5 blob: [6-byte magic "tc\x05\x10\x00\x00"][32-byte salt][AES-128-CBC ct] + * kdf_buf = SHA-512(salt) || (JG xor KG) // 128 bytes + * kdf_out = SHA-512(kdf_buf) // 64 bytes + * key = kdf_out[0..16], iv = kdf_out[16..32] + * plaintext = [SHA-512(json)][json] // integrity prefix stripped + */ + +const crypto = require("node:crypto"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); + +const TRAE_CN_HOME_ENV = "TOKENTRACKER_TRAE_CN_HOME"; +// The TRAE CN usage read transmits the locally stored sign-in JWT to TRAE's +// official endpoint, so it is strictly opt-in: nothing is sent unless the +// user explicitly sets this flag. Any value other than 1/true keeps it off. +const TRAE_CN_USAGE_ENV = "TOKENTRACKER_TRAE_CN_USAGE"; + +function isTraeCnUsageEnabled(env = process.env) { + const value = env ? env[TRAE_CN_USAGE_ENV] : undefined; + const normalized = typeof value === "string" ? value.trim().toLowerCase() : ""; + return normalized === "1" || normalized === "true"; +} +const TRAE_CN_APP_DIR = "TRAE SOLO CN"; +const TRAE_CN_AUTH_KEY = "iCubeAuthInfo://icube.cloudide"; + +const TRAE_CN_USAGE_URL = "https://api.trae.cn/trae/api/v1/pay/query_user_usage_group_by_session"; +// usage_type [7] mirrors the constant the official TRAE client sends when its +// usage page queries this endpoint (observed in the client's outbound +// requests; the enum is undocumented server-side). Other values unverified. +const TRAE_CN_USAGE_TYPE = [7]; +const TRAE_CN_USAGE_PAGE_SIZE = 20; +const TRAE_CN_USAGE_MAX_PAGES = 100; +const TRAE_CN_USAGE_DELAY_MS = 300; +const TRAE_CN_USAGE_TIMEOUT_MS = 30 * 1000; +// Capacity-adaptive window splitting: 30 days halves at most 8 times +// (finest sub-window ~2.8h, aggregate ceiling ~512k sessions/30d). A window +// still over capacity at that depth fails closed — never a partial import. +const TRAE_CN_USAGE_MAX_SPLIT_DEPTH = 8; + +const TRAE_CN_MAGIC = Buffer.from([0x74, 0x63, 0x05, 0x10, 0x00, 0x00]); +const TRAE_CN_SALT_LEN = 32; +const TRAE_CN_HASH_LEN = 64; +const TRAE_CN_AES_KEY_LEN = 16; +const TRAE_CN_IV_LEN = 16; + +// JG / KG — the two halves of the hardcoded (obfuscated) password in the TRAE +// client's byteCrypto.js (XOR the pair to recover the KDF secret). Failure +// signature when TRAE ships a new key: readTraeCnAuthFromStorage throws +// "Trae CN auth payload is malformed", sync skips the source, and +// `tokentracker status` reports auth "malformed" — re-extract the pair +// from the updated client to restore decryption. +const TRAE_CN_JG = Buffer.from([ + 82, 9, 106, 213, 48, 54, 165, 56, 191, 64, 163, 158, 129, 243, 215, 251, 124, 227, 57, 130, + 155, 47, 255, 135, 52, 142, 67, 68, 196, 222, 233, 203, 84, 123, 148, 50, 166, 194, 35, 61, + 238, 76, 149, 11, 66, 250, 195, 78, 8, 46, 161, 102, 40, 217, 36, 178, 118, 91, 162, 73, + 109, 139, 209, 37, +]); +const TRAE_CN_KG = Buffer.from([ + 31, 221, 168, 51, 136, 7, 199, 49, 177, 18, 16, 89, 39, 128, 236, 95, 96, 81, 127, 169, 25, + 181, 74, 13, 45, 229, 122, 159, 147, 201, 156, 239, 160, 224, 59, 77, 174, 42, 245, 176, + 200, 235, 187, 60, 131, 83, 153, 97, 23, 43, 4, 126, 186, 119, 214, 38, 225, 105, 20, 99, + 85, 33, 12, 125, +]); + +// --------------------------------------------------------------------------- +// Path resolution +// --------------------------------------------------------------------------- + +function resolveTraeCnHome({ env = process.env, home = os.homedir(), platform = process.platform } = {}) { + const override = env[TRAE_CN_HOME_ENV]; + if (typeof override === "string" && override.trim()) { + return path.resolve(override.trim()); + } + if (platform === "darwin") { + return path.join(home, "Library", "Application Support", TRAE_CN_APP_DIR); + } + // ponytail: this phase is macOS + test-injected paths only. + return null; +} + +function resolveTraeCnStoragePath(options = {}) { + const home = resolveTraeCnHome(options); + return home ? path.join(home, "User", "globalStorage", "storage.json") : null; +} + +// --------------------------------------------------------------------------- +// tc v5 decryption +// --------------------------------------------------------------------------- + +function traeCnHardcodedKdfSecret() { + const secret = Buffer.alloc(64); + for (let i = 0; i < 64; i++) secret[i] = TRAE_CN_JG[i] ^ TRAE_CN_KG[i]; + return secret; +} + +function deriveTraeCnKeyIv(salt) { + if (!Buffer.isBuffer(salt) || salt.length !== TRAE_CN_SALT_LEN) { + throw new Error("Trae CN auth salt is malformed."); + } + const kdfBuf = Buffer.concat([ + crypto.createHash("sha512").update(salt).digest(), + traeCnHardcodedKdfSecret(), + ]); + // Vendor tc-v5 compatibility KDF, not password storage. + const kdfOut = crypto.createHash("sha512").update(kdfBuf).digest(); + return { + key: kdfOut.subarray(0, TRAE_CN_AES_KEY_LEN), + iv: kdfOut.subarray(TRAE_CN_AES_KEY_LEN, TRAE_CN_AES_KEY_LEN + TRAE_CN_IV_LEN), + }; +} + +/** Decrypt a raw tc v5 blob; returns the JSON bytes (integrity prefix stripped). */ +function decryptTraeCnBlob(blob) { + const minLen = TRAE_CN_MAGIC.length + TRAE_CN_SALT_LEN + 16; + if (!Buffer.isBuffer(blob) || blob.length < minLen) { + throw new Error("Trae CN auth blob is too short."); + } + if (!blob.subarray(0, TRAE_CN_MAGIC.length).equals(TRAE_CN_MAGIC)) { + throw new Error("Trae CN auth blob has an unknown format."); + } + const salt = blob.subarray(TRAE_CN_MAGIC.length, TRAE_CN_MAGIC.length + TRAE_CN_SALT_LEN); + const ciphertext = blob.subarray(TRAE_CN_MAGIC.length + TRAE_CN_SALT_LEN); + if (ciphertext.length % 16 !== 0) { + throw new Error("Trae CN auth blob is not block-aligned."); + } + const { key, iv } = deriveTraeCnKeyIv(salt); + const decipher = crypto.createDecipheriv("aes-128-cbc", key, iv); // PKCS7 default + let plaintext; + try { + plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]); + } catch (_error) { + throw new Error("Trae CN auth blob could not be decrypted."); + } + if (plaintext.length < TRAE_CN_HASH_LEN) { + throw new Error("Trae CN auth blob is missing its integrity digest."); + } + const expected = plaintext.subarray(0, TRAE_CN_HASH_LEN); + const data = plaintext.subarray(TRAE_CN_HASH_LEN); + const actual = crypto.createHash("sha512").update(data).digest(); + if (expected.length !== actual.length || !crypto.timingSafeEqual(expected, actual)) { + throw new Error("Trae CN auth blob failed its integrity check."); + } + return data; +} + +/** Decrypt a Base64-encoded tc v5 blob and return the plaintext UTF-8 string. */ +function decryptTraeCnBase64(value) { + const blob = Buffer.from(String(value).trim(), "base64"); + return decryptTraeCnBlob(blob).toString("utf8"); +} + +// --------------------------------------------------------------------------- +// Credential reading +// --------------------------------------------------------------------------- + +/** + * Parse a stored auth value: a plaintext JSON object / JSON string, or a + * Base64-encoded tc v5 blob. Fails closed with a credential-safe error. + */ +function parseTraeCnAuthValue(value) { + if (value && typeof value === "object") { + return value; + } + if (typeof value === "string") { + const trimmed = value.trim(); + if (!trimmed) { + throw new Error("Trae CN auth data is empty."); + } + if (trimmed.startsWith("{")) { + let parsed; + try { + parsed = JSON.parse(trimmed); + } catch (_error) { + throw new Error("Trae CN auth JSON is malformed."); + } + if (!parsed || typeof parsed !== "object") { + throw new Error("Trae CN auth JSON is malformed."); + } + return parsed; + } + let decrypted; + try { + decrypted = decryptTraeCnBase64(trimmed); + } catch (error) { + if (error && error.message.startsWith("Trae CN auth")) throw error; + throw new Error("Trae CN auth payload is malformed."); + } + let parsed; + try { + parsed = JSON.parse(decrypted); + } catch (_error) { + throw new Error("Trae CN auth payload is malformed."); + } + if (!parsed || typeof parsed !== "object") { + throw new Error("Trae CN auth payload is malformed."); + } + return parsed; + } + throw new Error("Trae CN auth data is missing or malformed."); +} + +/** + * Read + decrypt the auth object from storage.json. Returns null when the + * file or the auth key is absent (not signed in). Throws a credential-safe + * error when present data is malformed, and a distinct unreadable error for + * real IO failures (permission denied, I/O error) - an unreadable storage + * file must not be reported as "not signed in". + */ +function readTraeCnAuthFromStorage({ env, home, platform, fsModule = fs } = {}) { + const storagePath = resolveTraeCnStoragePath({ env, home, platform }); + if (!storagePath) return null; + let raw; + try { + raw = fsModule.readFileSync(storagePath, "utf8"); + } catch (error) { + // A missing file (or missing path component) simply means the app is not + // installed / not signed in. Any other failure - EACCES, EISDIR, a real + // I/O error - fails closed with a generic, credential-safe error. Only a + // stable machine-readable code is attached, never the OS detail, so + // storage contents cannot leak through logs or error messages. + if (error && (error.code === "ENOENT" || error.code === "ENOTDIR")) { + return null; + } + const unreadable = new Error("Trae CN storage.json could not be read."); + unreadable.code = "TRAE_CN_STORAGE_UNREADABLE"; + throw unreadable; + } + let parsed; + try { + parsed = JSON.parse(raw); + } catch (_error) { + throw new Error("Trae CN storage.json is not valid JSON."); + } + const value = parsed && typeof parsed === "object" ? parsed[TRAE_CN_AUTH_KEY] : undefined; + if (value === undefined || value === null) return null; + return parseTraeCnAuthValue(value); +} + +/** Extract a nonempty JWT from a parsed auth object. */ +function extractTraeCnToken(auth) { + const token = auth && typeof auth === "object" ? auth.token : undefined; + if (typeof token === "string" && token.trim()) { + return token.trim(); + } + throw new Error("Trae CN auth data has no token."); +} + +// --------------------------------------------------------------------------- +// Usage API +// --------------------------------------------------------------------------- + +function usageError(message, { status, apiCode } = {}) { + const error = new Error(message); + if (status !== undefined) error.status = status; + if (apiCode !== undefined) error.apiCode = apiCode; + if (status === 401 || status === 403) error.code = "AUTH_EXPIRED"; + return error; +} + +function assertUsageParams({ start_time, end_time, page_size }) { + if (!Number.isInteger(start_time) || start_time <= 0) { + throw new Error("Trae CN usage start_time must be a positive integer epoch second."); + } + if (!Number.isInteger(end_time) || end_time <= 0) { + throw new Error("Trae CN usage end_time must be a positive integer epoch second."); + } + if (start_time > end_time) { + throw new Error("Trae CN usage start_time must not be after end_time."); + } + if (!Number.isInteger(page_size) || page_size <= 0 || page_size > TRAE_CN_USAGE_PAGE_SIZE) { + throw new Error( + `Trae CN usage page_size must be a positive integer no greater than ${TRAE_CN_USAGE_PAGE_SIZE}.`, + ); + } +} + +/** + * Fetch one page of TRAE CN usage. Returns { sessions, total, page_num, page_size } + * where `total` is null when the API omits it. Errors never contain the JWT + * or the raw request/response body. + */ +async function fetchTraeCnUsagePage({ + jwt, + start_time, + end_time, + page_num, + page_size = TRAE_CN_USAGE_PAGE_SIZE, + fetchImpl = fetch, + timeoutMs = TRAE_CN_USAGE_TIMEOUT_MS, +} = {}) { + if (typeof jwt !== "string" || !jwt.trim()) { + throw new Error("Trae CN usage request requires a token."); + } + if (!Number.isInteger(page_num) || page_num < 1) { + throw new Error("Trae CN usage page_num must be a positive integer."); + } + assertUsageParams({ start_time, end_time, page_size }); + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + try { + let response; + try { + // Security note (CodeQL "File data -> outbound network request"): + // the credential read from TRAE's local storage.json is sent ONLY to + // TRAE_CN_USAGE_URL, a compile-time constant HTTPS endpoint of the + // official API - callers cannot redirect the destination (the url + // parameter is ignored; pinned by the "sends the exact request" test). + // The whole flow is behind the explicit TOKENTRACKER_TRAE_CN_USAGE + // opt-in, the JWT is never logged, never written to the queue/cursor + // state, and never uploaded to the TokenTracker cloud (queue rows carry + // token counters only; transport exceptions are sanitized). + response = await fetchImpl(TRAE_CN_USAGE_URL, { + method: "POST", + headers: { + "Content-Type": "application/json", + Authorization: `Cloud-IDE-JWT ${jwt.trim()}`, + }, + body: JSON.stringify({ + usage_type: TRAE_CN_USAGE_TYPE, + start_time, + end_time, + page_num, + page_size, + }), + signal: controller.signal, + }); + } catch (_error) { + throw new Error("Trae CN usage API request failed."); + } + if (response.status === 401 || response.status === 403) { + throw usageError(`Trae CN usage API returned HTTP ${response.status}.`, { status: response.status }); + } + if (!response.ok) { + throw usageError(`Trae CN usage API returned HTTP ${response.status}.`, { status: response.status }); + } + let parsed; + try { + parsed = await response.json(); + } catch (_error) { + throw new Error("Trae CN usage API returned a non-JSON response."); + } + const apiCode = parsed?.data?.code ?? parsed?.code; + if (apiCode !== undefined && Number(apiCode) !== 0) { + throw usageError(`Trae CN usage API returned error code ${apiCode}.`, { + status: response.status, + apiCode: Number(apiCode), + }); + } + // Documented wrapper: data.user_usage_group_by_sessions, or the top-level + // equivalent. Only this module parses the wrapper; token rows/buckets are + // normalized elsewhere. A body without the session-array wrapper fails + // closed as a schema error instead of being misread as an empty result; + // an explicit empty array remains valid. + const data = parsed?.data && typeof parsed.data === "object" ? parsed.data : parsed; + if (!Array.isArray(data?.user_usage_group_by_sessions)) { + throw new Error("Trae CN usage API response is missing the session list."); + } + const sessions = data.user_usage_group_by_sessions; + // The declared total gates pagination termination (sessions.length >= + // total stops the walk), so a malformed value must fail closed as a + // schema error - never be coerced into a legal-looking stop condition. + // A negative, fractional, non-finite, or non-number total (the API + // speaks JSON numbers; a quoted digit string is a schema anomaly) would + // otherwise silently truncate the snapshot after page 1 and still look + // authoritative. + const rawTotal = data?.total; + let total = null; + if (rawTotal !== undefined && rawTotal !== null) { + if ( + typeof rawTotal !== "number" || + !Number.isSafeInteger(rawTotal) || + rawTotal < 0 + ) { + throw new Error("Trae CN usage API returned an invalid total."); + } + total = rawTotal; + } + return { sessions, total, page_num, page_size }; + } finally { + clearTimeout(timer); + } +} + +/** + * Fetch usage across pages serially. Starts at the 1-based page_num, at most + * maxPages pages, with a delay between pages (pass delayMs = 0 to disable in + * tests). Stops at the stated total when present, otherwise on an empty page. + * Throws if maxPages is reached while another page would still be needed. + */ +async function fetchTraeCnUsage({ + jwt, + start_time, + end_time, + page_num = 1, + page_size = TRAE_CN_USAGE_PAGE_SIZE, + fetchImpl = fetch, + maxPages = TRAE_CN_USAGE_MAX_PAGES, + delayMs = TRAE_CN_USAGE_DELAY_MS, + pageFetcher = fetchTraeCnUsagePage, +} = {}) { + const sessions = []; + let total = null; + let pagesFetched = 0; + let page = page_num; + for (;;) { + if (pagesFetched >= maxPages) { + // A further page would be needed but the cap prevents it: fail closed + // rather than return a partial snapshot that looks authoritative. Cap + // exhaustion is the same over-capacity signal as the declared-total + // check below (which the API may omit) — tag it identically so window + // splitting engages either way. + const error = new Error("Trae CN usage pagination exceeded the maximum page count."); + error.code = "TRAE_CN_USAGE_CAPACITY_EXCEEDED"; + throw error; + } + const result = await pageFetcher({ jwt, start_time, end_time, page_num: page, page_size, fetchImpl }); + pagesFetched += 1; + total = result.total ?? null; + if (pagesFetched === 1 && Number.isFinite(total) && total > page_size * maxPages) { + const error = new Error("Trae CN usage snapshot exceeds the supported capacity."); + error.code = "TRAE_CN_USAGE_CAPACITY_EXCEEDED"; + throw error; + } + if (Array.isArray(result.sessions)) sessions.push(...result.sessions); + if (result.sessions.length === 0) break; + if (total !== null && sessions.length >= total) break; + if (delayMs > 0) await new Promise((resolve) => setTimeout(resolve, delayMs)); + page += 1; + } + return { sessions, total, pages_fetched: pagesFetched }; +} + +/** + * Capacity-adaptive window fetch. Probing the official API confirmed the + * start_time/end_time range is closed on both ends ([a,b] and [b,c] both + * return the row at b) and that a row returned by two overlapping windows is + * byte-identical (idempotent duplicate, never cumulative). Splitting therefore + * staggers adjacent sub-windows by one second ([start,mid] + [mid+1,end]) so + * the union equals the full window exactly; any residual duplicate would be + * absorbed by the session-level reconciliation. On TRAE_CN_USAGE_CAPACITY_ + * EXCEEDED the window halves recursively up to maxSplitDepth levels; a window + * still over capacity at the finest allowed granularity re-throws so the + * caller keeps its fail-closed guarantee. + */ +async function fetchTraeCnUsageWindowed(options = {}, depth = 0) { + const { start_time, end_time } = options; + try { + return await fetchTraeCnUsage(options); + } catch (error) { + if (error?.code !== "TRAE_CN_USAGE_CAPACITY_EXCEEDED") throw error; + if (end_time - start_time < 1 || depth >= TRAE_CN_USAGE_MAX_SPLIT_DEPTH) throw error; + const mid = start_time + Math.floor((end_time - start_time) / 2); + const left = await fetchTraeCnUsageWindowed({ ...options, end_time: mid }, depth + 1); + const right = await fetchTraeCnUsageWindowed({ ...options, start_time: mid + 1 }, depth + 1); + const total = + Number.isFinite(left.total) && Number.isFinite(right.total) ? left.total + right.total : null; + return { + sessions: [...left.sessions, ...right.sessions], + total, + pages_fetched: left.pages_fetched + right.pages_fetched, + }; + } +} + +/** + * High-level storage-backed fetch: read the JWT from storage.json, fetch + * usage, and on a 401/403 re-read + re-decrypt storage.json and retry exactly + * once. Never persists or logs the JWT / refresh token; no refreshToken flow. + */ +async function fetchTraeCnUsageWithAuth({ + start_time, + end_time, + page_num = 1, + page_size = TRAE_CN_USAGE_PAGE_SIZE, + fetchImpl = fetch, + delayMs = TRAE_CN_USAGE_DELAY_MS, + maxPages = TRAE_CN_USAGE_MAX_PAGES, + readAuth = readTraeCnAuthFromStorage, + usageFetcher = fetchTraeCnUsageWindowed, + ...storageOptions +} = {}) { + let auth = readAuth(storageOptions); + let jwt = auth ? extractTraeCnToken(auth) : null; + if (!jwt) { + throw new Error("Trae CN credentials are not configured."); + } + const run = () => + usageFetcher({ jwt, start_time, end_time, page_num, page_size, fetchImpl, delayMs, maxPages }); + try { + return await run(); + } catch (error) { + const authFailure = + error?.code === "AUTH_EXPIRED" || error?.status === 401 || error?.status === 403; + if (!authFailure) throw error; + // Session may have rotated since the first read; re-read storage once. + auth = readAuth(storageOptions); + jwt = auth ? extractTraeCnToken(auth) : null; + if (!jwt) throw error; + return await run(); + } +} + +module.exports = { + TRAE_CN_HOME_ENV, + TRAE_CN_USAGE_ENV, + isTraeCnUsageEnabled, + TRAE_CN_APP_DIR, + TRAE_CN_AUTH_KEY, + TRAE_CN_USAGE_URL, + TRAE_CN_USAGE_TYPE, + TRAE_CN_USAGE_PAGE_SIZE, + TRAE_CN_USAGE_MAX_PAGES, + TRAE_CN_USAGE_DELAY_MS, + TRAE_CN_MAGIC, + resolveTraeCnHome, + resolveTraeCnStoragePath, + traeCnHardcodedKdfSecret, + deriveTraeCnKeyIv, + decryptTraeCnBlob, + decryptTraeCnBase64, + parseTraeCnAuthValue, + readTraeCnAuthFromStorage, + extractTraeCnToken, + fetchTraeCnUsagePage, + fetchTraeCnUsage, + fetchTraeCnUsageWindowed, + fetchTraeCnUsageWithAuth, +}; diff --git a/test/account-source-parity.test.js b/test/account-source-parity.test.js index da7d9256e..f33578f77 100644 --- a/test/account-source-parity.test.js +++ b/test/account-source-parity.test.js @@ -5,12 +5,18 @@ // devices) is hardcoded in several places that MUST agree: // - src/lib/source-metadata.js (authoritative, used by the CLI) // - scripts/ops/account-usage-grouped-rpc.sql (account view RPC) -// - scripts/ops/leaderboard-usage-grouped-rpc.sql (leaderboard RPC) // - dashboard/edge-patches/tokentracker-leaderboard-profile.ts (profile edge) // - dashboard/edge-patches/tokentracker-account-devices.ts (device breakdown edge) // A drift (e.g. adding a new account-level source to source-metadata.js but // forgetting the SQL) silently re-introduces the cross-device double-count bug // that v0.44 fixed. This test fails loudly on any mismatch. +// +// NOT covered here: scripts/ops/leaderboard-usage-grouped-rpc.sql (SUPERSEDED +// rollback reference — do not edit or verify). Since the account +// session-states work, leaderboard_hourly_dedup_v2's account_sources array +// IS versioned in migrations/20260817120000_account-session-states.sql, so +// it is verified below; applying that migration to production stays a +// maintainer-side deployment step. const fs = require("node:fs"); const path = require("node:path"); @@ -46,7 +52,7 @@ function extractSqlAccountSources(content) { .sort(); } -test("account-level source list is identical across source-metadata, both RPCs, and the profile edge", () => { +test("account-level source list is identical across source-metadata, the account RPC, and the profile edge", () => { const authoritative = extractJsSet( readFile("src/lib/source-metadata.js"), "ACCOUNT_LEVEL_SOURCES", @@ -60,8 +66,8 @@ test("account-level source list is identical across source-metadata, both RPCs, "account-usage-grouped-rpc.sql": extractSqlAccountSources( readFile("scripts/ops/account-usage-grouped-rpc.sql"), ), - "leaderboard-usage-grouped-rpc.sql": extractSqlAccountSources( - readFile("scripts/ops/leaderboard-usage-grouped-rpc.sql"), + "20260817120000_account-session-states.sql (leaderboard_hourly_dedup_v2)": extractSqlAccountSources( + readFile("migrations/20260817120000_account-session-states.sql"), ), "tokentracker-leaderboard-profile.ts": extractJsSet( readFile("dashboard/edge-patches/tokentracker-leaderboard-profile.ts"), diff --git a/test/account-usage-dedup.test.js b/test/account-usage-dedup.test.js new file mode 100644 index 000000000..15b595873 --- /dev/null +++ b/test/account-usage-dedup.test.js @@ -0,0 +1,495 @@ +/** + * Account-level cross-device truth regression tests (PR #474 review). + * + * TRAE CN's usage API returns CORRECTABLE snapshots, not append-only events: + * a session's totals can be revised downward, moved to another model, or + * shifted to another half-hour. Hour-level dedup cannot express those + * corrections (a fresh device's first data bucket cannot safely displace + * earlier hours, and a 10:00 -> 10:30 migration strands the old hour), so + * canonical truth for trae-cn lives at the SESSION level. These tests pin + * the semantics implemented by src/lib/account-usage-dedup.js (executable + * spec) and mirrored by: + * - migrations/20260817120000_account-session-states.sql + * (tokentracker_account_session_states + LWW upsert + + * leaderboard_hourly_dedup_v2) + * - scripts/ops/account-usage-grouped-rpc.sql (account_usage_grouped) + * - dashboard/edge-patches/tokentracker-ingest.ts (batch upload) + * + * The merge-blocker scenarios run the REAL pipeline: parser -> queue -> + * ingest batch -> LWW persistence -> aggregation. + */ +"use strict"; + +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const test = require("node:test"); + +const { + SESSION_STATE_KIND, + upsertAccountSessionStates, + aggregateSessionStates, + dedupeAccountLevelRows, + sumTotalTokens, +} = require("../src/lib/account-usage-dedup"); +const { parseTraeCnApiIncremental } = require("../src/lib/rollout"); + +const ROOT = path.join(__dirname, ".."); + +// Device ids / hours used across the scenarios. +const OLD = "11111111-1111-1111-1111-111111111111"; +const FRESH = "22222222-2222-2222-2222-222222222222"; +const H10 = "2027-01-10T10:00:00.000Z"; +const H1030 = "2027-01-10T10:30:00.000Z"; + +// --------------------------------------------------------------------------- +// Real-pipeline harness: parser -> queue -> ingest batch -> LWW table. +// +// Each "device" run drives the REAL parseTraeCnApiIncremental against its +// own append-only queue, then drains the queue exactly like the production +// upload path (src/commands/sync.js readQueueBatch collects session-state +// records last-wins per (source, session_id); the ingest edge dedupes the +// same way before the SQL batch upsert), then applies them to the shared +// cloud table through the JS mirror of tokentracker_upsert_account_ +// session_states. Aggregation mirrors the RPC's trae-cn branch. +// --------------------------------------------------------------------------- +function tempQueue(prefix) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + return { dir, queuePath: path.join(dir, "queue.jsonl") }; +} + +const WINDOW_START_MS = 1_799_000_000_000; // well before both devices' data +const WINDOW_END_MS = 1_801_000_000_000; + +async function deviceSync(cloudStates, device, { sessions, verifiedAtMs, cursors = {} }) { + const { dir, queuePath } = tempQueue("tokentracker-session-pipeline-"); + try { + await parseTraeCnApiIncremental({ + sessions, + cursors, + queuePath, + windowStartMs: WINDOW_START_MS, + windowEndMs: WINDOW_END_MS, + snapshotVerifiedAtMs: verifiedAtMs, + }); + // Drain the queue like the production upload path (last-wins per + // session, mirroring sync.js readQueueBatch + the edge's stateMap). + // An empty payload writes NO queue file at all (the parser asserts + // nothing), so the ingest batch is simply empty. + const lines = fs.existsSync(queuePath) + ? fs + .readFileSync(queuePath, "utf8") + .trim() + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line)) + : []; + const stateMap = new Map(); + for (const row of lines) { + if (row.kind === SESSION_STATE_KIND) { + stateMap.set(row.source + "|" + row.session_id, row); + } + } + upsertAccountSessionStates(cloudStates, [...stateMap.values()]); + return lines; + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +function traeSession(sessionId, usageTimeSec, overrides = {}) { + return { + session_id: sessionId, + model_name: "model-a", + usage_time: usageTimeSec, + input_token: 100, + output_token: 10, + cache_read_token: 0, + cache_write_token: 0, + ...overrides, + }; +} + +const T10 = Date.parse("2027-01-10T10:00:00.000Z") / 1000; // -> the 10:00 bucket +const T1030 = Date.parse("2027-01-10T10:30:00.000Z") / 1000; // -> the 10:30 bucket +const VERIFIED_OLD = Date.parse("2027-01-15T00:00:00.000Z"); +const VERIFIED_FRESH = Date.parse("2027-01-16T00:00:00.000Z"); + +function totalsByHour(rows) { + return new Map(rows.map((r) => [r.hour_start, r.total_tokens])); +} + +// --------------------------------------------------------------------------- +// 1. Fresh-device bucket migration (THE P0): old device uploaded S@10:00, +// the API later corrected S to 10:30, and the fresh device's snapshot only +// contains S@10:30 (its FIRST data bucket) - no hour-coverage inference +// can reclaim 10:00, but the session identity can. +// --------------------------------------------------------------------------- +test("R1. fresh-device bucket migration: S@10:00=100 then S@10:30=100 -> only 10:30, total 100", async () => { + const cloudStates = new Map(); + // Device OLD synced first: session s1 lived at 10:00. + await deviceSync(cloudStates, OLD, { + sessions: [traeSession("s1", T10)], + verifiedAtMs: VERIFIED_OLD, + }); + // Fresh device, no cursor history: the API now reports s1 at 10:30. + await deviceSync(cloudStates, FRESH, { + sessions: [traeSession("s1", T1030)], + verifiedAtMs: VERIFIED_FRESH, + cursors: {}, // fresh device: no local TRAE history + }); + const rows = aggregateSessionStates(cloudStates); + const byHour = totalsByHour(rows); + assert.equal(byHour.get(H10), undefined, "the 10:00 tuple is GONE (whole-row replace)"); + assert.equal(byHour.get(H1030), 100 + 10, "only the corrected 10:30 bucket remains"); + assert.equal(sumTotalTokens(rows), 110, "total is 110 tokens (session counted once), never 220"); +}); + +// --------------------------------------------------------------------------- +// 2. Fresh-device model migration. +// --------------------------------------------------------------------------- +test("R2. fresh-device model migration: S/A=100 then S/B=100 -> only B, total 100", async () => { + const cloudStates = new Map(); + await deviceSync(cloudStates, OLD, { + sessions: [traeSession("s1", T10)], + verifiedAtMs: VERIFIED_OLD, + }); + await deviceSync(cloudStates, FRESH, { + sessions: [traeSession("s1", T10, { model_name: "model-b" })], + verifiedAtMs: VERIFIED_FRESH, + cursors: {}, + }); + const rows = aggregateSessionStates(cloudStates); + const byModel = new Map(rows.map((r) => [r.model, r.total_tokens])); + assert.equal(byModel.get("model-a"), undefined, "stale model-a state is replaced, not kept"); + assert.equal(byModel.get("model-b"), 110, "only model-b remains"); + assert.equal(sumTotalTokens(rows), 110, "total 110, never the stitched 220"); +}); + +// --------------------------------------------------------------------------- +// 3. Fresh-device downward correction. +// --------------------------------------------------------------------------- +test("R3. fresh-device downward correction: S=100 then S=60 -> 60, never MAX", async () => { + const cloudStates = new Map(); + await deviceSync(cloudStates, OLD, { + sessions: [traeSession("s1", T10)], + verifiedAtMs: VERIFIED_OLD, + }); + await deviceSync(cloudStates, FRESH, { + sessions: [traeSession("s1", T10, { input_token: 50 })], + verifiedAtMs: VERIFIED_FRESH, + cursors: {}, + }); + const rows = aggregateSessionStates(cloudStates); + assert.equal(sumTotalTokens(rows), 60, "the corrected 60 wins, never the MAX 110"); + assert.equal(rows.length, 1, "one canonical session row"); +}); + +// --------------------------------------------------------------------------- +// 4. Missing session: absence is NOT deletion (contract NOT PROVEN). +// --------------------------------------------------------------------------- +test("R4. a session missing from the next non-empty snapshot is RETAINED", async () => { + const cloudStates = new Map(); + await deviceSync(cloudStates, OLD, { + sessions: [traeSession("s1", T10)], + verifiedAtMs: VERIFIED_OLD, + }); + // A later NON-EMPTY snapshot that simply lacks s1 (it only reports s2). + await deviceSync(cloudStates, FRESH, { + sessions: [traeSession("s2", T1030, { input_token: 20, output_token: 2 })], + verifiedAtMs: VERIFIED_FRESH, + cursors: {}, + }); + const rows = aggregateSessionStates(cloudStates); + assert.equal(cloudStates.get("trae-cn|s1").total_tokens, 110, "s1 is untouched by absence"); + assert.equal(sumTotalTokens(rows), 110 + 22, "both sessions count once"); +}); + +// --------------------------------------------------------------------------- +// 5. Empty response: asserts NOTHING (contract NOT PROVEN). +// --------------------------------------------------------------------------- +test("R5. an empty snapshot asserts nothing: existing sessions survive", async () => { + const cloudStates = new Map(); + await deviceSync(cloudStates, OLD, { + sessions: [traeSession("s1", T10)], + verifiedAtMs: VERIFIED_OLD, + }); + // The parser short-circuits an empty payload before ANY mutation, so the + // ingest batch is empty and the LWW table is untouched. + await deviceSync(cloudStates, FRESH, { + sessions: [], + verifiedAtMs: VERIFIED_FRESH, + cursors: {}, + }); + const rows = aggregateSessionStates(cloudStates); + assert.equal(sumTotalTokens(rows), 110, "empty response never deletes anything"); +}); + +// --------------------------------------------------------------------------- +// 6. Transport retry idempotence: the same observation replayed N times +// yields ONE canonical state with unchanged numbers. +// --------------------------------------------------------------------------- +test("R6. replaying the same observation is idempotent", async () => { + const cloudStates = new Map(); + const lines = await deviceSync(cloudStates, OLD, { + sessions: [traeSession("s1", T10)], + verifiedAtMs: VERIFIED_OLD, + }); + const observations = lines.filter((r) => r.kind === SESSION_STATE_KIND); + assert.equal(observations.length, 1); + + // Transport retries re-deliver the EXACT same batch (same bytes, same + // snapshot_verified_at - the append-only queue replays it verbatim). + const second = upsertAccountSessionStates(cloudStates, observations); + const third = upsertAccountSessionStates(cloudStates, observations); + assert.equal(second, 0, "first replay applies nothing"); + assert.equal(third, 0, "further replays apply nothing"); + assert.equal(aggregateSessionStates(cloudStates).length, 1, "still one canonical session"); + assert.equal(sumTotalTokens(aggregateSessionStates(cloudStates)), 110, "numbers unchanged"); + + // A genuinely NEWER fetch of the same session still lands (LWW >). + const applied = upsertAccountSessionStates(cloudStates, [ + { + ...observations[0], + input_token: 60, + total_tokens: 70, + snapshot_verified_at: new Date(VERIFIED_FRESH).toISOString(), + }, + ]); + assert.equal(applied, 1); + assert.equal(sumTotalTokens(aggregateSessionStates(cloudStates)), 70, "a newer stamp replaces"); +}); + +// --------------------------------------------------------------------------- +// LWW unit semantics (mirroring the SQL upsert guard exactly). +// --------------------------------------------------------------------------- +test("LWW: strictly newer stamps replace, equal stamps keep the first-applied row", () => { + const states = new Map(); + const base = { + source: "trae-cn", + session_id: "s1", + model: "model-a", + bucket_start: H10, + input_tokens: 100, + output_tokens: 0, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: 100, + snapshot_verified_at: "2027-01-15T00:00:00.000Z", + }; + assert.equal(upsertAccountSessionStates(states, [base]), 1); + // EQUAL stamp: first-applied wins (stable under retries). + assert.equal( + upsertAccountSessionStates(states, [{ ...base, model: "model-b" }]), + 0, + "equal stamp never replaces", + ); + assert.equal(states.get("trae-cn|s1").model, "model-a"); + // OLDER stamp: never displaces. + assert.equal( + upsertAccountSessionStates(states, [ + { ...base, model: "model-c", snapshot_verified_at: "2027-01-14T00:00:00.000Z" }, + ]), + 0, + "older observation is a no-op", + ); + assert.equal(states.get("trae-cn|s1").model, "model-a"); + // STRICTLY newer stamp: whole-row replace. + assert.equal( + upsertAccountSessionStates(states, [ + { + ...base, + model: "model-b", + bucket_start: H1030, + input_tokens: 60, + total_tokens: 60, + snapshot_verified_at: "2027-01-16T00:00:00.000Z", + }, + ]), + 1, + ); + const replaced = states.get("trae-cn|s1"); + assert.equal(replaced.model, "model-b"); + assert.equal(replaced.bucket_start, H1030); + assert.equal(replaced.total_tokens, 60, "downward replacement applied, not MAXed"); +}); + +test("LWW: malformed observations (bad dates / missing identity) assert nothing", () => { + const states = new Map(); + assert.equal(upsertAccountSessionStates(states, [null, {}, { source: "trae-cn" }]), 0); + assert.equal(states.size, 0); + assert.equal( + upsertAccountSessionStates(states, [ + { + source: "trae-cn", + session_id: "s1", + model: "model-a", + bucket_start: "not-a-date", + total_tokens: 10, + snapshot_verified_at: "2027-01-15T00:00:00.000Z", + }, + ]), + 0, + ); + assert.equal(states.size, 0, "an unparseable bucket_start never lands"); +}); + +test("aggregation groups by (bucket, model) and counts sessions, with an optional [from, to) window", () => { + const states = new Map(); + upsertAccountSessionStates(states, [ + { + source: "trae-cn", + session_id: "s1", + model: "model-a", + bucket_start: H10, + input_tokens: 100, + output_tokens: 0, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: 100, + snapshot_verified_at: "2027-01-15T00:00:00.000Z", + }, + { + source: "trae-cn", + session_id: "s2", + model: "model-a", + bucket_start: H10, + input_tokens: 5, + output_tokens: 0, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: 5, + snapshot_verified_at: "2027-01-15T00:00:00.000Z", + }, + { + source: "trae-cn", + session_id: "s3", + model: "model-b", + bucket_start: H1030, + input_tokens: 7, + output_tokens: 0, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: 7, + snapshot_verified_at: "2027-01-15T00:00:00.000Z", + }, + ]); + const rows = aggregateSessionStates(states); + assert.deepEqual( + rows.map((r) => [r.hour_start, r.model, r.total_tokens, r.conversations]), + [ + [H10, "model-a", 105, 2], + [H1030, "model-b", 7, 1], + ], + ); + const windowed = aggregateSessionStates(states, { from: H1030 }); + assert.equal(windowed.length, 1, "window filter excludes earlier buckets"); + assert.equal(windowed[0].model, "model-b"); +}); + +// --------------------------------------------------------------------------- +// Legacy dedup: account-level sources WITHOUT a session identity ('cursor'). +// --------------------------------------------------------------------------- +test("cursor-style rows (no session identity) keep the legacy whole-row MAX dedup", () => { + const rows = [ + { device_id: OLD, hour_start: H10, source: "cursor", model: "model-a", total_tokens: 100, updated_at: "2027-01-15T00:00:00.000Z" }, + { device_id: FRESH, hour_start: H10, source: "cursor", model: "model-a", total_tokens: 120, updated_at: "2027-01-15T00:00:00.000Z" }, + { device_id: OLD, hour_start: H10, source: "cursor", model: "model-b", total_tokens: 30, updated_at: "2027-01-15T00:00:00.000Z" }, + { device_id: FRESH, hour_start: H10, source: "cursor", model: "model-b", total_tokens: 30, updated_at: "2027-01-15T00:00:00.000Z" }, + ]; + const deduped = dedupeAccountLevelRows(rows); + assert.equal(sumTotalTokens(deduped), 150, "per-model MAX pick, identical rows count once"); + assert.equal(deduped.length, 2); +}); + +test("sumTotalTokens tolerates junk input", () => { + assert.equal(sumTotalTokens(null), 0); + assert.equal(sumTotalTokens([null, { total_tokens: "x" }, { total_tokens: 5 }]), 5); +}); + +// --------------------------------------------------------------------------- +// SQL / deployment parity: the deployed pipeline must implement the SAME +// session-state semantics as the JS spec above. +// --------------------------------------------------------------------------- +function readRepoFile(rel) { + return fs.readFileSync(path.join(ROOT, rel), "utf8"); +} + +test("migration: session-state table identity is (user_id, source, session_id) and the upsert guard is strictly newer", () => { + const sql = readRepoFile("migrations/20260817120000_account-session-states.sql"); + assert.match(sql, /CREATE TABLE IF NOT EXISTS public\.tokentracker_account_session_states/); + assert.match( + sql, + /PRIMARY KEY \(user_id, source, session_id\)/, + "device_id is NOT part of the canonical identity", + ); + assert.match(sql, /CHECK \(\s*total_tokens =\s*input_tokens \+ cached_input_tokens \+ cache_creation_input_tokens \+ output_tokens\s*\)/); + assert.match(sql, /ENABLE ROW LEVEL SECURITY/); + // The LWW guard. + assert.match( + sql, + /WHERE EXCLUDED\.snapshot_verified_at > t\.snapshot_verified_at/, + "only STRICTLY newer observations replace (retries are no-ops)", + ); + // Nothing may ever delete a session row. + assert.doesNotMatch(sql, /\bDELETE\b/i, "absence is not proven - no deletion path"); + // Leaderboard aggregates trae-cn from the session states. + assert.match(sql, /leaderboard_hourly_dedup_v2/); + const traeBranch = sql.split("trae-cn: canonical account truth aggregated from session states")[1]; + assert.ok(traeBranch, "leaderboard has a trae-cn session-state branch"); + assert.match(traeBranch.split("$func$")[0], /FROM tokentracker_account_session_states/); +}); + +test("account_usage_grouped RPC aggregates trae-cn from session states (no watermark path)", () => { + const sql = readRepoFile("scripts/ops/account-usage-grouped-rpc.sql"); + assert.match(sql, /tokentracker_account_session_states/, "RPC reads the session-state table"); + assert.match( + sql, + /GROUP BY s\.bucket_start, s\.source, s\.model/, + "aggregation groups by (bucket, source, model) - same as the JS spec", + ); + assert.doesNotMatch( + sql, + /account_sync_watermark|first_covered_hour/, + "watermarks must be fully out of the correctness path", + ); + // 'cursor' keeps the legacy whole-row MAX pick (DISTINCT ON + ORDER BY + // total_tokens DESC), NOT the session-state path. + const cursorBranch = sql.split("'cursor' (account-level, no session identity)")[1] || ""; + assert.match(cursorBranch.split("UNION ALL")[0], /source = 'cursor'/); + assert.match(cursorBranch.split("UNION ALL")[0], /total_tokens DESC/); +}); + +test("ingest edge validates account_session_states and upserts them via the LWW RPC after the hourly rows", () => { + const ts = readRepoFile("dashboard/edge-patches/tokentracker-ingest.ts"); + assert.match(ts, /body\.account_session_states/, "edge reads the account_session_states payload"); + assert.match(ts, /Invalid account session state/, "malformed states fail closed"); + assert.match(ts, /r\.source \+ "\|" \+ r\.session_id/, "batch dedup is last-wins per session"); + assert.match(ts, /tokentracker_upsert_account_session_states/, "edge calls the LWW upsert RPC"); + // Session states land AFTER the hourly upsert in file order. + assert.ok( + ts.indexOf('from("tokentracker_hourly")') < ts.indexOf("tokentracker_upsert_account_session_states"), + "bucket rows land before the canonical states describing them", + ); +}); + +test("local queue readers never surface session-state control records as usage rows", () => { + const localApi = readRepoFile("src/lib/local-api.js"); + assert.match( + localApi, + /kind === "account_session_state"/, + "readQueueData must skip account_session_state records", + ); + const sync = readRepoFile("src/commands/sync.js"); + assert.match( + sync, + /kind === "account_session_state"/, + "the upload path collects account_session_state records separately from bucket rows", + ); + assert.match(sync, /account_session_states/, "the upload body carries account_session_states"); +}); diff --git a/test/discovery-metadata.test.js b/test/discovery-metadata.test.js index f8b448318..725f3cca5 100644 --- a/test/discovery-metadata.test.js +++ b/test/discovery-metadata.test.js @@ -9,14 +9,14 @@ const ROOT = path.resolve(__dirname, ".."); const read = (relativePath) => fs.readFileSync(path.join(ROOT, relativePath), "utf8"); const README_EXPECTATIONS = [ - ["README.md", /32 AI coding tools/, /\|\s+\*\*AI tools supported\*\*\s+\|\s+\*\*32\*\*/, /Rate-limit tracking.*✅ 13 providers/], - ["README.zh-CN.md", /32 款 AI 编码工具/, /\|\s+\*\*支持的 AI 工具数\*\*\s+\|\s+\*\*32\*\*/, /限额追踪.*✅ 13 家 provider/], - ["README.ja.md", /32 種類の AI コーディングツール/, /\|\s+\*\*対応 AI ツール数\*\*\s+\|\s+\*\*32\*\*/, /レート制限トラッキング.*✅ 13 プロバイダー/], - ["README.ko.md", /32개의 AI 코딩 도구/, /\|\s+\*\*지원하는 AI 도구 수\*\*\s+\|\s+\*\*32\*\*/, /레이트 제한 추적.*✅ 13개 프로바이더/], - ["README.de.md", /32 KI-Coding-Tools/, /\|\s+\*\*Unterstützte KI-Tools\*\*\s+\|\s+\*\*32\*\*/, /Rate-Limit-Tracking.*✅ 13 Provider/], + ["README.md", /33 AI coding tools/, /\|\s+\*\*AI tools supported\*\*\s+\|\s+\*\*33\*\*/, /Rate-limit tracking.*✅ 13 providers/], + ["README.zh-CN.md", /33 款 AI 编码工具/, /\|\s+\*\*支持的 AI 工具数\*\*\s+\|\s+\*\*33\*\*/, /限额追踪.*✅ 13 家 provider/], + ["README.ja.md", /33 種類の AI コーディングツール/, /\|\s+\*\*対応 AI ツール数\*\*\s+\|\s+\*\*33\*\*/, /レート制限トラッキング.*✅ 13 プロバイダー/], + ["README.ko.md", /33개의 AI 코딩 도구/, /\|\s+\*\*지원하는 AI 도구 수\*\*\s+\|\s+\*\*33\*\*/, /레이트 제한 추적.*✅ 13개 프로바이더/], + ["README.de.md", /33 KI-Coding-Tools/, /\|\s+\*\*Unterstützte KI-Tools\*\*\s+\|\s+\*\*33\*\*/, /Rate-Limit-Tracking.*✅ 13 Provider/], ]; -test("public discovery surfaces describe all 32 supported tools", () => { +test("public discovery surfaces describe all 33 supported tools", () => { for (const [file, countPattern, comparisonPattern, limitCountPattern] of README_EXPECTATIONS) { const source = read(file); assert.match(source, countPattern, `${file} has the current provider count`); @@ -25,12 +25,15 @@ test("public discovery surfaces describe all 32 supported tools", () => { assert.match(source, /AnythingLLM Desktop/, `${file} lists AnythingLLM Desktop`); assert.match(source, /Qoder/, `${file} lists Qoder`); assert.match(source, /DeepSeek Harness/, `${file} lists DeepSeek Harness`); + assert.match(source, /Prime Agent/, `${file} lists Prime Agent`); + assert.match(source, /TRAE Work CN/, `${file} lists TRAE Work CN`); assert.match(source, limitCountPattern, `${file} rate-limit row carries the current usage-limits provider count`); } const index = read("dashboard/index.html"); assert.doesNotMatch(index, /13 AI coding/); - assert.match(index, /Supported AI coding tools \(32\)/); + assert.match(index, /Supported AI coding tools \(33\)/); + assert.match(index, /TRAE Work CN/); assert.match(index, /Desktop pet/); assert.match(index, /Four desktop widgets/); assert.match(index, /Achievements/); @@ -38,41 +41,44 @@ test("public discovery surfaces describe all 32 supported tools", () => { assert.match(index, /usage limits for 13 providers/i); const llms = read("dashboard/public/llms.txt"); - assert.match(llms, /Supported AI coding tools \(32\)/); + assert.match(llms, /Supported AI coding tools \(33\)/); + assert.match(llms, /TRAE Work CN/); assert.match(llms, /desktop pet/i); assert.match(llms, /four desktop widgets/i); assert.match(llms, /achievements/i); }); -test("marketing logo wall includes the same 32 product integrations", () => { +test("marketing logo wall includes the same 33 product integrations", () => { const source = read("dashboard/src/ui/marketing/agent-logos.js"); const providers = [...source.matchAll(/provider:\s*"([^"]+)"/g)].map((match) => match[1]); - assert.equal(providers.length, 32); - assert.equal(new Set(providers).size, 32); + assert.equal(providers.length, 33); + assert.equal(new Set(providers).size, 33); - for (const provider of ["every-code", "reasonix", "kilocode", "roocode", "zed", "goose", "droid", "qoder", "anythingllm", "dsh"]) { + for (const provider of ["every-code", "reasonix", "kilocode", "roocode", "zed", "goose", "droid", "qoder", "anythingllm", "dsh", "prime-agent", "trae-cn"]) { assert.ok(providers.includes(provider), `logo wall includes ${provider}`); } }); -test("CLI onboarding advertises the same 32 supported integrations", () => { +test("CLI onboarding advertises the same 33 supported integrations", () => { const source = read("src/commands/init.js"); const block = source.match(/const SUPPORTED_PROVIDERS = \[([\s\S]*?)\];/); assert.ok(block, "init defines SUPPORTED_PROVIDERS"); const providers = [...block[1].matchAll(/^\s*"([^"]+)",?$/gm)].map((match) => match[1]); - assert.equal(providers.length, 32); - assert.equal(new Set(providers).size, 32); + assert.equal(providers.length, 33); + assert.equal(new Set(providers).size, 33); assert.ok(providers.includes("Droid")); assert.ok(providers.includes("AnythingLLM Desktop")); assert.ok(providers.includes("Qoder")); assert.ok(providers.includes("Reasonix")); assert.ok(providers.includes("DeepSeek Harness")); + assert.ok(providers.includes("Prime Agent")); + assert.ok(providers.includes("TRAE Work CN")); }); test("npm metadata carries the current product hook", () => { const pkg = JSON.parse(read("package.json")); - assert.match(pkg.description, /32 tools/); + assert.match(pkg.description, /33 tools/); assert.match(pkg.description, /desktop pet/); assert.ok(pkg.keywords.includes("desktop-widget")); assert.ok(pkg.keywords.includes("ai-coding-tools")); diff --git a/test/leaderboard-rollup-correction.test.js b/test/leaderboard-rollup-correction.test.js new file mode 100644 index 000000000..ee0fafc55 --- /dev/null +++ b/test/leaderboard-rollup-correction.test.js @@ -0,0 +1,531 @@ +/** + * >7-day TRAE CN corrections vs the HISTORICAL rollup paths (PR #474). + * + * Product read paths under test (mirrored from the deployed SQL): + * + * account usage / profile / dashboard history + * -> account_usage_grouped(_cached) LIVE session states + * leaderboard week / month + * -> leaderboard_usage_grouped(bounded) LIVE leaderboard_hourly_dedup_v2 + * leaderboard TOTAL + * -> leaderboard_rollup_daily_v2 (closed days, materialized) + * UNION leaderboard_hourly_dedup_v2(v_through, now) (live tail) + * + * leaderboard_hourly_dedup_v2 aggregates trae-cn from + * tokentracker_account_session_states (migrations/ + * 20260817120000_account-session-states.sql), so LIVE reads reflect a + * correction immediately. Materialized rollup days only change when the + * cyclic repair in leaderboard_rollup_daily_advance_v2 (7 days per run, + * from the OLDEST history day, wrapping) reaches the corrected day - the + * claim "historical days self-heal" is true, but the lag scales with TOTAL + * history length (7 days repaired per scheduled 6h run), not with the + * correction's age. These tests pin BOTH facts: + * + * A. >7d downward correction rollup 100 -> 60 once repair covers the day + * B. >7d model migration rollup loses model-A, keeps model-B=100 + * C. cross-day bucket migration Day A 23:30 -> Day B 00:00: Day A drops + * the 100, Day B gains it, total stays 100 (transient dip possible + * while only one of the two days has been repaired) + * D. FIRST 30-day seed on a caught-up rollup: live paths (account / + * profile / bounded boards) include the seed immediately, but the + * leaderboard TOTAL (rollup UNION live tail) misses every CLOSED seed + * day until a repair rebuilds them. The seed-gap PRIORITIZATION in + * advance_v2 (repair jumps to the earliest uncovered trae-cn day) + * heals the seed deterministically in ceil(seed_days / 7) scheduled + * runs instead of waiting a full cycle. + * E. MULTI-USER isolation of that prioritization: gap coverage is PER + * USER (r.user_id = s.user_id) - another user's same-day trae-cn + * rollup row does not cover a new user's first seed day. + * + * The session-state inputs come from the REAL parser + * (parseTraeCnApiIncremental -> queue -> LWW upsert mirror); the rollup/repair + * cycle below is the executable mirror of leaderboard_rollup_daily_advance_v2 + * / _replace_v2 / leaderboard_usage_grouped, and the trailing tests pin the + * deployed SQL itself so the mirror cannot drift silently. + */ +"use strict"; + +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const test = require("node:test"); + +const { + upsertAccountSessionStates, + aggregateSessionStates, +} = require("../src/lib/account-usage-dedup"); +const { parseTraeCnApiIncremental } = require("../src/lib/rollout"); + +const ROOT = path.join(__dirname, ".."); +const DAY_MS = 24 * 60 * 60 * 1000; +const HALF_HOUR_MS = 30 * 60 * 1000; + +// Everything UTC-midnight aligned, like the SQL (date_trunc('day' ... UTC)). +const midnight = (ms) => Math.floor(ms / DAY_MS) * DAY_MS; + +function isoDay(ms) { + return new Date(midnight(ms)).toISOString().slice(0, 10); +} + +// --------------------------------------------------------------------------- +// Real parser -> queue -> LWW table (same harness shape as +// test/account-usage-dedup.test.js). +// --------------------------------------------------------------------------- +function tempQueue(prefix) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + return { dir, queuePath: path.join(dir, "queue.jsonl") }; +} + +async function syncSessions(cloudStates, { sessions, verifiedAtMs }) { + const { dir, queuePath } = tempQueue("tokentracker-rollup-"); + try { + await parseTraeCnApiIncremental({ + sessions, + cursors: {}, + queuePath, + windowStartMs: verifiedAtMs - 40 * DAY_MS, + windowEndMs: verifiedAtMs + DAY_MS, + snapshotVerifiedAtMs: verifiedAtMs, + }); + const lines = fs + .readFileSync(queuePath, "utf8") + .trim() + .split("\n") + .filter(Boolean) + .map((l) => JSON.parse(l)); + const batch = lines.filter((r) => r.kind === "account_session_state"); + upsertAccountSessionStates(cloudStates, batch); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +function traeSession(sessionId, usageTimeMs, overrides = {}) { + return { + session_id: sessionId, + model_name: "model-a", + usage_time: Math.floor(usageTimeMs / 1000), + input_token: 100, + output_token: 10, + cache_read_token: 0, + cache_write_token: 0, + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// Executable mirror of the rollup machinery. +// --------------------------------------------------------------------------- +const REPAIR_CHUNK_DAYS = 7; // advance_v2: interval '7 days' per run + +// getAccounts() -> [{ userId, states }] - one entry per user, mirroring +// the real shared tables (rollup PK (user_id, source, model, day); session +// states PK (user_id, source, session_id)). +function createRollupSim(getAccounts, { minDayMs, nowMs }) { + // meta mirrors tokentracker_leaderboard_rollup_meta_v2; assume catch-up + // done (through = today), repair_from cycling from the oldest history day. + const target = () => midnight(nowMs); + const meta = { through: target(), repairFrom: midnight(minDayMs) }; + // rollup mirrors tokentracker_leaderboard_rollup_daily_v2: + // Map "day|source|model|user" -> {user_id, day, source, model, total_tokens} + const rollup = new Map(); + + function replaceDay(dayMs) { + const dayKey = isoDay(dayMs); + for (const key of [...rollup.keys()]) { + if (key.startsWith(dayKey + "|")) rollup.delete(key); + } + for (const { userId, states } of getAccounts()) { + for (const row of aggregateSessionStates(states, { + from: new Date(dayMs).toISOString(), + to: new Date(dayMs + DAY_MS).toISOString(), + })) { + rollup.set(`${dayKey}|${row.source}|${row.model}|${userId}`, { + user_id: userId, + day: dayKey, + source: row.source, + model: row.model, + total_tokens: row.total_tokens, + }); + } + } + } + + return { + // Seed-gap prioritization (mirrors the superseding advance_v2): the + // earliest CLOSED day where SOME user has trae-cn session states but NO + // trae-cn rollup row OF THEIR OWN jumps the repair window there. + // Coverage is PER USER (r.user_id = s.user_id in the SQL): another + // user's same-day row does not cover this user's seed. Stale-value + // corrections (the user's own row exists) keep the cyclic schedule. + minSeedGapDay() { + const t = target(); + let gap = null; + for (const { userId, states } of getAccounts()) { + for (const state of states.values()) { + const dayMs = midnight(Date.parse(state.bucket_start)); + if (dayMs >= t) continue; + const covered = [...rollup.keys()].some( + (key) => + key.startsWith(isoDay(dayMs) + "|trae-cn|") && + key.endsWith("|" + userId), + ); + if (!covered && (gap === null || dayMs < gap)) gap = dayMs; + } + } + return gap; + }, + /** One scheduled advance run: gap-prioritized, else the cyclic chunk. */ + advanceRun() { + const t = target(); + const gap = this.minSeedGapDay(); + let from = gap !== null ? gap : (meta.repairFrom >= t ? minDayMs : meta.repairFrom); + const until = Math.min(t, from + REPAIR_CHUNK_DAYS * DAY_MS); + for (let d = from; d < until; d += DAY_MS) replaceDay(d); + meta.repairFrom = until >= t ? minDayMs : until; + }, + /** Materialize [minDay, today) in one go (initial full build). */ + buildAll() { + const t = target(); + for (let d = minDayMs; d < t; d += DAY_MS) replaceDay(d); + meta.repairFrom = minDayMs; + }, + /** leaderboard TOTAL: materialized rollup UNION live tail [through, now). */ + leaderboardTotal() { + const perModel = new Map(); + for (const row of rollup.values()) { + perModel.set(row.model, (perModel.get(row.model) || 0) + row.total_tokens); + } + for (const { states } of getAccounts()) { + for (const row of aggregateSessionStates(states, { + from: new Date(meta.through).toISOString(), + })) { + perModel.set(row.model, (perModel.get(row.model) || 0) + row.total_tokens); + } + } + return perModel; + }, + rollupDay(dayMs, model, userId = "user-a") { + return rollup.get(`${isoDay(dayMs)}|trae-cn|${model}|${userId}`) || null; + }, + }; +} + +// Fixed "now" so days are deterministic: TODAY = 2027-02-01T00:00:00Z. +const NOW_MS = midnight(Date.parse("2027-02-01T12:00:00.000Z")); +const DAY = (offsetDays) => NOW_MS - offsetDays * DAY_MS; + +// A 120-day history: the repair cycle needs ceil/worst ~17 runs to cover a +// day-20 correction - the honest lag characterization for the total board. +const HISTORY_MIN_DAY = DAY(120); + +function scenarioSetup() { + const cloudStates = new Map(); // user a (scenarios A-D are single-user) + const cloudStatesB = new Map(); // user b (multi-user scenario E) + const accounts = () => [ + { userId: "user-a", states: cloudStates }, + { userId: "user-b", states: cloudStatesB }, + ]; + const sim = createRollupSim(accounts, { minDayMs: HISTORY_MIN_DAY, nowMs: NOW_MS }); + return { cloudStates, cloudStatesB, sim }; +} + +const V1 = Date.parse("2027-01-12T00:00:00.000Z"); // original fetch stamp +const V2 = Date.parse("2027-02-01T00:00:00.000Z"); // correction fetch stamp + +// --------------------------------------------------------------------------- +// A. >7-day downward correction. +// --------------------------------------------------------------------------- +test("A. >7d downward: live paths show 60 immediately; total rollup heals when the repair cycle reaches the day", async () => { + const { cloudStates, sim } = scenarioSetup(); + const correctedDay = DAY(20); + await syncSessions(cloudStates, { + sessions: [traeSession("s1", correctedDay + 10 * HALF_HOUR_MS)], + verifiedAtMs: V1, + }); + sim.buildAll(); + assert.equal(sim.leaderboardTotal().get("model-a"), 110, "rollup materialized 100+10"); + assert.ok(sim.rollupDay(correctedDay, "model-a"), "day-20 rollup row exists"); + + // TRAE revises S down to input 50 (=total 60) 20 days later. + await syncSessions(cloudStates, { + sessions: [traeSession("s1", correctedDay + 10 * HALF_HOUR_MS, { input_token: 50 })], + verifiedAtMs: V2, + }); + + // LIVE paths (account_usage_grouped / bounded leaderboard windows): + // corrected immediately, zero rollup involvement. + const liveRows = aggregateSessionStates(cloudStates); + assert.equal(liveRows.length, 1); + assert.equal(liveRows[0].total_tokens, 60, "live path shows 60 immediately"); + + // TOTAL path: stale until the cyclic repair reaches day-20. + let runs = 0; + const maxRuns = 200; + while (sim.leaderboardTotal().get("model-a") !== 60 && runs < maxRuns) { + sim.advanceRun(); + runs += 1; + } + assert.ok(runs > 0, "the total board did NOT flip without repair work"); + assert.equal(sim.leaderboardTotal().get("model-a"), 60, `total healed to 60 after ${runs} repair runs`); + // Honest lag pin: with 120 days of history and 7 days per run the + // corrected day is reached near the END of the cycle (>= 10 runs). + assert.ok(runs >= 10, `repair-cycle lag is real (took ${runs} scheduled runs, not immediate)`); + assert.equal(sim.rollupDay(correctedDay, "model-a").total_tokens, 60, "materialized day-20 row itself is 60"); +}); + +// --------------------------------------------------------------------------- +// B. >7-day model migration. +// --------------------------------------------------------------------------- +test("B. >7d model migration: repaired rollup keeps ONLY model-B (no permanent model-A residue)", async () => { + const { cloudStates, sim } = scenarioSetup(); + const correctedDay = DAY(20); + await syncSessions(cloudStates, { + sessions: [traeSession("s1", correctedDay + 10 * HALF_HOUR_MS)], + verifiedAtMs: V1, + }); + sim.buildAll(); + assert.ok(sim.rollupDay(correctedDay, "model-a"), "model-A row materialized"); + + await syncSessions(cloudStates, { + sessions: [traeSession("s1", correctedDay + 10 * HALF_HOUR_MS, { model_name: "model-b" })], + verifiedAtMs: V2, + }); + + // Live bounded path already migrated. + const liveByModel = new Map(aggregateSessionStates(cloudStates).map((r) => [r.model, r.total_tokens])); + assert.deepEqual([...liveByModel.keys()].sort(), ["model-b"]); + + let runs = 0; + while ((!sim.leaderboardTotal().has("model-b") || sim.leaderboardTotal().has("model-a")) && runs < 200) { + sim.advanceRun(); + runs += 1; + } + const total = sim.leaderboardTotal(); + assert.equal(total.get("model-a"), undefined, "no permanent model-A residue after repair"); + assert.equal(total.get("model-b"), 110, "model-B carries the session exactly once"); + assert.equal(sim.rollupDay(correctedDay, "model-a"), null, "the materialized model-A row is deleted by the day rebuild"); + assert.equal(sim.rollupDay(correctedDay, "model-b").total_tokens, 110); +}); + +// --------------------------------------------------------------------------- +// C. Cross-day bucket migration (Day A 23:30 -> Day B 00:00). +// --------------------------------------------------------------------------- +test("C. cross-day bucket migration: Day A drops the 100, Day B gains it, total stays 100", async () => { + const { cloudStates, sim } = scenarioSetup(); + const dayA = DAY(21); // 23:30 slot lives on day A + const dayB = DAY(20); // 00:00 slot lives on day B (the next UTC day) + const slotA = dayA + 23 * HALF_HOUR_MS + HALF_HOUR_MS; // 23:30 on day A + const slotB = dayB; // 00:00 on day B + await syncSessions(cloudStates, { + sessions: [traeSession("s1", slotA)], + verifiedAtMs: V1, + }); + sim.buildAll(); + assert.equal(sim.rollupDay(dayA, "model-a").total_tokens, 110, "100+10 sits on day A"); + assert.equal(sim.rollupDay(dayB, "model-a"), null, "day B empty"); + + // TRAE moves the session across the UTC-day boundary. + await syncSessions(cloudStates, { + sessions: [traeSession("s1", slotB)], + verifiedAtMs: V2, + }); + + // Live path: only day B's bucket carries the session now. + const liveRows = aggregateSessionStates(cloudStates); + assert.equal(liveRows.length, 1); + assert.equal(Date.parse(liveRows[0].hour_start), Math.floor(slotB / HALF_HOUR_MS) * HALF_HOUR_MS); + assert.equal(liveRows[0].total_tokens, 110); + + // Total path: day A and day B sit in the same 7-day repair chunk here, so + // one run covering both heals them together; a chunk BOUNDARY between the + // two days would transiently dip (documented, not a correctness bug - the + // canonical state itself is single-copy). + let runs = 0; + const healed = () => sim.rollupDay(dayA, "model-a") === null + && sim.rollupDay(dayB, "model-a") + && sim.rollupDay(dayB, "model-a").total_tokens === 110; + while (!healed() && runs < 200) { + sim.advanceRun(); + runs += 1; + } + assert.ok(healed(), `cross-day migration healed after ${runs} repair runs`); + assert.equal(sim.leaderboardTotal().get("model-a"), 110, "total still counts the session exactly once"); +}); + +// --------------------------------------------------------------------------- +// D. FIRST 30-day seed vs the leaderboard TOTAL (rollup UNION live tail). +// --------------------------------------------------------------------------- +test("D. first 30-day seed: account/profile/bounded immediate; TOTAL undercounts closed days until the seed-gap-prioritized repair completes", async () => { + const { cloudStates, sim } = scenarioSetup(); + // Preconditions (review scenario): rollup caught up (through = today), + // no trae-cn session states, no trae-cn rollup rows. The 120-day history + // below is OTHER sources; the cyclic repair position sits at the OLDEST + // day, so without prioritization the seed would wait a full cycle. + sim.buildAll(); + + // First TRAE sync: one real parser call seeds ~30 CLOSED days at once + // (one session per day, 110 tokens each). + const seedDays = 30; + const sessions = []; + for (let k = seedDays; k >= 1; k -= 1) { + sessions.push(traeSession(`seed-${k}`, DAY(k) + 10 * 60 * 60 * 1000)); + } + await syncSessions(cloudStates, { sessions, verifiedAtMs: V2 }); + assert.equal(cloudStates.size, seedDays, "the first sync seeded 30 canonical sessions"); + + // LIVE paths: everything except the leaderboard TOTAL is immediate. + const accountTotal = aggregateSessionStates(cloudStates).reduce((acc, r) => acc + r.total_tokens, 0); + assert.equal(accountTotal, seedDays * 110, "account usage total = 3300 immediately (profile reads the same RPC)"); + const weekRows = aggregateSessionStates(cloudStates, { + from: new Date(DAY(7)).toISOString(), + to: new Date(NOW_MS).toISOString(), + }); + assert.equal(weekRows.reduce((acc, r) => acc + r.total_tokens, 0), 7 * 110, "bounded leaderboard week includes its seed days immediately"); + + // TOTAL before any repair: rollup has no trae-cn rows and the live tail + // only covers [today 00:00, now) - every seed day is closed, so the TOTAL + // UNDERCOUNTS by the full seed. + const before = sim.leaderboardTotal().get("model-a") || 0; + assert.equal(before, 0, "leaderboard TOTAL undercounts the first seed entirely (all 30 days are closed days)"); + + // The seed-gap prioritization must repair from the SEED's oldest day (not + // the cyclic position 120 days back): after one run exactly the first + // 7 seed days are materialized. + sim.advanceRun(); + const afterRun1 = sim.leaderboardTotal().get("model-a") || 0; + assert.equal(afterRun1, 7 * 110, "run 1 repairs the seed's oldest 7 days (gap-first, 7-day chunk)"); + + // Deterministic completion: ceil(30/7) = 5 scheduled runs total. + let runs = 1; + while ((sim.leaderboardTotal().get("model-a") || 0) !== seedDays * 110 && runs < 50) { + sim.advanceRun(); + runs += 1; + } + assert.equal(sim.leaderboardTotal().get("model-a"), seedDays * 110, "TOTAL now equals the account/profile total"); + assert.equal(runs, 5, "a 30-day seed heals in ceil(30/7)=5 scheduled runs (not a full 120-day cycle)"); +}); + +// --------------------------------------------------------------------------- +// E. MULTI-USER first-seed isolation: the seed-gap check must be PER USER. +// User A already has a trae-cn rollup row for day X; User B's first sync +// seeds day X with NO row of their own. Day X must STILL be a seed gap +// for User B (gap identity is (user_id, day), not just day) so the +// prioritized repair window covers User B's seed instead of falling back +// to the ordinary cyclic position. +// --------------------------------------------------------------------------- +test("E. multi-user first-seed: User B's day X stays a seed gap despite User A's same-day trae-cn rollup row", async () => { + const { cloudStates: userA, cloudStatesB: userB, sim } = scenarioSetup(); + // Both users' rows live in the SAME deployed tables; the gap check must + // correlate coverage by user_id, not by day alone. + + const seedDay = DAY(10); + // User A: day X already covered by the materialized rollup. + await syncSessions(userA, { + sessions: [traeSession("usera-day10", seedDay + 5 * 60 * 60 * 1000)], + verifiedAtMs: V1, + }); + sim.buildAll(); + assert.ok(sim.rollupDay(seedDay, "model-a"), "User A already has a trae-cn rollup row for day X"); + assert.equal(sim.leaderboardTotal().get("model-a"), 110, "TOTAL covers User A only"); + + // User B: FIRST trae-cn sync seeds the SAME day, no user-b rollup row. + await syncSessions(userB, { + sessions: [traeSession("userb-day10", seedDay + 12 * 60 * 60 * 1000)], + verifiedAtMs: V2, + }); + assert.equal( + aggregateSessionStates(userB)[0].total_tokens, + 110, + "User B's live account total is immediate (session states)", + ); + assert.equal( + sim.leaderboardTotal().get("model-a"), + 110, + "TOTAL still undercounts User B before repair (closed day, no user-b rollup row)", + ); + + // The regression: day X is a gap FOR USER B even though User A's row + // covers the same (source, day). A day-only existence check (the 672f02a6 + // SQL) would see "covered" and silently demote User B to cyclic repair. + assert.equal(sim.minSeedGapDay(), seedDay, "day X is still a seed gap for User B (per-user coverage)"); + + // The prioritized repair must jump to User B's seed day, not the cyclic + // position 120 days back: one run materializes user-b's day-X row while + // User A's own row is rebuilt unchanged. + sim.advanceRun(); + const userBRow = sim.rollupDay(seedDay, "model-a", "user-b"); + assert.ok(userBRow, "user-b's own day-X rollup row is materialized"); + assert.equal(userBRow.total_tokens, 110, "user-b's row carries their seed usage"); + assert.equal(sim.rollupDay(seedDay, "model-a", "user-a").total_tokens, 110, "user-a's row is untouched"); + assert.equal( + sim.leaderboardTotal().get("model-a"), + 220, + "run 1 prioritized User B's seed day (110 A + 110 B)", + ); +}); + +// --------------------------------------------------------------------------- +// SQL contract pins: the deployed functions must actually implement the +// semantics mirrored above (the JS mirror cannot be allowed to drift). +// --------------------------------------------------------------------------- +function readRepoFile(rel) { + return fs.readFileSync(path.join(ROOT, rel), "utf8"); +} + +test("SQL pin: replace_v2 rebuilds closed days from leaderboard_hourly_dedup_v2 (which reads trae-cn session states)", () => { + const sql = readRepoFile("migrations/20260804043427_align-leaderboard-machine-clusters.sql"); + const replaceFn = sql.split("CREATE OR REPLACE FUNCTION public.leaderboard_rollup_daily_replace_v2")[1].split("$func$;")[0]; + assert.match(replaceFn, /DELETE FROM public\.tokentracker_leaderboard_rollup_daily_v2/, "a rebuild DELETEs the materialized day (downward/model corrections clean up)"); + assert.match(replaceFn, /FROM public\.leaderboard_hourly_dedup_v2\(v_day, v_day \+ interval '1 day'\)/, "days are re-aggregated from the dedup function"); + + const sessionStates = readRepoFile("migrations/20260817120000_account-session-states.sql"); + const dedupFn = sessionStates.split("CREATE OR REPLACE FUNCTION public.leaderboard_hourly_dedup_v2")[1].split("$func$;")[0]; + assert.match(dedupFn, /FROM tokentracker_account_session_states s/, "the dedup function's trae-cn branch reads canonical session states"); + assert.match(dedupFn, /s\.source = 'trae-cn'/); +}); + +test("SQL pin: advance_v2 (superseding definition) repairs 7 days per run, cyclically, with first-seed gap prioritization", () => { + // The 20260817120000 migration supersedes the 0804 advance_v2: pin the + // LATEST definition the deployment will actually leave in the database. + const sql = readRepoFile("migrations/20260817120000_account-session-states.sql"); + const advanceFn = sql.split("CREATE OR REPLACE FUNCTION public.leaderboard_rollup_daily_advance_v2")[1].split("$func$;")[0]; + assert.match(advanceFn, /interval '7 days'/, "7 days per scheduled run (catch-up AND repair)"); + assert.match(advanceFn, /v_min_day/, "repair starts at the oldest history day"); + assert.match(advanceFn, /v_repair_from := v_min_day/, "repair wraps to the oldest day after completing a cycle"); + // First-seed prioritization: jump to the earliest closed trae-cn day that + // has session states but no rollup row yet. + assert.match(advanceFn, /v_seed_gap_day/, "seed-gap day is computed"); + assert.match(advanceFn, /FROM public\.tokentracker_account_session_states s/, "gaps come from the canonical session states"); + assert.match(advanceFn, /NOT EXISTS[\s\S]*?r\.source = 'trae-cn'/, "a day is a gap only when NO trae-cn rollup row exists (stale values are NOT gaps)"); + assert.match(advanceFn, /r\.user_id = s\.user_id/, "gap coverage is PER USER: another user's same-day row must not hide this user's seed (scenario E)"); + assert.match(advanceFn, /v_repair_from := v_seed_gap_day/, "the repair window jumps to the seed gap"); + // Only the window POSITION changes: the per-run chunk stays 7 days. + assert.match(advanceFn, /v_seed_gap_day[\s\S]*?v_until := LEAST\([\s\S]*?interval '7 days'/, "gap-jump still repairs a bounded 7-day chunk"); +}); + +test("SQL pin: leaderboard TOTAL reads rollup UNION live tail; bounded windows read live", () => { + const sql = readRepoFile("migrations/20260804043427_align-leaderboard-machine-clusters.sql"); + const groupedFn = sql.split("CREATE OR REPLACE FUNCTION public.leaderboard_usage_grouped")[1].split("$func$;")[0]; + assert.match(groupedFn, /FROM public\.tokentracker_leaderboard_rollup_daily_v2 r/, "total = materialized rollup..."); + assert.match(groupedFn, /FROM public\.leaderboard_hourly_dedup_v2\(v_through, p_to\) t/, "...UNION the live tail"); + // The rollup+tail branch requires an all-time range; bounded windows take + // the live-only ELSE branch. + assert.match(groupedFn, /p_from < '1980-01-01'::timestamptz AND p_to >= v_through/); + + const refresh = readRepoFile("dashboard/edge-patches/tokentracker-leaderboard-refresh.ts"); + assert.match(refresh, /if \(period === "total"\)/, "only the total period advances the rollup"); + assert.match(refresh, /leaderboard_rollup_daily_advance_v2/); + assert.match(refresh, /from_day: "1970-01-01"/, "total uses the all-time sentinel that selects the rollup branch"); +}); + +test("SQL pin: profile / account views read the LIVE account RPC (corrections visible immediately)", () => { + const profile = readRepoFile("dashboard/edge-patches/tokentracker-leaderboard-profile.ts"); + assert.match(profile, /account_usage_grouped/, "profile uses the live account RPC"); + const summary = readRepoFile("dashboard/edge-patches/tokentracker-account-summary.ts"); + assert.match(summary, /account_usage_grouped/, "account summary uses the live account RPC"); + const rpc = readRepoFile("scripts/ops/account-usage-grouped-rpc.sql"); + assert.match(rpc, /FROM tokentracker_account_session_states s/, "the account RPC's trae-cn branch reads session states"); + assert.doesNotMatch(rpc, /leaderboard_rollup_daily_v2/, "the account RPC never reads the materialized rollup"); +}); diff --git a/test/source-metadata.test.js b/test/source-metadata.test.js new file mode 100644 index 000000000..560823989 --- /dev/null +++ b/test/source-metadata.test.js @@ -0,0 +1,54 @@ +/** + * Source-metadata classification tests. Verifies TRAE Work CN is an + * account-level source (case-insensitively) and that personal scope filters it + * out while preserving local sources. + */ +"use strict"; + +const test = require("node:test"); +const assert = require("node:assert/strict"); + +const { + getSourceScope, + isAccountLevelSource, + normalizeUsageScope, + filterRowsByUsageScope, + listExcludedSources, +} = require("../src/lib/source-metadata"); + +test("trae-cn is classified as an account-level source case-insensitively", () => { + assert.equal(getSourceScope("trae-cn"), "account"); + assert.equal(getSourceScope("TRAE-CN"), "account"); + assert.equal(getSourceScope(" Trae-Cn "), "account"); + assert.equal(isAccountLevelSource("trae-cn"), true); + // Pre-existing classification is preserved. + assert.equal(getSourceScope("cursor"), "account"); + assert.equal(getSourceScope("codex"), "local"); + assert.equal(getSourceScope("claude"), "local"); +}); + +test("personal scope filters account-level trae-cn rows while preserving local sources", () => { + const rows = [ + { source: "trae-cn", total_tokens: 10 }, + { source: "TRAE-CN", total_tokens: 20 }, + { source: "codex", total_tokens: 30 }, + { source: "claude", total_tokens: 40 }, + ]; + const personal = filterRowsByUsageScope(rows, "personal"); + assert.deepEqual(personal, [ + { source: "codex", total_tokens: 30 }, + { source: "claude", total_tokens: 40 }, + ]); + assert.equal(filterRowsByUsageScope(rows, "all").length, 4); + assert.equal(normalizeUsageScope("personal"), "personal"); +}); + +test("listExcludedSources reports trae-cn as excluded from personal scope", () => { + const excluded = listExcludedSources( + [{ source: "trae-cn" }, { source: "TRAE-CN" }, { source: "codex" }], + "personal", + ); + assert.deepEqual(excluded, [ + { source: "trae-cn", source_scope: "account", reason: "account_level_source" }, + ]); +}); diff --git a/test/status.test.js b/test/status.test.js index 78d5d0825..e7db8d4ba 100644 --- a/test/status.test.js +++ b/test/status.test.js @@ -474,3 +474,200 @@ test("status renders the Trae SOLO entitlement snapshot from Local State", async await fs.rm(tmp, { recursive: true, force: true }); } }); + + +// --------------------------------------------------------------------------- +// TRAE Work CN install detection + auth state (PR #474): a RESOLVABLE default +// path does not mean the app is installed - the macOS resolver always derives +// one. installed must mean the storage file actually EXISTS (same semantics +// as the sync path), and the auth state must distinguish not-signed-in / +// readable / malformed / unreadable. +// --------------------------------------------------------------------------- + +test("status trae-cn installed=false when the default path resolves but no storage file exists", async () => { + const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "tokentracker-status-traecn-")); + const prevHome = process.env.HOME; + const prevUserProfile = process.env.USERPROFILE; + const prevTraeCnHome = process.env.TOKENTRACKER_TRAE_CN_HOME; + const prevTraeCnUsage = process.env.TOKENTRACKER_TRAE_CN_USAGE; + const prevWrite = process.stdout.write; + + try { + process.env.HOME = tmp; + process.env.USERPROFILE = tmp; + delete process.env.TOKENTRACKER_TRAE_CN_HOME; + delete process.env.TOKENTRACKER_TRAE_CN_USAGE; + + // Nothing is created under the default TRAE SOLO CN path: the resolver + // still derives it, but the file is absent -> NOT installed. + let out = ""; + process.stdout.write = (chunk, enc, cb) => { + out += typeof chunk === "string" ? chunk : chunk.toString(enc || "utf8"); + if (typeof cb === "function") cb(); + return true; + }; + await cmdStatus(["--json"]); + const summary = JSON.parse(out); + assert.equal(summary.providers["trae-cn"].installed, false, "absent storage file must not report installed"); + + out = ""; + await cmdStatus(); + assert.doesNotMatch(out, /- Trae SOLO CN: usage sync/); + } finally { + process.stdout.write = prevWrite; + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + if (prevUserProfile === undefined) delete process.env.USERPROFILE; + else process.env.USERPROFILE = prevUserProfile; + if (prevTraeCnHome === undefined) delete process.env.TOKENTRACKER_TRAE_CN_HOME; + else process.env.TOKENTRACKER_TRAE_CN_HOME = prevTraeCnHome; + if (prevTraeCnUsage === undefined) delete process.env.TOKENTRACKER_TRAE_CN_USAGE; + else process.env.TOKENTRACKER_TRAE_CN_USAGE = prevTraeCnUsage; + await fs.rm(tmp, { recursive: true, force: true }); + } +}); + +test("status trae-cn installed=true with auth readable when the storage file exists", async () => { + const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "tokentracker-status-traecn-")); + const prevHome = process.env.HOME; + const prevUserProfile = process.env.USERPROFILE; + const prevTraeCnHome = process.env.TOKENTRACKER_TRAE_CN_HOME; + const prevTraeCnUsage = process.env.TOKENTRACKER_TRAE_CN_USAGE; + const prevWrite = process.stdout.write; + + try { + process.env.HOME = tmp; + process.env.USERPROFILE = tmp; + // Platform-independent: the default-path resolver is darwin-only, so point + // the env override at a synthetic install. Same semantics under test: + // storage.json exists -> installed=true, auth readable. + process.env.TOKENTRACKER_TRAE_CN_HOME = path.join(tmp, "trae-install"); + delete process.env.TOKENTRACKER_TRAE_CN_USAGE; + + const storageDir = path.join( + process.env.TOKENTRACKER_TRAE_CN_HOME, "User", "globalStorage", + ); + await fs.mkdir(storageDir, { recursive: true }); + await fs.writeFile( + path.join(storageDir, "storage.json"), + JSON.stringify({ "iCubeAuthInfo://icube.cloudide": { token: "fake.jwt.value", refreshToken: "synthetic" } }), + "utf8", + ); + + let out = ""; + process.stdout.write = (chunk, enc, cb) => { + out += typeof chunk === "string" ? chunk : chunk.toString(enc || "utf8"); + if (typeof cb === "function") cb(); + return true; + }; + await cmdStatus(["--json"]); + const summary = JSON.parse(out); + assert.equal(summary.providers["trae-cn"].installed, true); + assert.equal(summary.providers["trae-cn"].auth, "readable"); + assert.equal(summary.providers["trae-cn"].usage_opt_in, false, "opt-in stays off by default"); + + out = ""; + await cmdStatus(); + assert.match(out, /- Trae SOLO CN: usage sync off \(set TOKENTRACKER_TRAE_CN_USAGE=1 to enable\), auth readable/); + } finally { + process.stdout.write = prevWrite; + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + if (prevUserProfile === undefined) delete process.env.USERPROFILE; + else process.env.USERPROFILE = prevUserProfile; + if (prevTraeCnHome === undefined) delete process.env.TOKENTRACKER_TRAE_CN_HOME; + else process.env.TOKENTRACKER_TRAE_CN_HOME = prevTraeCnHome; + if (prevTraeCnUsage === undefined) delete process.env.TOKENTRACKER_TRAE_CN_USAGE; + else process.env.TOKENTRACKER_TRAE_CN_USAGE = prevTraeCnUsage; + await fs.rm(tmp, { recursive: true, force: true }); + } +}); + +test("status trae-cn installed=false for a custom TOKENTRACKER_TRAE_CN_HOME that does not exist", async () => { + const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "tokentracker-status-traecn-")); + const prevHome = process.env.HOME; + const prevUserProfile = process.env.USERPROFILE; + const prevTraeCnHome = process.env.TOKENTRACKER_TRAE_CN_HOME; + const prevWrite = process.stdout.write; + + try { + process.env.HOME = tmp; + process.env.USERPROFILE = tmp; + process.env.TOKENTRACKER_TRAE_CN_HOME = path.join(tmp, "no-such-install"); + + let out = ""; + process.stdout.write = (chunk, enc, cb) => { + out += typeof chunk === "string" ? chunk : chunk.toString(enc || "utf8"); + if (typeof cb === "function") cb(); + return true; + }; + await cmdStatus(["--json"]); + const summary = JSON.parse(out); + assert.equal(summary.providers["trae-cn"].installed, false, "env override pointing at nothing is not installed"); + } finally { + process.stdout.write = prevWrite; + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + if (prevUserProfile === undefined) delete process.env.USERPROFILE; + else process.env.USERPROFILE = prevUserProfile; + if (prevTraeCnHome === undefined) delete process.env.TOKENTRACKER_TRAE_CN_HOME; + else process.env.TOKENTRACKER_TRAE_CN_HOME = prevTraeCnHome; + await fs.rm(tmp, { recursive: true, force: true }); + } +}); + +test("status trae-cn distinguishes malformed vs unreadable storage", async () => { + const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "tokentracker-status-traecn-")); + const prevHome = process.env.HOME; + const prevUserProfile = process.env.USERPROFILE; + const prevTraeCnHome = process.env.TOKENTRACKER_TRAE_CN_HOME; + const prevWrite = process.stdout.write; + + try { + process.env.HOME = tmp; + process.env.USERPROFILE = tmp; + const traeCnHome = path.join(tmp, "trae-cn-data"); + process.env.TOKENTRACKER_TRAE_CN_HOME = traeCnHome; + + const capture = () => { + let out = ""; + process.stdout.write = (chunk, enc, cb) => { + out += typeof chunk === "string" ? chunk : chunk.toString(enc || "utf8"); + if (typeof cb === "function") cb(); + return true; + }; + return async () => { + await cmdStatus(["--json"]); + return JSON.parse(out); + }; + }; + + // Present but malformed JSON -> malformed (NOT not-signed-in, NOT unreadable). + const storageDir = path.join(traeCnHome, "User", "globalStorage"); + await fs.mkdir(storageDir, { recursive: true }); + await fs.writeFile(path.join(storageDir, "storage.json"), "{not json", "utf8"); + let run = capture(); + let summary = await run(); + assert.equal(summary.providers["trae-cn"].installed, true); + assert.equal(summary.providers["trae-cn"].auth, "malformed"); + + // A storage path that exists but cannot be READ as a file (EISDIR: the + // resolved storage.json is itself a directory) -> unreadable, the + // fail-closed IO state, distinct from malformed. + await fs.rm(path.join(storageDir, "storage.json"), { force: true }); + await fs.mkdir(path.join(storageDir, "storage.json"), { recursive: true }); + run = capture(); + summary = await run(); + assert.equal(summary.providers["trae-cn"].installed, true); + assert.equal(summary.providers["trae-cn"].auth, "unreadable"); + } finally { + process.stdout.write = prevWrite; + if (prevHome === undefined) delete process.env.HOME; + else process.env.HOME = prevHome; + if (prevUserProfile === undefined) delete process.env.USERPROFILE; + else process.env.USERPROFILE = prevUserProfile; + if (prevTraeCnHome === undefined) delete process.env.TOKENTRACKER_TRAE_CN_HOME; + else process.env.TOKENTRACKER_TRAE_CN_HOME = prevTraeCnHome; + await fs.rm(tmp, { recursive: true, force: true }); + } +}); diff --git a/test/sync-trae-cn.test.js b/test/sync-trae-cn.test.js new file mode 100644 index 000000000..07444765c --- /dev/null +++ b/test/sync-trae-cn.test.js @@ -0,0 +1,438 @@ +/** + * TRAE Work CN sync wiring tests. + * + * All synthetic: no live API, no real JWT. Exercises the non-background auto + * sync block (rolling 30-day range, storage-backed auth, single parser call) + * and the gating that keeps every background / all-local variant from fetching. + * The injected fetch + fixed now come through cmdSync's narrow context seam + * (context.traeCnFetchImpl / context.traeCnNowMs). + */ +"use strict"; + +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const test = require("node:test"); + +const { cmdSync } = require("../src/commands/sync"); +const { getSourceScope } = require("../src/lib/source-metadata"); + +const TRAE_CN_AUTH_KEY = "iCubeAuthInfo://icube.cloudide"; +const TRAE_CN_USAGE_URL = "https://api.trae.cn/trae/api/v1/pay/query_user_usage_group_by_session"; +const NOW_MS = 1_800_000_000_000; +const ROLLING_DAYS = 30 * 24 * 60 * 60; + +function tokenCountLine({ ts, totalTokens }) { + const usage = { + input_tokens: totalTokens, + cached_input_tokens: 0, + output_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: totalTokens, + }; + return JSON.stringify({ + type: "event_msg", + timestamp: ts, + payload: { + type: "token_count", + info: { last_token_usage: usage, total_token_usage: usage }, + }, + }); +} + +async function withTempTraeEnv(fn) { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "tokentracker-trae-sync-")); + const traeCnHome = path.join(home, "trae-cn-data"); + const saved = { + HOME: process.env.HOME, + USERPROFILE: process.env.USERPROFILE, + CODEX_HOME: process.env.CODEX_HOME, + CODE_HOME: process.env.CODE_HOME, + GEMINI_HOME: process.env.GEMINI_HOME, + OPENCODE_HOME: process.env.OPENCODE_HOME, + XDG_DATA_HOME: process.env.XDG_DATA_HOME, + TOKENTRACKER_REASONIX_HOME: process.env.TOKENTRACKER_REASONIX_HOME, + REASONIX_STATE_HOME: process.env.REASONIX_STATE_HOME, + TOKENTRACKER_DEVICE_TOKEN: process.env.TOKENTRACKER_DEVICE_TOKEN, + TOKENTRACKER_INSFORGE_BASE_URL: process.env.TOKENTRACKER_INSFORGE_BASE_URL, + TOKENTRACKER_OPENCLAW_HOME: process.env.TOKENTRACKER_OPENCLAW_HOME, + TOKENTRACKER_OPENCLAW_AGENT_ID: process.env.TOKENTRACKER_OPENCLAW_AGENT_ID, + TOKENTRACKER_OPENCLAW_PREV_SESSION_ID: process.env.TOKENTRACKER_OPENCLAW_PREV_SESSION_ID, + TOKENTRACKER_OPENCLAW_SESSION_KEY: process.env.TOKENTRACKER_OPENCLAW_SESSION_KEY, + TOKENTRACKER_TRAE_CN_HOME: process.env.TOKENTRACKER_TRAE_CN_HOME, + TOKENTRACKER_TRAE_CN_USAGE: process.env.TOKENTRACKER_TRAE_CN_USAGE, + TOKENTRACKER_WSL_MODE: process.env.TOKENTRACKER_WSL_MODE, + }; + try { + process.env.HOME = home; + process.env.USERPROFILE = home; + process.env.CODEX_HOME = path.join(home, ".codex"); + process.env.CODE_HOME = path.join(home, ".code"); + process.env.GEMINI_HOME = path.join(home, ".gemini"); + process.env.OPENCODE_HOME = path.join(home, ".opencode"); + process.env.XDG_DATA_HOME = path.join(home, ".local", "share"); + process.env.TOKENTRACKER_OPENCLAW_HOME = path.join(home, ".openclaw"); + process.env.TOKENTRACKER_TRAE_CN_HOME = traeCnHome; + process.env.TOKENTRACKER_TRAE_CN_USAGE = "1"; + delete process.env.TOKENTRACKER_REASONIX_HOME; + delete process.env.REASONIX_STATE_HOME; + delete process.env.TOKENTRACKER_DEVICE_TOKEN; + delete process.env.TOKENTRACKER_INSFORGE_BASE_URL; + delete process.env.TOKENTRACKER_OPENCLAW_AGENT_ID; + delete process.env.TOKENTRACKER_OPENCLAW_PREV_SESSION_ID; + delete process.env.TOKENTRACKER_OPENCLAW_SESSION_KEY; + delete process.env.TOKENTRACKER_WSL_MODE; + fs.mkdirSync(path.join(traeCnHome, "User", "globalStorage"), { recursive: true }); + return await fn({ home, traeCnHome }); + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + fs.rmSync(home, { recursive: true, force: true }); + } +} + +function writeTraeCnStorage(traeCnHome, auth = { token: "fake-jwt-token", refreshToken: "synthetic" }) { + fs.writeFileSync( + path.join(traeCnHome, "User", "globalStorage", "storage.json"), + JSON.stringify({ [TRAE_CN_AUTH_KEY]: auth }), + ); +} + +function readSessionStateRows(home) { + return readQueueRows(home).filter((row) => row.kind === "account_session_state"); +} + +function readQueueRows(home) { + const queuePath = path.join(home, ".tokentracker", "tracker", "queue.jsonl"); + if (!fs.existsSync(queuePath)) return []; + return fs.readFileSync(queuePath, "utf8") + .trim() + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line)); +} + +function readCursors(home) { + const cursorsPath = path.join(home, ".tokentracker", "tracker", "cursors.json"); + if (!fs.existsSync(cursorsPath)) return {}; + return JSON.parse(fs.readFileSync(cursorsPath, "utf8")); +} + +function traeFetchOnce(body, requests = []) { + return async (url, options) => { + requests.push({ url, options }); + return { status: 200, ok: true, json: async () => body }; + }; +} + +function halfHourBucket(epochSeconds) { + const d = new Date(epochSeconds * 1000); + const half = d.getUTCMinutes() >= 30 ? 30 : 0; + return new Date( + Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate(), d.getUTCHours(), half, 0, 0), + ).toISOString(); +} + +function sampleSession(usageTime) { + return { + session_id: "s1", + model_name: "doubao-pro", + usage_time: usageTime, + input_token: 100, + output_token: 10, + cache_read_token: 0, + cache_write_token: 0, + }; +} + +test("non-background auto trae-cn sync fetches the exact rolling range and queues the row", async () => { + await withTempTraeEnv(async ({ home, traeCnHome }) => { + writeTraeCnStorage(traeCnHome); + const endTime = Math.floor(NOW_MS / 1000); + const startTime = Math.max(1, endTime - ROLLING_DAYS); + const usageTime = startTime + 3600; + const requests = []; + const fetchImpl = traeFetchOnce( + { user_usage_group_by_sessions: [sampleSession(usageTime)], total: 1 }, + requests, + ); + + await cmdSync(["--auto", "--source=trae-cn"], { traeCnFetchImpl: fetchImpl, traeCnNowMs: NOW_MS }); + + assert.equal(requests.length, 1, "single page fetch"); + assert.equal(requests[0].url, TRAE_CN_USAGE_URL); + assert.equal(requests[0].options.method, "POST"); + assert.equal(requests[0].options.headers["Content-Type"], "application/json"); + assert.equal(requests[0].options.headers.Authorization, "Cloud-IDE-JWT fake-jwt-token"); + const body = JSON.parse(requests[0].options.body); + assert.deepEqual(body.usage_type, [7]); + assert.equal(body.start_time, startTime); + assert.equal(body.end_time, endTime); + assert.equal(body.page_num, 1); + assert.equal(body.page_size, 20); + + const traeRows = readQueueRows(home).filter( + (row) => row.source === "trae-cn" && !row.kind, + ); + assert.equal(traeRows.length, 1); + assert.equal(traeRows[0].model, "doubao-pro"); + assert.equal(traeRows[0].input_tokens, 100); + assert.equal(traeRows[0].output_tokens, 10); + assert.equal(traeRows[0].total_tokens, 110); + assert.equal(traeRows[0].conversation_count, 1); + assert.equal(traeRows[0].hour_start, halfHourBucket(usageTime)); + + // The sync appends exactly one canonical session-state observation + // after the bucket row (queue order: a device's rows land before the + // states describing them). + const queueLines = fs + .readFileSync(path.join(home, ".tokentracker", "tracker", "queue.jsonl"), "utf8") + .trim() + .split("\n"); + assert.equal(queueLines.length, 2, "bucket row then session state"); + const states = readSessionStateRows(home); + assert.equal(states.length, 1); + assert.equal(states[0].source, "trae-cn"); + assert.equal(states[0].session_id, "s1"); + assert.equal(states[0].model, "doubao-pro"); + assert.equal(states[0].bucket_start, halfHourBucket(usageTime)); + assert.equal(states[0].total_tokens, 110); + assert.equal( + Date.parse(states[0].snapshot_verified_at), + NOW_MS, + "logical fetch stamp is the fixed sync clock, not the enqueue moment", + ); + + const cursors = readCursors(home); + assert.equal(cursors.traeCn.version, 1); + assert.equal(cursors.traeCn.sessions["s1"].totals.input_tokens, 100); + assert.equal(cursors.traeCn.sessions["s1"].bucketStart, halfHourBucket(usageTime)); + assert.equal(getSourceScope("trae-cn"), "account", "source-scope account"); + }); +}); + +test("repeated fixed-now sync is idempotent (no token growth / no new queue row)", async () => { + await withTempTraeEnv(async ({ home, traeCnHome }) => { + writeTraeCnStorage(traeCnHome); + const endTime = Math.floor(NOW_MS / 1000); + const fetchImpl = traeFetchOnce({ + user_usage_group_by_sessions: [sampleSession(endTime - 86400)], + total: 1, + }); + await cmdSync(["--auto", "--source=trae-cn"], { traeCnFetchImpl: fetchImpl, traeCnNowMs: NOW_MS }); + + const queuePath = path.join(home, ".tokentracker", "tracker", "queue.jsonl"); + const before = fs.readFileSync(queuePath, "utf8"); + await cmdSync(["--auto", "--source=trae-cn"], { traeCnFetchImpl: fetchImpl, traeCnNowMs: NOW_MS }); + assert.equal(fs.readFileSync(queuePath, "utf8"), before, "queue unchanged on repeat"); + + const traeRows = readQueueRows(home).filter( + (row) => row.source === "trae-cn" && !row.kind, + ); + assert.equal(traeRows.length, 1); + assert.equal(traeRows[0].total_tokens, 110); + // The repeat re-appends nothing: unchanged sessions emit no session + // state observation, so the queue stays byte-identical. + assert.equal(readSessionStateRows(home).length, 1); + }); +}); + +test("background and all-local syncs never call the TRAE CN fetch", async () => { + await withTempTraeEnv(async ({ home, traeCnHome }) => { + writeTraeCnStorage(traeCnHome); + let fetchCalls = 0; + const fetchImpl = async () => { + fetchCalls += 1; + throw new Error("should not fetch"); + }; + const opts = { traeCnFetchImpl: fetchImpl, traeCnNowMs: NOW_MS }; + + await cmdSync(["--auto", "--background"], opts); + assert.equal(fetchCalls, 0); + await cmdSync(["--auto", "--background", "--all-local-sources"], opts); + assert.equal(fetchCalls, 0); + await cmdSync(["--auto", "--background", "--source=trae-cn"], opts); + assert.equal(fetchCalls, 0); + + assert.equal( + readQueueRows(home).some((row) => row.source === "trae-cn"), + false, + ); + }); +}); + +test("TRAE CN usage read stays off unless TOKENTRACKER_TRAE_CN_USAGE opts in", async () => { + await withTempTraeEnv(async ({ home }) => { + writeTraeCnStorage(path.join(home, "trae-cn-data")); + let fetchCalls = 0; + const fetchImpl = async () => { + fetchCalls += 1; + throw new Error("should not fetch"); + }; + + const savedFlag = process.env.TOKENTRACKER_TRAE_CN_USAGE; + delete process.env.TOKENTRACKER_TRAE_CN_USAGE; + try { + await cmdSync(["--auto", "--source=trae-cn"], { + traeCnFetchImpl: fetchImpl, + traeCnNowMs: NOW_MS, + }); + } finally { + if (savedFlag !== undefined) process.env.TOKENTRACKER_TRAE_CN_USAGE = savedFlag; + } + + assert.equal(fetchCalls, 0, "no request without the opt-in flag"); + assert.equal(readQueueRows(home).some((row) => row.source === "trae-cn"), false); + assert.equal(readCursors(home).traeCn, undefined); + }); +}); + +test("an absent TRAE CN storage file does not call the fetch", async () => { + await withTempTraeEnv(async ({ home }) => { + let fetchCalls = 0; + const fetchImpl = async () => { + fetchCalls += 1; + throw new Error("should not fetch"); + }; + + await cmdSync(["--auto", "--source=trae-cn"], { + traeCnFetchImpl: fetchImpl, + traeCnNowMs: NOW_MS, + }); + + assert.equal(fetchCalls, 0); + assert.equal(readQueueRows(home).some((row) => row.source === "trae-cn"), false); + assert.equal(readCursors(home).traeCn, undefined); + }); +}); + +test("perpetually over-capacity TRAE CN fetch (split depth exhausted) preserves the prior queue and cursor bytes", async () => { + await withTempTraeEnv(async ({ home, traeCnHome }) => { + writeTraeCnStorage(traeCnHome); + const initialFetch = traeFetchOnce({ + user_usage_group_by_sessions: [sampleSession(Math.floor(NOW_MS / 1000) - 86400)], + total: 1, + }); + await cmdSync(["--auto", "--source=trae-cn"], { + traeCnFetchImpl: initialFetch, + traeCnNowMs: NOW_MS, + }); + + const queuePath = path.join(home, ".tokentracker", "tracker", "queue.jsonl"); + const cursorsPath = path.join(home, ".tokentracker", "tracker", "cursors.json"); + const queueBefore = fs.readFileSync(queuePath, "utf8"); + const traeCursorBefore = JSON.stringify(JSON.parse(fs.readFileSync(cursorsPath, "utf8")).traeCn); + const overCapacityRequests = []; + const overCapacityFetch = traeFetchOnce( + { + user_usage_group_by_sessions: [sampleSession(Math.floor(NOW_MS / 1000) - 86400)], + total: 2001, + }, + overCapacityRequests, + ); + + await cmdSync(["--auto", "--source=trae-cn"], { + traeCnFetchImpl: overCapacityFetch, + traeCnNowMs: NOW_MS, + }); + + // Every window (down to the split-depth ceiling) declares 2001 rows, so + // the fetch fails closed after probing the bounded split tree. + assert.ok(overCapacityRequests.length > 1, "window splitting was attempted"); + assert.ok(overCapacityRequests.length <= 511, "probe count stays within the depth ceiling"); + assert.equal(fs.readFileSync(queuePath, "utf8"), queueBefore, "prior queue remains byte-identical"); + const traeCursorAfter = JSON.stringify(JSON.parse(fs.readFileSync(cursorsPath, "utf8")).traeCn); + assert.equal(traeCursorAfter, traeCursorBefore, "prior TRAE cursor remains byte-identical"); + }); +}); + +test("over-capacity TRAE CN window is split and the staggered halves import atomically", async () => { + await withTempTraeEnv(async ({ home, traeCnHome }) => { + writeTraeCnStorage(traeCnHome); + const endTime = Math.floor(NOW_MS / 1000); + const startTime = Math.max(1, endTime - ROLLING_DAYS); + const mid = startTime + Math.floor((endTime - startTime) / 2); + const requests = []; + const fetchImpl = async (url, options) => { + requests.push({ url, options }); + const body = JSON.parse(options.body); + const overCapacity = body.start_time === startTime && body.end_time === endTime; + if (overCapacity) { + return { status: 200, ok: true, json: async () => ({ user_usage_group_by_sessions: [], total: 2001 }) }; + } + const side = body.start_time === startTime ? "left" : "right"; + return { + status: 200, + ok: true, + json: async () => ({ + user_usage_group_by_sessions: [ + { + ...sampleSession(body.end_time), + session_id: `split-${side}`, + }, + ], + total: 1, + }), + }; + }; + + await cmdSync(["--auto", "--source=trae-cn"], { traeCnFetchImpl: fetchImpl, traeCnNowMs: NOW_MS }); + + // Full window probes once, then the two staggered halves each paginate. + assert.deepEqual( + requests.map(({ options }) => { + const body = JSON.parse(options.body); + return [body.start_time, body.end_time]; + }), + [[startTime, endTime], [startTime, mid], [mid + 1, endTime]], + ); + const traeRows = readQueueRows(home).filter( + (row) => row.source === "trae-cn" && !row.kind, + ); + assert.equal(traeRows.length, 2, "both halves queue their row"); + const cursorSessions = readCursors(home).traeCn.sessions; + assert.deepEqual(Object.keys(cursorSessions).sort(), ["split-left", "split-right"]); + // Both halves emit their canonical session observations (one per + // session), all stamped with the SAME logical fetch clock. + const states = readSessionStateRows(home); + assert.deepEqual( + states.map((r) => r.session_id).sort(), + ["split-left", "split-right"], + ); + for (const state of states) { + assert.equal(Date.parse(state.snapshot_verified_at), NOW_MS); + } + }); +}); + +test("failed TRAE CN fetch skips parser/queue/cursor mutation while other providers continue", async () => { + await withTempTraeEnv(async ({ home, traeCnHome }) => { + writeTraeCnStorage(traeCnHome); + const fetchImpl = async () => { + throw new Error("network down"); + }; + const opts = { traeCnFetchImpl: fetchImpl, traeCnNowMs: NOW_MS }; + + // Scoped run: failing fetch must not create trae-cn queue rows or cursor state. + await cmdSync(["--auto", "--source=trae-cn"], opts); + assert.equal(readQueueRows(home).filter((row) => row.source === "trae-cn").length, 0); + assert.equal(readCursors(home).traeCn, undefined); + + // Full run: an unrelated local provider still parses while trae-cn is skipped. + const codexHome = process.env.CODEX_HOME; + fs.mkdirSync(path.join(codexHome, "sessions", "2026", "06", "30"), { recursive: true }); + fs.writeFileSync( + path.join(codexHome, "sessions", "2026", "06", "30", "rollout-2026-06-30T00-00-00-aaaa-ffff.jsonl"), + tokenCountLine({ ts: "2026-06-30T00:00:00.000Z", totalTokens: 31 }) + "\n", + "utf8", + ); + await cmdSync(["--auto"], opts); + const rows = readQueueRows(home); + assert.ok(rows.some((row) => row.source === "codex"), "unrelated provider continued"); + assert.ok(!rows.some((row) => row.source === "trae-cn"), "no trae-cn rows after failure"); + assert.equal(readCursors(home).traeCn, undefined); + }); +}); diff --git a/test/sync-upload-batching.test.js b/test/sync-upload-batching.test.js new file mode 100644 index 000000000..364a867d9 --- /dev/null +++ b/test/sync-upload-batching.test.js @@ -0,0 +1,268 @@ +/** + * Production-path batching regression for account_session_state uploads. + * + * Real chain under test (only the network boundary is mocked): + * + * queue.jsonl -> readQueueBatch() -> drainQueueToCloud() -> fetch(ingest) + * + * The ingest edge (dashboard/edge-patches/tokentracker-ingest.ts) rejects a + * request with > 500 account_session_states (HTTP 400) and > 500 hourly + * buckets. Before this regression, session-state records did NOT count + * toward readQueueBatch's per-batch record cap, so a states-heavy queue + * (e.g. a fresh device's first 30-day TRAE sync) produced ONE oversized + * request; the 400 left the queue offset untouched, and every retry re-read + * the identical oversized batch - a PERMANENT upload failure. + * + * These tests enforce the fixed contract instead: + * - every request carries <= 500 states (with the default batchSize of + * 200, batching follows the existing design: <= 200 records per batch), + * - batching makes bounded forward progress and loses nothing, + * - nextOffset only crosses records actually included in the request, + * - a states-only batch uploads normally, + * - an HTTP failure freezes the offset at the failed batch; a retry + * resumes from exactly there (no skip, no permanent 400), + * - observations stay canonical per request (no duplicate session ids in + * one request), keeping cloud LWW replay idempotent. + */ +"use strict"; + +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const test = require("node:test"); + +const { readQueueBatch, drainQueueToCloud } = require("../src/commands/sync"); + +const INGEST_STATE_LIMIT = 500; // edge contract: account_session_states.length <= 500 +const BATCH_SIZE = 200; // production default (cmdSync autoUploadDecision.batchSize) + +function tempQueue(prefix) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + return { dir, queuePath: path.join(dir, "queue.jsonl"), queueStatePath: path.join(dir, "queue-state.json") }; +} + +function stateRecord(sessionId, { model = "model-a", bucketStart = "2027-01-10T10:00:00.000Z", input = 100, verifiedAt = "2027-01-15T00:00:00.000Z" } = {}) { + return { + kind: "account_session_state", + source: "trae-cn", + session_id: sessionId, + model, + bucket_start: bucketStart, + input_tokens: input, + output_tokens: 10, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: input + 10, + snapshot_verified_at: verifiedAt, + }; +} + +function bucketRow(hourStart, total = 50) { + return { source: "trae-cn", model: "model-a", hour_start: hourStart, total_tokens: total, input_tokens: total, output_tokens: 0 }; +} + +/** + * Mock ingest that mirrors the REAL edge contract (payload caps + response + * shape). Records every request's buckets/states for assertions. + */ +function installIngestMock({ failFirstNBatches = 0 } = {}) { + const calls = []; + const realFetch = globalThis.fetch; + let failuresLeft = failFirstNBatches; + globalThis.fetch = async (_url, init) => { + const body = JSON.parse(init.body); + const buckets = Array.isArray(body.hourly) ? body.hourly : []; + const states = Array.isArray(body.account_session_states) ? body.account_session_states : []; + if (buckets.length > 500 || states.length > INGEST_STATE_LIMIT) { + calls.push({ buckets, states, rejected: true }); + return { + ok: false, + status: 400, + headers: { get: () => null }, + text: () => Promise.resolve(JSON.stringify({ error: "Too many buckets or account session states" })), + }; + } + if (failuresLeft > 0) { + failuresLeft -= 1; + calls.push({ buckets, states, failed: true }); + return { + ok: false, + status: 503, + headers: { get: () => null }, + text: () => Promise.resolve(JSON.stringify({ error: "transient" })), + }; + } + calls.push({ buckets, states }); + return { + ok: true, + status: 200, + headers: { get: () => null }, + text: () => Promise.resolve(JSON.stringify({ inserted: buckets.length + states.length, skipped: 0 })), + }; + }; + return { + calls, + restore() { + globalThis.fetch = realFetch; + }, + }; +} + +function writeQueue(queuePath, records) { + fs.writeFileSync(queuePath, records.map((r) => JSON.stringify(r)).join("\n") + "\n"); +} + +test("1200 states-only queue: bounded batches, every request <= 500, nothing lost, no permanent 400", async (t) => { + const { dir, queuePath, queueStatePath } = tempQueue("tokentracker-batching-"); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const total = 1200; + writeQueue(queuePath, Array.from({ length: total }, (_, i) => stateRecord(`s-${i}`))); + + const ingest = installIngestMock(); + t.after(() => ingest.restore()); + + // Production drain defaults: maxBatches=5 per invocation, batchSize=200. + let drained = 0; + let guard = 0; + while (drained < total && guard < 20) { + const result = await drainQueueToCloud({ baseUrl: "https://t.example", deviceToken: "tok", queuePath, queueStatePath, maxBatches: 5, batchSize: BATCH_SIZE }); + const state = JSON.parse(fs.readFileSync(queueStatePath, "utf8")); + drained = state.offset >= fs.statSync(queuePath).size ? total : drained + result.inserted + result.skipped; + guard += 1; + if (result.batches === 0) break; + } + + // No request ever exceeded the ingest cap... + assert.ok(ingest.calls.length > 0, "requests were made"); + for (const call of ingest.calls) { + assert.ok(!call.rejected, `request rejected by ingest cap (states=${call.states.length})`); + assert.ok(call.states.length <= INGEST_STATE_LIMIT, `states per request <= ${INGEST_STATE_LIMIT}`); + assert.ok(call.buckets.length <= 500, "buckets per request <= 500"); + } + // ...all 1200 distinct sessions were delivered exactly once... + const delivered = ingest.calls.flatMap((c) => c.states.map((s) => s.session_id)); + assert.equal(delivered.length, total, "no loss, no duplicate delivery"); + assert.equal(new Set(delivered).size, total, "every session delivered exactly once"); + // ...no request duplicated a session id (canonical per request)... + for (const call of ingest.calls) { + assert.equal(new Set(call.states.map((s) => s.session_id)).size, call.states.length, "one observation per session per request"); + } + // ...and the drain made bounded forward progress to the end of the queue. + const finalState = JSON.parse(fs.readFileSync(queueStatePath, "utf8")); + assert.ok(finalState.offset >= fs.statSync(queuePath).size, "offset reached the end of the queue"); +}); + +test("states-only batch uploads normally (no bucket rows required)", async (t) => { + const { dir, queuePath, queueStatePath } = tempQueue("tokentracker-states-only-"); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + writeQueue(queuePath, Array.from({ length: 10 }, (_, i) => stateRecord(`s-${i}`))); + + const ingest = installIngestMock(); + t.after(() => ingest.restore()); + + const result = await drainQueueToCloud({ baseUrl: "https://t.example", deviceToken: "tok", queuePath, queueStatePath, maxBatches: 5, batchSize: BATCH_SIZE }); + assert.ok(result.batches >= 1, "states-only queue still uploads"); + assert.equal(ingest.calls.filter((c) => !c.failed && !c.rejected).length, 1); + assert.equal(ingest.calls[0].buckets.length, 0, "no bucket rows in a states-only queue"); + assert.equal(ingest.calls[0].states.length, 10); +}); + +test("mid-drain HTTP failure: offset freezes at the failed batch, retry resumes exactly there", async (t) => { + const { dir, queuePath, queueStatePath } = tempQueue("tokentracker-recover-"); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const total = 450; // 3 batches of 200/200/50 at batchSize=200 + const records = Array.from({ length: total }, (_, i) => stateRecord(`s-${i}`)); + writeQueue(queuePath, records); + // Cumulative byte offset after the first N records (file bytes, incl. \n). + const offsetAfter = (n) => records.slice(0, n).reduce((acc, r) => acc + JSON.stringify(r).length + 1, 0); + + // First drain: batch 1 succeeds, batch 2 fails (503), drain aborts. + // Custom mock: only the SECOND request fails. + const realFetch = globalThis.fetch; + const calls = []; + let requestIndex = 0; + globalThis.fetch = async (_url, init) => { + const body = JSON.parse(init.body); + const states = Array.isArray(body.account_session_states) ? body.account_session_states : []; + const buckets = Array.isArray(body.hourly) ? body.hourly : []; + requestIndex += 1; + const fail = requestIndex === 2; // first batch OK, second batch fails + calls.push({ buckets, states, failed: fail }); + if (fail) { + return { ok: false, status: 503, headers: { get: () => null }, text: () => Promise.resolve(JSON.stringify({ error: "transient" })) }; + } + return { ok: true, status: 200, headers: { get: () => null }, text: () => Promise.resolve(JSON.stringify({ inserted: states.length, skipped: 0 })) }; + }; + t.after(() => { + globalThis.fetch = realFetch; + }); + + await assert.rejects( + () => drainQueueToCloud({ baseUrl: "https://t.example", deviceToken: "tok", queuePath, queueStatePath, maxBatches: 5, batchSize: BATCH_SIZE }), + (err) => err.status === 503, + ); + // Exactly one batch landed; the offset crossed ONLY that batch's records. + assert.equal(calls.filter((c) => !c.failed).length, 1); + const stateAfterFailure = JSON.parse(fs.readFileSync(queueStatePath, "utf8")); + assert.equal(stateAfterFailure.offset, offsetAfter(200), "offset crossed exactly the first batch's 200 records"); + + // Retry: everything from the failed batch onward now succeeds. + globalThis.fetch = async (_url, init) => { + const body = JSON.parse(init.body); + const states = Array.isArray(body.account_session_states) ? body.account_session_states : []; + calls.push({ buckets: [], states }); + return { ok: true, status: 200, headers: { get: () => null }, text: () => Promise.resolve(JSON.stringify({ inserted: states.length, skipped: 0 })) }; + }; + await drainQueueToCloud({ baseUrl: "https://t.example", deviceToken: "tok", queuePath, queueStatePath, maxBatches: 5, batchSize: BATCH_SIZE }); + + const successful = calls.filter((c) => !c.failed); + const delivered = successful.flatMap((c) => c.states.map((s) => s.session_id)); + assert.equal(new Set(delivered).size, total, "every session eventually delivered exactly once across the failure boundary"); + assert.equal(delivered.length, total, "no loss and no duplicate delivery"); + const finalState = JSON.parse(fs.readFileSync(queueStatePath, "utf8")); + assert.ok(finalState.offset >= fs.statSync(queuePath).size, "retry drains to the end"); +}); + +test("mixed queue: buckets and states both count toward the batch cap; nextOffset crosses only included records", async (t) => { + const { dir, queuePath, queueStatePath } = tempQueue("tokentracker-mixed-"); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + // 150 buckets then 100 states: with the cap counting BOTH record kinds, + // the first batch must stop at 200 records total (150 buckets + 50 + // states), and the second batch carries the remaining 50 states. + const records = [ + // Distinct half-hours: bucket rows are keyed by (source, model, hour) + // and would otherwise legitimately dedup in place. + ...Array.from({ length: 150 }, (_, i) => + bucketRow(new Date(Date.parse("2027-01-10T00:00:00.000Z") + i * 30 * 60 * 1000).toISOString()), + ), + ...Array.from({ length: 100 }, (_, i) => stateRecord(`s-${i}`)), + ]; + writeQueue(queuePath, records); + + const first = await readQueueBatch(queuePath, 0, BATCH_SIZE); + assert.equal(first.buckets.length, 150); + assert.equal(first.sessionStates.length, 50, "states fill the remainder of the capped batch"); + assert.equal(first.sessionStates[0].session_id, "s-0"); + + const second = await readQueueBatch(queuePath, first.nextOffset, BATCH_SIZE); + assert.equal(second.buckets.length, 0); + assert.equal(second.sessionStates.length, 50, "the tail states come back in the next batch"); + assert.equal(second.sessionStates[0].session_id, "s-50"); +}); + +test("duplicate observations of one session: last-wins within a batch, never duplicated inside one request", async (t) => { + const { dir, queuePath, queueStatePath } = tempQueue("tokentracker-dup-"); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const older = stateRecord("dup-1", { input: 100, verifiedAt: "2027-01-15T00:00:00.000Z" }); + const newer = stateRecord("dup-1", { input: 60, verifiedAt: "2027-01-16T00:00:00.000Z" }); + writeQueue(queuePath, [older, newer, stateRecord("other-1")]); + + const batch = await readQueueBatch(queuePath, 0, BATCH_SIZE); + const ids = batch.sessionStates.map((s) => s.session_id); + assert.equal(ids.length, 2); + assert.equal(new Set(ids).size, 2, "no duplicate session id inside one request"); + const dup = batch.sessionStates.find((s) => s.session_id === "dup-1"); + assert.equal(dup.input_tokens, 60, "last-wins: the newer observation is the one delivered"); +}); diff --git a/test/trae-cn-config.test.js b/test/trae-cn-config.test.js new file mode 100644 index 000000000..422db412f --- /dev/null +++ b/test/trae-cn-config.test.js @@ -0,0 +1,962 @@ +/** + * TRAE Work CN config tests. Synthetic data only — no real JWTs / fixtures. + * + * The encrypted-tc-v5 coverage relies on a frozen static fixture (FIXTURE_B64) + * produced once with the documented KDF + a fixed 32-byte salt, plus an + * independent encryptor in this file that writes its own KDF. Neither imports + * production `deriveTraeCnKeyIv`, so a broken production KDF fails the + * decrypt assertions instead of being hidden by test-only encryption. + */ +"use strict"; + +const test = require("node:test"); +const assert = require("node:assert/strict"); +const crypto = require("node:crypto"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); + +const { + TRAE_CN_HOME_ENV, + TRAE_CN_AUTH_KEY, + TRAE_CN_USAGE_URL, + TRAE_CN_USAGE_DELAY_MS, + resolveTraeCnHome, + resolveTraeCnStoragePath, + decryptTraeCnBase64, + parseTraeCnAuthValue, + readTraeCnAuthFromStorage, + extractTraeCnToken, + fetchTraeCnUsagePage, + fetchTraeCnUsage, + fetchTraeCnUsageWindowed, + fetchTraeCnUsageWithAuth, +} = require("../src/lib/trae-cn-config"); + +const CN_APP_DIR = "TRAE SOLO CN"; + +// Frozen fixture: encrypted with the documented KDF, fixed 32-byte salt. +const FIXTURE_B64 = + "dGMFEAAA3q2+796tvu/erb7v3q2+796tvu/erb7v3q2+796tvu/43q0BL3Fo+gI3mEsCEdP2c66sO1HLQB7bKIlHUL7xJg0PVJHLSzn/gDVQmH7Ksh0wflKSFHtHgz7rwynP4aOOx3CPgDR5lQ8neR7CWouH94Dh99HlsWwqo29etbVBVbTU18KIS74UPyA5ASAWnYHqKPQSfv8uCn6GOMfOyJPNPMI6Yz666PBNVFTastLi70ThZ6Lb/Gs+bjyTDrQnSfjDnDJPinjJejWWlboznbnME/rNkmc2LLMftj8jGqorIq4="; +const FIXTURE_TOKEN = "fake.jwt.eyJzdWIiOiJ0ZXN0IiwiZXhwIjo5OTk5OTk5OTk5fQ.sig"; +const FIXTURE_AUTH = { + token: FIXTURE_TOKEN, + refreshToken: "synthetic-refresh", + userId: "user-123", +}; + +// Independent encryptor — own copy of the documented constants + KDF. +const TEST_MAGIC = Buffer.from([0x74, 0x63, 0x05, 0x10, 0x00, 0x00]); +const TEST_JG = Buffer.from([ + 82, 9, 106, 213, 48, 54, 165, 56, 191, 64, 163, 158, 129, 243, 215, 251, 124, 227, 57, 130, + 155, 47, 255, 135, 52, 142, 67, 68, 196, 222, 233, 203, 84, 123, 148, 50, 166, 194, 35, 61, + 238, 76, 149, 11, 66, 250, 195, 78, 8, 46, 161, 102, 40, 217, 36, 178, 118, 91, 162, 73, + 109, 139, 209, 37, +]); +const TEST_KG = Buffer.from([ + 31, 221, 168, 51, 136, 7, 199, 49, 177, 18, 16, 89, 39, 128, 236, 95, 96, 81, 127, 169, 25, + 181, 74, 13, 45, 229, 122, 159, 147, 201, 156, 239, 160, 224, 59, 77, 174, 42, 245, 176, + 200, 235, 187, 60, 131, 83, 153, 97, 23, 43, 4, 126, 186, 119, 214, 38, 225, 105, 20, 99, + 85, 33, 12, 125, +]); + +function testKdfSecret() { + const secret = Buffer.alloc(64); + for (let i = 0; i < 64; i++) secret[i] = TEST_JG[i] ^ TEST_KG[i]; + return secret; +} + +function testDerive(salt) { + const kdfBuf = Buffer.concat([crypto.createHash("sha512").update(salt).digest(), testKdfSecret()]); + // Vendor tc-v5 compatibility KDF, not password storage. + const out = crypto.createHash("sha512").update(kdfBuf).digest(); + return { key: out.subarray(0, 16), iv: out.subarray(16, 32) }; +} + +function testEncrypt(plainText, salt = Buffer.alloc(32, 0x11)) { + const plain = Buffer.from(plainText, "utf8"); + const { key, iv } = testDerive(salt); + const hash = crypto.createHash("sha512").update(plain).digest(); + const cipher = crypto.createCipheriv("aes-128-cbc", key, iv); + const ct = Buffer.concat([cipher.update(Buffer.concat([hash, plain])), cipher.final()]); + return Buffer.concat([TEST_MAGIC, salt, ct]); +} + +function makeTraeCnHome() { + return fs.mkdtempSync(path.join(os.tmpdir(), "trae-cn-test-")); +} + +function writeStorage(traeHome, value, { authKey = TRAE_CN_AUTH_KEY } = {}) { + const dir = path.join(traeHome, "User", "globalStorage"); + fs.mkdirSync(dir, { recursive: true }); + const storagePath = path.join(dir, "storage.json"); + const payload = {}; + if (value !== undefined) payload[authKey] = value; + fs.writeFileSync(storagePath, JSON.stringify(payload)); + return storagePath; +} + +function jsonResponse(status, body) { + return { + status, + ok: status >= 200 && status < 300, + json: async () => body, + }; +} + +// --------------------------------------------------------------------------- +// Path resolution +// --------------------------------------------------------------------------- + +test("resolveTraeCnHome honors TOKENTRACKER_TRAE_CN_HOME override", () => { + assert.equal( + resolveTraeCnHome({ env: { [TRAE_CN_HOME_ENV]: "/custom/cn-home" } }), + path.resolve("/custom/cn-home"), + ); + // Whitespace-only override falls back to the platform default. + const fallback = resolveTraeCnHome({ env: { [TRAE_CN_HOME_ENV]: " " }, platform: "darwin", home: "/Users/t" }); + assert.equal(fallback, path.join("/Users/t", "Library", "Application Support", CN_APP_DIR)); +}); + +test("resolveTraeCnHome resolves the macOS default and rejects other platforms", () => { + const home = "/Users/tester"; + assert.equal( + resolveTraeCnHome({ platform: "darwin", home }), + path.join(home, "Library", "Application Support", CN_APP_DIR), + ); + // Phase A is macOS + test-injected paths only. + assert.equal(resolveTraeCnHome({ platform: "win32", home, env: {} }), null); + assert.equal(resolveTraeCnHome({ platform: "linux", home, env: {} }), null); +}); + +test("resolveTraeCnStoragePath points at User/globalStorage/storage.json", () => { + const override = "/tmp/cn-install"; + assert.equal( + resolveTraeCnStoragePath({ env: { [TRAE_CN_HOME_ENV]: override } }), + path.join(path.resolve(override), "User", "globalStorage", "storage.json"), + ); + assert.equal( + resolveTraeCnStoragePath({ platform: "darwin", home: "/Users/tester", env: {} }), + path.join("/Users/tester", "Library", "Application Support", CN_APP_DIR, "User", "globalStorage", "storage.json"), + ); + assert.equal(resolveTraeCnStoragePath({ platform: "win32", home: "/Users/tester", env: {} }), null); +}); + +// --------------------------------------------------------------------------- +// Storage auth reading: plaintext + encrypted tc v5 +// --------------------------------------------------------------------------- + +test("frozen tc v5 fixture decrypts to the expected auth (broken-KDF detector)", () => { + assert.deepEqual(parseTraeCnAuthValue(FIXTURE_B64), FIXTURE_AUTH); + assert.equal(JSON.parse(decryptTraeCnBase64(FIXTURE_B64)).token, FIXTURE_TOKEN); +}); + +test("independent-encryptor round trip decrypts through production", () => { + const blob = testEncrypt(JSON.stringify({ token: "roundtrip.jwt.abc", refreshToken: "x" })); + assert.deepEqual(parseTraeCnAuthValue(blob.toString("base64")), { token: "roundtrip.jwt.abc", refreshToken: "x" }); +}); + +test("readTraeCnAuthFromStorage reads plaintext JSON object auth", () => { + const home = makeTraeCnHome(); + writeStorage(home, { token: "plain.jwt.1", refreshToken: "r1" }); + assert.deepEqual( + readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }), + { token: "plain.jwt.1", refreshToken: "r1" }, + ); +}); + +test("readTraeCnAuthFromStorage reads plaintext JSON string auth", () => { + const home = makeTraeCnHome(); + writeStorage(home, JSON.stringify({ token: "plain.jwt.2", refreshToken: "r2" })); + assert.deepEqual( + readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }), + { token: "plain.jwt.2", refreshToken: "r2" }, + ); +}); + +test("readTraeCnAuthFromStorage reads the encrypted tc v5 blob", () => { + const home = makeTraeCnHome(); + writeStorage(home, FIXTURE_B64); + const auth = readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }); + assert.equal(auth.token, FIXTURE_TOKEN); + assert.equal(extractTraeCnToken(auth), FIXTURE_TOKEN); +}); + +test("readTraeCnAuthFromStorage returns null when not configured", () => { + const home = makeTraeCnHome(); + const storagePath = path.join(home, "User", "globalStorage", "storage.json"); + assert.equal(readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }), null); + fs.mkdirSync(path.dirname(storagePath), { recursive: true }); + fs.writeFileSync(storagePath, JSON.stringify({})); + assert.equal(readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }), null); +}); + +// --------------------------------------------------------------------------- +// Malformed / tampered rejection (credential-safe errors) +// --------------------------------------------------------------------------- + +test("malformed storage.json fails closed with a credential-safe error", () => { + const home = makeTraeCnHome(); + const storagePath = path.join(home, "User", "globalStorage", "storage.json"); + fs.mkdirSync(path.dirname(storagePath), { recursive: true }); + fs.writeFileSync(storagePath, "{not json"); + assert.throws( + () => readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }), + /not valid JSON/, + ); +}); + +test("malformed plaintext JSON fails closed", () => { + const home = makeTraeCnHome(); + writeStorage(home, "{broken"); + assert.throws( + () => readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }), + /auth JSON is malformed/, + ); +}); + +test("tampered ciphertext is rejected and leaks nothing", () => { + const blob = Buffer.from(FIXTURE_B64, "base64"); + blob[blob.length - 16] ^= 0x01; // flip a byte in the ciphertext + const tampered = blob.toString("base64"); + assert.throws(() => parseTraeCnAuthValue(tampered)); + try { + parseTraeCnAuthValue(tampered); + assert.fail("expected throw"); + } catch (error) { + assert.ok(!error.message.includes(FIXTURE_TOKEN), "error must not contain the token"); + assert.ok(!error.message.includes(tampered), "error must not contain the blob"); + } +}); + +test("wrong magic / too-short / bad base64 are rejected", () => { + const blob = Buffer.from(FIXTURE_B64, "base64"); + blob[0] = 0x78; // "x" + assert.throws(() => parseTraeCnAuthValue(blob.toString("base64")), /unknown format/); + assert.throws(() => parseTraeCnAuthValue(Buffer.alloc(16).toString("base64")), /too short|unknown format/); + assert.throws(() => parseTraeCnAuthValue("not-base64@#$"), /Trae CN auth/); + assert.throws(() => parseTraeCnAuthValue(""), /empty|malformed/); + assert.throws(() => parseTraeCnAuthValue(42), /missing or malformed/); +}); + +test("extractTraeCnToken extracts a nonempty JWT and rejects empty/missing", () => { + assert.equal(extractTraeCnToken({ token: " abc.jwt.xyz " }), "abc.jwt.xyz"); + assert.throws(() => extractTraeCnToken({ token: "" }), /has no token/); + assert.throws(() => extractTraeCnToken({}), /has no token/); + assert.throws(() => extractTraeCnToken(null), /has no token/); +}); + +// --------------------------------------------------------------------------- +// Usage API page fetch +// --------------------------------------------------------------------------- + +const JWT = "test.jwt.usage-123"; +const START = 1700000000; +const END = 1700086400; + +test("fetchTraeCnUsagePage sends the exact request", async () => { + let captured; + const fetchImpl = async (url, options) => { + captured = { url, options }; + return jsonResponse(200, { user_usage_group_by_sessions: [{ id: 1 }], total: 1 }); + }; + await fetchTraeCnUsagePage({ + jwt: JWT, + start_time: START, + end_time: END, + page_num: 1, + url: "https://attacker.example/redirect", + fetchImpl, + }); + assert.equal(captured.url, TRAE_CN_USAGE_URL); + assert.equal(captured.options.method, "POST"); + assert.equal(captured.options.headers["Content-Type"], "application/json"); + assert.equal(captured.options.headers.Authorization, `Cloud-IDE-JWT ${JWT}`); + assert.deepEqual(JSON.parse(captured.options.body), { + usage_type: [7], + start_time: START, + end_time: END, + page_num: 1, + page_size: 20, + }); +}); + +test("fetchTraeCnUsagePage parses both documented wrappers", async () => { + const topLevel = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [{ id: 1 }, { id: 2 }], total: 2 }), + }); + assert.equal(topLevel.sessions.length, 2); + assert.equal(topLevel.total, 2); + + const nested = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { data: { user_usage_group_by_sessions: [{ id: 3 }], total: 1 } }), + }); + assert.equal(nested.sessions.length, 1); + assert.equal(nested.total, 1); + + const noTotal = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [{ id: 9 }] }), + }); + assert.equal(noTotal.sessions.length, 1); + assert.equal(noTotal.total, null); +}); + +test("fetchTraeCnUsagePage fails closed when the session wrapper is missing", async () => { + const bodies = [ + {}, + { data: {} }, + { data: { other: 1 } }, + { user_usage_group_by_sessions: "not-an-array" }, + { data: { user_usage_group_by_sessions: null } }, + ]; + for (const body of bodies) { + await assert.rejects( + fetchTraeCnUsagePage({ jwt: JWT, start_time: START, end_time: END, page_num: 1, fetchImpl: async () => jsonResponse(200, body) }), + (error) => { + assert.match(error.message, /missing the session list/); + assert.ok(!error.message.includes(JWT), "schema error must not leak the JWT"); + return true; + }, + ); + } +}); + +test("fetchTraeCnUsagePage accepts an explicit empty session array", async () => { + const result = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [], total: 0 }), + }); + assert.deepEqual(result.sessions, []); + assert.equal(result.total, 0); +}); + +test("fetchTraeCnUsagePage validates start_time / end_time / page_size locally", async () => { + const ok = () => async () => jsonResponse(200, { user_usage_group_by_sessions: [] }); + const page = (overrides) => + fetchTraeCnUsagePage({ jwt: JWT, start_time: START, end_time: END, page_num: 1, fetchImpl: ok(), ...overrides }); + + await assert.rejects(page({ start_time: 0 }), /start_time must be a positive integer/); + await assert.rejects(page({ start_time: -1 }), /start_time must be a positive integer/); + await assert.rejects(page({ start_time: 1.5 }), /start_time must be a positive integer/); + await assert.rejects(page({ start_time: Infinity }), /start_time must be a positive integer/); + await assert.rejects(page({ start_time: "1700000000" }), /start_time must be a positive integer/); + + await assert.rejects(page({ end_time: 0 }), /end_time must be a positive integer/); + await assert.rejects(page({ end_time: 1.5 }), /end_time must be a positive integer/); + + await assert.rejects(page({ start_time: END, end_time: START }), /start_time must not be after end_time/); + + await assert.rejects(page({ page_size: 0 }), /page_size must be a positive integer no greater than 20/); + await assert.rejects(page({ page_size: -5 }), /page_size must be a positive integer no greater than 20/); + await assert.rejects(page({ page_size: 21 }), /page_size must be a positive integer no greater than 20/); + await assert.rejects(page({ page_size: 2.5 }), /page_size must be a positive integer no greater than 20/); + + const valid = await page({ page_size: 20, start_time: START, end_time: END }); + assert.deepEqual(valid.sessions, []); + const smallPage = await page({ page_size: 5 }); + assert.equal(smallPage.page_size, 5); +}); + +test("fetchTraeCnUsagePage errors are useful and credential-safe", async () => { + await assert.rejects( + fetchTraeCnUsagePage({ jwt: JWT, start_time: START, end_time: END, page_num: 1, fetchImpl: async () => jsonResponse(401, {}) }), + (error) => { + assert.match(error.message, /HTTP 401/); + assert.equal(error.status, 401); + assert.equal(error.code, "AUTH_EXPIRED"); + assert.ok(!error.message.includes(JWT)); + return true; + }, + ); + + await assert.rejects( + fetchTraeCnUsagePage({ jwt: JWT, start_time: START, end_time: END, page_num: 1, fetchImpl: async () => jsonResponse(500, {}) }), + (error) => { + assert.match(error.message, /HTTP 500/); + assert.ok(!error.message.includes(JWT)); + return true; + }, + ); + + await assert.rejects( + fetchTraeCnUsagePage({ jwt: JWT, start_time: START, end_time: END, page_num: 1, fetchImpl: async () => jsonResponse(200, { code: 10001, data: {} }) }), + (error) => { + assert.match(error.message, /error code 10001/); + assert.equal(error.apiCode, 10001); + assert.ok(!error.message.includes(JWT)); + return true; + }, + ); + + await assert.rejects( + fetchTraeCnUsagePage({ jwt: "", start_time: START, end_time: END, page_num: 1 }), + /requires a token/, + ); + await assert.rejects( + fetchTraeCnUsagePage({ jwt: JWT, start_time: START, end_time: END, page_num: 0 }), + /positive integer/, + ); +}); + +test("fetchTraeCnUsagePage sanitizes transport exceptions containing the JWT", async () => { + const jwtCanary = "jwt-transport-canary"; + await assert.rejects( + fetchTraeCnUsagePage({ + jwt: jwtCanary, + start_time: START, + end_time: END, + page_num: 1, + fetchImpl: async (_url, options) => { + throw new Error(`request failed ${jwtCanary} ${options.headers.Authorization}`); + }, + }), + (error) => { + assert.equal(error.message, "Trae CN usage API request failed."); + assert.ok(!error.message.includes(jwtCanary)); + return true; + }, + ); +}); + +test("fetchTraeCnUsagePage keeps the abort timeout active through body consumption", async () => { + const originalSetTimeout = global.setTimeout; + const originalClearTimeout = global.clearTimeout; + let timerHandle; + let bodyStarted = false; + let bodySawCancelledTimer = false; + let bodyAborted = false; + global.setTimeout = (callback) => { + timerHandle = { callback, cancelled: false }; + return timerHandle; + }; + global.clearTimeout = (handle) => { + if (handle) handle.cancelled = true; + }; + + try { + const pending = fetchTraeCnUsagePage({ + jwt: JWT, + start_time: START, + end_time: END, + page_num: 1, + fetchImpl: (_url, options) => ({ + status: 200, + ok: true, + json: () => { + bodyStarted = true; + bodySawCancelledTimer = timerHandle.cancelled; + if (bodySawCancelledTimer) return Promise.resolve({ user_usage_group_by_sessions: [] }); + return new Promise((resolve) => { + options.signal.addEventListener("abort", () => { + bodyAborted = true; + resolve({ user_usage_group_by_sessions: [] }); + }, { once: true }); + }); + }, + }), + }); + await Promise.resolve(); + assert.equal(bodyStarted, true); + timerHandle.callback(); + const result = await pending; + assert.deepEqual(result.sessions, []); + assert.equal(bodySawCancelledTimer, false); + assert.equal(bodyAborted, true); + assert.equal(timerHandle.cancelled, true, "timer clears after body consumption"); + } finally { + global.setTimeout = originalSetTimeout; + global.clearTimeout = originalClearTimeout; + } +}); + +// --------------------------------------------------------------------------- +// Pagination +// --------------------------------------------------------------------------- + +function pagedFetch({ rowsPerPage, total } = {}) { + const requestedPages = []; + const fetchImpl = async (_url, options) => { + const body = JSON.parse(options.body); + requestedPages.push(body.page_num); + const rows = Array.from({ length: rowsPerPage(body) }, (_v, i) => ({ id: body.page_num * 1000 + i })); + const responseBody = { user_usage_group_by_sessions: rows }; + if (total !== undefined) responseBody.total = total; + return jsonResponse(200, responseBody); + }; + return { fetchImpl, requestedPages }; +} + +test("fetchTraeCnUsage paginates serially and stops at the stated total", async () => { + const total = 45; + const { fetchImpl, requestedPages } = pagedFetch({ + rowsPerPage: (body) => { + const remaining = total - (body.page_num - 1) * body.page_size; + return Math.max(0, Math.min(body.page_size, remaining)); + }, + total, + }); + const result = await fetchTraeCnUsage({ jwt: JWT, start_time: START, end_time: END, fetchImpl, delayMs: 0 }); + assert.deepEqual(requestedPages, [1, 2, 3]); + assert.equal(result.sessions.length, 45); + assert.equal(result.total, 45); + assert.equal(result.pages_fetched, 3); +}); + +test("fetchTraeCnUsage stops on an empty page when total is absent", async () => { + const requestedPages = []; + const fetchImpl = async (_url, options) => { + const body = JSON.parse(options.body); + requestedPages.push(body.page_num); + const rows = body.page_num === 1 ? [{ id: 1 }] : []; + return jsonResponse(200, { user_usage_group_by_sessions: rows }); + }; + const result = await fetchTraeCnUsage({ jwt: JWT, start_time: START, end_time: END, fetchImpl, delayMs: 0 }); + assert.deepEqual(requestedPages, [1, 2]); + assert.equal(result.sessions.length, 1); + assert.equal(result.total, null); + assert.equal(result.pages_fetched, 2); +}); + +test("fetchTraeCnUsage fails closed when maxPages is reached and more is needed", async () => { + const { fetchImpl, requestedPages } = pagedFetch({ rowsPerPage: () => 20 }); + await assert.rejects( + fetchTraeCnUsage({ jwt: JWT, start_time: START, end_time: END, fetchImpl, delayMs: 0, maxPages: 3 }), + (error) => { + assert.match(error.message, /exceeded the maximum page count/); + // Same capacity tag as the declared-total path so window splitting + // engages even when the API omits total. + assert.equal(error.code, "TRAE_CN_USAGE_CAPACITY_EXCEEDED"); + return true; + }, + ); + // Never returns a partial snapshot: all 3 cap pages were fetched, then it + // throws because page 4 would still be needed. + assert.deepEqual(requestedPages, [1, 2, 3]); +}); + +test("fetchTraeCnUsage succeeds when maxPages exactly covers the total", async () => { + const { fetchImpl, requestedPages } = pagedFetch({ rowsPerPage: () => 10, total: 30 }); + const result = await fetchTraeCnUsage({ jwt: JWT, start_time: START, end_time: END, fetchImpl, delayMs: 0, maxPages: 3 }); + assert.deepEqual(requestedPages, [1, 2, 3]); + assert.equal(result.sessions.length, 30); + assert.equal(result.pages_fetched, 3); +}); + +test("fetchTraeCnUsage rejects a declared snapshot over capacity before page two", async () => { + const { fetchImpl, requestedPages } = pagedFetch({ rowsPerPage: () => 20, total: 2001 }); + await assert.rejects( + fetchTraeCnUsage({ jwt: JWT, start_time: START, end_time: END, fetchImpl, delayMs: 0, maxPages: 100 }), + (error) => { + assert.equal(error.code, "TRAE_CN_USAGE_CAPACITY_EXCEEDED"); + assert.match(error.message, /supported capacity/); + assert.ok(!error.message.includes(JWT)); + return true; + }, + ); + assert.deepEqual(requestedPages, [1], "declared over-capacity total makes one request"); +}); + +// --------------------------------------------------------------------------- +// Capacity-adaptive window splitting +// --------------------------------------------------------------------------- + +function windowedFetch(handler, requests = []) { + const fetchImpl = async (_url, options) => { + const body = JSON.parse(options.body); + requests.push(body); + return jsonResponse(200, handler(body)); + }; + return { fetchImpl, requests }; +} + +test("fetchTraeCnUsageWindowed returns a within-capacity window without splitting", async () => { + const { fetchImpl, requests } = windowedFetch(() => ({ + user_usage_group_by_sessions: [{ session_id: "only" }], + total: 1, + })); + const result = await fetchTraeCnUsageWindowed({ + jwt: JWT, + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + }); + assert.equal(requests.length, 1, "no probe beyond the single window"); + assert.equal(requests[0].start_time, START); + assert.equal(requests[0].end_time, END); + assert.equal(result.sessions.length, 1); + assert.equal(result.total, 1); +}); + +test("fetchTraeCnUsageWindowed splits an over-capacity window into staggered halves", async () => { + const mid = START + Math.floor((END - START) / 2); + const leftRow = { session_id: "left" }; + const rightRow = { session_id: "right" }; + const { fetchImpl, requests } = windowedFetch((body) => { + if (body.start_time === START && body.end_time === END) { + return { user_usage_group_by_sessions: [leftRow], total: 2001 }; + } + if (body.start_time === START && body.end_time === mid) { + return { user_usage_group_by_sessions: [leftRow], total: 1 }; + } + if (body.start_time === mid + 1 && body.end_time === END) { + return { user_usage_group_by_sessions: [rightRow], total: 1 }; + } + throw new Error(`unexpected window ${body.start_time}-${body.end_time}`); + }); + const result = await fetchTraeCnUsageWindowed({ + jwt: JWT, + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + }); + // Full window fails on page one; the halves are staggered by one second so + // their union equals the full window without overlap. + assert.deepEqual( + requests.map((b) => [b.start_time, b.end_time]), + [[START, END], [START, mid], [mid + 1, END]], + ); + assert.deepEqual(result.sessions, [leftRow, rightRow]); + assert.equal(result.total, 2); + assert.equal(result.pages_fetched, 2); +}); + +test("fetchTraeCnUsageWindowed splits recursively until a sub-window fits", async () => { + const mid = START + Math.floor((END - START) / 2); + const quarter = START + Math.floor((mid - START) / 2); + const { fetchImpl, requests } = windowedFetch((body) => { + if (body.start_time === START && (body.end_time === END || body.end_time === mid)) { + return { user_usage_group_by_sessions: [], total: 2001 }; + } + if (body.start_time === START && body.end_time === quarter) { + return { user_usage_group_by_sessions: [{ session_id: "q1" }], total: 1 }; + } + if (body.start_time === quarter + 1 && body.end_time === mid) { + return { user_usage_group_by_sessions: [{ session_id: "q2" }], total: 1 }; + } + if (body.start_time === mid + 1 && body.end_time === END) { + return { user_usage_group_by_sessions: [{ session_id: "q3" }], total: 1 }; + } + throw new Error(`unexpected window ${body.start_time}-${body.end_time}`); + }); + const result = await fetchTraeCnUsageWindowed({ + jwt: JWT, + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + }); + assert.deepEqual( + requests.map((b) => [b.start_time, b.end_time]), + [[START, END], [START, mid], [START, quarter], [quarter + 1, mid], [mid + 1, END]], + ); + assert.equal(result.sessions.length, 3); + assert.equal(result.total, 3); +}); + +test("fetchTraeCnUsageWindowed fails closed at the split-depth ceiling", async () => { + const { fetchImpl, requests } = windowedFetch(() => ({ + user_usage_group_by_sessions: [], + total: 2001, + })); + await assert.rejects( + fetchTraeCnUsageWindowed({ + jwt: JWT, + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + }), + (error) => { + assert.equal(error.code, "TRAE_CN_USAGE_CAPACITY_EXCEEDED"); + assert.match(error.message, /supported capacity/); + return true; + }, + ); + // Every sub-window stays over capacity, so the full depth-8 binary tree is + // probed: at most 2^9 - 1 first-page requests before failing closed. + assert.ok(requests.length <= 511, `probe count ${requests.length} exceeds the ceiling`); + assert.ok(requests.length > 1, "splitting was attempted"); +}); + +test("fetchTraeCnUsageWindowed splits when cap exhaustion signals capacity without a declared total", async () => { + const mid = START + Math.floor((END - START) / 2); + const fullPage = Array.from({ length: 20 }, (_, i) => ({ session_id: `row-${i}` })); + const { fetchImpl, requests } = windowedFetch((body) => { + if (body.start_time === START && body.end_time === END) { + // Full pages with total omitted: the ONLY over-capacity signal is + // maxPages exhaustion, which must carry the capacity code too. + return { user_usage_group_by_sessions: fullPage }; + } + return { user_usage_group_by_sessions: [{ session_id: "ok" }], total: 1 }; + }); + const result = await fetchTraeCnUsageWindowed({ + jwt: JWT, + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + maxPages: 2, + }); + assert.deepEqual( + requests.map((b) => [b.start_time, b.end_time]), + [ + [START, END], // page 1 + [START, END], // page 2 -> cap exhausted, split + [START, mid], + [mid + 1, END], + ], + ); + assert.equal(result.sessions.length, 2); + assert.equal(result.total, 2); +}); + +test("fetchTraeCnUsageWindowed rethrows non-capacity errors untouched", async () => { + const requests = []; + const fetchImpl = async (_url, options) => { + requests.push(JSON.parse(options.body)); + return jsonResponse(500, {}); + }; + await assert.rejects( + fetchTraeCnUsageWindowed({ + jwt: JWT, + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + }), + /HTTP 500/, + ); + assert.deepEqual( + requests.map((b) => [b.start_time, b.end_time]), + [[START, END]], + "no split attempt on a non-capacity failure", + ); +}); + +test("production inter-page delay constant is 300ms (tests inject delayMs: 0)", () => { + assert.equal(TRAE_CN_USAGE_DELAY_MS, 300); +}); + +// --------------------------------------------------------------------------- +// Storage-backed fetch with single 401/403 retry +// --------------------------------------------------------------------------- + +test("fetchTraeCnUsageWithAuth retries exactly once on 401 by re-reading storage", async () => { + const home = makeTraeCnHome(); + const storagePath = writeStorage(home, { token: "rotated.jwt.2", refreshToken: "r2" }); + const authCalls = []; + const usageCalls = []; + const readAuth = () => { + authCalls.push(fs.readFileSync(storagePath, "utf8")); + return readTraeCnAuthFromStorage({ env: { [TRAE_CN_HOME_ENV]: home }, platform: "darwin" }); + }; + const usageFetcher = async ({ jwt }) => { + usageCalls.push(jwt); + if (usageCalls.length === 1) { + const error = new Error("Trae CN usage API returned HTTP 401."); + error.status = 401; + error.code = "AUTH_EXPIRED"; + throw error; + } + return { sessions: [{ id: 7 }], total: 1, pages_fetched: 1, truncated: false }; + }; + const result = await fetchTraeCnUsageWithAuth({ + start_time: START, + end_time: END, + fetchImpl: async () => jsonResponse(200, {}), + delayMs: 0, + readAuth, + usageFetcher, + env: { [TRAE_CN_HOME_ENV]: home }, + platform: "darwin", + }); + assert.equal(usageCalls.length, 2); + assert.equal(usageCalls[0], "rotated.jwt.2"); + assert.equal(usageCalls[1], "rotated.jwt.2"); + assert.equal(authCalls.length, 2, "storage re-read once"); + assert.equal(result.sessions[0].id, 7); +}); + +test("fetchTraeCnUsageWithAuth never retries more than once", async () => { + const usageCalls = []; + const usageFetcher = async () => { + usageCalls.push(1); + const error = new Error("Trae CN usage API returned HTTP 403."); + error.status = 403; + error.code = "AUTH_EXPIRED"; + throw error; + }; + await assert.rejects( + fetchTraeCnUsageWithAuth({ + start_time: START, end_time: END, fetchImpl: async () => jsonResponse(200, {}), delayMs: 0, + readAuth: () => ({ token: "a.jwt.1", refreshToken: "r" }), + usageFetcher, + }), + /HTTP 403/, + ); + assert.equal(usageCalls.length, 2, "exactly two attempts"); +}); + +test("fetchTraeCnUsageWithAuth splits over-capacity windows by default", async () => { + const home = makeTraeCnHome(); + writeStorage(home, { token: "split.jwt.1", refreshToken: "r1" }); + const mid = START + Math.floor((END - START) / 2); + const requests = []; + const fetchImpl = async (_url, options) => { + const body = JSON.parse(options.body); + requests.push([body.start_time, body.end_time]); + if (body.start_time === START && body.end_time === END) { + return jsonResponse(200, { user_usage_group_by_sessions: [], total: 2001 }); + } + const row = { session_id: `s-${body.start_time}`, model_name: "doubao-pro", usage_time: body.end_time, input_token: 1, output_token: 1, cache_read_token: 0, cache_write_token: 0 }; + return jsonResponse(200, { user_usage_group_by_sessions: [row], total: 1 }); + }; + const result = await fetchTraeCnUsageWithAuth({ + start_time: START, + end_time: END, + fetchImpl, + delayMs: 0, + env: { [TRAE_CN_HOME_ENV]: home }, + platform: "darwin", + }); + assert.deepEqual(requests, [[START, END], [START, mid], [mid + 1, END]]); + assert.equal(result.sessions.length, 2); + assert.equal(result.total, 2); +}); + +test("fetchTraeCnUsageWithAuth does not retry on non-auth errors", async () => { + const usageCalls = []; + const usageFetcher = async () => { + usageCalls.push(1); + const error = new Error("Trae CN usage API returned HTTP 500."); + error.status = 500; + throw error; + }; + await assert.rejects( + fetchTraeCnUsageWithAuth({ + start_time: START, end_time: END, fetchImpl: async () => jsonResponse(200, {}), delayMs: 0, + readAuth: () => ({ token: "a.jwt.1", refreshToken: "r" }), + usageFetcher, + }), + /HTTP 500/, + ); + assert.equal(usageCalls.length, 1); +}); + +test("fetchTraeCnUsageWithAuth fails when credentials are missing", async () => { + await assert.rejects( + fetchTraeCnUsageWithAuth({ start_time: START, end_time: END, readAuth: () => null, usageFetcher: async () => ({}) }), + /credentials are not configured/, + ); +}); + + +// --------------------------------------------------------------------------- +// Usage API total validation (PR #474): the declared total gates pagination +// termination, so a malformed value must fail closed as a schema error - +// never be coerced into a legal-looking stop condition that silently +// truncates the snapshot after page 1. +// --------------------------------------------------------------------------- + +test("fetchTraeCnUsagePage rejects a malformed declared total", async () => { + const badTotals = [ + -1, // negative: 20 >= -1 would stop pagination after page 1 + -0.5, // negative fraction + 2.5, // fractional row count + "45", // quoted digits: the API speaks JSON numbers (see the + // start_time convention above - string digits are a schema + // anomaly, not a numeric representation we accept) + "", // empty string + true, // wrong primitive + ]; + for (const total of badTotals) { + await assert.rejects( + fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [{ id: 1 }], total }), + }), + (error) => { + assert.match(error.message, /invalid total/); + assert.ok(!error.message.includes(JWT), "schema error must not leak the JWT"); + return true; + }, + `total ${JSON.stringify(total)} must be rejected`, + ); + } +}); + +test("fetchTraeCnUsagePage still accepts valid totals (0, exact page multiple) and absent total", async () => { + const zero = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [], total: 0 }), + }); + assert.equal(zero.total, 0); + + const big = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [{ id: 1 }], total: 9007199254740991 }), + }); + assert.equal(big.total, Number.MAX_SAFE_INTEGER); + + const absent = await fetchTraeCnUsagePage({ + jwt: JWT, start_time: START, end_time: END, page_num: 1, + fetchImpl: async () => jsonResponse(200, { user_usage_group_by_sessions: [{ id: 1 }] }), + }); + assert.equal(absent.total, null); +}); + +test("fetchTraeCnUsage fails closed on a malformed total instead of returning a partial snapshot", async () => { + // Page 1 is full (20 rows) and declares total = -1: the pre-fix code + // evaluated 20 >= -1 and silently stopped, returning a truncated snapshot. + const rows = Array.from({ length: 20 }, (_, i) => ({ id: i + 1 })); + const fetchImpl = async () => jsonResponse(200, { user_usage_group_by_sessions: rows, total: -1 }); + await assert.rejects( + fetchTraeCnUsage({ jwt: JWT, start_time: START, end_time: END, fetchImpl, delayMs: 0 }), + /invalid total/, + ); +}); + +// --------------------------------------------------------------------------- +// Auth storage IO errors (PR #474): a missing file is "not signed in", but a +// real IO failure (EACCES / EISDIR / I/O error) must fail closed with a +// generic, credential-safe error - never be reported as "not signed in" and +// never leak the OS detail, the path, or the storage contents. +// --------------------------------------------------------------------------- + +test("readTraeCnAuthFromStorage: ENOENT / ENOTDIR mean absent (null), other IO errors fail closed", () => { + const home = "/Users/tester"; + const env = { [TRAE_CN_HOME_ENV]: home }; + + const ioError = (code) => ({ + readFileSync: () => { + const err = new Error(`os-level detail ${code} /Users/tester/secret-path`); + err.code = code; + throw err; + }, + }); + + // Absent file / absent path component -> not signed in. + assert.equal(readTraeCnAuthFromStorage({ env, platform: "darwin", fsModule: ioError("ENOENT") }), null); + assert.equal(readTraeCnAuthFromStorage({ env, platform: "darwin", fsModule: ioError("ENOTDIR") }), null); + + // Real IO failures -> distinct unreadable error, credential-safe message. + for (const code of ["EACCES", "EISDIR", "EIO"]) { + assert.throws( + () => readTraeCnAuthFromStorage({ env, platform: "darwin", fsModule: ioError(code) }), + (error) => { + assert.equal(error.code, "TRAE_CN_STORAGE_UNREADABLE", code); + assert.match(error.message, /could not be read/); + // No OS detail, no filesystem path, no synthetic-JWT leak. + assert.ok(!error.message.includes(code), "must not surface the OS error code"); + assert.ok(!error.message.includes("secret-path"), "must not surface the storage path"); + return true; + }, + code, + ); + } +}); diff --git a/test/trae-cn-parser.test.js b/test/trae-cn-parser.test.js new file mode 100644 index 000000000..4d424cb85 --- /dev/null +++ b/test/trae-cn-parser.test.js @@ -0,0 +1,711 @@ +/** + * Trae Work CN usage-API incremental parser tests. + * + * Synthetic rows only. Verifies the parser's contract: + * - raw normalization / prevalidation before any cursor/bucket mutation + * - per-session reconciliation (subtract old contribution, add new) with a + * versioned cursors.traeCn state and source "trae-cn" + * - fail-closed corruption / malformed-state handling + */ +"use strict"; + +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const test = require("node:test"); + +const { parseTraeCnApiIncremental } = require("../src/lib/rollout"); + +function tempQueue() { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "tokentracker-trae-cn-")); + return { + dir, + queuePath: path.join(dir, "queue.jsonl"), + }; +} + +function queueRows(queuePath) { + if (!fs.existsSync(queuePath)) return []; + return fs.readFileSync(queuePath, "utf8") + .trim() + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line)); +} + +function lastTraeRow(queuePath, model, hourStart) { + return queueRows(queuePath) + .filter( + (row) => + // Usage rows only: the queue also carries account_session_state + // control records (same source/model, no hour_start). + !row.kind && + row.source === "trae-cn" && + row.model === model && + (hourStart === undefined || row.hour_start === hourStart), + ) + .at(-1); +} + +// Compare only the deterministic, meaningful state (drops updatedAt timestamps). +function stripState(cursors) { + const buckets = {}; + for (const [key, bucket] of Object.entries(cursors.hourly?.buckets || {})) { + buckets[key] = bucket?.totals; + } + const sessions = {}; + for (const [id, entry] of Object.entries(cursors.traeCn?.sessions || {})) { + sessions[id] = { model: entry.model, bucketStart: entry.bucketStart, totals: entry.totals }; + } + return { hourlyBuckets: buckets, sessions }; +} + +const T1 = 1_700_000_000; // 2023-11-14T22:13:20Z -> 22:00 bucket +const T2 = 1_700_002_000; // 2023-11-14T22:46:40Z -> 22:30 bucket +const B1 = "2023-11-14T22:00:00.000Z"; +const B2 = "2023-11-14T22:30:00.000Z"; + +function sessionRow(overrides = {}) { + return { + session_id: "s1", + model_name: "doubao-pro", + usage_time: T1, + input_token: 100, + output_token: 10, + cache_read_token: 0, + cache_write_token: 0, + ...overrides, + }; +} + +test("first insert maps direct and extra_info shapes into canonical buckets", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + let progressCalls = 0; + const onProgress = () => { + progressCalls += 1; + }; + const direct = sessionRow({ session_id: "s-direct", cache_read_token: 30, cache_write_token: 40 }); + const extra = { + session_id: "s-extra", + model_name: "doubao-lite", + usage_time: T2, + extra_info: { input_token: 5, output_token: 6, cache_read_token: 7, cache_write_token: 8 }, + }; + + const result = await parseTraeCnApiIncremental({ sessions: [direct, extra], cursors, queuePath, onProgress }); + assert.equal(result.recordsProcessed, 2); + assert.equal(result.eventsAggregated, 2); + assert.ok(result.bucketsQueued >= 1); + assert.ok(progressCalls > 0, "progress callback fires"); + + const rows = queueRows(queuePath).filter((row) => row.source === "trae-cn"); + const pro = rows.find((row) => row.model === "doubao-pro"); + const lite = rows.find((row) => row.model === "doubao-lite"); + assert.ok(pro, "direct row queued"); + assert.ok(lite, "extra_info row queued"); + + assert.equal(pro.hour_start, B1); + assert.deepEqual( + { + input: pro.input_tokens, + cached: pro.cached_input_tokens, + cacheCreation: pro.cache_creation_input_tokens, + output: pro.output_tokens, + reasoning: pro.reasoning_output_tokens, + total: pro.total_tokens, + billable: pro.billable_total_tokens, + conv: pro.conversation_count, + }, + // input_token is cache-inclusive: fresh = 100-30-40, total = 100+10. + { input: 30, cached: 30, cacheCreation: 40, output: 10, reasoning: 0, total: 110, billable: 110, conv: 1 }, + ); + + assert.equal(lite.hour_start, B2); + // Clamped to the cache-inclusive input: cached=min(5,7)=5, creation + // gets no room left (min(0,8)=0), fresh=0, total=5+6. + assert.equal(lite.input_tokens, 0); + assert.equal(lite.output_tokens, 6); + assert.equal(lite.cached_input_tokens, 5); + assert.equal(lite.cache_creation_input_tokens, 0); + assert.equal(lite.total_tokens, 11); + + assert.equal(cursors.traeCn.version, 1); + assert.equal(cursors.traeCn.sessions["s-direct"].model, "doubao-pro"); + assert.equal(cursors.traeCn.sessions["s-direct"].bucketStart, B1); + assert.equal(cursors.traeCn.sessions["s-direct"].totals.input_tokens, 30); + assert.equal(cursors.traeCn.sessions["s-extra"].bucketStart, B2); +}); + +test("cache-inclusive input_token is peeled into subset columns (real-data shape)", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + // Shape observed on real GLM rows: every prompt token hit the cache, so + // input_token equals cache_read_token exactly (ratio floor 1.0 across 135 + // real rows). Ordinary input must be 0, not input_token again. + const fullHit = sessionRow({ session_id: "full-hit", input_token: 500, output_token: 7, cache_read_token: 500 }); + // Partial hit: 200 of 500 cached -> fresh 300, total = 500 + 7. + const partial = sessionRow({ session_id: "partial", input_token: 500, output_token: 7, cache_read_token: 200 }); + // No cache activity at all (Doubao/DeepSeek shape): columns stay as-is. + const noCache = sessionRow({ session_id: "no-cache", input_token: 500, output_token: 7 }); + + await parseTraeCnApiIncremental({ sessions: [fullHit, partial, noCache], cursors, queuePath }); + assert.equal(cursors.traeCn.sessions["full-hit"].totals.input_tokens, 0); + assert.equal(cursors.traeCn.sessions["full-hit"].totals.cached_input_tokens, 500); + assert.equal(cursors.traeCn.sessions["full-hit"].totals.total_tokens, 507); + assert.equal(cursors.traeCn.sessions["partial"].totals.input_tokens, 300); + assert.equal(cursors.traeCn.sessions["partial"].totals.cached_input_tokens, 200); + assert.equal(cursors.traeCn.sessions["partial"].totals.total_tokens, 507); + assert.equal(cursors.traeCn.sessions["no-cache"].totals.input_tokens, 500); + assert.equal(cursors.traeCn.sessions["no-cache"].totals.cached_input_tokens, 0); + assert.equal(cursors.traeCn.sessions["no-cache"].totals.total_tokens, 507); +}); + +test("identical payload twice appends no queue row and no token growth", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const row = sessionRow(); + + const first = await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + assert.equal(first.eventsAggregated, 1); + const before = fs.readFileSync(queuePath, "utf8"); + + const second = await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + assert.equal(second.recordsProcessed, 1); + assert.equal(second.eventsAggregated, 0); + assert.equal(second.bucketsQueued, 0); + assert.equal(fs.readFileSync(queuePath, "utf8"), before, "queue unchanged"); +}); + +test("growth and downward correction preserve a shared-bucket sibling", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const a = sessionRow({ session_id: "A" }); + const b = sessionRow({ session_id: "B", input_token: 200, output_token: 20 }); + + await parseTraeCnApiIncremental({ sessions: [a, b], cursors, queuePath }); + + const grow = await parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: "A", input_token: 150 })], cursors, queuePath }); + assert.equal(grow.eventsAggregated, 1); + let latest = lastTraeRow(queuePath, "doubao-pro"); + assert.equal(latest.input_tokens, 200 + 150, "sibling B kept while A grows"); + assert.equal(latest.output_tokens, 20 + 10); + + const shrink = await parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: "A", input_token: 60 })], cursors, queuePath }); + assert.equal(shrink.eventsAggregated, 1); + latest = lastTraeRow(queuePath, "doubao-pro"); + assert.equal(latest.input_tokens, 200 + 60, "sibling B kept while A shrinks"); + assert.equal(latest.total_tokens, 200 + 20 + 60 + 10); + assert.equal(cursors.traeCn.sessions.B.totals.input_tokens, 200, "sibling contribution preserved in cursor"); +}); + +test("model and time-bucket moves retract the old tuple and write the new one", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const row = sessionRow(); + + await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + assert.equal(lastTraeRow(queuePath, "doubao-pro", B1).total_tokens, 110); + + // Model move: same bucket, different model. + const moved = await parseTraeCnApiIncremental({ sessions: [sessionRow({ model_name: "doubao-lite" })], cursors, queuePath }); + assert.equal(moved.eventsAggregated, 1); + assert.equal(lastTraeRow(queuePath, "doubao-pro", B1).total_tokens, 0, "old model tuple retracted to zero"); + assert.equal(lastTraeRow(queuePath, "doubao-lite", B1).total_tokens, 110); + + // Time-bucket move: same model, shifted half-hour bucket. + const shifted = await parseTraeCnApiIncremental({ sessions: [sessionRow({ model_name: "doubao-lite", usage_time: T2 })], cursors, queuePath }); + assert.equal(shifted.eventsAggregated, 1); + assert.equal(lastTraeRow(queuePath, "doubao-lite", B1).total_tokens, 0, "old time tuple retracted to zero"); + assert.equal(lastTraeRow(queuePath, "doubao-lite", B2).total_tokens, 110); +}); + +test("missing model_name values use the TRAE sentinel and reconcile without changing totals", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const missing = sessionRow({ session_id: "missing-model" }); + delete missing.model_name; + const nullModel = sessionRow({ session_id: "null-model", model_name: null }); + const blankModel = sessionRow({ session_id: "blank-model", model_name: " ", mode: "" }); + + await parseTraeCnApiIncremental({ sessions: [missing, nullModel, blankModel], cursors, queuePath }); + assert.equal(cursors.traeCn.sessions["missing-model"].model, "trae-cn-unknown"); + assert.equal(cursors.traeCn.sessions["null-model"].model, "trae-cn-unknown"); + assert.equal(cursors.traeCn.sessions["blank-model"].model, "trae-cn-unknown"); + assert.equal(lastTraeRow(queuePath, "trae-cn-unknown", B1).total_tokens, 330); + + await parseTraeCnApiIncremental({ + sessions: [sessionRow({ session_id: "transition", model_name: "doubao-pro" })], + cursors, + queuePath, + }); + await parseTraeCnApiIncremental({ + sessions: [sessionRow({ session_id: "transition", model_name: "" })], + cursors, + queuePath, + }); + assert.equal(lastTraeRow(queuePath, "doubao-pro", B1).total_tokens, 0); + assert.equal(lastTraeRow(queuePath, "trae-cn-unknown", B1).total_tokens, 440); + + await parseTraeCnApiIncremental({ + sessions: [sessionRow({ session_id: "transition", model_name: "doubao-pro" })], + cursors, + queuePath, + }); + assert.equal(lastTraeRow(queuePath, "trae-cn-unknown", B1).total_tokens, 330); + assert.equal(lastTraeRow(queuePath, "doubao-pro", B1).total_tokens, 110); +}); + +test("reordered payload and identical duplicate session records yield the same result", async (t) => { + const dirA = tempQueue(); + const dirB = tempQueue(); + t.after(() => { + fs.rmSync(dirA.dir, { recursive: true, force: true }); + fs.rmSync(dirB.dir, { recursive: true, force: true }); + }); + const cursorsA = {}; + const cursorsB = {}; + const a = sessionRow({ session_id: "A", input_token: 10, output_token: 1 }); + const b = sessionRow({ session_id: "B", model_name: "doubao-lite", usage_time: T2, input_token: 20, output_token: 2 }); + + await parseTraeCnApiIncremental({ sessions: [a, b], cursors: cursorsA, queuePath: dirA.queuePath }); + await parseTraeCnApiIncremental({ sessions: [b, a], cursors: cursorsB, queuePath: dirB.queuePath }); + assert.deepEqual(stripState(cursorsB), stripState(cursorsA), "order does not change the result"); + + // Identical duplicate session records are accepted once within one payload. + const dirC = tempQueue(); + t.after(() => fs.rmSync(dirC.dir, { recursive: true, force: true })); + const cursorsC = {}; + const dup = await parseTraeCnApiIncremental({ + sessions: [a, { ...a }], + cursors: cursorsC, + queuePath: dirC.queuePath, + }); + assert.equal(dup.recordsProcessed, 2); + assert.equal(dup.eventsAggregated, 1, "exact duplicate accepted once"); + assert.equal(cursorsC.traeCn.sessions.A.totals.input_tokens, 10); +}); + +test("conflicting duplicate ids and malformed rows fail before any mutation", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const canary = "session-id-canary"; + const row = sessionRow({ session_id: canary }); + + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [row, sessionRow({ session_id: canary, input_token: 200 })], cursors, queuePath }), + (error) => { + assert.match(error.message, /conflicting contributions/); + assert.ok(!error.message.includes(canary)); + return true; + }, + ); + assert.equal(cursors.hourly, undefined, "no hourly state assigned"); + assert.equal(cursors.traeCn, undefined, "no traeCn state assigned"); + assert.equal(fs.existsSync(queuePath), false, "no queue rows written"); + + // Cache fields are optional: models without a prompt-cache concept (Doubao / + // DeepSeek on TRAE CN) legitimately omit them — absent means 0, not malformed. + // Uses its own queue dir so the no-mutation assertions below stay meaningful. + const optionalQueue = tempQueue(); + const cursorsOptional = {}; + const missing = sessionRow({ session_id: canary }); + delete missing.cache_read_token; + delete missing.cache_write_token; + const optionalResult = await parseTraeCnApiIncremental({ + sessions: [missing], + cursors: cursorsOptional, + queuePath: optionalQueue.queuePath, + }); + assert.equal(optionalResult.skippedRows, 0); + assert.equal(cursorsOptional.traeCn.sessions[canary].totals.cached_input_tokens, 0); + fs.rmSync(optionalQueue.dir, { recursive: true, force: true }); + + // A malformed row FAILS THE WHOLE SNAPSHOT closed (P0: a partially + // understood snapshot must never become the authoritative window state - + // no bucket rows, no session states, no cursor commit). Confirmed-legal + // variations (absent cache fields) are handled above and stay legal. + const mixedQueue = tempQueue(); + const cursorsMixed = {}; + await assert.rejects( + parseTraeCnApiIncremental({ + sessions: [ + sessionRow({ session_id: canary, cache_read_token: "lots" }), + sessionRow({ session_id: "healthy-row" }), + ], + cursors: cursorsMixed, + queuePath: mixedQueue.queuePath, + }), + (error) => { + assert.match(error.message, /not authoritative/); + assert.match(error.message, /1 malformed row/); + assert.match(error.message, /invalid cache_read_token/); + assert.ok(!error.message.includes(canary)); + return true; + }, + ); + assert.equal(cursorsMixed.hourly, undefined, "no state assigned on a partial snapshot"); + assert.equal(cursorsMixed.traeCn, undefined, "no traeCn state on a partial snapshot"); + assert.equal(fs.existsSync(mixedQueue.queuePath), false, "no queue rows written (no buckets, no watermark)"); + fs.rmSync(mixedQueue.dir, { recursive: true, force: true }); + // A payload where EVERY row is malformed fails with the same unified + // not-authoritative error (reason stays diagnosable + id-free). + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: canary, cache_read_token: "lots" })], cursors, queuePath }), + (error) => { + assert.match(error.message, /not authoritative/); + assert.match(error.message, /invalid cache_read_token/); + assert.ok(!error.message.includes(canary)); + return true; + }, + ); + // Model names containing "|" are rejected because bucket keys are unescaped. + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: canary, model_name: "bad|model" })], cursors, queuePath }), + (error) => { + assert.match(error.message, /not authoritative/); + assert.match(error.message, /unsupported model name/); + assert.ok(!error.message.includes(canary)); + return true; + }, + ); + // Invalid usage_time / sessions / cursors. + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: canary, usage_time: -5 })], cursors, queuePath }), + (error) => { + assert.match(error.message, /invalid usage_time/); + assert.ok(!error.message.includes(canary)); + return true; + }, + ); + await assert.rejects(parseTraeCnApiIncremental({ sessions: "nope", cursors, queuePath }), /sessions must be an array/); + await assert.rejects(parseTraeCnApiIncremental({ sessions: [], cursors: null, queuePath }), /cursors must be a writable object/); + + assert.equal(cursors.hourly, undefined); + assert.equal(cursors.traeCn, undefined); + assert.equal(fs.existsSync(queuePath), false); +}); + +test("empty payload is a no-op and preserves existing contributions", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const row = sessionRow(); + + await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + const before = fs.readFileSync(queuePath, "utf8"); + + const empty = await parseTraeCnApiIncremental({ sessions: [], cursors, queuePath }); + assert.equal(empty.recordsProcessed, 0); + assert.equal(empty.eventsAggregated, 0); + assert.equal(empty.bucketsQueued, 0); + assert.equal(fs.readFileSync(queuePath, "utf8"), before, "no queue change"); + assert.equal(cursors.traeCn.sessions["s1"].totals.input_tokens, 100, "contribution preserved"); + assert.equal(lastTraeRow(queuePath, "doubao-pro").total_tokens, 110); +}); + +test("empty payload asserts nothing: no usage mutation, no session states (absence contract NOT PROVEN)", async (t) => { + // Evidence: two real fetches 17min apart (137 -> 141 sessions, 0 + // disappeared, 0 changed) show a stable session set, but nothing proves + // that an absent session means deleted/zero. An empty response therefore + // asserts NOTHING - no usage mutation AND no account_session_state + // observation that could displace another device's canonical rows. + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + await parseTraeCnApiIncremental({ sessions: [sessionRow()], cursors, queuePath }); + const before = fs.readFileSync(queuePath, "utf8"); + + const empty = await parseTraeCnApiIncremental({ + sessions: [], + cursors, + queuePath, + windowStartMs: 1_700_000_000_000, + windowEndMs: 1_700_000_000_000 + 30 * 24 * 3600 * 1000, + }); + assert.equal(empty.recordsProcessed, 0); + assert.equal(empty.bucketsQueued, 0); + assert.equal(fs.readFileSync(queuePath, "utf8"), before, "no session state appended for an empty response"); + assert.equal(cursors.traeCn.sessions.s1.totals.input_tokens, 100, "contribution preserved"); +}); + +test("a partial snapshot (99 valid + 1 malformed) publishes no session states even with a window", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + await assert.rejects( + parseTraeCnApiIncremental({ + sessions: [sessionRow({ session_id: "bad-row", usage_time: "x" })].concat( + Array.from({ length: 3 }, (_, i) => sessionRow({ session_id: "ok-" + i })), + ), + cursors, + queuePath, + windowStartMs: 1_700_000_000_000, + windowEndMs: 1_700_000_000_000 + 30 * 24 * 3600 * 1000, + }), + /not authoritative/, + ); + assert.equal(fs.existsSync(queuePath), false, "neither bucket rows nor session states may land"); + assert.equal(cursors.hourly, undefined, "previous canonical state untouched locally"); +}); +test("cursors survive a serialize/parse restart and a later correction", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + let cursors = {}; + const row = sessionRow(); + + await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + cursors = JSON.parse(JSON.stringify(cursors)); + + // Identical re-run after restart is a no-op. + const before = fs.readFileSync(queuePath, "utf8"); + const noop = await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + assert.equal(noop.eventsAggregated, 0); + assert.equal(noop.bucketsQueued, 0); + assert.equal(fs.readFileSync(queuePath, "utf8"), before); + + // Correction after restart retracts the old tuple and writes the new one. + const corrected = await parseTraeCnApiIncremental({ sessions: [sessionRow({ input_token: 120 })], cursors, queuePath }); + assert.equal(corrected.eventsAggregated, 1); + const latest = lastTraeRow(queuePath, "doubao-pro"); + assert.equal(latest.input_tokens, 120); + assert.equal(latest.total_tokens, 130); +}); + +test("unexpected cursor version fails closed instead of resetting", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = { traeCn: { version: 99, sessions: {} } }; + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow()], cursors, queuePath }), + /version 99 is not supported/, + ); + assert.equal(cursors.traeCn.version, 99, "malformed state is not reset"); +}); + +test("a stored contribution exceeding its bucket fails closed without mutation", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const bucketKey = `trae-cn|doubao-pro|${B1}`; + const cursors = { + hourly: { + version: 3, + buckets: { + [bucketKey]: { + totals: { + input_tokens: 5, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + output_tokens: 5, + reasoning_output_tokens: 0, + total_tokens: 10, + billable_total_tokens: 10, + conversation_count: 1, + }, + queuedKey: null, + }, + }, + }, + traeCn: { + version: 1, + sessions: { + "session-id-canary": { + model: "doubao-pro", + bucketStart: B1, + totals: { + input_tokens: 100, + output_tokens: 10, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + reasoning_output_tokens: 0, + total_tokens: 110, + billable_total_tokens: 110, + conversation_count: 1, + }, + updatedAt: "x", + }, + }, + }, + }; + const before = JSON.stringify(cursors); + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: "session-id-canary", input_token: 120 })], cursors, queuePath }), + (error) => { + assert.match(error.message, /corruption/); + assert.ok(!error.message.includes("session-id-canary")); + return true; + }, + ); + assert.equal(JSON.stringify(cursors), before, "cursor state untouched"); + assert.equal(fs.existsSync(queuePath), false); +}); + +test("failed enqueue leaves the serialized cursors byte-identical", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const row = sessionRow(); + + await parseTraeCnApiIncremental({ sessions: [row], cursors, queuePath }); + const before = JSON.stringify(cursors); + + // Corrected snapshot whose queuePath is an existing directory: enqueue's + // appendFile fails (EISDIR) after reconciliation already mutated the working + // state. The deep-cloned working states must keep cursors untouched. + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow({ input_token: 120 })], cursors, queuePath: dir }), + ); + assert.equal(JSON.stringify(cursors), before, "cursors byte-identical after failed enqueue"); +}); + +test("an unrelated malformed stored session fails a new valid payload before mutation", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = { + traeCn: { + version: 1, + sessions: { + good: { + model: "doubao-pro", + bucketStart: B1, + totals: { + input_tokens: 100, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + output_tokens: 10, + reasoning_output_tokens: 0, + total_tokens: 110, + billable_total_tokens: 110, + conversation_count: 1, + }, + updatedAt: "x", + }, + bad: { + model: "doubao-lite", + bucketStart: B1, + totals: { + input_tokens: 1, + cached_input_tokens: 0, + cache_creation_input_tokens: 0, + output_tokens: 1, + reasoning_output_tokens: 1, // violates the canonical invariant + total_tokens: 2, + billable_total_tokens: 2, + conversation_count: 1, + }, + updatedAt: "x", + }, + }, + }, + }; + const before = JSON.stringify(cursors); + await assert.rejects( + parseTraeCnApiIncremental({ sessions: [sessionRow({ session_id: "new-session" })], cursors, queuePath }), + /stored session totals are malformed/, + ); + assert.equal(JSON.stringify(cursors), before, "no mutation on malformed stored state"); + assert.equal(fs.existsSync(queuePath), false, "no queue rows written"); +}); + +test("window prune drops pre-window sessions monotonically and never rewinds", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + await parseTraeCnApiIncremental({ + sessions: [ + sessionRow({ session_id: "old-s" }), // bucket B1 = 22:00 + sessionRow({ session_id: "new-s", usage_time: T2 }), // bucket B2 = 22:30 + ], + cursors, + queuePath, + }); + assert.ok(cursors.traeCn.sessions["old-s"]); + assert.ok(cursors.traeCn.sessions["new-s"]); + assert.equal(cursors.traeCn.prunedBeforeMs, 0, "no prune before a window is supplied"); + + // Window starts at 22:33:20 (T1 + 20min): old-s's whole bucket (22:00) is + // before it and can never reappear; new-s (22:46:40, bucket 22:30) is in-window. + const windowStartMs = (T1 + 20 * 60) * 1000; + await parseTraeCnApiIncremental({ + sessions: [sessionRow({ session_id: "new-s", usage_time: T2, output_token: 20 })], + cursors, + queuePath, + windowStartMs, + }); + assert.equal(cursors.traeCn.sessions["old-s"], undefined, "pre-window entry pruned"); + assert.ok(cursors.traeCn.sessions["new-s"], "in-window entry kept"); + assert.equal(cursors.traeCn.prunedBeforeMs, windowStartMs); + + // An earlier window start must not rewind the prune watermark + // (prunedBeforeMs) or re-prune; a re-reported in-window session still + // reconciles against its stored entry. + await parseTraeCnApiIncremental({ + sessions: [sessionRow({ session_id: "new-s", usage_time: T2 })], + cursors, + queuePath, + windowStartMs: (T1 + 60) * 1000, + }); + assert.equal(cursors.traeCn.prunedBeforeMs, windowStartMs, "prune watermark never rewinds"); + assert.ok(cursors.traeCn.sessions["new-s"]); +}); + +test("session state records carry the canonical observation + snapshot_verified_at (logical freshness)", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const T_LATE = 1_700_003_500; // 23:05:00Z -> 23:00 bucket + const verifiedAt = 1_700_010_000_000; + await parseTraeCnApiIncremental({ + sessions: [sessionRow(), sessionRow({ session_id: "s2", usage_time: T_LATE })], + cursors, + queuePath, + windowStartMs: 1_699_000_000_000, + windowEndMs: 1_700_010_000_000, + snapshotVerifiedAtMs: verifiedAt, + }); + const states = queueRows(queuePath).filter((r) => r.kind === "account_session_state"); + assert.equal(states.length, 2, "one canonical observation per session in the snapshot"); + const byId = new Map(states.map((r) => [r.session_id, r])); + const s1 = byId.get("s1"); + assert.ok(s1, "s1 observation queued"); + assert.equal(s1.source, "trae-cn"); + assert.equal(s1.model, "doubao-pro"); + assert.equal(s1.bucket_start, B1, "bucket_start is the session's half-hour floor (22:00)"); + assert.equal(s1.input_tokens, 100, "token columns mirror the session totals"); + assert.equal(s1.total_tokens, 110); + assert.equal( + s1.snapshot_verified_at, + new Date(verifiedAt).toISOString(), + "logical fetch stamp is the injected real-fetch time", + ); + assert.equal(byId.get("s2").bucket_start, "2023-11-14T23:00:00.000Z"); +}); + +test("transport retry replays the ORIGINAL session state record verbatim (append-only queue)", async (t) => { + const { dir, queuePath } = tempQueue(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const cursors = {}; + const verifiedAt = 1_700_010_000_000; + await parseTraeCnApiIncremental({ + sessions: [sessionRow()], + cursors, + queuePath, + windowStartMs: 1_699_000_000_000, + windowEndMs: 1_700_010_000_000, + snapshotVerifiedAtMs: verifiedAt, + }); + const first = fs.readFileSync(queuePath, "utf8").trim().split("\n").at(-1); + // A "retry" re-reads the same append-only record: the bytes - including + // snapshot_verified_at - are identical, so no new logical version exists. + const replayed = fs.readFileSync(queuePath, "utf8").trim().split("\n").at(-1); + assert.equal(replayed, first, "re-delivery carries the original logical fetch stamp"); + assert.ok(JSON.parse(first).snapshot_verified_at); +});