From 339d38f1baf8260fa22f538650d1864c2b3de9a1 Mon Sep 17 00:00:00 2001 From: Sahil Ahuja Date: Wed, 10 Jun 2026 21:18:05 +0530 Subject: [PATCH 1/2] fix: stop multi-machine sync from destroying historical cost data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Never-shrink guard in writeMetrics: skip the day-file write when the incoming entry's totalCost is below the existing snapshot's — Claude Code purges transcripts >30 days old, so every sync was overwriting correct per-day JSONL history with post-purge residue (~$5,160 measured destroyed across machines) - New standalone scripts/repair-metrics.mjs (dry-run default, --write) restoring shrunk day-files to their historical max from the metrics repo's git history; run manually post-release - Self-view max-merge: own-machine repo snapshots are max-merged into the live local view via new pure maxMergeEntries in fetcher.ts, so a machine sees its own synced history after local transcript purge (fixes --by-machine and watch mode too) --- docs/memory/cli/data-pipeline.md | 4 + docs/memory/cli/index.md | 2 +- docs/memory/sync/index.md | 2 +- docs/memory/sync/multi-machine.md | 10 +- .../.history.jsonl | 14 + .../.status.yaml | 49 ++++ .../intake.md | 110 ++++++++ .../plan.md | 198 ++++++++++++++ scripts/repair-metrics.mjs | 245 ++++++++++++++++++ src/node/core/__tests__/fetcher.test.ts | 74 ++++++ .../core/__tests__/self-view-merge.test.ts | 177 +++++++++++++ src/node/core/cli.ts | 27 +- src/node/core/fetcher.ts | 22 ++ .../sync/__tests__/repair-metrics.test.ts | 197 ++++++++++++++ src/node/sync/__tests__/sync.test.ts | 85 ++++++ src/node/sync/sync.ts | 27 ++ 16 files changed, 1234 insertions(+), 9 deletions(-) create mode 100644 fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl create mode 100644 fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml create mode 100644 fab/changes/260610-srmi-fix-metrics-history-destruction/intake.md create mode 100644 fab/changes/260610-srmi-fix-metrics-history-destruction/plan.md create mode 100644 scripts/repair-metrics.mjs create mode 100644 src/node/core/__tests__/self-view-merge.test.ts create mode 100644 src/node/sync/__tests__/repair-metrics.test.ts diff --git a/docs/memory/cli/data-pipeline.md b/docs/memory/cli/data-pipeline.md index 10faf13..8718751 100644 --- a/docs/memory/cli/data-pipeline.md +++ b/docs/memory/cli/data-pipeline.md @@ -15,6 +15,8 @@ Entry point: `src/node/core/cli.ts`. Data types: `src/node/core/types.ts`. Data - Global flags (parsed by `parseGlobalFlags` into `GlobalFlags`): `--json`, `--csv`, `--md`, `--sync`, `--fresh`/`-f`, `--watch`/`-w`, `--interval`/`-i `, `--user`/`-u `, `--by-machine`, `--no-color`, `--no-rain`, `--version`/`-V`/`-v`. Separately, `--skip-brew-update` is an `update`-scoped command-specific flag detected via raw-argv membership — it is NOT in `GlobalFlags` - `--by-machine` MUST show per-machine cost breakdown columns in tables; works with snapshot and single-tool history; incompatible with all-tools history pivot (warn on stderr and ignore); compatible with `--watch`, `--json`, and `-u`; in single mode, shows one machine (hostname); uses `fetchToolMergedWithMachines` which returns `{ entries, machineMap }` where machineMap is `Map` (machine → entries) - `--user`/`-u` MUST set a target user for data display (multi mode only); in single mode, warn on stderr and ignore; when targeting a different user, display only metrics repo data (no local ccusage data); when targeting the same user as `config.user`, behave identically to the default (no `-u`) path — perform a local fetch (using cache unless `--fresh`/`-f` is provided), write to the metrics repo, then merge with other machines +- `maxMergeEntries(a: UsageEntry[], b: UsageEntry[])` (exported from `src/node/core/fetcher.ts`, alongside `mergeEntries`) MUST pick, per date label, whichever whole entry has the greater `totalCost` — no field mixing, no summing; on equal `totalCost` the entry from `a` wins (call sites pass the live local fetch as `a`); output MUST be sorted ascending by label and inputs MUST NOT be mutated (pure function, Constitution V) +- In multi mode, the own-user merge pipeline (`fetchToolMerged` and `fetchToolMergedWithMachines` in `src/node/core/cli.ts`) MUST run: live local fetch → `writeMetrics` (never-shrink guarded — see `sync/multi-machine.md`) → `readRemoteEntriesByMachine(metricsDir, config.user, null, toolKey)` (all machines, one walk) → split the own machine out of the map → `effectiveLocal = maxMergeEntries(local, ownSnapshots)` → `mergeEntries(effectiveLocal, otherMachines)`. `fetchToolMergedWithMachines` MUST set the own-machine `machineMap` entry to the max-merged view so `--by-machine` shows the corrected own-machine column and the flattened entries reflect it. Single mode (live entries only, no repo reads) and the `-u ` path (repo-only, `excludeMachine = null`) are unaffected - `--json`, `--csv`, and `--md` MUST be mutually exclusive with each other; `--watch` MUST be incompatible with any of them. Violations print `Error: {flag-a} and {flag-b} are incompatible` to stderr and exit 1 (`parseGlobalFlags` in `src/node/core/cli.ts`) - `--interval` range: 5-3600 seconds, default 10 - Non-data commands (`init-conf`, `init-metrics`, `sync`, `status`, `update`, `shell-init`, `help`) MUST be dispatched before grammar parsing @@ -40,6 +42,7 @@ Entry point: `src/node/core/cli.ts`. Data types: `src/node/core/types.ts`. Data - **Output format as a single enum plumbed through dispatch** (260423-lx0g): `outputFormat: "table" | "json" | "csv" | "md"` is resolved once in `parseGlobalFlags` and each dispatch function switches on it in one place. Avoids multiplying `if (jsonFlag) ... else ...` branches as new formats are added. - **Static shell completion scripts** (260423-lx0g): Bash/zsh/fish completion scripts are hardcoded string constants in `src/node/core/completions.ts`. No dynamic lookup (the grammar is stable and runtime `tu --list-*` calls would add 50-200ms per tab-press); no `$SHELL` auto-detection (explicit arg avoids silent mismatches when `$SHELL` and the running shell diverge). - **`--skip-brew-update` detected via raw-argv membership, not `parseGlobalFlags`** (260531-e96v): The `update` dispatch passes `process.argv.includes("--skip-brew-update")` into `runUpdate` rather than threading the flag through `parseGlobalFlags`/`GlobalFlags`. It is command-specific to `update` (which ignores positional/data args); routing it through the shared data-flag path would broaden the blast radius and risk perturbing unrelated flag handling. Membership testing matches the existing `rawArgs.includes(...)` idiom (e.g. `--sync`). The cross-toolkit contract mandated the exact flag name `--skip-brew-update` (no alias). +- **Own-machine snapshots max-merged into the live view** (260610-srmi): Claude Code purges transcripts older than ~30 days, so the live ccusage fetch under-reports old days; the previous pipeline excluded the machine's own repo dir from remote reads (`excludeMachine = config.machine`), leaving its own synced history invisible to itself once transcripts were purged. The multi-mode pipeline now reads all machines in one `readRemoteEntriesByMachine(..., null, ...)` walk, splits out the own machine, and applies per-day whole-entry max (`maxMergeEntries`) to the own share before the existing sum-merge with other machines. Max, not sum: a partially-purged day still yields a residual live entry, and summing residual + snapshot would double-count the surviving transcripts. Reuses the existing by-machine walk unchanged (minimum pathways) instead of adding a single-machine read helper; merged totals only ever increase relative to the pre-change pipeline. Watch mode shares the same fetch path and is corrected for free. ## Changelog @@ -54,3 +57,4 @@ Entry point: `src/node/core/cli.ts`. Data types: `src/node/core/types.ts`. Data | 2026-04-01 | Added `-v` (lowercase) as version flag alias alongside `--version` and `-V` (260401-kuuh) | | 2026-04-23 | Migrated child process spawning from `exec` to `execFile` with argv arrays; `TOOLS` shape changed from `{name, command, needsFilter}` to `{name, binary, prefixArgs, needsFilter}`; added `--csv`/`--md` global flags and `outputFormat` enum dispatch; added `tu completions ` non-data subcommand with static bash/zsh/fish scripts (260423-lx0g) | | 2026-05-31 | Added `--skip-brew-update` flag to `tu update` — skips only the internal `brew update --quiet` refresh; version check and `brew upgrade` unaffected (260531-e96v) | +| 2026-06-10 | Self-view fix: added pure `maxMergeEntries` to `fetcher.ts`; multi-mode `fetchToolMerged`/`fetchToolMergedWithMachines` read all machines (`excludeMachine = null`) and max-merge own-machine snapshots into the live view before the sum-merge; single mode and `-u ` unchanged (260610-srmi) | diff --git a/docs/memory/cli/index.md b/docs/memory/cli/index.md index a2c333f..a4c9a94 100644 --- a/docs/memory/cli/index.md +++ b/docs/memory/cli/index.md @@ -2,4 +2,4 @@ | File | Description | Last Updated | |------|-------------|--------------| -| [data-pipeline.md](data-pipeline.md) | CLI argument parsing, data fetching, caching, tool registry | 2026-05-31 | +| [data-pipeline.md](data-pipeline.md) | CLI argument parsing, data fetching, caching, tool registry | 2026-06-10 | diff --git a/docs/memory/sync/index.md b/docs/memory/sync/index.md index 15d38e3..319cc83 100644 --- a/docs/memory/sync/index.md +++ b/docs/memory/sync/index.md @@ -2,4 +2,4 @@ | File | Description | Last Updated | |------|-------------|--------------| -| [multi-machine.md](multi-machine.md) | Git-based metrics sync, JSONL storage, remote merging, auto-clone | 2026-03-06 | +| [multi-machine.md](multi-machine.md) | Git-based metrics sync, JSONL high-water-mark storage (never-shrink writes, self-view max-merge), remote merging, auto-clone, repair script | 2026-06-10 | diff --git a/docs/memory/sync/multi-machine.md b/docs/memory/sync/multi-machine.md index 4a08f76..026f24e 100644 --- a/docs/memory/sync/multi-machine.md +++ b/docs/memory/sync/multi-machine.md @@ -2,7 +2,7 @@ ## Overview -Multi-machine sync (`src/node/sync/sync.ts`) enables aggregating AI usage costs across multiple machines via a shared git repository. Each machine writes per-day JSONL files to a structured directory hierarchy, then syncs via git commit/pull/push. +Multi-machine sync (`src/node/sync/sync.ts`) enables aggregating AI usage costs across multiple machines via a shared git repository. Each machine writes per-day JSONL files to a structured directory hierarchy, then syncs via git commit/pull/push. Day-file snapshots are high-water marks: `writeMetrics` never shrinks them, and at display time each machine max-merges its own snapshots back into its live view (the live view under-reports old dates once Claude Code purges transcripts older than ~30 days). ## Requirements @@ -10,9 +10,11 @@ Multi-machine sync (`src/node/sync/sync.ts`) enables aggregating AI usage costs - Metrics directory structure MUST follow: `{metricsDir}/{user}/{year}/{machine}/{toolKey}-{YYYY-MM-DD}.jsonl` - Each JSONL file MUST contain a single `UsageEntry` JSON object - `writeMetrics()` MUST write local entries to the metrics directory (creates dirs as needed) +- `writeMetrics()` MUST NOT shrink a day-file: when the existing file parses as JSON with a finite numeric `totalCost` and the incoming entry's `totalCost` is strictly lower, the write is skipped silently (no stderr). Absent, empty, or unparseable files — including JSON without a finite `totalCost` — are treated as absent (write proceeds); equal or greater incoming `totalCost` overwrites as before, so today's file keeps refreshing as the day grows. The guard (`isShrinkingWrite`) lives inside `writeMetrics` itself, covering every call site (`fetchToolMerged` and `fetchToolMergedWithMachines` in `src/node/core/cli.ts`, and `fullSync`) - `readRemoteEntriesByMachine(metricsDir, targetUser, excludeMachine, toolKey)` MUST return `Map` grouped by machine directory name; when `excludeMachine` is non-null, that machine is excluded from the map - `readRemoteEntries(metricsDir, targetUser, excludeMachine, toolKey)` MUST delegate to `readRemoteEntriesByMachine` and flatten the result into a single `UsageEntry[]`; signature and behavior unchanged from callers' perspective - `mergeEntries()` MUST sum token counts and costs for entries with matching labels [INFERRED] +- In multi mode, the own-user display path MUST merge the machine's own repo snapshots back into its live view: `fetchToolMerged`/`fetchToolMergedWithMachines` (`src/node/core/cli.ts`) read ALL machines via one `readRemoteEntriesByMachine(metricsDir, config.user, null, toolKey)` walk, split the own machine's entries out of the map, and use `effectiveLocal = maxMergeEntries(local, ownSnapshots)` (per-day whole-entry max on `totalCost`, defined in `src/node/core/fetcher.ts`) as the own-machine share before sum-merging the other machines; `-u ` (repo-only) and single mode are unchanged - `syncMetrics()` MUST: (1) git add user dir, (2) commit if changes, (3) pull --rebase, (4) push (with one retry). All git commands MUST be invoked via `execFile("git", [...argv])` (no shell subprocess) — the internal `git` helper is `(args: string[]) => execFileAsync("git", ["-C", metricsDir, ...args])`. Argv entries (including `metricsDir`) are passed as literal strings; no quote parsing is performed. Interrupted-rebase recovery (`execFile("git", ["-C", metricsDir, "rebase", "--abort"])`) and single-retry push preserve their pre-change semantics - `fullSync()` MUST: fetch all tools locally, write metrics, sync via git, touch `.last-sync` timestamp - `isStale()` MUST return true if `.last-sync` is older than 3 hours or missing @@ -21,12 +23,15 @@ Multi-machine sync (`src/node/sync/sync.ts`) enables aggregating AI usage costs - Clone failures MUST write a marker file (`.clone-failed`) with ISO timestamp; retry suppressed for 3 hours - `init-metrics` MUST clone the metrics repo and clear any clone-failed marker - When metrics dir is missing and can't be cloned, MUST fall back to single mode with a warning +- `scripts/repair-metrics.mjs` (one-time repair tooling; standalone `node` script, NOT bundled into `dist/tu.mjs`, no imports from `src/`) MUST restore shrunk day-files from the metrics repo's git history: one `git log --format=%H%x09%cs --name-only -- '*.jsonl'` walk builds per-file commit lists (no per-file `git log`), `git show :` finds each tracked day-file's historical-max `totalCost`, and files below their max by more than `CENT_TOLERANCE` ($0.01) are reported with path, current, max, max-commit short SHA + date, delta, per-user subtotals, and a grand total. Dry-run is the default (no modification); `--write` restores the full historical-max blob byte-exact into the working tree only (no commit/push — left to the user; idempotent). `--repo` defaults to `~/.tu/metrics_repo`; missing/non-git repo and unknown args fail loud (stderr + exit 1); deleted-at-commit paths and unparseable historical blobs are skipped without crashing ## Design Decisions - **Git as sync transport**: Using a git repository avoids building a custom sync server. Commit/pull/push is simple and works over SSH. Rebase on pull avoids merge commits. - **Per-day JSONL files**: One file per tool per day enables efficient incremental writes and avoids conflicts when multiple machines sync. -- **User-scoped remote reads**: `readRemoteEntries` reads only from the target user's directory (default: config user). The `excludeMachine` parameter (default: config machine) prevents double-counting with local data. When `-u` targets a different user, `excludeMachine` is `null` to include all of that user's machines. When `-u` targets the same user as `config.user`, the default path is used instead (local fetch, cached unless `--fresh`, plus merge), ensuring identical behavior to no `-u` flag. +- **User-scoped remote reads**: `readRemoteEntries`/`readRemoteEntriesByMachine` read only from the target user's directory (default: config user). Since 260610-srmi every production call site passes `excludeMachine = null` (all machines in one walk): double-counting with local data is prevented by max-merging the own machine's snapshots into the live view instead of excluding the own dir — the old exclusion created a self-view blind spot where a machine could not see its own synced history once its local transcripts were purged. When `-u` targets a different user, the path is repo-only across all of that user's machines. When `-u` targets the same user as `config.user`, the default path is used instead (local fetch, cached unless `--fresh`, plus merge), ensuring identical behavior to no `-u` flag. The now test-only `excludeMachine` parameter is flagged as a deletion candidate for a follow-up signature cleanup (260610-srmi plan). +- **Day-files are high-water marks** (260610-srmi): Claude Code purges session transcripts older than ~30 days, so a live ccusage fetch for an old date collapses toward zero. Treating each day-file as the fullest complete snapshot ever observed fixes two bugs with one principle: (1) writes — `writeMetrics` skips shrinking writes, where unconditional overwrites had silently destroyed correct history on every post-purge sync (measured before the guard: $5,160.63 across 21 day-files for one user); (2) display — each machine max-merges its own repo snapshots back into its live view so its synced history stays visible to itself after the purge. Whole entries only — never per-field max (fabricates chimera entries mixing token/cost fields from different snapshots, violating Constitution V) and never residual+snapshot sums (double-counts the surviving transcripts of partially-purged days). +- **One-time repair from git history, manual and working-tree only** (260610-srmi): nothing was ever deleted from the metrics repo's history — only overwritten — so `scripts/repair-metrics.mjs` restores each shrunk day-file losslessly by writing back the full blob of the commit where its `totalCost` peaked (never patching just the cost field). Dry-run by default; `--write` touches the working tree only, leaving review/commit/push to the user. Standalone unbundled script (precedent: `scripts/help-dump.mjs`) keeps Constitution III intact. Sequencing constraint: run only after the guarded binary is installed on every actively-syncing machine, otherwise an old binary re-clobbers restored values at the rolling retention edge. - **Graceful degradation**: If multi mode is configured but the metrics repo is unavailable, the tool falls back to single mode rather than failing. This ensures the tool always works for local data. - **Clone-failed marker with cooldown**: Prevents repeated clone attempts on every invocation when the repo is unreachable (e.g., no network). 3-hour cooldown matches the staleness threshold. @@ -40,3 +45,4 @@ Multi-machine sync (`src/node/sync/sync.ts`) enables aggregating AI usage costs | 2026-03-07 | Fixed `-u` same-user: falls through to fresh-fetch path instead of reading stale repo data | | 2026-03-07 | Added `readRemoteEntriesByMachine` returning grouped `Map`; refactored `readRemoteEntries` to delegate and flatten | | 2026-04-23 | Migrated git invocations from `exec("git -C ... ...")` to `execFile("git", [...argv])` — no shell fork, paths with spaces/quotes pass through as literal argv entries (260423-lx0g) | +| 2026-06-10 | Never-shrink guard in `writeMetrics` (skip silently when incoming `totalCost` is lower than the existing day-file's); display path max-merges own-machine snapshots back into the live view (all production reads now pass `excludeMachine = null`); added standalone one-time repair script `scripts/repair-metrics.mjs` (dry-run default, `--write` working-tree only) (260610-srmi) | diff --git a/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl b/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl new file mode 100644 index 0000000..d4e2993 --- /dev/null +++ b/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl @@ -0,0 +1,14 @@ +{"action":"enter","driver":"fab-new","event":"stage-transition","stage":"intake","ts":"2026-06-10T14:43:30Z"} +{"args":"Fix metrics history destruction and self-exclusion blind spot in multi-machine sync, plus one-time repo repair. Three ordered parts (backlog items r01f, 87hw, gsji in main worktree backlog): (1) Guard writeMetrics so a sync can never shrink history — Claude Code purges transcripts \u003e30 days old, live ccusage data for old dates collapses toward zero, and every sync/merged-fetch overwrites correct historical per-day JSONL files with post-purge residue (~$5,160 destroyed across sahil machines, ongoing). (2) One-time repair script restoring shrunk day-files to historical max from git history, all users/machines, run after guard ships. (3) Fix self-exclusion blind spot: merge own-machine repo entries into local view via per-day max so purged local history doesn't vanish from a machine's own view.","cmd":"fab-new","event":"command","ts":"2026-06-10T14:43:30Z"} +{"delta":"+3.5","event":"confidence","score":3.5,"trigger":"calc-score","ts":"2026-06-10T14:45:44Z"} +{"delta":"+0.0","event":"confidence","score":3.5,"trigger":"calc-score","ts":"2026-06-10T14:45:55Z"} +{"cmd":"fab-fff","event":"command","ts":"2026-06-10T14:58:49Z"} +{"action":"enter","driver":"fab-fff","event":"stage-transition","stage":"apply","ts":"2026-06-10T14:59:05Z"} +{"cmd":"fab-continue","event":"command","ts":"2026-06-10T15:00:54Z"} +{"action":"enter","driver":"fab-continue","event":"stage-transition","stage":"review","ts":"2026-06-10T15:19:20Z"} +{"cmd":"fab-continue","event":"command","ts":"2026-06-10T15:21:30Z"} +{"action":"enter","driver":"fab-fff","event":"stage-transition","stage":"hydrate","ts":"2026-06-10T15:36:59Z"} +{"event":"review","result":"passed","ts":"2026-06-10T15:36:59Z"} +{"cmd":"fab-continue","event":"command","ts":"2026-06-10T15:38:21Z"} +{"action":"enter","driver":"fab-fff","event":"stage-transition","stage":"ship","ts":"2026-06-10T15:44:08Z"} +{"cmd":"git-pr","event":"command","ts":"2026-06-10T15:45:52Z"} diff --git a/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml b/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml new file mode 100644 index 0000000..f8f98bd --- /dev/null +++ b/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml @@ -0,0 +1,49 @@ +id: srmi +name: 260610-srmi-fix-metrics-history-destruction +created: 2026-06-10T14:43:30Z +created_by: sahil-noon +change_type: fix +issues: [] +progress: + intake: done + apply: done + review: done + hydrate: done + ship: active + review-pr: pending +plan: + generated: true + task_count: 10 + acceptance_count: 21 + acceptance_completed: 21 +confidence: + certain: 6 + confident: 5 + tentative: 0 + unresolved: 0 + score: 3.5 + fuzzy: true + dimensions: + signal: 81.4 + reversibility: 83.2 + competence: 85.5 + disambiguation: 79.1 +stage_metrics: + intake: {started_at: "2026-06-10T14:43:30Z", driver: fab-new, iterations: 1, completed_at: "2026-06-10T14:59:05Z"} + apply: {started_at: "2026-06-10T14:59:05Z", driver: fab-fff, iterations: 1, completed_at: "2026-06-10T15:19:20Z"} + review: {started_at: "2026-06-10T15:19:20Z", driver: fab-continue, iterations: 1, completed_at: "2026-06-10T15:36:59Z"} + hydrate: {started_at: "2026-06-10T15:36:59Z", driver: fab-fff, iterations: 1, completed_at: "2026-06-10T15:44:08Z"} + ship: {started_at: "2026-06-10T15:44:08Z", driver: fab-fff, iterations: 1} +prs: [] +true_impact: + added: 0 + deleted: 0 + net: 0 + excluding: + added: 0 + deleted: 0 + net: 0 + computed_at: "2026-06-10T15:44:08Z" + computed_at_stage: hydrate +# true_impact: lazily created on first apply-finish (no placeholder here). +last_updated: 2026-06-10T15:44:08Z diff --git a/fab/changes/260610-srmi-fix-metrics-history-destruction/intake.md b/fab/changes/260610-srmi-fix-metrics-history-destruction/intake.md new file mode 100644 index 0000000..645b37d --- /dev/null +++ b/fab/changes/260610-srmi-fix-metrics-history-destruction/intake.md @@ -0,0 +1,110 @@ +# Intake: Fix Metrics History Destruction & Self-Exclusion Blind Spot + +**Change**: 260610-srmi-fix-metrics-history-destruction +**Created**: 2026-06-10 +**Status**: Draft + +## Origin + +Conversational. A `/fab-discuss` session investigated why `tu dh` output changed drastically between two runs on the same day (2026-06-10), focusing on 2026-04-24 dropping from $308.12 to $9.46. The investigation reconstructed the full causal chain from the metrics repo's git history (`~/.tu/metrics_repo`), the local ccusage cache, and the source code. The user then queued three fixes as ordered backlog items (`r01f`, `87hw`, `gsji` — in the **main worktree's** `fab/backlog.md`, not this worktree's) and requested this draft bundling all three into one change: + +> Fix metrics history destruction and self-exclusion blind spot in multi-machine sync, plus one-time repo repair. Three ordered parts: (1) Guard writeMetrics (src/node/sync/sync.ts:22-36) so a sync can never shrink history — Claude Code purges transcripts >30 days old, so live ccusage data for old dates collapses toward zero, and every fullSync currently overwrites correct historical per-day JSONL files with post-purge residue. Fix: skip the write when the existing file's totals exceed the new entry's. (2) One-time repair script that walks the metrics repo git history and restores every shrunk day-file to its historical max value, across all users/machines; run manually after the guard ships. (3) Fix the self-exclusion blind spot: a machine's own repo dir is excluded from remote reads and replaced by the live local fetch, so once local transcripts are purged a machine cannot see its own synced history. Fix: merge own-machine repo entries into the local view via per-day max. + +Key decisions from the discussion: bundle all three parts in one change (user approved); guard = skip-write (not per-field max); repair = standalone script in `scripts/`, dry-run by default, executed manually post-release; self-view fix = per-day whole-entry **max** (not sum, to avoid double-counting partially-purged days). + +## Why + +**The problem.** Claude Code deletes session transcripts older than ~30 days. ccusage computes daily costs by scanning those transcripts, so a machine's *live* view of any day older than the retention window collapses toward $0. tu's multi-machine sync treats the live fetch as authoritative and **unconditionally overwrites** the per-day JSONL snapshots in the shared metrics repo (`writeMetrics`, `src/node/sync/sync.ts:29-35`). The result is permanent, silent destruction of correct historical data, re-triggered every time the retention window rolls forward. + +**Measured damage** (metrics repo, user `sahil`, diff `f509276` → HEAD as of 2026-06-10): **$5,160.63 destroyed across 21 day-files**. Example: commit `0bb81ca` (2026-05-30, from `dev-ws-sahil02`) rewrote 16 April files in one sync — `cc-2026-04-24.jsonl` went $308.12 → $9.46, `cc-2026-04-23.jsonl` $1,016.32 → $49.24, `cc-2026-04-22.jsonl` $840.12 → $5.27. `Sahils-Mac-mini.local` shows the same pattern (`cc-2026-04-26.jsonl` $18.78 → $2.73 on 2026-06-03). Other users (akshay, pulkit, shreyas, vivek) almost certainly have the same rot. The destruction is **ongoing**: early-June syncs shrank 2026-05-03..05-09 files as those dates crossed the retention horizon. + +**A second, compounding bug.** When displaying data, a machine excludes its *own* directory from repo reads (`readRemoteEntriesByMachine`, `src/node/sync/sync.ts:125`) and substitutes the live local fetch — under the assumption that live local data is a superset of the machine's repo snapshots. The retention purge breaks that assumption: once transcripts are gone, the machine's own synced history is invisible *to itself*, even though it sits intact in the repo. Concretely: `Sahils-MacBook-Pro.local` has $236.00 recorded for 2026-04-24 in the repo, but `tu dh` on that MacBook shows $0 of it. The true 2026-04-24 spend was ~$544 (devws $308.12 + MacBook $236.00); the user never saw a correct number — old output $308.12 (Bug B hid the MacBook's share), new output $9.46 (Bug A destroyed the devws share, Bug B still hides the MacBook's). + +**If we don't fix it:** every machine destroys another slice of shared history each time it syncs after a purge, and every machine under-reports its own past. The repo's git history still holds the true values today; the longer the wait, the more noise accumulates on top. + +**Why this approach:** the day-file snapshots are exactly a high-water mark of complete data — "never shrink" restores their intended semantics at the single choke point both callers share. Restoring from git history is lossless because nothing was ever deleted, only overwritten in newer commits. Per-day max for the self-view reuses the same high-water-mark principle for display. + +## What Changes + +### 1. Never-shrink guard in `writeMetrics` (`src/node/sync/sync.ts`) + +Current behavior (sync.ts:29-35): for every entry in the live fetch, unconditionally `writeFileSync` the day-file. Called from **two** sites — `fetchToolMerged` (`src/node/core/cli.ts:467`, i.e. on *every* data-displaying invocation in multi mode) and `fullSync` (`src/node/sync/sync.ts:188`). The guard must therefore live inside `writeMetrics` itself, covering both. + +New behavior, per entry: + +- If the day-file does not exist → write (unchanged). +- If it exists and parses as a `UsageEntry`: write the incoming entry **only if** `incoming.totalCost >= existing.totalCost`; otherwise skip silently, keeping the existing file. +- If it exists but is empty/unparseable → write (treat as absent; matches the read path's skip-silently posture). +- Equal values → write (idempotent refresh; keeps today's file updating normally as the day grows). + +Rejected alternative: per-field `max` across the two entries — it fabricates a chimera entry whose token fields and cost come from different snapshots, violating Constitution V (consistent data model). The whole-entry rule keeps every file an atomic snapshot that was real at some point in time. + +Out of scope (possible follow-up, discussed but excluded by the user's backlog selection): a stderr warning when the guard skips shrunk entries. + +### 2. One-time repair script (`scripts/repair-metrics.mjs`, new) + +Standalone Node script (precedent: `scripts/help-dump.mjs`) — **not** bundled into `dist/tu.mjs`; Constitution III untouched. Run manually: `node scripts/repair-metrics.mjs [--repo ~/.tu/metrics_repo] [--write]`. + +Algorithm: + +1. Enumerate every commit touching `*.jsonl` once (`git log --format=%H --name-only -- '*/2026/**'` style walk over the full history of `main`), building a per-file commit list — avoids a `git log` per file. +2. For each tracked day-file, `git show :` across its commits to find the version with **maximum `totalCost`** (the historical high-water mark). +3. Compare with the working-tree/HEAD value. If HEAD is lower by more than a cent, the file is "shrunk". +4. **Dry-run (default):** print a per-file table — path, HEAD value, max value, commit/date of max, delta — plus per-user and grand totals. No writes. +5. **`--write`:** restore each shrunk file's content (the full original JSON line, not just the cost field) in the working tree. Committing and pushing are deliberately left to the user for review. Idempotent — re-running reports nothing left to repair. + +Scope: all users and machines in the repo (akshay/pulkit/shreyas/vivek included, not just sahil). + +**Sequencing constraint (the reason the backlog items are ordered):** the repair must run only after part 1 has shipped and the actively-syncing machines have upgraded — otherwise the next sync from an old binary re-clobbers restored values at the rolling retention edge (currently ~mid-May dates). April-era dates are already outside every machine's live window and are safe either way. + +### 3. Self-view max-merge (`src/node/core/cli.ts` + `src/node/core/fetcher.ts`) + +Current behavior (`fetchToolMerged`, cli.ts:449-472): `local = fetchHistory(...)` (live ccusage) → `writeMetrics(local)` → `remote = readRemoteEntries(metricsDir, config.user, /* excludeMachine */ config.machine, toolKey)` → `mergeEntries(local, remote)`, where `mergeEntries` (fetcher.ts:260) **sums** token/cost fields by date label. The machine's own repo snapshots are never read back. + +New behavior: + +1. Add a pure helper in `fetcher.ts` (Constitution V — pure function over `UsageEntry[]`): `maxMergeEntries(a, b)` — per date label, pick **whichever whole entry has the greater `totalCost`** (no field mixing, no summing). Output sorted by label like `mergeEntries`. +2. In `fetchToolMerged`: read the machine's own snapshots (its single machine dir) and compute `effectiveLocal = maxMergeEntries(local, ownSnapshots)`; then `mergeEntries(effectiveLocal, remote)` as before. For dates within the live window, live wins (equal or greater — it includes in-flight today data); for purged dates, the repo snapshot resurfaces. +3. Same treatment in `fetchToolMergedWithMachines` (cli.ts:481+), so `--by-machine` shows the corrected own-machine column from the same `effectiveLocal`. +4. Unaffected paths: `-u ` (targetUser ≠ config.user) is already repo-only with `excludeMachine = null`; single mode has no repo. Watch mode uses the same fetch path and is fixed for free. + +Why max, not sum: for a partially-purged date the live fetch still returns a residual entry (e.g. $9.46 of the true $308.12); summing residual + snapshot would double-count the surviving transcripts. + +Expected user-visible outcome (real data): on the MacBook, 2026-04-24 displays its own $236.00 again (plus whatever the repo holds for other machines — $9.46 pre-repair, $308.12 post-repair). Merged totals only ever increase relative to today's behavior. + +## Affected Memory + +- `sync/multi-machine.md`: (modify) — `writeMetrics()` requirement gains the never-shrink guard; new requirement for own-machine snapshot max-merge in the display path; design decision "day-files are high-water marks". +- `cli/data-pipeline.md`: (modify) — merge pipeline now includes `maxMergeEntries` step for own-machine entries in `fetchToolMerged`/`fetchToolMergedWithMachines`. + +## Impact + +- **Code**: `src/node/sync/sync.ts` (writeMetrics guard), `src/node/core/fetcher.ts` (new `maxMergeEntries` pure helper), `src/node/core/cli.ts` (`fetchToolMerged`, `fetchToolMergedWithMachines`), `scripts/repair-metrics.mjs` (new, unbundled). +- **Tests** (node:test via tsx, co-located): `src/node/sync/__tests__/` (guard: fresh write / shrink-skip / grow-overwrite / corrupt-file / equal-value), `src/node/core/__tests__/` (maxMergeEntries; merged-fetch behavior with purged local). Repair script: unit-testable against a fixture git repo (follow `cli-sync.test.ts` bare-repo fixture pattern); keep hermetic per the open backlog note on env isolation. +- **Data flow**: all three tools (cc/codex/oc) — code paths are toolKey-generic. +- **Output stability**: table/JSON *format* unchanged; displayed *values* increase for purge-affected dates. Constitution's Output Stability clause → release as a **minor** version bump (0.4.17 → 0.5.0). +- **Deployment sequencing**: part 2's repair run happens manually after the release is installed on actively-syncing machines (brew upgrade). +- **Backlog**: at archive time, mark `r01f`, `87hw`, `gsji` done in the **main worktree's** `fab/backlog.md` (`idea done --main`). +- **Specs**: `docs/specs/usage.md` describes sync/merge semantics — human-curated; flag for review during hydrate. + +## Open Questions + +None — the investigation resolved the mechanism end-to-end, and the user confirmed scope and ordering. + +## Assumptions + +| # | Grade | Decision | Rationale | Scores | +|---|-------|----------|-----------|--------| +| 1 | Certain | All three backlog items (r01f, 87hw, gsji) bundled into this single change, plan tasks ordered 1→2→3 | Discussed — user approved bundling ("all 3 in one fab change?" → "just go ahead") | S:90 R:80 A:85 D:90 | +| 2 | Confident | Guard = skip whole-entry write when `incoming.totalCost < existing.totalCost`; rejected per-field max (chimera entry violates Constitution V) | One obvious interpretation of "never shrink" that keeps snapshots atomic | S:75 R:80 A:80 D:65 | +| 3 | Confident | Comparison key is `totalCost` alone (not totalTokens) | Cost is the user-facing quantity; live values are monotonic within a day; token/cost divergence has no realistic source here | S:70 R:75 A:75 D:60 | +| 4 | Certain | Guard lives inside `writeMetrics()`, covering both call sites (cli.ts:467 per-invocation write and sync.ts:188 fullSync) | Code analysis — single choke point; both callers verified during investigation | S:85 R:85 A:95 D:90 | +| 5 | Certain | Repair is a standalone `scripts/repair-metrics.mjs`, not bundled into `dist/tu.mjs`, run manually | Matches `scripts/help-dump.mjs` precedent; Constitution III (single bundle) untouched; one-time ops task doesn't belong in the CLI surface | S:85 R:90 A:90 D:85 | +| 6 | Confident | Repair defaults to dry-run report; `--write` restores working tree only; commit/push left to user; restores full historical-max file content | Safest reviewable flow for destructive-adjacent data ops; user didn't specify CLI shape | S:70 R:85 A:80 D:70 | +| 7 | Confident | Self-view fix = per-label whole-entry `max(live, own snapshot)` via new pure `maxMergeEntries`, then existing sum-merge with other machines; applied in both `fetchToolMerged` and `fetchToolMergedWithMachines` | Discussed — max-not-sum explicitly chosen to avoid double-counting partially-purged days | S:80 R:75 A:80 D:70 | +| 8 | Certain | Applies to all three tools (cc/codex/oc) | Code is toolKey-generic at every touched site | S:90 R:85 A:95 D:90 | +| 9 | Confident | Release as minor version bump (0.5.0) | Constitution Output Stability: values change materially for affected dates even though format is identical | S:65 R:90 A:75 D:70 | +| 10 | Certain | Tests: node:test runner, co-located `__tests__/` dirs; repair script tested against a seeded bare-repo git fixture | Constitution Test Runner + Test Location clauses are deterministic | S:95 R:90 A:95 D:95 | +| 11 | Certain | Non-goal: stderr warning when guard skips shrunk entries (discussed hardening) is excluded | Discussed — user selected only items 1–3 for the backlog; trivial to add later | S:90 R:80 A:90 D:85 | + +11 assumptions (6 certain, 5 confident, 0 tentative, 0 unresolved). diff --git a/fab/changes/260610-srmi-fix-metrics-history-destruction/plan.md b/fab/changes/260610-srmi-fix-metrics-history-destruction/plan.md new file mode 100644 index 0000000..825e93f --- /dev/null +++ b/fab/changes/260610-srmi-fix-metrics-history-destruction/plan.md @@ -0,0 +1,198 @@ +# Plan: Fix Metrics History Destruction & Self-Exclusion Blind Spot + +**Change**: 260610-srmi-fix-metrics-history-destruction +**Status**: In Progress +**Intake**: `intake.md` + +## Requirements + +### Sync: Never-Shrink Guard in `writeMetrics` (Part 1) + +#### R1: writeMetrics never shrinks a day-file +`writeMetrics()` (`src/node/sync/sync.ts`) MUST skip writing a per-day JSONL file when the existing file parses as a `UsageEntry` and the incoming entry's `totalCost` is strictly less than the existing entry's `totalCost`. The skip MUST be silent (no stderr output). In all other cases (file absent, file empty, file unparseable as a `UsageEntry`, incoming `totalCost >= existing totalCost`) the write MUST proceed exactly as before. The guard MUST live inside `writeMetrics` itself so both call sites (`fetchToolMerged` in `src/node/core/cli.ts` and `fullSync` in `src/node/sync/sync.ts`) are covered. The comparison key is `totalCost` alone. + +- **GIVEN** a day-file `cc-2026-04-24.jsonl` containing an entry with `totalCost: 308.12` +- **WHEN** `writeMetrics` is called with an incoming entry for `2026-04-24` with `totalCost: 9.46` +- **THEN** the file is left untouched (still contains the `308.12` entry) and no error or warning is emitted + +- **GIVEN** a day-file containing an entry with `totalCost: 1.0` +- **WHEN** `writeMetrics` is called with an incoming entry for the same date with `totalCost: 2.0` (or exactly `1.0`) +- **THEN** the file is overwritten with the incoming entry (grow and equal-value refresh both write) + +- **GIVEN** no day-file exists for the entry's date, OR the existing file is empty, OR its content does not parse as a `UsageEntry` with a numeric `totalCost` +- **WHEN** `writeMetrics` is called +- **THEN** the incoming entry is written (treat as absent — matches the read path's skip-silently posture) + +### Repair: One-Time History Restoration Script (Part 2) + +#### R2: Standalone repair script with single-walk history discovery +A new standalone script `scripts/repair-metrics.mjs` MUST walk the metrics repo's git history **once** (a single `git log --format=... --name-only -- '*.jsonl'` invocation, not one `git log` per file) to build a per-file commit list, then for each tracked day-file find the historical version with **maximum `totalCost`** via `git show :`. The script MUST be runnable as `node scripts/repair-metrics.mjs [--repo ] [--write]` with `--repo` defaulting to `~/.tu/metrics_repo`. It MUST cover all users and machines in the repo. It MUST NOT be bundled into `dist/tu.mjs` and MUST NOT be imported by anything under `src/` (Constitution III untouched; precedent: `scripts/help-dump.mjs`). Historical versions that are unparseable, and commits where a path was deleted, MUST be skipped without crashing. Operational errors (missing repo, not a git repo) MUST print a message to stderr and exit 1. + +- **GIVEN** a metrics repo where `sahil/2026/devws/cc-2026-04-24.jsonl` was committed with `totalCost: 308.12` and later overwritten by a commit with `totalCost: 9.46` +- **WHEN** the script scans the repo +- **THEN** it identifies `308.12` as the historical maximum for that file using one history walk plus per-version `git show` reads + +- **GIVEN** `--repo` points at a directory that is not a git repository +- **WHEN** the script runs +- **THEN** it prints an error to stderr and exits with code 1 + +#### R3: Dry-run report is the default +By default (no `--write`), the script MUST print a per-file report of every "shrunk" file — path, current (HEAD/working-tree) `totalCost`, historical max `totalCost`, the commit (short SHA) and date of the max, and the delta — plus per-user subtotals and a grand total, and MUST NOT modify any file. A file is "shrunk" when its current value is lower than its historical max by **more than one cent** (`delta > 0.01`). Files at their historical max (or within a cent) MUST NOT be listed. When nothing is shrunk, the script MUST say so explicitly. + +- **GIVEN** a repo with one shrunk file (max `308.12`, current `9.46`) and one never-shrunk file +- **WHEN** the script runs without `--write` +- **THEN** the report lists only the shrunk file with current/max/delta and the max commit reference, prints per-user and grand totals, and the working tree is byte-identical to before the run + +- **GIVEN** a file whose current value is lower than its max by `0.005` +- **WHEN** the script runs +- **THEN** the file is not reported as shrunk (within-a-cent tolerance) + +#### R4: `--write` restores full content in the working tree only +With `--write`, the script MUST restore each shrunk file's **full original content** (the exact bytes of the historical-max version, not just the cost field) into the working tree. It MUST NOT commit or push — review, commit, and push are deliberately left to the user. The operation MUST be idempotent: a second run after a successful `--write` reports nothing left to repair. + +- **GIVEN** a shrunk day-file whose historical-max version had specific token fields and cost +- **WHEN** the script runs with `--write` +- **THEN** the working-tree file is byte-identical to the historical-max blob, no git commit is created, and a re-run reports nothing to repair + +### CLI: Self-View Max-Merge (Part 3) + +#### R5: `maxMergeEntries` pure helper +`src/node/core/fetcher.ts` MUST export a new pure function `maxMergeEntries(a: UsageEntry[], b: UsageEntry[]): UsageEntry[]` that, per date label, picks **whichever whole entry has the greater `totalCost`** — no field mixing, no summing. On equal `totalCost`, the entry from `a` (the live local fetch at the call sites) wins. Output MUST be sorted ascending by label (like `mergeEntries`) and the function MUST NOT mutate its inputs (Constitution V: pure functions over `UsageEntry` types). + +- **GIVEN** `a = [{label: "2026-04-24", totalCost: 9.46, ...}]` and `b = [{label: "2026-04-24", totalCost: 236.00, ...}]` +- **WHEN** `maxMergeEntries(a, b)` is called +- **THEN** the result contains exactly `b`'s whole entry for `2026-04-24` (every field from `b`, none from `a`) — not the sum + +- **GIVEN** entries with non-overlapping labels in `a` and `b` +- **WHEN** `maxMergeEntries(a, b)` is called +- **THEN** all entries appear once, sorted ascending by label, and neither input array is mutated + +#### R6: `fetchToolMerged` merges own-machine snapshots into the local view +In `fetchToolMerged` (`src/node/core/cli.ts`), the default (own-user) path MUST read the machine's own repo snapshots back and compute `effectiveLocal = maxMergeEntries(local, ownSnapshots)` before the existing sum-merge with other machines (`mergeEntries(effectiveLocal, remote)`). For dates within the live window, live data wins (equal or greater); for purged dates, the repo snapshot resurfaces. The `-u ` path (repo-only, `excludeMachine = null`) MUST remain unchanged. The change applies to all three tools (the code is toolKey-generic). + +- **GIVEN** a machine whose live fetch returns `totalCost: 9.46` for `2026-04-24` (post-purge residue), whose own repo snapshot for that date holds `236.00`, and another machine's snapshot holds `308.12` +- **WHEN** the merged entries are computed +- **THEN** the `2026-04-24` total is `236.00 + 308.12 = 544.12` (own max, then sum across machines) — not `9.46 + 308.12` and not `9.46 + 236.00 + 308.12` + +- **GIVEN** a date within the live window where live `totalCost` equals or exceeds the own snapshot (e.g. today, still growing) +- **WHEN** the merged entries are computed +- **THEN** the live entry is used for the own-machine share (idempotent with today's behavior; merged totals only ever increase relative to the pre-change pipeline) + +#### R7: `fetchToolMergedWithMachines` applies the same self-view correction +`fetchToolMergedWithMachines` (`src/node/core/cli.ts`) MUST apply the same own-snapshot max-merge in multi mode: the own-machine entry in the returned `machineMap` MUST be `maxMergeEntries(local, ownSnapshots)`, so `--by-machine` shows the corrected own-machine column, and the flattened `entries` sum reflects it. Single mode (no repo) MUST remain unchanged (machineMap contains only the live local entries). + +- **GIVEN** multi mode, an own-machine snapshot of `236.00` for a purged date, live residue `9.46`, and a remote machine snapshot of `308.12` +- **WHEN** `--by-machine` data is computed +- **THEN** the own-machine column shows `236.00` for that date, the remote machine column shows `308.12`, and the merged total is `544.12` + +- **GIVEN** single mode (`config.mode !== "multi"`) +- **WHEN** `fetchToolMergedWithMachines` runs +- **THEN** behavior is byte-identical to before this change (one machine, live entries only, no repo reads) + +### Non-Goals + +- No stderr warning when the guard skips shrunk entries — discussed hardening, explicitly excluded by the user's backlog selection (trivial follow-up). +- No per-field `max` across entries — it fabricates a chimera entry violating Constitution V; whole-entry semantics only. +- No automatic execution of the repair script — it ships in `scripts/` and is run manually after the release is installed on actively-syncing machines. This apply stage MUST NOT run it against the real `~/.tu/metrics_repo`. +- No version bump in `package.json` — release (0.5.0 minor, per Output Stability) is handled outside this change. + +### Design Decisions + +1. **Guard inside `writeMetrics`**: single choke point covering both callers (`fetchToolMerged` per-invocation write and `fullSync`) — *Why*: any caller-side guard would have to be duplicated and can drift — *Rejected*: guarding at each call site. +2. **Skip-write, not per-field max**: when the incoming entry is smaller, keep the existing file verbatim — *Why*: every day-file remains an atomic snapshot that was real at some point in time (Constitution V) — *Rejected*: per-field `max` (chimera entries mixing token/cost fields from different snapshots). +3. **Self-view merge = per-day whole-entry max, then existing sum-merge**: `maxMergeEntries(local, ownSnapshots)` feeds the unchanged `mergeEntries` — *Why*: a partially-purged day still yields a residual live entry; summing residual + snapshot would double-count surviving transcripts — *Rejected*: summing own snapshots into the remote set. +4. **Repair restores full file content from the max commit**: `git show :` bytes written verbatim — *Why*: lossless; keeps token fields and cost from one real snapshot — *Rejected*: patching only `totalCost` into the current file. +5. **Single directory walk for own + remote snapshots**: the rewired fetch paths call `readRemoteEntriesByMachine(metricsDir, user, /* excludeMachine */ null, toolKey)` once and split the own machine out of the returned map — *Why*: reuses the existing utility unchanged and keeps one well-exercised path (code-quality: minimum pathways) instead of adding a second single-machine read helper — *Rejected*: new `readMachineEntries()` helper plus a second exclude-walk. + +## Tasks + +### Phase 1: Never-Shrink Guard (intake part 1) + +- [x] T001 Add the never-shrink guard inside `writeMetrics` in `src/node/sync/sync.ts`: before each `writeFileSync`, read the existing day-file; if it parses as a `UsageEntry` with numeric `totalCost` and `incoming.totalCost < existing.totalCost`, skip silently; absent/empty/unparseable files and grow/equal cases write as before +- [x] T002 Add guard tests to `src/node/sync/__tests__/sync.test.ts` (`writeMetrics` describe block): fresh write, shrink-skip, grow-overwrite, equal-value refresh, empty file, corrupt/non-entry JSON file; run `npx tsx --test src/node/sync/__tests__/sync.test.ts` + +### Phase 2: Repair Script (intake part 2) + +- [x] T003 Create `scripts/repair-metrics.mjs` (standalone, plain `node`, `node:`-prefixed builtins only, no imports from `src/`): parse `--repo` (default `~/.tu/metrics_repo`) and `--write`; enumerate tracked day-files via `git ls-files`; build per-file commit lists from one `git log --format --name-only -- '*.jsonl'` walk; find each file's historical-max `totalCost` via `git show`; dry-run report (path, current, max, max-commit short SHA + date, delta, per-user + grand totals, cent tolerance as a named constant); `--write` restores full max-version content into the working tree only; idempotent; stderr + exit 1 on operational errors +- [x] T004 Add `src/node/sync/__tests__/repair-metrics.test.ts` driving the script via `spawnSync(process.execPath, [script, "--repo", fixture])` against a seeded local git fixture repo (per-repo `user.email`/`user.name`, `main` branch — follow the `cli-sync.test.ts`/`sync.test.ts` fixture pattern; hermetic: no HOME/TU_* dependence, never touches the real metrics repo): dry-run reporting + no modification, within-a-cent tolerance, never-shrunk files omitted, multi-user totals, max across 3+ versions, unparseable historical version skipped, `--write` byte-exact restore + no commit + idempotent re-run, non-repo error path; run `npx tsx --test src/node/sync/__tests__/repair-metrics.test.ts` + +### Phase 3: Self-View Max-Merge (intake part 3) + +- [x] T005 Add pure `maxMergeEntries(a, b)` to `src/node/core/fetcher.ts` next to `mergeEntries`: per-label whole-entry max on `totalCost`, ties keep `a`'s entry, output sorted by label, no input mutation +- [x] T006 Add `maxMergeEntries` tests to `src/node/core/__tests__/fetcher.test.ts`: picks larger whole entry (no field mixing/summing), tie keeps first argument's entry, non-overlapping union, empty inputs, sorted output, no mutation; run `npx tsx --test src/node/core/__tests__/fetcher.test.ts` +- [x] T007 Rewire `fetchToolMerged` in `src/node/core/cli.ts` (default own-user path): replace the `readRemoteEntries(..., config.machine, ...)` call with one `readRemoteEntriesByMachine(..., null, ...)` read; split own-machine snapshots from other machines; `effectiveLocal = maxMergeEntries(local, ownSnapshots)`; `mergeEntries(effectiveLocal, remote)`; `-u` other-user branch untouched; import `maxMergeEntries` from `./fetcher.js` +- [x] T008 Rewire `fetchToolMergedWithMachines` in `src/node/core/cli.ts` (multi-mode branch): read all machines with `excludeMachine = null`, set `machineMap.set(config.machine, maxMergeEntries(local, ownSnapshots))`, copy other machines as-is; single-mode branch untouched +- [x] T009 Add `src/node/core/__tests__/self-view-merge.test.ts`: composition test simulating the rewired pipeline against a temp metrics dir (seeded own-machine + remote-machine day-files, purged live view) — guarded `writeMetrics` → `readRemoteEntriesByMachine(null)` → own/others split → `maxMergeEntries` → `mergeEntries`; asserts purged-date resurfacing (own 236.00 + remote 308.12 = 544.12, not residue-summed), live-window dominance for today, and by-machine own-column correction; hermetic temp dirs only; run `npx tsx --test src/node/core/__tests__/self-view-merge.test.ts` + +### Phase 4: Validation + +- [x] T010 Run the full suite (`env -u TU_METRICS_REPO npm test`) and the bundle build (`npm run build`); confirm all tests green, the bundle compiles, and `dist/tu.mjs` does not contain repair-script code + +## Execution Order + +- Phases run strictly 1 → 2 → 3 → 4 (user-mandated ordering of the three intake parts). +- T005 blocks T007/T008 (cli.ts imports the new helper); T001 blocks T009 (composition test exercises the guarded `writeMetrics`). +- Running the repair script against the real `~/.tu/metrics_repo` is NOT part of this plan — it happens manually after release, once actively-syncing machines have upgraded (sequencing constraint from the intake). + +## Acceptance + +### Functional Completeness + +- [x] A-001 R1: `writeMetrics` skips silently when the incoming entry's `totalCost` is lower than the existing parsed day-file's; writes in all other cases; guard lives inside `writeMetrics` covering both call sites +- [x] A-002 R2: `scripts/repair-metrics.mjs` exists, runs standalone under plain `node` with `--repo`/`--write` flags, walks history with a single `git log` pass (no per-file `git log`), covers all users/machines, and is neither bundled into `dist/tu.mjs` nor imported by `src/` +- [x] A-003 R3: default invocation prints the shrunk-file report (path, current, max, max commit + date, delta, per-user and grand totals) and modifies nothing +- [x] A-004 R4: `--write` restores byte-exact historical-max content into the working tree, creates no commit, and a re-run reports nothing to repair +- [x] A-005 R5: `maxMergeEntries` is exported from `fetcher.ts`, picks whole entries by greater `totalCost` per label, sorts output, never mutates inputs +- [x] A-006 R6: `fetchToolMerged` computes `effectiveLocal = maxMergeEntries(local, ownSnapshots)` before `mergeEntries` with other machines; `-u` other-user path unchanged +- [x] A-007 R7: `fetchToolMergedWithMachines` sets the own-machine `machineMap` entry to the max-merged view in multi mode; single mode unchanged + +### Behavioral Correctness + +- [x] A-008 R1: equal-value and growing writes still overwrite (today's file keeps refreshing as the day grows) — the existing "overwrites existing file on re-run" test still passes +- [x] A-009 R6: for a purged date, merged output equals own-snapshot max plus other machines' sum (no double count of residual live data); merged totals never decrease relative to the pre-change pipeline + +### Scenario Coverage + +- [x] A-010 R1: tests cover fresh write / shrink-skip / grow-overwrite / equal-value / empty file / corrupt file +- [x] A-011 R2: repair tests run against a seeded fixture git repo (never the real metrics repo) and cover dry-run, tolerance, multi-user totals, max-across-3+-versions, unparseable version skip, `--write`, idempotence, and the non-repo error path +- [x] A-012 R5: `maxMergeEntries` tests cover overlap (whole-entry pick), tie-keeps-`a`, non-overlap union, empties, sort order, input immutability +- [x] A-013 R6 R7: composition test exercises the full rewired pipeline (guarded write → read-back → split → max-merge → sum-merge) including the by-machine own-column + +### Edge Cases & Error Handling + +- [x] A-014 R1: empty and unparseable existing day-files are treated as absent (write proceeds, no crash, no warning) +- [x] A-015 R2: deleted-at-some-commit paths and unparseable historical blobs are skipped without crashing; operational errors (missing/non-git `--repo`) warn on stderr and exit 1 (no unhandled throw) +- [x] A-016 R6: machines with no own snapshots (fresh machine, empty repo dir) behave as today (`ownSnapshots = []` → `effectiveLocal = local`) + +### Code Quality + +- [x] A-017 Pattern consistency: new code follows surrounding naming/structure (`node:` imports, `.js` extensions, `type` imports, functional style, no classes) +- [x] A-018 No unnecessary duplication: reuses `readRemoteEntriesByMachine`, `mergeEntries` shape, and existing test fixture patterns instead of new parallel utilities +- [x] A-019 Constitution V: `maxMergeEntries` is a pure function over `UsageEntry[]`; no chimera entries anywhere +- [x] A-020 No magic numbers: the one-cent repair tolerance is a named constant in the script +- [x] A-021 No swallowed errors beyond spec: repair script fails loud (stderr + exit 1) on operational errors; guard's silent skip is the specified behavior, mirroring the read path's posture + +## Notes + +- Check items as you review: `- [x]` +- All acceptance items must pass before `/fab-continue` (hydrate) +- If an item is not applicable, mark checked and prefix with **N/A**: `- [x] A-NNN **N/A**: {reason}` +- Test-environment caveat: an exported `TU_METRICS_REPO` or the developer's real `~/.tu.conf` leaks into pre-existing config-dependent suites — run validation with `env -u TU_METRICS_REPO npm test`. New tests in this change are hermetic (temp dirs, explicit `--repo`, no ambient TU_* reads). + +## Deletion Candidates + +- `excludeMachine` parameter of `readRemoteEntriesByMachine`/`readRemoteEntries` (`src/node/sync/sync.ts:121,152,184`) — after this change every production call site passes `null` (cli.ts:459, 474, 502, 527); the non-null filtering branch is exercised only by tests (including this change's pre/post comparison test). The parameter and its skip branch can be removed in a follow-up signature simplification. + +## Assumptions + +| # | Grade | Decision | Rationale | Scores | +|---|-------|----------|-----------|--------| +| 1 | Confident | Own + remote snapshots read in one `readRemoteEntriesByMachine(..., null, ...)` walk, own machine split from the map — instead of adding a single-machine read helper | Intake specifies *what* to read (own snapshots), not *how*; reuse of the existing utility honors minimum-pathways and avoids a second directory-walk path | S:70 R:85 A:85 D:70 | +| 2 | Confident | Repair script walks the history of HEAD (the checked-out branch — `main` on the real repo) rather than hardcoding the ref `main` | Intake says "full history of `main`" describing the real repo; HEAD is identical there, matches the working-tree/HEAD comparison baseline, and stays robust to fixture/clone branch names | S:65 R:85 A:80 D:65 | +| 3 | Confident | Repair-script tests live at `src/node/sync/__tests__/repair-metrics.test.ts` | `npm test` only discovers `src/node/**/__tests__/*.test.ts`, so a `scripts/__tests__/` location would never run; the metrics repo is sync-domain | S:70 R:90 A:85 D:75 | +| 4 | Confident | A day-file that parses as JSON but lacks a finite numeric `totalCost` counts as "unparseable as a UsageEntry" → write proceeds | One obvious reading of the intake's absent/empty/unparseable posture; keeps the guard from being wedged open by junk files | S:70 R:85 A:85 D:75 | +| 5 | Confident | Repair script exits 0 after both dry-run and `--write` regardless of findings; non-zero only for operational errors | Report-style ops tool; intake specifies no exit-code contract beyond fail-loud errors | S:60 R:90 A:80 D:70 | +| 6 | Certain | `fetchToolMergedWithMachines` applies the max-merge only inside its existing `config.mode === "multi"` branch | The repo read/write block is already gated on multi mode; single mode has no repo dir (intake: unaffected paths) | S:85 R:90 A:95 D:90 | +| 7 | Certain | On equal `totalCost`, `maxMergeEntries` keeps the first argument's (live local) entry | Intake: "For dates within the live window, live wins (equal or greater — it includes in-flight today data)" | S:90 R:90 A:95 D:90 | + +7 assumptions (2 certain, 5 confident, 0 tentative). diff --git a/scripts/repair-metrics.mjs b/scripts/repair-metrics.mjs new file mode 100644 index 0000000..f07115c --- /dev/null +++ b/scripts/repair-metrics.mjs @@ -0,0 +1,245 @@ +// scripts/repair-metrics.mjs — one-time repair: restore shrunk metrics +// day-files to their historical maximum from the metrics repo's git history. +// +// Why: Claude Code purges session transcripts older than ~30 days, so a +// machine's live ccusage view of an old day collapses toward zero. Until the +// never-shrink guard in writeMetrics() (src/node/sync/sync.ts) shipped, every +// sync overwrote correct per-day JSONL snapshots in the shared metrics repo +// with that post-purge residue. Nothing was ever deleted from git history — +// only overwritten in newer commits — so every shrunk day-file can be +// restored losslessly by picking the commit where its totalCost was highest. +// +// Standalone ops script (precedent: scripts/help-dump.mjs): NOT bundled into +// dist/tu.mjs and not imported by anything under src/ (Constitution III +// untouched). Runs under plain `node` with `node:`-prefixed built-ins only. +// +// IMPORTANT sequencing: run this only AFTER the guarded binary is installed +// on every actively-syncing machine — otherwise the next sync from an old +// binary re-clobbers restored values at the rolling retention edge. +// +// Usage: +// node scripts/repair-metrics.mjs [--repo ] [--write] +// +// --repo Metrics repo to scan (default: ~/.tu/metrics_repo) +// --write Restore shrunk files in the working tree. Default is a +// dry-run report. Committing/pushing is left to the user +// for review. Idempotent — re-running reports nothing left. + +import { readFileSync, writeFileSync, existsSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { join, resolve } from "node:path"; +import { homedir } from "node:os"; + +// A file is "shrunk" only when HEAD is below its historical max by more than +// a cent — float noise within a cent is not worth touching. +const CENT_TOLERANCE = 0.01; + +// Day-file names follow {toolKey}-{YYYY-MM-DD}.jsonl (see writeMetrics). +const DAY_FILE_RE = /-\d{4}-\d{2}-\d{2}\.jsonl$/; + +// git log over a long history can be large; well above any realistic size. +const MAX_GIT_BUFFER = 64 * 1024 * 1024; + +const USAGE = "Usage: node scripts/repair-metrics.mjs [--repo ] [--write]"; + +function fail(msg) { + process.stderr.write(`repair-metrics: ${msg}\n`); + process.exit(1); +} + +function resolveHome(p) { + if (p === "~") return homedir(); + if (p.startsWith("~/")) return resolve(homedir(), p.slice(2)); + return resolve(p); +} + +function parseArgs(argv) { + let repo = resolveHome("~/.tu/metrics_repo"); + let write = false; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === "--repo") { + const value = argv[++i]; + if (!value) fail(`--repo requires a path\n${USAGE}`); + repo = resolveHome(value); + } else if (arg === "--write") { + write = true; + } else { + fail(`unknown argument: ${arg}\n${USAGE}`); + } + } + return { repo, write }; +} + +/** Run git against the repo; returns stdout or null on failure. */ +function git(repo, args) { + // core.quotePath=false → raw UTF-8 paths (no C-style quoting to unescape). + const result = spawnSync("git", ["-C", repo, "-c", "core.quotePath=false", ...args], { + encoding: "utf-8", + maxBuffer: MAX_GIT_BUFFER, + }); + if (result.error || result.status !== 0) return null; + return result.stdout; +} + +/** Parse the first line of a day-file blob; returns a finite totalCost or null. */ +function parseCost(content) { + if (typeof content !== "string" || !content.trim()) return null; + try { + const parsed = JSON.parse(content.trim().split("\n")[0]); + const cost = Number(parsed?.totalCost); + return Number.isFinite(cost) ? cost : null; + } catch { + return null; + } +} + +/** Tracked day-files at HEAD (paths relative to the repo root). */ +function listTrackedDayFiles(repo) { + const out = git(repo, ["ls-files", "-z", "--", "*.jsonl"]); + if (out === null) fail("git ls-files failed — is this a metrics repo checkout?"); + return out.split("\0").filter((p) => p && DAY_FILE_RE.test(p)); +} + +/** + * One history walk (avoids a `git log` per file): every commit on the + * checked-out branch that touches a *.jsonl file, with the paths it touched. + * Returns { commitCount, fileCommits: Map> } — + * commit lists are newest-first (git log order). + */ +function buildFileCommitMap(repo) { + const out = git(repo, ["log", "--format=%H%x09%cs", "--name-only", "--", "*.jsonl"]); + if (out === null) fail("git log failed — is this a metrics repo checkout?"); + const fileCommits = new Map(); + let current = null; + let commitCount = 0; + for (const line of out.split("\n")) { + const commitMatch = line.match(/^([0-9a-f]{40})\t(\d{4}-\d{2}-\d{2})$/); + if (commitMatch) { + current = { sha: commitMatch[1], date: commitMatch[2] }; + commitCount++; + continue; + } + if (!line || !current || !DAY_FILE_RE.test(line)) continue; + if (!fileCommits.has(line)) fileCommits.set(line, []); + fileCommits.get(line).push(current); + } + return { commitCount, fileCommits }; +} + +/** + * Historical maximum for one file: highest parseable totalCost across every + * commit that touched it. Deleted-at-commit paths and unparseable blobs are + * skipped. Returns { cost, sha, date, content } or null when no version parses. + */ +function findHistoricalMax(repo, path, commits) { + let max = null; + for (const { sha, date } of commits) { + const content = git(repo, ["show", `${sha}:${path}`]); + if (content === null) continue; // path deleted in this commit — skip + const cost = parseCost(content); + if (cost === null) continue; // unparseable historical version — skip + if (max === null || cost > max.cost) max = { cost, sha, date, content }; + } + return max; +} + +/** Working-tree totalCost; missing/unparseable counts as 0 (fully shrunk). */ +function currentCost(repo, path) { + try { + return parseCost(readFileSync(join(repo, path), "utf-8")) ?? 0; + } catch { + return 0; + } +} + +const money = (v) => `$${v.toFixed(2)}`; + +function printReport(repo, shrunk, dayFileCount, commitCount) { + const out = []; + out.push(`repair-metrics: scanned ${repo}`); + out.push(` ${dayFileCount} tracked day-files, ${commitCount} commits touching *.jsonl`); + out.push(""); + + if (shrunk.length === 0) { + out.push("Nothing to repair — every day-file is at its historical maximum."); + process.stdout.write(out.join("\n") + "\n"); + return; + } + + const pathWidth = Math.max(...shrunk.map((s) => s.path.length), "FILE".length); + out.push(`Shrunk day-files (${shrunk.length}):`); + out.push(""); + out.push( + ` ${"FILE".padEnd(pathWidth)} ${"CURRENT".padStart(10)} ${"MAX".padStart(10)} ${"DELTA".padStart(10)} MAX COMMIT`, + ); + for (const s of shrunk) { + out.push( + ` ${s.path.padEnd(pathWidth)} ${money(s.current).padStart(10)} ${money(s.max.cost).padStart(10)} ${("+" + money(s.delta)).padStart(10)} ${s.max.sha.slice(0, 7)} (${s.max.date})`, + ); + } + + // Per-user totals — the user is the first path segment. + const byUser = new Map(); + for (const s of shrunk) { + const user = s.path.split("/")[0]; + const agg = byUser.get(user) ?? { delta: 0, files: 0 }; + agg.delta += s.delta; + agg.files += 1; + byUser.set(user, agg); + } + out.push(""); + out.push("Per-user totals:"); + for (const [user, agg] of [...byUser.entries()].sort()) { + out.push(` ${user}: +${money(agg.delta)} across ${agg.files} file(s)`); + } + + const grand = shrunk.reduce((sum, s) => sum + s.delta, 0); + out.push(""); + out.push(`Grand total: +${money(grand)} across ${shrunk.length} file(s)`); + process.stdout.write(out.join("\n") + "\n"); +} + +function main() { + const { repo, write } = parseArgs(process.argv.slice(2)); + + if (!existsSync(repo)) fail(`repo not found: ${repo}`); + if (git(repo, ["rev-parse", "--is-inside-work-tree"]) === null) { + fail(`not a git repository: ${repo}`); + } + + const dayFiles = listTrackedDayFiles(repo); + const { commitCount, fileCommits } = buildFileCommitMap(repo); + + const shrunk = []; + for (const path of dayFiles) { + const max = findHistoricalMax(repo, path, fileCommits.get(path) ?? []); + if (max === null) continue; // no parseable history — nothing to compare + const current = currentCost(repo, path); + const delta = max.cost - current; + if (delta > CENT_TOLERANCE) shrunk.push({ path, current, max, delta }); + } + shrunk.sort((a, b) => a.path.localeCompare(b.path)); + + printReport(repo, shrunk, dayFiles.length, commitCount); + if (shrunk.length === 0) return; + + if (!write) { + process.stdout.write("\nDry run — nothing modified. Re-run with --write to restore shrunk files.\n"); + return; + } + + // Restore the full original content (the exact historical-max blob), not + // just the cost field — each day-file stays an atomic snapshot that was + // real at some point in time. Working tree only: review, commit, and push + // are deliberately left to the user. + for (const s of shrunk) { + writeFileSync(join(repo, s.path), s.max.content); + } + process.stdout.write( + `\nRestored ${shrunk.length} file(s) in the working tree.\n` + + `Review with: git -C ${repo} diff\nThen commit and push manually.\n`, + ); +} + +main(); diff --git a/src/node/core/__tests__/fetcher.test.ts b/src/node/core/__tests__/fetcher.test.ts index 097d098..2181027 100644 --- a/src/node/core/__tests__/fetcher.test.ts +++ b/src/node/core/__tests__/fetcher.test.ts @@ -10,6 +10,7 @@ import { currentLabel, pickCurrentEntry, mergeEntries, + maxMergeEntries, aggregateMonthly, TOOLS, EMPTY, @@ -499,6 +500,79 @@ describe("mergeEntries", () => { }); }); +// --------------------------------------------------------------------------- +// maxMergeEntries — per-label whole-entry max (own-machine self-view merge) +// --------------------------------------------------------------------------- + +describe("maxMergeEntries", () => { + it("picks the whole entry with the greater totalCost (no summing)", () => { + // Purged live view vs own repo snapshot — snapshot must resurface intact + const live = [mkEntry("2026-04-24", 9.46, 12)]; + const snapshot = [mkEntry("2026-04-24", 236.0, 9999)]; + const result = maxMergeEntries(live, snapshot); + assert.equal(result.length, 1); + assert.equal(result[0].totalCost, 236.0); + assert.equal(result[0].inputTokens, 9999); // every field from the winner + assert.equal(result[0].totalTokens, 9999); + }); + + it("never mixes fields across entries (atomic snapshots, not per-field max)", () => { + const a = [{ ...mkEntry("2026-04-24", 10.0, 100), outputTokens: 1 }]; + const b = [{ ...mkEntry("2026-04-24", 5.0, 50), outputTokens: 9999 }]; + const result = maxMergeEntries(a, b); + // a wins on totalCost — b's larger outputTokens must NOT leak in + assert.equal(result[0].totalCost, 10.0); + assert.equal(result[0].outputTokens, 1); + }); + + it("keeps the first argument's entry on equal totalCost (live wins in the live window)", () => { + const live = [{ ...mkEntry("2026-06-10", 3.0, 100), outputTokens: 42 }]; + const snapshot = [{ ...mkEntry("2026-06-10", 3.0, 100), outputTokens: 7 }]; + const result = maxMergeEntries(live, snapshot); + assert.equal(result.length, 1); + assert.equal(result[0].outputTokens, 42); + }); + + it("preserves non-overlapping entries from both sides", () => { + const live = [mkEntry("2026-06-10", 1.0)]; + const snapshot = [mkEntry("2026-04-24", 236.0)]; + const result = maxMergeEntries(live, snapshot); + assert.equal(result.length, 2); + assert.equal(result[0].label, "2026-04-24"); + assert.equal(result[1].label, "2026-06-10"); + }); + + it("handles empty inputs", () => { + const only = [mkEntry("2026-02-20", 1.5)]; + assert.deepEqual(maxMergeEntries([], []), []); + assert.equal(maxMergeEntries(only, [])[0].totalCost, 1.5); + assert.equal(maxMergeEntries([], only)[0].totalCost, 1.5); + }); + + it("sorts result ascending by label", () => { + const a = [mkEntry("2026-02-20", 1.0)]; + const b = [mkEntry("2026-02-18", 0.5), mkEntry("2026-02-22", 0.3)]; + const result = maxMergeEntries(a, b); + assert.deepEqual( + result.map((e) => e.label), + ["2026-02-18", "2026-02-20", "2026-02-22"] + ); + }); + + it("does not mutate input arrays and returns copies", () => { + const a = [mkEntry("2026-02-20", 1.0)]; + const b = [mkEntry("2026-02-20", 2.0)]; + const aCopy = JSON.parse(JSON.stringify(a)); + const bCopy = JSON.parse(JSON.stringify(b)); + const result = maxMergeEntries(a, b); + assert.deepEqual(a, aCopy); + assert.deepEqual(b, bCopy); + // Winner is copied, not aliased — mutating the result must not touch inputs + result[0].totalCost = 999; + assert.equal(b[0].totalCost, 2.0); + }); +}); + // --------------------------------------------------------------------------- // Unified monthly path: aggregateMonthly + currentLabel (replaces fetchTotals monthly) // --------------------------------------------------------------------------- diff --git a/src/node/core/__tests__/self-view-merge.test.ts b/src/node/core/__tests__/self-view-merge.test.ts new file mode 100644 index 0000000..a5b7070 --- /dev/null +++ b/src/node/core/__tests__/self-view-merge.test.ts @@ -0,0 +1,177 @@ +import { describe, it, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { mkdirSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { writeMetrics, readRemoteEntriesByMachine } from "../../sync/sync.js"; +import { mergeEntries, maxMergeEntries } from "../fetcher.js"; +import type { UsageEntry } from "../types.js"; + +// Composition test for the self-view max-merge (fetchToolMerged / +// fetchToolMergedWithMachines in src/node/core/cli.ts). Those functions are +// not exported and shell out to ccusage for the live fetch, so — following +// the cli-user-flag.test.ts precedent — the rewired pipeline is mirrored +// here with the live `local` entries injected, exercising the real fs-backed +// pieces end-to-end against a temp metrics dir: +// +// guarded writeMetrics → readRemoteEntriesByMachine(excludeMachine = null) +// → own/others split → maxMergeEntries(local, own) → mergeEntries(..., others) +// +// If the production pipeline shape changes, update this mirror to match. + +const TEST_DIR = join(tmpdir(), "tu-self-view-test-" + process.pid); + +const USER = "sahil"; +const MACHINE = "macbook"; +const TOOL = "cc"; + +const entry = (label: string, cost: number, tokens = 150): UsageEntry => ({ + label, + totalCost: cost, + inputTokens: 100, + outputTokens: 50, + cacheCreationTokens: 0, + cacheReadTokens: 0, + totalTokens: tokens, +}); + +function seedSnapshot(machine: string, e: UsageEntry): void { + const dir = join(TEST_DIR, USER, e.label.slice(0, 4), machine); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, `${TOOL}-${e.label}.jsonl`), JSON.stringify(e) + "\n"); +} + +interface PipelineResult { + merged: UsageEntry[]; + machineMap: Map; +} + +// Mirrors the multi-mode own-user path of fetchToolMerged; machineMap mirrors +// what fetchToolMergedWithMachines exposes for --by-machine. +function mergedPipeline(local: UsageEntry[]): PipelineResult { + writeMetrics(TEST_DIR, USER, MACHINE, TOOL, local); + const byMachine = readRemoteEntriesByMachine(TEST_DIR, USER, null, TOOL); + const ownSnapshots = byMachine.get(MACHINE) ?? []; + const remote: UsageEntry[] = []; + const machineMap = new Map(); + machineMap.set(MACHINE, maxMergeEntries(local, ownSnapshots)); + for (const [machine, machineEntries] of byMachine) { + if (machine !== MACHINE) { + remote.push(...machineEntries); + machineMap.set(machine, machineEntries); + } + } + const effectiveLocal = maxMergeEntries(local, ownSnapshots); + return { merged: mergeEntries(effectiveLocal, remote), machineMap }; +} + +const byLabel = (entries: UsageEntry[], label: string) => entries.find((e) => e.label === label); + +describe("self-view max-merge pipeline (purged local + own snapshots + remote)", () => { + beforeEach(() => mkdirSync(TEST_DIR, { recursive: true })); + afterEach(() => rmSync(TEST_DIR, { recursive: true, force: true })); + + it("resurfaces a purged date from the own snapshot without double-counting", () => { + // Intake's real numbers: own snapshot $236.00, remote machine $308.12, + // post-purge live residue $9.46. + seedSnapshot(MACHINE, entry("2026-04-24", 236.0, 9999)); + seedSnapshot("devws", entry("2026-04-24", 308.12)); + const local = [entry("2026-04-24", 9.46, 12)]; + + const { merged } = mergedPipeline(local); + const day = byLabel(merged, "2026-04-24"); + assert.ok(day); + // 236.00 + 308.12 — NOT 9.46 + 308.12 (blind spot) and NOT + // 9.46 + 236.00 + 308.12 (double count of surviving transcripts) + assert.equal(day.totalCost.toFixed(2), "544.12"); + }); + + it("keeps the live view dominant inside the live window (today keeps growing)", () => { + seedSnapshot(MACHINE, entry("2026-06-10", 3.0, 100)); + const local = [entry("2026-06-10", 5.0, 500)]; + + const { merged } = mergedPipeline(local); + const today = byLabel(merged, "2026-06-10"); + assert.ok(today); + assert.equal(today.totalCost, 5.0); + assert.equal(today.totalTokens, 500); // whole live entry, not the stale snapshot + + // The grow-write went through: the repo snapshot now holds the live value + const file = join(TEST_DIR, USER, "2026", MACHINE, "cc-2026-06-10.jsonl"); + assert.equal(JSON.parse(readFileSync(file, "utf-8").trim()).totalCost, 5.0); + }); + + it("write guard keeps the own snapshot intact while the merge resurfaces it", () => { + seedSnapshot(MACHINE, entry("2026-04-24", 236.0)); + const local = [entry("2026-04-24", 9.46)]; + + mergedPipeline(local); + + // The residual live entry must NOT have overwritten the snapshot file + const file = join(TEST_DIR, USER, "2026", MACHINE, "cc-2026-04-24.jsonl"); + assert.equal(JSON.parse(readFileSync(file, "utf-8").trim()).totalCost, 236.0); + }); + + it("carries the whole snapshot entry (token fields included), not a chimera", () => { + seedSnapshot(MACHINE, { ...entry("2026-04-24", 236.0), totalTokens: 8888, outputTokens: 777 }); + const local = [{ ...entry("2026-04-24", 9.46), totalTokens: 12, outputTokens: 3 }]; + + const { merged } = mergedPipeline(local); + const day = byLabel(merged, "2026-04-24"); + assert.ok(day); + assert.equal(day.totalTokens, 8888); + assert.equal(day.outputTokens, 777); + }); + + it("shows the corrected own-machine column for --by-machine", () => { + seedSnapshot(MACHINE, entry("2026-04-24", 236.0)); + seedSnapshot("devws", entry("2026-04-24", 308.12)); + const local = [entry("2026-04-24", 9.46), entry("2026-06-10", 5.0)]; + + const { machineMap } = mergedPipeline(local); + const own = machineMap.get(MACHINE); + const devws = machineMap.get("devws"); + assert.ok(own && devws); + assert.equal(byLabel(own, "2026-04-24")?.totalCost, 236.0); // corrected, not 9.46 + assert.equal(byLabel(own, "2026-06-10")?.totalCost, 5.0); // live window untouched + assert.equal(byLabel(devws, "2026-04-24")?.totalCost, 308.12); + }); + + it("behaves as before for a fresh machine with no own snapshots", () => { + seedSnapshot("devws", entry("2026-06-09", 2.0)); + const local = [entry("2026-06-10", 1.0)]; + + const { merged, machineMap } = mergedPipeline(local); + assert.equal(byLabel(merged, "2026-06-10")?.totalCost, 1.0); + assert.equal(byLabel(merged, "2026-06-09")?.totalCost, 2.0); + // effectiveLocal degenerates to local... plus the file writeMetrics just + // wrote for today, which is the same entry — still exactly local. + assert.deepEqual(machineMap.get(MACHINE), local); + }); + + it("merged totals never decrease relative to the pre-change pipeline", () => { + // Pre-change: mergeEntries(local, remote-excluding-own). The new pipeline + // adds max(local, own) — which is >= local per label — so every merged + // label total is >= the old one. + seedSnapshot(MACHINE, entry("2026-04-24", 236.0)); + seedSnapshot(MACHINE, entry("2026-05-03", 40.0)); + seedSnapshot("devws", entry("2026-04-24", 308.12)); + const local = [entry("2026-04-24", 9.46), entry("2026-05-03", 41.0), entry("2026-06-10", 5.0)]; + + const oldRemote = readRemoteEntriesByMachine(TEST_DIR, USER, MACHINE, TOOL); + const oldFlat: UsageEntry[] = []; + for (const machineEntries of oldRemote.values()) oldFlat.push(...machineEntries); + const oldMerged = mergeEntries(local, oldFlat); + + const { merged } = mergedPipeline(local); + for (const oldEntry of oldMerged) { + const newEntry = byLabel(merged, oldEntry.label); + assert.ok(newEntry, `label ${oldEntry.label} missing from new merge`); + assert.ok( + newEntry.totalCost >= oldEntry.totalCost, + `${oldEntry.label}: ${newEntry.totalCost} < ${oldEntry.totalCost}`, + ); + } + }); +}); diff --git a/src/node/core/cli.ts b/src/node/core/cli.ts index 2af3324..34b5e53 100644 --- a/src/node/core/cli.ts +++ b/src/node/core/cli.ts @@ -1,4 +1,4 @@ -import { TOOLS, EMPTY, fetchHistory, fetchAllTotals, fetchAllHistory, aggregateMonthly, mergeEntries, currentLabel } from "./fetcher.js"; +import { TOOLS, EMPTY, fetchHistory, fetchAllTotals, fetchAllHistory, aggregateMonthly, mergeEntries, maxMergeEntries, currentLabel } from "./fetcher.js"; import { printHistory, printTotal, printTotalHistory, renderHistory, renderTotal, renderTotalHistory, emitCsv, emitMarkdown } from "../tui/formatter.js"; import type { FormatOptions } from "../tui/formatter.js"; import { readConfig, CONFIG_PATH, TU_HOME, THREE_HOURS_MS, resolveHome, DEFAULT_CONFIG_PATH } from "./config.js"; @@ -466,9 +466,20 @@ async function fetchToolMerged( _mark(`fetchToolMerged(${toolKey}) → fetchHistory done (${local.length} entries)`); writeMetrics(config.metricsDir, config.user, config.machine, toolKey, local); _mark(`fetchToolMerged(${toolKey}) → writeMetrics done`); - const remote = readRemoteEntries(config.metricsDir, config.user, config.machine, toolKey); + // Read ALL machines (excludeMachine = null) in one walk, then split out this + // machine's own snapshots: once Claude Code purges old transcripts, the live + // fetch under-reports old days, so the machine's own synced history must be + // merged back via per-day whole-entry max (sum would double-count the + // surviving transcripts of partially-purged days). + const byMachine = readRemoteEntriesByMachine(config.metricsDir, config.user, null, toolKey); + const ownSnapshots = byMachine.get(config.machine) ?? []; + const remote: UsageEntry[] = []; + for (const [machine, machineEntries] of byMachine) { + if (machine !== config.machine) remote.push(...machineEntries); + } _mark(`fetchToolMerged(${toolKey}) → readRemote done (${remote.length} entries)`); - const merged = mergeEntries(local, remote); + const effectiveLocal = maxMergeEntries(local, ownSnapshots); + const merged = mergeEntries(effectiveLocal, remote); if (period === "monthly") return aggregateMonthly(merged); return merged; } @@ -510,8 +521,14 @@ async function fetchToolMergedWithMachines( if (config.mode === "multi") { writeMetrics(config.metricsDir, config.user, config.machine, toolKey, local); - const remoteMachines = readRemoteEntriesByMachine(config.metricsDir, config.user, config.machine, toolKey); - for (const [machine, mEntries] of remoteMachines) machineMap.set(machine, mEntries); + // Same self-view correction as fetchToolMerged: read all machines in one + // walk and max-merge this machine's own snapshots into its live view, so + // the own-machine column resurfaces purge-collapsed days. + const allMachines = readRemoteEntriesByMachine(config.metricsDir, config.user, null, toolKey); + machineMap.set(config.machine, maxMergeEntries(local, allMachines.get(config.machine) ?? [])); + for (const [machine, mEntries] of allMachines) { + if (machine !== config.machine) machineMap.set(machine, mEntries); + } } const allEntries: UsageEntry[] = []; diff --git a/src/node/core/fetcher.ts b/src/node/core/fetcher.ts index 70d3117..f83f0e5 100644 --- a/src/node/core/fetcher.ts +++ b/src/node/core/fetcher.ts @@ -280,6 +280,28 @@ export function mergeEntries( return [...map.values()].sort((a, b) => a.label.localeCompare(b.label)); } +// --- Max-merge: per-label whole-entry high-water mark (own-machine self-view) --- +// +// Picks, per date label, whichever whole entry has the greater totalCost — +// never mixing fields across entries and never summing; on ties the entry +// from `a` wins. Used to merge a machine's live fetch (`a`) with its own +// synced repo snapshots (`b`): once Claude Code purges old transcripts, the +// live view of an old day collapses toward zero while the snapshot still +// holds the full value — and summing them would double-count the surviving +// transcripts of partially-purged days. +export function maxMergeEntries(a: UsageEntry[], b: UsageEntry[]): UsageEntry[] { + const map = new Map(); + + for (const e of [...a, ...b]) { + const existing = map.get(e.label); + if (!existing || e.totalCost > existing.totalCost) { + map.set(e.label, { ...e }); + } + } + + return [...map.values()].sort((x, y) => x.label.localeCompare(y.label)); +} + export async function fetchAllHistory( period: string, extraArgs: string[] = [], diff --git a/src/node/sync/__tests__/repair-metrics.test.ts b/src/node/sync/__tests__/repair-metrics.test.ts new file mode 100644 index 0000000..7671c06 --- /dev/null +++ b/src/node/sync/__tests__/repair-metrics.test.ts @@ -0,0 +1,197 @@ +import { describe, it, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { mkdirSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { tmpdir } from "node:os"; +import { execSync, spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +// scripts/repair-metrics.mjs is a standalone ops script (not bundled, not +// imported by src/), so it is driven end-to-end as a child process against a +// seeded local git fixture — hermetic: explicit --repo, no HOME/TU_* reads, +// and never the real ~/.tu/metrics_repo. Fixture pattern follows +// sync.test.ts / cli-sync.test.ts. +const SCRIPT = fileURLToPath(new URL("../../../../scripts/repair-metrics.mjs", import.meta.url)); + +const TEST_DIR = join(tmpdir(), "tu-repair-test-" + process.pid); +const REPO = join(TEST_DIR, "metrics"); + +const opts = { stdio: "pipe" as const }; + +function initRepo(): void { + mkdirSync(REPO, { recursive: true }); + execSync(`git init "${REPO}"`, opts); + execSync(`git -C "${REPO}" config user.email "test@test.com"`, opts); + execSync(`git -C "${REPO}" config user.name "Test"`, opts); + // The developer's global config could enable signing and stall on a missing + // key — pin it off for the fixture. + execSync(`git -C "${REPO}" config commit.gpgsign false`, opts); +} + +function writeDay(relPath: string, content: string): void { + const full = join(REPO, relPath); + mkdirSync(dirname(full), { recursive: true }); + writeFileSync(full, content); +} + +function commitAll(msg: string): string { + execSync(`git -C "${REPO}" add -A`, opts); + execSync(`git -C "${REPO}" commit -m "${msg}"`, opts); + return execSync(`git -C "${REPO}" rev-parse HEAD`, { encoding: "utf-8" }).trim(); +} + +function commitCount(): number { + return Number(execSync(`git -C "${REPO}" rev-list --count HEAD`, { encoding: "utf-8" }).trim()); +} + +const entryLine = (label: string, cost: number, tokens = 150): string => + JSON.stringify({ + label, + totalCost: cost, + inputTokens: 100, + outputTokens: 50, + cacheCreationTokens: 0, + cacheReadTokens: 0, + totalTokens: tokens, + }) + "\n"; + +function runScript(...args: string[]): { status: number | null; stdout: string; stderr: string } { + const result = spawnSync(process.execPath, [SCRIPT, ...args], { encoding: "utf-8" }); + return { status: result.status, stdout: result.stdout ?? "", stderr: result.stderr ?? "" }; +} + +describe("repair-metrics script", () => { + beforeEach(() => { + rmSync(TEST_DIR, { recursive: true, force: true }); + initRepo(); + }); + afterEach(() => rmSync(TEST_DIR, { recursive: true, force: true })); + + const SHRUNK = "sahil/2026/devws/cc-2026-04-24.jsonl"; + const HIGH = entryLine("2026-04-24", 308.12, 9999); + const LOW = entryLine("2026-04-24", 9.46, 12); + + function seedShrunkRepo(): void { + writeDay(SHRUNK, HIGH); + writeDay("sahil/2026/devws/cc-2026-05-01.jsonl", entryLine("2026-05-01", 10.0)); + writeDay("bob/2026/laptop/cc-2026-04-24.jsonl", entryLine("2026-04-24", 50.0)); + commitAll("initial high-water marks"); + writeDay(SHRUNK, LOW); + writeDay("bob/2026/laptop/cc-2026-04-24.jsonl", entryLine("2026-04-24", 2.0)); + commitAll("post-purge shrink"); + } + + it("dry run reports shrunk files with current, max, and delta — and modifies nothing", () => { + seedShrunkRepo(); + const result = runScript("--repo", REPO); + assert.equal(result.status, 0, result.stderr); + assert.ok(result.stdout.includes(SHRUNK), `expected shrunk path in report:\n${result.stdout}`); + assert.ok(result.stdout.includes("$9.46"), "expected current value"); + assert.ok(result.stdout.includes("$308.12"), "expected historical max"); + assert.ok(result.stdout.includes("+$298.66"), "expected delta"); + assert.ok(result.stdout.includes("Dry run"), "expected dry-run notice"); + // Working tree untouched + assert.equal(readFileSync(join(REPO, SHRUNK), "utf-8"), LOW); + }); + + it("dry run omits files that never shrank", () => { + seedShrunkRepo(); + const result = runScript("--repo", REPO); + assert.equal(result.status, 0, result.stderr); + assert.ok(!result.stdout.includes("cc-2026-05-01.jsonl"), "never-shrunk file must not be listed"); + }); + + it("reports per-user subtotals and a grand total across users", () => { + seedShrunkRepo(); + const result = runScript("--repo", REPO); + assert.equal(result.status, 0, result.stderr); + assert.ok(result.stdout.includes("Per-user totals:"), "expected per-user section"); + assert.ok(/sahil: \+\$298\.66 across 1 file/.test(result.stdout), `expected sahil subtotal:\n${result.stdout}`); + assert.ok(/bob: \+\$48\.00 across 1 file/.test(result.stdout), `expected bob subtotal:\n${result.stdout}`); + // 298.66 + 48.00 + assert.ok(/Grand total: \+\$346\.66 across 2 file/.test(result.stdout), `expected grand total:\n${result.stdout}`); + }); + + it("does not flag files within a cent of their historical max", () => { + const path = "sahil/2026/devws/cc-2026-03-01.jsonl"; + writeDay(path, entryLine("2026-03-01", 1.005)); + commitAll("high"); + writeDay(path, entryLine("2026-03-01", 1.0)); + commitAll("within tolerance"); + const result = runScript("--repo", REPO); + assert.equal(result.status, 0, result.stderr); + assert.ok(!result.stdout.includes(path), "within-a-cent file must not be flagged"); + assert.ok(result.stdout.includes("Nothing to repair"), `expected nothing-to-repair:\n${result.stdout}`); + }); + + it("picks the maximum across 3+ versions (max in the middle of history)", () => { + const path = "sahil/2026/devws/cc-2026-02-10.jsonl"; + writeDay(path, entryLine("2026-02-10", 5.0)); + commitAll("v1"); + writeDay(path, entryLine("2026-02-10", 100.0, 7777)); + commitAll("v2 — the high-water mark"); + writeDay(path, entryLine("2026-02-10", 20.0)); + commitAll("v3"); + const result = runScript("--repo", REPO); + assert.equal(result.status, 0, result.stderr); + assert.ok(result.stdout.includes("$100.00"), `expected middle-commit max:\n${result.stdout}`); + assert.ok(result.stdout.includes("+$80.00"), "expected delta against v3"); + }); + + it("skips unparseable historical versions without crashing", () => { + const path = "sahil/2026/devws/cc-2026-02-11.jsonl"; + writeDay(path, "totally not json\n"); + commitAll("garbage version"); + writeDay(path, entryLine("2026-02-11", 50.0)); + commitAll("good version"); + writeDay(path, entryLine("2026-02-11", 5.0)); + commitAll("shrunk version"); + const result = runScript("--repo", REPO); + assert.equal(result.status, 0, result.stderr); + assert.ok(result.stdout.includes("$50.00"), `expected max from parseable versions:\n${result.stdout}`); + assert.ok(result.stdout.includes("+$45.00"), "expected delta vs current"); + }); + + it("--write restores the full historical-max content byte-exactly, without committing", () => { + seedShrunkRepo(); + const commitsBefore = commitCount(); + const result = runScript("--repo", REPO, "--write"); + assert.equal(result.status, 0, result.stderr); + assert.ok(/Restored 2 file/.test(result.stdout), `expected restore summary:\n${result.stdout}`); + // Full original content — token fields included, not just the cost + assert.equal(readFileSync(join(REPO, SHRUNK), "utf-8"), HIGH); + // Working tree only: no commit created, changes left for review + assert.equal(commitCount(), commitsBefore); + const status = execSync(`git -C "${REPO}" status --porcelain`, { encoding: "utf-8" }); + assert.ok(status.includes(SHRUNK), "expected restored file to show as modified"); + }); + + it("is idempotent — a re-run after --write reports nothing to repair", () => { + seedShrunkRepo(); + runScript("--repo", REPO, "--write"); + const second = runScript("--repo", REPO, "--write"); + assert.equal(second.status, 0, second.stderr); + assert.ok(second.stdout.includes("Nothing to repair"), `expected idempotent re-run:\n${second.stdout}`); + assert.equal(readFileSync(join(REPO, SHRUNK), "utf-8"), HIGH); + }); + + it("exits 1 with a stderr message when --repo is not a git repository", () => { + const plain = join(TEST_DIR, "plain"); + mkdirSync(plain, { recursive: true }); + const result = runScript("--repo", plain); + assert.equal(result.status, 1); + assert.ok(result.stderr.includes("not a git repository"), `expected error, got: ${result.stderr}`); + }); + + it("exits 1 with a stderr message when --repo does not exist", () => { + const result = runScript("--repo", join(TEST_DIR, "nonexistent")); + assert.equal(result.status, 1); + assert.ok(result.stderr.includes("repo not found"), `expected error, got: ${result.stderr}`); + }); + + it("exits 1 on unknown arguments", () => { + const result = runScript("--frobnicate"); + assert.equal(result.status, 1); + assert.ok(result.stderr.includes("unknown argument"), `expected usage error, got: ${result.stderr}`); + }); +}); diff --git a/src/node/sync/__tests__/sync.test.ts b/src/node/sync/__tests__/sync.test.ts index 71d5804..1dd3f52 100644 --- a/src/node/sync/__tests__/sync.test.ts +++ b/src/node/sync/__tests__/sync.test.ts @@ -55,6 +55,91 @@ describe("writeMetrics", () => { const parsed = JSON.parse(readFileSync(filePath, "utf-8").trim()); assert.equal(parsed.totalCost, 2.0); }); + + // --- Never-shrink guard: day-files are high-water marks. A live fetch after + // the transcript retention purge collapses toward zero; writeMetrics must + // never let that residue overwrite correct historical snapshots. --- + + it("skips the write when incoming totalCost is lower than existing (shrink)", () => { + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-04-24", 308.12)]); + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-04-24", 9.46)]); + const filePath = join(TEST_DIR, "sahil", "2026", "macbook", "cc-2026-04-24.jsonl"); + const parsed = JSON.parse(readFileSync(filePath, "utf-8").trim()); + assert.equal(parsed.totalCost, 308.12); + }); + + it("skips shrinking writes silently (no stderr warning)", () => { + const errors: string[] = []; + const origError = console.error; + const origWrite = process.stderr.write; + console.error = (...args: unknown[]) => errors.push(String(args[0])); + process.stderr.write = ((chunk: string) => { errors.push(String(chunk)); return true; }) as typeof process.stderr.write; + try { + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-04-24", 100.0)]); + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-04-24", 1.0)]); + } finally { + console.error = origError; + process.stderr.write = origWrite; + } + assert.deepEqual(errors, []); + }); + + it("writes when incoming totalCost equals existing (idempotent refresh)", () => { + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-02-20", 1.5)]); + const filePath = join(TEST_DIR, "sahil", "2026", "macbook", "cc-2026-02-20.jsonl"); + // Equal cost but different token counts — the newer snapshot must win + const updated = { ...entry("2026-02-20", 1.5), totalTokens: 999 }; + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [updated]); + const parsed = JSON.parse(readFileSync(filePath, "utf-8").trim()); + assert.equal(parsed.totalCost, 1.5); + assert.equal(parsed.totalTokens, 999); + }); + + it("writes when existing file is empty (treated as absent)", () => { + const dir = join(TEST_DIR, "sahil", "2026", "macbook"); + mkdirSync(dir, { recursive: true }); + const filePath = join(dir, "cc-2026-02-20.jsonl"); + writeFileSync(filePath, ""); + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-02-20", 0.5)]); + const parsed = JSON.parse(readFileSync(filePath, "utf-8").trim()); + assert.equal(parsed.totalCost, 0.5); + }); + + it("writes when existing file is not valid JSON (treated as absent)", () => { + const dir = join(TEST_DIR, "sahil", "2026", "macbook"); + mkdirSync(dir, { recursive: true }); + const filePath = join(dir, "cc-2026-02-20.jsonl"); + writeFileSync(filePath, "not json at all\n"); + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-02-20", 0.5)]); + const parsed = JSON.parse(readFileSync(filePath, "utf-8").trim()); + assert.equal(parsed.totalCost, 0.5); + }); + + it("writes when existing JSON has no numeric totalCost (not a UsageEntry)", () => { + const dir = join(TEST_DIR, "sahil", "2026", "macbook"); + mkdirSync(dir, { recursive: true }); + const filePath = join(dir, "cc-2026-02-20.jsonl"); + writeFileSync(filePath, JSON.stringify({ label: "2026-02-20", totalCost: "junk" }) + "\n"); + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [entry("2026-02-20", 0.5)]); + const parsed = JSON.parse(readFileSync(filePath, "utf-8").trim()); + assert.equal(parsed.totalCost, 0.5); + }); + + it("guards per entry within one batch (skips shrunk, writes grown)", () => { + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [ + entry("2026-04-24", 308.12), + entry("2026-04-25", 1.0), + ]); + writeMetrics(TEST_DIR, "sahil", "macbook", "cc", [ + entry("2026-04-24", 9.46), // shrunk → skipped + entry("2026-04-25", 5.0), // grown → written + ]); + const dir = join(TEST_DIR, "sahil", "2026", "macbook"); + const day24 = JSON.parse(readFileSync(join(dir, "cc-2026-04-24.jsonl"), "utf-8").trim()); + const day25 = JSON.parse(readFileSync(join(dir, "cc-2026-04-25.jsonl"), "utf-8").trim()); + assert.equal(day24.totalCost, 308.12); + assert.equal(day25.totalCost, 5.0); + }); }); describe("readRemoteEntries", () => { diff --git a/src/node/sync/sync.ts b/src/node/sync/sync.ts index 000ceb1..15f98cb 100644 --- a/src/node/sync/sync.ts +++ b/src/node/sync/sync.ts @@ -19,6 +19,32 @@ function execFileAsync(file: string, args: string[]): Promise { }); } +// Never-shrink guard: day-file snapshots are high-water marks of complete +// data. Claude Code purges transcripts older than ~30 days, so a live fetch +// for an old date collapses toward zero — overwriting would silently destroy +// correct history. Skip the write (whole-entry, keeping the file an atomic +// snapshot) when the incoming entry's totalCost is lower than the existing +// one's. Absent/empty/unparseable files are treated as absent, matching the +// read path's skip-silently posture; equal values still write so today's +// file keeps refreshing as the day grows. +function isShrinkingWrite(filePath: string, incoming: UsageEntry): boolean { + let raw: string; + try { + raw = readFileSync(filePath, "utf-8").trim(); + } catch { + return false; // file absent or unreadable → write + } + if (!raw) return false; // empty file → treat as absent + try { + const existing = JSON.parse(raw) as Partial; + const existingCost = Number(existing?.totalCost); + if (!Number.isFinite(existingCost)) return false; // not a UsageEntry → treat as absent + return incoming.totalCost < existingCost; + } catch { + return false; // unparseable → treat as absent + } +} + export function writeMetrics( metricsDir: string, user: string, @@ -31,6 +57,7 @@ export function writeMetrics( const dir = join(metricsDir, user, yyyy, machine); mkdirSync(dir, { recursive: true }); const filePath = join(dir, `${toolKey}-${entry.label}.jsonl`); + if (isShrinkingWrite(filePath, entry)) continue; writeFileSync(filePath, JSON.stringify(entry) + "\n"); } } From ad830842f490b1bed3f683dc989109011a537043 Mon Sep 17 00:00:00 2001 From: Sahil Ahuja Date: Wed, 10 Jun 2026 21:19:26 +0530 Subject: [PATCH 2/2] Update ship status and record PR URL --- .../.history.jsonl | 1 + .../.status.yaml | 12 +++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl b/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl index d4e2993..f26612c 100644 --- a/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl +++ b/fab/changes/260610-srmi-fix-metrics-history-destruction/.history.jsonl @@ -12,3 +12,4 @@ {"cmd":"fab-continue","event":"command","ts":"2026-06-10T15:38:21Z"} {"action":"enter","driver":"fab-fff","event":"stage-transition","stage":"ship","ts":"2026-06-10T15:44:08Z"} {"cmd":"git-pr","event":"command","ts":"2026-06-10T15:45:52Z"} +{"action":"enter","driver":"git-pr","event":"stage-transition","stage":"review-pr","ts":"2026-06-10T15:49:11Z"} diff --git a/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml b/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml index f8f98bd..cebc93f 100644 --- a/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml +++ b/fab/changes/260610-srmi-fix-metrics-history-destruction/.status.yaml @@ -9,8 +9,8 @@ progress: apply: done review: done hydrate: done - ship: active - review-pr: pending + ship: done + review-pr: active plan: generated: true task_count: 10 @@ -33,8 +33,10 @@ stage_metrics: apply: {started_at: "2026-06-10T14:59:05Z", driver: fab-fff, iterations: 1, completed_at: "2026-06-10T15:19:20Z"} review: {started_at: "2026-06-10T15:19:20Z", driver: fab-continue, iterations: 1, completed_at: "2026-06-10T15:36:59Z"} hydrate: {started_at: "2026-06-10T15:36:59Z", driver: fab-fff, iterations: 1, completed_at: "2026-06-10T15:44:08Z"} - ship: {started_at: "2026-06-10T15:44:08Z", driver: fab-fff, iterations: 1} -prs: [] + ship: {started_at: "2026-06-10T15:44:08Z", driver: fab-fff, iterations: 1, completed_at: "2026-06-10T15:49:11Z"} + review-pr: {started_at: "2026-06-10T15:49:11Z", driver: git-pr, iterations: 1} +prs: + - https://github.com/sahil87/tu/pull/34 true_impact: added: 0 deleted: 0 @@ -46,4 +48,4 @@ true_impact: computed_at: "2026-06-10T15:44:08Z" computed_at_stage: hydrate # true_impact: lazily created on first apply-finish (no placeholder here). -last_updated: 2026-06-10T15:44:08Z +last_updated: 2026-06-10T15:49:11Z