From d25f7ca7ef2666fb8fb162f20551bc175fb29620 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:01:32 +0200 Subject: [PATCH 01/13] feat(telemetry)!: report envelope v2 keeps the figures apart from the run Envelope v1 mixed the figures with facts about the read that printed them (files read, shapes met, calls not stored, the oldest transcript, ledger lines skipped before settling), so two reports over the same data were not byte-identical. Version 2 keeps every figure at top level: period, axis, filters, rows, totals, unknown_records, and coverage.records and coverage.left_out. They are a function of the ledger, the declarations, the identity and the request alone. The run's facts move to `run`, which is always last. The CLI's ReportResult makes the same split, and the text report prints what it printed before. Readers updated: the 01-usage skill, the usage contract, the CLI memory, and the token end-to-end check, which now refuses a version other than 2. BREAKING CHANGE: `aidd telemetry report --json` prints `version: 2`. `coverage.files_read`, `unrecognised_shapes`, `not_stored`, `consent_log_damaged`, `oldest_transcript_at` and `skipped_ledger_lines` moved to `run`. The refusal is `{"version": 2, "refused": ...}`. There is no v1 shape. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- aidd_docs/product/usage-contract.md | 20 +- cli/aidd_docs/memory/telemetry.md | 3 +- .../application/report/report-envelope.ts | 69 ++-- .../report/report-usage-use-case.ts | 40 +- .../telemetry/telemetry-report-display.ts | 11 +- .../report/report-envelope.unit.test.ts | 40 +- ...eport-figures.property.integration.test.ts | 342 ++++++++++++++++++ .../report/report-usage-use-case.unit.test.ts | 21 +- ...report-window.property.integration.test.ts | 5 +- cli/tests/e2e/telemetry-journey.e2e.test.ts | 14 +- .../e2e/telemetry-report-figures.e2e.test.ts | 38 ++ .../e2e/telemetry-report-narrowed.e2e.test.ts | 2 +- cli/tests/e2e/telemetry-report-sandbox.ts | 3 +- cli/tests/e2e/telemetry-report.e2e.test.ts | 4 +- .../telemetry-wiring.integration.test.ts | 10 +- .../telemetry-report-display.unit.test.ts | 9 +- .../skills/01-usage/actions/answer.md | 6 +- .../skills/01-usage/references/envelope.md | 28 +- .../check-claude-tokens-end-to-end.test.js | 16 +- scripts/check-claude-tokens-end-to-end.cjs | 8 + 20 files changed, 567 insertions(+), 122 deletions(-) create mode 100644 cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts create mode 100644 cli/tests/e2e/telemetry-report-figures.e2e.test.ts diff --git a/aidd_docs/product/usage-contract.md b/aidd_docs/product/usage-contract.md index 92d4f8420..b9a2c05cc 100644 --- a/aidd_docs/product/usage-contract.md +++ b/aidd_docs/product/usage-contract.md @@ -49,7 +49,7 @@ The ledger stores `StoredUsage`: the record plus `repository_id`, resolved at in **Null is not zero.** - Unknown is `null`, never `0`. A `null` counter is left out of its sum and counted in that counter's `unknown_records`. - A usage line missing a counter is still a record, that counter `null`. -- A line that is no usage line (not JSON, no `message.usage`, no `message.id`, `sessionId` or `timestamp`) is never a record. It counts in the run's `coverage.unrecognised_shapes`, and, when it names a clone that consents, in `coverage.left_out.unrecognised` of every report (see [Left out](#left-out)). +- A line that is no usage line (not JSON, no `message.usage`, no `message.id`, `sessionId` or `timestamp`) is never a record. It counts in `run.unrecognised_shapes` of the report that read it, and, when it names a clone that consents, in `coverage.left_out.unrecognised` of every report (see [Left out](#left-out)). - Ingest and report say: "not recognised by this read: a call missing a counter is kept with that counter unknown; a line that is no usage line is never a call." (`UNRECOGNISED_WORDS`, `presentation/display/telemetry-display.ts`). What is kept of such lines is said apart: "Kept from every read: …". **Key and fold** (`domain/formats/claude-transcript-usage.ts`). @@ -66,7 +66,7 @@ The ledger stores `StoredUsage`: the record plus `repository_id`, resolved at in **Repository.** The call's working directory resolved at ingest: its git root, named by the hash of the `origin` address (`host/owner/repo`), else the root commit. The address is never kept. -**Not stored.** Storing follows [Consent](#consent). A call read and left out counts in the run's `coverage.not_stored` under one reason, and nothing of it is kept, except an `undated` call of a clone that consents (see [Left out](#left-out)): +**Not stored.** Storing follows [Consent](#consent). A call read and left out counts in `run.not_stored` of the report that read it, under one reason, and nothing of it is kept, except an `undated` call of a clone that consents (see [Left out](#left-out)): | Reason | When | | --- | --- | @@ -98,18 +98,19 @@ What an ingest measured and could not store is kept in `ledger/left-out.txt`, so ## Report envelope -`aidd telemetry report --json` prints one object, `version: 1`. A field may be added to version 1; never renamed, retyped or removed without a new `version`. Type: `ReportEnvelope`, `cli/src/contexts/telemetry/application/report/report-envelope.ts`. +`aidd telemetry report --json` prints one object, `version: 2`. A field may be added to version 2, a figure outside `run`, a fact of the read inside it; never renamed, retyped, moved or removed without a new `version`. Type: `ReportEnvelope`, `cli/src/contexts/telemetry/application/report/report-envelope.ts`. | Field | Meaning | | --- | --- | -| `version` | `1` | +| `version` | `2` | | `period` | `{from, to}`, UTC days, `null` for open | | `axis` | `total`, `person`, `session`, `model`, `agent`, `day`, `repository`, `task`, `ticket` | | `filters` | `task`, `ticket`, `person`, `model`, `agent`, `session`, `repository`, `absent`, `unattributed`, in that order: `null` when not given, else as resolved (see [Filters](#filters)) | | `rows` | one per axis value, of the calls that match | | `totals` | the calls of the period that match | | `unknown_records` | calls with at least one counter unknown | -| `coverage` | `files_read`, `records`, `unrecognised_shapes`, `not_stored`, `consent_log_damaged`, `oldest_transcript_at`, `skipped_ledger_lines`, `left_out`. `records` counts the calls of the period, matching or not; `left_out` (`undated`, `unrecognised`) what every ingest so far measured and could not store, see [Left out](#left-out) | +| `coverage` | `records`, `left_out`. `records` counts the calls of the period, matching or not; `left_out` (`undated`, `unrecognised`) what every ingest so far measured and could not store, see [Left out](#left-out) | +| `run` | last: `files_read`, `unrecognised_shapes`, `not_stored`, `consent_log_damaged`, `oldest_transcript_at`, `skipped_ledger_lines`, what the read that printed the report found | Row and `totals`: `records`, `input`, `output`, `cache_read`, `cache_write` (each `{tokens, unknown_records}`), `total`. A row also has: @@ -120,10 +121,11 @@ Row and `totals`: `records`, `input`, `output`, `cache_read`, `cache_write` (eac | `reason` | `no-binding` or `declared-none` for `unattributed`, else `null` | - Every axis partitions the same calls: each adds up to `totals`, counter by counter. -- `AIDD_TELEMETRY=0`: nothing read, object is `{"version": 1, "refused": "AIDD_TELEMETRY=0"}`. +- **The figures** are every field but `run`: a function of the ledger (its calls and `left-out.txt`), the declarations, the chosen identity, the period, the filters and the axis. Two reports over the same of those print them byte for byte the same, whatever ingests ran before and in whatever order, run again or not, in any time zone and locale. Envelope version 1 mixed the two; version 2 has no compatibility shape. +- `AIDD_TELEMETRY=0`: nothing read, object is `{"version": 2, "refused": "AIDD_TELEMETRY=0"}`. - The text report labels an `absent` person `anonymous`; the envelope keeps `kind: absent`, `key: null`. -- A report ingests first. `files_read`, `unrecognised_shapes` and `not_stored` describe that run; `records` the period; `left_out` every ingest so far. -- A report with `--days`, `--from` or `--to` reads only the days of its period and the day on each side, plus any monthly file one of them falls in, with every day of its month; with none it reads the whole ledger. The copies of one call have been observed less than a day apart, and under that bound both print the same figures. Outside it, only a report with no period folds every copy of a call. `skipped_ledger_lines` counts the lines of the partitions read. +- A report ingests first. `run` describes that read: what it opened, met and left out, whether the consent log let it store, how old the transcripts on disk are, and the lines that are not records in the partitions it loaded, counted before it settled them (a rewrite keeps each once, so the next report can count fewer). `coverage.records` describes the period; `coverage.left_out` every ingest so far. +- A report with `--days`, `--from` or `--to` reads only the days of its period and the day on each side, plus any monthly file one of them falls in, with every day of its month; with none it reads the whole ledger. The copies of one call have been observed less than a day apart, and under that bound both print the same figures. Outside it, only a report with no period folds every copy of a call. `run.skipped_ledger_lines` counts the lines of the partitions read. - No path, working directory or branch. `session` keys are full session ids, `repository` keys are hashes. ### Filters @@ -321,7 +323,7 @@ stateDiagram-v2 **`forget --yes`** unsets `aidd.telemetry` and branch task keys in every clone in `roots.json` or `consents.jsonl` that still exists. Names only gone clones that had consented. A clone that never ran `on` is recorded only if a call from it was read after some clone opted in, so a hand-set key there may stay. **Fail closed.** -- A terminated `consents.jsonl` line not exactly one of the two shapes: log untrusted. Storage refused for every clone; `on` and `off` refuse and change nothing; every `ingest` and `report` says so, `coverage.consent_log_damaged` in `--json`. Calls read meanwhile are lost (offsets move on). +- A terminated `consents.jsonl` line not exactly one of the two shapes: log untrusted. Storage refused for every clone; `on` and `off` refuse and change nothing; every `ingest` and `report` says so, `run.consent_log_damaged` in `--json`. Calls read meanwhile are lost (offsets move on). - An unterminated last line is not a line yet (cut write, or still writing): both readers ignore it, even if it parses. The next writer, CLI or hook, cuts it off before appending. A lost close is re-observed; a lost open leaves a key with no interval, measuring nothing. - Recovery: `aidd telemetry forget --yes`, then `aidd telemetry on` per clone. Safe: `forget` removes the log and offsets, no backfill, intervals open at `on`. Removing the line by hand is not recovery and is no longer suggested: a lost close would reopen a window turned off. diff --git a/cli/aidd_docs/memory/telemetry.md b/cli/aidd_docs/memory/telemetry.md index aad0899d0..0d5a377a0 100644 --- a/cli/aidd_docs/memory/telemetry.md +++ b/cli/aidd_docs/memory/telemetry.md @@ -26,11 +26,12 @@ flowchart LR Task["telemetry task"] -->|write| Bindings Ledger --> Report["report: attribute, keep what the filters name, then split by one axis"] Bindings --> Report - Report --> Out["text or the version 1 envelope"] + Report --> Out["text or the version 2 envelope"] ``` - A command that reports or ingests reads new lines first, under the ledger lock. `ingest` is the same step on its own, run quietly by the plugin at session start. - A transcript is read a block at a time (1 MiB), each complete line handed on as it is found, never the unread tail whole. An ingest holds a block, up to about three copies of the longest line at once (its pieces, the joined bytes, the decoded string), and what the reading keeps: every record and every unrecognised line naming a directory. An unterminated last line waits for the next read. A transcript that shrank while it was read counts none of that read and keeps its position, to be read again. One replaced at its path meanwhile is read as it was opened, whole. +- A report's envelope keeps its figures apart from its `run`. The figures are every field but `run`, a function of the ledger, the declarations, the identity and what was asked, alone: byte for byte the same whatever ran before, in any time zone and locale. `run` is what the read that printed it found. - Reading is one pass: tool-specific only up to the usage record. Resolving, storing, attributing and reporting are written once for every tool. - The hooks and the CLI never call each other's code. They meet in the files, whose formats one fixture pins on both sides. diff --git a/cli/src/contexts/telemetry/application/report/report-envelope.ts b/cli/src/contexts/telemetry/application/report/report-envelope.ts index 587265d22..591851178 100644 --- a/cli/src/contexts/telemetry/application/report/report-envelope.ts +++ b/cli/src/contexts/telemetry/application/report/report-envelope.ts @@ -39,34 +39,40 @@ export interface EnvelopeRow extends EnvelopeTally { readonly reason: UnattributedReason | null; } -/** What `aidd telemetry report --json` prints. Version 1. +/** What `aidd telemetry report --json` prints. Version 2. + * + * Every field but `run` is a figure: a function of the ledger (its calls and what was left out + * of it), the declarations, the chosen identity, the period, the filters and the axis, alone. + * Two reports over the same of those print them byte for byte the same, whatever ingest ran + * before, in any time zone and locale. `run`, last, is what the read that printed the envelope + * found, and may differ between two such reports. * * `rows` split the period along `axis`; the rows of every axis add up to the same `totals`, * counter by counter. `unknown_records` is how many calls had at least one counter unknown. - * `coverage` says what the figures rest on: `files_read`, `unrecognised_shapes` and `not_stored` - * are of this run alone, `records` of the period. `left_out` is what every ingest so far, - * whoever ran it (the plugin's included), measured and could not store, of a clone that - * consents: `undated` calls and `unrecognised` lines that are no usage line, each once; none has - * a day, so every report counts all of them. `consent_log_damaged` is true while - * `ledger/consents.jsonl` cannot be trusted and nothing is stored, `oldest_transcript_at` is when + * `coverage.records` counts every call of the period, matching the filters or not, and + * `coverage.left_out` what every ingest so far, whoever ran it (the plugin's included), + * measured and could not store, of a clone that consents: `undated` calls and `unrecognised` + * lines that are no usage line, each once; none has a day, so every report counts all of them. + * + * `run`: `files_read` transcripts this read opened, `unrecognised_shapes` and `not_stored` (by + * reason) what it met and did not store, `consent_log_damaged` true while + * `ledger/consents.jsonl` cannot be trusted and nothing is stored, `oldest_transcript_at` when * the oldest transcript still on disk was last written (`null` when there is none), and - * `skipped_ledger_lines` counts the lines that were not records, kept as found, in the ledger - * partitions the report read: a bounded report reads its period and the day on each side, so - * damage outside them is not counted. Nothing here is a path, a working directory or a branch - * name. + * `skipped_ledger_lines` the lines that were not records, kept as found, in the ledger + * partitions it read before settling them: a bounded report reads its period and the day on + * each side. Nothing here is a path, a working directory or a branch name. * * `filters` names what the report was narrowed to, every flag in a fixed order, `null` when it * was not given, else as resolved: the full id for a start or `.`, the start as typed when it - * names no call of the period. `rows` and `totals` count the calls that match; - * `coverage.records` counts every call of the period, matching or not. + * names no call of the period. `rows` and `totals` count the calls that match. * - * Under `AIDD_TELEMETRY=0` nothing is read and the envelope is `{ "version": 1, "refused": + * Under `AIDD_TELEMETRY=0` nothing is read and the envelope is `{ "version": 2, "refused": * "AIDD_TELEMETRY=0" }` (`REFUSED_ENVELOPE`). * - * A new field may appear in a version 1 envelope; a field is never renamed, retyped or removed - * without a new `version`. */ + * A new field may appear in a version 2 envelope, a figure outside `run`, a fact of the read in + * it; a field is never renamed, retyped, moved or removed without a new `version`. */ export interface ReportEnvelope { - readonly version: 1; + readonly version: 2; readonly period: Period; readonly axis: ReportAxis; readonly filters: ReportFilters; @@ -74,14 +80,16 @@ export interface ReportEnvelope { readonly totals: EnvelopeTally; readonly unknown_records: number; readonly coverage: { - readonly files_read: number; readonly records: number; + readonly left_out: Readonly>; + }; + readonly run: { + readonly files_read: number; readonly unrecognised_shapes: number; readonly not_stored: Readonly>; readonly consent_log_damaged: boolean; readonly oldest_transcript_at: string | null; readonly skipped_ledger_lines: number; - readonly left_out: Readonly>; }; } @@ -111,27 +119,26 @@ function rowOf(row: ReportRow): EnvelopeRow { export function reportEnvelopeOf( result: Extract ): ReportEnvelope { - const { period, filters, report, coverage } = result; + const { period, filters, report, coverage, run } = result; return { - version: 1, + version: 2, period, axis: report.axis, filters: filtersOf(filters), rows: report.rows.map(rowOf), totals: tallyOf(report.totals), unknown_records: report.totals.total.unknownRecords, - coverage: { - files_read: coverage.filesRead, - records: coverage.records, - unrecognised_shapes: coverage.unrecognised, - not_stored: coverage.notStored, - consent_log_damaged: coverage.consentLogDamaged, - oldest_transcript_at: coverage.oldestTranscriptAt, - skipped_ledger_lines: coverage.skippedLedgerLines, - left_out: coverage.leftOut, + coverage: { records: coverage.records, left_out: coverage.leftOut }, + run: { + files_read: run.filesRead, + unrecognised_shapes: run.unrecognised, + not_stored: run.notStored, + consent_log_damaged: run.consentLogDamaged, + oldest_transcript_at: run.oldestTranscriptAt, + skipped_ledger_lines: run.skippedLedgerLines, }, }; } /** What `--json` prints under `AIDD_TELEMETRY=0`, when nothing was read. */ -export const REFUSED_ENVELOPE = { version: 1, refused: "AIDD_TELEMETRY=0" } as const; +export const REFUSED_ENVELOPE = { version: 2, refused: "AIDD_TELEMETRY=0" } as const; diff --git a/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts b/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts index d020b09cb..7baf36e2e 100644 --- a/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts +++ b/cli/src/contexts/telemetry/application/report/report-usage-use-case.ts @@ -18,9 +18,8 @@ import { type UsageReport, } from "../../domain/report/usage-report.js"; import { repositoryIdOf } from "../../domain/repository-identity.js"; -import type { NotStoredReason } from "../../domain/repository-resolution.js"; import type { StoredUsage } from "../../domain/stored-usage.js"; -import type { IngestUsageUseCase } from "../ingest-usage-use-case.js"; +import type { IngestResult, IngestUsageUseCase } from "../ingest-usage-use-case.js"; import { settleLedger } from "../ledger-upkeep.js"; import type { DeclaredBindings } from "./declared-bindings.js"; @@ -41,28 +40,28 @@ export interface ReportRequest { /** The flags typed as the start of an id. */ type PrefixFlag = "session" | "repository"; -/** What the figures rest on, so a reader can tell a small number from a missing one. */ +/** What the figures rest on, so a reader can tell a small number from a missing one. Like the + * figures, a function of what is stored and asked alone. */ export interface ReportCoverage { - /** Transcripts read by this run: new, or grown since the last. */ - readonly filesRead: number; /** Billed calls in the report's period. */ readonly records: number; - /** Shapes this run met and did not recognise, so did not count. */ - readonly unrecognised: number; - /** Billed calls this run read and did not store, by the reason. */ - readonly notStored: Readonly>; - /** A line of `ledger/consents.jsonl` was not an event: nothing is stored until it is recovered. */ - readonly consentLogDamaged: boolean; - /** When the oldest transcript still on disk was last written: nothing older can be read. */ - readonly oldestTranscriptAt: string | null; - /** Lines of the partitions read that were not records: a bounded report reads its period - * and the day on each side. */ - readonly skippedLedgerLines: number; /** What every ingest so far, whoever ran it, measured and could not store, by reason: none of * it has a day, so every report counts all of it. */ readonly leftOut: Readonly>; } +/** What the read that made the report found: two reports over the same stored calls can find + * different things. */ +export interface ReportRun + extends Pick< + IngestResult, + "filesRead" | "unrecognised" | "notStored" | "consentLogDamaged" | "oldestTranscriptAt" + > { + /** Lines of the partitions read that were not records, counted before the report settled + * them: a bounded report reads its period and the day on each side. */ + readonly skippedLedgerLines: number; +} + export type ReportResult = | { readonly status: "refused" } /** Two flags given select on one axis, so together they would keep nothing. */ @@ -87,6 +86,7 @@ export type ReportResult = readonly filters: ReportFilters; readonly report: UsageReport; readonly coverage: ReportCoverage; + readonly run: ReportRun; }; /** Reads what is new, then answers what the stored calls consumed in a period, split along one @@ -103,7 +103,8 @@ export type ReportResult = * anything is read: refusal means no git is run either. */ export class ReportUsageUseCase { constructor( - private readonly ingest: IngestUsageUseCase, + /** Only its run is used: whatever it found is the report's `run`, never a figure. */ + private readonly ingest: Pick, private readonly ledger: UsageLedger, private readonly declared: DeclaredBindings, private readonly identity: PersonIdentityStore, @@ -144,15 +145,14 @@ export class ReportUsageUseCase { period: request.period, filters, report: buildReport(selected, facts, person, request.axis), - coverage: { + coverage: { records: records.length, leftOut }, + run: { filesRead, - records: records.length, unrecognised, notStored, consentLogDamaged, oldestTranscriptAt, skippedLedgerLines: held.skippedLines, - leftOut, }, }; } diff --git a/cli/src/presentation/display/telemetry/telemetry-report-display.ts b/cli/src/presentation/display/telemetry/telemetry-report-display.ts index a1fa07038..554ac94b0 100644 --- a/cli/src/presentation/display/telemetry/telemetry-report-display.ts +++ b/cli/src/presentation/display/telemetry/telemetry-report-display.ts @@ -1,6 +1,7 @@ import type { ReportCoverage, ReportResult, + ReportRun, } from "../../../contexts/telemetry/application/report/report-usage-use-case.js"; import type { Period } from "../../../contexts/telemetry/domain/report/period.js"; import { @@ -120,11 +121,11 @@ function periodWords(period: Period): string { return period.to === null ? "all days" : `until ${period.to}`; } -function coverageLines(coverage: ReportCoverage): string[] { +function coverageLines(coverage: ReportCoverage, run: ReportRun): string[] { const { undated, unrecognised } = coverage.leftOut; return [ - `Read ${count(coverage.filesRead, "transcript", "transcripts")} just now; ${count(coverage.records, "call", "calls")} in this period; oldest transcript on disk: ${coverage.oldestTranscriptAt?.slice(0, 10) ?? "none"}.`, - ...leftOutLines(coverage).map(({ text }) => text), + `Read ${count(run.filesRead, "transcript", "transcripts")} just now; ${count(coverage.records, "call", "calls")} in this period; oldest transcript on disk: ${run.oldestTranscriptAt?.slice(0, 10) ?? "none"}.`, + ...leftOutLines(run).map(({ text }) => text), ...(unrecognised > 0 ? [ `Kept from every read: ${count(unrecognised, "line that is", "lines that are")} no usage line.`, @@ -159,7 +160,7 @@ export function printUsageReport(output: CLIOutput, result: ReportResult): void ); return; } - const { report, coverage, period, filters } = result; + const { report, coverage, run, period, filters } = result; output.print(`Usage, ${periodWords(period)}, by ${report.axis}`); const selection = selectionLine(filters); if (selection !== null) output.print(selection); @@ -181,5 +182,5 @@ export function printUsageReport(output: CLIOutput, result: ReportResult): void } } output.print(""); - for (const line of coverageLines(coverage)) output.print(line); + for (const line of coverageLines(coverage, run)) output.print(line); } diff --git a/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts b/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts index c70546114..559612ebd 100644 --- a/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts @@ -40,9 +40,9 @@ const reported: Extract = { ], totals: tally(4, 1), }, - coverage: { + coverage: { records: 4, leftOut: { undated: 2, unrecognised: 5 } }, + run: { filesRead: 3, - records: 4, unrecognised: 1, notStored: { "outside-repo": 0, @@ -55,8 +55,7 @@ const reported: Extract = { }, consentLogDamaged: false, oldestTranscriptAt: "2026-09-20T08:00:00.000Z", - skippedLedgerLines: 0, - leftOut: { undated: 2, unrecognised: 5 }, + skippedLedgerLines: 6, }, }; @@ -65,7 +64,7 @@ describe("the JSON envelope of a report", () => { const envelope = reportEnvelopeOf(reported); expect(Object.keys(envelope)[0]).toBe("version"); expect(envelope).toMatchObject({ - version: 1, + version: 2, period: { from: "2026-10-01", to: null }, axis: "task", }); @@ -97,17 +96,34 @@ describe("the JSON envelope of a report", () => { expect(envelope.unknown_records).toBe(1); }); - it("states the coverage in snake case, and nothing about where the data lives", () => { + it("states what the figures rest on, in snake case, and nothing about where the data lives", () => { const envelope = reportEnvelopeOf(reported); - expect(envelope.coverage).toEqual({ - files_read: 3, + expect(envelope.coverage).toStrictEqual({ records: 4, + left_out: { undated: 2, unrecognised: 5 }, + }); + }); + + it("puts what the read that printed it found apart, last, so the rest is the figures", () => { + const envelope = reportEnvelopeOf(reported); + expect(Object.keys(envelope)).toEqual([ + "version", + "period", + "axis", + "filters", + "rows", + "totals", + "unknown_records", + "coverage", + "run", + ]); + expect(envelope.run).toStrictEqual({ + files_read: 3, unrecognised_shapes: 1, - not_stored: reported.coverage.notStored, + not_stored: reported.run.notStored, consent_log_damaged: false, oldest_transcript_at: "2026-09-20T08:00:00.000Z", - skipped_ledger_lines: 0, - left_out: { undated: 2, unrecognised: 5 }, + skipped_ledger_lines: 6, }); }); @@ -128,6 +144,6 @@ describe("the JSON envelope of a report", () => { }); it("is versioned when nothing was read under AIDD_TELEMETRY=0", () => { - expect(REFUSED_ENVELOPE).toEqual({ version: 1, refused: "AIDD_TELEMETRY=0" }); + expect(REFUSED_ENVELOPE).toEqual({ version: 2, refused: "AIDD_TELEMETRY=0" }); }); }); diff --git a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts new file mode 100644 index 000000000..8f43f64a7 --- /dev/null +++ b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts @@ -0,0 +1,342 @@ +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import * as fc from "fast-check"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import type { + IngestResult, + IngestUsageUseCase, +} from "../../../../../src/contexts/telemetry/application/ingest-usage-use-case.js"; +import { storeCalls } from "../../../../../src/contexts/telemetry/application/ledger-upkeep.js"; +import { DeclaredBindings } from "../../../../../src/contexts/telemetry/application/report/declared-bindings.js"; +import { reportEnvelopeOf } from "../../../../../src/contexts/telemetry/application/report/report-envelope.js"; +import { + type ReportRequest, + ReportUsageUseCase, +} from "../../../../../src/contexts/telemetry/application/report/report-usage-use-case.js"; +import { declarationOf } from "../../../../../src/contexts/telemetry/domain/declaration/task-declaration.js"; +import type { LeftOutEntry } from "../../../../../src/contexts/telemetry/domain/ports/usage-ledger.js"; +import type { Period } from "../../../../../src/contexts/telemetry/domain/report/period.js"; +import { + conflictsOf, + type ReportFilters, +} from "../../../../../src/contexts/telemetry/domain/report/report-filter.js"; +import { REPORT_AXES } from "../../../../../src/contexts/telemetry/domain/report/usage-report.js"; +import type { StoredUsage } from "../../../../../src/contexts/telemetry/domain/stored-usage.js"; +import { USAGE_AGENTS } from "../../../../../src/contexts/telemetry/domain/usage-record.js"; +import { UsageLedgerAdapter } from "../../../../../src/contexts/telemetry/infrastructure/usage-ledger-adapter.js"; +import { PrivateStorageAdapter } from "../../../../../src/runtime/filesystem/private-storage-adapter.js"; +import { + FakeLocator, + InMemoryIdentity, + InMemorySessions, + InMemorySnapshots, +} from "../../../../helpers/ports/in-memory-telemetry.js"; + +/* The figures of a report, every field of its envelope but `run`, depend on what is stored, + * declared and asked, and on nothing else: not on how the calls reached the ledger, not on what + * the read before it found, not on the zone of the machine's clock. */ + +let root: string; +let ledgers = 0; + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "aidd-report-figures-")); +}); +afterEach(async () => { + await rm(root, { recursive: true, force: true }); +}); + +/** UTC, and two zones 25 hours apart: whatever reads the local day reads another one. */ +const ZONES = ["UTC", "Pacific/Kiritimati", "America/Adak"] as const; +type Zone = (typeof ZONES)[number]; +/** The offset `getTimezoneOffset` gives each zone on a day of the calls, so a switch that did + * not take fails here instead of passing for nothing. */ +const OFFSETS: Readonly> = { + UTC: 0, + "Pacific/Kiritimati": -840, + "America/Adak": 540, +}; + +/** A month's end inside the days, so a local day crosses a month as well as a day. */ +const DAYS = ["2026-09-30", "2026-10-01", "2026-10-02"]; +const SESSIONS = ["s-a1", "s-a2", "s-b"]; +const REPOSITORIES = ["r-a1", "r-a2", "r-b"]; + +const snapshot = fc.record({ + minutes: fc.integer({ min: -90, max: 90 }), + output: fc.option(fc.nat(50), { nil: null, freq: 5 }), + cacheRead: fc.option(fc.nat(50), { nil: null, freq: 5 }), +}); +const call = fc.record({ + day: fc.constantFrom(...DAYS), + // The first and last hours of a day: a local day is another day in one of the zones. + hour: fc.constantFrom(0, 1, 11, 22, 23), + session: fc.constantFrom(...SESSIONS), + repository: fc.constantFrom(...REPOSITORIES), + agent: fc.constantFrom(...USAGE_AGENTS), + model: fc.constantFrom("m-1", "m-2", null), + snapshots: fc.array(snapshot, { minLength: 1, maxLength: 3 }), +}); +const declaration = fc.record({ + session: fc.constantFrom(...SESSIONS), + task: fc.constantFrom("t-1", "t-2", null), + ticket: fc.constantFrom("T-1", null), + day: fc.constantFrom("2026-09-29", ...DAYS), + hour: fc.constantFrom(0, 12, 23), +}); +const edge = fc.option(fc.constantFrom(...DAYS), { nil: null }); +const period = fc + .tuple(edge, edge) + .map( + ([a, b]): Period => + a !== null && b !== null && a > b ? { from: b, to: a } : { from: a, to: b } + ); +const filters = fc + .record({ + task: fc.constantFrom(null, "t-1"), + ticket: fc.constantFrom(null, "T-1"), + person: fc.constantFrom(null, "p-1"), + model: fc.constantFrom(null, "m-1"), + agent: fc.constantFrom(null, ...USAGE_AGENTS), + session: fc.constantFrom(null, "s-a1", "s-b", "zz"), + repository: fc.constantFrom(null, "r-a1", "r-b"), + absent: fc.constantFrom(null, "person" as const, "model" as const, "ticket" as const), + unattributed: fc.constantFrom(null, "any" as const, "no-binding" as const), + }) + .filter((f: ReportFilters) => conflictsOf(f).length === 0); + +const NOT_STORED_REASONS = [ + "outside-repo", + "never-seen-alive", + "no-consent", + "consent-closed", + "unreadable-consent", + "no-cwd", + "undated", +] as const; +/** What the read before a report may have found: anything, since none of it is a figure. */ +const run = fc.record({ + filesRead: fc.nat(9), + unrecognised: fc.nat(9), + notStored: fc.tuple(...NOT_STORED_REASONS.map(() => fc.nat(9))), + consentLogDamaged: fc.boolean(), + oldestTranscriptAt: fc.option( + fc.constantFrom("2026-09-01T10:00:00.000Z", "2026-09-30T23:30:00.000Z"), + { nil: null } + ), +}); +type Run = typeof run extends fc.Arbitrary ? T : never; + +const scenario = fc.record({ + calls: fc.array(call, { maxLength: 8 }), + declarations: fc.array(declaration, { maxLength: 4 }), + person: fc.constantFrom("p-1", null), + leftOut: fc.array( + fc.record({ + reason: fc.constantFrom("undated" as const, "unrecognised" as const), + of: fc.constantFrom("a", "b", "c", "d"), + }), + { maxLength: 5 } + ), + // Lines no version reads, already on disk, some twice: settling keeps each once. + damaged: fc.array(fc.record({ day: fc.constantFrom(...DAYS), text: fc.constantFrom("x", "y") }), { + maxLength: 3, + }), + request: fc.record({ axis: fc.constantFrom(...REPORT_AXES), period, filters }), + // How another history reached the same data: each snapshot and left-out entry in a batch of + // its own choosing, the batches in any order, and stores that bring nothing in between. + history: fc.record({ + batchOf: fc.array(fc.nat(3), { minLength: 40, maxLength: 40 }), + order: fc.shuffledSubarray([0, 1, 2, 3], { minLength: 4, maxLength: 4 }), + noOps: fc.nat(2), + }), + zones: fc.tuple(fc.constantFrom(...ZONES), fc.constantFrom(...ZONES)), + runs: fc.tuple(run, run, run), +}); +type Scenario = typeof scenario extends fc.Arbitrary ? T : never; + +function recordsOf(calls: Scenario["calls"]): StoredUsage[] { + return calls.flatMap((c, index) => + c.snapshots.map( + (s): StoredUsage => ({ + tool: "claude-code", + tool_version: "1.0.0", + key: `k${index}`, + session_id: c.session, + agent: c.agent, + at: new Date( + Date.parse(`${c.day}T${String(c.hour).padStart(2, "0")}:30:00.000Z`) + s.minutes * 60_000 + ).toISOString(), + model: c.model, + input: 1, + output: s.output, + cache_read: s.cacheRead, + cache_write: 0, + cache_write_1h: null, + reasoning: null, + cwd: null, + git_branch: null, + repository_id: c.repository, + }) + ) + ); +} + +/** A ledger directory holding the damaged lines, as an earlier version or a hand edit left. */ +async function ledgerWith(damaged: Scenario["damaged"]): Promise { + ledgers += 1; + const dir = join(root, `ledger-${ledgers}`); + await mkdir(dir, { recursive: true }); + const byDay = new Map(); + for (const { day, text } of damaged) byDay.set(day, `${byDay.get(day) ?? ""}${text}\n`); + for (const [day, text] of byDay) await writeFile(join(dir, `${day}.jsonl`), text); + return new UsageLedgerAdapter(dir, new PrivateStorageAdapter()); +} + +function ingestResultOf(facts: Run): IngestResult { + return { + refused: false, + filesRead: facts.filesRead, + added: 0, + updated: 0, + unrecognised: facts.unrecognised, + skippedLedgerLines: 0, + snapshots: 0, + notStored: Object.fromEntries( + NOT_STORED_REASONS.map((reason, n) => [reason, facts.notStored[n]]) + ) as IngestResult["notStored"], + consentLogDamaged: facts.consentLogDamaged, + oldestTranscriptAt: facts.oldestTranscriptAt, + }; +} + +async function inZone(zone: Zone, work: () => Promise): Promise { + const before = process.env.TZ; + process.env.TZ = zone; + try { + expect(new Date("2026-10-01T12:00:00.000Z").getTimezoneOffset(), zone).toBe(OFFSETS[zone]); + return await work(); + } finally { + if (before === undefined) delete process.env.TZ; + else process.env.TZ = before; + } +} + +/** Reports once over the ledger, after a read that found `facts`, with the clock in `zone`. */ +async function report( + held: Scenario, + ledger: UsageLedgerAdapter, + request: ReportRequest, + facts: Run, + zone: Zone +) { + const sessions = new InMemorySessions(); + for (const d of held.declarations) { + await sessions.append( + d.session, + declarationOf( + d.task === null ? { kind: "none" } : { kind: "task", task: d.task, ticket: d.ticket }, + new Date(`${d.day}T${String(d.hour).padStart(2, "0")}:00:00.000Z`), + "command" + ) + ); + } + const identity = new InMemoryIdentity(); + identity.personId = held.person; + const ingest: Pick = { + execute: async () => ingestResultOf(facts), + }; + const result = await inZone(zone, () => + new ReportUsageUseCase( + ingest, + ledger, + new DeclaredBindings(sessions, new InMemorySnapshots()), + identity, + new FakeLocator(), + { refusedByEnvironment: false } + ).execute(request) + ); + return result.status === "reported" ? reportEnvelopeOf(result) : result; +} + +/** The figures' bytes: the envelope as printed, but its `run`. */ +function figuresOf(envelope: Awaited>): string { + const { run: _run, ...figures } = envelope as Record; + return JSON.stringify(figures, null, 2); +} + +describe("the figures of a report", () => { + it("are byte for byte the same whatever history stored the same calls, whatever the read before found, on a second report, and in any zone", async () => { + await fc.assert( + fc.asyncProperty(scenario, async (held) => { + const records = recordsOf(held.calls); + const leftOut: LeftOutEntry[] = held.leftOut; + + // One history: every call and left-out entry stored at once, reported in UTC. + const once = await ledgerWith(held.damaged); + await once.exclusively(() => storeCalls(once, records)); + await once.leftOut.add(leftOut); + const reference = await report(held, once, held.request, held.runs[0], "UTC"); + + // Another: in batches, in another order, with stores that bring nothing. + const batched = await ledgerWith(held.damaged); + const { batchOf, order, noOps } = held.history; + const nth = (n: number) => batchOf[n % batchOf.length] as number; + for (const batch of order) { + const calls = records.filter((_, n) => nth(n) === batch); + await batched.exclusively(() => storeCalls(batched, calls)); + await batched.leftOut.add(leftOut.filter((_, n) => nth(n + 7) === batch)); + if (batch < noOps) { + await batched.exclusively(() => storeCalls(batched, [])); + await batched.leftOut.add([]); + } + } + const [first, second] = held.zones; + const later = await report(held, batched, held.request, held.runs[1], first); + const again = await report(held, batched, held.request, held.runs[2], second); + + expect(figuresOf(later)).toBe(figuresOf(reference)); + expect(figuresOf(again)).toBe(figuresOf(reference)); + if (!("run" in again)) return; + // What the read found is in `run`, and only there. + const facts = held.runs[2]; + expect(again.run).toEqual({ + files_read: facts.filesRead, + unrecognised_shapes: facts.unrecognised, + not_stored: ingestResultOf(facts).notStored, + consent_log_damaged: facts.consentLogDamaged, + oldest_transcript_at: facts.oldestTranscriptAt, + skipped_ledger_lines: again.run.skipped_ledger_lines, + }); + expect(Object.keys(again).at(-1)).toBe("run"); + }), + { numRuns: 60 } + ); + }, 120_000); + + it("leave out the ledger lines that are no record, whose count changes once a report has settled them", async () => { + const held = fc.sample(scenario, { numRuns: 1, seed: 1 })[0] as Scenario; + const quiet: Run = { + filesRead: 0, + unrecognised: 0, + notStored: [0, 0, 0, 0, 0, 0, 0], + consentLogDamaged: false, + oldestTranscriptAt: null, + }; + const ledger = await ledgerWith([ + { day: "2026-10-01", text: "x" }, + { day: "2026-10-01", text: "x" }, + ]); + const request: ReportRequest = { + axis: "day", + period: { from: "2026-10-01", to: "2026-10-01" }, + }; + const first = await report(held, ledger, request, quiet, "UTC"); + const second = await report(held, ledger, request, quiet, "UTC"); + expect(figuresOf(second)).toBe(figuresOf(first)); + // The rewrite kept the line once: the same ledger, read again, counts one fewer. + expect(first).toMatchObject({ run: { skipped_ledger_lines: 2 } }); + expect(second).toMatchObject({ run: { skipped_ledger_lines: 1 } }); + }); +}); diff --git a/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts b/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts index 60cc76803..1084fd49d 100644 --- a/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-usage-use-case.unit.test.ts @@ -141,7 +141,7 @@ describe("reporting usage", () => { const s = setup(); s.ledger.damaged.add("2026-01-15"); const result = await s.report.execute({ axis: "total", period: ANY }); - expect(result).toMatchObject({ coverage: { skippedLedgerLines: 1 } }); + expect(result).toMatchObject({ run: { skippedLedgerLines: 1 } }); expect(s.ledger.savedPartitions).toEqual(new Set(["2026-01-15"])); expect((await s.ledger.load()).skippedLines).toBe(1); }); @@ -175,7 +175,11 @@ describe("reporting usage", () => { const s = setup(); s.transcripts.files.set("/t/1.jsonl", [line("A", 10, "2026-10-07T10:00:00.000Z")]); const result = await s.report.execute({ axis: "total", period: ANY }); - expect(result).toMatchObject({ status: "reported", coverage: { filesRead: 1, records: 1 } }); + expect(result).toMatchObject({ + status: "reported", + coverage: { records: 1 }, + run: { filesRead: 1 }, + }); expect(s.events.indexOf("save")).toBeLessThan(s.events.lastIndexOf("load")); }); @@ -287,9 +291,9 @@ describe("reporting usage", () => { s.transcripts.modified.set("/t/1.jsonl", "2026-09-20T08:00:00.000Z"); const result = await s.report.execute({ axis: "total", period: ANY }); expect(result).toMatchObject({ - coverage: { + coverage: { records: 1 }, + run: { filesRead: 1, - records: 1, unrecognised: 0, oldestTranscriptAt: "2026-09-20T08:00:00.000Z", notStored: { "no-consent": 0, "outside-repo": 0 }, @@ -546,7 +550,8 @@ describe("saying what was measured and left out, whoever ingested it", () => { for (const period of [OCTOBER, ANY]) { const result = await s.report.execute({ axis: "total", period }); expect(result).toMatchObject({ - coverage: { filesRead: 0, unrecognised: 0, leftOut: { undated: 1, unrecognised: 2 } }, + coverage: { leftOut: { undated: 1, unrecognised: 2 } }, + run: { filesRead: 0, unrecognised: 0 }, }); } }); @@ -573,12 +578,14 @@ describe("a call whose consent cannot be read yet", () => { s.consents.unreadableClones.add(cloneKey(cloneOf("/work/a/.git"))); for (const _ of [1, 2]) { expect(await s.report.execute({ axis: "total", period: ANY })).toMatchObject({ - coverage: { records: 0, notStored: { "unreadable-consent": 1 } }, + coverage: { records: 0 }, + run: { notStored: { "unreadable-consent": 1 } }, }); } s.consents.unreadableClones.clear(); expect(await s.report.execute({ axis: "total", period: ANY })).toMatchObject({ - coverage: { records: 1, notStored: { "unreadable-consent": 0 } }, + coverage: { records: 1 }, + run: { notStored: { "unreadable-consent": 0 } }, }); }); }); diff --git a/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts index c4c816c8f..46013b1ee 100644 --- a/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-window.property.integration.test.ts @@ -273,15 +273,14 @@ describe("a report on a bounded period", () => { period, filters: resolved, report: buildReport(selected, facts, person, axis), - coverage: { + coverage: { records: kept.length, leftOut: await fullLedger.leftOut.counts() }, + run: { filesRead: NOTHING_NEW.filesRead, - records: kept.length, unrecognised: NOTHING_NEW.unrecognised, notStored: NOTHING_NEW.notStored, consentLogDamaged: NOTHING_NEW.consentLogDamaged, oldestTranscriptAt: NOTHING_NEW.oldestTranscriptAt, skippedLedgerLines: full.skippedLines, - leftOut: await fullLedger.leftOut.counts(), }, }); expect(JSON.stringify(reportEnvelopeOf(bounded))).toBe(JSON.stringify(reference)); diff --git a/cli/tests/e2e/telemetry-journey.e2e.test.ts b/cli/tests/e2e/telemetry-journey.e2e.test.ts index d3f127332..e80c7faf1 100644 --- a/cli/tests/e2e/telemetry-journey.e2e.test.ts +++ b/cli/tests/e2e/telemetry-journey.e2e.test.ts @@ -362,7 +362,7 @@ describe("the telemetry journey, on the built binary", () => { const totals = new Set(); for (const axis of AXES) { const envelope = await report(axis, first, last); - expect(envelope.version).toBe(1); + expect(envelope.version).toBe(2); expect(envelope.axis).toBe(axis); expect(envelope.totals.records, axis).toBe(CALLS.length); expect( @@ -452,13 +452,13 @@ describe("the telemetry journey, on the built binary", () => { runHook("catch-up.cjs", { id: S_MAIN }, { hook_event_name: "SessionStart", source: "startup" }); const run = await aidd(["report", "--json"]); expect(run.exitCode, run.stderr).toBe(0); - const { coverage } = JSON.parse(run.stdout) as { - coverage: { files_read: number; records: number; left_out: Record }; + const envelope = JSON.parse(run.stdout) as { + coverage: { records: number; left_out: Record }; + run: { files_read: number }; }; - expect(coverage).toMatchObject({ - files_read: 0, - records: 1, - left_out: { undated: 1, unrecognised: 1 }, + expect(envelope).toMatchObject({ + coverage: { records: 1, left_out: { undated: 1, unrecognised: 1 } }, + run: { files_read: 0 }, }); }); diff --git a/cli/tests/e2e/telemetry-report-figures.e2e.test.ts b/cli/tests/e2e/telemetry-report-figures.e2e.test.ts new file mode 100644 index 000000000..40e3e49b3 --- /dev/null +++ b/cli/tests/e2e/telemetry-report-figures.e2e.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from "vitest"; +import { aidd, declareBranch, useReportSandbox, WINDOW } from "./telemetry-report-sandbox.js"; + +useReportSandbox(); + +/** UTC, and two zones 25 hours apart: a call at 21:00 UTC is on another local day in each. */ +const ZONES = ["UTC", "Pacific/Kiritimati", "America/Adak"]; +/** English, and two locales that write numbers, dates and order otherwise. */ +const LOCALES = ["C", "fr_FR.UTF-8", "ar_SA.UTF-8"]; + +async function report(axis: string, zone: string, locale: string) { + const run = await aidd(["report", ...WINDOW, "--axis", axis, "--json"], { + TZ: zone, + LC_ALL: locale, + LANG: locale, + }); + expect(run.exitCode, run.stderr).toBe(0); + const { run: facts, ...figures } = JSON.parse(run.stdout) as Record; + return { figures: JSON.stringify(figures, null, 2), facts }; +} + +describe("aidd telemetry report --json", () => { + it("prints the same figures, byte for byte, on the read that ingests and every one after, in any zone and locale", async () => { + await declareBranch(); + for (const axis of ["day", "task"]) { + const first = await report(axis, "UTC", "C"); + for (const zone of ZONES) { + for (const locale of LOCALES) { + const later = await report(axis, zone, locale); + expect(later.figures, `${axis} in ${zone}, ${locale}`).toBe(first.figures); + expect(later.facts).toMatchObject({ files_read: 0 }); + } + } + // The first read of the first axis opened both transcripts; the run says so, apart. + expect(first.facts).toMatchObject({ files_read: axis === "day" ? 2 : 0 }); + } + }); +}); diff --git a/cli/tests/e2e/telemetry-report-narrowed.e2e.test.ts b/cli/tests/e2e/telemetry-report-narrowed.e2e.test.ts index 7dcb07cb9..8993ffb6b 100644 --- a/cli/tests/e2e/telemetry-report-narrowed.e2e.test.ts +++ b/cli/tests/e2e/telemetry-report-narrowed.e2e.test.ts @@ -132,6 +132,6 @@ describe("aidd telemetry report, narrowed", () => { outside ); expect(run.exitCode).toBe(0); - expect(JSON.parse(run.stdout)).toEqual({ version: 1, refused: "AIDD_TELEMETRY=0" }); + expect(JSON.parse(run.stdout)).toEqual({ version: 2, refused: "AIDD_TELEMETRY=0" }); }); }); diff --git a/cli/tests/e2e/telemetry-report-sandbox.ts b/cli/tests/e2e/telemetry-report-sandbox.ts index 9aff5b0af..ae662dff2 100644 --- a/cli/tests/e2e/telemetry-report-sandbox.ts +++ b/cli/tests/e2e/telemetry-report-sandbox.ts @@ -329,7 +329,8 @@ export interface Envelope { rows: Row[]; totals: Row; unknown_records: number; - coverage: { records: number; oldest_transcript_at: string | null }; + coverage: { records: number }; + run: { files_read: number; oldest_transcript_at: string | null }; } export async function json(axis: string): Promise { const run = await aidd(["report", ...WINDOW, "--axis", axis, "--json"]); diff --git a/cli/tests/e2e/telemetry-report.e2e.test.ts b/cli/tests/e2e/telemetry-report.e2e.test.ts index 143c0e788..9d862db5b 100644 --- a/cli/tests/e2e/telemetry-report.e2e.test.ts +++ b/cli/tests/e2e/telemetry-report.e2e.test.ts @@ -32,7 +32,7 @@ describe("aidd telemetry report", () => { const sums: Record = {}; for (const axis of AXES) { const envelope = await json(axis); - expect(envelope.version).toBe(1); + expect(envelope.version).toBe(2); expect(envelope.axis).toBe(axis); expect(envelope.totals.records).toBe(CALLS.length); expect(envelope.rows.reduce((n, row) => n + row.records, 0)).toBe(CALLS.length); @@ -183,7 +183,7 @@ describe("aidd telemetry report", () => { expect(text.exitCode).toBe(0); expect(text.stdout).toContain("AIDD_TELEMETRY=0"); const asJson = await aidd(["report", "--json"], { AIDD_TELEMETRY: "0" }); - expect(JSON.parse(asJson.stdout)).toEqual({ version: 1, refused: "AIDD_TELEMETRY=0" }); + expect(JSON.parse(asJson.stdout)).toEqual({ version: 2, refused: "AIDD_TELEMETRY=0" }); expect(await held()).toEqual(before); }); }); diff --git a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts index 19629355f..f87a1ba92 100644 --- a/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts +++ b/cli/tests/presentation/commands/telemetry-wiring.integration.test.ts @@ -121,14 +121,14 @@ const reported = { total: ZERO, }, }, - coverage: { + coverage: { records: 0, leftOut: { undated: 0, unrecognised: 0 } }, + run: { filesRead: 0, - records: 0, unrecognised: 0, notStored: NOT_STORED, + consentLogDamaged: false, oldestTranscriptAt: null, skippedLedgerLines: 0, - leftOut: { undated: 0, unrecognised: 0 }, }, }; @@ -165,13 +165,13 @@ describe("aidd telemetry report", () => { it("prints the versioned envelope with --json, and nothing else", async () => { report.mockResolvedValue(reported); await run("report", "--json"); - expect(JSON.parse(stdout())).toMatchObject({ version: 1, axis: "total", rows: [] }); + expect(JSON.parse(stdout())).toMatchObject({ version: 2, axis: "total", rows: [] }); }); it("prints the refusal envelope with --json when nothing was read", async () => { report.mockResolvedValue({ status: "refused" }); await run("report", "--json"); - expect(JSON.parse(stdout())).toEqual({ version: 1, refused: "AIDD_TELEMETRY=0" }); + expect(JSON.parse(stdout())).toEqual({ version: 2, refused: "AIDD_TELEMETRY=0" }); }); it("refuses a bad period before reading anything, with exit 1", async () => { diff --git a/cli/tests/presentation/display/telemetry/telemetry-report-display.unit.test.ts b/cli/tests/presentation/display/telemetry/telemetry-report-display.unit.test.ts index 950127ce6..5fc36316a 100644 --- a/cli/tests/presentation/display/telemetry/telemetry-report-display.unit.test.ts +++ b/cli/tests/presentation/display/telemetry/telemetry-report-display.unit.test.ts @@ -35,7 +35,9 @@ const NOT_STORED = { "no-cwd": 0, undated: 0, }; -type Coverage = Extract["coverage"]; +type Reported = Extract; +/** What the figures rest on and what the read found, given flat: `result` splits them. */ +type Coverage = Reported["coverage"] & Reported["run"]; const COVERAGE: Coverage = { filesRead: 2, records: 3, @@ -61,9 +63,12 @@ function result( period, filters, report: { axis, rows: rows.map(([value, t]) => ({ value, ...t })), totals }, - coverage: { ...COVERAGE, ...coverage }, + ...split({ ...COVERAGE, ...coverage }), }; } +function split({ records, leftOut, ...run }: Coverage): Pick { + return { coverage: { records, leftOut }, run }; +} const printed = (r: ReportResult): string[] => { const output = new CapturingOutput(false); printUsageReport(output, r); diff --git a/plugins/aidd-telemetry/skills/01-usage/actions/answer.md b/plugins/aidd-telemetry/skills/01-usage/actions/answer.md index f7ff16aee..8e6228a6c 100644 --- a/plugins/aidd-telemetry/skills/01-usage/actions/answer.md +++ b/plugins/aidd-telemetry/skills/01-usage/actions/answer.md @@ -16,17 +16,17 @@ The figures the report holds for the question, each counter apart, followed by e - A non-zero exit: show its message and stop. 2. **Check.** Read [the envelope reference](../references/envelope.md) for the fields. - `refused` present: measurement is refused by the environment. Say so and give no figure. - - `version` other than `1`: say the report has a shape this skill does not know, and give no figure. + - `version` other than `2`: say the report has a shape this skill does not know, and give no figure. 3. **State.** Give the rows and the `totals` as the report holds them: input, output, cache read, cache write and `total`, each with its token count. Name the period, the axis, and each filter as `filters` prints it. 4. **Say the unknowns.** Say each of these that applies, in this order: - a counter whose `unknown_records` is above `0`: its figure is a lower bound, and that many calls did not report it; - `unknown_records` on the totals above `0`: the same for the total; - a row of kind `unattributed`: its `reason`, with its meaning from the reference; - a row of kind `absent`: that the axis had no value for those calls; - - `coverage.unrecognised_shapes`, `coverage.skipped_ledger_lines`, any `coverage.not_stored` entry or any `coverage.left_out` entry above `0`: that some lines were read and left out, with the counts; + - `run.unrecognised_shapes`, `run.skipped_ledger_lines`, any `run.not_stored` entry or any `coverage.left_out` entry above `0`: that some lines were read and left out, with the counts; - `coverage.records` of `0`: that no call is recorded for the period, and why that may be, from the reference. - `totals.records` of `0` with `coverage.records` above `0`: that a filter matched no call of the period, naming each as `filters` prints it, from the reference. -5. **Bound the past.** When the period starts before `coverage.oldest_transcript_at`, say that Claude Code may have removed older session files, so earlier figures can be incomplete. +5. **Bound the past.** When the period starts before `run.oldest_transcript_at`, say that Claude Code may have removed older session files, so earlier figures can be incomplete. 6. **Offer the fix.** For work with no task, say it can be attributed afterwards by declaring the task from that branch with `aidd telemetry task [--ticket ]`. A branch renamed or copied since it was declared is undeclared under its new name (unless declared by `aidd` 5.5.0 or earlier), and is declared the same way. Declare nothing yourself. ## Test diff --git a/plugins/aidd-telemetry/skills/01-usage/references/envelope.md b/plugins/aidd-telemetry/skills/01-usage/references/envelope.md index 5a02b235a..31b4e4f7c 100644 --- a/plugins/aidd-telemetry/skills/01-usage/references/envelope.md +++ b/plugins/aidd-telemetry/skills/01-usage/references/envelope.md @@ -1,12 +1,14 @@ # Envelope -What `aidd telemetry report --json` prints, and what each part means. Version `1`. +What `aidd telemetry report --json` prints, and what each part means. Version `2`. + +Every field but `run` is a figure: two reports over the same stored calls, declarations, period and filters print them byte for byte the same, whatever ran before, in any time zone or locale. `run` is what the read that printed the report found, and may differ from one report to the next. ## Top level | Field | Meaning | | --- | --- | -| `version` | `1`. Any other value is a shape not known here | +| `version` | `2`. Any other value is a shape not known here | | `refused` | Only present, with `version`, when measurement is refused by the environment; nothing was read | | `period` | `from` and `to`, UTC days, `null` for open | | `axis` | What `rows` split by | @@ -15,6 +17,7 @@ What `aidd telemetry report --json` prints, and what each part means. Version `1 | `totals` | The calls of the period the filters keep | | `unknown_records` | Calls with at least one counter unknown | | `coverage` | What the figures rest on | +| `run` | What the read that printed the report found, last | ## A row and the totals @@ -39,22 +42,27 @@ What `aidd telemetry report --json` prints, and what each part means. Version `1 | Field | Meaning | | --- | --- | -| `files_read` | Session files read by this run | | `records` | Calls recorded in the period, whether the filters keep them or not | +| `left_out` | What every read so far, one at session start included, measured in a clone that consents and could not store: `undated` calls and `unrecognised` lines that are no usage line, each counted once in every report | + +## Run + +| Field | Meaning | +| --- | --- | +| `files_read` | Session files read by this run | | `unrecognised_shapes` | Lines of a shape the reader did not recognise in this run, reported and never guessed | | `not_stored` | Calls this run read and left out, by reason: `outside-repo`, `never-seen-alive`, `no-consent`, `consent-closed`, `unreadable-consent`, `no-cwd`, `undated` | -| `left_out` | What every read so far, one at session start included, measured in a clone that consents and could not store: `undated` calls and `unrecognised` lines that are no usage line, each counted once in every report | | `consent_log_damaged` | `true` while `ledger/consents.jsonl` cannot be trusted and nothing is stored | | `oldest_transcript_at` | When the oldest session file still on disk was last written, `null` when there is none | -| `skipped_ledger_lines` | Lines that were not records, kept as found (a newer version may have written them), in the stored days this report read: with a period, those days and the day on each side, plus every day of a month still kept in one file | +| `skipped_ledger_lines` | Lines that were not records, kept as found (a newer version may have written them), in the stored days this run read before tidying them: with a period, those days and the day on each side, plus every day of a month still kept in one file | ## When there is nothing | Sign | Likely reason | | --- | --- | -| `records` of `0` and `not_stored.no-consent` above `0` | The clone has not opted in, or the calls were made before it did or while it was off: measurement starts at `aidd telemetry on` and reads nothing back | -| `not_stored.consent-closed` above `0` | The clone opted in once and its consent was closed: `aidd telemetry on` in that clone measures again, from then on | -| `not_stored.unreadable-consent` above `0` | A clone's git config or git directory cannot be read: nothing is stored for it until it can. For a clone that had opted in, its calls are then stored, read again by the next run; for any other directory, they are not | -| `consent_log_damaged` is `true` | A line of the consent log `ledger/consents.jsonl` is not an event, so nothing is stored for any clone. Recovery is `aidd telemetry forget --yes`, then `aidd telemetry on` in each clone: nothing from before is stored. Do not edit the log by hand | +| `records` of `0` and `run.not_stored.no-consent` above `0` | The clone has not opted in, or the calls were made before it did or while it was off: measurement starts at `aidd telemetry on` and reads nothing back | +| `run.not_stored.consent-closed` above `0` | The clone opted in once and its consent was closed: `aidd telemetry on` in that clone measures again, from then on | +| `run.not_stored.unreadable-consent` above `0` | A clone's git config or git directory cannot be read: nothing is stored for it until it can. For a clone that had opted in, its calls are then stored, read again by the next run; for any other directory, they are not | +| `run.consent_log_damaged` is `true` | A line of the consent log `ledger/consents.jsonl` is not an event, so nothing is stored for any clone. Recovery is `aidd telemetry forget --yes`, then `aidd telemetry on` in each clone: nothing from before is stored. Do not edit the log by hand | | `totals.records` of `0` and `coverage.records` above `0`, with a filter given | No call of the period matches the filters: a value no call has, or a start of a session or repository id that names none, printed as typed in `filters` | -| `records` of `0` and `files_read` of `0` | No clone on this machine has opted in, so no session file is read (`aidd telemetry status` says so in the clone); or no Claude Code session file was written since the first `aidd telemetry on` | +| `records` of `0` and `run.files_read` of `0` | No clone on this machine has opted in, so no session file is read (`aidd telemetry status` says so in the clone); or no Claude Code session file was written since the first `aidd telemetry on` | diff --git a/scripts/__tests__/check-claude-tokens-end-to-end.test.js b/scripts/__tests__/check-claude-tokens-end-to-end.test.js index 30816be9c..275fc59f2 100644 --- a/scripts/__tests__/check-claude-tokens-end-to-end.test.js +++ b/scripts/__tests__/check-claude-tokens-end-to-end.test.js @@ -18,9 +18,9 @@ function exact(overrides = {}) { function reports(row = exact(), totals = exact()) { return { billed: BILLED, - byTask: { totals, rows: [{ kind: "value", key: "demo", ...row }] }, - byTicket: { totals, rows: [{ kind: "value", key: "T-1", ...row }] }, - here: { filters: { task: "demo", repository: "a".repeat(64) }, totals: row }, + byTask: { version: 2, totals, rows: [{ kind: "value", key: "demo", ...row }] }, + byTicket: { version: 2, totals, rows: [{ kind: "value", key: "T-1", ...row }] }, + here: { version: 2, filters: { task: "demo", repository: "a".repeat(64) }, totals: row }, }; } @@ -67,6 +67,16 @@ describe("the report of a real claude run, judged against what the API billed", assert.match(judge(narrowed).message, /--repository \. resolved to "\."/u); }); + it("calls itself broken on a report of a version it does not read, figures right or not", () => { + for (const which of ["byTask", "byTicket", "here"]) { + const older = reports(); + older[which].version = 1; + const verdict = judge(older); + assert.equal(verdict.code, 2, which); + assert.match(verdict.message, /version 1, not 2/u); + } + }); + it("calls itself broken when the API billed fewer calls than prompts were sent", () => { assert.equal(judge({ ...reports(), billed: BILLED.slice(0, 1) }).code, 2); }); diff --git a/scripts/check-claude-tokens-end-to-end.cjs b/scripts/check-claude-tokens-end-to-end.cjs index c0781a72d..4233d5719 100644 --- a/scripts/check-claude-tokens-end-to-end.cjs +++ b/scripts/check-claude-tokens-end-to-end.cjs @@ -26,6 +26,8 @@ const path = require("node:path"); const WRONG = 1; const BROKEN = 2; +/** The `aidd telemetry report --json` shape this check reads. */ +const ENVELOPE_VERSION = 2; const RUN_TIMEOUT_MS = 120_000; const WAIT_MS = 60_000; const API_KEY = "sk-ant-e2e-not-a-real-key-0000000000"; @@ -263,6 +265,12 @@ function compare(label, actual, expected) { * repository, against the billed calls. */ function judge({ billed, byTask, byTicket, here, prompts = PROMPTS.length }) { if (billed.length < prompts) return { code: BROKEN, message: `the API billed ${billed.length} calls for ${prompts} prompts` }; + // The figures are read as envelope version 2 shapes them: another version cannot be judged. + for (const [label, report] of [["--axis task", byTask], ["--axis ticket", byTicket], ["--task demo --repository .", here]]) { + if (report?.version !== ENVELOPE_VERSION) { + return { code: BROKEN, message: `the report ${label} is envelope version ${report?.version}, not ${ENVELOPE_VERSION}` }; + } + } const expected = expectedTotals(billed); const rowOf = (report, key) => report?.rows?.find((row) => row.kind === "value" && row.key === key); const wrong = [ From c0fee29ec4e9bab4773bd8e5131d686fd89c092d Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:02:30 +0200 Subject: [PATCH 02/13] test(cli): run the telemetry tests under a foreign clock and locale CI runs in UTC and English, so code that reads the machine's time zone or language passed everywhere it was checked. `pnpm test:foreign-clock` runs the telemetry tests with TZ=Pacific/Kiritimati (UTC+14) and LC_ALL=LANG=ar_SA.UTF-8. It first checks that a child Node really sees both, and fails if it does not. CI runs it as one step in the existing ubuntu cli-coverage job: no new job, no matrix. Hand mutations prove the guard (see the task evidence): - the ledger day read in local time: 36 tests fail under the guard, where the default suite on origin/next caught nothing; - counts grouped by the default locale: 24 tests fail under the guard, and the default suite stays green. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .github/workflows/cli-ci.yml | 4 +++ cli/aidd_docs/memory/telemetry.md | 1 + cli/package.json | 1 + cli/scripts/test-foreign-clock.mjs | 47 ++++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+) create mode 100644 cli/scripts/test-foreign-clock.mjs diff --git a/.github/workflows/cli-ci.yml b/.github/workflows/cli-ci.yml index 5414579d2..15875fe42 100644 --- a/.github/workflows/cli-ci.yml +++ b/.github/workflows/cli-ci.yml @@ -297,6 +297,10 @@ jobs: restore-keys: pnpm-${{ runner.os }}- - run: cd cli && pnpm install --frozen-lockfile - run: cd cli && pnpm test:coverage + # The runner is UTC and English: the telemetry tests again with the clock at UTC+14 and an + # Arabic locale, so a figure that reads the machine's zone or language cannot pass unseen. + - name: Telemetry tests under a foreign clock and locale + run: cd cli && pnpm test:foreign-clock cli-smoke: name: cli / Smoke diff --git a/cli/aidd_docs/memory/telemetry.md b/cli/aidd_docs/memory/telemetry.md index 0d5a377a0..c98c6e8c6 100644 --- a/cli/aidd_docs/memory/telemetry.md +++ b/cli/aidd_docs/memory/telemetry.md @@ -75,4 +75,5 @@ flowchart LR - Unit and integration tests mirror `src/` under `tests/contexts/telemetry/`. - The `telemetry` end-to-end tests under `tests/e2e/` run the built binary; `tests/e2e/telemetry-journey.e2e.test.ts` drives the plugin's real hooks through a shim to it and checks that every axis adds up to the total. +- `pnpm test:foreign-clock` (`scripts/test-foreign-clock.mjs`) runs the telemetry tests with the clock in `Pacific/Kiritimati` and the locale `ar_SA.UTF-8`, after checking that a child Node really sees both. CI runs it in `cli-coverage`. A test that compares figures across zones sets `TZ` itself; the locale is fixed at process start, so only a child process can vary it. - The skills' commands are checked against the help golden by `scripts/__tests__/telemetry-skills-name-real-commands.test.js`. diff --git a/cli/package.json b/cli/package.json index 2c5dfa5a5..f88b4807f 100644 --- a/cli/package.json +++ b/cli/package.json @@ -46,6 +46,7 @@ "test:unit": "vitest run --project=unit", "test:integration": "vitest run --project=integration", "test:e2e": "vitest run --project=e2e", + "test:foreign-clock": "node scripts/test-foreign-clock.mjs", "test:e2e:kilo": "KILO_RUNTIME_SMOKE=1 vitest run --project=e2e tests/e2e/kilo-runtime.e2e.test.ts", "test:coverage": "vitest run --coverage", "test:kanban": "pnpm --dir ../kanban test", diff --git a/cli/scripts/test-foreign-clock.mjs b/cli/scripts/test-foreign-clock.mjs new file mode 100644 index 000000000..8a2a26785 --- /dev/null +++ b/cli/scripts/test-foreign-clock.mjs @@ -0,0 +1,47 @@ +#!/usr/bin/env node +/* Runs the telemetry tests with the clock far from UTC and the locale far from English, so code + * that reads the machine's zone or language fails here instead of passing wherever CI runs. A + * test that compares across zones sets `TZ` itself; this catches the ones that do not, and the + * locale, which a process cannot change once started. */ +import { spawnSync } from "node:child_process"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const CLI_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +/** UTC+14, the farthest ahead: a call before 10:00 UTC is on the next local day. */ +const ZONE = "Pacific/Kiritimati"; +/** Arabic digits and separators, right to left. */ +const LOCALE = "ar_SA.UTF-8"; +const TESTS = [ + "tests/contexts/telemetry", + "tests/presentation/commands/telemetry", + "tests/presentation/display/telemetry", + "tests/e2e/telemetry", +]; + +const env = { ...process.env, TZ: ZONE, LC_ALL: LOCALE, LANG: LOCALE }; + +// A zone or locale the platform ignores would run the tests in UTC and English, and pass. +const probe = spawnSync( + process.execPath, + [ + "-p", + 'JSON.stringify([new Date("2026-10-01T12:00:00Z").getTimezoneOffset(), Intl.DateTimeFormat().resolvedOptions().locale])', + ], + { env, encoding: "utf8" } +); +const [offset, locale] = probe.status === 0 ? JSON.parse(probe.stdout) : [0, "unknown"]; +if (offset === 0 || locale.startsWith("en")) { + console.error( + `A child Node sees UTC offset ${-offset} min and locale ${locale} under TZ=${ZONE} LC_ALL=${LOCALE}: this guard would prove nothing here.` + ); + process.exit(1); +} +console.log(`Telemetry tests under TZ=${ZONE}, ${-offset} min from UTC, locale ${locale}`); + +const run = spawnSync( + process.execPath, + [join(CLI_ROOT, "node_modules", "vitest", "vitest.mjs"), "run", ...TESTS], + { cwd: CLI_ROOT, env, stdio: "inherit" } +); +process.exit(run.status ?? 1); From 25c9f8570d1eadce74a06b284bd89a396121cd16 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:05:13 +0200 Subject: [PATCH 03/13] test(telemetry): vary the clock's zone only where node can move it Node applies a new `TZ` only on a process's main thread. Stryker runs Vitest on worker threads, where the figures property failed its own precondition: "UTC: expected -120 to be +0". On a worker the property now runs in the zone the thread started in. The forked suite still varies the zone, and `pnpm test:foreign-clock` covers the rest. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .../report/report-figures.property.integration.test.ts | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts index 8f43f64a7..99c3404e5 100644 --- a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts @@ -1,6 +1,7 @@ import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { isMainThread } from "node:worker_threads"; import * as fc from "fast-check"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import type { @@ -211,7 +212,11 @@ function ingestResultOf(facts: Run): IngestResult { }; } +/** Node moves the clock's zone when `TZ` is set, but only on a process's main thread: a worker + * thread (Stryker runs Vitest on them) keeps the zone it started in. There the zone is not + * varied, and the zones are left to the forked suite and to `pnpm test:foreign-clock`. */ async function inZone(zone: Zone, work: () => Promise): Promise { + if (!isMainThread) return work(); const before = process.env.TZ; process.env.TZ = zone; try { From 70260b6c952bf606d12861a55d1c9107581baf18 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:20:03 +0200 Subject: [PATCH 04/13] docs(telemetry): frame, plan and evidence for report envelope v2 Contract with a field-by-field v1 to v2 table and the reason for each move, the plan and its phases, the red runs, the hand mutations that prove the foreign clock guard, and the gates. Review pending. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .../2026_10_17_report-envelope-v2/contract.md | 101 ++++++++++++++++++ .../2026_10_17_report-envelope-v2/evidence.md | 89 +++++++++++++++ .../2026_10_17_report-envelope-v2/phase-1.md | 17 +++ .../2026_10_17_report-envelope-v2/phase-2.md | 13 +++ .../2026_10_17_report-envelope-v2/phase-3.md | 17 +++ .../2026_10_17_report-envelope-v2/phase-4.md | 20 ++++ .../2026_10_17_report-envelope-v2/plan.md | 32 ++++++ .../2026_10_17_report-envelope-v2/review.md | 3 + 8 files changed, 292 insertions(+) create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-1.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-2.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-4.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/plan.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md new file mode 100644 index 000000000..b43d32a6b --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md @@ -0,0 +1,101 @@ +--- +status: implemented +--- + +# Report envelope v2: figures apart from the run + +## Need + +Found by the R1 telemetry audit (M5, M9). + +- **M5.** `aidd telemetry report --json` (envelope v1) puts the figures next to facts about the run that printed them. `coverage.files_read`, `unrecognised_shapes` and `not_stored` are "of this run", and `oldest_transcript_at` depends on transcripts on disk. So two reports over the same measured data are not byte-identical: the second reads 0 files where the first read 2. +- **M9.** No test runs under a non-UTC time zone or a non-English locale. CI runs UTC and English, so code that reads the local clock or locale passes everywhere it is checked. + +Maintainer decision: move to envelope **v2**, the clean final target. The tool is in beta, so breaking the envelope is accepted, with no v1 shim. + +## Decision + +```mermaid +flowchart LR + L[("ledger, left-out.txt")] --> F + D[("declarations, identity")] --> F + Q["period, filters, axis"] --> F + F["figures: everything but run"] + I["the ingest this report ran first, transcripts on disk, partitions read"] --> R["run"] +``` + +- **The figures** are every top-level field except `run`. They are a pure function of the ledger (records and `left-out.txt`), the declarations, the chosen identity, the period, the filters and the axis. Their bytes are `JSON.stringify` of the envelope with `run` removed. +- **`run`**, always last, holds what the read that printed the envelope found. It can differ between two reports over the same data. +- **Inside the CLI**, `ReportResult` makes the same split: `coverage` (`records`, `leftOut`) and `run` (`filesRead`, `unrecognised`, `notStored`, `consentLogDamaged`, `oldestTranscriptAt`, `skippedLedgerLines`). The text report prints exactly what it printed before. +- **Refused** becomes `{"version": 2, "refused": "AIDD_TELEMETRY=0"}`. + +## v1 to v2, field by field + +| v1 | v2 | Why | +| --- | --- | --- | +| `version: 1` | `version: 2` | breaking shape | +| `refused` | `refused`, with `version: 2` | nothing read, so no figure and no run | +| `period` | `period` | an input | +| `axis` | `axis` | an input | +| `filters` | `filters` | inputs, resolved against the period's calls (a function of the ledger). `--repository .` resolves from the working directory, which is part of the request | +| `rows` | `rows` | ledger, declarations and identity, filtered | +| `totals` | `totals` | same | +| `unknown_records` | `unknown_records` | same as `totals.total.unknown_records`, kept for readers that check it alone | +| `coverage.records` | `coverage.records` | the calls of the period in the ledger | +| `coverage.left_out` | `coverage.left_out` | a set of hashes, each kept once. Neither order nor a repeated ingest changes it, and none of it has a day, so every report counts all of it | +| `coverage.files_read` | `run.files_read` | what this read opened: 2 then 0 over the same data | +| `coverage.unrecognised_shapes` | `run.unrecognised_shapes` | lines this read met. The lasting count is `coverage.left_out.unrecognised` | +| `coverage.not_stored` | `run.not_stored` | calls this read left out. A later read never sees them again | +| `coverage.consent_log_damaged` | `run.consent_log_damaged` | says whether this read could store anything. It describes the store at run time, not any figure | +| `coverage.oldest_transcript_at` | `run.oldest_transcript_at` | transcripts' mtimes on disk: it moves as Claude Code deletes files | +| `coverage.skipped_ledger_lines` | `run.skipped_ledger_lines` | counts the damaged lines in the partitions this read loaded, before its own settle rewrote them. A rewrite keeps a line once and moves a month's lines to its first day, which a later bounded window may not load. So a re-run can count differently (phase 1 proves it) | + +Nothing is dropped: the 01-usage skill reads every run field (unknowns, "when there is nothing", bounding the past). + +## Readers + +| Reader | Change | +| --- | --- | +| `cli/src/contexts/telemetry/application/report/report-envelope.ts` | shape v2 | +| `cli/src/contexts/telemetry/application/report/report-usage-use-case.ts` | `coverage` and `run` apart | +| `cli/src/presentation/display/telemetry/telemetry-report-display.ts` | reads both, prints the same text | +| `plugins/aidd-telemetry/skills/01-usage` (`answer.md`, `references/envelope.md`) | version 2, `run.*` | +| `aidd_docs/product/usage-contract.md` | the envelope section, and every line naming a moved field | +| `cli/aidd_docs/memory/telemetry.md` | the envelope, figures vs run | +| `scripts/check-claude-tokens-end-to-end.cjs` | checks `version: 2`, reads `totals` and `rows` as before | +| `aidd telemetry explore` | none on `origin/next`, nothing to change | + +## Properties + +| # | Property | +| --- | --- | +| P1 | For any ledger, left-out entries, declarations, identity, period, axis and filters, the figures are byte-identical across: the order and grouping the calls and left-out entries were stored in, extra no-op stores, a second report right after, and `TZ` in `UTC`, `Pacific/Kiritimati`, `America/Adak`. `run` is exactly what the ingest and the load reported, never part of the figures | +| P2 | Through the built CLI, over one sandbox with real transcripts, the figures of the first report (which reads 2 transcripts) and of every later one are byte-identical across `TZ` ∈ {`UTC`, `Pacific/Kiritimati`, `America/Adak`} × `LC_ALL`/`LANG` ∈ {`C`, `fr_FR.UTF-8`, `ar_SA.UTF-8`}, on the `day` and `task` axes. `run.files_read` is 2, then 0 | + +## The time zone and locale guard (M9) + +- `cli/scripts/test-foreign-clock.mjs` (`pnpm test:foreign-clock`) runs the telemetry tests (`tests/contexts/telemetry`, `tests/e2e/telemetry-report`) under `TZ=Pacific/Kiritimati` and `LC_ALL=LANG=ar_SA.UTF-8`. First it checks that a child Node sees a UTC offset other than 0 and a locale that is not `en`, and fails if it does not. +- CI: one step in the existing `cli-coverage` job (ubuntu), after the suite. No new job, no matrix. +- Proven by hand: `dayOf` switched to local getters. The default suite stays green under that mutation and the guard goes red (`evidence.md`). + +## Acceptance + +| # | Criterion | Proof | +| --- | --- | --- | +| 1 | P1, P2 hold, red on `origin/next` for a behavioural reason | fast-check, e2e, `evidence.md` | +| 2 | The envelope test pins v2: `run` last, every figure field, refused v2 | unit test, red first | +| 3 | The guard runs locally and in CI, and goes red on a hand-made TZ dependence that the default suite misses | `evidence.md` | +| 4 | Every reader updated, and the text report unchanged | e2e text assertions untouched, scripts tests | +| 5 | `cd cli && pnpm test`, `node --test 'scripts/__tests__/**/*.test.js'`, `pnpm test:mutation:telemetry` ≥ 96 with changed-line survivors explained, `pnpm smoke`, both token end-to-end checks | `evidence.md` | +| 6 | Bundle within 709 KB, never weakened. A raise only per `check-bundle-size.mjs`, in its own `chore(cli):` commit | `evidence.md` | + +## Out of scope + +M10 (fallback key without `requestId`) and M7 (repository id case) change stored keys and get their own migration PR. + +## Limits + +- **Locale inside one process.** ICU fixes the locale when the process starts, so P1 varies `TZ` in-process and P2 varies the locale in child processes. The guard runs P1 and the rest under one foreign locale. +- **Worker threads.** Node applies a new `TZ` only on a process's main thread. Under Stryker, which runs Vitest on worker threads, P1 runs in the zone the thread started in. The forked suite and the guard still vary it. +- **Windows.** ICU takes its locale from the Windows API, not `LC_ALL`. The guard runs on ubuntu only, and no test asserts the locale. +- **Purity of the figures** assumes the ledger has settled the way it always has. A call with copies more than a day apart is still folded differently by a bounded report and an unbounded one. That rule is unchanged (`windowOf`). diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md new file mode 100644 index 000000000..19982701b --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md @@ -0,0 +1,89 @@ +# Evidence + +## Red runs, tests first, against `origin/next` code (`616dbbea`) + +### P1, `report-figures.property.integration.test.ts` + +The figures change with what the read before found (v1 keeps `oldest_transcript_at` among them): + +```txt +× are byte for byte the same whatever history stored the same calls, whatever the read before found, on a second report, and in any zone +- "oldest_transcript_at": null, ++ "oldest_transcript_at": "2026-09-01T10:00:00.000Z", +``` + +A second report over the same ledger counts fewer lines that are no record, because the first settled them (v1 keeps `skipped_ledger_lines` among the figures): + +```txt +× leave out the ledger lines that are no record, whose count changes once a report has settled them +- "skipped_ledger_lines": 2, ++ "skipped_ledger_lines": 1, +``` + +### P2, `telemetry-report-figures.e2e.test.ts` (built CLI) + +The second report, same sandbox, same zone and locale: + +```txt +AssertionError: day in UTC, C: expected '{\n "version": 1, … to be '{\n "version": 1, … +- "files_read": 2, ++ "files_read": 0, +- "unrecognised_shapes": 1, ++ "unrecognised_shapes": 0, +``` + +### Envelope shape, `report-envelope.unit.test.ts` + +```txt +× carries its version first, and the period and axis it answers +× states what the figures rest on, in snake case, and nothing about where the data lives +× puts what the read that printed it found apart, last, so the rest is the figures +× is versioned when nothing was read under AIDD_TELEMETRY=0 +Tests 4 failed | 3 passed (7) +``` + +The first version of the coverage test passed on v1: `toEqual` ignores `undefined` fields. It now uses `toStrictEqual`, and it fails. + +### `scripts/check-claude-tokens-end-to-end.cjs` + +```txt +✖ calls itself broken on a report of a version it does not read, figures right or not +``` + +## The guard is a guard + +Each mutation was made by hand, run once under the default conditions (`TZ=UTC LC_ALL=C`, as CI runs) and once under `pnpm test:foreign-clock`, then reverted (`git checkout --` the file). Default scope: `tests/contexts/telemetry tests/presentation/commands/telemetry tests/presentation/display/telemetry tests/e2e/telemetry`, the guard's own. + +| Mutation | Default (UTC, C) | Guard (UTC+14, ar-SA) | +| --- | --- | --- | +| none | `Tests 1497 passed` | `Tests 1497 passed`, 32 s | +| `stored-usage.ts` `dayOf`: the local day, `new Date(instant - new Date(instant).getTimezoneOffset() * 60_000).toISOString().slice(0, 10)` | `1 failed`: P1 only, which sets `TZ` itself. On `origin/next`, which has no P1, nothing catches it | `Tests 36 failed`, 7 files (ledger adapter, ingest e2e, report e2e…) | +| `telemetry-display.ts` `grouped`: `n.toLocaleString()` | `Tests 1497 passed`: English groups the way the code did | `Tests 24 failed`, 5 files (`1,200` printed as `١٬٢٠٠`) | + +The guard checks itself first: + +```txt +Telemetry tests under TZ=Pacific/Kiritimati, 840 min from UTC, locale ar-SA +``` + +On a platform that ignored `TZ` or `LC_ALL`, it would exit 1 before running anything. + +## Mutation run, first attempt + +The first `pnpm test:mutation:telemetry` stopped in Stryker's initial run: `UTC: expected -120 to be +0`. Stryker runs Vitest on worker threads, and Node applies a new `TZ` only on a process's main thread. The property's own precondition caught it instead of passing for nothing. Fix `25c9f857`: on a worker thread the property runs in the zone the thread started in. Confirmed with `vitest run --pool=threads` (red before, `Tests 2 passed` after) and `--pool=forks`. + +## Gates, at `25c9f857` + +| Gate | Result | +| --- | --- | +| `cd cli && pnpm test` | `Test Files 516 passed \| 1 skipped (517)`, `Tests 6744 passed \| 1 skipped (6745)`, first run green | +| `node --test 'scripts/__tests__/**/*.test.js'` | `tests 461`, `pass 461`, `fail 0` | +| `pnpm test:foreign-clock` | `Tests 1497 passed (1497)` under `TZ=Pacific/Kiritimati, 840 min from UTC, locale ar-SA` | +| `pnpm test:mutation:telemetry` | `score 96.7, floor 96`. 16 mutants on changed source lines (`report-envelope.ts`, `report-usage-use-case.ts`, `telemetry-report-display.ts`), every one killed or timed out: no survivor on a changed line | +| `pnpm smoke` (sandboxed `HOME`, `XDG_CONFIG_HOME`, `AIDD_*`, `CLAUDE_CONFIG_DIR`) | `PASS: 132 FAIL: 0 SKIP: 1`, coverage 100% | +| `node scripts/check-claude-tokens-end-to-end.cjs cli/dist/cli.js` | exit 0: `2 calls, 393 tokens, all on task demo, ticket T-1, and on task demo in this repository` | +| same, `--attended` | exit 0: `…; asked, answered by the hook, kept across /clear` | +| Bundle | `616dbbea` 712,722 B; this branch 712,764 B (+42 B), `696.1 KB / budget: 709 KB`. Not raised | +| `pnpm lint`, `pnpm typecheck`, type honesty, markdown links | green in the pre-commit hook of each commit | + +Not run here: Windows. The new tests use `join` for paths and rename no open file. `test-foreign-clock.mjs` spawns `node` directly, never a shell. diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-1.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-1.md new file mode 100644 index 000000000..4a6dbd1ce --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-1.md @@ -0,0 +1,17 @@ +--- +status: done +--- + +# Instruction: Tests first + +## Architecture projection + +```txt +cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts 🆕 P1 +cli/tests/e2e/telemetry-report-figures.e2e.test.ts 🆕 P2 +cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts ✏️ v2 shape +``` + +## Red + +Recorded in [`evidence.md`](./evidence.md), each against `origin/next` code. diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-2.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-2.md new file mode 100644 index 000000000..3a3c165f6 --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-2.md @@ -0,0 +1,13 @@ +--- +status: done +--- + +# Instruction: `coverage` and `run` apart + +## Architecture projection + +```txt +cli/src/contexts/telemetry/application/report/report-usage-use-case.ts ✏️ ReportCoverage {records, leftOut}, ReportRun +cli/src/contexts/telemetry/application/report/report-envelope.ts ✏️ version 2, run last, refused v2 +cli/src/presentation/display/telemetry/telemetry-report-display.ts ✏️ reads both, same text +``` diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md new file mode 100644 index 000000000..ac7bf99e1 --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md @@ -0,0 +1,17 @@ +--- +status: done +--- + +# Instruction: The foreign clock guard + +## Architecture projection + +```txt +cli/scripts/test-foreign-clock.mjs 🆕 checks the child sees a foreign zone and locale, then runs the telemetry tests under them +cli/package.json ✏️ test:foreign-clock +.github/workflows/cli-ci.yml ✏️ one step in cli-coverage +``` + +## Proof + +A hand mutation of `dayOf` to local getters: the default suite stays green, the guard goes red. See [`evidence.md`](./evidence.md). diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-4.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-4.md new file mode 100644 index 000000000..a63dd836d --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-4.md @@ -0,0 +1,20 @@ +--- +status: done +--- + +# Instruction: Readers, docs, gates + +## Architecture projection + +```txt +plugins/aidd-telemetry/skills/01-usage/actions/answer.md ✏️ version 2, run.* +plugins/aidd-telemetry/skills/01-usage/references/envelope.md ✏️ figures and run +aidd_docs/product/usage-contract.md ✏️ Report envelope, every moved field +cli/aidd_docs/memory/telemetry.md ✏️ envelope v2, the guard +scripts/check-claude-tokens-end-to-end.cjs ✏️ version 2 +scripts/__tests__/check-claude-tokens-end-to-end.test.js ✏️ fixtures +``` + +## Gates + +See [`evidence.md`](./evidence.md). diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/plan.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/plan.md new file mode 100644 index 000000000..f63ab2120 --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/plan.md @@ -0,0 +1,32 @@ +--- +objective: "Envelope v2: the figures of `aidd telemetry report --json` are byte-identical over the same data, whatever ran before, the clock's zone or the locale; the run's facts sit apart, in `run`." +status: implemented +--- + +# Plan: Report envelope v2 + +## Overview + +| Field | Value | +| --- | --- | +| **Goal** | M5 and M9 of the R1 telemetry audit closed | +| **Source** | [`contract.md`](./contract.md) | + +## Phases + +| # | Phase | File | +| --- | --- | --- | +| 1 | Tests first, red on `origin/next` | [`phase-1.md`](./phase-1.md) | +| 2 | `coverage` and `run` apart, envelope v2 | [`phase-2.md`](./phase-2.md) | +| 3 | The foreign clock guard, locally and in CI, proven by a mutation | [`phase-3.md`](./phase-3.md) | +| 4 | Readers, docs, gates | [`phase-4.md`](./phase-4.md) | + +## Decisions + +| Decision | Why | +| --- | --- | +| Figures stay at top level, `run` added last | one definition of the figures' bytes, the envelope without `run`, and readers of `rows`/`totals` keep their paths | +| `skipped_ledger_lines` and `consent_log_damaged` go to `run` | the first changes on a re-run (settling rewrites what it counted), the second describes whether this read could store | +| The split made in `ReportResult`, not only in the envelope | the names say what each part is wherever it is read; `leftOutLines` already takes exactly the run's fields | +| `TZ` varied in-process, the locale in child processes | Node re-reads `TZ` when it is set; ICU reads the locale once, at start | +| One guard step in the existing ubuntu test job | the lean option: no matrix, no extra install | diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md new file mode 100644 index 000000000..acc00262d --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md @@ -0,0 +1,3 @@ +# Review: Report envelope v2 + +Pending independent review. From 80293f4d0e3d7da291e2a8e81829b0ec62cbf5cb Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:24:03 +0200 Subject: [PATCH 05/13] docs(telemetry): say what the guard proof observed, and name the run The contract and phase 3 claimed the default suite misses the zone mutation. With P1 in place it does not: one test fails. What was observed is now on record: on 616dbbea the same mutation passes all 1493 tests in scope under UTC, while the guard fails 36. The locale mutation passes the default suite and fails 24 under the guard. The guard's scope is listed in full, and a comment in attribution.ts now points to `run.not_stored`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .../2026_10/2026_10_17_report-envelope-v2/contract.md | 8 +++++--- .../2026_10/2026_10_17_report-envelope-v2/evidence.md | 4 +++- .../2026_10/2026_10_17_report-envelope-v2/phase-3.md | 2 +- cli/src/contexts/telemetry/domain/report/attribution.ts | 2 +- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md index b43d32a6b..e74372197 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md @@ -74,9 +74,11 @@ Nothing is dropped: the 01-usage skill reads every run field (unknowns, "when th ## The time zone and locale guard (M9) -- `cli/scripts/test-foreign-clock.mjs` (`pnpm test:foreign-clock`) runs the telemetry tests (`tests/contexts/telemetry`, `tests/e2e/telemetry-report`) under `TZ=Pacific/Kiritimati` and `LC_ALL=LANG=ar_SA.UTF-8`. First it checks that a child Node sees a UTC offset other than 0 and a locale that is not `en`, and fails if it does not. +- `cli/scripts/test-foreign-clock.mjs` (`pnpm test:foreign-clock`) runs the telemetry tests (`tests/contexts/telemetry`, `tests/presentation/commands/telemetry`, `tests/presentation/display/telemetry`, `tests/e2e/telemetry`) under `TZ=Pacific/Kiritimati` and `LC_ALL=LANG=ar_SA.UTF-8`. First it checks that a child Node sees a UTC offset other than 0 and a locale that is not `en`, and fails if it does not. - CI: one step in the existing `cli-coverage` job (ubuntu), after the suite. No new job, no matrix. -- Proven by hand: `dayOf` switched to local getters. The default suite stays green under that mutation and the guard goes red (`evidence.md`). +- Proven by hand (`evidence.md`): + - the ledger's day read in local time: under UTC it changes nothing, so on `origin/next` the default suite passes it, and with P1 one test fails. The guard fails 36; + - counts grouped by the default locale: the default suite passes it, and the guard fails 24. ## Acceptance @@ -84,7 +86,7 @@ Nothing is dropped: the 01-usage skill reads every run field (unknowns, "when th | --- | --- | --- | | 1 | P1, P2 hold, red on `origin/next` for a behavioural reason | fast-check, e2e, `evidence.md` | | 2 | The envelope test pins v2: `run` last, every figure field, refused v2 | unit test, red first | -| 3 | The guard runs locally and in CI, and goes red on a hand-made TZ dependence that the default suite misses | `evidence.md` | +| 3 | The guard runs locally and in CI, and goes red on a hand-made dependence on the local zone, and on one on the locale that the default suite misses | `evidence.md` | | 4 | Every reader updated, and the text report unchanged | e2e text assertions untouched, scripts tests | | 5 | `cd cli && pnpm test`, `node --test 'scripts/__tests__/**/*.test.js'`, `pnpm test:mutation:telemetry` ≥ 96 with changed-line survivors explained, `pnpm smoke`, both token end-to-end checks | `evidence.md` | | 6 | Bundle within 709 KB, never weakened. A raise only per `check-bundle-size.mjs`, in its own `chore(cli):` commit | `evidence.md` | diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md index 19982701b..0ec674a57 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md @@ -57,9 +57,11 @@ Each mutation was made by hand, run once under the default conditions (`TZ=UTC L | Mutation | Default (UTC, C) | Guard (UTC+14, ar-SA) | | --- | --- | --- | | none | `Tests 1497 passed` | `Tests 1497 passed`, 32 s | -| `stored-usage.ts` `dayOf`: the local day, `new Date(instant - new Date(instant).getTimezoneOffset() * 60_000).toISOString().slice(0, 10)` | `1 failed`: P1 only, which sets `TZ` itself. On `origin/next`, which has no P1, nothing catches it | `Tests 36 failed`, 7 files (ledger adapter, ingest e2e, report e2e…) | +| `stored-usage.ts` `dayOf`: the local day, `new Date(instant - new Date(instant).getTimezoneOffset() * 60_000).toISOString().slice(0, 10)` | `1 failed`: P1 only, which sets `TZ` itself. The same mutation on `616dbbea`, same scope, `TZ=UTC LC_ALL=C`: `Tests 1493 passed (1493)` | `Tests 36 failed`, 7 files (ledger adapter, ingest e2e, report e2e…) | | `telemetry-display.ts` `grouped`: `n.toLocaleString()` | `Tests 1497 passed`: English groups the way the code did | `Tests 24 failed`, 5 files (`1,200` printed as `١٬٢٠٠`) | +A first TZ variant, built from `getFullYear`/`getMonth`/`getDate`, also broke two year-edge tests of `dayOf` under UTC, which has nothing to do with the zone. So the offset variant above is the one on record. + The guard checks itself first: ```txt diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md index ac7bf99e1..e7bf48826 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/phase-3.md @@ -14,4 +14,4 @@ cli/package.json ✏️ test:foreign-clock ## Proof -A hand mutation of `dayOf` to local getters: the default suite stays green, the guard goes red. See [`evidence.md`](./evidence.md). +Two hand mutations: the ledger's day read in local time (the guard fails 36 tests; on `origin/next` the default suite passes it), and counts grouped by the default locale (the default suite passes it, the guard fails 24). See [`evidence.md`](./evidence.md). diff --git a/cli/src/contexts/telemetry/domain/report/attribution.ts b/cli/src/contexts/telemetry/domain/report/attribution.ts index 990d5b3ec..d19a75f43 100644 --- a/cli/src/contexts/telemetry/domain/report/attribution.ts +++ b/cli/src/contexts/telemetry/domain/report/attribution.ts @@ -3,7 +3,7 @@ import { resolveBinding } from "../declaration/binding-resolution.js"; import type { SessionCarry, SessionDeclaration } from "../declaration/task-declaration.js"; /** Why a billed call has no task. A call from outside any repository, or from a directory never - * seen alive, is never stored, so it reaches a report only as a count in `coverage.not_stored` + * seen alive, is never stored, so it reaches a report only as a count in `run.not_stored` * and is not a reason here. */ export const UNATTRIBUTED_REASONS = ["no-binding", "declared-none"] as const; export type UnattributedReason = (typeof UNATTRIBUTED_REASONS)[number]; From 425f80efea926e0939d5bfb3735d64d2ab287f2c Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:45:04 +0200 Subject: [PATCH 06/13] fix(cli): the foreign clock guard checks how the locale writes, not its name A locale ICU has no data for keeps its name and formats as English, so LC_ALL=xx_BOGUS.UTF-8 passed the old check while the tests ran in English. The check, now in scripts/foreign-clock.mjs, has a child Node write a number and a date and requires both to differ from en-US, and the offset to differ from 0. A bogus locale or zone makes the guard exit 1 before any test runs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- cli/scripts/foreign-clock.d.mts | 1 + cli/scripts/foreign-clock.mjs | 25 +++++++++++++++ cli/scripts/test-foreign-clock.mjs | 17 +++------- .../scripts/foreign-clock.integration.test.ts | 32 +++++++++++++++++++ 4 files changed, 63 insertions(+), 12 deletions(-) create mode 100644 cli/scripts/foreign-clock.d.mts create mode 100644 cli/scripts/foreign-clock.mjs create mode 100644 cli/tests/scripts/foreign-clock.integration.test.ts diff --git a/cli/scripts/foreign-clock.d.mts b/cli/scripts/foreign-clock.d.mts new file mode 100644 index 000000000..36ce9aa74 --- /dev/null +++ b/cli/scripts/foreign-clock.d.mts @@ -0,0 +1 @@ +export function foreignClockProblem(env: NodeJS.ProcessEnv): string | null; diff --git a/cli/scripts/foreign-clock.mjs b/cli/scripts/foreign-clock.mjs new file mode 100644 index 000000000..17e3b8e83 --- /dev/null +++ b/cli/scripts/foreign-clock.mjs @@ -0,0 +1,25 @@ +import { spawnSync } from "node:child_process"; + +/* What a child Node writes for one instant and one number, in its own zone and locale and in + * English: a locale is foreign only if it writes them otherwise, whatever it is called. A locale + * ICU has no data for keeps its name and formats as English. */ +const PROBE = ` +const at = new Date("2026-10-01T12:00:00Z"); +const n = 1234567.5; +process.stdout.write(JSON.stringify({ + offset: at.getTimezoneOffset(), + local: [n.toLocaleString(), at.toLocaleDateString()], + english: [n.toLocaleString("en-US"), at.toLocaleDateString("en-US")], +}));`; + +/** Why a child Node run with `env` would not see a foreign clock and locale, or `null` when it + * would. */ +export function foreignClockProblem(env) { + const probe = spawnSync(process.execPath, ["-e", PROBE], { env, encoding: "utf8" }); + if (probe.status !== 0) return `the probe failed: ${probe.stderr.trim()}`; + const { offset, local, english } = JSON.parse(probe.stdout); + if (offset === 0) return "a child Node runs in UTC"; + const same = local.filter((text, n) => text === english[n]); + if (same.length > 0) return `a child Node formats as English: ${same.join(", ")}`; + return null; +} diff --git a/cli/scripts/test-foreign-clock.mjs b/cli/scripts/test-foreign-clock.mjs index 8a2a26785..e86c2f827 100644 --- a/cli/scripts/test-foreign-clock.mjs +++ b/cli/scripts/test-foreign-clock.mjs @@ -6,6 +6,7 @@ import { spawnSync } from "node:child_process"; import { dirname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; +import { foreignClockProblem } from "./foreign-clock.mjs"; const CLI_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); /** UTC+14, the farthest ahead: a call before 10:00 UTC is on the next local day. */ @@ -22,22 +23,14 @@ const TESTS = [ const env = { ...process.env, TZ: ZONE, LC_ALL: LOCALE, LANG: LOCALE }; // A zone or locale the platform ignores would run the tests in UTC and English, and pass. -const probe = spawnSync( - process.execPath, - [ - "-p", - 'JSON.stringify([new Date("2026-10-01T12:00:00Z").getTimezoneOffset(), Intl.DateTimeFormat().resolvedOptions().locale])', - ], - { env, encoding: "utf8" } -); -const [offset, locale] = probe.status === 0 ? JSON.parse(probe.stdout) : [0, "unknown"]; -if (offset === 0 || locale.startsWith("en")) { +const problem = foreignClockProblem(env); +if (problem !== null) { console.error( - `A child Node sees UTC offset ${-offset} min and locale ${locale} under TZ=${ZONE} LC_ALL=${LOCALE}: this guard would prove nothing here.` + `Under TZ=${ZONE} LC_ALL=${LOCALE}, ${problem}: this guard would prove nothing here.` ); process.exit(1); } -console.log(`Telemetry tests under TZ=${ZONE}, ${-offset} min from UTC, locale ${locale}`); +console.log(`Telemetry tests under TZ=${ZONE} and LC_ALL=${LOCALE}, both seen by Node`); const run = spawnSync( process.execPath, diff --git a/cli/tests/scripts/foreign-clock.integration.test.ts b/cli/tests/scripts/foreign-clock.integration.test.ts new file mode 100644 index 000000000..b49381d3a --- /dev/null +++ b/cli/tests/scripts/foreign-clock.integration.test.ts @@ -0,0 +1,32 @@ +import { describe, expect, it } from "vitest"; +import { foreignClockProblem } from "../../scripts/foreign-clock.mjs"; + +/** The environment `pnpm test:foreign-clock` runs the telemetry tests in, with one part swapped. */ +const guarded = (zone: string, locale: string): NodeJS.ProcessEnv => ({ + ...process.env, + TZ: zone, + LC_ALL: locale, + LANG: locale, +}); + +describe("the foreign clock guard's own check", () => { + it("accepts a zone far from UTC and a locale that writes numbers and dates otherwise", () => { + expect(foreignClockProblem(guarded("Pacific/Kiritimati", "ar_SA.UTF-8"))).toBeNull(); + }); + + it("refuses a locale ICU has no data for, which formats as English whatever its name", () => { + expect(foreignClockProblem(guarded("Pacific/Kiritimati", "xx_BOGUS.UTF-8"))).toMatch( + /formats as English/ + ); + }); + + it("refuses English", () => { + expect(foreignClockProblem(guarded("Pacific/Kiritimati", "en_US.UTF-8"))).toMatch( + /formats as English/ + ); + }); + + it("refuses a zone the platform does not know, which runs in UTC", () => { + expect(foreignClockProblem(guarded("Not/AZone", "ar_SA.UTF-8"))).toMatch(/UTC/); + }); +}); From e5ac7e60ae82e16ae18022f5bb47d34704acc359 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:45:10 +0200 Subject: [PATCH 07/13] test(telemetry): build the envelope in the zone it is compared in P1 built the envelope after restoring TZ, so a zone dependence in report-envelope.ts escaped it. The envelope and the figures' bytes are now made inside the zone. The ledger also starts with monthly partitions, holding calls and lines that are no record, and the second history is stored with the clock in another zone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- ...eport-figures.property.integration.test.ts | 98 ++++++++++++------- 1 file changed, 60 insertions(+), 38 deletions(-) diff --git a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts index 99c3404e5..daf70a9d4 100644 --- a/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts @@ -140,10 +140,19 @@ const scenario = fc.record({ }), { maxLength: 5 } ), - // Lines no version reads, already on disk, some twice: settling keeps each once. - damaged: fc.array(fc.record({ day: fc.constantFrom(...DAYS), text: fc.constantFrom("x", "y") }), { - maxLength: 3, - }), + // Lines no version reads, already on disk, some twice, some in a month an earlier version + // kept in one file: settling keeps each once, in its day or its month's first day. + damaged: fc.array( + fc.record({ + day: fc.constantFrom(...DAYS), + text: fc.constantFrom("x", "y"), + inMonth: fc.boolean(), + }), + { maxLength: 3 } + ), + // Snapshots an earlier version already wrote, in their month's single file: both histories + // start from them, and the first read that reaches the month splits it into days. + inMonth: fc.array(fc.boolean(), { minLength: 40, maxLength: 40 }), request: fc.record({ axis: fc.constantFrom(...REPORT_AXES), period, filters }), // How another history reached the same data: each snapshot and left-out entry in a batch of // its own choosing, the batches in any order, and stores that bring nothing in between. @@ -184,14 +193,20 @@ function recordsOf(calls: Scenario["calls"]): StoredUsage[] { ); } -/** A ledger directory holding the damaged lines, as an earlier version or a hand edit left. */ -async function ledgerWith(damaged: Scenario["damaged"]): Promise { +/** A ledger directory as an earlier version or a hand edit left it: records in their month's + * single file, and lines that are no record in a day's file or a month's. */ +async function ledgerWith( + damaged: readonly { day: string; text: string; inMonth?: boolean }[], + monthly: readonly StoredUsage[] = [] +): Promise { ledgers += 1; const dir = join(root, `ledger-${ledgers}`); await mkdir(dir, { recursive: true }); - const byDay = new Map(); - for (const { day, text } of damaged) byDay.set(day, `${byDay.get(day) ?? ""}${text}\n`); - for (const [day, text] of byDay) await writeFile(join(dir, `${day}.jsonl`), text); + const files = new Map(); + const put = (name: string, line: string) => files.set(name, `${files.get(name) ?? ""}${line}\n`); + for (const record of monthly) put(record.at.slice(0, 7), JSON.stringify(record)); + for (const { day, text, inMonth } of damaged) put(inMonth ? day.slice(0, 7) : day, text); + for (const [name, text] of files) await writeFile(join(dir, `${name}.jsonl`), text); return new UsageLedgerAdapter(dir, new PrivateStorageAdapter()); } @@ -228,7 +243,8 @@ async function inZone(zone: Zone, work: () => Promise): Promise { } } -/** Reports once over the ledger, after a read that found `facts`, with the clock in `zone`. */ +/** Reports once over the ledger, after a read that found `facts`, with the clock in `zone`: + * the envelope, and the figures' bytes, are made in that zone too. */ async function report( held: Scenario, ledger: UsageLedgerAdapter, @@ -252,21 +268,22 @@ async function report( const ingest: Pick = { execute: async () => ingestResultOf(facts), }; - const result = await inZone(zone, () => - new ReportUsageUseCase( + return inZone(zone, async () => { + const result = await new ReportUsageUseCase( ingest, ledger, new DeclaredBindings(sessions, new InMemorySnapshots()), identity, new FakeLocator(), { refusedByEnvironment: false } - ).execute(request) - ); - return result.status === "reported" ? reportEnvelopeOf(result) : result; + ).execute(request); + const envelope = result.status === "reported" ? reportEnvelopeOf(result) : result; + return { envelope, figures: figuresOf(envelope) }; + }); } /** The figures' bytes: the envelope as printed, but its `run`. */ -function figuresOf(envelope: Awaited>): string { +function figuresOf(envelope: object): string { const { run: _run, ...figures } = envelope as Record; return JSON.stringify(figures, null, 2); } @@ -275,46 +292,51 @@ describe("the figures of a report", () => { it("are byte for byte the same whatever history stored the same calls, whatever the read before found, on a second report, and in any zone", async () => { await fc.assert( fc.asyncProperty(scenario, async (held) => { - const records = recordsOf(held.calls); + const all = recordsOf(held.calls); + const monthly = all.filter((_, n) => held.inMonth[n % held.inMonth.length]); + const records = all.filter((_, n) => !held.inMonth[n % held.inMonth.length]); const leftOut: LeftOutEntry[] = held.leftOut; // One history: every call and left-out entry stored at once, reported in UTC. - const once = await ledgerWith(held.damaged); + const once = await ledgerWith(held.damaged, monthly); await once.exclusively(() => storeCalls(once, records)); await once.leftOut.add(leftOut); const reference = await report(held, once, held.request, held.runs[0], "UTC"); - // Another: in batches, in another order, with stores that bring nothing. - const batched = await ledgerWith(held.damaged); + // Another: in batches, in another order, with stores that bring nothing, stored with + // the clock in another zone. + const batched = await ledgerWith(held.damaged, monthly); const { batchOf, order, noOps } = held.history; const nth = (n: number) => batchOf[n % batchOf.length] as number; - for (const batch of order) { - const calls = records.filter((_, n) => nth(n) === batch); - await batched.exclusively(() => storeCalls(batched, calls)); - await batched.leftOut.add(leftOut.filter((_, n) => nth(n + 7) === batch)); - if (batch < noOps) { - await batched.exclusively(() => storeCalls(batched, [])); - await batched.leftOut.add([]); - } - } const [first, second] = held.zones; + await inZone(first, async () => { + for (const batch of order) { + const calls = records.filter((_, n) => nth(n) === batch); + await batched.exclusively(() => storeCalls(batched, calls)); + await batched.leftOut.add(leftOut.filter((_, n) => nth(n + 7) === batch)); + if (batch < noOps) { + await batched.exclusively(() => storeCalls(batched, [])); + await batched.leftOut.add([]); + } + } + }); const later = await report(held, batched, held.request, held.runs[1], first); const again = await report(held, batched, held.request, held.runs[2], second); - expect(figuresOf(later)).toBe(figuresOf(reference)); - expect(figuresOf(again)).toBe(figuresOf(reference)); - if (!("run" in again)) return; + expect(later.figures).toBe(reference.figures); + expect(again.figures).toBe(reference.figures); + if (!("run" in again.envelope)) return; // What the read found is in `run`, and only there. const facts = held.runs[2]; - expect(again.run).toEqual({ + expect(again.envelope.run).toEqual({ files_read: facts.filesRead, unrecognised_shapes: facts.unrecognised, not_stored: ingestResultOf(facts).notStored, consent_log_damaged: facts.consentLogDamaged, oldest_transcript_at: facts.oldestTranscriptAt, - skipped_ledger_lines: again.run.skipped_ledger_lines, + skipped_ledger_lines: again.envelope.run.skipped_ledger_lines, }); - expect(Object.keys(again).at(-1)).toBe("run"); + expect(Object.keys(again.envelope).at(-1)).toBe("run"); }), { numRuns: 60 } ); @@ -339,9 +361,9 @@ describe("the figures of a report", () => { }; const first = await report(held, ledger, request, quiet, "UTC"); const second = await report(held, ledger, request, quiet, "UTC"); - expect(figuresOf(second)).toBe(figuresOf(first)); + expect(second.figures).toBe(first.figures); // The rewrite kept the line once: the same ledger, read again, counts one fewer. - expect(first).toMatchObject({ run: { skipped_ledger_lines: 2 } }); - expect(second).toMatchObject({ run: { skipped_ledger_lines: 1 } }); + expect(first.envelope).toMatchObject({ run: { skipped_ledger_lines: 2 } }); + expect(second.envelope).toMatchObject({ run: { skipped_ledger_lines: 1 } }); }); }); From dbc358e88512123c2d89e71124787bf4c0f9f5f4 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:45:17 +0200 Subject: [PATCH 08/13] feat(telemetry): envelope v2 states unknown records once, in the totals The top-level unknown_records copied totals.total.unknown_records. The copy is dropped from v2 before any release, and readers use the totals. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .../application/report/report-envelope.ts | 39 +++---------------- .../report/report-envelope.unit.test.ts | 5 +-- cli/tests/e2e/telemetry-journey.e2e.test.ts | 1 - cli/tests/e2e/telemetry-report-sandbox.ts | 1 - cli/tests/e2e/telemetry-report.e2e.test.ts | 2 +- 5 files changed, 8 insertions(+), 40 deletions(-) diff --git a/cli/src/contexts/telemetry/application/report/report-envelope.ts b/cli/src/contexts/telemetry/application/report/report-envelope.ts index 591851178..f6f53fd42 100644 --- a/cli/src/contexts/telemetry/application/report/report-envelope.ts +++ b/cli/src/contexts/telemetry/application/report/report-envelope.ts @@ -39,38 +39,11 @@ export interface EnvelopeRow extends EnvelopeTally { readonly reason: UnattributedReason | null; } -/** What `aidd telemetry report --json` prints. Version 2. - * - * Every field but `run` is a figure: a function of the ledger (its calls and what was left out - * of it), the declarations, the chosen identity, the period, the filters and the axis, alone. - * Two reports over the same of those print them byte for byte the same, whatever ingest ran - * before, in any time zone and locale. `run`, last, is what the read that printed the envelope - * found, and may differ between two such reports. - * - * `rows` split the period along `axis`; the rows of every axis add up to the same `totals`, - * counter by counter. `unknown_records` is how many calls had at least one counter unknown. - * `coverage.records` counts every call of the period, matching the filters or not, and - * `coverage.left_out` what every ingest so far, whoever ran it (the plugin's included), - * measured and could not store, of a clone that consents: `undated` calls and `unrecognised` - * lines that are no usage line, each once; none has a day, so every report counts all of them. - * - * `run`: `files_read` transcripts this read opened, `unrecognised_shapes` and `not_stored` (by - * reason) what it met and did not store, `consent_log_damaged` true while - * `ledger/consents.jsonl` cannot be trusted and nothing is stored, `oldest_transcript_at` when - * the oldest transcript still on disk was last written (`null` when there is none), and - * `skipped_ledger_lines` the lines that were not records, kept as found, in the ledger - * partitions it read before settling them: a bounded report reads its period and the day on - * each side. Nothing here is a path, a working directory or a branch name. - * - * `filters` names what the report was narrowed to, every flag in a fixed order, `null` when it - * was not given, else as resolved: the full id for a start or `.`, the start as typed when it - * names no call of the period. `rows` and `totals` count the calls that match. - * - * Under `AIDD_TELEMETRY=0` nothing is read and the envelope is `{ "version": 2, "refused": - * "AIDD_TELEMETRY=0" }` (`REFUSED_ENVELOPE`). - * - * A new field may appear in a version 2 envelope, a figure outside `run`, a fact of the read in - * it; a field is never renamed, retyped, moved or removed without a new `version`. */ +/** What `aidd telemetry report --json` prints, version 2: every field but `run` is a figure, + * `run` is what the read that printed it found. What each field means, and what the figures + * depend on, is defined once, in `plugins/aidd-telemetry/skills/01-usage/references/envelope.md`. + * Under `AIDD_TELEMETRY=0` nothing is read: `REFUSED_ENVELOPE`. A field is never renamed, + * retyped, moved or removed without a new `version`. */ export interface ReportEnvelope { readonly version: 2; readonly period: Period; @@ -78,7 +51,6 @@ export interface ReportEnvelope { readonly filters: ReportFilters; readonly rows: readonly EnvelopeRow[]; readonly totals: EnvelopeTally; - readonly unknown_records: number; readonly coverage: { readonly records: number; readonly left_out: Readonly>; @@ -127,7 +99,6 @@ export function reportEnvelopeOf( filters: filtersOf(filters), rows: report.rows.map(rowOf), totals: tallyOf(report.totals), - unknown_records: report.totals.total.unknownRecords, coverage: { records: coverage.records, left_out: coverage.leftOut }, run: { files_read: run.filesRead, diff --git a/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts b/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts index 559612ebd..15af19551 100644 --- a/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts +++ b/cli/tests/contexts/telemetry/application/report/report-envelope.unit.test.ts @@ -87,13 +87,13 @@ describe("the JSON envelope of a report", () => { expect(rows[2]).toMatchObject({ kind: "absent", key: null, reason: null }); }); - it("states the totals and how many records had an unknown counter", () => { + it("states how many records had an unknown counter once, in the totals", () => { const envelope = reportEnvelopeOf(reported); expect(envelope.totals).toMatchObject({ records: 4, total: { tokens: 10, unknown_records: 1 }, }); - expect(envelope.unknown_records).toBe(1); + expect(envelope).not.toHaveProperty("unknown_records"); }); it("states what the figures rest on, in snake case, and nothing about where the data lives", () => { @@ -113,7 +113,6 @@ describe("the JSON envelope of a report", () => { "filters", "rows", "totals", - "unknown_records", "coverage", "run", ]); diff --git a/cli/tests/e2e/telemetry-journey.e2e.test.ts b/cli/tests/e2e/telemetry-journey.e2e.test.ts index e80c7faf1..e7cbd924f 100644 --- a/cli/tests/e2e/telemetry-journey.e2e.test.ts +++ b/cli/tests/e2e/telemetry-journey.e2e.test.ts @@ -280,7 +280,6 @@ interface Envelope { axis: string; rows: Row[]; totals: Row; - unknown_records: number; } async function report(axis: string, from: string, to: string): Promise { diff --git a/cli/tests/e2e/telemetry-report-sandbox.ts b/cli/tests/e2e/telemetry-report-sandbox.ts index ae662dff2..772c3e519 100644 --- a/cli/tests/e2e/telemetry-report-sandbox.ts +++ b/cli/tests/e2e/telemetry-report-sandbox.ts @@ -328,7 +328,6 @@ export interface Envelope { filters: Record; rows: Row[]; totals: Row; - unknown_records: number; coverage: { records: number }; run: { files_read: number; oldest_transcript_at: string | null }; } diff --git a/cli/tests/e2e/telemetry-report.e2e.test.ts b/cli/tests/e2e/telemetry-report.e2e.test.ts index 9d862db5b..60f03c3a9 100644 --- a/cli/tests/e2e/telemetry-report.e2e.test.ts +++ b/cli/tests/e2e/telemetry-report.e2e.test.ts @@ -46,7 +46,7 @@ describe("aidd telemetry report", () => { expect(envelope.rows.reduce((n, row) => n + row.total.tokens, 0)).toBe( envelope.totals.total.tokens ); - expect(envelope.unknown_records).toBe(1); + expect(envelope.totals.total.unknown_records).toBe(1); expect(envelope.totals.cache_read.unknown_records).toBe(1); sums[axis] = envelope.totals.total.tokens; } From 092aac564fb309f115265a8dc9368fb8456df87b Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:46:06 +0200 Subject: [PATCH 09/13] docs(telemetry): one definition of the figures, and a fix named on a mismatch The skill's envelope reference now defines, once, what a figure depends on (ledger records, left-out file, declarations, identity, period, filters, axis) and what run holds. The usage contract and the CLI memory link to it. On an envelope version it does not read, the 01-usage skill names the fix: update the CLI below 2, the plugin above 2. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- aidd_docs/product/usage-contract.md | 5 ++--- cli/aidd_docs/memory/telemetry.md | 2 +- plugins/aidd-telemetry/skills/01-usage/actions/answer.md | 6 ++++-- .../skills/01-usage/references/envelope.md | 9 ++++++--- 4 files changed, 13 insertions(+), 9 deletions(-) diff --git a/aidd_docs/product/usage-contract.md b/aidd_docs/product/usage-contract.md index b9a2c05cc..f7823674f 100644 --- a/aidd_docs/product/usage-contract.md +++ b/aidd_docs/product/usage-contract.md @@ -98,7 +98,7 @@ What an ingest measured and could not store is kept in `ledger/left-out.txt`, so ## Report envelope -`aidd telemetry report --json` prints one object, `version: 2`. A field may be added to version 2, a figure outside `run`, a fact of the read inside it; never renamed, retyped, moved or removed without a new `version`. Type: `ReportEnvelope`, `cli/src/contexts/telemetry/application/report/report-envelope.ts`. +`aidd telemetry report --json` prints one object, `version: 2`. What a figure is, what it depends on, what `run` holds, and how the envelope may change are defined once, in [the envelope reference](../../plugins/aidd-telemetry/skills/01-usage/references/envelope.md#figures-and-run). Type: `ReportEnvelope`, `cli/src/contexts/telemetry/application/report/report-envelope.ts`. | Field | Meaning | | --- | --- | @@ -108,7 +108,6 @@ What an ingest measured and could not store is kept in `ledger/left-out.txt`, so | `filters` | `task`, `ticket`, `person`, `model`, `agent`, `session`, `repository`, `absent`, `unattributed`, in that order: `null` when not given, else as resolved (see [Filters](#filters)) | | `rows` | one per axis value, of the calls that match | | `totals` | the calls of the period that match | -| `unknown_records` | calls with at least one counter unknown | | `coverage` | `records`, `left_out`. `records` counts the calls of the period, matching or not; `left_out` (`undated`, `unrecognised`) what every ingest so far measured and could not store, see [Left out](#left-out) | | `run` | last: `files_read`, `unrecognised_shapes`, `not_stored`, `consent_log_damaged`, `oldest_transcript_at`, `skipped_ledger_lines`, what the read that printed the report found | @@ -121,7 +120,7 @@ Row and `totals`: `records`, `input`, `output`, `cache_read`, `cache_write` (eac | `reason` | `no-binding` or `declared-none` for `unattributed`, else `null` | - Every axis partitions the same calls: each adds up to `totals`, counter by counter. -- **The figures** are every field but `run`: a function of the ledger (its calls and `left-out.txt`), the declarations, the chosen identity, the period, the filters and the axis. Two reports over the same of those print them byte for byte the same, whatever ingests ran before and in whatever order, run again or not, in any time zone and locale. Envelope version 1 mixed the two; version 2 has no compatibility shape. +- Envelope version 1 put the figures and the run together. Version 2 keeps them apart and has no compatibility shape. - `AIDD_TELEMETRY=0`: nothing read, object is `{"version": 2, "refused": "AIDD_TELEMETRY=0"}`. - The text report labels an `absent` person `anonymous`; the envelope keeps `kind: absent`, `key: null`. - A report ingests first. `run` describes that read: what it opened, met and left out, whether the consent log let it store, how old the transcripts on disk are, and the lines that are not records in the partitions it loaded, counted before it settled them (a rewrite keeps each once, so the next report can count fewer). `coverage.records` describes the period; `coverage.left_out` every ingest so far. diff --git a/cli/aidd_docs/memory/telemetry.md b/cli/aidd_docs/memory/telemetry.md index c98c6e8c6..33c9be313 100644 --- a/cli/aidd_docs/memory/telemetry.md +++ b/cli/aidd_docs/memory/telemetry.md @@ -31,7 +31,7 @@ flowchart LR - A command that reports or ingests reads new lines first, under the ledger lock. `ingest` is the same step on its own, run quietly by the plugin at session start. - A transcript is read a block at a time (1 MiB), each complete line handed on as it is found, never the unread tail whole. An ingest holds a block, up to about three copies of the longest line at once (its pieces, the joined bytes, the decoded string), and what the reading keeps: every record and every unrecognised line naming a directory. An unterminated last line waits for the next read. A transcript that shrank while it was read counts none of that read and keeps its position, to be read again. One replaced at its path meanwhile is read as it was opened, whole. -- A report's envelope keeps its figures apart from its `run`. The figures are every field but `run`, a function of the ledger, the declarations, the identity and what was asked, alone: byte for byte the same whatever ran before, in any time zone and locale. `run` is what the read that printed it found. +- A report's envelope keeps its figures apart from its `run`: defined once in [the envelope reference](../../../plugins/aidd-telemetry/skills/01-usage/references/envelope.md#figures-and-run). - Reading is one pass: tool-specific only up to the usage record. Resolving, storing, attributing and reporting are written once for every tool. - The hooks and the CLI never call each other's code. They meet in the files, whose formats one fixture pins on both sides. diff --git a/plugins/aidd-telemetry/skills/01-usage/actions/answer.md b/plugins/aidd-telemetry/skills/01-usage/actions/answer.md index 8e6228a6c..254cf928c 100644 --- a/plugins/aidd-telemetry/skills/01-usage/actions/answer.md +++ b/plugins/aidd-telemetry/skills/01-usage/actions/answer.md @@ -16,11 +16,11 @@ The figures the report holds for the question, each counter apart, followed by e - A non-zero exit: show its message and stop. 2. **Check.** Read [the envelope reference](../references/envelope.md) for the fields. - `refused` present: measurement is refused by the environment. Say so and give no figure. - - `version` other than `2`: say the report has a shape this skill does not know, and give no figure. + - `version` other than `2`: the CLI and this plugin read different envelope versions. Give no figure, and name the fix: below `2`, update the CLI (`@ai-driven-dev/cli`); above `2`, update the `aidd-telemetry` plugin, so both are on the same version. 3. **State.** Give the rows and the `totals` as the report holds them: input, output, cache read, cache write and `total`, each with its token count. Name the period, the axis, and each filter as `filters` prints it. 4. **Say the unknowns.** Say each of these that applies, in this order: - a counter whose `unknown_records` is above `0`: its figure is a lower bound, and that many calls did not report it; - - `unknown_records` on the totals above `0`: the same for the total; + - `totals.total.unknown_records` above `0`: the same for the total; - a row of kind `unattributed`: its `reason`, with its meaning from the reference; - a row of kind `absent`: that the axis had no value for those calls; - `run.unrecognised_shapes`, `run.skipped_ledger_lines`, any `run.not_stored` entry or any `coverage.left_out` entry above `0`: that some lines were read and left out, with the counts; @@ -36,4 +36,6 @@ The figures the report holds for the question, each counter apart, followed by e | A report with `unknown_records` above `0` | The answer says the figure is a lower bound, and never shows `0` for it | | A report with a `no-binding` row | The answer names the row and what it means | | A `refused` envelope | No figure is given | +| An envelope of `version` `1` | No figure; the answer says to update the CLI so it prints version `2` | +| An envelope of `version` `3` | No figure; the answer says to update the `aidd-telemetry` plugin so it reads that version | | Every figure in the answer | It appears in the JSON; none was computed here | diff --git a/plugins/aidd-telemetry/skills/01-usage/references/envelope.md b/plugins/aidd-telemetry/skills/01-usage/references/envelope.md index 31b4e4f7c..561f0a4bf 100644 --- a/plugins/aidd-telemetry/skills/01-usage/references/envelope.md +++ b/plugins/aidd-telemetry/skills/01-usage/references/envelope.md @@ -2,7 +2,11 @@ What `aidd telemetry report --json` prints, and what each part means. Version `2`. -Every field but `run` is a figure: two reports over the same stored calls, declarations, period and filters print them byte for byte the same, whatever ran before, in any time zone or locale. `run` is what the read that printed the report found, and may differ from one report to the next. +## Figures and run + +Every field but `run` is a figure. The figures depend on these inputs and nothing else: the ledger's records, its left-out file, the declarations, the chosen identity, the period, the filters and the axis. Two reports with the same inputs print them byte for byte the same, whatever ingest ran before and in whatever order, run again or not, in any time zone or locale. `run`, always last, is what the read that printed the report found, and may differ from one report to the next. + +A field may be added to version `2`: a figure outside `run`, a fact of the read inside it. A field is never renamed, retyped, moved or removed without a new `version`. ## Top level @@ -14,8 +18,7 @@ Every field but `run` is a figure: two reports over the same stored calls, decla | `axis` | What `rows` split by | | `filters` | What the report was narrowed to: `task`, `ticket`, `person`, `model`, `agent`, `session`, `repository`, `absent`, `unattributed`. `null` when not given, else the value as resolved: a full id for a start or `.`, the axis for `absent`, the reason or `any` for `unattributed` | | `rows` | One per value of the axis, of the calls the filters keep; every axis adds up to the same `totals` | -| `totals` | The calls of the period the filters keep | -| `unknown_records` | Calls with at least one counter unknown | +| `totals` | The calls of the period the filters keep; `totals.total.unknown_records` is how many had at least one counter unknown | | `coverage` | What the figures rest on | | `run` | What the read that printed the report found, last | From 4a3cdc560584f384d72a3558aad3ef6814de04ee Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 06:46:58 +0200 Subject: [PATCH 10/13] docs(telemetry): review round 1 and its fixes, with their red runs Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .../2026_10_17_report-envelope-v2/contract.md | 8 ++-- .../2026_10_17_report-envelope-v2/evidence.md | 43 +++++++++++++++++++ .../2026_10_17_report-envelope-v2/review.md | 35 ++++++++++++++- 3 files changed, 81 insertions(+), 5 deletions(-) diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md index e74372197..8a3d9ea9e 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/contract.md @@ -24,7 +24,7 @@ flowchart LR I["the ingest this report ran first, transcripts on disk, partitions read"] --> R["run"] ``` -- **The figures** are every top-level field except `run`. They are a pure function of the ledger (records and `left-out.txt`), the declarations, the chosen identity, the period, the filters and the axis. Their bytes are `JSON.stringify` of the envelope with `run` removed. +- **The figures** are every top-level field except `run`. They are a pure function of the ledger (records and `left-out.txt`), the declarations, the chosen identity, the period, the filters and the axis. Their bytes are `JSON.stringify` of the envelope with `run` removed. The canonical definition, linked from the usage contract and the CLI memory, is the skill's envelope reference (`plugins/aidd-telemetry/skills/01-usage/references/envelope.md#figures-and-run`): a skill links nowhere outside itself, so the definition lives there. - **`run`**, always last, holds what the read that printed the envelope found. It can differ between two reports over the same data. - **Inside the CLI**, `ReportResult` makes the same split: `coverage` (`records`, `leftOut`) and `run` (`filesRead`, `unrecognised`, `notStored`, `consentLogDamaged`, `oldestTranscriptAt`, `skippedLedgerLines`). The text report prints exactly what it printed before. - **Refused** becomes `{"version": 2, "refused": "AIDD_TELEMETRY=0"}`. @@ -40,7 +40,7 @@ flowchart LR | `filters` | `filters` | inputs, resolved against the period's calls (a function of the ledger). `--repository .` resolves from the working directory, which is part of the request | | `rows` | `rows` | ledger, declarations and identity, filtered | | `totals` | `totals` | same | -| `unknown_records` | `unknown_records` | same as `totals.total.unknown_records`, kept for readers that check it alone | +| `unknown_records` | dropped: `totals.total.unknown_records` | a copy of a totals field; review round 1 kept one | | `coverage.records` | `coverage.records` | the calls of the period in the ledger | | `coverage.left_out` | `coverage.left_out` | a set of hashes, each kept once. Neither order nor a repeated ingest changes it, and none of it has a day, so every report counts all of it | | `coverage.files_read` | `run.files_read` | what this read opened: 2 then 0 over the same data | @@ -69,12 +69,12 @@ Nothing is dropped: the 01-usage skill reads every run field (unknowns, "when th | # | Property | | --- | --- | -| P1 | For any ledger, left-out entries, declarations, identity, period, axis and filters, the figures are byte-identical across: the order and grouping the calls and left-out entries were stored in, extra no-op stores, a second report right after, and `TZ` in `UTC`, `Pacific/Kiritimati`, `America/Adak`. `run` is exactly what the ingest and the load reported, never part of the figures | +| P1 | For any ledger (daily and monthly partitions, lines that are no record), left-out entries, declarations, identity, period, axis and filters, the figures are byte-identical across: the order and grouping the calls and left-out entries were stored in, the zone they were stored in, extra no-op stores, a second report right after, and `TZ` in `UTC`, `Pacific/Kiritimati`, `America/Adak`, the envelope built in that zone. `run` is exactly what the ingest and the load reported, never part of the figures | | P2 | Through the built CLI, over one sandbox with real transcripts, the figures of the first report (which reads 2 transcripts) and of every later one are byte-identical across `TZ` ∈ {`UTC`, `Pacific/Kiritimati`, `America/Adak`} × `LC_ALL`/`LANG` ∈ {`C`, `fr_FR.UTF-8`, `ar_SA.UTF-8`}, on the `day` and `task` axes. `run.files_read` is 2, then 0 | ## The time zone and locale guard (M9) -- `cli/scripts/test-foreign-clock.mjs` (`pnpm test:foreign-clock`) runs the telemetry tests (`tests/contexts/telemetry`, `tests/presentation/commands/telemetry`, `tests/presentation/display/telemetry`, `tests/e2e/telemetry`) under `TZ=Pacific/Kiritimati` and `LC_ALL=LANG=ar_SA.UTF-8`. First it checks that a child Node sees a UTC offset other than 0 and a locale that is not `en`, and fails if it does not. +- `cli/scripts/test-foreign-clock.mjs` (`pnpm test:foreign-clock`) runs the telemetry tests (`tests/contexts/telemetry`, `tests/presentation/commands/telemetry`, `tests/presentation/display/telemetry`, `tests/e2e/telemetry`) under `TZ=Pacific/Kiritimati` and `LC_ALL=LANG=ar_SA.UTF-8`. First it checks, in a child Node, that the offset is not 0 and that a number and a date are written otherwise than in `en-US`, and exits 1 if not: a locale ICU has no data for keeps its name and formats as English (review round 1). - CI: one step in the existing `cli-coverage` job (ubuntu), after the suite. No new job, no matrix. - Proven by hand (`evidence.md`): - the ledger's day read in local time: under UTC it changes nothing, so on `origin/next` the default suite passes it, and with P1 one test fails. The guard fails 36; diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md index 0ec674a57..07039472a 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md @@ -89,3 +89,46 @@ The first `pnpm test:mutation:telemetry` stopped in Stryker's initial run: `UTC: | `pnpm lint`, `pnpm typecheck`, type honesty, markdown links | green in the pre-commit hook of each commit | Not run here: Windows. The new tests use `join` for paths and rename no open file. `test-foreign-clock.mjs` spawns `node` directly, never a shell. + +## Review round 1 fixes + +### Red runs, tests first + +`report-envelope.unit.test.ts`, before `unknown_records` left the top level: + +```txt +× states how many records had an unknown counter once, in the totals +× puts what the read that printed it found apart, last, so the rest is the figures +Tests 2 failed | 5 passed (7) +``` + +`tests/scripts/foreign-clock.integration.test.ts`, against the round 0 check, which only looked at the locale's name (extracted unchanged into `scripts/foreign-clock.mjs`): + +```txt +× refuses a locale ICU has no data for, which formats as English whatever its name +TypeError: .toMatch() expects to receive a string, but got object +``` + +The check returned `null` for `xx_BOGUS.UTF-8`, so it accepted that locale. + +### The guard's own check fails closed + +`scripts/test-foreign-clock.mjs`, each constant swapped by hand, then restored: + +```txt +Under TZ=Pacific/Kiritimati LC_ALL=xx_BOGUS.UTF-8, a child Node formats as English: 1,234,567.5: this guard would prove nothing here. +bogus locale exit 1 +Under TZ=Not/AZone LC_ALL=ar_SA.UTF-8, a child Node runs in UTC: this guard would prove nothing here. +bogus zone exit 1 +``` + +### P1 sees a zone dependence in the envelope + +Hand mutation in `report-envelope.ts`: `period.from` written as `new Date(`${period.from}T00:00:00.000Z`).toLocaleDateString("sv")`, the local day of a UTC midnight. Reverted after. + +| P1 | Result | +| --- | --- | +| round 0 (`HEAD`, envelope built after the zone is restored) | `Tests 2 passed (2)`: missed | +| round 1 (envelope and figures' bytes built in the zone) | `Tests 1 failed`, `- "from": "2026-09-30"` `+ "from": "2026-09-29"` | + +P1 now also starts from monthly partitions (calls and lines that are no record) and stores its second history with the clock in another zone. diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md index acc00262d..c79c4c080 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md @@ -1,3 +1,36 @@ # Review: Report envelope v2 -Pending independent review. +- Rounds: 1 +- Started: 2026-10-17 + +## Round 1 · r-7c3e + +- Date: 2026-10-17 +- By: Baptiste LAFOURCADE +- Diff: `616dbbea...80293f4d` +- Axes: code, functional, relevancy +- Verdict: changes-requested +- Score: 4/6 met, 0 unmet, 2 unverified at runtime, 0 out of the diff + +### Criteria + +- [x] P1, P2 hold, red on `origin/next` for a behavioural reason — P1 green here (`Tests 65 passed`, sandboxed targeted run of the four report tests); P2 inside `cli / Test & Coverage thresholds` on head 80293f4d (`Tests 6744 passed | 1 skipped`); red on `616dbbea` recorded in `evidence.md` (P1 `oldest_transcript_at`, `skipped_ledger_lines 2 → 1`; P2 `files_read 2 → 0` on the built CLI). P1's main arm is red by construction (its `run` generator varies v1's figure fields); the `skipped_ledger_lines` case and P2 are the genuinely behavioural reds +- [x] The envelope test pins v2: `run` last, every figure field, refused v2 — `report-envelope.unit.test.ts` with `toStrictEqual`, green in the targeted run; red on v1 recorded (`Tests 4 failed | 3 passed`) +- Unverified at runtime: the guard runs locally and in CI, and goes red on a hand-made zone and locale dependence — runs on head in CI (`Telemetry tests under TZ=Pacific/Kiritimati, 840 min from UTC, locale ar-SA`, `Tests 1497 passed (1497)`, job 117308487679); the two red proofs (36 and 24 failures) are recorded at `25c9f857`, not on the reviewed head. The guard script is unchanged since then +- [x] Every reader updated, and the text report unchanged — no stale `coverage.files_read`/`not_stored`/`oldest_transcript_at`/`skipped_ledger_lines`/`consent_log_damaged`/`unrecognised_shapes` outside `run.` in the repo; `answer.md:19`, `envelope.md`, `usage-contract.md:101-128`, `telemetry.md:29,34`, `check-claude-tokens-end-to-end.cjs:29,268-273` updated; e2e text assertions untouched in the diff; `node --test scripts/__tests__/check-claude-tokens-end-to-end.test.js` `pass 15, fail 0`; no other consumer (hooks, kanban, sibling `telemetry-screen` branch) reads the moved fields +- Unverified at runtime: `pnpm test`, scripts tests, `pnpm test:mutation:telemetry` ≥ 96, `pnpm smoke`, both token end-to-end checks — CI on head: coverage, `cli / Smoke`, the three `Tokens end to end` jobs and telemetry mutation shards 1/3 and 3/3 pass; shard 2/3 still pending when this round was written. The 96.7 score is recorded at `25c9f857`, not on head +- [x] Bundle within 709 KB, never weakened — `cli / Build & Bundle Budget` passes on head; `evidence.md` 696.1 KB / 709 KB, +42 B, budget file untouched by the diff + +### Findings + +- `release-please-config.json:17` : the PR's release heads-up names CLI 5.5.0 → 6.0.0 and `aidd-telemetry` 0.3.0 → 1.0.0 only. The root package `.` takes every commit: the 5.12.0 `CHANGELOG.md` lists `cli:` and `telemetry:` entries, and #1037 pinned `.` at 5.12.0 next to cli and telemetry for exactly this reason, then #1040 removed those pins on `next`. So `feat(telemetry)!`, as configured, also takes `marketplace.json` from 5.12.0 to **6.0.0**. That is three majors, against the last release's precedent of pinning a telemetry break to a minor → state all three in the PR heads-up and get an explicit maintainer decision: accept 6.0.0 everywhere, or add `release-as` pins at promote time as #1037 did — 🟡 major, fit +- `cli/scripts/test-foreign-clock.mjs:34` : the probe checks the locale's tag, not its behaviour, so the locale half does not fail closed. Observed: `LC_ALL=xx_BOGUS.UTF-8` resolves to `xx-bogus`, which passes `!startsWith("en")`, while `(1200).toLocaleString()` still prints `1,200`. A locale ICU has no data for, or a Node built with less ICU, would run the guard in English and report it as foreign. The zone half does fail closed (`TZ=Not/AZone` gives offset 0) → probe the behaviour: require `(1200).toLocaleString()` to differ from `"1,200"` (it prints `١٬٢٠٠` under `ar_SA`), or require the resolved locale to equal `ar-SA` exactly — 🟡 major, error-handling +- `cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts:255-265` : `inZone` wraps only `execute`. `reportEnvelopeOf(result)`, and the `JSON.stringify` in `figuresOf`, run after `TZ` is restored. A zone dependence added in `report-envelope.ts`, the file this PR rewrites, escapes P1's zone arm; only P2 and the guard would see it → build the envelope, or the figures' bytes, inside `inZone` — 🟢 minor, code-health +- `cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts:219` : on a worker thread `inZone` silently runs in the thread's zone, so under Stryker P1 still reports a pass with its zone arm off. This is documented in the contract's Limits and is no regression, and the default forks pool does vary the zone (`evidence.md`: the `dayOf` mutation fails P1 under the default run) → make the skip visible, e.g. a `skipIf(!isMainThread)` zone case or a logged note, so a pool change cannot drop the arm unseen — 🟢 minor, code-health +- `cli/tests/contexts/telemetry/application/report/report-figures.property.integration.test.ts:188-196` : `ledgerWith` writes only `.jsonl` and `storeCalls` writes day files. P1 never has a legacy monthly partition, yet the contract cites settling a monthly file as what makes `skipped_ledger_lines` a run fact. All histories also store in the process zone, so the zone is varied for the report only, never for the store. `numRuns: 60` → add a monthly partition to the generator, and store one history inside another `inZone` — 🟢 minor, code-health +- `plugins/aidd-telemetry/skills/01-usage/actions/answer.md:19` : a version mismatch fails closed in both directions, since the 0.3.0 skill checks `other than 1`. But the message names no fix: a v2 skill on cli 5.5.0 passes the `--help` probe and then only says "a shape this skill does not know". `00-init` names `@ai-driven-dev/cli` in the same situation, and the action's `## Test` has no row for a non-2 envelope → for `version` below 2, say to update `@ai-driven-dev/cli`; above 2, to update the plugin; add a Test row — 🟢 minor, fit +- `plugins/aidd-telemetry/skills/01-usage/references/envelope.md:5` : the figures' inputs are given as "stored calls, declarations, period and filters". This omits the identity, the axis and `left-out.txt`, which `usage-contract.md` and `report-envelope.ts` name → list the same inputs as the contract — 🟢 minor, rot +- `cli/aidd_docs/memory/telemetry.md:34` : this is the fourth statement of the figures-versus-run definition, after `report-envelope.ts:42-73`, `usage-contract.md:124` and `envelope.md:5`. The skill's copy is forced, because a skill cannot link out, but the CLI memory can → link to the usage contract's Report envelope section instead of restating it — 🟢 minor, rot +- `cli/src/contexts/telemetry/application/report/report-envelope.ts:130` : v2 is the declared clean, no-shim target, yet it keeps top-level `unknown_records`, a copy of `totals.total.unknown_records`, "for readers that check it alone" → drop it in the break, or record why a duplicate survives a clean break — 🟢 minor, rot +- `aidd_docs/product/usage-contract.md:106` : `period` is an input. Under `--days` it is resolved from the UTC clock at run time (`periodOf`, `dayOf(now.toISOString())`), and the resolved `{from, to}` is printed in the figures, so it is TZ-independent and not a run fact. But nothing says the purity holds for the resolved period, not for the flags: `--days 7` across UTC midnight prints different figures by design → add "under `--days`, the UTC days ending today" to the `period` row — 🟢 minor, fit +- `aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md:1` : tests and code land in one commit (`d25f7ca7`), so git cannot show red-first; only the recorded output does. The gates are recorded at `25c9f857`, not at head (since then, head changed only a comment in `attribution.ts` and the docs) → re-record the gates at head, or say they carry over and why — 🟢 minor, code-health From 63bda50670d0264e4a77803957a0d9126e8710db Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 07:00:47 +0200 Subject: [PATCH 11/13] docs(telemetry): gates after review round 1 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- .../2026_10/2026_10_17_report-envelope-v2/evidence.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md index 07039472a..0cac2e0d4 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md @@ -132,3 +132,14 @@ Hand mutation in `report-envelope.ts`: `period.from` written as `new Date(`${per | round 1 (envelope and figures' bytes built in the zone) | `Tests 1 failed`, `- "from": "2026-09-30"` `+ "from": "2026-09-29"` | P1 now also starts from monthly partitions (calls and lines that are no record) and stores its second history with the clock in another zone. + +### Gates after round 1, at `4a3cdc56` + +| Gate | Result | +| --- | --- | +| `cd cli && pnpm test` | `Test Files 517 passed \| 1 skipped (518)`, `Tests 6748 passed \| 1 skipped (6749)`, first run green | +| `pnpm test:foreign-clock` | `Telemetry tests under TZ=Pacific/Kiritimati and LC_ALL=ar_SA.UTF-8, both seen by Node`, `Tests 1497 passed (1497)` | +| targeted (`tests/contexts/telemetry tests/presentation tests/scripts tests/e2e/telemetry`) | `Tests 2008 passed (2008)` | +| Bundle | 712,713 B, `696.0 KB / budget: 709 KB` | +| `pnpm test:mutation:telemetry` | not completed: stopped by the machine running low on memory at 318/339 mutants. Not rerun | +| `pnpm smoke` | not rerun after round 1, with memory short | From 44c74da5e4f1ae55e466e8cca3c410555ee1135e Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 07:11:14 +0200 Subject: [PATCH 12/13] test(cli): accept a foreign locale only where LC_ALL can set one Windows ICU takes its locale from the system and ignores LC_ALL, so the case accepting ar_SA failed on windows-latest. It is skipped there. The three refusal cases still run on every platform. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- cli/tests/scripts/foreign-clock.integration.test.ts | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/cli/tests/scripts/foreign-clock.integration.test.ts b/cli/tests/scripts/foreign-clock.integration.test.ts index b49381d3a..2a93f6f5d 100644 --- a/cli/tests/scripts/foreign-clock.integration.test.ts +++ b/cli/tests/scripts/foreign-clock.integration.test.ts @@ -10,9 +10,13 @@ const guarded = (zone: string, locale: string): NodeJS.ProcessEnv => ({ }); describe("the foreign clock guard's own check", () => { - it("accepts a zone far from UTC and a locale that writes numbers and dates otherwise", () => { - expect(foreignClockProblem(guarded("Pacific/Kiritimati", "ar_SA.UTF-8"))).toBeNull(); - }); + // Windows ICU takes its locale from the system, never LC_ALL: there no locale can be accepted. + it.skipIf(process.platform === "win32")( + "accepts a zone far from UTC and a locale that writes numbers and dates otherwise", + () => { + expect(foreignClockProblem(guarded("Pacific/Kiritimati", "ar_SA.UTF-8"))).toBeNull(); + } + ); it("refuses a locale ICU has no data for, which formats as English whatever its name", () => { expect(foreignClockProblem(guarded("Pacific/Kiritimati", "xx_BOGUS.UTF-8"))).toMatch( From 1b38af2c39cfb66c0f7c0766a188c8881e07cb43 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Sat, 17 Oct 2026 07:12:44 +0200 Subject: [PATCH 13/13] docs(telemetry): review round 2, and figures pure in the period's days --days resolves against today's UTC day, so the figures are pure in the resolved from and to, not in the flags. The usage contract no longer restates what run holds: it links to the envelope reference. Also records the checker's round 2. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp --- aidd_docs/product/usage-contract.md | 4 +-- .../2026_10_17_report-envelope-v2/review.md | 34 ++++++++++++++++++- .../skills/01-usage/references/envelope.md | 4 +-- 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/aidd_docs/product/usage-contract.md b/aidd_docs/product/usage-contract.md index f7823674f..cb6f72b0e 100644 --- a/aidd_docs/product/usage-contract.md +++ b/aidd_docs/product/usage-contract.md @@ -103,7 +103,7 @@ What an ingest measured and could not store is kept in `ledger/left-out.txt`, so | Field | Meaning | | --- | --- | | `version` | `2` | -| `period` | `{from, to}`, UTC days, `null` for open | +| `period` | `{from, to}`, UTC days, `null` for open, as resolved: `--days` against today's UTC day. The figures depend on these days, not on the flags | | `axis` | `total`, `person`, `session`, `model`, `agent`, `day`, `repository`, `task`, `ticket` | | `filters` | `task`, `ticket`, `person`, `model`, `agent`, `session`, `repository`, `absent`, `unattributed`, in that order: `null` when not given, else as resolved (see [Filters](#filters)) | | `rows` | one per axis value, of the calls that match | @@ -123,7 +123,7 @@ Row and `totals`: `records`, `input`, `output`, `cache_read`, `cache_write` (eac - Envelope version 1 put the figures and the run together. Version 2 keeps them apart and has no compatibility shape. - `AIDD_TELEMETRY=0`: nothing read, object is `{"version": 2, "refused": "AIDD_TELEMETRY=0"}`. - The text report labels an `absent` person `anonymous`; the envelope keeps `kind: absent`, `key: null`. -- A report ingests first. `run` describes that read: what it opened, met and left out, whether the consent log let it store, how old the transcripts on disk are, and the lines that are not records in the partitions it loaded, counted before it settled them (a rewrite keeps each once, so the next report can count fewer). `coverage.records` describes the period; `coverage.left_out` every ingest so far. +- A report ingests first; what that read found is its `run`, see [the envelope reference](../../plugins/aidd-telemetry/skills/01-usage/references/envelope.md#run). `coverage.records` describes the period; `coverage.left_out` every ingest so far. - A report with `--days`, `--from` or `--to` reads only the days of its period and the day on each side, plus any monthly file one of them falls in, with every day of its month; with none it reads the whole ledger. The copies of one call have been observed less than a day apart, and under that bound both print the same figures. Outside it, only a report with no period folds every copy of a call. `run.skipped_ledger_lines` counts the lines of the partitions read. - No path, working directory or branch. `session` keys are full session ids, `repository` keys are hashes. diff --git a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md index c79c4c080..7315923e2 100644 --- a/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md +++ b/aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/review.md @@ -1,6 +1,6 @@ # Review: Report envelope v2 -- Rounds: 1 +- Rounds: 2 - Started: 2026-10-17 ## Round 1 · r-7c3e @@ -34,3 +34,35 @@ - `cli/src/contexts/telemetry/application/report/report-envelope.ts:130` : v2 is the declared clean, no-shim target, yet it keeps top-level `unknown_records`, a copy of `totals.total.unknown_records`, "for readers that check it alone" → drop it in the break, or record why a duplicate survives a clean break — 🟢 minor, rot - `aidd_docs/product/usage-contract.md:106` : `period` is an input. Under `--days` it is resolved from the UTC clock at run time (`periodOf`, `dayOf(now.toISOString())`), and the resolved `{from, to}` is printed in the figures, so it is TZ-independent and not a run fact. But nothing says the purity holds for the resolved period, not for the flags: `--days 7` across UTC midnight prints different figures by design → add "under `--days`, the UTC days ending today" to the `period` row — 🟢 minor, fit - `aidd_docs/tasks/2026_10/2026_10_17_report-envelope-v2/evidence.md:1` : tests and code land in one commit (`d25f7ca7`), so git cannot show red-first; only the recorded output does. The gates are recorded at `25c9f857`, not at head (since then, head changed only a comment in `attribution.ts` and the docs) → re-record the gates at head, or say they carry over and why — 🟢 minor, code-health + +## Round 2 · r-2b91 + +- Date: 2026-10-17 +- By: Baptiste LAFOURCADE +- Diff: `80293f4d...63bda506` (round-1 fixes), whole PR re-checked against `origin/next` +- Axes: code, functional, relevancy +- Verdict: changes-requested. `cli / Windows` fails on head, and `cli / gate`, the required check on `next`, needs it (`cli-ci.yml:666-683`). On ubuntu, `Test & Coverage thresholds` passes, including the step `Telemetry tests under TZ=Pacific/Kiritimati and LC_ALL=ar_SA.UTF-8, both seen by Node` (job 117314876160). `Mutation (telemetry 3/3)` passes; shards 1/3 and 2/3 were still pending when this round was written +- Score: 6.5/10 + +### Round-1 findings + +- [x] Release heads-up: the PR body now names CLI 6.0.0, `aidd-telemetry` 1.0.0 and root `.` 6.0.0, cites #1037 and #1040, and leaves the decision to the maintainer. Checked against `release-please-config.json`: `bump-minor-pre-major: false` (so 0.3.0 → 1.0.0 holds), and `.` has no path exclusion except the manifest +- [x] The probe checks how the locale writes, not its name (`cli/scripts/foreign-clock.mjs`). Run sandboxed on this machine: `ar_SA` and `fr_FR` → `null`; `xx_BOGUS`, `en_GB`, `C`, `POSIX` and empty → `formats as English`; `Not/AZone`, `UTC` and `Etc/GMT` → `runs in UTC`. A probe that crashes exits non-zero. CI wiring is `cli-ci.yml:302-303`, after `pnpm test:coverage` +- [x] P1 builds the envelope and the figures' bytes inside `inZone`. The red run is recorded: a `toLocaleDateString("sv")` mutation in `report-envelope.ts` is missed by round 0 and caught by round 1 +- [x] P1 now has monthly partitions, holding both calls and lines that are not records, and stores the second history inside `inZone(first)`. Sandboxed targeted run: `Test Files 3 passed`, `Tests 13 passed`. Not proved: no hand mutation shows that the monthly arm catches a split that loses or duplicates a call +- [x] Skill mismatch: `answer.md:19` names the fix in both directions (below 2, update `@ai-driven-dev/cli`; above 2, update `aidd-telemetry`), and `## Test` now has rows for version 1 and version 3 +- [x] `envelope.md` "Figures and run" lists the same inputs as the contract +- [x] One definition: `usage-contract.md:101` and `cli/aidd_docs/memory/telemetry.md:34` link to `envelope.md#figures-and-run` (the anchor exists, and both relative paths resolve), and the JSDoc in `report-envelope.ts` points there +- [x] Top-level `unknown_records` removed. `git grep unknown_records` outside `aidd_docs/tasks` finds only per-counter and `totals.total` reads: `answer.md:23`, `envelope.md:21`, the e2e sandbox and the journey `Envelope` types, and `check-claude-tokens-end-to-end.cjs:259` (it reads the totals counter). `node --test scripts/__tests__/check-claude-tokens-end-to-end.test.js`: `pass 15, fail 0`. The sibling `brainstorm-telemetry` worktree has no reader in `cli/src`, `plugins` or `scripts` +- [x] Gates re-recorded at `4a3cdc56`. `git diff --stat 4a3cdc56 63bda506` shows only `evidence.md`, so they carry over to head +- [ ] Worker-thread zone skip made visible: not done. `inZone` still returns `work()` silently when `!isMainThread` (`report-figures.property.integration.test.ts:234`), and the gap is documented only in a comment and in the PR Limits. I accept it: the forked default pool, P2 and the guard all vary the zone, so under Stryker a zone mutant can only survive (score too low), never hide a failure from CI. It stays an open minor +- [ ] `period` under `--days`: not done. Neither `usage-contract.md:106` nor `envelope.md:17` says that the figures are pure in the resolved `{from, to}`, not in the flags + +### Findings + +- `cli/tests/scripts/foreign-clock.integration.test.ts:13-15` : the "accepts … `ar_SA`" case asserts `null`, which holds only where ICU honours `LC_ALL`/`LANG`. On windows-latest, `pnpm test:integration` runs it and it fails: `expected 'a child Node formats as English: 1,23…' to be null`, received `"a child Node formats as English: 1,234,567.5, 10/2/2026"` (job 117314876110). The PR's own Limits say Windows ICU ignores `LC_ALL`, yet the test sits in the default integration tier, which runs on every platform. `cli / gate` therefore fails and the PR cannot merge into `next`. The probe is right to refuse there; it is the test that assumes the platform → skip the acceptance case where ICU ignores the environment (`it.skipIf(process.platform === "win32")`, with the reason in the title), or derive what it expects from a probe of the platform. Keep the three refusal cases unconditional, since they hold everywhere. Then rerun `cli / Windows` — 🔴 critical, error-handling +- PR body, Heads-up, "Release impact": the body says the squash title decides. But the repository allows merge commits (`allow_merge_commit=true`), and a merge commit carries `d25f7ca7 feat(telemetry)!` into `next`, then through the promote merge to `main`, which ships three majors whatever the title says → state that this PR must be squash-merged, or reword `d25f7ca7` before merging if the majors are not wanted — 🟡 major, fit +- `report-figures.property.integration.test.ts:234`: the worker-thread zone skip is still silent (carried over from round 1) → add a `skipIf(!isMainThread)` zone case, or one logged line — 🟢 minor, code-health +- `envelope.md:17`, `usage-contract.md:106`: `--days` resolves from the UTC clock at run time. Two runs across UTC midnight print different figures by design (carried over from round 1) → add "under `--days`, the UTC days ending today" to the `period` row — 🟢 minor, fit +- `usage-contract.md:126`: this restates what `run` holds ("what it opened, met and left out, whether the consent log let it store…") next to the link that calls `envelope.md` the one definition. The contract's field table also repeats `envelope.md`'s top-level table → keep the storage-side facts the contract owns (partitions read, settling), and link for what each field means — 🟢 minor, rot +- `evidence.md` "Gates after round 1": telemetry mutation and `pnpm smoke` were not rerun after round 1. CI `Mutation (telemetry 3/3)` passes on head. Shards 1/3 and 2/3 and `Test & Coverage thresholds` were pending when this round was written → confirm all are green before marking the PR ready — 🟢 minor, code-health diff --git a/plugins/aidd-telemetry/skills/01-usage/references/envelope.md b/plugins/aidd-telemetry/skills/01-usage/references/envelope.md index 561f0a4bf..6cedb47ea 100644 --- a/plugins/aidd-telemetry/skills/01-usage/references/envelope.md +++ b/plugins/aidd-telemetry/skills/01-usage/references/envelope.md @@ -4,7 +4,7 @@ What `aidd telemetry report --json` prints, and what each part means. Version `2 ## Figures and run -Every field but `run` is a figure. The figures depend on these inputs and nothing else: the ledger's records, its left-out file, the declarations, the chosen identity, the period, the filters and the axis. Two reports with the same inputs print them byte for byte the same, whatever ingest ran before and in whatever order, run again or not, in any time zone or locale. `run`, always last, is what the read that printed the report found, and may differ from one report to the next. +Every field but `run` is a figure. The figures depend on these inputs and nothing else: the ledger's records, its left-out file, the declarations, the chosen identity, the period as resolved, the filters and the axis. Two reports with the same inputs print them byte for byte the same, whatever ingest ran before and in whatever order, run again or not, in any time zone or locale. `run`, always last, is what the read that printed the report found, and may differ from one report to the next. A field may be added to version `2`: a figure outside `run`, a fact of the read inside it. A field is never renamed, retyped, moved or removed without a new `version`. @@ -14,7 +14,7 @@ A field may be added to version `2`: a figure outside `run`, a fact of the read | --- | --- | | `version` | `2`. Any other value is a shape not known here | | `refused` | Only present, with `version`, when measurement is refused by the environment; nothing was read | -| `period` | `from` and `to`, UTC days, `null` for open | +| `period` | `from` and `to`, UTC days, `null` for open: resolved from the flags, `--days` against today's UTC day. The figures depend on these days, not on the flags | | `axis` | What `rows` split by | | `filters` | What the report was narrowed to: `task`, `ticket`, `person`, `model`, `agent`, `session`, `repository`, `absent`, `unattributed`. `null` when not given, else the value as resolved: a full id for a start or `.`, the axis for `absent`, the reason or `any` for `unattributed` | | `rows` | One per value of the axis, of the calls the filters keep; every axis adds up to the same `totals` |