From 96710e6d3e00f4c96f3f6fbca800b95dff9ac2b4 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Thu, 19 Nov 2026 01:51:43 -0500 Subject: [PATCH 1/3] feat(lane-transcripts): open a leg's sealed telemetry into the sink The puller opens otel.jsonl.gz.sealed, when an artifact carries it beside the transcript, under the payload name otel.jsonl.gz with the transcript's run and job binding, and stores it next to the transcript. The row gains telemetry, telemetry_outcome, telemetry_bytes, telemetry_spans and telemetry_joined_spans, the last counting the spans whose traceId is the row's trace_id. Telemetry that does not open keeps the transcript. Prune removes a row's telemetry with its transcript and clears a missing path. Refs: #17261 Co-Authored-By: Claude Opus 5.5 --- docs/contracts/lane-transcripts/README.md | 14 ++++- scripts/lane-transcripts/README.md | 18 ++++-- scripts/lane-transcripts/pull.mjs | 52 +++++++++++++--- scripts/lane-transcripts/pull.test.mjs | 74 ++++++++++++++++++++++- scripts/lane-transcripts/store.mjs | 38 ++++++++++-- 5 files changed, 172 insertions(+), 24 deletions(-) diff --git a/docs/contracts/lane-transcripts/README.md b/docs/contracts/lane-transcripts/README.md index 6c4de404954..82fc2aedae1 100644 --- a/docs/contracts/lane-transcripts/README.md +++ b/docs/contracts/lane-transcripts/README.md @@ -16,6 +16,7 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`. /metrics.jsonl one JSON row per leg, no transcript text /outcomes.jsonl one JSON row per lane-outcome record /transcripts///.json.gz one leg's transcript +/transcripts///.jsonl.gz one leg's telemetry, when the leg kept it /state.json the writer's own state; readers ignore it ``` @@ -37,8 +38,12 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`. less of GitHub's 1 MB per job (counted at two bytes a character); one over its cap, or one the credential scan blocks, arrives as a marker, never cut short. When the other outputs leave no room even for a marker, nothing is carried. -- The file name under a lane's date directory is the writer's; readers take a transcript's path - from its row, never by building it. +- A leg's telemetry is gzipped JSONL, one OTLP/JSON export request per line, traces and logs + together; a line's top-level `resourceSpans` or `resourceLogs` key says which. Prompt and tool + content are redacted at the source. + - **Pointer**: for the line shapes, read the [OTLP specification](https://opentelemetry.io/docs/specs/otlp/). +- The file name under a lane's date directory is the writer's; readers take a transcript's or + telemetry's path from its row, never by building it. ## Row fields @@ -66,6 +71,9 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`. | `failure_class_source` | Which case set `failure_class`: `lane-outcome` (the records agree), `disagree` (they differ) or `none` (no record held) | yes | | `transcript` | The transcript's path relative to the sink, or null once retention removed it | yes | | `pruned` | Why `transcript` is null (`age`, `cap`, `missing`), absent while it is set | yes | +| `telemetry` | The leg's telemetry path relative to the sink, or null when the leg kept none, it did not open, or retention removed it | yes | +| `telemetry_outcome` | `stored`, `none` (the leg kept no telemetry), `sealed_mismatch` (it did not open) or `unreadable` (it opened but did not decompress); absent on a row written before the field existed | yes | +| `telemetry_spans`, `telemetry_joined_spans` | How many spans the telemetry holds, and how many carry the row's `trace_id`; null unless `telemetry_outcome` is `stored` | yes | A reader asking for `--outcome error` reads `is_error`, not `outcome`: a row stored as `success` with `is_error: true` is an error, and is not a `success`. @@ -80,7 +88,7 @@ neither field below. A writer whose keys follow another scheme writes `leg_kind` ### GitHub only `artifact_id`, `run_id` (the integer behind `run_ref`), `job_id` (the same value as `leg_key`), -`workflow`, `event`, `head_branch`, `run_conclusion`, `transcript_bytes` and `pulled_at`. A row +`workflow`, `event`, `head_branch`, `run_conclusion`, `transcript_bytes`, `telemetry_bytes` and `pulled_at`. A row written before the provider-neutral fields existed has only these: a reader takes `run_ref` from `run_id`, `leg_key` from `job_id`, `provider` as `github` and `run_url` as null. diff --git a/scripts/lane-transcripts/README.md b/scripts/lane-transcripts/README.md index 070b9962f28..51990d21673 100644 --- a/scripts/lane-transcripts/README.md +++ b/scripts/lane-transcripts/README.md @@ -42,6 +42,7 @@ machine or object storage, is any directory it mounts at; nothing else changes. ```text /transcripts///--.json.gz the CLI messages, gzipped +/transcripts///--.otel.jsonl.gz the leg's telemetry, gzipped /metrics.jsonl one row per transcript, no content /outcomes.jsonl one row per lane-outcome record /state.json last success, last attempt, last error kind @@ -93,10 +94,16 @@ listed newest first, back to the last successful pull less 6 hours (7 days on th re-run attempt of an older run is still pulled. It skips an artifact it already holds, an expired one, one over 64 MiB, and one whose run's head repository is not this repository: the public key is public, so a fork's run could seal a record. A zip must hold `transcript.json.gz.sealed` and at most one other entry, which must be -a sealed file (`.sealed`, such as a leg's telemetry); the pull opens only the transcript, -and any other zip counts as `bad_zip`. A record that does not open for the -key, the artifact's run id and its job id (moved, tampered with, or sealed under a rotated key) is -counted as `sealed_mismatch` and skipped. The summary line counts each kind. +a sealed file (`.sealed`); any other zip counts as `bad_zip`. A transcript record that does +not open for the key, the artifact's run id and its job id (moved, tampered with, or sealed under a +rotated key) is counted as `sealed_mismatch` and skipped. The summary line counts each kind. + +When the other entry is `otel.jsonl.gz.sealed` (`upload-lane-transcript`'s `otel-file`), the pull +opens it under the payload name `otel.jsonl.gz` with the same run and job binding and stores it +beside the transcript. The row's `telemetry_outcome` says what happened (`stored`, `none`, +`sealed_mismatch`, `unreadable`); a telemetry record that does not open keeps the transcript. +`telemetry_spans` counts the spans and `telemetry_joined_spans` those whose `traceId` is the row's +`trace_id`, so a row shows whether the leg's spans joined its run attempt. A `gh` call that fails is retried, the wait doubling from 2 seconds: a listing page gets 10 tries with the wait capped at 5 minutes, any other call 5 tries. An artifact whose download, unzip or run read still fails counts as `fetch_failed` and the pull goes on. It has no metrics row @@ -145,7 +152,8 @@ resuming at that workflow. It records each run attempt it finished reading as `r Every pull, and `prune`, applies it in this order: -1. A transcript over 30 days old is deleted; its row stays, with `transcript: null`. +1. A transcript over 30 days old is deleted; its row stays, with `transcript: null`. Its + telemetry goes with it (`telemetry: null`). 2. A metrics row, or a lane-outcome record, over 365 days old is dropped. 3. While every file under the sink totals more than 5 GB (5,000,000,000 bytes), the oldest transcript is deleted. diff --git a/scripts/lane-transcripts/pull.mjs b/scripts/lane-transcripts/pull.mjs index 492b24be47e..103babe78a7 100644 --- a/scripts/lane-transcripts/pull.mjs +++ b/scripts/lane-transcripts/pull.mjs @@ -32,6 +32,8 @@ import { readState, sinkBytes, staleness, + telemetryPath, + telemetrySpans, traceIdOf, transcriptMetrics, transcriptPath, @@ -43,6 +45,8 @@ import { export const REPO = "melodic-software/claude-code-plugins"; export const PAYLOAD = "transcript.json.gz"; export const ENTRY = `${PAYLOAD}.sealed`; +export const TELEMETRY_PAYLOAD = "otel.jsonl.gz"; +export const TELEMETRY_ENTRY = `${TELEMETRY_PAYLOAD}.sealed`; const HOUR = 60 * 60 * 1000; const ARTIFACT_DAYS = 7; const OVERLAP = 6 * HOUR; @@ -209,15 +213,15 @@ function fetched(kind, call) { /** * A second sealed file the writer may add beside the transcript, such as the - * leg's telemetry; the pull does not open it. + * leg's telemetry; the pull opens only the telemetry. */ const EXTRA_SEALED = /^[A-Za-z0-9][A-Za-z0-9._-]{0,99}\.sealed$/; /** - * The sealed record inside one artifact zip: the sealed transcript, and at - * most one other sealed file. + * The sealed records inside one artifact zip: the sealed transcript, and the + * telemetry when it is the one other sealed file. */ -function sealedRecord(deps, sink, zip) { +function sealedRecords(deps, sink, zip) { const dir = mkdtempSync(path.join(paths(sink).tmp, "zip-")); try { const file = path.join(dir, "artifact.zip"); @@ -226,7 +230,7 @@ function sealedRecord(deps, sink, zip) { const entries = unzip(["-Z1", file]).toString("utf8").split("\n").filter(Boolean); const extra = entries.filter((entry) => entry !== ENTRY); if (entries.length - extra.length !== 1 || extra.length > 1 || !extra.every((entry) => EXTRA_SEALED.test(entry))) return null; - return unzip(["-p", file, ENTRY]); + return { transcript: unzip(["-p", file, ENTRY]), telemetry: extra[0] === TELEMETRY_ENTRY ? unzip(["-p", file, TELEMETRY_ENTRY]) : null }; } finally { rmSync(dir, { recursive: true, force: true }); } @@ -323,11 +327,11 @@ function pullWithin({ sink, repo, recipient, deps, log }) { function store({ sink, repo, recipient, deps, artifact, named, runs, byTrace, now }) { const runId = String(artifact.workflow_run.id); - const record = sealedRecord(deps, sink, fetched("artifact-zip", () => ghRetried(deps, ["api", `repos/${repo}/actions/artifacts/${artifact.id}/zip`]))); - if (!record) return "bad_zip"; + const records = sealedRecords(deps, sink, fetched("artifact-zip", () => ghRetried(deps, ["api", `repos/${repo}/actions/artifacts/${artifact.id}/zip`]))); + if (!records) return "bad_zip"; let gz; try { - gz = openRecord(recipient, [recipient.keyId], PAYLOAD, record, runId, named.jobId); + gz = openRecord(recipient, [recipient.keyId], PAYLOAD, records.transcript, runId, named.jobId); } catch (error) { if (error instanceof OpenError) return "sealed_mismatch"; throw error; @@ -367,10 +371,42 @@ function store({ sink, repo, recipient, deps, artifact, named, runs, byTrace, no const file = path.join(sink, row.transcript); ensureDir(path.dirname(file)); writeFileSync(file, gz, { mode: 0o600 }); + const telemetry = openTelemetry(recipient, records.telemetry, runId, named.jobId, row.trace_id); + Object.assign(row, telemetry.fields); + if (telemetry.gz) { + row.telemetry = telemetryPath(row); + writeFileSync(path.join(sink, row.telemetry), telemetry.gz, { mode: 0o600 }); + } appendRow(sink, row); return "pulled"; } +const NO_TELEMETRY = { telemetry: null, telemetry_bytes: null, telemetry_spans: null, telemetry_joined_spans: null }; + +/** + * The leg's telemetry, opened under its own payload name and the transcript's + * run and job binding, and the row fields it gives. `telemetry_outcome` is + * `none` (no file), `sealed_mismatch` (it did not open), `unreadable` (it + * opened but is not gzip) or `stored`. + */ +function openTelemetry(recipient, record, runId, jobId, traceId) { + if (!record) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "none" } }; + let gz; + try { + gz = openRecord(recipient, [recipient.keyId], TELEMETRY_PAYLOAD, record, runId, jobId); + } catch (error) { + if (error instanceof OpenError) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "sealed_mismatch" } }; + throw error; + } + let text; + try { + text = gunzipSync(gz, { maxOutputLength: MAX_PLAIN }).toString("utf8"); + } catch { + return { fields: { ...NO_TELEMETRY, telemetry_outcome: "unreadable" } }; + } + return { gz, fields: { telemetry_bytes: gz.length, ...telemetrySpans(text, traceId), telemetry_outcome: "stored" } }; +} + /** * Each completed run of every lane workflow created since the cutoff, newest * first per workflow, starting at `at.workflow` (where a budget-stopped pull diff --git a/scripts/lane-transcripts/pull.test.mjs b/scripts/lane-transcripts/pull.test.mjs index 8448c8873b3..4de28dc1a48 100644 --- a/scripts/lane-transcripts/pull.test.mjs +++ b/scripts/lane-transcripts/pull.test.mjs @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import { execFileSync } from "node:child_process"; -import { cpSync, existsSync, readdirSync, readFileSync, statSync, utimesSync, writeFileSync, mkdirSync } from "node:fs"; +import { cpSync, existsSync, readdirSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync, mkdirSync } from "node:fs"; import path from "node:path"; import { test } from "node:test"; import { gunzipSync, gzipSync } from "node:zlib"; @@ -8,8 +8,8 @@ import { gunzipSync, gzipSync } from "node:zlib"; import { generate } from "../../.github/actions/open-lane-output/keygen.mjs"; import { parsePublicKey, sealRecord } from "../../.github/actions/seal-lane-output/seal.mjs"; import { tempDir } from "../../lib/test-tmp.mjs"; -import { ENTRY, LANE_WORKFLOWS, main, PAYLOAD, REPO, TRANSCRIPT_JOB_IDS, TRANSCRIPT_NAMES } from "./pull.mjs"; -import { CAP_BYTES, parseArtifactName, paths, prune, query, readOutcomes, readRows, traceIdOf, transcriptMetrics, writeRows } from "./store.mjs"; +import { ENTRY, LANE_WORKFLOWS, main, PAYLOAD, REPO, TELEMETRY_ENTRY, TELEMETRY_PAYLOAD, TRANSCRIPT_JOB_IDS, TRANSCRIPT_NAMES } from "./pull.mjs"; +import { CAP_BYTES, parseArtifactName, paths, prune, query, readOutcomes, readRows, telemetrySpans, traceIdOf, transcriptMetrics, writeRows } from "./store.mjs"; const MARKER = "plaintext-marker-5d21"; const NOW = Date.parse("2026-11-02T12:00:00Z"); @@ -766,6 +766,74 @@ test("a zip holding a second sealed file beside the transcript is pulled; a thir assert.equal(result.code, 0, result.err); assert.match(result.out, /"pulled":1,.*"bad_zip":3/); assert.deepEqual(readRows(sink).map((r) => r.artifact_id), [91]); + assert.deepEqual(pickTelemetry(readRows(sink)[0]), { telemetry: null, telemetry_bytes: null, telemetry_spans: null, telemetry_joined_spans: null, telemetry_outcome: "sealed_mismatch" }, "telemetry that does not open keeps the transcript"); +}); + +const pickTelemetry = ({ telemetry, telemetry_bytes, telemetry_spans, telemetry_joined_spans, telemetry_outcome }) => ({ telemetry, telemetry_bytes, telemetry_spans, telemetry_joined_spans, telemetry_outcome }); + +function telemetryLines(traceIds) { + const spans = traceIds.map((traceId, i) => ({ traceId, spanId: `${i}`.padStart(16, "0"), name: MARKER })); + return `${JSON.stringify({ resourceSpans: [{ scopeSpans: [{ spans }] }] })}\n${JSON.stringify({ resourceLogs: [{ scopeLogs: [{ logRecords: [{ body: { stringValue: MARKER } }] }] }] })}\n`; +} + +const sealedTelemetry = (publicLine, text, { runId = 42, jobId = "check-intent", payload = TELEMETRY_PAYLOAD } = {}) => sealRecord(parsePublicKey(publicLine), payload, gzipSync(Buffer.from(text)), String(runId), jobId); + +test("a pull opens the sealed telemetry beside the transcript into the sink and counts the spans that join the row's trace id", () => { + const { publicLine, privateText } = generate(); + const sink = sinkDir(); + const trace = traceIdOf(42, 1); + const text = telemetryLines([trace, trace.toUpperCase(), "0".repeat(32)]); + const deps = fakeDeps([artifact(95, { entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, text) } })]); + const result = run(["pull", "--sink", sink], { env: { LANE_TRANSCRIPT_PRIVATE_KEY: privateText }, deps }); + assert.equal(result.code, 0, result.err); + for (const out of [result.out, result.err]) assert.ok(!out.includes(MARKER)); + const [row] = readRows(sink); + assert.equal(row.trace_id, trace); + const { telemetry, telemetry_bytes: bytes, ...counts } = pickTelemetry(row); + assert.deepEqual(counts, { telemetry_spans: 3, telemetry_joined_spans: 2, telemetry_outcome: "stored" }); + assert.equal(telemetry, row.transcript.replace(/\.json\.gz$/, ".otel.jsonl.gz")); + const file = path.join(sink, telemetry); + assert.equal(gunzipSync(readFileSync(file)).toString("utf8"), text); + assert.equal(bytes, statSync(file).size); + assert.equal(statSync(file).mode & 0o777, 0o600); + assert.ok(!JSON.stringify(row).includes(MARKER), "the metrics row holds no telemetry text"); +}); + +test("telemetry bound to another job or payload name is not stored, and a transcript alone records none", () => { + const { publicLine, privateText } = generate(); + const sink = sinkDir(); + const text = telemetryLines([traceIdOf(42, 1)]); + const page = [ + artifact(96, { entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, text, { jobId: "other" }) } }), + artifact(97, { attempt: 2, entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, text, { payload: PAYLOAD }) } }), + artifact(98, { attempt: 3, record: sealed(publicLine) }), + ]; + assert.equal(run(["pull", "--sink", sink], { env: { LANE_TRANSCRIPT_PRIVATE_KEY: privateText }, deps: fakeDeps(page) }).code, 0); + const byId = Object.fromEntries(readRows(sink).map((r) => [r.artifact_id, r.telemetry_outcome])); + assert.deepEqual(byId, { 96: "sealed_mismatch", 97: "sealed_mismatch", 98: "none" }); + assert.ok(readRows(sink).every((r) => r.telemetry === null && r.transcript)); +}); + +test("telemetrySpans counts spans across lines and skips a line that is not JSON", () => { + const trace = "a".repeat(32); + assert.deepEqual(telemetrySpans(`not json\n${telemetryLines([trace, "b".repeat(32)])}`, trace), { telemetry_spans: 2, telemetry_joined_spans: 1 }); + assert.deepEqual(telemetrySpans("", trace), { telemetry_spans: 0, telemetry_joined_spans: 0 }); +}); + +test("prune drops a row's telemetry with its transcript, and a telemetry file gone missing clears its path", () => { + const sink = sinkDir(); + const rows = seedStore(sink, [1, 31]); + for (const row of rows) { + row.telemetry = row.transcript.replace(/\.json\.gz$/, ".otel.jsonl.gz"); + writeFileSync(path.join(sink, row.telemetry), "t"); + } + writeRows(sink, rows); + rmSync(path.join(sink, rows[0].telemetry)); + prune(sink, NOW); + const after = readRows(sink); + assert.deepEqual(after.map((r) => [r.telemetry, r.pruned ?? null]), [[null, null], [null, "age"]]); + assert.ok(existsSync(path.join(sink, after[0].transcript)), "the fresh transcript stays"); + assert.ok(!existsSync(path.join(sink, rows[1].telemetry)), "the aged telemetry goes with its transcript"); }); test("a pulled row carries its run attempt's trace id, and no failure class while no lane-outcome record is held", () => { diff --git a/scripts/lane-transcripts/store.mjs b/scripts/lane-transcripts/store.mjs index b4786193c32..bcd69bdae84 100644 --- a/scripts/lane-transcripts/store.mjs +++ b/scripts/lane-transcripts/store.mjs @@ -120,6 +120,30 @@ export function transcriptPath({ date, lane, run_id: runId, run_attempt: attempt return path.join("transcripts", date, lane, `${runId}-${attempt}-${jobId}.json.gz`); } +export const telemetryPath = (row) => transcriptPath(row).replace(/\.json\.gz$/, ".otel.jsonl.gz"); + +export function telemetrySpans(text, traceId) { + let spans = 0; + let joined = 0; + for (const line of text.split("\n")) { + let request; + try { + request = JSON.parse(line); + } catch { + continue; + } + for (const resource of Array.isArray(request?.resourceSpans) ? request.resourceSpans : []) { + for (const scope of Array.isArray(resource?.scopeSpans) ? resource.scopeSpans : []) { + for (const span of Array.isArray(scope?.spans) ? scope.spans : []) { + spans++; + if (typeof span?.traceId === "string" && span.traceId.toLowerCase() === traceId) joined++; + } + } + } + } + return { telemetry_spans: spans, telemetry_joined_spans: joined }; +} + // The rows every complete line holds. An append cut short (a full disk, a // kill) leaves a last line with no newline: it is not a row, so its artifact // is pulled again and the next append replaces it. @@ -224,21 +248,25 @@ export function prune(sink, now = Date.now(), { capBytes = CAP_BYTES } = {}) { const keptOutcomes = outcomes.filter((o) => now - Date.parse(o.created_at) <= METRICS_DAYS * DAY); if (keptOutcomes.length < outcomes.length) writeOutcomes(sink, keptOutcomes); const rows = readRows(sink); - const byPath = new Map(rows.filter((r) => r.transcript).map((r) => [path.join(sink, r.transcript), r])); + const byPath = new Map(rows.flatMap((row) => ["transcript", "telemetry"].filter((field) => row[field]).map((field) => [path.join(sink, row[field]), { row, field }]))); const stored = files(paths(sink).transcripts).map((f) => { - const row = byPath.get(f.file); - return { ...f, row, at: row ? Date.parse(row.created_at) : f.mtimeMs }; + const { row, field } = byPath.get(f.file) ?? {}; + return { ...f, row, field, at: row ? Date.parse(row.created_at) : f.mtimeMs }; }); const removed = { age: 0, cap: 0, rows: 0, outcomes: outcomes.length - keptOutcomes.length }; const drop = (entry, reason) => { rmSync(entry.file, { force: true }); entry.gone = true; removed[reason]++; - if (entry.row) Object.assign(entry.row, { transcript: null, pruned: reason }); + if (entry.field === "transcript") Object.assign(entry.row, { transcript: null, pruned: reason }); + else if (entry.field === "telemetry") entry.row.telemetry = null; }; for (const entry of stored) if (now - entry.at > TRANSCRIPT_DAYS * DAY) drop(entry, "age"); // A row whose file is gone keeps its metrics but no path. - for (const row of rows) if (row.transcript && !existsSync(path.join(sink, row.transcript))) Object.assign(row, { transcript: null, pruned: row.pruned ?? "missing" }); + for (const row of rows) { + if (row.transcript && !existsSync(path.join(sink, row.transcript))) Object.assign(row, { transcript: null, pruned: row.pruned ?? "missing" }); + if (row.telemetry && !existsSync(path.join(sink, row.telemetry))) row.telemetry = null; + } const kept = rows.filter((row) => now - Date.parse(row.created_at) <= METRICS_DAYS * DAY); removed.rows = rows.length - kept.length; writeRows(sink, kept); From 9813e9a5485a00dd022828a931e6b392477049f2 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Thu, 19 Nov 2026 02:01:05 -0500 Subject: [PATCH 2/3] fix(lane-transcripts): keep the transcript when its telemetry cannot be read Telemetry that fails to extract from the zip, or decompresses past a 64 MiB cap of its own, is recorded as telemetry_outcome unreadable and the transcript row is still written, so one bad telemetry entry no longer turns the artifact into a fetch failure or aborts the pull. Spans are counted line by line rather than through one split array. Refs: #17261 Co-Authored-By: Claude Opus 5.5 --- docs/contracts/lane-transcripts/README.md | 5 +++-- scripts/lane-transcripts/pull.mjs | 16 ++++++++++++---- scripts/lane-transcripts/pull.test.mjs | 19 +++++++++++++++++++ scripts/lane-transcripts/store.mjs | 6 +++++- 4 files changed, 39 insertions(+), 7 deletions(-) diff --git a/docs/contracts/lane-transcripts/README.md b/docs/contracts/lane-transcripts/README.md index 82fc2aedae1..ef27e89b934 100644 --- a/docs/contracts/lane-transcripts/README.md +++ b/docs/contracts/lane-transcripts/README.md @@ -39,8 +39,9 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`. credential scan blocks, arrives as a marker, never cut short. When the other outputs leave no room even for a marker, nothing is carried. - A leg's telemetry is gzipped JSONL, one OTLP/JSON export request per line, traces and logs - together; a line's top-level `resourceSpans` or `resourceLogs` key says which. Prompt and tool - content are redacted at the source. + together; a line's top-level `resourceSpans` or `resourceLogs` key says which. The writer + (ci-workflows run-claude-cli's `otel-receiver`) leaves the CLI's redaction of prompt and tool + content on. - **Pointer**: for the line shapes, read the [OTLP specification](https://opentelemetry.io/docs/specs/otlp/). - The file name under a lane's date directory is the writer's; readers take a transcript's or telemetry's path from its row, never by building it. diff --git a/scripts/lane-transcripts/pull.mjs b/scripts/lane-transcripts/pull.mjs index 103babe78a7..4bf43194ab8 100644 --- a/scripts/lane-transcripts/pull.mjs +++ b/scripts/lane-transcripts/pull.mjs @@ -53,6 +53,8 @@ const OVERLAP = 6 * HOUR; const PER_PAGE = 100; const MAX_SEALED = 64 * 1024 * 1024; const MAX_PLAIN = 256 * 1024 * 1024; +const MAX_TELEMETRY_PLAIN = 64 * 1024 * 1024; +const UNEXTRACTED = Symbol("unextracted"); export const GH_ATTEMPTS = 5; export const GH_LIST_ATTEMPTS = 10; export const GH_DELAY_MS = 2000; @@ -230,7 +232,13 @@ function sealedRecords(deps, sink, zip) { const entries = unzip(["-Z1", file]).toString("utf8").split("\n").filter(Boolean); const extra = entries.filter((entry) => entry !== ENTRY); if (entries.length - extra.length !== 1 || extra.length > 1 || !extra.every((entry) => EXTRA_SEALED.test(entry))) return null; - return { transcript: unzip(["-p", file, ENTRY]), telemetry: extra[0] === TELEMETRY_ENTRY ? unzip(["-p", file, TELEMETRY_ENTRY]) : null }; + const transcript = unzip(["-p", file, ENTRY]); + if (extra[0] !== TELEMETRY_ENTRY) return { transcript, telemetry: null }; + try { + return { transcript, telemetry: deps.unzip(["-p", file, TELEMETRY_ENTRY]) }; + } catch { + return { transcript, telemetry: UNEXTRACTED }; + } } finally { rmSync(dir, { recursive: true, force: true }); } @@ -391,6 +399,7 @@ const NO_TELEMETRY = { telemetry: null, telemetry_bytes: null, telemetry_spans: */ function openTelemetry(recipient, record, runId, jobId, traceId) { if (!record) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "none" } }; + if (record === UNEXTRACTED) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "unreadable" } }; let gz; try { gz = openRecord(recipient, [recipient.keyId], TELEMETRY_PAYLOAD, record, runId, jobId); @@ -398,13 +407,12 @@ function openTelemetry(recipient, record, runId, jobId, traceId) { if (error instanceof OpenError) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "sealed_mismatch" } }; throw error; } - let text; try { - text = gunzipSync(gz, { maxOutputLength: MAX_PLAIN }).toString("utf8"); + const text = gunzipSync(gz, { maxOutputLength: MAX_TELEMETRY_PLAIN }).toString("utf8"); + return { gz, fields: { telemetry_bytes: gz.length, ...telemetrySpans(text, traceId), telemetry_outcome: "stored" } }; } catch { return { fields: { ...NO_TELEMETRY, telemetry_outcome: "unreadable" } }; } - return { gz, fields: { telemetry_bytes: gz.length, ...telemetrySpans(text, traceId), telemetry_outcome: "stored" } }; } /** diff --git a/scripts/lane-transcripts/pull.test.mjs b/scripts/lane-transcripts/pull.test.mjs index 4de28dc1a48..0ed45f50505 100644 --- a/scripts/lane-transcripts/pull.test.mjs +++ b/scripts/lane-transcripts/pull.test.mjs @@ -814,6 +814,25 @@ test("telemetry bound to another job or payload name is not stored, and a transc assert.ok(readRows(sink).every((r) => r.telemetry === null && r.transcript)); }); +test("telemetry that cannot be extracted or is over its size cap is unreadable, and the transcript is kept", () => { + const { publicLine, privateText } = generate(); + const sink = sinkDir(); + const page = [ + artifact(81, { entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, "\n".repeat(64 * 1024 * 1024 + 1)) } }), + artifact(82, { attempt: 2, entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: "z" } }), + ]; + const deps = fakeDeps(page); + const unzip = deps.unzip; + deps.unzip = (args) => { + if (args[2] === TELEMETRY_ENTRY && JSON.parse(readFileSync(args[1], "utf8"))[TELEMETRY_ENTRY] === Buffer.from("z").toString("base64")) throw Object.assign(new Error("maxBuffer"), { status: 9 }); + return unzip(args); + }; + const result = run(["pull", "--sink", sink], { env: { LANE_TRANSCRIPT_PRIVATE_KEY: privateText }, deps }); + assert.equal(result.code, 0, result.err); + const byId = Object.fromEntries(readRows(sink).map((r) => [r.artifact_id, [r.telemetry_outcome, r.telemetry, Boolean(r.transcript)]])); + assert.deepEqual(byId, { 81: ["unreadable", null, true], 82: ["unreadable", null, true] }); +}); + test("telemetrySpans counts spans across lines and skips a line that is not JSON", () => { const trace = "a".repeat(32); assert.deepEqual(telemetrySpans(`not json\n${telemetryLines([trace, "b".repeat(32)])}`, trace), { telemetry_spans: 2, telemetry_joined_spans: 1 }); diff --git a/scripts/lane-transcripts/store.mjs b/scripts/lane-transcripts/store.mjs index bcd69bdae84..6d886e7be9e 100644 --- a/scripts/lane-transcripts/store.mjs +++ b/scripts/lane-transcripts/store.mjs @@ -125,7 +125,11 @@ export const telemetryPath = (row) => transcriptPath(row).replace(/\.json\.gz$/, export function telemetrySpans(text, traceId) { let spans = 0; let joined = 0; - for (const line of text.split("\n")) { + for (let at = 0; at < text.length; ) { + const end = text.indexOf("\n", at); + const line = text.slice(at, end === -1 ? text.length : end); + at = end === -1 ? text.length : end + 1; + if (line.trim() === "") continue; let request; try { request = JSON.parse(line); From 41ce1f0af989cb78aa274c8d35dc055fcf67eb33 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Thu, 19 Nov 2026 02:31:02 -0500 Subject: [PATCH 3/3] docs(lane-transcripts): define unreadable telemetry to cover extraction failure A telemetry entry that unzip cannot extract also yields `unreadable` without ever being opened, so the contract row and the openTelemetry JSDoc now say so. Co-Authored-By: Claude Opus 5.5 --- docs/contracts/lane-transcripts/README.md | 2 +- scripts/lane-transcripts/pull.mjs | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/contracts/lane-transcripts/README.md b/docs/contracts/lane-transcripts/README.md index ef27e89b934..6511430cb06 100644 --- a/docs/contracts/lane-transcripts/README.md +++ b/docs/contracts/lane-transcripts/README.md @@ -73,7 +73,7 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`. | `transcript` | The transcript's path relative to the sink, or null once retention removed it | yes | | `pruned` | Why `transcript` is null (`age`, `cap`, `missing`), absent while it is set | yes | | `telemetry` | The leg's telemetry path relative to the sink, or null when the leg kept none, it did not open, or retention removed it | yes | -| `telemetry_outcome` | `stored`, `none` (the leg kept no telemetry), `sealed_mismatch` (it did not open) or `unreadable` (it opened but did not decompress); absent on a row written before the field existed | yes | +| `telemetry_outcome` | `stored`, `none` (the leg kept no telemetry), `sealed_mismatch` (it did not open) or `unreadable` (it could not be extracted from the zip, or it opened but did not decompress within its cap); absent on a row written before the field existed | yes | | `telemetry_spans`, `telemetry_joined_spans` | How many spans the telemetry holds, and how many carry the row's `trace_id`; null unless `telemetry_outcome` is `stored` | yes | A reader asking for `--outcome error` reads `is_error`, not `outcome`: a row stored as `success` diff --git a/scripts/lane-transcripts/pull.mjs b/scripts/lane-transcripts/pull.mjs index 4bf43194ab8..d83e9fdca7d 100644 --- a/scripts/lane-transcripts/pull.mjs +++ b/scripts/lane-transcripts/pull.mjs @@ -395,7 +395,8 @@ const NO_TELEMETRY = { telemetry: null, telemetry_bytes: null, telemetry_spans: * The leg's telemetry, opened under its own payload name and the transcript's * run and job binding, and the row fields it gives. `telemetry_outcome` is * `none` (no file), `sealed_mismatch` (it did not open), `unreadable` (it - * opened but is not gzip) or `stored`. + * could not be extracted from the zip, or it opened but did not decompress + * within its cap) or `stored`. */ function openTelemetry(recipient, record, runId, jobId, traceId) { if (!record) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "none" } };