Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions docs/contracts/lane-transcripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`.
<sink>/metrics.jsonl one JSON row per leg, no transcript text
<sink>/outcomes.jsonl one JSON row per lane-outcome record
<sink>/transcripts/<YYYY-MM-DD>/<lane>/<file>.json.gz one leg's transcript
<sink>/transcripts/<YYYY-MM-DD>/<lane>/<file>.jsonl.gz one leg's telemetry, when the leg kept it
<sink>/state.json the writer's own state; readers ignore it
```

Expand All @@ -37,8 +38,13 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`.
less of GitHub's 1 MB per job (counted at two bytes a character); one over its cap, or one the
credential scan blocks, arrives as a marker, never cut short. When the other outputs leave no
room even for a marker, nothing is carried.
- The file name under a lane's date directory is the writer's; readers take a transcript's path
from its row, never by building it.
- A leg's telemetry is gzipped JSONL, one OTLP/JSON export request per line, traces and logs
together; a line's top-level `resourceSpans` or `resourceLogs` key says which. The writer
(ci-workflows run-claude-cli's `otel-receiver`) leaves the CLI's redaction of prompt and tool
content on.
- **Pointer**: for the line shapes, read the [OTLP specification](https://opentelemetry.io/docs/specs/otlp/).
- The file name under a lane's date directory is the writer's; readers take a transcript's or
telemetry's path from its row, never by building it.

## Row fields

Expand Down Expand Up @@ -66,6 +72,9 @@ The current reader is the session-flow plugin's `scripts/lane_legs.py`.
| `failure_class_source` | Which case set `failure_class`: `lane-outcome` (the records agree), `disagree` (they differ) or `none` (no record held) | yes |
| `transcript` | The transcript's path relative to the sink, or null once retention removed it | yes |
| `pruned` | Why `transcript` is null (`age`, `cap`, `missing`), absent while it is set | yes |
| `telemetry` | The leg's telemetry path relative to the sink, or null when the leg kept none, it did not open, or retention removed it | yes |
| `telemetry_outcome` | `stored`, `none` (the leg kept no telemetry), `sealed_mismatch` (it did not open) or `unreadable` (it could not be extracted from the zip, or it opened but did not decompress within its cap); absent on a row written before the field existed | yes |
| `telemetry_spans`, `telemetry_joined_spans` | How many spans the telemetry holds, and how many carry the row's `trace_id`; null unless `telemetry_outcome` is `stored` | yes |

A reader asking for `--outcome error` reads `is_error`, not `outcome`: a row stored as `success`
with `is_error: true` is an error, and is not a `success`.
Expand All @@ -80,7 +89,7 @@ neither field below. A writer whose keys follow another scheme writes `leg_kind`
### GitHub only

`artifact_id`, `run_id` (the integer behind `run_ref`), `job_id` (the same value as `leg_key`),
`workflow`, `event`, `head_branch`, `run_conclusion`, `transcript_bytes` and `pulled_at`. A row
`workflow`, `event`, `head_branch`, `run_conclusion`, `transcript_bytes`, `telemetry_bytes` and `pulled_at`. A row
written before the provider-neutral fields existed has only these: a reader takes `run_ref` from
`run_id`, `leg_key` from `job_id`, `provider` as `github` and `run_url` as null.

Expand Down
18 changes: 13 additions & 5 deletions scripts/lane-transcripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ machine or object storage, is any directory it mounts at; nothing else changes.

```text
<sink>/transcripts/<YYYY-MM-DD>/<lane>/<run id>-<attempt>-<job id>.json.gz the CLI messages, gzipped
<sink>/transcripts/<YYYY-MM-DD>/<lane>/<run id>-<attempt>-<job id>.otel.jsonl.gz the leg's telemetry, gzipped
<sink>/metrics.jsonl one row per transcript, no content
<sink>/outcomes.jsonl one row per lane-outcome record
<sink>/state.json last success, last attempt, last error kind
Expand Down Expand Up @@ -93,10 +94,16 @@ listed newest first, back to the last successful pull less 6 hours (7 days on th
re-run attempt of an older run is still pulled. It skips an artifact it already holds, an expired one, one over 64 MiB, and one whose
run's head repository is not this repository: the public key is public, so a fork's run could seal
a record. A zip must hold `transcript.json.gz.sealed` and at most one other entry, which must be
a sealed file (`<name>.sealed`, such as a leg's telemetry); the pull opens only the transcript,
and any other zip counts as `bad_zip`. A record that does not open for the
key, the artifact's run id and its job id (moved, tampered with, or sealed under a rotated key) is
counted as `sealed_mismatch` and skipped. The summary line counts each kind.
a sealed file (`<name>.sealed`); any other zip counts as `bad_zip`. A transcript record that does
not open for the key, the artifact's run id and its job id (moved, tampered with, or sealed under a
rotated key) is counted as `sealed_mismatch` and skipped. The summary line counts each kind.

When the other entry is `otel.jsonl.gz.sealed` (`upload-lane-transcript`'s `otel-file`), the pull
opens it under the payload name `otel.jsonl.gz` with the same run and job binding and stores it
beside the transcript. The row's `telemetry_outcome` says what happened (`stored`, `none`,
`sealed_mismatch`, `unreadable`); a telemetry record that does not open keeps the transcript.
`telemetry_spans` counts the spans and `telemetry_joined_spans` those whose `traceId` is the row's
`trace_id`, so a row shows whether the leg's spans joined its run attempt.

A `gh` call that fails is retried, the wait doubling from 2 seconds: a listing page gets 10 tries with the wait capped at 5 minutes, any other call 5 tries. An artifact whose download,
unzip or run read still fails counts as `fetch_failed` and the pull goes on. It has no metrics row
Expand Down Expand Up @@ -145,7 +152,8 @@ resuming at that workflow. It records each run attempt it finished reading as `r

Every pull, and `prune`, applies it in this order:

1. A transcript over 30 days old is deleted; its row stays, with `transcript: null`.
1. A transcript over 30 days old is deleted; its row stays, with `transcript: null`. Its
telemetry goes with it (`telemetry: null`).
2. A metrics row, or a lane-outcome record, over 365 days old is dropped.
3. While every file under the sink totals more than 5 GB (5,000,000,000 bytes), the oldest
transcript is deleted.
Expand Down
61 changes: 53 additions & 8 deletions scripts/lane-transcripts/pull.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ import {
readState,
sinkBytes,
staleness,
telemetryPath,
telemetrySpans,
traceIdOf,
transcriptMetrics,
transcriptPath,
Expand All @@ -43,12 +45,16 @@ import {
export const REPO = "melodic-software/claude-code-plugins";
export const PAYLOAD = "transcript.json.gz";
export const ENTRY = `${PAYLOAD}.sealed`;
export const TELEMETRY_PAYLOAD = "otel.jsonl.gz";
export const TELEMETRY_ENTRY = `${TELEMETRY_PAYLOAD}.sealed`;
const HOUR = 60 * 60 * 1000;
const ARTIFACT_DAYS = 7;
const OVERLAP = 6 * HOUR;
const PER_PAGE = 100;
const MAX_SEALED = 64 * 1024 * 1024;
const MAX_PLAIN = 256 * 1024 * 1024;
const MAX_TELEMETRY_PLAIN = 64 * 1024 * 1024;
const UNEXTRACTED = Symbol("unextracted");
export const GH_ATTEMPTS = 5;
export const GH_LIST_ATTEMPTS = 10;
export const GH_DELAY_MS = 2000;
Expand Down Expand Up @@ -209,15 +215,15 @@ function fetched(kind, call) {

/**
* A second sealed file the writer may add beside the transcript, such as the
* leg's telemetry; the pull does not open it.
* leg's telemetry; the pull opens only the telemetry.
*/
const EXTRA_SEALED = /^[A-Za-z0-9][A-Za-z0-9._-]{0,99}\.sealed$/;

/**
* The sealed record inside one artifact zip: the sealed transcript, and at
* most one other sealed file.
* The sealed records inside one artifact zip: the sealed transcript, and the
* telemetry when it is the one other sealed file.
*/
function sealedRecord(deps, sink, zip) {
function sealedRecords(deps, sink, zip) {
const dir = mkdtempSync(path.join(paths(sink).tmp, "zip-"));
try {
const file = path.join(dir, "artifact.zip");
Expand All @@ -226,7 +232,13 @@ function sealedRecord(deps, sink, zip) {
const entries = unzip(["-Z1", file]).toString("utf8").split("\n").filter(Boolean);
const extra = entries.filter((entry) => entry !== ENTRY);
if (entries.length - extra.length !== 1 || extra.length > 1 || !extra.every((entry) => EXTRA_SEALED.test(entry))) return null;
return unzip(["-p", file, ENTRY]);
const transcript = unzip(["-p", file, ENTRY]);
if (extra[0] !== TELEMETRY_ENTRY) return { transcript, telemetry: null };
try {
return { transcript, telemetry: deps.unzip(["-p", file, TELEMETRY_ENTRY]) };
} catch {
return { transcript, telemetry: UNEXTRACTED };
}
} finally {
rmSync(dir, { recursive: true, force: true });
}
Expand Down Expand Up @@ -323,11 +335,11 @@ function pullWithin({ sink, repo, recipient, deps, log }) {

function store({ sink, repo, recipient, deps, artifact, named, runs, byTrace, now }) {
const runId = String(artifact.workflow_run.id);
const record = sealedRecord(deps, sink, fetched("artifact-zip", () => ghRetried(deps, ["api", `repos/${repo}/actions/artifacts/${artifact.id}/zip`])));
if (!record) return "bad_zip";
const records = sealedRecords(deps, sink, fetched("artifact-zip", () => ghRetried(deps, ["api", `repos/${repo}/actions/artifacts/${artifact.id}/zip`])));
if (!records) return "bad_zip";
let gz;
try {
gz = openRecord(recipient, [recipient.keyId], PAYLOAD, record, runId, named.jobId);
gz = openRecord(recipient, [recipient.keyId], PAYLOAD, records.transcript, runId, named.jobId);
} catch (error) {
if (error instanceof OpenError) return "sealed_mismatch";
throw error;
Expand Down Expand Up @@ -367,10 +379,43 @@ function store({ sink, repo, recipient, deps, artifact, named, runs, byTrace, no
const file = path.join(sink, row.transcript);
ensureDir(path.dirname(file));
writeFileSync(file, gz, { mode: 0o600 });
const telemetry = openTelemetry(recipient, records.telemetry, runId, named.jobId, row.trace_id);
Object.assign(row, telemetry.fields);
if (telemetry.gz) {
row.telemetry = telemetryPath(row);
writeFileSync(path.join(sink, row.telemetry), telemetry.gz, { mode: 0o600 });
}
appendRow(sink, row);
return "pulled";
}

const NO_TELEMETRY = { telemetry: null, telemetry_bytes: null, telemetry_spans: null, telemetry_joined_spans: null };

/**
* The leg's telemetry, opened under its own payload name and the transcript's
* run and job binding, and the row fields it gives. `telemetry_outcome` is
* `none` (no file), `sealed_mismatch` (it did not open), `unreadable` (it
* could not be extracted from the zip, or it opened but did not decompress
* within its cap) or `stored`.
*/
function openTelemetry(recipient, record, runId, jobId, traceId) {
if (!record) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "none" } };
if (record === UNEXTRACTED) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "unreadable" } };
let gz;
try {
gz = openRecord(recipient, [recipient.keyId], TELEMETRY_PAYLOAD, record, runId, jobId);
} catch (error) {
if (error instanceof OpenError) return { fields: { ...NO_TELEMETRY, telemetry_outcome: "sealed_mismatch" } };
throw error;
}
try {
const text = gunzipSync(gz, { maxOutputLength: MAX_TELEMETRY_PLAIN }).toString("utf8");
return { gz, fields: { telemetry_bytes: gz.length, ...telemetrySpans(text, traceId), telemetry_outcome: "stored" } };
} catch {
return { fields: { ...NO_TELEMETRY, telemetry_outcome: "unreadable" } };
}
}

/**
* Each completed run of every lane workflow created since the cutoff, newest
* first per workflow, starting at `at.workflow` (where a budget-stopped pull
Expand Down
93 changes: 90 additions & 3 deletions scripts/lane-transcripts/pull.test.mjs
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import { cpSync, existsSync, readdirSync, readFileSync, statSync, utimesSync, writeFileSync, mkdirSync } from "node:fs";
import { cpSync, existsSync, readdirSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync, mkdirSync } from "node:fs";
import path from "node:path";
import { test } from "node:test";
import { gunzipSync, gzipSync } from "node:zlib";

import { generate } from "../../.github/actions/open-lane-output/keygen.mjs";
import { parsePublicKey, sealRecord } from "../../.github/actions/seal-lane-output/seal.mjs";
import { tempDir } from "../../lib/test-tmp.mjs";
import { ENTRY, LANE_WORKFLOWS, main, PAYLOAD, REPO, TRANSCRIPT_JOB_IDS, TRANSCRIPT_NAMES } from "./pull.mjs";
import { CAP_BYTES, parseArtifactName, paths, prune, query, readOutcomes, readRows, traceIdOf, transcriptMetrics, writeRows } from "./store.mjs";
import { ENTRY, LANE_WORKFLOWS, main, PAYLOAD, REPO, TELEMETRY_ENTRY, TELEMETRY_PAYLOAD, TRANSCRIPT_JOB_IDS, TRANSCRIPT_NAMES } from "./pull.mjs";
import { CAP_BYTES, parseArtifactName, paths, prune, query, readOutcomes, readRows, telemetrySpans, traceIdOf, transcriptMetrics, writeRows } from "./store.mjs";

const MARKER = "plaintext-marker-5d21";
const NOW = Date.parse("2026-11-02T12:00:00Z");
Expand Down Expand Up @@ -766,6 +766,93 @@ test("a zip holding a second sealed file beside the transcript is pulled; a thir
assert.equal(result.code, 0, result.err);
assert.match(result.out, /"pulled":1,.*"bad_zip":3/);
assert.deepEqual(readRows(sink).map((r) => r.artifact_id), [91]);
assert.deepEqual(pickTelemetry(readRows(sink)[0]), { telemetry: null, telemetry_bytes: null, telemetry_spans: null, telemetry_joined_spans: null, telemetry_outcome: "sealed_mismatch" }, "telemetry that does not open keeps the transcript");
});

const pickTelemetry = ({ telemetry, telemetry_bytes, telemetry_spans, telemetry_joined_spans, telemetry_outcome }) => ({ telemetry, telemetry_bytes, telemetry_spans, telemetry_joined_spans, telemetry_outcome });

function telemetryLines(traceIds) {
const spans = traceIds.map((traceId, i) => ({ traceId, spanId: `${i}`.padStart(16, "0"), name: MARKER }));
return `${JSON.stringify({ resourceSpans: [{ scopeSpans: [{ spans }] }] })}\n${JSON.stringify({ resourceLogs: [{ scopeLogs: [{ logRecords: [{ body: { stringValue: MARKER } }] }] }] })}\n`;
}

const sealedTelemetry = (publicLine, text, { runId = 42, jobId = "check-intent", payload = TELEMETRY_PAYLOAD } = {}) => sealRecord(parsePublicKey(publicLine), payload, gzipSync(Buffer.from(text)), String(runId), jobId);

test("a pull opens the sealed telemetry beside the transcript into the sink and counts the spans that join the row's trace id", () => {
const { publicLine, privateText } = generate();
const sink = sinkDir();
const trace = traceIdOf(42, 1);
const text = telemetryLines([trace, trace.toUpperCase(), "0".repeat(32)]);
const deps = fakeDeps([artifact(95, { entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, text) } })]);
const result = run(["pull", "--sink", sink], { env: { LANE_TRANSCRIPT_PRIVATE_KEY: privateText }, deps });
assert.equal(result.code, 0, result.err);
for (const out of [result.out, result.err]) assert.ok(!out.includes(MARKER));
const [row] = readRows(sink);
assert.equal(row.trace_id, trace);
const { telemetry, telemetry_bytes: bytes, ...counts } = pickTelemetry(row);
assert.deepEqual(counts, { telemetry_spans: 3, telemetry_joined_spans: 2, telemetry_outcome: "stored" });
assert.equal(telemetry, row.transcript.replace(/\.json\.gz$/, ".otel.jsonl.gz"));
const file = path.join(sink, telemetry);
assert.equal(gunzipSync(readFileSync(file)).toString("utf8"), text);
assert.equal(bytes, statSync(file).size);
assert.equal(statSync(file).mode & 0o777, 0o600);
assert.ok(!JSON.stringify(row).includes(MARKER), "the metrics row holds no telemetry text");
});

test("telemetry bound to another job or payload name is not stored, and a transcript alone records none", () => {
const { publicLine, privateText } = generate();
const sink = sinkDir();
const text = telemetryLines([traceIdOf(42, 1)]);
const page = [
artifact(96, { entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, text, { jobId: "other" }) } }),
artifact(97, { attempt: 2, entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, text, { payload: PAYLOAD }) } }),
artifact(98, { attempt: 3, record: sealed(publicLine) }),
];
assert.equal(run(["pull", "--sink", sink], { env: { LANE_TRANSCRIPT_PRIVATE_KEY: privateText }, deps: fakeDeps(page) }).code, 0);
const byId = Object.fromEntries(readRows(sink).map((r) => [r.artifact_id, r.telemetry_outcome]));
assert.deepEqual(byId, { 96: "sealed_mismatch", 97: "sealed_mismatch", 98: "none" });
assert.ok(readRows(sink).every((r) => r.telemetry === null && r.transcript));
});

test("telemetry that cannot be extracted or is over its size cap is unreadable, and the transcript is kept", () => {
const { publicLine, privateText } = generate();
const sink = sinkDir();
const page = [
artifact(81, { entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: sealedTelemetry(publicLine, "\n".repeat(64 * 1024 * 1024 + 1)) } }),
artifact(82, { attempt: 2, entries: { [ENTRY]: sealed(publicLine), [TELEMETRY_ENTRY]: "z" } }),
];
const deps = fakeDeps(page);
const unzip = deps.unzip;
deps.unzip = (args) => {
if (args[2] === TELEMETRY_ENTRY && JSON.parse(readFileSync(args[1], "utf8"))[TELEMETRY_ENTRY] === Buffer.from("z").toString("base64")) throw Object.assign(new Error("maxBuffer"), { status: 9 });
return unzip(args);
};
const result = run(["pull", "--sink", sink], { env: { LANE_TRANSCRIPT_PRIVATE_KEY: privateText }, deps });
assert.equal(result.code, 0, result.err);
const byId = Object.fromEntries(readRows(sink).map((r) => [r.artifact_id, [r.telemetry_outcome, r.telemetry, Boolean(r.transcript)]]));
assert.deepEqual(byId, { 81: ["unreadable", null, true], 82: ["unreadable", null, true] });
});

test("telemetrySpans counts spans across lines and skips a line that is not JSON", () => {
const trace = "a".repeat(32);
assert.deepEqual(telemetrySpans(`not json\n${telemetryLines([trace, "b".repeat(32)])}`, trace), { telemetry_spans: 2, telemetry_joined_spans: 1 });
assert.deepEqual(telemetrySpans("", trace), { telemetry_spans: 0, telemetry_joined_spans: 0 });
});

test("prune drops a row's telemetry with its transcript, and a telemetry file gone missing clears its path", () => {
const sink = sinkDir();
const rows = seedStore(sink, [1, 31]);
for (const row of rows) {
row.telemetry = row.transcript.replace(/\.json\.gz$/, ".otel.jsonl.gz");
writeFileSync(path.join(sink, row.telemetry), "t");
}
writeRows(sink, rows);
rmSync(path.join(sink, rows[0].telemetry));
prune(sink, NOW);
const after = readRows(sink);
assert.deepEqual(after.map((r) => [r.telemetry, r.pruned ?? null]), [[null, null], [null, "age"]]);
assert.ok(existsSync(path.join(sink, after[0].transcript)), "the fresh transcript stays");
assert.ok(!existsSync(path.join(sink, rows[1].telemetry)), "the aged telemetry goes with its transcript");
});

test("a pulled row carries its run attempt's trace id, and no failure class while no lane-outcome record is held", () => {
Expand Down
Loading
Loading