Skip to content

ci(lanes): keep the lane sweep's telemetry through the sealed receiver - #17400

Merged
kyle-sexton merged 2 commits into
mainfrom
ci/lane-otel-receiver
Nov 19, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
ci/lane-otel-receiver

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Refs: #17261

Summary

Opts the lane sweep (maintenance-audit-lane-runs.yml, its three assess-units legs) into run-claude-cli's sealed telemetry receiver, and teaches this repository's upload-lane-transcript to carry the leg's otel-file beside the transcript.

The re-pin half of the issue is already on main: the scheduled re-pin job merged #17376, which moved every melodic-software/ci-workflows action pin from v0.75.0 to v0.76.3, a release after v0.76.0 that carries ci-workflows#1136. This PR does not touch a pin. The one v0.75.0 pin left, check-managed-files in pr-check-managed-files-hosted.yml, is left to the standards sync, as the re-pin job states.

Fix

  • upload-lane-transcript takes an optional otel-file (empty by default, so every other caller is unchanged). After the transcript is sealed, the telemetry gets its own scan and its own seal, both continue-on-error. It is sealed under the payload name otel.jsonl.gz to the same key, run id and job id, and uploaded in the same artifact as otel.jsonl.gz.sealed. A scan hit, a seal failure, or an artifact that would pass the puller's 64 MiB limit leaves the telemetry out with an annotation, and the transcript still uploads. This follows the ci-workflows copy of the action at v0.76.3. The local copy is extended because every lane here uploads through it, and callers.test.mjs matches only the local path, so switching one lane to the ci-workflows copy would add a second way to upload.

  • scan-upload takes an optional SCAN_WITHHOLDS. When the telemetry scan finds something, the error line says the telemetry is left out and the transcript still uploads, not "nothing is uploaded".

  • The lane. maintenance-audit-lane-runs.yml sets otel-receiver: "true" on its key, subscription and fallback assess-units legs, and passes each leg's own otel-file to its upload. Only one leg judges each unit, so turning on all three keeps a handoff from silently dropping telemetry.

  • Lane choice (the issue leaves it open; the intake triage suggests a low-traffic lane). I picked the lane with the lowest blast radius among the 13 workflows that run run-claude-cli. Its legs:

    • have Read, Grep and Glob only, with no Bash;
    • run with read permissions only;
    • post nothing (a separate job without a model writes the digest);
    • use the default managed settings and sandbox, with no key proxy;
    • run once a day and can be dispatched for the check.

    The other candidates write pull requests or comments (maintenance-tidy-code-upkeep.yml, the pr-* lanes), or run only on an admin dispatch (maintenance-measure-skill-edit.yml, maintenance-verify-whole-map.yml).

  • Trust boundary: unchanged. Basis: ci-workflows run-claude-cli/lane.cjs at v0.76.3, legEnv:

    • With the receiver on, it deletes every inherited OTEL_*, CLAUDE_CODE_OTEL_*, BETA_TRACING_ENDPOINT and ENABLE_BETA_TRACING_DETAILED variable, so OTEL_LOG_USER_PROMPTS and OTEL_LOG_TOOL_DETAILS stay unset and prompt and tool content stay redacted.
    • It points the exporter at a receiver bound to 127.0.0.1 (otlp-receiver.cjs).
    • The lane's apply-claude-managed-settings calls pass no settings, so no managed OTLP endpoint overrides this.

    Telemetry therefore leaves the runner only as ciphertext, through the same scan, key, artifact and 7-day retention as the transcript, to the same puller.

Verification

  • node --test --test-concurrency=4 over the 11 files that read the changed actions or the lane workflow: 1353 tests, 1353 passed, 0 failed. The files: callers.test.mjs, seal-transcript.test.mjs, scan-upload.test.mjs, check-kill-switch.test.mjs, admit.test.mjs, the three maintenance-audit-lane-runs/*.test.mjs, pr-merge/workflow.test.mjs, security-outside-actor/flag.test.mjs and lane-transcripts/pull.test.mjs.
  • New tests:
    • The telemetry seals beside the transcript and opens to the same bytes only under otel.jsonl.gz and its own run.
    • A missing telemetry file seals nothing.
    • A bad key only warns.
    • The artifact size cap leaves the telemetry out.
    • The upload path lists both files.
    • The telemetry scan comes after the transcript seal and before its own seal.
    • SCAN_WITHHOLDS replaces "nothing is uploaded" in the error line.
    • callers.test.mjs checks that every upload's otel-file is the same leg's output, from a launcher with otel-receiver: "true", and that otel-receiver appears only on a launcher.
  • actionlint passes on the lane workflow.
  • Consumers checked:
    • the 13 workflows that call run-claude-cli: none set otel-receiver before this PR, and the input defaults to false;
    • this repository's upload-lane-transcript, scan-upload and seal-lane-output;
    • scripts/lane-transcripts/pull.mjs, which accepts one extra *.sealed entry in an artifact and does not open it.
  • The ci-workflows releases v0.76.0 to v0.76.3 carry no breaking note.

Not verified here: a run of the lane with the receiver on. Its jobs run only on refs/heads/main.

Remaining

  • Teach scripts/lane-transcripts/pull.mjs to open otel.jsonl.gz.sealed and write the telemetry and its trace join to the sink (a follow-up PR from this session).
  • After both merge: dispatch maintenance-audit-lane-runs.yml on main, run the pull, and confirm the sink holds the sealed telemetry and its spans join the row's trace_id.
  • pr-pipeline-sandbox syncs upload-lane-transcript, scan-upload and resolve-config byte for byte (scripts/lanes/check-sync.sh there), so it needs the same files once this merges.

Related

  • ci-workflows#1136: the receiver and the otel-file input of the ci-workflows upload action.
  • #17376: the re-pin to v0.76.3.
  • #17046, #17075: the sink rows' cost, cache tokens, failure class and trace id.

🤖 Generated with Claude Code

upload-lane-transcript takes an optional otel-file: the leg's telemetry is
scanned on its own after the transcript is sealed, sealed under the payload
name otel.jsonl.gz to the same run and job, and uploaded in the same
artifact. A scan hit, a seal failure or an artifact past the puller's
64 MiB limit leaves the telemetry out with an annotation; the transcript
still uploads. scan-upload's SCAN_WITHHOLDS names what a hit withholds.

maintenance-audit-lane-runs.yml's three assess-units legs set
otel-receiver: "true" and pass their otel-file to the upload.
callers.test.mjs checks that every otel-file is its own leg's, from a
launcher with the receiver on.

Refs: #17261

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T06:59:07.263830Z 9e1c0ca Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 8be72d4d9cf59e7287c41fb1800f6dd8bedc6626: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #17400 (incremental, head 8be72d4)

No security issues found. This was an incremental, blocking-only review, because this lane already reviewed head 9e1c0ca2. I read every hunk of both files in scope in .claude-lane/incremental.diff:

  • .github/actions/scan-upload/scan-upload.mjs
  • .github/actions/scan-upload/scan-upload.test.mjs

What changed

When a scan hit withholds a file, the step-summary header now reads ### scan-upload hit: <SCAN_WITHHOLDS> if SCAN_WITHHOLDS is set. Otherwise it still reads ### scan-upload withheld the upload (scan-upload.mjs#L100). The test adds two assertions on that header.

What I checked

  • Where the value comes from. The only place that sets SCAN_WITHHOLDS is a fixed literal in upload-lane-transcript/action.yml#L60: SCAN_WITHHOLDS: the telemetry is left out and the transcript still uploads. No input, pull request text or file content reaches it, so nothing can be injected into the step summary through it.
  • What reaches the summary. The body written under the header is still only withheld.fingerprint, which holds the hit's rule, path and line and no character of the match. The diff does not change that, so no matched secret can reach the summary.
  • Gating is unchanged. The withhold decision, the ::error:: line and the clean=true output that gates the seal are all untouched. The upload stays fail-closed.
  • Trust controls are untouched. The diff does not touch the trusted-trigger check, the lane token, the kill switch, any merge hold or any manifest.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 9e1c0ca2a237cbd7ce333187a62efac729ef00a6; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 8be72d4d9cf59e7287c41fb1800f6dd8bedc6626: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review, incremental (9e1c0ca → 8be72d4)

No new findings. I read every hunk of both files in scope: .github/actions/scan-upload/scan-upload.mjs and .github/actions/scan-upload/scan-upload.test.mjs. This lane already reviewed an earlier head, so this pass looked for blocking findings only.

What changed: the step-summary header in main() now uses the same SCAN_WITHHOLDS value as the annotation, ### scan-upload hit: <SCAN_WITHHOLDS>. When the variable is unset, the header is still ### scan-upload withheld the upload.

This fixes the earlier finding (discussion_r4625386841), which the owner's reply on that thread marks as fixed in this commit.

What I checked by reading the code (no tests or commands run):

  • The new test reads the real summary. The run() helper writes the step summary to a file and returns its contents as result.summary (scan-upload.test.mjs lines 23–30). The new assertions therefore check the header that actually gets written:
    • the ^### scan-upload hit: … match;
    • the absence of withheld the upload.
  • The unset case is still covered. The existing exact-equality tests (lines 83, 101, 117, 127, 136 and 146) still expect ### scan-upload withheld the upload, so the default wording is still tested.
  • The inserted value is fixed text. SCAN_WITHHOLDS comes from the action's own step env, not from the scanned content. The header only adds that text and adds no character of a match.
  • Author's claim: the owner reply says 13 of 13 tests pass. That is author-claimed and unverified here.

Closing issue: the PR says Refs: #17261 and has no closing keyword, so I checked no acceptance criteria.

Loosened checks: none. Both test edits only add assertions.

Loosened-check verdict: fail. That is the default, because the prompt had no loosened_check_verdict: or loosened_check_patterns: line. Patterns: 0. The prompt also had no loosened_check_decision_record: line, so I did not check the citation requirement.

Verifier dropped 0 of 0 candidates.

With SCAN_WITHHOLDS set, the summary header said the whole upload was
withheld while the transcript still uploads. Build the header from the
same value as the annotation, and assert on it in the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 26bb3d7 Nov 19, 2026
58 checks passed
@kyle-sexton
kyle-sexton deleted the ci/lane-otel-receiver branch November 19, 2026 08:15
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant