Draft: dispatch immutable Benchmarks release images

[borgesius]

Summary

Retain the exact runner and API digests emitted by each build and dispatch those immutable @sha256: references to Benchmark Infra. Release/version tags remain metadata only.

The caller discovers the infra run by the exact release correlation and fails closed if that run is absent; it cannot treat an unrelated recent success as this transaction.

Dependency

Consumer cutover and backwards-compatible tag-to-digest resolution are consolidated in coval-ai/benchmark-infra#91. The combined Infra draft safely accepts the current tag caller until this PR lands, then uses the already-retained digests directly.

Safety and validation

• no build, image push, dispatch, deployment, or marker update was triggered during validation;
• focused release workflow contracts, Ruff, mypy, actionlint, and prior hosted CodeQL/Switchboard checks are green;
• exact-head workflow-dispatch run 31285901223 passed three consecutive complete attempts; the PR remains draft and no release path was invoked.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b8c4c2c-e3da-4770-8f06-b4ba2a86a760

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[callumreid]

Both release paths — runner-release.yml (Switchboard dispatch) and tag.yml (v* tag push) — now capture the exact manifest digest each docker/build-push-action emits, pass it between jobs via a short-lived artifact, and dispatch .../coval-bench-{runner,api}@sha256:... to benchmark-infra instead of a mutable :<sha> / :<version> tag. The correlation-matching loop in dispatch_infra loses its "newest recent run" fallback and now fails closed.

What I verified against the tree

• The correlation match is exact, not lucky. benchmark-infra's bump-images.yml on main sets run-name: format('Image bump {0} {1}', sha||ref||run_id, switchboard_correlation_id||run_id), so a dispatched run's displayTitle genuinely ends with a space followed by the correlation id. The new endswith(" " + $correlation) predicate is therefore correct, and tightening contains → endswith also closes the case where one correlation id is a substring of another. Dropping the gh run view --log grep removes a 30-run log download per poll iteration — a real cost win, not just cleanup.
• Removing the fallback is the point of the PR and is safe. The old branch adopted the newest repository_dispatch run after attempt 6, which — under benchmark-infra's bump-images concurrency group — could attach gh run watch to an unrelated rollout and then fast-forward production-runner on its result. New behaviour is 30x5s of polling then a hard exit 1, so promote_marker never runs (it gates on needs.dispatch_infra.result == 'success'). Strictly fail-closed.
• Digest plumbing holds. Both matrix legs are runner and api; artifact names are per-image and per-workflow (benchmarks-image-digest-* vs benchmarks-tag-image-digest-*), so a concurrent tag release cannot cross-contaminate; if-no-files-found: error prevents a silently empty download; the ^sha256:[0-9a-f]{64}$ check is applied at both the producer and the consumer. resolve_images' permissions: contents: read is sufficient — same-run download-artifact@v4 uses the Actions runtime token, not actions: read.
• Skip path preserved. With context.outputs.skip == 'true', build_and_push is skipped, so resolve_images is skipped, so dispatch_infra's needs.resolve_images.result == 'success' is false. Same behaviour as the previous build_and_push.result gate. summarize still uses if: always() with the new job added to needs.
• Test file resolves correctly. runner/tests/unit/test_release_workflow_contracts.py uses parents[3], which from runner/tests/unit/<file> is the repo root. The negative assertions (fallback_run_id, newest repository_dispatch run, gh run view "$candidate_id" all absent) are the useful ones — they would fail if someone reintroduced the ambiguous-run fallback. These are text-matching contract tests rather than behavioural ones, which is the right tool here since GitHub Actions expressions aren't otherwise executable in CI.
• Claimed validation run is real. Run 31285901223 is workflow_dispatch, conclusion: success, on head SHA e865d9b2 exactly — and it is the Switchboard workflow, consistent with the body's claim that no release path was invoked.

Known gate (not a reason to withhold approval, but do not merge before it clears): benchmark-infra main currently validates the incoming payload with ^...coval-bench-runner:[A-Za-z0-9._-]{1,128}$ — a colon-tag-only regex that rejects an @sha256: reference outright. Merging this PR before coval-ai/benchmark-infra#91 lands would make every dispatch fail at "Validate payload". The PR body names this dependency and the PR is correctly still a draft, so this is ordering, not a defect. Land infra#91 first, then this.

Clean, well-scoped supply-chain hardening. CI is fully green on the head SHA (Analyze Python, Runner CI, CodeQL, Plan checks, Switchboard Gate); CodeRabbit is skipped only because the PR is a draft.

[callumreid]

P3 - context job's runner_image / api_image outputs are now dead, and its step summary still advertises them as the deployed images.

These two lines were the last consumers of needs.context.outputs.runner_image / .api_image. After this change nothing reads them, but the context job still computes both :<deploy_sha> tag refs, still exports them as job outputs, and still writes them to $GITHUB_STEP_SUMMARY as - Runner image: / - API image:. Someone doing release forensics from the run summary will read a mutable tag reference that is no longer what was dispatched to benchmark-infra — the summarize job reports only job results, not the resolved digests, so the tag refs are the only image strings a reader sees.

Existing checks miss this because dead workflow outputs are not a lint failure and test_release_workflow_contracts.py only asserts on the new digest wiring, never on what context still emits.

Smallest correction: drop runner_image / api_image from the context job's outputs: and from its step-summary block, and instead echo ${{ needs.resolve_images.outputs.runner_image }} / .api_image in the summarize job so the summary names the reference that was actually deployed. Non-blocking — nothing behaves incorrectly today.