Part H — CI action: run the corpus against one backend without cloning - #10
Merged
Merged
Conversation
This was referenced Sep 9, 2026
`action.yml` — a composite GitHub Action a backend maintainer adds to their own CI, taking `backend` (a name from this repository's own `backends/`, or a path to an adapter already checked out in the caller's repository), `suite`, `url` of the already-running backend, and `version` (a release tag, default `latest`). It fetches the corpus (cases/, built-in backends/) at that same tag, downloads the matching runner binary, runs the suite, and writes a job summary — never failing the job on a verdict, only on a harness error. `.github/workflows/release.yml`: on a `v*` tag, builds and attaches `specmatrix` for `x86_64-unknown-linux-gnu` and `aarch64-unknown-linux-gnu` via `taiki-e/upload-rust-binary-action`, matching this repository's existing convention of pinning every action to a commit, not a tag. `allow:` on `Backend` (`src/backend.rs`): case ids a maintainer has read and accepted, each with a one-line reason. It never changes a verdict — `CheckResult` now carries an `allowed_reason` alongside the verdict the runner already decided from the wire, so a badly behaved backend cannot turn its own `ALTER` into a `PASS` by adding an entry. What it changes is only `tools/summarize.py`'s job-summary output: allowed rows move to their own section instead of reading as unreviewed failures. `.github/workflows/dogfood.yml`: runs the action against every (backend, suite) pair this repository declares (`tools/list_backend_protocols.py` reads them from the adapters themselves, so a new one is picked up without editing a workflow), on a pull request touching `cases/` or `backends/` — using two inputs that exist only for this workflow, `ref` and `binary-path`, to run the pull request's own corpus and binary rather than the last published release, so a wrong case goes red before it ships. Scoped to same-repository pull requests: the corpus-fetch step asks this repository for the pull request's SHA, which a fork's commits do not exist here until pushed to it. Verified: the action's core logic (bare-name and path-form `backend` inputs, binary-path override, summary generation) run by hand against a real Loki container, matching a `specmatrix run --json` output byte for byte with what `tools/summarize.py` expects. 168 tests, 0 clippy warnings, corpus gate passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK
Two bugs, both only visible once the dogfood workflow actually ran:
- "resolve version" always called the GitHub releases API, even when
`ref`/`binary-path` (the two inputs that exist only for the dogfood
workflow) make that lookup unneeded. This repository has never pushed a
release tag yet, so `releases/latest` 404'd and every dogfood job failed
before it reached the suite. Guarded the step to skip entirely when both
overrides are given.
- The "run the suite" step referenced
`.specmatrix-corpus-${{ github.action_ref || 'action' }}`, left over from
an earlier revision; the checkout step writes to the plain
`.specmatrix-corpus` an earlier edit renamed it to, but a replace_all at
the time matched only the checkout step's own `path:` line, not this
second occurrence — so the two never agreed and `--cases` would have
pointed at a directory that does not exist. Never caught because the CLI
logic was dry-run by hand against a hardcoded path rather than through the
action script itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK
…el legs One leg hit the 20-minute limit purely from queueing behind the same shared Rust build cache as the other 19; nothing was actually hung. Measured: the slowest legitimate leg took 17m3s in the same run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK
DeviousCardi
force-pushed
the
ci-action
branch
from
September 9, 2026 08:37
6becf4a to
3bfd969
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part H of PLAN-1.0.md: a composite GitHub Action a backend maintainer can add
to their own CI, plus the release and dogfood workflows around it.
action.yml— takesbackend(a name from this repository's ownbackends/, orpath/to/adapter.yamlfor one not carried here),suite,url, andversion(release tag, defaultlatest). Fetches the corpus atthat tag, downloads the matching binary, runs the suite, writes a job
summary. Fails the job only on a harness error — a verdict never does.
.github/workflows/release.yml— on av*tag, builds and attachesspecmatrixforx86_64-unknown-linux-gnuandaarch64-unknown-linux-gnu.allow:on an adapter (src/backend.rs) — case ids a maintainer hasread and accepted, each with a reason. Never changes a verdict (the runner
decides that from the wire, unchanged);
CheckResultnow carries thereason alongside it, and
tools/summarize.py's job summary moves thoserows to their own section instead of reading as unreviewed failures.
.github/workflows/dogfood.yml— runs the action against every(backend, suite) pair this repository declares, on a pull request touching
cases/orbackends/, using that pull request's own binary and corpus(
ref/binary-path, two inputs that exist only for this workflow) ratherthan the last published release — so a wrong case goes red before it ships.
Scoped to same-repository pull requests, since the corpus-fetch step needs
the PR's SHA to already exist in this repository.
Verified by hand against a real Loki container: both
backendinput forms(bare name and path), the
binary-pathoverride, and the summary script'soutput — matched byte for byte against a real
specmatrix run --json.168 tests, 0 clippy warnings, corpus gate passes.
🤖 Generated with Claude Code
https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK