From 466dd45062c628e62dc6e6dca27a59cb20a9af00 Mon Sep 17 00:00:00 2001 From: DeviousCardi <115358213+DeviousCardi@users.noreply.github.com> Date: Wed, 9 Sep 2026 13:19:54 +0530 Subject: [PATCH 1/3] CI action: run the corpus against one backend without cloning (Part H) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `action.yml` — a composite GitHub Action a backend maintainer adds to their own CI, taking `backend` (a name from this repository's own `backends/`, or a path to an adapter already checked out in the caller's repository), `suite`, `url` of the already-running backend, and `version` (a release tag, default `latest`). It fetches the corpus (cases/, built-in backends/) at that same tag, downloads the matching runner binary, runs the suite, and writes a job summary — never failing the job on a verdict, only on a harness error. `.github/workflows/release.yml`: on a `v*` tag, builds and attaches `specmatrix` for `x86_64-unknown-linux-gnu` and `aarch64-unknown-linux-gnu` via `taiki-e/upload-rust-binary-action`, matching this repository's existing convention of pinning every action to a commit, not a tag. `allow:` on `Backend` (`src/backend.rs`): case ids a maintainer has read and accepted, each with a one-line reason. It never changes a verdict — `CheckResult` now carries an `allowed_reason` alongside the verdict the runner already decided from the wire, so a badly behaved backend cannot turn its own `ALTER` into a `PASS` by adding an entry. What it changes is only `tools/summarize.py`'s job-summary output: allowed rows move to their own section instead of reading as unreviewed failures. `.github/workflows/dogfood.yml`: runs the action against every (backend, suite) pair this repository declares (`tools/list_backend_protocols.py` reads them from the adapters themselves, so a new one is picked up without editing a workflow), on a pull request touching `cases/` or `backends/` — using two inputs that exist only for this workflow, `ref` and `binary-path`, to run the pull request's own corpus and binary rather than the last published release, so a wrong case goes red before it ships. Scoped to same-repository pull requests: the corpus-fetch step asks this repository for the pull request's SHA, which a fork's commits do not exist here until pushed to it. Verified: the action's core logic (bare-name and path-form `backend` inputs, binary-path override, summary generation) run by hand against a real Loki container, matching a `specmatrix run --json` output byte for byte with what `tools/summarize.py` expects. 168 tests, 0 clippy warnings, corpus gate passes. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK --- .github/workflows/dogfood.yml | 74 +++++++++++++++ .github/workflows/release.yml | 40 +++++++++ AGENTS.md | 40 +++++++++ README.md | 15 ++++ action.yml | 153 ++++++++++++++++++++++++++++++++ src/backend.rs | 11 +++ src/matrix.rs | 1 + src/runner.rs | 84 +++++++++++++++++- tools/list_backend_protocols.py | 27 ++++++ tools/summarize.py | 64 +++++++++++++ 10 files changed, 508 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/dogfood.yml create mode 100644 .github/workflows/release.yml create mode 100644 action.yml create mode 100644 tools/list_backend_protocols.py create mode 100644 tools/summarize.py diff --git a/.github/workflows/dogfood.yml b/.github/workflows/dogfood.yml new file mode 100644 index 0000000..d036c20 --- /dev/null +++ b/.github/workflows/dogfood.yml @@ -0,0 +1,74 @@ +name: dogfood + +# Runs this repository's own `action.yml` against every declared (backend, +# suite) pair, using the pull request's own binary and corpus rather than a +# published release — so a case or adapter that is wrong goes red before it +# reaches a tag, on the columns already committed here. +# +# Only for pull requests on this repository, not forks: the corpus-fetch step +# inside the action asks for the pull request's SHA from this repository, and +# a fork's commits do not exist here until the branch is pushed to it. A fork +# PR still gets the fast static checks in ci.yml on every commit; this workflow +# adds the containers. +on: + pull_request: + paths: + - "cases/**" + - "backends/**" + - "action.yml" + - "tools/summarize.py" + +permissions: + contents: read + +concurrency: + group: dogfood-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + matrix: + if: github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + outputs: + pairs: ${{ steps.pairs.outputs.pairs }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - id: pairs + run: echo "pairs=$(python3 tools/list_backend_protocols.py)" >> "$GITHUB_OUTPUT" + + run: + needs: matrix + if: needs.matrix.outputs.pairs != '[]' + runs-on: ubuntu-latest + timeout-minutes: 20 + strategy: + fail-fast: false + matrix: + pair: ${{ fromJson(needs.matrix.outputs.pairs) }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: dtolnay/rust-toolchain@6bed0761d98439e5a578e2877258200ad565ba87 # stable + - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + - run: cargo build --release + + - name: start ${{ matrix.pair.backend }} + id: up + run: | + url="$(cargo run --release -q -- up --backend ${{ matrix.pair.backend }} \ + | sed -n 's/^.* ready at //p')" + echo "url=$url" >> "$GITHUB_OUTPUT" + + # The action itself, exercised the same way an external caller would + # use it — `ref`/`binary-path` are the two escape hatches that exist + # only for this job, so this run reads the pull request's own corpus + # and binary instead of the last published release. + - uses: ./ + with: + backend: ${{ matrix.pair.backend }} + suite: ${{ matrix.pair.suite }} + url: ${{ steps.up.outputs.url }} + ref: ${{ github.event.pull_request.head.sha }} + binary-path: target/release/specmatrix + + - if: always() + run: cargo run --release -q -- down --backend ${{ matrix.pair.backend }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..b61ff94 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,40 @@ +name: release + +# A maintainer running the CI action, or the corpus by hand, needs a binary +# that is not "clone this repository and build it". This builds one for each +# architecture the action supports and attaches it to the release the tag +# names. +on: + push: + tags: ["v*"] + +permissions: + contents: write + +jobs: + build: + strategy: + fail-fast: false + matrix: + target: [x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu] + runs-on: ubuntu-latest + steps: + # Actions are pinned to a commit, not a tag, the same as every other + # workflow in this repository. + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: dtolnay/rust-toolchain@6bed0761d98439e5a578e2877258200ad565ba87 # stable + with: + targets: ${{ matrix.target }} + - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + with: + key: ${{ matrix.target }} + # Builds (cross-compiling for aarch64 via `cross`, which this action + # drives itself), archives, and uploads to the release the pushed tag + # names, creating it if this is the first artefact for it. + - uses: taiki-e/upload-rust-binary-action@f0d45ae91ee7b8ee928de7a9d04d893a08bcbec6 # v1.30.2 + with: + bin: specmatrix + target: ${{ matrix.target }} + archive: specmatrix-$tag-$target + checksum: sha256 + token: ${{ secrets.GITHUB_TOKEN }} diff --git a/AGENTS.md b/AGENTS.md index 1a1e1d3..3824421 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -247,6 +247,43 @@ Two `container:` fields exist for settings a CLI flag cannot reach: in the API says so — only `docker logs` names the bound address, and every ingest from outside answers a bare connection reset with no HTTP status. +`allow:` names case ids a maintainer has read and accepted, each with a +one-line reason — for a backend maintainer running `action.yml` in their own +CI, not for this repository's own adapters. It never changes a verdict: the +runner decides `PASS`/`REJECT`/`ALTER` from the wire alone, exactly as it +would without the entry. What it changes is only how the action's job summary +presents the row, moving it to its own section instead of reading as an +unreviewed failure — because the job already never fails on a verdict, only +on a harness error. Silence is not acceptance; a case id absent from `allow:` +is printed with the others. + +```yaml +allow: + otlp-logs/body-invalid-utf8: "documented, tracked at our-org/our-store#123" +``` + +## Running the suite without cloning this repository + +`action.yml` at the repository root is a composite GitHub Action a backend +maintainer adds to their own CI, so a commit that breaks conformance is +visible without anyone cloning this repository by hand. It takes `backend` +(a name from this repository's own `backends/`, or a `path/to/adapter.yaml` +already checked out in the caller's own repository, for a backend not carried +here), `suite`, `url` of the already-running backend, and `version` (a +release tag of this project, default `latest`) — pinning `version` pins both +the runner binary and the corpus it is paired with, so a result names an +exact version of both rather than a mix. It fails the job only on a harness +error; verdicts never fail it, which is what makes `allow:` above meaningful +rather than a way to silence CI. + +`.github/workflows/release.yml` builds and attaches the binaries `action.yml` +downloads, for `x86_64-unknown-linux-gnu` and `aarch64-unknown-linux-gnu`, on +every `v*` tag. `.github/workflows/dogfood.yml` runs the action against every +adapter this repository carries, on a pull request that touches `cases/` or +`backends/` — using the pull request's own binary and corpus (`action.yml`'s +`ref`/`binary-path` inputs, which exist only for that workflow) rather than +the last published release, so a wrong case goes red before it ships. + ## Quarterly reruns A matrix without a date is a claim about the past that reads as a claim about @@ -296,6 +333,9 @@ than assert. - Every case cites a rule with a basis (`tools/check_corpus.py`) - Every adapter pins its image and can be started - `cargo audit` +- A pull request touching `cases/` or `backends/` runs `action.yml` against + every adapter, using that pull request's own binary and corpus + (`.github/workflows/dogfood.yml`) ## Never diff --git a/README.md b/README.md index aab5899..01e54d0 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,21 @@ and deletes the stream or index each case uses before running it. Point it at a store you are willing to have written to, and read [`SECURITY.md`](SECURITY.md) first. +## Running it in CI, without cloning this repository + +```yaml +- uses: DeviousCardi/specmatrix@v1.0.0 + with: + backend: loki # or path/to/your-adapter.yaml for one not carried here + suite: otlp-logs + url: http://localhost:3100 +``` + +Fails the job only on a harness error — a verdict never does, so a maintainer +who has read and accepted a divergence names it in their adapter's `allow:` +list rather than the job going red on a result they already know about. See +[`AGENTS.md`](AGENTS.md#running-the-suite-without-cloning-this-repository). + ## Backends | Backend | OTLP logs | Elasticsearch `_bulk` | diff --git a/action.yml b/action.yml new file mode 100644 index 0000000..7206d90 --- /dev/null +++ b/action.yml @@ -0,0 +1,153 @@ +name: specmatrix +description: > + Run a SpecMatrix conformance suite against one running backend, without + cloning this repository. +author: DeviousCardi + +inputs: + backend: + description: > + Which adapter to run. Either a bare name resolved against this + repository's own `backends/.yaml` at `version`, or a path (must + contain a `/`) to an adapter file already checked out in the caller's + own repository — for a backend not carried here. + required: true + suite: + description: Protocol suite, resolved to this repository's own cases//*.yaml at `version`. + required: true + url: + description: Base URL of the already-running backend to test. + required: true + version: + description: > + Release tag of this project to run — pins both the runner binary and + the corpus (cases and built-in adapters) it is paired with, so a result + names an exact version of both. `latest` resolves to the newest + published release. + required: false + default: latest + ref: + description: > + Internal: overrides which commit the corpus (cases/, backends/) is + fetched at, independent of `version`. Used only by this repository's + own dogfood workflow, to run a pull request's own corpus rather than + the last tagged one — every other caller should leave this unset, which + fetches the corpus at `version` like the description above says. + required: false + binary-path: + description: > + Internal: a specmatrix binary already on disk to run instead of + downloading a release. Used only by this repository's own dogfood + workflow, which builds the pull request's own code; every other caller + should leave this unset. + required: false + +outputs: + verdict-summary: + description: One line — counts of pass/reject/alter/n/a — the same line the table prints. + value: ${{ steps.run.outputs.verdict-summary }} + matrix-json: + description: Path to the JSON result this run wrote. + value: ${{ steps.run.outputs.matrix-json }} + +runs: + using: composite + steps: + - name: resolve version + id: version + shell: bash + run: | + set -euo pipefail + version="${{ inputs.version }}" + if [ "$version" = "latest" ]; then + version="$(curl --proto '=https' --tlsv1.2 --silent --show-error --fail \ + https://api.github.com/repos/DeviousCardi/specmatrix/releases/latest \ + | grep -m1 '"tag_name"' | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/')" + if [ -z "$version" ]; then + echo "::error::could not resolve the latest specmatrix release" >&2 + exit 1 + fi + fi + echo "resolved to $version" + echo "version=$version" >> "$GITHUB_OUTPUT" + + # The corpus (cases/, and the built-in backends/) is fetched at the same + # tag as the binary, into a directory the caller's own checkout cannot + # collide with. A result names one version of the runner and one version + # of the corpus it ran, never a mix. + - name: fetch the corpus at that version + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + repository: DeviousCardi/specmatrix + ref: ${{ inputs.ref || steps.version.outputs.version }} + path: .specmatrix-corpus + sparse-checkout: | + cases + backends + sparse-checkout-cone-mode: false + + - name: download the runner binary + if: inputs.binary-path == '' + shell: bash + run: | + set -euo pipefail + version="${{ steps.version.outputs.version }}" + case "$(uname -m)" in + x86_64|amd64) target=x86_64-unknown-linux-gnu ;; + aarch64|arm64) target=aarch64-unknown-linux-gnu ;; + *) echo "::error::no specmatrix release binary for $(uname -m)" >&2; exit 1 ;; + esac + archive="specmatrix-${version}-${target}.tar.gz" + url="https://github.com/DeviousCardi/specmatrix/releases/download/${version}/${archive}" + echo "downloading $url" + curl --proto '=https' --tlsv1.2 --location --silent --show-error --fail "$url" -o "$RUNNER_TEMP/${archive}" + tar -xzf "$RUNNER_TEMP/${archive}" -C "$RUNNER_TEMP" + chmod +x "$RUNNER_TEMP/specmatrix" + echo "$RUNNER_TEMP" >> "$GITHUB_PATH" + + - name: use the given binary + if: inputs.binary-path != '' + shell: bash + run: | + set -euo pipefail + mkdir -p "$RUNNER_TEMP/specmatrix-bin" + cp "${{ inputs.binary-path }}" "$RUNNER_TEMP/specmatrix-bin/specmatrix" + chmod +x "$RUNNER_TEMP/specmatrix-bin/specmatrix" + echo "$RUNNER_TEMP/specmatrix-bin" >> "$GITHUB_PATH" + + - name: run the suite + id: run + shell: bash + run: | + set -euo pipefail + corpus=".specmatrix-corpus-${{ github.action_ref || 'action' }}" + backend_input="${{ inputs.backend }}" + if [[ "$backend_input" == */* ]]; then + # A path into the caller's own checkout: split it into the + # directory `--backends` reads and the bare name `--backend` names, + # since the CLI resolves the two together as /.yaml. + backends_dir="$(dirname "$backend_input")" + backend_name="$(basename "$backend_input" .yaml)" + else + backends_dir="$corpus/backends" + backend_name="$backend_input" + fi + + specmatrix run \ + --backend "$backend_name" \ + --backends "$backends_dir" \ + --suite "${{ inputs.suite }}" \ + --cases "$corpus/cases" \ + --url "${{ inputs.url }}" \ + --json > matrix.json + specmatrix run \ + --backend "$backend_name" \ + --backends "$backends_dir" \ + --suite "${{ inputs.suite }}" \ + --cases "$corpus/cases" \ + --url "${{ inputs.url }}" + + echo "matrix-json=matrix.json" >> "$GITHUB_OUTPUT" + python3 "${{ github.action_path }}/tools/summarize.py" matrix.json >> "$GITHUB_STEP_SUMMARY" + summary="$(python3 "${{ github.action_path }}/tools/summarize.py" matrix.json --line-only)" + echo "verdict-summary=$summary" >> "$GITHUB_OUTPUT" diff --git a/src/backend.rs b/src/backend.rs index 6fdaf26..c1d7d9c 100644 --- a/src/backend.rs +++ b/src/backend.rs @@ -53,6 +53,17 @@ pub struct Backend { /// difference between measuring the backend and measuring a race. pub setup_verify: Option, pub teardown: Option, + /// Case ids a maintainer has read and accepted, each with a one-line + /// reason. Read by the CI action (`Part H` of the 1.0 plan), never by the + /// runner's own verdict logic: a verdict here is decided from the wire + /// alone, the same as every other column, so a badly behaved backend + /// cannot quietly turn its own `ALTER` into a `PASS` by adding an entry. + /// What `allow` changes is only how the action's job summary presents the + /// result — the allowed cases move to their own section instead of + /// reading as unreviewed failures — because the job already never fails + /// on a verdict, only on a harness error. + #[serde(default)] + pub allow: HashMap, } #[derive(Debug, Deserialize, Clone)] diff --git a/src/matrix.rs b/src/matrix.rs index f701f1a..71a41ea 100644 --- a/src/matrix.rs +++ b/src/matrix.rs @@ -269,6 +269,7 @@ mod tests { title: String::new(), verdict, detail: detail.to_string(), + allowed_reason: None, }) .collect(), } diff --git a/src/runner.rs b/src/runner.rs index c207e70..a6cda6b 100644 --- a/src/runner.rs +++ b/src/runner.rs @@ -35,6 +35,12 @@ pub struct CheckResult { pub verdict: Verdict, /// One line saying what happened. Shown beside the verdict. pub detail: String, + /// The maintainer's own reason, from the adapter's `allow:` list, for + /// having read this verdict and accepted it. Never affects the verdict + /// itself — only how a caller such as the CI action chooses to present + /// the result. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub allowed_reason: Option, } #[derive(Debug, Serialize, Deserialize)] @@ -135,6 +141,7 @@ impl Runner { title: case.title.clone(), verdict: Verdict::NotApplicable, detail: reason.clone(), + allowed_reason: self.backend.allow.get(&case.id).cloned(), }, None => self.run_or_report(suite, case), }); @@ -161,6 +168,7 @@ impl Runner { title: case.title.clone(), verdict: Verdict::NotApplicable, detail: format!("harness error: {e:#}"), + allowed_reason: self.backend.allow.get(&case.id).cloned(), }, } } @@ -524,7 +532,14 @@ impl Runner { } fn result(&self, case: &Case, verdict: Verdict, detail: String) -> CheckResult { - CheckResult { id: case.id.clone(), title: case.title.clone(), verdict, detail } + let allowed_reason = self.backend.allow.get(&case.id).cloned(); + CheckResult { + id: case.id.clone(), + title: case.title.clone(), + verdict, + detail, + allowed_reason, + } } /// Variables available to payloads and adapter templates. @@ -1260,6 +1275,73 @@ expect: assert!(result.detail.contains("round trip intact"), "{}", result.detail); } + /// The adapter's `allow:` list is carried onto the result untouched — it + /// never changes the verdict, only names the reason a maintainer already + /// accepted it. A CI action reads this to move the row into its own + /// section rather than reporting it as unreviewed. + #[test] + fn an_allowed_case_id_carries_its_reason_without_changing_the_verdict() { + let adapter: Backend = serde_yaml::from_str( + r#" +name: stub +allow: + otlp-logs/minimal-record: "known, tracked upstream at example.com/issues/1" +protocols: + otlp-logs: + formats: [otlp-json] + ingest: + request: POST /ingest + readback: + request: POST /search + records: /hits + fields: + body: /message + poll: + interval_ms: 10 + timeout_ms: 120 +"#, + ) + .expect("adapter parses"); + let stub = stub::start(vec![ + ("/ingest", vec![Reply::json(200, "{}")]), + ( + "/search", + vec![Reply::json( + 200, + r#"{"hits":[{"message":"TRUNCATED","specmatrix.run":"{{RUNKEY}}"}]}"#, + )], + ), + ]); + let runner = Runner::new(adapter, stub.url.clone(), false).expect("runner builds"); + let case = case_yaml(" readback:\n match: exact\n on: [body]"); + let result = runner.run_case("otlp-logs", &case).expect("no harness error"); + assert_eq!(result.verdict, Verdict::Alter, "detail: {}", result.detail); + assert_eq!( + result.allowed_reason.as_deref(), + Some("known, tracked upstream at example.com/issues/1") + ); + } + + /// A case id absent from `allow:` carries no reason — silence is not + /// acceptance. + #[test] + fn a_case_not_named_in_allow_carries_no_reason() { + let stub = stub::start(vec![ + ("/ingest", vec![Reply::json(200, "{}")]), + ( + "/search", + vec![Reply::json( + 200, + r#"{"hits":[{"message":"specmatrix minimal record","severity":"INFO","specmatrix.run":"{{RUNKEY}}"}]}"#, + )], + ), + ]); + let runner = runner_for(&stub.url, vec![], Reply::json(200, "{}")); + let case = case_yaml(" readback:\n match: exact\n on: [body]"); + let result = runner.run_case("otlp-logs", &case).expect("no harness error"); + assert_eq!(result.allowed_reason, None); + } + /// A value that comes back changed is an ALTER naming both sides, so a /// reader can see what happened without rerunning anything. #[test] diff --git a/tools/list_backend_protocols.py b/tools/list_backend_protocols.py new file mode 100644 index 0000000..63ecae8 --- /dev/null +++ b/tools/list_backend_protocols.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +"""Emit every (backend, suite) pair this repository's adapters declare, as a +JSON array, for the dogfood workflow's matrix. + +Read from the adapters themselves rather than kept as a hand-maintained list: +a new adapter or a new protocol block on an existing one is picked up the next +run without anyone updating a workflow file to match. +""" +import glob +import json + +import yaml + + +def main() -> None: + pairs = [] + for path in sorted(glob.glob("backends/*.yaml")): + with open(path) as handle: + adapter = yaml.safe_load(handle) + name = adapter["name"] + for suite in adapter.get("protocols", {}): + pairs.append({"backend": name, "suite": suite}) + print(json.dumps(pairs)) + + +if __name__ == "__main__": + main() diff --git a/tools/summarize.py b/tools/summarize.py new file mode 100644 index 0000000..0d396b4 --- /dev/null +++ b/tools/summarize.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Render one `specmatrix run --json` outcome as a job-summary table. + +Used by `action.yml`. Every row printed here came from the wire — the +verdict was decided the same way whether or not the case id appears in the +adapter's `allow:` list. What `allow:` changes is only where the row is +printed: reviewed rows move to their own section so a maintainer's dashboard +does not read every accepted, understood divergence as an unreviewed one. +""" +import argparse +import json +import sys + + +def table(rows: list[dict]) -> str: + if not rows: + return "_none_\n" + lines = ["| Check | Verdict | Detail |", "| --- | --- | --- |"] + for row in rows: + detail = row["detail"].replace("|", "\\|").replace("\n", " ") + lines.append(f"| `{row['id']}` | {row['verdict'].upper()} | {detail} |") + return "\n".join(lines) + "\n" + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("matrix_json") + parser.add_argument("--line-only", action="store_true") + args = parser.parse_args() + + with open(args.matrix_json) as handle: + outcome = json.load(handle) + + results = outcome["results"] + counts: dict[str, int] = {} + for row in results: + counts[row["verdict"]] = counts.get(row["verdict"], 0) + 1 + total = len(results) + line = ( + f"{total} checks, {counts.get('pass', 0)} pass, {counts.get('reject', 0)} reject, " + f"{counts.get('alter', 0)} alter, {counts.get('n/a', 0)} n/a" + ) + + if args.line_only: + print(line) + return 0 + + reviewed = [r for r in results if r.get("allowed_reason")] + unreviewed = [r for r in results if not r.get("allowed_reason")] + + version = outcome.get("backend_version") or "version unknown" + print(f"## specmatrix: {outcome['backend']} {version} — {outcome['suite']}\n") + print(f"{line}\n") + print(table(unreviewed)) + if reviewed: + print("\n### Allowed — read and accepted by this adapter's maintainer\n") + for row in reviewed: + reason = row["allowed_reason"] + print(f"- `{row['id']}` ({row['verdict'].upper()}): {reason}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 5e86e593065403e8e2a6bb14f005467edb71f2d1 Mon Sep 17 00:00:00 2001 From: DeviousCardi <115358213+DeviousCardi@users.noreply.github.com> Date: Wed, 9 Sep 2026 13:34:15 +0530 Subject: [PATCH 2/3] Fix action.yml: dogfood run 404'd resolving "latest" with no release yet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two bugs, both only visible once the dogfood workflow actually ran: - "resolve version" always called the GitHub releases API, even when `ref`/`binary-path` (the two inputs that exist only for the dogfood workflow) make that lookup unneeded. This repository has never pushed a release tag yet, so `releases/latest` 404'd and every dogfood job failed before it reached the suite. Guarded the step to skip entirely when both overrides are given. - The "run the suite" step referenced `.specmatrix-corpus-${{ github.action_ref || 'action' }}`, left over from an earlier revision; the checkout step writes to the plain `.specmatrix-corpus` an earlier edit renamed it to, but a replace_all at the time matched only the checkout step's own `path:` line, not this second occurrence — so the two never agreed and `--cases` would have pointed at a directory that does not exist. Never caught because the CLI logic was dry-run by hand against a hardcoded path rather than through the action script itself. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK --- action.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/action.yml b/action.yml index 7206d90..d9326f8 100644 --- a/action.yml +++ b/action.yml @@ -53,8 +53,14 @@ outputs: runs: using: composite steps: + # Only actually needed to pick a corpus ref or a binary to download — + # both of which `ref`/`binary-path` can override. Skipped when both are + # given, so the dogfood workflow (which always gives both) never depends + # on a release existing at all, which matters before this project's own + # first tag is ever pushed. - name: resolve version id: version + if: inputs.ref == '' || inputs.binary-path == '' shell: bash run: | set -euo pipefail @@ -120,7 +126,7 @@ runs: shell: bash run: | set -euo pipefail - corpus=".specmatrix-corpus-${{ github.action_ref || 'action' }}" + corpus=".specmatrix-corpus" backend_input="${{ inputs.backend }}" if [[ "$backend_input" == */* ]]; then # A path into the caller's own checkout: split it into the From 3bfd9697d87910a45c4285b11b20ba0f20e2e301 Mon Sep 17 00:00:00 2001 From: DeviousCardi <115358213+DeviousCardi@users.noreply.github.com> Date: Wed, 9 Sep 2026 13:57:27 +0530 Subject: [PATCH 3/3] =?UTF-8?q?dogfood:=20raise=20job=20timeout=20to=2035m?= =?UTF-8?q?=20=E2=80=94=20cache=20contention=20across=2020=20parallel=20le?= =?UTF-8?q?gs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One leg hit the 20-minute limit purely from queueing behind the same shared Rust build cache as the other 19; nothing was actually hung. Measured: the slowest legitimate leg took 17m3s in the same run. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK --- .github/workflows/dogfood.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/dogfood.yml b/.github/workflows/dogfood.yml index d036c20..d546676 100644 --- a/.github/workflows/dogfood.yml +++ b/.github/workflows/dogfood.yml @@ -40,7 +40,10 @@ jobs: needs: matrix if: needs.matrix.outputs.pairs != '[]' runs-on: ubuntu-latest - timeout-minutes: 20 + # Twenty (backend, suite) pairs run as one matrix, all restoring and + # saving the same Rust build cache at once — measured contention pushed + # one leg past 20 minutes with nothing wrong, just queued behind others. + timeout-minutes: 35 strategy: fail-fast: false matrix: