feat(ga): make registry clean-room acceptance mandatory — test what npm and PyPI actually serve #79
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| name: registry clean-room acceptance | ||
|
|
||
| # Tests ONLY what the public registries serve — never this checkout. | ||
| # | ||
| # `registry parity` (the sibling workflow) answers "does the declared version equal the published | ||
| # version". That is necessary and not sufficient: it never installs anything, so it cannot see a | ||
| # package whose version number is right and whose CONTENTS are broken. Every artifact regression in | ||
| # the pre-GA audit was of that second kind — CI green on source while npm and PyPI served a broken | ||
| # build for days. This workflow installs the published artifact into a throwaway directory or venv | ||
| # and asserts it actually works. | ||
| # | ||
| # WHY NIGHTLY AND NOT ONLY ON RELEASE | ||
| # A published package can break with NO commit anywhere. @wave-av/cli's own published dependency | ||
| # range on @wave-av/sdk is a caret range, so what a customer receives is decided by npm's resolver | ||
| # on the day they install, not by anything in this repo. Only a scheduled run catches that. | ||
| # | ||
| # BLOCKING BEHAVIOUR | ||
| # schedule / workflow_run(release) / workflow_dispatch -> HARD FAIL, and open-or-update an issue. | ||
| # pull_request -> informational only. | ||
| # A PR is not the cause of an already-published defect, so a PR is not blocked by one. The PR run | ||
| # exists so that a change to the gate itself is exercised before it merges. There is deliberately | ||
| # NO path filter: the check reports on every PR, which is what makes it eligible to become a | ||
| # required status check later (a path-filtered required check stays permanently unreported and | ||
| # blocks every PR that misses the filter — the lesson already recorded in registry-parity.yml). | ||
|
|
||
| on: | ||
| schedule: | ||
| # 09:00 UTC — deliberately offset from `registry parity` (14:00) so a registry outage does not | ||
| # take out both signals in the same window. | ||
| - cron: "0 9 * * *" | ||
| workflow_dispatch: | ||
| inputs: | ||
| versions: | ||
| description: 'Exact versions to accept, e.g. "@wave-av/cli=1.0.9,wave-sdk=2.1.0" (default: registry latest)' | ||
| type: string | ||
| required: false | ||
| only: | ||
| description: 'Comma-separated target ids to run (default: all). Ids are in scripts/ga/cleanroom-targets.json' | ||
| type: string | ||
| required: false | ||
| workflow_run: | ||
| # Runs after a real publish so a release is verified against the registry it just wrote to. | ||
| workflows: ["npm publish (OIDC + provenance)"] | ||
| types: [completed] | ||
| pull_request: | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| cleanroom: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| # On workflow_run, only a SUCCESSFUL publish is worth verifying; a failed publish has its own | ||
| # error and would produce a confusing second failure here. | ||
| if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' | ||
| permissions: | ||
| contents: read | ||
| issues: write # fail-loud: open or update the tracking issue | ||
| steps: | ||
| - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 | ||
| with: | ||
| node-version: "22" | ||
|
|
||
| - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Run clean-room acceptance against the public registries | ||
| id: cleanroom | ||
| env: | ||
| CLEANROOM_VERSIONS: ${{ inputs.versions }} | ||
|
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift Pin the npm versions from the triggering publish. For Have 🤖 Prompt for AI Agents |
||
| ONLY: ${{ inputs.only }} | ||
| run: | | ||
| set -uo pipefail | ||
| args=(--python python3 --out-dir "$GITHUB_WORKSPACE/ga-out") | ||
| [ -n "${ONLY:-}" ] && args+=(--only "$ONLY") | ||
| set +e | ||
| node scripts/ga/registry-cleanroom.mjs "${args[@]}" 2>&1 | tee "$RUNNER_TEMP/cleanroom.log" | ||
| code=${PIPESTATUS[0]} | ||
| set -e | ||
| echo "exit_code=$code" >> "$GITHUB_OUTPUT" | ||
| { | ||
| echo "## Registry clean-room acceptance" | ||
| echo | ||
| echo "Exit code \`$code\` (0 = all checks passed, 1 = an artifact failed, 2 = the gate could not run)." | ||
| echo | ||
| echo '```' | ||
| cat "$RUNNER_TEMP/cleanroom.log" | ||
| echo '```' | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| exit 0 | ||
|
|
||
| - name: Upload GA evidence | ||
| if: always() | ||
| uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 | ||
| with: | ||
| name: ga-evidence-registry-cleanroom | ||
| path: ga-out/ | ||
| if-no-files-found: warn | ||
| retention-days: 90 | ||
|
|
||
| - name: Fail loudly — open or update the tracking issue | ||
| # A nightly that fails quietly is worse than no nightly at all. | ||
| if: steps.cleanroom.outputs.exit_code != '0' && github.event_name != 'pull_request' | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | ||
| EXIT_CODE: ${{ steps.cleanroom.outputs.exit_code }} | ||
| run: | | ||
| set -euo pipefail | ||
| title="registry clean-room acceptance is failing" | ||
| # Only the failing-check lines: the issue must say WHAT is broken, not page a human to | ||
| # go read a log to find out. | ||
| summary=$(sed -n '/^REGISTRY CLEAN-ROOM FAILED/,$p' "$RUNNER_TEMP/cleanroom.log" | head -40) | ||
| [ -z "$summary" ] && summary="The gate did not complete (exit $EXIT_CODE). See the run log." | ||
| body=$(printf '%s\n\n```\n%s\n```\n\nRun: %s\nEvent: %s\n\nThe published artifacts do not satisfy ART-001 / SUPPLY-001 / VER-001. A green source branch does not certify these — the registry is what customers install.\n' \ | ||
| "Clean-room acceptance against the public registries failed." "$summary" "$RUN_URL" "$GITHUB_EVENT_NAME") | ||
| existing=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "$title in:title" --json number --jq '.[0].number // empty') | ||
| if [ -n "$existing" ]; then | ||
| gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" --body "$body" | ||
| echo "updated issue #$existing" | ||
| else | ||
| gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" --body "$body" \ | ||
| || echo "::warning::could not open the tracking issue; the job still fails below" | ||
| fi | ||
|
|
||
| - name: Enforce | ||
| # Hard-fail everywhere the result is actionable. A pull request is informational: it did not | ||
| # publish the artifact under test and cannot fix it. | ||
| if: github.event_name != 'pull_request' | ||
| env: | ||
| CODE: ${{ steps.cleanroom.outputs.exit_code }} | ||
| run: | | ||
| if [ "$CODE" != "0" ]; then | ||
| echo "::error title=registry clean-room::published artifacts failed clean-room acceptance (exit $CODE) — see the job summary" | ||
| exit 1 | ||
| fi | ||
| echo "registry clean-room acceptance passed" | ||
|
|
||
| - name: Report (pull request, informational) | ||
| if: github.event_name == 'pull_request' | ||
| env: | ||
| CODE: ${{ steps.cleanroom.outputs.exit_code }} | ||
| run: | | ||
| [ "$CODE" = "0" ] && echo "clean-room acceptance passed" \ | ||
| || echo "::warning title=registry clean-room::published artifacts fail clean-room acceptance (exit $CODE). Not blocking this PR — see the job summary." | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # WAVE GA compliance — wave-av/sdks | ||
|
|
||
| > Criterion IDs are global and immutable. `unknown` and a waiver are **not** a pass. A green | ||
| > repository is necessary but not sufficient for platform GA. | ||
|
|
||
| ## Repository declaration | ||
|
|
||
| ```yaml | ||
| repository: wave-av/sdks | ||
| revision: 70b2a04a205658a25b3723adec9faf665e204af6 | ||
| repo_class: sdk | ||
| owner: WAVE platform / SDK publishing | ||
| last_evaluated_at: 2026-09-04T00:31:56Z | ||
| evaluator_version: scripts/ga/registry-cleanroom.mjs | ||
| spec_version: 1.0.0 | ||
| overall_status: fail | ||
| ``` | ||
|
|
||
| `overall_status: fail` is the honest reading, and it is an improvement on what came before it: | ||
| these criteria were **unknown** until this repository could install what the registries serve and | ||
| look. The gate now produces evidence. The evidence says the published artifacts are broken. | ||
|
|
||
| ## How this repo proves compliance | ||
|
|
||
| `scripts/ga/registry-cleanroom.mjs` installs each published artifact from its **public registry** | ||
| into a throwaway directory or venv — never from this checkout, never `npm link`, never | ||
| `pip install -e` — and asserts it behaves. It emits `ga-out/ga-evidence.json` keyed to criterion | ||
| IDs, plus `ga-out/cleanroom-report.json` with per-check detail and artifact digests. The run | ||
| output is a CI artifact and is never committed; a committed report would let a stale file | ||
| masquerade as current evidence. | ||
|
|
||
| The distinction from the sibling `registry parity` workflow matters. Parity asks *"does the | ||
| declared version equal the published version"* and never installs anything, so it cannot see a | ||
| package whose version number is correct and whose contents are broken. Every artifact regression in | ||
| the pre-GA audit was of that second kind. | ||
|
|
||
| ```bash | ||
| # everything the registries currently serve | ||
| node scripts/ga/registry-cleanroom.mjs | ||
|
|
||
| # a specific release, pinned rather than whatever `latest` points at | ||
| node scripts/ga/registry-cleanroom.mjs --versions '@wave-av/cli=1.0.9,wave-sdk=2.1.0' | ||
| ``` | ||
|
|
||
| Exit `0` all checks passed · `1` an artifact failed · `2` the gate could not run. Exit 2 is never | ||
| to be read as a pass. | ||
|
|
||
| Schedule: nightly at 09:00 UTC, after every successful npm publish, on demand, and informationally | ||
| on every pull request (`.github/workflows/registry-cleanroom.yml`). Nightly is not decoration — a | ||
| published package can break with no commit anywhere, because a published dependency **range** is | ||
| resolved on the day a customer installs. | ||
|
|
||
| | Criterion | Title | GA must-pass | Repo status | Evidence | | ||
| |---|---|---:|---|---| | ||
| | ART-001 | Published artifacts install, import, start, identify themselves, and match source | true | **fail** | `ga-out/ga-evidence.json` (evaluator: `scripts/ga/registry-cleanroom.mjs`) | | ||
| | SUPPLY-001 | Builds have provenance, signatures, SBOMs, dependency policy, and protected release identity | true | **fail** (partial coverage — see note) | `ga-out/ga-evidence.json` | | ||
| | VER-001 | Version and release truth agree from source through deployment | true | **fail** (registry half) | `ga-out/ga-evidence.json` | | ||
| | CONTRACT-001 | One promoted contract is the source of truth across spec, gateway, registry, MCP, SDK and CLI | true | unknown | not evaluated by this repo | | ||
| | COMPAT-001 | Backward compatibility, versioning, deprecation, and sunset policy are enforced | true | unknown | not evaluated by this repo | | ||
| | DX-001 | A new developer can complete one honest golden path from published materials | true | unknown | not evaluated by this repo | | ||
| | STATUS-001 | Marketing, registry, preview labels, availability, and status tell the same truth | true | unknown | not evaluated by this repo | | ||
|
|
||
| Criteria absent from this table are owned by other repositories and surfaces; this repository makes | ||
| no claim about them. Absence here is `unknown` at the platform gate, not `not_applicable`. | ||
|
|
||
| ## Per-criterion evidence | ||
|
|
||
| ```yaml | ||
| criterion_id: ART-001 | ||
| status: fail | ||
| owner: WAVE platform / SDK publishing | ||
| verification_command: node scripts/ga/registry-cleanroom.mjs | ||
| verified_revision: 70b2a04a205658a25b3723adec9faf665e204af6 | ||
| verified_at: 2026-09-04T00:31:56Z | ||
| evidence: | ||
| - uri: ci://wave-av/sdks/.github/workflows/registry-cleanroom.yml#cleanroom-report.json | ||
| sha256: 0f03603df2bc33f50958a75c8998777463507c1f9e3ee0fec811d82a3ca1206d | ||
| pass_condition_from_spec: > | ||
| All supported runtimes pass from public registries; Python uses a non-stdlib-colliding import; | ||
| source, package metadata, CLI banner, tag and GitHub release agree. | ||
| notes: | | ||
| Observed against the live registries on 2026-09-04. Passing: | ||
| @wave-av/sdk@2.1.3 ESM import, CJS require, all 46 declared subpath exports resolve. | ||
| @wave-av/adk@1.0.15 installs and imports. | ||
| @wave-av/mcp-server@0.2.0 starts over stdio, lists 18 tools, serves every tool its shipped | ||
| README advertises. | ||
| Failing: | ||
| @wave-av/cli@1.0.8 `wave --version` prints 1.0.0. Installs cleanly and `--help` exits 0, | ||
| so nothing short of running the binary detects this. | ||
| wave-sdk@2.0.0 (PyPI) `from wave_sdk import Wave` raises ModuleNotFoundError. The wheel's | ||
| only top-level name is `wave`, which collides with the CPython stdlib | ||
| module of that name; because the stdlib directory precedes | ||
| site-packages, `import wave` returns the stdlib WAV reader and the SDK | ||
| is unreachable by any name. The artifact is unusable as published. | ||
| wave-av-sdk@2.0.0 (PyPI) identical defect. | ||
| ``` | ||
|
|
||
| ```yaml | ||
| criterion_id: VER-001 | ||
| status: fail | ||
| owner: WAVE platform / SDK publishing | ||
| verification_command: node scripts/ga/registry-cleanroom.mjs | ||
| verified_revision: 70b2a04a205658a25b3723adec9faf665e204af6 | ||
| verified_at: 2026-09-04T00:31:56Z | ||
| evidence: | ||
| - uri: ci://wave-av/sdks/.github/workflows/registry-cleanroom.yml#cleanroom-report.json | ||
| sha256: 0f03603df2bc33f50958a75c8998777463507c1f9e3ee0fec811d82a3ca1206d | ||
| pass_condition_from_spec: > | ||
| Every shipped component resolves to one source revision and version; no newer source is | ||
| represented as deployed; mutable channels are labeled; deployment receipt identifies artifact | ||
| digest. | ||
| notes: | | ||
| This repository covers the REGISTRY half of VER-001 — does the artifact agree with itself about | ||
| which build it is. Two published artifacts do not: | ||
| @wave-av/cli@1.0.8 binary self-reports 1.0.0 | ||
| @wave-av/mcp-server@0.2.0 serverInfo.version reports 0.1.0 | ||
| In both cases the package metadata is correct and the code inside it disagrees, so a version | ||
| comparison against the registry cannot see the defect — only running the artifact can. | ||
| Tag/GitHub-release/deployed-endpoint agreement is NOT covered here and remains unknown; it belongs | ||
| to the release-ledger check named in the spec's runnable_command. | ||
| ``` | ||
|
|
||
| ```yaml | ||
| criterion_id: SUPPLY-001 | ||
| status: fail | ||
| owner: WAVE platform / SDK publishing | ||
| verification_command: node scripts/ga/registry-cleanroom.mjs | ||
| verified_revision: 70b2a04a205658a25b3723adec9faf665e204af6 | ||
| verified_at: 2026-09-04T00:31:56Z | ||
| evidence: | ||
| - uri: ci://wave-av/sdks/.github/workflows/registry-cleanroom.yml#cleanroom-report.json | ||
| sha256: 0f03603df2bc33f50958a75c8998777463507c1f9e3ee0fec811d82a3ca1206d | ||
| pass_condition_from_spec: > | ||
| Release artifacts are built by approved CI from an immutable source revision, provenance is | ||
| verifiable, SBOM is attached, critical known vulnerabilities are resolved or explicitly | ||
| risk-accepted, and publisher accounts require strong MFA. | ||
| notes: | | ||
| PARTIAL COVERAGE — this evaluator checks two of the five clauses. A `fail` here is therefore | ||
| sound, but a future `pass` would NOT be sufficient to pass SUPPLY-001 on its own. | ||
| Covered and failing: | ||
| provenance @wave-av/cli@1.0.8 carries no npm provenance attestation (dist.attestations is | ||
| null), while sdk, mcp-server and adk each carry a | ||
| https://slsa.dev/provenance/v1 attestation. The CLI was published outside the | ||
| provenance-emitting pipeline and cannot be traced to an approved CI build. | ||
| dependency @wave-av/cli@1.0.8 declares `@wave-av/sdk: "^2.0.11"`. What a customer receives | ||
| policy is decided by npm's resolver on the day they install; today that is 2.1.3. The | ||
| published artifact is not reproducible, and this is the precise mechanism by | ||
| which a broken SDK shipped inside a CLI that no one had changed. | ||
| NOT covered here, still unknown: SBOM attachment, vulnerability posture, publisher MFA and | ||
| branch-protection attestation. | ||
| ``` | ||
|
|
||
| ## Operator actions to finish these criteria | ||
|
|
||
| 1. **Make the release gate blocking.** The clean-room job hard-fails on schedule, release and | ||
| dispatch, but it is not yet a *required* status check. Add | ||
| `registry clean-room acceptance / cleanroom` to the default branch's required checks. It runs on | ||
| every pull request with no path filter precisely so it can be made required without going | ||
| permanently unreported. | ||
| 2. **Fix the three artifact defects the gate found** (each needs a publish, which is a named floor | ||
| and not this lane's to cross): the CLI version constant, the MCP server's `serverInfo.version`, | ||
| and the Python distribution's top-level module name. | ||
| 3. **Republish the CLI through the provenance-emitting workflow** so `dist.attestations` is | ||
| populated, and exact-pin its first-party dependency. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Clean-room run output (cleanroom-report.json, ga-evidence.json) is a CI artifact, never a | ||
| # committed one: evidence must describe a run that actually happened, in the environment that | ||
| # happened to run it. Committing a locally generated report would let a stale file masquerade as | ||
| # current evidence — the exact failure mode the GA gate exists to prevent. | ||
| # | ||
| # Self-ignoring so the root .gitignore does not need to change. | ||
| * | ||
| !.gitignore |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Edge Case: workflow_run trigger only watches npm publish, missing PyPI
The
workflow_runtrigger references only"npm publish (OIDC + provenance)", so a PyPI-only release ofwave-sdk/wave-av-sdk(no accompanying npm publish) does not trigger an immediate post-publish clean-room run — the two PyPI packages this PR flags as broken would only be re-verified on the next 09:00 UTC nightly, not right after the publish that shipped the defect. Add the PyPI publish workflow name to theworkflow_run.workflowslist so both ecosystems get a same-day check after a fresh publish.Fix:
Was this helpful? React with 👍 / 👎