From 37e2fd33586f520e4c6739c376571a50aacbcfad Mon Sep 17 00:00:00 2001 From: Tom Cobb Date: Fri, 21 Aug 2026 14:33:34 +0000 Subject: [PATCH 1/3] chore: restore devcontainer and package-lock to their pre-#64 state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two local working-tree edits were swept into #64 by a `git add -A`. A commit on that branch reverted them, and a later `git add -A` in the same branch put them back, so they reached main inside the squash merge. Neither belongs there. The devcontainer gained `--device=/dev/net/tun`, a host-specific tweak that fails on hosts without that device. The package-lock change is a regression: it stripped the `libc` fields from Biome's platform-specific optional dependencies, which is what an older npm does when it rewrites the file — those fields are how npm picks the glibc vs musl build. Restored both to a354613, the state main had before #64 and the state CI was green on. Also ignores node_modules.stale, left behind by an NFS lock that blocks unlink. Co-Authored-By: Claude Opus 5 --- .devcontainer/devcontainer.json | 3 +-- .gitignore | 1 + package-lock.json | 45 +++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 2 deletions(-) diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 121af3a..44ecbee 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -24,8 +24,7 @@ // Allow the container to access the host X11 display and EPICS CA "--net=host", // Make sure SELinux does not disable with access to host filesystems like tmp - "--security-opt=label=disable", - "--device=/dev/net/tun" + "--security-opt=label=disable" ], "mounts": [ // Mount in the user terminal config folder so it can be edited diff --git a/.gitignore b/.gitignore index d7f23b7..f182f24 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ node_modules/ docs/_build/ *.log __pycache__/ +node_modules.stale/ diff --git a/package-lock.json b/package-lock.json index 552bf3f..e5be372 100644 --- a/package-lock.json +++ b/package-lock.json @@ -81,6 +81,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT OR Apache-2.0", "optional": true, "os": [ @@ -98,6 +101,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT OR Apache-2.0", "optional": true, "os": [ @@ -115,6 +121,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT OR Apache-2.0", "optional": true, "os": [ @@ -132,6 +141,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT OR Apache-2.0", "optional": true, "os": [ @@ -266,6 +278,9 @@ "arm" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -283,6 +298,9 @@ "arm" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -300,6 +318,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -317,6 +338,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -334,6 +358,9 @@ "arm" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -351,6 +378,9 @@ "ia32" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -368,6 +398,9 @@ "ia32" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -385,6 +418,9 @@ "riscv64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -402,6 +438,9 @@ "s390x" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -419,6 +458,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -436,6 +478,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ From 3cb6f420d6780ecabe8199ca2fc17d7b5afc819f Mon Sep 17 00:00:00 2001 From: Tom Cobb Date: Fri, 21 Aug 2026 14:37:28 +0000 Subject: [PATCH 2/3] feat!: publish on workflow_run, deleting the trampoline, shim and injection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The live experiment on 2026-08-21 settled the question this design was built around. Pages stamps every deployment with pages_build_version = GITHUB_SHA and silently no-ops a repeat (actions/deploy-pages#383) — but only for some events. Four consecutive deploys at the identical build version b24237484c3b445469c2db4ef161410a185fcdbc — a push to main, a tag cut on that same commit, and two pushes to one PR — EVERY one of which updated the live origin. workflow_run forces a re-serve exactly as workflow_dispatch does. That removes the reason for most of this machinery: - The `re-dispatch` trampoline is gone: a tag's deploy re-serves directly. - `publish-dispatch.yml` is gone. It existed only because a reusable workflow cannot be workflow_dispatch'd cross-repo, so the trampoline needed a local file to re-fire. - The `version-name` injection is gone, along with its threading through ci.yml and the shim. workflow_run fires after the triggering run has COMPLETED, so that run's `docs` artifact is discoverable by the ordinary gather — verified in the experiment's baseline, where both injection steps were skipped and the site still assembled correctly. - The `warn` job is gone: a fork's CI never reaches the engine. - `retry-until` and the wedged-origin retry-dispatch are gone, and with them the engine's need for `actions: write`. publish.yml is now the file each consumer carries: workflow_run on their CI completing, plus workflow_dispatch for fork previews and manual re-deploys. The engine it calls is renamed publish-gh-pages.yml so this repo can dogfood that filename exactly rather than approximating it. ci.yml no longer publishes at all and needs no elevated permissions. Publishing also leaves every PR's critical path, which is the point. Its cost is O(the whole site) and independent of the change, which is why the first consumer switched PR previews off a day after adopting; and a Pages failure now stops showing as a red check against an author with no control over that infrastructure. One trap this introduces, and the reason for the new test_shape.py: under workflow_run, github.ref and GITHUB_SHA are ALWAYS the default branch's HEAD, never the commit that was built. Both cache-save conditions gated on github.ref, which would have been true for every PR-triggered deploy — each one writing a cache scoped to a PR that nothing else can read. They now test github.event.workflow_run.head_branch. That is an `if:` expression, so the gather harness cannot reach it; test_shape.py asserts it, along with the fork guard, whose failure mode is to fail open. cancel-in-progress is now true. It was unsafe while publish was nested in ci.yml, where cancelling marked an unrelated PR's CI run cancelled; with publish on its own run there is no such collateral, and every deploy reconstructs the whole site so a superseded one holds nothing unique. BREAKING CHANGE: consumers must add publish.yml, delete publish-dispatch.yml, remove the publish job from ci.yml, and re-pin to publish-gh-pages.yml. Co-Authored-By: Claude Opus 5 --- .github/workflows/_probe-workflow-run.yml | 100 --- .github/workflows/ci.yml | 44 +- .github/workflows/publish-dispatch.yml | 55 -- .github/workflows/publish-gh-pages.yml | 596 +++++++++++++++++ .github/workflows/publish.yml | 742 ++-------------------- package.json | 2 +- test/workflow-harness/run.py | 4 +- test/workflow-harness/test_gather.py | 2 +- test/workflow-harness/test_shape.py | 95 +++ 9 files changed, 765 insertions(+), 875 deletions(-) delete mode 100644 .github/workflows/_probe-workflow-run.yml delete mode 100644 .github/workflows/publish-dispatch.yml create mode 100644 .github/workflows/publish-gh-pages.yml create mode 100644 test/workflow-harness/test_shape.py diff --git a/.github/workflows/_probe-workflow-run.yml b/.github/workflows/_probe-workflow-run.yml deleted file mode 100644 index a707323..0000000 --- a/.github/workflows/_probe-workflow-run.yml +++ /dev/null @@ -1,100 +0,0 @@ -name: Probe (workflow_run) - -# TEMPORARY EXPERIMENT — delete once the question below is answered. -# -# QUESTION: does a Pages deploy triggered by `workflow_run` update the served ORIGIN -# when that SHA has already been deployed? -# -# Why we can't reason it out. `workflow_run` sets GITHUB_SHA to the DEFAULT BRANCH's HEAD, -# not the triggering commit (GitHub's events table; confirmed on scikit-learn, whose -# consecutive workflow_run runs from different PRs all carry the same head_sha). And -# actions/deploy-pages stamps pages_build_version = GITHUB_SHA, which per -# actions/deploy-pages#383 is silently no-op'd when it repeats. Together that *suggests* -# at most one serving deploy per merge to the default branch. -# -# But that inference is unsafe: a repeat `workflow_dispatch` at an already-deployed SHA -# demonstrably DOES re-serve — the whole re-dispatch trampoline in publish.yml depends on -# it and it works in production. So event type matters beyond pages_build_version, in a -# way #383 doesn't explain, and workflow_run might behave like workflow_dispatch. The -# Pages deployment RECORD is created either way (so the API can't tell us); only the -# origin can. Hence a real deploy against the real site. -# -# THE OBSERVATIONS (see the plan; read the answer from switcher.json at the origin): -# 1. push to main → first deploy of that SHA; should serve. Baseline, and it -# proves the gather works with no `version-name` injection. -# 2. tag v0.0.0a1 on that same main commit → SECOND deploy of the SAME SHA, but now -# assembling a site that CONTAINS v0.0.0a1. Different bytes, -# same SHA — the real-world tag case. If the origin gains -# v0.0.0a1, workflow_run re-serves and the migration is on. -# 3. a docs PR, pushed twice → both deploys land at the same unchanged main HEAD. Does -# /pr-/ appear, and does it UPDATE on the second push? -# Highest-volume case; first to break if the collision is real. -# -# Safety: this deploys the normal, correctly assembled site — the only question is whether -# the origin moves. Worst case it goes stale (not broken), and publish.yml's -# "Verify the deployed origin" step turns that into a red check rather than a silent lie. -# Recovery is a manual workflow_dispatch of publish-dispatch.yml, which we know re-serves. -on: - workflow_run: - workflows: [CI] # matched by workflow NAME (ci.yml's `name:`), not filename - types: [completed] - -jobs: - # Always runs, even for a failed/fork CI run: the context dump is half the experiment. - # `github.sha` here is the value deploy-pages will use as pages_build_version. - context: - runs-on: ubuntu-latest - permissions: {} - steps: - - name: Log the workflow_run context - env: - WR: ${{ toJSON(github.event.workflow_run) }} - run: | - set -euo pipefail - printf '%-34s %s\n' \ - 'github.sha (→ pages_build_version)' '${{ github.sha }}' \ - 'github.ref' '${{ github.ref }}' \ - 'github.ref_name' '${{ github.ref_name }}' \ - 'github.event_name' '${{ github.event_name }}' \ - '— triggering run —' '' \ - 'workflow_run.id' '${{ github.event.workflow_run.id }}' \ - 'workflow_run.event' '${{ github.event.workflow_run.event }}' \ - 'workflow_run.conclusion' '${{ github.event.workflow_run.conclusion }}' \ - 'workflow_run.head_branch' '${{ github.event.workflow_run.head_branch }}' \ - 'workflow_run.head_sha' '${{ github.event.workflow_run.head_sha }}' \ - 'head_repository' '${{ github.event.workflow_run.head_repository.full_name }}' - # The comparison that matters: if github.sha != workflow_run.head_sha, the deploy - # is stamped with something other than the commit that was actually built. - if [ '${{ github.sha }}' = '${{ github.event.workflow_run.head_sha }}' ]; then - echo "::notice::github.sha MATCHES the triggering commit — no collision risk" - else - echo "::notice::github.sha (${{ github.sha }}) differs from the built commit \ - (${{ github.event.workflow_run.head_sha }}) — every deploy between merges reuses this SHA" - fi - echo "--- full workflow_run payload ---" - printf '%s\n' "$WR" - - # The actual experiment: deploy through the REAL engine, so the site content is correct - # and publish.yml's Verify step reports the answer in-run. - # - # version-name: "" → pure durable gather. Unlike the inline path, the triggering run has - # COMPLETED, so its `docs` artifact is discoverable by the normal gather (by default-branch - # or by PR head SHA) with no injection. Whether that actually holds is observation 1. - # - # The fork guard is not optional: workflow_run hands a write token to a job triggered by - # fork-PR code (the pwn-request footgun that publish.yml's read-only `warn` job exists to - # avoid). Fork CI runs are excluded here and stay excluded. - deploy: - needs: [context] - if: >- - github.event.workflow_run.conclusion == 'success' && - github.event.workflow_run.head_repository.full_name == github.repository - uses: ./.github/workflows/publish.yml - with: - version-name: "" - permissions: - contents: read - actions: write - pages: write - id-token: write - statuses: write diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d0d9ea7..fd710df 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,15 +1,14 @@ name: CI -# Verify + build, then publish on INTERNAL events. Runs on every PR (including -# forks), pushes to main, and tags — lint, test, build the docs, and on tags -# publish the release assets. It uploads each build's `docs` artifact. +# Verify + build. Runs on every PR (including forks), pushes to main, and tags — lint, +# test, build the docs, and on tags publish the release assets. It uploads each build's +# `docs` artifact. # -# Publishing is nested here (the `publish` job → publish-dispatch.yml shim → publish.yml) -# so its status is visible on the PR / commit. A single `publish` job covers every event; -# publish.yml branches internally into deploy (internal PR / main push), tag re-dispatch -# (a same-SHA tag deploy would be silently dropped by Pages unless dispatched), or a -# fork-PR warning (forks build with a read-only token and never deploy). See the -# architecture explanation in docs/. +# It does NOT publish the site. publish.yml picks this run up on `workflow_run` once it +# completes and deploys from there, so the site's reconstruction cost stays out of every +# PR's critical path and a Pages failure never shows as a red check against an author who +# cannot act on it. That also means this workflow needs no elevated permissions, and its +# `name:` below is load-bearing — publish.yml matches on it. on: pull_request: push: @@ -36,30 +35,3 @@ jobs: uses: ./.github/workflows/release.yml permissions: contents: write # create the GitHub Release + attach assets - - # One publish job for every event — publish.yml (reached via the publish-dispatch.yml - # shim) owns the branching (deploy inline / tag re-dispatch / fork-PR warn), so there's - # no separate tag or fork job and no ref/fork guards here. It runs for fork PRs too, - # but only to surface the opt-in hint: publish.yml's `deploy` job excludes forks and a - # fork PR's token is read-only, so untrusted code never reaches a write token. - # `actions: write` lets the re-dispatch job re-fire the shim; `version-name` is the - # in-run build for the inline path (ignored by the re-dispatch/warn branches). - # TEMPORARILY DISABLED — _probe-workflow-run.yml's experiment needs the workflow_run path - # to be the ONLY deploy path, so the experiment has one variable. Tags are disabled here - # too, deliberately: the re-dispatch job would deploy via workflow_dispatch, which we - # already know re-serves, and that is precisely the confound observation 2 must avoid. - # RESTORE to: - # if: github.repository == 'DiamondLightSource/myst-version-switcher-plugin' - # when the probe is deleted. - publish: - needs: [lint, test, docs] - if: false - uses: ./.github/workflows/publish-dispatch.yml - with: - version-name: ${{ needs.docs.outputs.version-name }} - permissions: - contents: read - actions: write # the wrapper's tag re-dispatch dispatches via gh workflow run - pages: write - id-token: write - statuses: write diff --git a/.github/workflows/publish-dispatch.yml b/.github/workflows/publish-dispatch.yml deleted file mode 100644 index a40c3e3..0000000 --- a/.github/workflows/publish-dispatch.yml +++ /dev/null @@ -1,55 +0,0 @@ -name: Publish (dispatch) - -# Thin shim — the one dispatchable file in this repo (and the one consumers copy). It -# forwards to publish.yml's engine, exposing it two ways: -# workflow_call — ci.yml's `publish` job, for every event (publish.yml branches into -# deploy / tag re-dispatch / fork warn). -# workflow_dispatch — the tag re-dispatch, the fork-PR preview (`pr`), and -# manual re-deploys. -# A reusable workflow can't be `workflow_dispatch`'d cross-repo, so this shim must live -# in the consumer's repo; publish.yml's re-dispatch job re-fires it by name -# (`dispatch-workflow`). This repo dogfoods the exact shim a consumer adds — theirs only -# differs in the `uses:` target (…/publish.yml@), the single place they pin . -on: - workflow_call: - inputs: - version-name: - description: Version name of the in-run build to inject (passed by ci.yml's inline publish). - required: false - default: "" - type: string - workflow_dispatch: - inputs: - pr: - description: External fork PR number to approve (pins its head SHA) and preview. Leave empty to just re-deploy. - required: false - default: "" - retry-until: - description: >- - Internal — epoch-seconds deadline for auto-retrying a wedged Pages origin. - Don't set manually. - required: false - default: "" - -jobs: - publish: - uses: ./.github/workflows/publish.yml - with: - version-name: ${{ inputs.version-name }} # "" on dispatch → pure durable gather - # Site-size policy, so it applies on EVERY path (call and dispatch alike) — hence a - # literal here rather than another input threaded down from ci.yml. - # This repo deliberately differs from the value the tutorial ships ("20"): our own - # site is ~223 MB packed across 24 releases, a fifth of the Pages cap, so capping - # would drop old versions for no gain. The capped path is covered by the unit tests - # and the workflow harness rather than by our own deploys. Set a real number here - # once a site nears the cap — see docs/how-to/keep-the-site-small.md. - max-releases: "0" - pr: ${{ inputs.pr }} # set (dispatch only) → pin that fork head SHA - dispatch-workflow: publish-dispatch.yml # the file the tag re-dispatch re-fires - retry-until: ${{ inputs.retry-until }} # "" on workflow_call/no-retry dispatch → no auto-retry - permissions: - contents: read - actions: write # publish.yml's re-dispatch job re-fires this shim via gh - pages: write - id-token: write - statuses: write diff --git a/.github/workflows/publish-gh-pages.yml b/.github/workflows/publish-gh-pages.yml new file mode 100644 index 0000000..c64d344 --- /dev/null +++ b/.github/workflows/publish-gh-pages.yml @@ -0,0 +1,596 @@ +name: Publish to GitHub Pages + +# Reconstruct the WHOLE versioned docs site from durable sources (the consumer's +# default-branch build, release docs.zip assets, open-PR build artifacts) and deploy it +# directly to GitHub Pages — no gh-pages branch. The PRIVILEGED half, and a reusable +# workflow: its only caller is a `publish.yml` in the consumer's repo (this repo has one +# too, identical but for the pin), triggered by `workflow_run` when their CI completes. +# +# WHY workflow_run, and why this file is so much smaller than it used to be. Pages stamps +# every deployment with `pages_build_version = GITHUB_SHA` and silently no-ops a repeat +# (actions/deploy-pages#383) — but only for SOME events. `workflow_dispatch` and +# `workflow_run` both force a re-serve. That was proven here on 2026-08-21, not reasoned: +# four consecutive deploys at the identical build version b24237484c3b445469c2db4ef161410a185fcdbc +# (a main push, a tag cut on that same commit, and two pushes to one PR) EVERY ONE of +# which updated the live origin. +# +# Because a tag's deploy re-serves directly, the old `re-dispatch` trampoline is gone, +# and with it the `publish-dispatch.yml` shim that existed only to give the trampoline a +# locally-dispatchable file. Because `workflow_run` fires after the triggering run has +# COMPLETED, that run's `docs` artifact is discoverable by the ordinary gather, so the +# `version-name` injection is gone too. And because a fork's CI never reaches this +# workflow (the caller guards on it), the read-only `warn` job is gone. +# +# One caveat that shapes the code below: under `workflow_run`, GITHUB_SHA and github.ref +# are ALWAYS the default branch's HEAD, never the commit that was built. Anything that +# needs the real branch must read `github.event.workflow_run.head_branch`. +# +# The assemble logic is THIS repo's assemble.mjs (the pure kernel); the IO plumbing +# lives as inline steps in this file. To version-match assemble.mjs to this workflow, +# we check out the reusable workflow's OWN repo at its OWN commit via the `job` context: +# unlike github.workflow_* (which resolve to the CALLER's entry workflow), +# `job.workflow_repository` + `job.workflow_sha` resolve to the file that defines THIS +# job — i.e. the reusable workflow at the ref a consumer pinned it to. So assemble.mjs +# always matches the workflow version, with no hardcoded repo and no manual release-time +# bump. assemble.mjs runs `git tag` in the cwd, so the CONSUMER's repo is checked out at +# the root (with tags) to supply their version list; assemble.mjs comes from .mvs. +on: + workflow_call: + inputs: + pr: + description: External fork PR number to approve (pins its head SHA) and preview. + required: false + default: "" + type: string + max-releases: + description: >- + Publish only the N most recent releases (by the tagged commit's date), or 0 for + all of them. The site is deployed as ONE Pages artifact and Pages caps that at + 1 GB, so an old project with hundreds of released docs.zips will eventually + fail to deploy at all — a repo with 131 of them was already at 452 MB. Set this + in your publish.yml's `with:` block so it applies on every path. + required: false + default: "0" + type: string + +jobs: + # The only job. The caller (publish.yml in the consumer's repo) has already gated on + # the triggering run having SUCCEEDED and not being from a fork, so there is nothing + # left to branch on here. + deploy: + runs-on: ubuntu-latest + permissions: + contents: read # checkout + read release assets + actions: read # gather cross-run docs artifacts (the retry-dispatch that needed write is gone) + pages: write # deploy to Pages + id-token: write # deploy-pages OIDC + statuses: write # set the preview-approved status on a fork PR head SHA + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + # Cancelling is safe now, and worth having: every deploy reconstructs the whole site + # from durable sources, so a superseded one has nothing unique in it — the next deploy + # gathers everything the cancelled one would have. This was NOT safe while publish was + # nested in ci.yml, where cancelling marked an unrelated PR's CI run as cancelled. + concurrency: + group: pages + cancel-in-progress: true + env: + REPO: ${{ github.repository }} + # The branch that was actually BUILT. Under workflow_run, github.ref and GITHUB_SHA + # are always the default branch's HEAD regardless of what triggered it, so github.ref + # is useless for "was this the default branch?" — it is true for a PR-triggered + # deploy too. head_branch is the honest answer; ref_name covers a manual dispatch, + # where there is no triggering run. + BUILT_BRANCH: ${{ github.event.workflow_run.head_branch || github.ref_name }} + DEFAULT: ${{ github.event.repository.default_branch }} + SEED_TAG: pages-default-seed + # Site-size guard thresholds, both on the deployed artifact's scale. PROBE is the + # uncompressed size past which it is worth actually packing the tree to find out + # (the pack costs real seconds); WARN is the packed size that earns a warning, + # 700 MB against Pages' hard 1 GB. Overridable so the guard is testable and so a + # site with unusual compressibility can be tuned. + SIZE_PROBE_BYTES: "1610612736" + SIZE_WARN_BYTES: "734003200" + # DEPRECATED: the in-site durable store. Nothing writes it any more (the default + # branch's zip lives in the Actions cache); this is only still read as a one-shot + # fallback for a site last deployed by an older publish.yml. + SOURCES_DIR: _sources + steps: + # The CONSUMER's repo, at the root cwd: assemble.mjs runs `git tag` here to + # order that repo's deployed versions, so it needs their tags (fetch-depth: 0). + - uses: actions/checkout@v7 + with: + fetch-depth: 0 + + # THIS reusable workflow's repo + commit (the `job` context resolves to the + # reusable file, not the caller), so assemble.mjs matches the ref the consumer + # pinned publish-gh-pages.yml@ to — automatically, no version bump. + # Sparse: only assemble/ is needed. + - name: Fetch assemble.mjs + uses: actions/checkout@v7 + with: + repository: ${{ job.workflow_repository }} + ref: ${{ job.workflow_sha }} + path: .mvs + sparse-checkout: assemble + + # Fork opt-in: pin the PR's head commit as approved. assemble gathers an external + # PR only when its head SHA carries this status, so a later push (new SHA) silently + # drops the preview until re-approved. Gated on `pr` alone, which the caller + # threads in from its workflow_dispatch input. + - name: Approve fork PR head SHA + if: inputs.pr != '' + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + sha=$(gh pr view "${{ inputs.pr }}" --repo "$REPO" --json headRefOid -q .headRefOid) + gh api --method POST "repos/$REPO/statuses/$sha" \ + -f state=success -f context=preview-approved \ + -f description="Fork docs preview approved" + + # BUILT_BRANCH vs github.ref is the one thing a post-mortem here always trips over, + # so state both up front rather than leaving someone to infer it from a cache miss. + - name: Create staging dirs + run: | + set -euo pipefail + mkdir -p "$RUNNER_TEMP/site" "$RUNNER_TEMP/gather" + printf '%-16s %s\n' \ + 'event' '${{ github.event_name }}' \ + 'built branch' "$BUILT_BRANCH" \ + 'default' "$DEFAULT" \ + 'github.ref' '${{ github.ref }} (always the default branch under workflow_run)' \ + 'github.sha' '${{ github.sha }} (→ pages_build_version)' \ + 'triggered by' '${{ github.event.workflow_run.html_url || "manual dispatch" }}' + + # The live Pages base URL, from the Pages API — respects a custom domain + # (CNAME); falls back to the conventional github.io URL if the API call + # fails. Used by the _sources restore below and baked into switcher.json. + - name: Resolve the Pages URL + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + url=$(gh api "repos/$REPO/pages" -q .html_url 2>/dev/null || true) + if [ -z "$url" ]; then + owner=$(printf '%s' "${REPO%%/*}" | tr '[:upper:]' '[:lower:]') + url="https://$owner.github.io/${REPO#*/}" + fi + echo "PAGES_URL=${url%/}" >> "$GITHUB_ENV" + echo "Pages URL: ${url%/}" + + # ── Which releases, and which of their zips do we already have? ───────── + # + # Selection is assemble.mjs's job (pure, unit-tested); this step only does IO. + # It runs BEFORE any download so the gather fetches only what will actually be + # published, and so the cache key describes exactly that set. + # + # Cached zips are named by ASSET ID, not by tag: an asset id identifies its bytes + # unambiguously, so a re-cut release can never be served from a stale cache entry. + - name: Select releases to publish + id: releases + env: + GH_TOKEN: ${{ github.token }} + MAX_RELEASES: ${{ inputs.max-releases }} + run: | + set -euo pipefail + gh api --paginate "repos/$REPO/releases" | jq -rs 'add // []' > "$RUNNER_TEMP/releases.json" + node "$GITHUB_WORKSPACE/.mvs/assemble/assemble.mjs" select-releases \ + --default-branch "$DEFAULT" --seed-tag "$SEED_TAG" \ + --max-releases "${MAX_RELEASES:-0}" \ + < "$RUNNER_TEMP/releases.json" > "$RUNNER_TEMP/releases.tsv" + # Key on the exact set of asset ids we intend to publish, so any change to + # that set misses and re-saves. restore-keys then falls back to the previous + # set, which shares nearly every zip — only the new release is downloaded. + digest=$(awk -F'\t' '$2 != "-" {print $5}' "$RUNNER_TEMP/releases.tsv" \ + | sort | sha256sum | cut -c1-32) + echo "cache-key=mvs-relzips-v1-$digest" >> "$GITHUB_OUTPUT" + + - name: Restore cached release zips + id: relcache + uses: actions/cache/restore@v6 + with: + path: ${{ runner.temp }}/relcache + key: ${{ steps.releases.outputs.cache-key }} + restore-keys: mvs-relzips-v1- + + # Releases are permanent sources; the seed release (if present) stands in for the + # default branch until the durable copy is established — store it as $DEFAULT.zip + # so the priority-overwrite chain below can supersede it with a fresher source. + # + # Prints a decision for every release the API listed, so a version missing from + # the site can be explained from this log alone. Downloads run in parallel: a repo + # with a hundred releases spent minutes here doing them one at a time. + - name: Gather release artifacts + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + relcache="$RUNNER_TEMP/relcache"; mkdir -p "$relcache" + todo="$RUNNER_TEMP/relcache-todo"; : > "$todo" + failed="$RUNNER_TEMP/relcache-failed"; mkdir -p "$failed" + + # Pass 1 — account for every release, and queue the cache misses. + printf '%-28s %-21s %-9s %s\n' TAG CREATED SIZE DECISION + while IFS=$'\t' read -r tag dest created size assetid decision; do + [ -n "$tag" ] || continue + if [ "$dest" != "-" ]; then + if [ -s "$relcache/$assetid.zip" ]; then + decision="$decision (cache hit)" + else + # Space-separated is safe: a git refname can never contain a space. + echo "$tag $assetid" >> "$todo" + decision="$decision (downloading)" + fi + fi + printf '%-28s %-21s %-9s %s\n' "$tag" "$created" "$size" "$decision" + # NB tab is IFS whitespace, so `read` COLLAPSES consecutive tabs — an empty + # field would shift every column after it. assemble.mjs emits "-", never "". + done < "$RUNNER_TEMP/releases.tsv" + + # Pass 2 — fetch the misses in parallel. Each worker is independent and + # records its own failure, so one bad asset cannot abort the rest. + n=$(grep -c . "$todo" || true) + echo "release zips: $(( $(awk -F'\t' '$2 != "-"' "$RUNNER_TEMP/releases.tsv" | wc -l) - n )) cached, $n to download" + if [ "$n" -gt 0 ]; then + export REPO relcache failed + # shellcheck disable=SC2016 + xargs -a "$todo" -P 8 -n 2 bash -c ' + set -uo pipefail + tag="$1"; assetid="$2" + if ! gh release download "$tag" --repo "$REPO" -p docs.zip \ + -O "$relcache/$assetid.zip" 2>"$failed/$assetid.err"; then + rm -f "$relcache/$assetid.zip" + touch "$failed/$assetid" + fi + ' _ + fi + + # Pass 3 — stage into the gather dir, in the TSV's order so the log is + # deterministic whatever order the downloads finished in. + while IFS=$'\t' read -r tag dest created size assetid decision; do + [ "$dest" = "-" ] && continue + if [ -f "$failed/$assetid" ]; then + echo "::warning::release $tag: docs.zip download failed — skipping. $(cat "$failed/$assetid.err" 2>/dev/null | head -1)" + elif [ -s "$relcache/$assetid.zip" ]; then + cp "$relcache/$assetid.zip" "$RUNNER_TEMP/gather/$dest.zip" + else + echo "::warning::release $tag: docs.zip missing after download — skipping" + fi + done < "$RUNNER_TEMP/releases.tsv" + + # Prune anything no longer selected, so a capped cache stays capped. + keep=$(awk -F'\t' '$2 != "-" {print $5 ".zip"}' "$RUNNER_TEMP/releases.tsv" | sort -u) + for f in "$relcache"/*.zip; do + [ -f "$f" ] || continue + grep -qxF "$(basename "$f")" <<<"$keep" || rm -f "$f" + done + + # Save only when the DEFAULT BRANCH was what got built. A cache written while a PR + # is the trigger is scoped to that PR and never reused by anything else, so saving + # there would just churn the 10 GB repo quota. This must test the BUILT branch, not + # github.ref — see BUILT_BRANCH above. Skipped on an exact key hit: the entry is + # already there and cache entries are immutable. + - name: Save release zips to the cache + if: >- + (github.event.workflow_run.head_branch || github.ref_name) == github.event.repository.default_branch && + steps.relcache.outputs.cache-hit != 'true' + uses: actions/cache/save@v6 + with: + path: ${{ runner.temp }}/relcache + key: ${{ steps.releases.outputs.cache-key }} + + # Branch CI artifacts overwrite the seed (higher priority). Found via the + # artifacts API by NAME (`docs` — the contract), never by workflow filename, + # so the consumer's entry workflow can be called anything. An artifact's + # workflow_run carries head_branch/head_sha/head_repository_id but no run + # conclusion — existence is the signal (docs.yml only uploads after a + # successful build). Default branch: newest non-expired docs artifact built + # from THIS repo's code on that branch (head_repository_id keeps a same-named + # fork branch out). PRs: keyed by head SHA — internal always; external forks + # only when the SHA carries a preview-approved status. No + # $ARTIFACT_VERSION_NAME skip — the current build (staged next) overwrites + # anything gathered here with the same name. + # + # The artifact list is paginated ONCE and indexed by head SHA. It used to be + # re-paginated per open PR, which on a busy repo (26 open PRs, thousands of + # artifacts) was ~175 s of identical API calls. + - name: Gather branch CI artifacts + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + repo_id=$(gh api "repos/$REPO" -q .id) + arts="$RUNNER_TEMP/artifacts.json" + gh api --paginate "repos/$REPO/actions/artifacts?name=docs&per_page=100" \ + | jq -rs '[.[].artifacts[] | select(.expired | not)]' > "$arts" + echo "$(jq length "$arts") non-expired 'docs' artifact(s) indexed" + + # Ascending by created_at so the last write per SHA is the newest artifact. + declare -A BY_SHA + while IFS=$'\t' read -r sha id created size run; do + [ -n "$sha" ] && BY_SHA["$sha"]="$id"$'\t'"$created"$'\t'"$size"$'\t'"$run" + done < <(jq -r 'sort_by(.created_at) | .[] + | [.workflow_run.head_sha, .id, .created_at, .size_in_bytes, .workflow_run.id] | @tsv' "$arts") + + # want.tsv: dest artifact-id label — resolved before any download, + # so the downloads can all run at once. + want="$RUNNER_TEMP/artifact-want.tsv"; : > "$want" + queue() { + local dest="$1" sha="$2" label="$3" meta="${BY_SHA[$2]:-}" + if [ -z "$meta" ]; then + echo " $label → no non-expired 'docs' artifact for ${sha:0:8} — skipping" + return 0 + fi + local id created size run + IFS=$'\t' read -r id created size run <<<"$meta" + echo " $label → artifact $id ($size B, built $created) from $GITHUB_SERVER_URL/$REPO/actions/runs/$run @ ${sha:0:8}" + printf '%s\t%s\t%s\n' "$dest" "$id" "$label" >> "$want" + } + + echo "default branch:" + dsha=$(jq -r --arg b "$DEFAULT" --argjson r "$repo_id" ' + [.[] | select(.workflow_run.head_branch == $b and .workflow_run.head_repository_id == $r)] + | sort_by(.created_at) | last | .workflow_run.head_sha // empty' "$arts") + if [ -n "$dsha" ]; then + queue "$DEFAULT" "$dsha" "default-branch CI ($DEFAULT)" + else + echo " default-branch CI ($DEFAULT) → no non-expired 'docs' artifact — skipping" + fi + + prs=$(gh pr list --repo "$REPO" --state open --limit 200 \ + --json number,headRefOid,isCrossRepository -q '.[] | [.number, .headRefOid, .isCrossRepository] | @tsv') + echo "open PRs: $(grep -c . <<<"$prs" || true)" + while IFS=$'\t' read -r num sha cross; do + [ -z "$num" ] && continue + if [ "$cross" = "true" ]; then + approved=$(gh api "repos/$REPO/commits/$sha/statuses" \ + -q 'any(.[]; .context=="preview-approved" and .state=="success")') + if [ "$approved" != "true" ]; then + echo " PR #$num → fork PR, head ${sha:0:8} not preview-approved — skipping" + continue + fi + fi + queue "pr-$num" "$sha" "PR #$num" + done <<< "$prs" + + # The API serves an artifact as a zip-of-the-artifact, with docs.zip inside. + # Three failure modes are reported distinctly because they need different + # fixes: download failure (transient), no docs.zip inside (an artifact built + # by a docs.yml older than the packing contract — re-run that build), and the + # not-found case already handled above. + if [ -s "$want" ]; then + export REPO + gather_dir="$RUNNER_TEMP/gather"; export gather_dir + # shellcheck disable=SC2016 + xargs -a "$want" -P 8 -d '\n' -I{} bash -c ' + set -uo pipefail + IFS=$'"'"'\t'"'"' read -r dest id label <<<"{}" + tmp=$(mktemp -d) + if ! gh api "repos/$REPO/actions/artifacts/$id/zip" > "$tmp/artifact.zip" 2>/dev/null; then + echo "::warning::$label: artifact $id download failed — skipping" + elif ! unzip -q "$tmp/artifact.zip" docs.zip -d "$tmp" >/dev/null 2>&1 || [ ! -f "$tmp/docs.zip" ]; then + # The FILE test, not unzip'"'"'s exit status: unzip exits 0 with only a + # "caution: filename not matched" note when the member is absent. + echo "::warning::$label: artifact $id contains no docs.zip — it predates the docs.zip packing contract. Re-run that build to refresh it. Skipping." + else + mv "$tmp/docs.zip" "$gather_dir/$dest.zip" + fi + rm -rf "$tmp" + ' + fi + + # The default branch's last known-good docs.zip. Releases are durable and PRs are + # ephemeral by design, but the DEFAULT BRANCH has no permanent source: it is + # gathered from its latest CI artifact, which expires, after which it would drop + # out and the guard below would hard-fail. + # + # This used to be solved by publishing the zip into the site itself + # (`_sources/.zip`) — permanent, but it shipped a multi-megabyte zip to + # every visitor's Pages artifact forever. The Actions cache holds it instead. The + # trade is real and deliberate: cache entries are evicted after 7 days without a + # read, and by LRU past 10 GB. A repo that goes quiet for over a week AND whose + # main CI artifact has expired loses this rung, where `_sources/` was forever. + # + # Key is a bare prefix so it never matches exactly; restore-keys then supplies the + # most recent entry. + - name: Restore the default branch's durable docs.zip + uses: actions/cache/restore@v6 + with: + path: ${{ runner.temp }}/defaultcache + key: mvs-default-v1- + restore-keys: mvs-default-v1- + + # Fallback chain for $DEFAULT.zip: current build → release/seed → branch CI (all + # above) → the cache → the deprecated in-site copy. Hard-fail if none of them + # produced it: a deploy missing the default branch publishes a hole. + - name: Ensure default-branch zip + id: ensure-default + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + cache="$RUNNER_TEMP/defaultcache"; mkdir -p "$cache" + zip="$RUNNER_TEMP/gather/$DEFAULT.zip" + if [ -f "$zip" ]; then + echo "$DEFAULT supplied by an earlier rung ($(stat -c%s "$zip") B)" + elif [ -s "$cache/default.zip" ]; then + cp "$cache/default.zip" "$zip" + echo "::notice::$DEFAULT restored from the Actions cache ($(stat -c%s "$zip") B) \ + — its CI artifact has expired, so this deploy is serving the last cached build" + # DEPRECATED rung, kept so a site deployed by an older publish.yml still has a + # source on its first upgraded deploy. Nothing writes _sources/ any more, so it + # disappears from the site on that very deploy; this read is one-shot insurance. + elif curl -fsSL "$PAGES_URL/$SOURCES_DIR/$DEFAULT.zip" -o "$zip" 2>/dev/null; then + echo "::notice::$DEFAULT restored from the deprecated in-site copy \ + ($PAGES_URL/$SOURCES_DIR/$DEFAULT.zip) — it will be cached from here on" + else + echo "::error::$DEFAULT: not provided by current build, releases, branch CI, \ + the Actions cache, or the deprecated in-site copy — deploy aborted" + exit 1 + fi + # Re-save only when the bytes actually changed: cache entries are immutable, so + # a content-addressed key means an unchanged default branch adds no entry. + new=$(sha256sum "$zip" | cut -c1-32) + old=$(cat "$cache/default.sha" 2>/dev/null || true) + if [ "$new" != "$old" ]; then + cp "$zip" "$cache/default.zip" + printf '%s' "$new" > "$cache/default.sha" + echo "cache-key=mvs-default-v1-$new" >> "$GITHUB_OUTPUT" + fi + + # Same rule as the release-zip cache, and the same trap: test the BUILT branch, not + # github.ref, which under workflow_run is always the default branch. + - name: Save the default branch's durable docs.zip + if: >- + steps.ensure-default.outputs.cache-key != '' && + (github.event.workflow_run.head_branch || github.ref_name) == github.event.repository.default_branch + uses: actions/cache/save@v6 + with: + path: ${{ runner.temp }}/defaultcache + key: ${{ steps.ensure-default.outputs.cache-key }} + + # Unzip every gathered zip into SITE/. + # + # THE ZIP-SHAPE CONTRACT: a docs.zip must unzip to exactly ONE top-level + # directory, whose CONTENTS become site/. The directory's NAME is + # ignored. docs.yml packs `html/`, but release assets are durable and long-lived + # — a repo migrating from python-copier-template's `_release.yml` has immutable + # releases carrying zips rooted at the TAG NAME (`mv html $GITHUB_REF_NAME; + # zip -r docs.zip $GITHUB_REF_NAME`), which cannot be re-cut. Keying on `html/` + # silently dropped every one of those releases from the switcher. Anything else + # — no entries, several entries, or files loose at the zip root — is malformed + # (nothing is known to produce it) and warns + skips rather than guessing. + - name: Extract artifacts into site + run: | + set -euo pipefail + for f in "$RUNNER_TEMP/gather"/*.zip; do + [ -f "$f" ] || continue + name=$(basename "$f" .zip) + tmp=$(mktemp -d) + if ! unzip -q "$f" -d "$tmp" 2>/dev/null; then + echo "::warning::$name: could not unzip docs.zip — skipping" + rm -rf "$tmp"; continue + fi + # -print0/-d '' so the count is exact whatever the entries are named, + # and dotfiles at the root count too (they would be a malformed zip). + roots=(); mapfile -d '' -t roots < <(find "$tmp" -mindepth 1 -maxdepth 1 -print0) + if [ "${#roots[@]}" -eq 1 ] && [ -d "${roots[0]}" ]; then + rm -rf "$RUNNER_TEMP/site/$name" + mv "${roots[0]}" "$RUNNER_TEMP/site/$name" + else + found=$(printf '%s ' "${roots[@]##*/}") + echo "::warning::$name: docs.zip must unzip to exactly one top-level directory — found [${found% }] — skipping" + fi + rm -rf "$tmp" + done + # upload-pages-artifact tars the whole site and GitHub Pages rejects the + # artifact over 1 GB. That ceiling arrives silently — deploys just start + # failing — so warn while there is still room to act. `max-releases` is the + # lever: it is unlimited by default, which is safe for a young project and a + # slow-motion footgun for an old one. + # + # The cap is on the COMPRESSED artifact, which is nothing like the size of the + # tree: this repo's own site is 734 MiB on disk but packs to 223 MiB, because + # HTML and JS compress ~3x. Comparing `du` against 1 GB would cry wolf at a + # third of the real usage. So measure the tree cheaply, and only pay for an + # actual pack when the tree is big enough for the answer to matter. + # + # -h matches upload-pages-artifact's --dereference: the `stable/` symlink is + # inflated to a full copy of a release in the real artifact, so a measurement + # that stored it as a symlink would under-read by a whole version. + bytes=$(du -sb "$RUNNER_TEMP/site" | cut -f1) + echo "assembled site: $(numfmt --to=iec --suffix=B "$bytes") uncompressed across \ + $(find "$RUNNER_TEMP/site" -mindepth 1 -maxdepth 1 -type d | wc -l) version dir(s)" + if [ "$bytes" -gt "$SIZE_PROBE_BYTES" ]; then + packed=$(tar -ch -C "$RUNNER_TEMP/site" . | gzip -1 -c | wc -c) + echo "packed artifact: ~$(numfmt --to=iec --suffix=B "$packed") (Pages rejects over 1 GB)" + if [ "$packed" -gt "$SIZE_WARN_BYTES" ]; then + echo "::warning title=Docs site approaching the Pages 1 GB limit::The \ + deployed artifact is ~$(numfmt --to=iec --suffix=B "$packed"), against a hard \ + 1 GB cap. Set publish.yml's \`max-releases\` input (in your publish-dispatch \ + shim) to publish only the N most recent releases." + fi + fi + + # `generate` orders versions, writes switcher.json + index.html, and prints the + # stable-alias source dir (the newest release), or "" when no release is deployed yet. + # Hard-fails if the default branch is absent (Ensure step guarantees the zip, but + # the extract could still fail if the zip was malformed). + - name: Generate switcher.json and stable alias + run: | + set -euo pipefail + stable_src=$(node "$GITHUB_WORKSPACE/.mvs/assemble/assemble.mjs" generate \ + --site-dir "$RUNNER_TEMP/site" --base-url "$PAGES_URL" --required "$DEFAULT") + [ -n "$stable_src" ] && ln -s "$stable_src" "$RUNNER_TEMP/site/stable" || true + + - name: Upload Pages artifact + uses: actions/upload-pages-artifact@v5 + with: + path: ${{ runner.temp }}/site + + # Deploy the uploaded artifact. deploy-pages stamps the deployment with + # pages_build_version = GITHUB_SHA and has no input to change it; the value is + # server-validated against the OIDC commit claim, so it CANNOT be overridden + # (a custom value 404s — actions/deploy-pages#383). The Pages backend silently + # DROPS a *second* deploy of an already-deployed SHA on a non-`workflow_dispatch` + # event (it reports success but the origin keeps serving the first artifact), + # EXCEPT a `workflow_dispatch` deploy of that SHA, which forces a re-serve. A + # release tag shares the default branch's just-deployed SHA, so an inline tag + # deploy would be dropped — which is why tags go through the `re-dispatch` job + # below (re-fired as workflow_dispatch) instead of here. main/internal-PR + # deploys are the FIRST of their SHA, so they serve fine inline. The Verify step + # below backstops any residual stale-origin case. + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v5 + + # deploy-pages reports success once the deployment RECORD flips active — even + # if the Pages backend silently fails to update the SERVED origin (a wedged + # site can serve a stale artifact indefinitely; see the 2026-06 incident where + # a gh-pages→Actions cutover froze the origin while every deploy reported OK). + # The re-dispatch (deploy via workflow_dispatch) prevents the tag-after-merge + # drop; this is the backstop that turns any *other* stale-origin failure into a + # red check instead of silently serving old docs. Confirm the live switcher.json + # byte-matches the one we just assembled, polling up to ~6 min — the window is + # generous because a release deploys TWICE in quick succession on the same SHA (the + # merge-to-default-branch push, then the tag re-dispatch), and the second (dispatch) + # re-serve can take a few minutes to propagate to the origin behind the first + # (observed ~4 min on 2026-06-25). The loop exits the instant the origin matches, + # so a fast/first-of-SHA deploy isn't slowed. On failure, dump the origin headers + # (a frozen etag/last-modified is the wedge signature). + - name: Verify the deployed origin matches what we assembled + id: verify + env: + PAGE_URL: ${{ steps.deployment.outputs.page_url }} + SITE_DIR: ${{ runner.temp }}/site + run: | + set -euo pipefail + [ -n "$PAGE_URL" ] || { echo "::error::deploy-pages did not report a page_url"; exit 1; } + local_json="$SITE_DIR/switcher.json" + [ -f "$local_json" ] || { echo "::error::no switcher.json at $local_json"; exit 1; } + # Command substitution strips trailing newlines from both sides, so the + # comparison is robust to a curl/file trailing-newline difference. + want=$(cat "$local_json") + url="${PAGE_URL%/}/switcher.json" + timeout_s=360 + deadline=$(( $(date +%s) + timeout_s )) + attempt=0 + while :; do + attempt=$((attempt + 1)) + got=$(curl -fsS "$url?cb=$(date +%s%N)" || true) # cache-bust → hit the origin + if [ -n "$got" ] && [ "$got" = "$want" ]; then + echo "origin matches the assembled switcher.json (attempt $attempt)" + exit 0 + fi + if [ "$(date +%s)" -ge "$deadline" ]; then + echo "::error::live switcher.json still does not match the assembled one after ${timeout_s}s — Pages origin likely stale/wedged" + echo "--- origin headers (frozen etag/last-modified ⇒ wedged) ---" + curl -sI "$url?cb=$(date +%s%N)" | grep -iE '^(HTTP/|etag|last-modified|age)' | tr -d '\r' + echo "--- assembled ($local_json) ---"; printf '%s\n' "$want" + echo "--- live ($url) ---"; printf '%s\n' "$got" + exit 1 + fi + sleep 5 + done diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index a5e0448..6d17214 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -1,695 +1,77 @@ name: Publish -# Reconstruct the WHOLE versioned docs site from durable sources (the consumer's -# default-branch build, release docs.zip assets, open-PR build artifacts) and deploy it -# directly to GitHub Pages — no gh-pages branch. The PRIVILEGED half. Its only caller is -# a thin `publish-dispatch.yml` shim (this repo's own, or a consumer's pinned at ) -# which exposes it as both `workflow_call` (ci.yml's `publish` job, for every event) and -# `workflow_dispatch` (the tag re-dispatch + fork-PR opt-in + manual re-deploy). -# This workflow OWNS the event branching, into one of three jobs: +# The one file each consuming repo carries, and the single place they pin . It does +# nothing but call the engine (publish-gh-pages.yml) when CI finishes. # -# deploy — internal PR / default-branch push / any workflow_dispatch → assemble + -# deploy + verify. The inline path injects the in-run build via -# `version-name`; the dispatch paths pure-gather (version-name ""). -# re-dispatch — a TAG push. An inline tag deploy is silently DROPPED by Pages (it reuses -# the default branch's just-deployed SHA on a non-dispatch event — -# deploy-pages#383), so wait for the release, then re-fire the shim -# (`inputs.dispatch-workflow`, a file in the CONSUMER's repo — a -# workflow_call run executes with the caller's repo + token, so we can -# dispatch it) → the deploy lands on a `workflow_dispatch` event and -# re-serves. A reusable workflow can't be dispatched cross-repo, which is -# why the shim (not this engine) is the dispatch target. -# warn — a FORK PR. Read-only token, never deploys; just surfaces the opt-in hint. +# This repo's copy is IDENTICAL to a consumer's but for the `uses:` path — theirs pins +# DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish-gh-pages.yml@, +# this one uses the local file so the working tree is what gets tested. # -# The assemble logic is THIS repo's assemble.mjs (the pure kernel); the IO plumbing -# lives as inline steps in this file. To version-match assemble.mjs to this workflow, -# we check out the reusable workflow's OWN repo at its OWN commit via the `job` context: -# unlike github.workflow_* (which resolve to the CALLER's entry workflow), -# `job.workflow_repository` + `job.workflow_sha` resolve to the file that defines THIS -# job — i.e. the reusable workflow at the ref a consumer pinned `publish.yml@` to. -# So assemble.mjs always matches the workflow version, with no hardcoded repo and no -# manual release-time bump. assemble.mjs runs `git tag` in the cwd, so the CONSUMER's -# repo is checked out at the root (with tags) to supply their version list; assemble.mjs -# comes from .mvs. +# WHY workflow_run rather than a job in ci.yml. Publishing is O(the whole site) — it +# reconstructs every version on every run — so nesting it in CI put a constant, sizeable +# cost inside every PR's critical path, which is exactly why the first consumer switched +# PR previews off a day after adopting. On workflow_run it runs after CI, on its own time, +# and a failed deploy no longer shows up as a red check against an author who has no +# control over the Pages infrastructure. +# +# It also DELETED machinery rather than adding any: because workflow_run fires once the +# triggering run has COMPLETED, that run's `docs` artifact is discoverable by the ordinary +# gather, so there is no in-run artifact to inject; and because a tag's deploy re-serves +# on this event (see the engine's header), there is no trampoline and so no need for a +# separately dispatchable shim. on: - workflow_call: + # `workflows:` matches the entry workflow by its `name:`, NOT its filename — if you + # rename your CI workflow, change it here too or publishing silently stops. + workflow_run: + workflows: [CI] + types: [completed] + + # Fork-PR previews (`pr`) and manual re-deploys. A maintainer runs this to publish a + # preview for an external contributor, which is the only way a fork's build reaches the + # site — see the guard below. + workflow_dispatch: inputs: - version-name: - description: Version name (pr- | | ) of the in-run build to inject; "" → pure durable gather. - required: false - default: "" - type: string pr: - description: External fork PR number to approve (pins its head SHA) and preview. - required: false - default: "" - type: string - max-releases: - description: >- - Publish only the N most recent releases (by the tagged commit's date), or 0 for - all of them. The site is deployed as ONE Pages artifact and Pages caps that at - 1 GB, so an old project with hundreds of released docs.zips will eventually - fail to deploy at all — a repo with 131 of them was already at 452 MB. Set this - in your publish-dispatch shim's `with:` block so it applies on every path. - required: false - default: "0" - type: string - dispatch-workflow: - description: >- - Filename of the publish-dispatch shim in the consumer's repo, re-fired by - the tag re-dispatch. Defaults to the conventional name; override if you renamed it. - required: false - default: publish-dispatch.yml - type: string - retry-until: - description: >- - Internal — epoch-seconds deadline for auto-retrying a wedged Pages origin. - Empty means no retry budget (set only by the tag re-dispatch job, from the - release's publish time); carried unchanged through any retries. Don't set manually. + description: "Fork PR to approve + preview (empty = plain re-deploy)" required: false default: "" - type: string jobs: - # Fork PR: read-only token, never deploys — surface the manual-opt-in hint on the PR. - warn: - if: >- - github.event_name == 'pull_request' && - github.event.pull_request.head.repo.full_name != github.repository - runs-on: ubuntu-latest - permissions: {} - steps: - - name: Explain the fork-preview opt-in - run: | - echo "::warning title=Docs preview not published::This is a fork PR, so the \ - versioned docs site is NOT auto-published (fork builds run with a read-only \ - token). A maintainer can publish a preview by running the \ - ${{ inputs.dispatch-workflow }} workflow for PR #${{ github.event.pull_request.number }}: \ - https://github.com/${{ github.repository }}/actions/workflows/${{ inputs.dispatch-workflow }}" - - # Internal PR / default-branch push (inline), or any workflow_dispatch → deploy. - deploy: - # `workflow_run` is TEMPORARY — _probe-workflow-run.yml's experiment (delete both once - # it has answered). Its caller already gates on conclusion + non-fork; the deploy - # itself needs no other change, because version-name "" is an existing supported mode. + publish: + # Two guards, both load-bearing: + # + # conclusion == 'success' — a failed CI run has no usable docs artifact, and + # workflow_run fires on failures too. + # + # head_repository == this repo — workflow_run runs with a WRITE token even when the + # triggering run came from a fork's pull request. Deploying a fork's build + # automatically would hand untrusted content a privileged path to the live site + # (the classic pwn-request shape). Fork builds reach the site only when a maintainer + # dispatches this workflow with `pr`, which records an approval against that exact + # head SHA — so a later push to the PR drops the preview until re-approved. + # + # A manual dispatch has no `workflow_run` payload, so both terms are empty and the + # first clause carries it. if: >- github.event_name == 'workflow_dispatch' || - github.event_name == 'workflow_run' || - (github.event_name == 'push' && github.ref_type != 'tag') || - (github.event_name == 'pull_request' && - github.event.pull_request.head.repo.full_name == github.repository) - runs-on: ubuntu-latest - permissions: - contents: read # checkout + read release assets - actions: write # gather cross-run docs artifacts (read) + retry-dispatch on a wedged origin (write) - pages: write # deploy to Pages - id-token: write # deploy-pages OIDC - statuses: write # set the preview-approved status on a fork PR head SHA - environment: - name: github-pages - url: ${{ steps.deployment.outputs.page_url }} - concurrency: - group: pages - cancel-in-progress: false - env: - REPO: ${{ github.repository }} - DEFAULT: ${{ github.event.repository.default_branch }} - SEED_TAG: pages-default-seed - # Site-size guard thresholds, both on the deployed artifact's scale. PROBE is the - # uncompressed size past which it is worth actually packing the tree to find out - # (the pack costs real seconds); WARN is the packed size that earns a warning, - # 700 MB against Pages' hard 1 GB. Overridable so the guard is testable and so a - # site with unusual compressibility can be tuned. - SIZE_PROBE_BYTES: "1610612736" - SIZE_WARN_BYTES: "734003200" - # DEPRECATED: the in-site durable store. Nothing writes it any more (the default - # branch's zip lives in the Actions cache); this is only still read as a one-shot - # fallback for a site last deployed by an older publish.yml. - SOURCES_DIR: _sources - steps: - # The CONSUMER's repo, at the root cwd: assemble.mjs runs `git tag` here to - # order that repo's deployed versions, so it needs their tags (fetch-depth: 0). - - uses: actions/checkout@v7 - with: - fetch-depth: 0 - - # THIS reusable workflow's repo + commit (the `job` context resolves to the - # reusable file, not the caller), so assemble.mjs matches the ref the consumer - # pinned publish.yml@ to — automatically, no version bump. - # Sparse: only assemble/ is needed. - - name: Fetch assemble.mjs - uses: actions/checkout@v7 - with: - repository: ${{ job.workflow_repository }} - ref: ${{ job.workflow_sha }} - path: .mvs - sparse-checkout: assemble - - # Fork opt-in: pin the PR's head commit as approved. assemble gathers an external - # PR only when its head SHA carries this status, so a later push (new SHA) silently - # drops the preview until re-approved. Gated on `pr` alone, which the shim threads - # in from its dispatch input. - - name: Approve fork PR head SHA - if: inputs.pr != '' - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - sha=$(gh pr view "${{ inputs.pr }}" --repo "$REPO" --json headRefOid -q .headRefOid) - gh api --method POST "repos/$REPO/statuses/$sha" \ - -f state=success -f context=preview-approved \ - -f description="Fork docs preview approved" - - - name: Create staging dirs - run: mkdir -p "$RUNNER_TEMP/site" "$RUNNER_TEMP/gather" - - # The live Pages base URL, from the Pages API — respects a custom domain - # (CNAME); falls back to the conventional github.io URL if the API call - # fails. Used by the _sources restore below and baked into switcher.json. - - name: Resolve the Pages URL - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - url=$(gh api "repos/$REPO/pages" -q .html_url 2>/dev/null || true) - if [ -z "$url" ]; then - owner=$(printf '%s' "${REPO%%/*}" | tr '[:upper:]' '[:lower:]') - url="https://$owner.github.io/${REPO#*/}" - fi - echo "PAGES_URL=${url%/}" >> "$GITHUB_ENV" - echo "Pages URL: ${url%/}" - - # ── Which releases, and which of their zips do we already have? ───────── + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.head_repository.full_name == github.repository) + uses: ./.github/workflows/publish-gh-pages.yml + with: + # Set (dispatch only) → approve and preview that fork PR's head SHA. + pr: ${{ inputs.pr }} + # Publish only the N most recent releases; 0 = all of them. A literal, because this + # is site-size policy and must hold on every path. The site deploys as ONE Pages + # artifact against a hard 1 GB cap — see docs/how-to/keep-the-site-small.md. # - # Selection is assemble.mjs's job (pure, unit-tested); this step only does IO. - # It runs BEFORE any download so the gather fetches only what will actually be - # published, and so the cache key describes exactly that set. - # - # Cached zips are named by ASSET ID, not by tag: an asset id identifies its bytes - # unambiguously, so a re-cut release can never be served from a stale cache entry. - - name: Select releases to publish - id: releases - env: - GH_TOKEN: ${{ github.token }} - MAX_RELEASES: ${{ inputs.max-releases }} - run: | - set -euo pipefail - gh api --paginate "repos/$REPO/releases" | jq -rs 'add // []' > "$RUNNER_TEMP/releases.json" - node "$GITHUB_WORKSPACE/.mvs/assemble/assemble.mjs" select-releases \ - --default-branch "$DEFAULT" --seed-tag "$SEED_TAG" \ - --max-releases "${MAX_RELEASES:-0}" \ - < "$RUNNER_TEMP/releases.json" > "$RUNNER_TEMP/releases.tsv" - # Key on the exact set of asset ids we intend to publish, so any change to - # that set misses and re-saves. restore-keys then falls back to the previous - # set, which shares nearly every zip — only the new release is downloaded. - digest=$(awk -F'\t' '$2 != "-" {print $5}' "$RUNNER_TEMP/releases.tsv" \ - | sort | sha256sum | cut -c1-32) - echo "cache-key=mvs-relzips-v1-$digest" >> "$GITHUB_OUTPUT" - - - name: Restore cached release zips - id: relcache - uses: actions/cache/restore@v6 - with: - path: ${{ runner.temp }}/relcache - key: ${{ steps.releases.outputs.cache-key }} - restore-keys: mvs-relzips-v1- - - # Releases are permanent sources; the seed release (if present) stands in for the - # default branch until the durable copy is established — store it as $DEFAULT.zip - # so the priority-overwrite chain below can supersede it with a fresher source. - # - # Prints a decision for every release the API listed, so a version missing from - # the site can be explained from this log alone. Downloads run in parallel: a repo - # with a hundred releases spent minutes here doing them one at a time. - - name: Gather release artifacts - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - relcache="$RUNNER_TEMP/relcache"; mkdir -p "$relcache" - todo="$RUNNER_TEMP/relcache-todo"; : > "$todo" - failed="$RUNNER_TEMP/relcache-failed"; mkdir -p "$failed" - - # Pass 1 — account for every release, and queue the cache misses. - printf '%-28s %-21s %-9s %s\n' TAG CREATED SIZE DECISION - while IFS=$'\t' read -r tag dest created size assetid decision; do - [ -n "$tag" ] || continue - if [ "$dest" != "-" ]; then - if [ -s "$relcache/$assetid.zip" ]; then - decision="$decision (cache hit)" - else - # Space-separated is safe: a git refname can never contain a space. - echo "$tag $assetid" >> "$todo" - decision="$decision (downloading)" - fi - fi - printf '%-28s %-21s %-9s %s\n' "$tag" "$created" "$size" "$decision" - # NB tab is IFS whitespace, so `read` COLLAPSES consecutive tabs — an empty - # field would shift every column after it. assemble.mjs emits "-", never "". - done < "$RUNNER_TEMP/releases.tsv" - - # Pass 2 — fetch the misses in parallel. Each worker is independent and - # records its own failure, so one bad asset cannot abort the rest. - n=$(grep -c . "$todo" || true) - echo "release zips: $(( $(awk -F'\t' '$2 != "-"' "$RUNNER_TEMP/releases.tsv" | wc -l) - n )) cached, $n to download" - if [ "$n" -gt 0 ]; then - export REPO relcache failed - # shellcheck disable=SC2016 - xargs -a "$todo" -P 8 -n 2 bash -c ' - set -uo pipefail - tag="$1"; assetid="$2" - if ! gh release download "$tag" --repo "$REPO" -p docs.zip \ - -O "$relcache/$assetid.zip" 2>"$failed/$assetid.err"; then - rm -f "$relcache/$assetid.zip" - touch "$failed/$assetid" - fi - ' _ - fi - - # Pass 3 — stage into the gather dir, in the TSV's order so the log is - # deterministic whatever order the downloads finished in. - while IFS=$'\t' read -r tag dest created size assetid decision; do - [ "$dest" = "-" ] && continue - if [ -f "$failed/$assetid" ]; then - echo "::warning::release $tag: docs.zip download failed — skipping. $(cat "$failed/$assetid.err" 2>/dev/null | head -1)" - elif [ -s "$relcache/$assetid.zip" ]; then - cp "$relcache/$assetid.zip" "$RUNNER_TEMP/gather/$dest.zip" - else - echo "::warning::release $tag: docs.zip missing after download — skipping" - fi - done < "$RUNNER_TEMP/releases.tsv" - - # Prune anything no longer selected, so a capped cache stays capped. - keep=$(awk -F'\t' '$2 != "-" {print $5 ".zip"}' "$RUNNER_TEMP/releases.tsv" | sort -u) - for f in "$relcache"/*.zip; do - [ -f "$f" ] || continue - grep -qxF "$(basename "$f")" <<<"$keep" || rm -f "$f" - done - - # Save only from the default branch. A cache written on a PR ref is scoped to that - # PR and never reused by anything else, so saving there would just churn the 10 GB - # repo quota. Skipped on an exact key hit — the entry is already there and cache - # entries are immutable. - - name: Save release zips to the cache - if: >- - github.ref == format('refs/heads/{0}', github.event.repository.default_branch) && - steps.relcache.outputs.cache-hit != 'true' - uses: actions/cache/save@v6 - with: - path: ${{ runner.temp }}/relcache - key: ${{ steps.releases.outputs.cache-key }} - - # Branch CI artifacts overwrite the seed (higher priority). Found via the - # artifacts API by NAME (`docs` — the contract), never by workflow filename, - # so the consumer's entry workflow can be called anything. An artifact's - # workflow_run carries head_branch/head_sha/head_repository_id but no run - # conclusion — existence is the signal (docs.yml only uploads after a - # successful build). Default branch: newest non-expired docs artifact built - # from THIS repo's code on that branch (head_repository_id keeps a same-named - # fork branch out). PRs: keyed by head SHA — internal always; external forks - # only when the SHA carries a preview-approved status. No - # $ARTIFACT_VERSION_NAME skip — the current build (staged next) overwrites - # anything gathered here with the same name. - # - # The artifact list is paginated ONCE and indexed by head SHA. It used to be - # re-paginated per open PR, which on a busy repo (26 open PRs, thousands of - # artifacts) was ~175 s of identical API calls. - - name: Gather branch CI artifacts - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - repo_id=$(gh api "repos/$REPO" -q .id) - arts="$RUNNER_TEMP/artifacts.json" - gh api --paginate "repos/$REPO/actions/artifacts?name=docs&per_page=100" \ - | jq -rs '[.[].artifacts[] | select(.expired | not)]' > "$arts" - echo "$(jq length "$arts") non-expired 'docs' artifact(s) indexed" - - # Ascending by created_at so the last write per SHA is the newest artifact. - declare -A BY_SHA - while IFS=$'\t' read -r sha id created size run; do - [ -n "$sha" ] && BY_SHA["$sha"]="$id"$'\t'"$created"$'\t'"$size"$'\t'"$run" - done < <(jq -r 'sort_by(.created_at) | .[] - | [.workflow_run.head_sha, .id, .created_at, .size_in_bytes, .workflow_run.id] | @tsv' "$arts") - - # want.tsv: dest artifact-id label — resolved before any download, - # so the downloads can all run at once. - want="$RUNNER_TEMP/artifact-want.tsv"; : > "$want" - queue() { - local dest="$1" sha="$2" label="$3" meta="${BY_SHA[$2]:-}" - if [ -z "$meta" ]; then - echo " $label → no non-expired 'docs' artifact for ${sha:0:8} — skipping" - return 0 - fi - local id created size run - IFS=$'\t' read -r id created size run <<<"$meta" - echo " $label → artifact $id ($size B, built $created) from $GITHUB_SERVER_URL/$REPO/actions/runs/$run @ ${sha:0:8}" - printf '%s\t%s\t%s\n' "$dest" "$id" "$label" >> "$want" - } - - echo "default branch:" - dsha=$(jq -r --arg b "$DEFAULT" --argjson r "$repo_id" ' - [.[] | select(.workflow_run.head_branch == $b and .workflow_run.head_repository_id == $r)] - | sort_by(.created_at) | last | .workflow_run.head_sha // empty' "$arts") - if [ -n "$dsha" ]; then - queue "$DEFAULT" "$dsha" "default-branch CI ($DEFAULT)" - else - echo " default-branch CI ($DEFAULT) → no non-expired 'docs' artifact — skipping" - fi - - prs=$(gh pr list --repo "$REPO" --state open --limit 200 \ - --json number,headRefOid,isCrossRepository -q '.[] | [.number, .headRefOid, .isCrossRepository] | @tsv') - echo "open PRs: $(grep -c . <<<"$prs" || true)" - while IFS=$'\t' read -r num sha cross; do - [ -z "$num" ] && continue - if [ "$cross" = "true" ]; then - approved=$(gh api "repos/$REPO/commits/$sha/statuses" \ - -q 'any(.[]; .context=="preview-approved" and .state=="success")') - if [ "$approved" != "true" ]; then - echo " PR #$num → fork PR, head ${sha:0:8} not preview-approved — skipping" - continue - fi - fi - queue "pr-$num" "$sha" "PR #$num" - done <<< "$prs" - - # The API serves an artifact as a zip-of-the-artifact, with docs.zip inside. - # Three failure modes are reported distinctly because they need different - # fixes: download failure (transient), no docs.zip inside (an artifact built - # by a docs.yml older than the packing contract — re-run that build), and the - # not-found case already handled above. - if [ -s "$want" ]; then - export REPO - gather_dir="$RUNNER_TEMP/gather"; export gather_dir - # shellcheck disable=SC2016 - xargs -a "$want" -P 8 -d '\n' -I{} bash -c ' - set -uo pipefail - IFS=$'"'"'\t'"'"' read -r dest id label <<<"{}" - tmp=$(mktemp -d) - if ! gh api "repos/$REPO/actions/artifacts/$id/zip" > "$tmp/artifact.zip" 2>/dev/null; then - echo "::warning::$label: artifact $id download failed — skipping" - elif ! unzip -q "$tmp/artifact.zip" docs.zip -d "$tmp" >/dev/null 2>&1 || [ ! -f "$tmp/docs.zip" ]; then - # The FILE test, not unzip'"'"'s exit status: unzip exits 0 with only a - # "caution: filename not matched" note when the member is absent. - echo "::warning::$label: artifact $id contains no docs.zip — it predates the docs.zip packing contract. Re-run that build to refresh it. Skipping." - else - mv "$tmp/docs.zip" "$gather_dir/$dest.zip" - fi - rm -rf "$tmp" - ' - fi - - # Highest priority: the current build overwrites anything gathered above. - # Skipped on the dispatch paths (version-name is ""), which gather purely from durable sources. - - name: Download this run's docs artifact - if: inputs.version-name != '' - uses: actions/download-artifact@v8 - with: - name: docs - path: ${{ runner.temp }}/gather-current - - name: Stage current build - if: inputs.version-name != '' - env: - ARTIFACT_VERSION_NAME: ${{ inputs.version-name }} - run: | - set -euo pipefail - src="$RUNNER_TEMP/gather-current/docs.zip" - [ -f "$src" ] || { echo "::error::in-run artifact not found at $src"; exit 1; } - mv "$src" "$RUNNER_TEMP/gather/$ARTIFACT_VERSION_NAME.zip" - - # The default branch's last known-good docs.zip. Releases are durable and PRs are - # ephemeral by design, but the DEFAULT BRANCH has no permanent source: it is - # gathered from its latest CI artifact, which expires, after which it would drop - # out and the guard below would hard-fail. - # - # This used to be solved by publishing the zip into the site itself - # (`_sources/.zip`) — permanent, but it shipped a multi-megabyte zip to - # every visitor's Pages artifact forever. The Actions cache holds it instead. The - # trade is real and deliberate: cache entries are evicted after 7 days without a - # read, and by LRU past 10 GB. A repo that goes quiet for over a week AND whose - # main CI artifact has expired loses this rung, where `_sources/` was forever. - # - # Key is a bare prefix so it never matches exactly; restore-keys then supplies the - # most recent entry. - - name: Restore the default branch's durable docs.zip - uses: actions/cache/restore@v6 - with: - path: ${{ runner.temp }}/defaultcache - key: mvs-default-v1- - restore-keys: mvs-default-v1- - - # Fallback chain for $DEFAULT.zip: current build → release/seed → branch CI (all - # above) → the cache → the deprecated in-site copy. Hard-fail if none of them - # produced it: a deploy missing the default branch publishes a hole. - - name: Ensure default-branch zip - id: ensure-default - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - cache="$RUNNER_TEMP/defaultcache"; mkdir -p "$cache" - zip="$RUNNER_TEMP/gather/$DEFAULT.zip" - if [ -f "$zip" ]; then - echo "$DEFAULT supplied by an earlier rung ($(stat -c%s "$zip") B)" - elif [ -s "$cache/default.zip" ]; then - cp "$cache/default.zip" "$zip" - echo "::notice::$DEFAULT restored from the Actions cache ($(stat -c%s "$zip") B) \ - — its CI artifact has expired, so this deploy is serving the last cached build" - # DEPRECATED rung, kept so a site deployed by an older publish.yml still has a - # source on its first upgraded deploy. Nothing writes _sources/ any more, so it - # disappears from the site on that very deploy; this read is one-shot insurance. - elif curl -fsSL "$PAGES_URL/$SOURCES_DIR/$DEFAULT.zip" -o "$zip" 2>/dev/null; then - echo "::notice::$DEFAULT restored from the deprecated in-site copy \ - ($PAGES_URL/$SOURCES_DIR/$DEFAULT.zip) — it will be cached from here on" - else - echo "::error::$DEFAULT: not provided by current build, releases, branch CI, \ - the Actions cache, or the deprecated in-site copy — deploy aborted" - exit 1 - fi - # Re-save only when the bytes actually changed: cache entries are immutable, so - # a content-addressed key means an unchanged default branch adds no entry. - new=$(sha256sum "$zip" | cut -c1-32) - old=$(cat "$cache/default.sha" 2>/dev/null || true) - if [ "$new" != "$old" ]; then - cp "$zip" "$cache/default.zip" - printf '%s' "$new" > "$cache/default.sha" - echo "cache-key=mvs-default-v1-$new" >> "$GITHUB_OUTPUT" - fi - - # Same rule as the release-zip cache: only the default branch writes, because a - # cache saved on a PR ref is scoped to that PR and no other run can read it. - - name: Save the default branch's durable docs.zip - if: >- - steps.ensure-default.outputs.cache-key != '' && - github.ref == format('refs/heads/{0}', github.event.repository.default_branch) - uses: actions/cache/save@v6 - with: - path: ${{ runner.temp }}/defaultcache - key: ${{ steps.ensure-default.outputs.cache-key }} - - # Unzip every gathered zip into SITE/. - # - # THE ZIP-SHAPE CONTRACT: a docs.zip must unzip to exactly ONE top-level - # directory, whose CONTENTS become site/. The directory's NAME is - # ignored. docs.yml packs `html/`, but release assets are durable and long-lived - # — a repo migrating from python-copier-template's `_release.yml` has immutable - # releases carrying zips rooted at the TAG NAME (`mv html $GITHUB_REF_NAME; - # zip -r docs.zip $GITHUB_REF_NAME`), which cannot be re-cut. Keying on `html/` - # silently dropped every one of those releases from the switcher. Anything else - # — no entries, several entries, or files loose at the zip root — is malformed - # (nothing is known to produce it) and warns + skips rather than guessing. - - name: Extract artifacts into site - run: | - set -euo pipefail - for f in "$RUNNER_TEMP/gather"/*.zip; do - [ -f "$f" ] || continue - name=$(basename "$f" .zip) - tmp=$(mktemp -d) - if ! unzip -q "$f" -d "$tmp" 2>/dev/null; then - echo "::warning::$name: could not unzip docs.zip — skipping" - rm -rf "$tmp"; continue - fi - # -print0/-d '' so the count is exact whatever the entries are named, - # and dotfiles at the root count too (they would be a malformed zip). - roots=(); mapfile -d '' -t roots < <(find "$tmp" -mindepth 1 -maxdepth 1 -print0) - if [ "${#roots[@]}" -eq 1 ] && [ -d "${roots[0]}" ]; then - rm -rf "$RUNNER_TEMP/site/$name" - mv "${roots[0]}" "$RUNNER_TEMP/site/$name" - else - found=$(printf '%s ' "${roots[@]##*/}") - echo "::warning::$name: docs.zip must unzip to exactly one top-level directory — found [${found% }] — skipping" - fi - rm -rf "$tmp" - done - # upload-pages-artifact tars the whole site and GitHub Pages rejects the - # artifact over 1 GB. That ceiling arrives silently — deploys just start - # failing — so warn while there is still room to act. `max-releases` is the - # lever: it is unlimited by default, which is safe for a young project and a - # slow-motion footgun for an old one. - # - # The cap is on the COMPRESSED artifact, which is nothing like the size of the - # tree: this repo's own site is 734 MiB on disk but packs to 223 MiB, because - # HTML and JS compress ~3x. Comparing `du` against 1 GB would cry wolf at a - # third of the real usage. So measure the tree cheaply, and only pay for an - # actual pack when the tree is big enough for the answer to matter. - # - # -h matches upload-pages-artifact's --dereference: the `stable/` symlink is - # inflated to a full copy of a release in the real artifact, so a measurement - # that stored it as a symlink would under-read by a whole version. - bytes=$(du -sb "$RUNNER_TEMP/site" | cut -f1) - echo "assembled site: $(numfmt --to=iec --suffix=B "$bytes") uncompressed across \ - $(find "$RUNNER_TEMP/site" -mindepth 1 -maxdepth 1 -type d | wc -l) version dir(s)" - if [ "$bytes" -gt "$SIZE_PROBE_BYTES" ]; then - packed=$(tar -ch -C "$RUNNER_TEMP/site" . | gzip -1 -c | wc -c) - echo "packed artifact: ~$(numfmt --to=iec --suffix=B "$packed") (Pages rejects over 1 GB)" - if [ "$packed" -gt "$SIZE_WARN_BYTES" ]; then - echo "::warning title=Docs site approaching the Pages 1 GB limit::The \ - deployed artifact is ~$(numfmt --to=iec --suffix=B "$packed"), against a hard \ - 1 GB cap. Set publish.yml's \`max-releases\` input (in your publish-dispatch \ - shim) to publish only the N most recent releases." - fi - fi - - # `generate` orders versions, writes switcher.json + index.html, and prints the - # stable-alias source dir (the newest release), or "" when no release is deployed yet. - # Hard-fails if the default branch is absent (Ensure step guarantees the zip, but - # the extract could still fail if the zip was malformed). - - name: Generate switcher.json and stable alias - run: | - set -euo pipefail - stable_src=$(node "$GITHUB_WORKSPACE/.mvs/assemble/assemble.mjs" generate \ - --site-dir "$RUNNER_TEMP/site" --base-url "$PAGES_URL" --required "$DEFAULT") - [ -n "$stable_src" ] && ln -s "$stable_src" "$RUNNER_TEMP/site/stable" || true - - - name: Upload Pages artifact - uses: actions/upload-pages-artifact@v5 - with: - path: ${{ runner.temp }}/site - - # Deploy the uploaded artifact. deploy-pages stamps the deployment with - # pages_build_version = GITHUB_SHA and has no input to change it; the value is - # server-validated against the OIDC commit claim, so it CANNOT be overridden - # (a custom value 404s — actions/deploy-pages#383). The Pages backend silently - # DROPS a *second* deploy of an already-deployed SHA on a non-`workflow_dispatch` - # event (it reports success but the origin keeps serving the first artifact), - # EXCEPT a `workflow_dispatch` deploy of that SHA, which forces a re-serve. A - # release tag shares the default branch's just-deployed SHA, so an inline tag - # deploy would be dropped — which is why tags go through the `re-dispatch` job - # below (re-fired as workflow_dispatch) instead of here. main/internal-PR - # deploys are the FIRST of their SHA, so they serve fine inline. The Verify step - # below backstops any residual stale-origin case. - - name: Deploy to GitHub Pages - id: deployment - uses: actions/deploy-pages@v5 - - # deploy-pages reports success once the deployment RECORD flips active — even - # if the Pages backend silently fails to update the SERVED origin (a wedged - # site can serve a stale artifact indefinitely; see the 2026-06 incident where - # a gh-pages→Actions cutover froze the origin while every deploy reported OK). - # The re-dispatch (deploy via workflow_dispatch) prevents the tag-after-merge - # drop; this is the backstop that turns any *other* stale-origin failure into a - # red check instead of silently serving old docs. Confirm the live switcher.json - # byte-matches the one we just assembled, polling up to ~6 min — the window is - # generous because a release deploys TWICE in quick succession on the same SHA (the - # merge-to-default-branch push, then the tag re-dispatch), and the second (dispatch) - # re-serve can take a few minutes to propagate to the origin behind the first - # (observed ~4 min on 2026-06-25). The loop exits the instant the origin matches, - # so a fast/first-of-SHA deploy isn't slowed. On failure, dump the origin headers - # (a frozen etag/last-modified is the wedge signature). - - name: Verify the deployed origin matches what we assembled - id: verify - env: - PAGE_URL: ${{ steps.deployment.outputs.page_url }} - SITE_DIR: ${{ runner.temp }}/site - run: | - set -euo pipefail - [ -n "$PAGE_URL" ] || { echo "::error::deploy-pages did not report a page_url"; exit 1; } - local_json="$SITE_DIR/switcher.json" - [ -f "$local_json" ] || { echo "::error::no switcher.json at $local_json"; exit 1; } - # Command substitution strips trailing newlines from both sides, so the - # comparison is robust to a curl/file trailing-newline difference. - want=$(cat "$local_json") - url="${PAGE_URL%/}/switcher.json" - timeout_s=360 - deadline=$(( $(date +%s) + timeout_s )) - attempt=0 - while :; do - attempt=$((attempt + 1)) - got=$(curl -fsS "$url?cb=$(date +%s%N)" || true) # cache-bust → hit the origin - if [ -n "$got" ] && [ "$got" = "$want" ]; then - echo "origin matches the assembled switcher.json (attempt $attempt)" - exit 0 - fi - if [ "$(date +%s)" -ge "$deadline" ]; then - echo "::error::live switcher.json still does not match the assembled one after ${timeout_s}s — Pages origin likely stale/wedged" - echo "--- origin headers (frozen etag/last-modified ⇒ wedged) ---" - curl -sI "$url?cb=$(date +%s%N)" | grep -iE '^(HTTP/|etag|last-modified|age)' | tr -d '\r' - echo "--- assembled ($local_json) ---"; printf '%s\n' "$want" - echo "--- live ($url) ---"; printf '%s\n' "$got" - exit 1 - fi - sleep 5 - done - - # A verify failure means the origin is wedged (see comment above) — and per - # observed external evidence (actions/deploy-pages#383: waiting longer does NOT - # recover a wedged origin, only a genuinely NEW deploy attempt does), the fix is a - # fresh workflow_dispatch, not a longer timeout. This must be a fresh dispatch and - # NOT a job rerun — rerunning would stack a second `github-pages` artifact under - # this same run and break deploy-pages' artifact lookup ("Multiple artifacts named - # github-pages"). Only tag/release deploys carry a retry-until budget (set by the - # re-dispatch job below, from the release's publish time, carried unchanged through - # each retry) — that's the same-SHA dedup wedge this targets, which a push/PR/manual - # deploy can't hit (those are always first-of-SHA). No budget → no auto-retry, just - # the red check. - - name: Retry via a fresh dispatch if the origin is wedged - if: failure() && steps.verify.outcome == 'failure' && inputs.retry-until != '' - env: - GH_TOKEN: ${{ github.token }} - REPO: ${{ github.repository }} - WF: ${{ inputs.dispatch-workflow }} - DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} - RETRY_UNTIL: ${{ inputs.retry-until }} - run: | - set -euo pipefail - if [ "$(date +%s)" -ge "$RETRY_UNTIL" ]; then - echo "::error::origin still wedged past the retry deadline ($(date -d "@$RETRY_UNTIL" -Is)) — giving up, manual re-dispatch needed" - exit 0 - fi - gh workflow run "$WF" --repo "$REPO" --ref "$DEFAULT_BRANCH" -f retry-until="$RETRY_UNTIL" - echo "::notice::origin wedged — dispatched a fresh retry (budget until $(date -d "@$RETRY_UNTIL" -Is))" - - # A tag can't deploy inline (same SHA as the default-branch push → Pages drops it - # unless the event is workflow_dispatch). Wait for the release, then re-dispatch the - # shim so the deploy runs on a workflow_dispatch event and re-serves. - re-dispatch: - if: github.event_name == 'push' && github.ref_type == 'tag' - runs-on: ubuntu-latest + # This repo deliberately differs from the "20" the tutorial ships: our own site is + # ~223 MB packed across 24 releases, a fifth of the cap, so capping would drop old + # versions for no gain. The capped path is covered by the unit tests and the + # workflow harness instead. + max-releases: "0" permissions: - contents: read # gh release view - actions: write # gh workflow run (re-dispatch the shim) - steps: - - name: Wait for the release, then re-deploy via workflow_dispatch - env: - GH_TOKEN: ${{ github.token }} - REPO: ${{ github.repository }} - TAG: ${{ github.ref_name }} - WF: ${{ inputs.dispatch-workflow }} - DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} - run: | - set -euo pipefail - # ci.yml's release job attaches the tag's docs.zip in parallel; the dispatched - # run gathers the tag from that asset, so don't dispatch until it exists. - published_at="" - for _ in $(seq 1 36); do - published_at=$(gh release view "$TAG" --repo "$REPO" --json publishedAt -q .publishedAt 2>/dev/null) && break - sleep 5 - done - [ -n "$published_at" ] \ - || { echo "::error::release $TAG not found after 3 min — did the release job fail?"; exit 1; } - # Retry budget anchored on the release's publish time, not the commit's — a tag - # can be cut long after its commit was merged (this repo's own release flow - # allows it), so the commit's own timestamp would wrongly disqualify the exact - # same-SHA dedup wedge (deploy-pages#383) this budget exists to retry through. - retry_until=$(( $(date -d "$published_at" +%s) + 900 )) - gh workflow run "$WF" --repo "$REPO" --ref "$DEFAULT_BRANCH" -f retry-until="$retry_until" + contents: read + actions: read # gather cross-run docs artifacts + pages: write + id-token: write + statuses: write # record the fork-preview approval on a head SHA diff --git a/package.json b/package.json index 8eaeecf..dc5f771 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,7 @@ "test": "node test/test-url-logic.mjs && node test/test-assemble.mjs", "docs": "cd docs && myst build --html --strict", "docs-dev": "cd docs && myst start", - "test:workflows": "uv run --quiet --with pyyaml python3 test/workflow-harness/test_gather.py" + "test:workflows": "uv run --quiet --with pyyaml python3 test/workflow-harness/test_gather.py && uv run --quiet --with pyyaml python3 test/workflow-harness/test_shape.py" }, "files": [ "plugins/version-switcher.mjs", diff --git a/test/workflow-harness/run.py b/test/workflow-harness/run.py index bf34dce..57c10c9 100644 --- a/test/workflow-harness/run.py +++ b/test/workflow-harness/run.py @@ -1,5 +1,5 @@ """ -Harness for the bash inside publish.yml's `deploy` job. +Harness for the bash inside publish-gh-pages.yml's `deploy` job. The gather steps are the workflow's most intricate logic — release selection, an asset cache, a SHA-indexed artifact lookup, parallel downloads, and four @@ -13,7 +13,7 @@ import yaml, re, subprocess, os, sys, json, shutil, pathlib H = os.path.dirname(os.path.abspath(__file__)) REPO_DIR = os.path.dirname(os.path.dirname(H)) -d = yaml.safe_load(open(f"{REPO_DIR}/.github/workflows/publish.yml")) +d = yaml.safe_load(open(f"{REPO_DIR}/.github/workflows/publish-gh-pages.yml")) steps = {s.get("name"): s for s in d["jobs"]["deploy"]["steps"] if s.get("run")} WANT = ["Select releases to publish", "Gather release artifacts", "Gather branch CI artifacts"] diff --git a/test/workflow-harness/test_gather.py b/test/workflow-harness/test_gather.py index 3ed2e34..45ddc82 100644 --- a/test/workflow-harness/test_gather.py +++ b/test/workflow-harness/test_gather.py @@ -1,4 +1,4 @@ -"""Behaviour tests for publish.yml's gather steps. See run.py for how they load.""" +"""Behaviour tests for publish-gh-pages.yml's gather steps. See run.py for how they load.""" import os, sys, tempfile diff --git a/test/workflow-harness/test_shape.py b/test/workflow-harness/test_shape.py new file mode 100644 index 0000000..357c2f1 --- /dev/null +++ b/test/workflow-harness/test_shape.py @@ -0,0 +1,95 @@ +""" +Structural invariants of the publish workflows. + +The gather bash is covered by test_gather.py. What that harness cannot reach are the +`if:` expressions, which are evaluated by GitHub, not by a shell — and those are where +the workflow_run migration is easiest to get quietly wrong. Each assertion here stands +for a specific way it has broken or could break. +""" + +import os +import sys + +import yaml + +H = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.dirname(os.path.dirname(H)) +WF = os.path.join(REPO, ".github", "workflows") + +fails = [] + + +def check(name, cond, detail=""): + print((" ok - " if cond else " FAIL - ") + name + (f"\n {detail}" if not cond and detail else "")) + if not cond: + fails.append(name) + + +def load(name): + with open(os.path.join(WF, name)) as f: + return yaml.safe_load(f) + + +engine = load("publish-gh-pages.yml") +caller = load("publish.yml") +ci = load("ci.yml") +# PyYAML parses the bare key `on` as the boolean True. +engine_on, caller_on, ci_on = engine[True], caller[True], ci[True] + +print("\n-- engine: publish-gh-pages.yml --") +check("is workflow_call only", list(engine_on) == ["workflow_call"], list(engine_on)) +check("exposes only pr + max-releases", + sorted(engine_on["workflow_call"]["inputs"]) == ["max-releases", "pr"], + sorted(engine_on["workflow_call"]["inputs"])) +check("has a single deploy job (no re-dispatch, no warn)", + list(engine["jobs"]) == ["deploy"], list(engine["jobs"])) + +steps = engine["jobs"]["deploy"]["steps"] +names = [s.get("name", s.get("uses", "")) for s in steps] +check("no in-run artifact injection remains", + not any("this run's docs artifact" in n or "Stage current build" == n for n in names), names) +check("no wedged-origin retry remains", not any("Retry via" in n for n in names), names) +check("still verifies the deployed origin", any("Verify" in n for n in names), names) + +# THE trap this migration introduces. Under workflow_run, github.ref is ALWAYS the +# default branch — even when a PR's CI triggered the run — so gating a cache SAVE on +# github.ref would have every PR-triggered deploy writing caches that nothing can read. +saves = [s for s in steps if "cache/save" in str(s.get("uses", ""))] +check("there are cache-save steps to check", len(saves) == 2, f"found {len(saves)}") +for s in saves: + cond = str(s.get("if", "")) + check(f"{s['name']!r} does not gate on github.ref", + "github.ref ==" not in cond and "format('refs/heads/" not in cond, cond) + check(f"{s['name']!r} gates on the branch that was built", + "workflow_run.head_branch" in cond, cond) + +check("deploy cancels superseded runs", + engine["jobs"]["deploy"]["concurrency"].get("cancel-in-progress") is True, + engine["jobs"]["deploy"]["concurrency"]) +check("deploy no longer needs actions:write", + engine["jobs"]["deploy"]["permissions"]["actions"] == "read", + engine["jobs"]["deploy"]["permissions"]) + +print("\n-- caller: publish.yml --") +check("triggers on workflow_run + workflow_dispatch", + sorted(caller_on) == ["workflow_dispatch", "workflow_run"], sorted(caller_on)) +check("listens for the CI workflow by name", + caller_on["workflow_run"]["workflows"] == [ci["name"]], + f'{caller_on["workflow_run"]["workflows"]} vs ci name {ci["name"]!r}') + +guard = str(caller["jobs"]["publish"]["if"]) +# workflow_run hands a WRITE token to a job triggered by fork-PR code. Losing either +# half of this guard is the pwn-request footgun the old read-only warn job existed to +# avoid, and it would fail open rather than loudly. +check("requires the triggering run to have succeeded", + "conclusion == 'success'" in guard, guard) +check("excludes forks", "head_repository.full_name == github.repository" in guard, guard) +check("still allows a manual dispatch", + "workflow_dispatch" in guard, guard) + +print("\n-- entry: ci.yml --") +check("no longer publishes", "publish" not in ci["jobs"], list(ci["jobs"])) +check("still uploads the docs artifact CI's consumers gather", "docs" in ci["jobs"], list(ci["jobs"])) + +print("\n" + ("FAILURES: " + ", ".join(fails) if fails else "all shape checks passed")) +sys.exit(1 if fails else 0) From e81a55af6d141051e5a0dab911ec908fdafea31b Mon Sep 17 00:00:00 2001 From: Tom Cobb Date: Fri, 21 Aug 2026 14:45:31 +0000 Subject: [PATCH 3/3] docs: rewrite the publish story around workflow_run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The trigger change deleted three pieces of machinery, and the docs described all of them at length: the tag re-dispatch trampoline, the publish-dispatch.yml shim, and the in-run artifact injection with its version-name threading. All of that is gone, so the explanations that justified it are replaced rather than amended. architecture.md gains "why publishing listens instead of being called" (the O(whole site) cost on every PR's critical path, and that a wedged Pages origin was never the PR author's to act on), "what listening deleted", and the trap it brings: under workflow_run github.ref is always the default branch, so anything asking "was this the default branch?" must read workflow_run.head_branch. The deferred release-layer cache section is now a description of the implemented one. choose-publish-events.md is rewritten: its whole premise was an if:-gated job in ci.yml, which no longer exists. Narrowing now means filtering in publish.yml's own if:, and the "it slows my PRs down" reason for narrowing is gone entirely. The tutorial and the reference carry the new two-file shape — ci.yml builds, publish.yml listens — and both spell out that the caller's two guards fail open, since that is the failure mode nobody notices. blueapi-issue.md (untracked) is rewritten as a migration guide rather than a bug report, since all three problems it described are now fixed upstream. Co-Authored-By: Claude Opus 5 --- .github/workflows/docs.yml | 7 +- .github/workflows/publish-gh-pages.yml | 4 +- CLAUDE.md | 238 +++++++++++------------ docs/explanations/architecture.md | 198 ++++++++++--------- docs/how-to/choose-publish-events.md | 101 +++++----- docs/how-to/keep-the-site-small.md | 15 +- docs/how-to/migrate-from-gh-pages.md | 2 +- docs/how-to/use-with-sphinx.md | 22 +-- docs/reference/workflows.md | 93 +++------ docs/tutorials/adding-to-a-fresh-repo.md | 116 +++++------ 10 files changed, 371 insertions(+), 425 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 11fad47..2dacba4 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -123,6 +123,7 @@ jobs: path: ${{ runner.temp }}/docs.zip compression-level: 0 - # NOTE: the fork-PR "preview not published" hint lives in publish.yml's `warn` - # job (reached via the publish-dispatch.yml shim), which runs for fork PRs too — - # so this build job stays purely about building + uploading the docs artifact. + # NOTE: nothing here tells a fork PR that its docs will not be published. There is + # no longer anywhere sensible to say it: publish.yml excludes forks before the + # engine is reached, so no job runs in that case at all. This build job stays + # purely about building + uploading the docs artifact. diff --git a/.github/workflows/publish-gh-pages.yml b/.github/workflows/publish-gh-pages.yml index c64d344..68ce45a 100644 --- a/.github/workflows/publish-gh-pages.yml +++ b/.github/workflows/publish-gh-pages.yml @@ -509,8 +509,8 @@ jobs: if [ "$packed" -gt "$SIZE_WARN_BYTES" ]; then echo "::warning title=Docs site approaching the Pages 1 GB limit::The \ deployed artifact is ~$(numfmt --to=iec --suffix=B "$packed"), against a hard \ - 1 GB cap. Set publish.yml's \`max-releases\` input (in your publish-dispatch \ - shim) to publish only the N most recent releases." + 1 GB cap. Set the \`max-releases\` input in your publish.yml to publish only the \ + N most recent releases." fi fi diff --git a/CLAUDE.md b/CLAUDE.md index d65c0d5..740fc21 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,15 +10,15 @@ to `publish.yml`, not a separately consumed action. ``` plugins/version-switcher.mjs # MyST directive + anywidget runtime (single file, no README — docs are in docs/) -assemble/assemble.mjs # INTERNAL: dependency-free Node kernel for publish.yml's "Generate" step (the `generate` subcommand writes switcher.json + index.html) +assemble/assemble.mjs # INTERNAL: dependency-free Node kernel for the engine's "Select releases" + "Generate" steps scripts/migrate.sh # operator gh-pages → durable-source migration (bash); two-phase: reversible cutover, then guarded --delete-gh-pages test/ # npm test suite (node, no framework) + test/workflow-harness/ (python: loads publish.yml's gather steps out of the YAML and runs them against a mock gh) docs/ # this repo's own docs (dogfoods the plugin) .github/workflows/docs.yml # PUBLIC reusable: build at the versioned BASE_URL → pack docs.zip → upload `docs` artifact (workflow_call; build-command input) -.github/workflows/publish.yml # PUBLIC reusable ENGINE (PRIVILEGED): branches on the event into 3 jobs — deploy (gather+extract inline bash + assemble.mjs@job.workflow_sha + Pages), tag-re-dispatch (re-fire the shim), fork-warn. workflow_call ONLY; reached via the publish-dispatch.yml shim -.github/workflows/publish-dispatch.yml # PUBLIC thin shim: workflow_call (ci.yml's publish, every event) + workflow_dispatch (tag re-dispatch + fork-PR opt-in + manual re-deploy) → forwards to publish.yml. The dispatchable file consumers copy (reusable workflows can't be dispatched cross-repo) +.github/workflows/publish-gh-pages.yml # PUBLIC reusable ENGINE (PRIVILEGED): ONE `deploy` job — gather+extract inline bash + assemble.mjs@job.workflow_sha + Pages + verify. workflow_call ONLY. No event branching left: the caller has already gated on success + non-fork +.github/workflows/publish.yml # PUBLIC: the file each consumer carries. workflow_run (their CI completing, matched by NAME) + workflow_dispatch (fork-PR preview via `pr`, manual re-deploy) → publish-gh-pages.yml. The one place is pinned .github/workflows/release.yml # PUBLIC reusable: attach the run's build artifacts (docs.zip; + version-switcher.mjs via _test.yml) to the tag's GitHub Release via gh (create-or-upload, immutable-safe). Consumers `uses:` it directly -.github/workflows/ci.yml # this repo's own entry: _lint / _test / docs.yml / release, then ONE publish job nesting publish-dispatch.yml (shim) → publish.yml for every event (publish.yml branches: deploy / tag-re-dispatch / fork-warn) +.github/workflows/ci.yml # this repo's own entry: _lint / _test / docs.yml / release. Does NOT publish — publish.yml picks the run up on workflow_run. Its `name: CI` is load-bearing (publish.yml matches on it) ``` ## Two halves, different lifecycles @@ -70,11 +70,11 @@ probe `_sources/.zip`, which passed even while the content came from th ### The site has a hard size ceiling — cap the releases `upload-pages-artifact` tars the WHOLE site into one artifact and Pages rejects it over **1 GB**. blueapi at 131 released `docs.zip`s was at 452 MB, +~5 MB/release: a few years -from deploys simply failing, silently, with no warning on the way. `publish.yml` takes +from deploys simply failing, silently, with no warning on the way. The engine takes **`max-releases`** (default `0` = unlimited, so upgrading never silently deletes -versions); the tutorial's shim ships `20`. Set it as a LITERAL in the shim's `with:`, -not threaded from `ci.yml` — the shim is the single caller on every path, so a literal -there can't be bypassed by a manual dispatch. A deploy past 700 MB warns — measured on +versions); the tutorial ships `20`. Set it as a LITERAL in `publish.yml`'s `with:` — +that is the single caller on every path, so a literal there can't be bypassed by a +manual dispatch. A deploy past 700 MB warns — measured on the PACKED artifact (the cap is on the gzipped tar; this repo's site is 734 MiB on disk and 223 MiB deployed, so `du` would cry wolf at a third of real usage). The tree is only packed to measure once it exceeds `SIZE_PROBE_BYTES`, since packing costs seconds; `tar @@ -108,50 +108,52 @@ gathered (a merged/closed PR, a deleted release) is correctly dropped — no `keep_files` drift. The **publish workflow** owns the Pages publish because `deploy-pages` is job-scoped. -### Split build (unprivileged) from publish (privileged), but nest publish for visibility -`ci.yml` builds + uploads the `docs` artifact for every event including fork PRs. -It then nests a single privileged `publish` job (`uses: publish-dispatch.yml` shim → -`publish.yml`) for **every** event in the canonical repo. `publish.yml` branches: an -internal PR / main push deploys inline (status **visible on the PR/commit**); a fork PR -hits a read-only **warn** job (untrusted fork-PR code never reaches a write token — the -`deploy` job is `if`-excluded for forks and the token is read-only anyway); a **tag** -hits the **re-dispatch** (a tag shares the merge commit's SHA and Pages drops a second -same-SHA deploy unless it's a `workflow_dispatch` — deploy-pages#383 — so it -re-dispatches the shim). One `publish` job, no ref/fork guards in `ci.yml`. - -Because the inline publish runs **inside the build's own CI run**, the current build -isn't a *completed* successful run the gather can discover (and for a main push the -gather would find the *previous* build). So `ci.yml` passes `publish.yml` the build's -version name (`needs.docs.outputs.version-name`); `publish.yml` downloads this run's -`docs` artifact and stages it into the gather dir as the **highest-priority** source — -overwriting any same-name zip gathered from durable sources. Everything else (other -releases, other open PRs, the rest) still comes from durable sources. - -`publish.yml`'s `deploy` job runs inline gather + extract steps, then calls -`assemble.mjs` (checked out from this repo at `job.workflow_sha`, so the kernel -matches the workflow's own ref) to generate + output the site dir, then -`deploy-pages`. Operator note: because internal PRs/tags now deploy from their -**own ref** (not only the default branch), the `github-pages` environment's -deployment-branch policy must allow those refs (or be unrestricted) — this is the -cost of nesting publish for visibility. Inline bash steps do the `gh`/`unzip`/`mv` -IO (gather + extract); the JS kernel does the pure logic (`generate` + the folded-in -`missingRequired` guard) and is unit-tested without git/network/fs. There is **no -sanitisation**: version names are clean by construction — `main`, `pr-`, or -a tag without `/` (the `tags: ['*']` trigger never builds `/`-tags). - -The extract step's zip-shape rule is deliberately **name-agnostic**: a `docs.zip` must -unzip to exactly one top-level directory, whose *contents* become `site/` -— the directory's *name* is ignored. `docs.yml` packs `html/`, but release assets are -durable and may predate this pipeline (python-copier-template's `_release.yml` roots -its zip at the tag name, and an immutable release can't be re-cut), so keying on -`html/` silently dropped every such release from the switcher. Zero entries, several -entries, or files loose at the zip root are malformed — warn and skip, never guess. +### Split build (unprivileged) from publish (privileged); publish LISTENS +`ci.yml` builds + uploads the `docs` artifact for every event including fork PRs, and +never publishes. `publish.yml` (in each consumer's repo; this repo carries an identical +one) triggers on **`workflow_run`** when their CI workflow completes — matched by its +`name:`, not filename — and calls the `publish-gh-pages.yml` engine. It also takes +`workflow_dispatch` for fork-PR previews (`pr`) and manual re-deploys. + +Publish used to be a job nested in `ci.yml`, for status visibility on the PR. That was +wrong twice over: reconstructing the site is **O(the whole site)** and independent of the +change, so it put a large constant on every PR's critical path (blueapi switched PR +previews off one day after adopting, a 654s deploy on a 40s build); and a wedged Pages +origin is not the PR author's to fix, so the red check was noise. + +**The trigger change was SUBTRACTION.** Proven live 2026-08-21 (see +[[pages-origin-wedge]]): `workflow_run` forces a Pages re-serve at an already-deployed +SHA exactly as `workflow_dispatch` does — four consecutive deploys at identical +`pages_build_version b24237484c3b445469c2db4ef161410a185fcdbc` all updated the origin. So: +- the **tag re-dispatch trampoline** is gone (a tag's deploy re-serves directly); +- **`publish-dispatch.yml`** is gone — it existed ONLY because a reusable workflow can't + be `workflow_dispatch`'d cross-repo, so the trampoline needed a local file to re-fire; +- the **`version-name` injection** is gone: `workflow_run` fires after the triggering run + COMPLETED, so its `docs` artifact is discoverable by the ordinary gather (verified — both + injection steps skipped and the site still assembled); +- the **`warn` job**, `retry-until` and the wedged-origin retry-dispatch are gone, and the + engine no longer needs `actions: write`. + +### THE workflow_run TRAP: `github.ref` is always the default branch +In a `workflow_run` run `GITHUB_SHA` and `github.ref` are **always the default branch's +HEAD**, never the built commit (a PR-triggered deploy reports `refs/heads/main`). Anything +asking "was this the default branch?" must read `github.event.workflow_run.head_branch`. +Both cache-save steps got this wrong initially — gating on `github.ref` would have every +PR-triggered deploy writing caches scoped to a PR nothing can read. + +`test/workflow-harness/test_shape.py` asserts this, plus the caller's two guards +(`conclusion == 'success'`, `head_repository == this repo`). All three **fail OPEN**, and +they live in `if:` expressions the gather harness can't reach. + +**The fork guard is not optional:** `workflow_run` runs with a WRITE token even when a +fork's PR triggered it (pwn-request). Fork builds reach the site only via a maintainer +dispatching `publish.yml` with `pr`, which pins approval to that head SHA. ### Self-referencing assemble.mjs (no separate action) `publish.yml` sparse-checks-out **this** repo's `assemble/` at `job.workflow_repository` + `job.workflow_sha` — the `job` context resolves to the file that defines the running job, i.e. the *reusable* workflow (unlike `github.workflow_*`, which resolve to the -*caller's* entry workflow). So a consumer pinning `publish.yml@vX` gets +*caller's* entry workflow). So a consumer pinning `publish-gh-pages.yml@vX` gets `assemble.mjs@vX` automatically, with no hardcoded repo and no release-time bump, and this repo's own publish job tests the working-tree script (no dogfood gap). The consumer's repo stays checked out at the root so `assemble.mjs`'s `git tag` lists @@ -189,36 +191,28 @@ stable inventory URL for cross-project `objects.inv`. `switcher.json` has no ## CI structure -One entry workflow (`ci.yml`) that nests the privileged publish: - -- `ci.yml` — **build + verify, then publish on internal events.** Triggers on - `pull_request` + push to `main`/tags (no other-branch pushes; `*` excludes - `/`-tags). Orchestrates `_lint` / `_test` / `docs` / `release`, runs for forks, - and uploads the `docs` artifact. It then nests a single `publish` job - (`uses: publish-dispatch.yml`) for **every** event in the canonical repo (the `if` - is just `repository == '…'`); grants `pages`/`id-token`/`statuses`/`actions:write` - perms to the call and passes `version-name: needs.docs.outputs.version-name`. No - ref/fork guards and no separate tag job — `publish.yml` does all the branching. -- `publish-dispatch.yml` — **the thin shim** (the dispatchable file each repo carries). - `workflow_call` (ci.yml's `publish` job, every event) + `workflow_dispatch` (the tag - re-dispatch + fork-PR opt-in + manual re-deploy); both just `uses: - publish.yml`, threading `version-name`/`pr` + its own filename as `dispatch-workflow`. - It exists only because a reusable workflow can't be dispatched cross-repo, so the - re-dispatch job needs a local file to re-fire. This repo dogfoods the exact shim - consumers add (theirs pins `publish.yml@`; this one uses the local path). -- `publish.yml` — **assemble + deploy ENGINE, privileged; owns the event branching.** - `workflow_call` ONLY (reached via the shim). Three jobs by originating event: - **deploy** (internal PR / default-branch push inline, or any dispatch → sparse-checks - out `assemble.mjs` at `job.workflow_sha` so it matches the pinned ref — see - "Self-referencing assemble.mjs" above — runs inline gather + extract steps then - `assemble.mjs generate`, then `upload-pages-artifact` + `deploy-pages` + verify, - carrying the `github-pages` environment + perms + `concurrency`), **re-dispatch** - (a tag — waits for the release, - then `gh workflow run ` in the consumer's repo with their token, so - the deploy lands as a `workflow_dispatch` that re-serves; a same-SHA tag deploy is - otherwise dropped — deploy-pages#383), and **warn** (a fork PR — read-only, posts the - opt-in hint). No canonical guard (it lives upstream in ci.yml). `version-name` injects - the in-run build on the inline path. See the architecture explanation in docs/. +Two top-level workflows: `ci.yml` builds, `publish.yml` listens for it finishing. + +- `ci.yml` — **build + verify only; never publishes.** Triggers on `pull_request` + + push to `main`/tags (no other-branch pushes; `*` excludes `/`-tags). Orchestrates + `_lint` / `_test` / `docs` / `release`, runs for forks, uploads the `docs` artifact. + Needs no elevated permissions. Its `name: CI` is **load-bearing** — `publish.yml`'s + `workflow_run` matches workflows by name, not filename, so renaming it silently stops + publishing. +- `publish.yml` — **the file each consumer carries**, and the one place `` is + pinned. `workflow_run` (CI completing) + `workflow_dispatch` (fork-PR preview via `pr`, + manual re-deploy) → `publish-gh-pages.yml`. Its job `if:` carries the two guards that + fail open (`conclusion == 'success'`, `head_repository == this repo`) and a + `max-releases` literal. This repo's copy is identical to a consumer's but for the + `uses:` path (local vs pinned) and `max-releases: "0"` vs the tutorial's `"20"` — our + site is ~223 MB packed across 24 releases, a fifth of the cap. +- `publish-gh-pages.yml` — **assemble + deploy ENGINE, privileged.** `workflow_call` + ONLY. **One** job, `deploy`: sparse-checks out `assemble.mjs` at `job.workflow_sha` so + it matches the pinned ref (see "Self-referencing assemble.mjs"), runs inline + gather + extract, then `assemble.mjs generate`, `upload-pages-artifact`, + `deploy-pages` and the origin verify — carrying the `github-pages` environment, perms, + and `concurrency: {group: pages, cancel-in-progress: true}`. No event branching: the + caller already established success + non-fork. Inputs are just `pr` and `max-releases`. Sub-workflows of `ci.yml`: - `_lint.yml` — biome @@ -243,23 +237,18 @@ Sub-workflows of `ci.yml`: artifact, verbatim) + `version-switcher.mjs` (uploaded by `_test.yml` as an artifact, so the generic workflow needs no plugin-specific step). Consumers `uses:` it directly. -**Publish flow.** The single `publish` job (→ `publish-dispatch.yml` shim → `publish.yml`) -runs for every event and `publish.yml` branches. Internal PRs and `main` deploy inline as part of the -same CI run once lint/test/docs pass (`publish.yml` injects this build + `assemble` -gathers `main`, releases, and every other open PR → deploy), a visible check on the -PR/commit. **Tags** hit the wrapper's `re-dispatch`: a release tag shares the merge -commit's SHA, and GitHub Pages silently drops a second deploy of an already-deployed -SHA *unless* the event is `workflow_dispatch` -([`actions/deploy-pages#383`](https://github.com/actions/deploy-pages/issues/383)) — -so it waits for the release, then `gh workflow run publish-dispatch.yml` -(`workflow_dispatch`), which re-gathers from durable sources (incl. the new release) -and deploys with the origin actually updating (the dispatched run carries the deploy -status, not the tag's CI run). An **external fork PR** hits the wrapper's `warn` job -(read-only, no deploy); a maintainer dispatches `publish-dispatch.yml` with the PR -number (`pr`), which sets a `preview-approved` commit status pinned to that **head -SHA** and assembles — so a later push to the PR (new SHA) drops the preview until -re-approved. `assemble` gathers an open PR's artifact via its head SHA; internal PRs -always, fork PRs only when the SHA carries that status. +**Publish flow.** `ci.yml` builds and stops. When that run *completes*, `publish.yml` +fires on `workflow_run` and calls the engine, which gathers `main`, the newest +`max-releases` releases, and every open PR's artifact from durable sources, then deploys +the whole site. Same path for every event — a PR, a `main` push and a tag all just +produce a completed CI run. A **tag** additionally has `release.yml` attach its +`docs.zip` first, within that same CI run, so by the time publish gathers, the asset is +already there (this is why the old tag/asset race is gone rather than fixed). An +**external fork PR** is excluded by the caller's guard and never deploys; a maintainer +dispatches `publish.yml` with the PR number (`pr`), which sets a `preview-approved` +commit status pinned to that **head SHA** — so a later push to the PR drops the preview +until re-approved. `assemble` gathers an open PR's artifact via its head SHA; internal +PRs always, fork PRs only when the SHA carries that status. `mystmd` is pinned at `1.10.1` (not `latest`). @@ -291,9 +280,11 @@ git push origin --tags # or: git push origin vX.Y.Z to push just this tag ``` CI runs lint + tests + docs build, then `release.yml` creates the GitHub Release with -`version-switcher.mjs` + the tag's `docs.zip` attached (via `gh`), and the tag -re-dispatch reconstructs + deploys the site including the new tag. -The plugin URL and the `uses:` refs for `docs.yml`/`publish.yml` all resolve to the +`version-switcher.mjs` + the tag's `docs.zip` attached (via `gh`). When that CI run +completes, `publish.yml` fires on `workflow_run` and reconstructs + deploys the site +including the new tag — the release asset is already attached by then, because +`release.yml` ran inside the CI run publish is waiting on. The plugin URL and the +`uses:` refs for `docs.yml`/`release.yml`/`publish-gh-pages.yml` all resolve to the same tag. ## Consuming this in another repo @@ -318,8 +309,7 @@ site: ::: ``` -Set the repo's **Pages source to "GitHub Actions"**, then add a `ci.yml` that calls -the two shared reusable workflows by full path at ``: +Set the repo's **Pages source to "GitHub Actions"**, then add a `ci.yml` that builds: ```yaml jobs: @@ -327,34 +317,34 @@ jobs: uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/docs.yml@ with: build-command: make docs # or: tox -e docs / npx … myst build / npm ci && npm run docs - publish: # every event — publish.yml branches + release: # tag-only; attaches docs.zip to the Release needs: [docs] - if: github.repository == 'ORG/REPO' - uses: ./.github/workflows/publish-dispatch.yml # your wrapper; it pins publish.yml@ - with: - version-name: ${{ needs.docs.outputs.version-name }} - permissions: { pages: write, id-token: write, contents: read, actions: write, statuses: write } + if: github.ref_type == 'tag' + uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/release.yml@ + permissions: { contents: write } +``` + +…and a **separate** `publish.yml` that listens for it: + +```yaml +on: + workflow_run: { workflows: [CI], types: [completed] } # matches ci.yml's `name:` + workflow_dispatch: { inputs: { pr: { required: false, default: "" } } } +jobs: + publish: + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.head_repository.full_name == github.repository) + uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish-gh-pages.yml@ + with: { max-releases: "20" } + permissions: { pages: write, id-token: write, contents: read, actions: read, statuses: write } ``` -Your `publish-dispatch.yml` wrapper's `workflow_dispatch` inputs must include `pr` -**and `retry-until`**, forwarding both into `publish.yml`'s `with:` — copy the shim -in [`docs/reference/workflows.md`](docs/reference/workflows.md) or the -[tutorial](docs/tutorials/adding-to-a-fresh-repo.md) verbatim rather than -hand-rolling it. `retry-until` is never set by a human; `publish.yml`'s -`re-dispatch` job always passes it when re-firing the shim after a tag release, so -a shim missing the input rejects that dispatch (breaking every tagged release's -re-publish, not just the wedged-origin retry it exists for). - -Add a `release` job that `uses:` this repo's `release.yml@` (with `permissions: -contents: write`) to attach each tag's built `docs.zip` (single `html/` root dir) as a -Release asset so `assemble` can reconstruct released versions, and the -`publish-dispatch.yml` wrapper around `publish.yml` — the single caller of the engine -and the one place you pin `publish.yml@`. The wrapper owns all -the branching: **deploy** (inline non-tag, or a dispatch), **re-dispatch** (a tag -re-fires itself so the deploy is a `workflow_dispatch` — required for a release's -same-SHA deploy to go live), and **warn** (a fork PR, read-only, posts the preview -opt-in hint; a maintainer dispatches the wrapper with `pr` to actually preview). See the -full how-to + snippets in `docs/`. +`ci.yml` has **no** publish job. Copy `publish.yml` verbatim from the +[tutorial](docs/tutorials/adding-to-a-fresh-repo.md) rather than hand-rolling it — both +`if:` guards fail OPEN, and `workflows: [CI]` matches the entry workflow by NAME, so +renaming it silently stops publishing. ## Upstreaming diff --git a/docs/explanations/architecture.md b/docs/explanations/architecture.md index c29a25d..d36797c 100644 --- a/docs/explanations/architecture.md +++ b/docs/explanations/architecture.md @@ -12,7 +12,7 @@ publishes it **directly to GitHub Pages** via `actions/upload-pages-artifact` + `actions/deploy-pages`. There is **no `gh-pages` branch** — `deploy-pages` publishes one artifact as the *entire* site, which is a whole-site-replace. The four source kinds and their durability are in the -[reference](../reference/workflows.md#what-publish-yml-gathers). +[reference](../reference/workflows.md#what-the-engine-gathers). Releases are permanent, so old versions never vanish. PR previews come from CI artifacts and silently drop if the artifact expires and nothing rebuilds — fine for @@ -42,7 +42,7 @@ with nothing on the way to say so. Two things follow. -`publish.yml` takes a **`max-releases`** input: publish only the N most recent releases, +The engine takes a **`max-releases`** input: publish only the N most recent releases, ranked by the tagged commit's date. It defaults to `0` (unlimited), so upgrading never silently deletes versions from an existing site — but the paved path in the [tutorial](../tutorials/adding-to-a-fresh-repo.md) sets it, and a deploy whose *packed* @@ -121,84 +121,101 @@ The architecture makes that boundary structural by splitting build from publish: - **CI (unprivileged)** runs `myst build` and uploads the `docs` artifact for *every* event, forks included. It never holds a write token. -- **`publish.yml` (privileged)** runs `assemble` + the Pages deploy. It runs only in +- **`publish-gh-pages.yml` (privileged)** runs `assemble` + the Pages deploy. It runs only in the trusted upstream context. So a fork's build can never reach a write token; only trusted code deploys. -### Why publish is *nested* in CI (for internal events) - -The deploy is surfaced as a **job inside the CI run** (`ci.yml`'s `publish` job → -`publish-dispatch.yml` shim → `publish.yml` via `workflow_call`) so its status and URL are -visible on the PR / commit — rather than running invisibly after the fact. A single -`publish` job runs for **every** event; `publish.yml` does the branching. - -`publish.yml` is the `workflow_call` **engine** and owns all the publish branching, -reached only through a thin `publish-dispatch.yml` shim that exposes it as both -`workflow_call` (the inline `publish` job) and `workflow_dispatch` (the tag -re-dispatch, the fork-PR opt-in, manual re-deploys). It routes each event to one of three -jobs: **deploy** (internal PR / default-branch push, or a dispatch), **re-dispatch** (a tag -— below), or **warn** (a fork PR — read-only, never deploys, just posts the manual-opt-in -hint; this replaced the old warning step in the build job). A fork PR can't reach a write -token (its `GITHUB_TOKEN` is read-only and `deploy` is `if`-excluded for forks), so the -security boundary holds while the hint lives next to the rest of the publish logic. The -shim exists only because a reusable workflow can't be `workflow_dispatch`'d cross-repo, so -the re-dispatch job needs a local file to re-fire — it `gh workflow run`s the shim by name -(`dispatch-workflow`), which works because a called workflow runs with the *caller's* repo -and token. This repo dogfoods the exact shim a consumer adds (theirs just pins -`publish.yml@`). - -The cost of nesting is an environment-policy change: because internal PRs now deploy -from **their own ref**, the `github-pages` environment's deployment-branch policy must -allow those refs. The alternative — triggering publish via `workflow_run` after CI -completes — keeps deploys on the default branch only, but at the price of the deploy -being invisible on the PR. This project chose visibility. (Tags are the exception — -they deploy from the default-branch ref via the re-dispatch below, not their own ref.) - -### Why tags deploy via a `workflow_dispatch` re-dispatch - -A release tag is cut on the merge commit, so it shares the default branch's -just-deployed SHA. `deploy-pages` stamps every deployment with -`pages_build_version = GITHUB_SHA` — it has no input to change it, and the value is -server-validated against the deploy OIDC token's commit claim, so a unique value -can't be forced (it 404s; see [`actions/deploy-pages#383`](https://github.com/actions/deploy-pages/issues/383)). -The Pages backend then silently **drops a second deploy of an already-deployed SHA** -on a non-`workflow_dispatch` event — it reports success and flips the deployment -record active, but the origin keeps serving the *first* artifact. So an inline tag -deploy would "succeed" while the site stayed on the pre-tag build. - -The one documented escape hatch: a `workflow_dispatch` deploy of that same SHA -*forces* a re-serve. So tags don't deploy inline — `publish.yml`'s `re-dispatch` job -waits for `ci.yml`'s parallel `release` job to attach the new `docs.zip`, then -`gh workflow run`s the shim as a `workflow_dispatch`; that run re-gathers from durable -sources (now including the new release) and deploys, and because its event is -`workflow_dispatch` the origin updates. `main` and internal-PR deploys are the *first* of -their SHA, so they serve fine inline and keep their PR/commit visibility. This all lives -in the shared engine — the consumer's only dispatch-related file is the thin shim (a -reusable workflow can't be dispatched cross-repo, so the re-dispatch job needs a local file -to re-fire). -The post-deploy origin-verify step in `publish.yml` backstops any residual stale -origin by failing the run instead of serving stale docs silently. - -### Why the current build is *injected* - -Because the inline publish runs *inside* the build's own CI run, that run isn't a -*completed* successful run yet — so `assemble`'s normal gather can't discover it. For -a `main` push it would be worse than missing: the gather would find the **previous** -successful run and publish a build behind by one commit. So `ci.yml` passes the -build's version name to `publish.yml`, which downloads this run's `docs` artifact and -stages it into the gather dir as the **highest-priority** source — overwriting any -same-name zip gathered from durable sources. Everything else still comes from durable -sources. The dispatch paths (the tag re-dispatch and the fork opt-in) pass no current -build — they gather entirely from durable sources, the tag from its just-attached -`docs.zip` Release asset and a fork from its approved head SHA's successful run. +(why-publishing-listens-instead-of-being-called)= +### Why publishing listens instead of being called + +Publishing is **not** a job in `ci.yml`. It is a separate `publish.yml` in the consumer's +repo, triggered by `workflow_run` when their CI workflow completes, which then calls the +`publish-gh-pages.yml` engine. + +It used to be nested, deliberately, so the deploy's status and URL showed on the PR. Two +things made that wrong. + +The first is cost. Reconstructing the site is **O(the whole site)** and completely +independent of what changed — the same 24 releases, the same open PRs, the same upload, +whichever PR triggered it. Nesting that put a large constant inside every PR's critical +path. The first consumer to adopt this switched PR previews off one day later, because a +650-second deploy had been bolted onto a 40-second docs build. + +The second is that the visibility was never the PR author's to act on. A red check for a +wedged Pages origin, on a dependency-bump PR, is noise that trains people to ignore CI. + +`workflow_run` fixes both: the deploy runs afterwards on its own run, and a failure is +visible where the people who can fix it are looking. + +### What listening deleted + +The trigger change was mostly **subtraction**, because three separate pieces of machinery +existed only to work around the old shape. + +**The tag trampoline.** A release tag is cut on the merge commit, so it shares the default +branch's just-deployed SHA, and `deploy-pages` stamps every deployment with +`pages_build_version = GITHUB_SHA` — no input to change it, and the value is +server-validated against the OIDC commit claim, so a unique one can't be forced (it 404s; +see [`actions/deploy-pages#383`](https://github.com/actions/deploy-pages/issues/383)). +Pages silently drops a *second* deploy of an already-deployed SHA on some events: it +reports success and flips the deployment record active, but the origin keeps serving the +first artifact. A tag deploy would "succeed" while the site stayed on the pre-tag build. + +The old fix was a trampoline: tags re-fired a locally dispatchable shim so the deploy +landed as a `workflow_dispatch`, which does force a re-serve. + +`workflow_run` forces a re-serve too. That was established by experiment rather than +inference, because the documented rule does not predict it — on 2026-08-21 four +consecutive deploys at the *identical* build version +`b24237484c3b445469c2db4ef161410a185fcdbc` (a push to `main`, a tag cut on that same +commit, and two pushes to one PR) each updated the live origin. So a tag's deploy re-serves +directly, and the trampoline is gone. + +**The shim.** `publish-dispatch.yml` existed *only* because a reusable workflow cannot be +`workflow_dispatch`'d cross-repo, so the trampoline needed a local file to re-fire. No +trampoline, no shim. What consumers carry now is one `publish.yml` that calls the engine. + +**The in-run artifact injection.** A nested publish runs *inside* the build's own run, so +that run is not yet a completed success and the gather cannot discover it — worse, on a +`main` push the gather would find the *previous* run and publish a build one commit behind. +So the build's version name was threaded through `ci.yml` and the shim, and its artifact +staged as the highest-priority source. `workflow_run` fires *after* the triggering run +completes, so the ordinary gather finds it. Nothing to inject, nothing to thread. + +The read-only `warn` job went the same way: a fork's CI never reaches the engine, because +the caller excludes it. + +### The trap `workflow_run` brings with it + +In a `workflow_run` run, `GITHUB_SHA` and `github.ref` are **always the default branch's +HEAD** — never the commit that was built. A PR-triggered deploy reports `refs/heads/main`. + +That is mostly harmless (it is why the same-SHA question mattered at all), but it silently +breaks anything that asks "was this the default branch?". Both of the engine's cache-save +steps did exactly that, and gating on `github.ref` would have had every PR-triggered deploy +writing caches scoped to a PR that nothing else can read. They test +`github.event.workflow_run.head_branch` instead. + +Because that lives in an `if:` expression rather than a shell script, the gather harness +cannot reach it; `test/workflow-harness/test_shape.py` asserts it structurally, along with +the caller's two guards — all of which fail *open*, and so would never announce themselves. + +### Fork PRs still cannot deploy themselves + +`workflow_run` runs with a **write token even when a fork's pull request triggered it** — +the classic pwn-request shape, and a real hazard rather than a theoretical one. The caller +therefore requires `head_repository.full_name == github.repository`, so a fork's build +never reaches the engine automatically. A maintainer publishes a preview by dispatching +`publish.yml` with the PR number, which records approval against that exact head SHA; a +later push to the PR drops the preview until re-approved. ## The inline-bash / JS split inside `assemble` -The `assemble` logic is split between `publish.yml` inline steps and `assemble.mjs` -(sparse-checked-out at `job.workflow_sha`): +The `assemble` logic is split between `publish-gh-pages.yml` inline steps and +`assemble.mjs` (sparse-checked-out at `job.workflow_sha`): -- **`publish.yml`'s gather and extract steps** do the IO plumbing — `gh` downloads, +- **the engine's gather and extract steps** do the IO plumbing — `gh` downloads, `unzip`, `mv`, the `stable/` symlink — as inline bash. The steps are individually named so each one's timing and failures are visible in the GH Actions UI. - **`assemble.mjs`** is the pure-ish kernel: ordering, prerelease detection, @@ -210,8 +227,8 @@ Pure bash is ruled out — semver ordering, prerelease detection and JSON render are not unit-testable in bash. Bash never parses JSON itself: every extraction uses `gh`'s built-in `-q`/`--jq` (it embeds real jq), never a piped standalone `jq` — a `gh … | jq` pipe would mask an API failure as empty output. Gather order encodes -priority — releases first, then branch CI overwrites them, then the current build -overwrites those; all version-ordering and prerelease logic lives in `generate`. +priority — releases first, then branch CI overwrites them; all version-ordering and +prerelease logic lives in `generate`. ## Fork-PR previews: per-commit maintainer opt-in @@ -284,30 +301,37 @@ The `stable` segment name is a fixed convention, hardcoded in the widget. following a digit, PEP 440 style — parity with the release workflow; a tag that merely contains those letters, like `release-1.0`, is not a prerelease), but still listed in the switcher if gathered. -- **Concurrency:** `concurrency: { group: pages, cancel-in-progress: false }` makes - deploys last-writer-wins; reconstructing from durable sources keeps that mostly - self-healing. +- **Concurrency:** `concurrency: { group: pages, cancel-in-progress: true }`. Cancelling + a superseded deploy is safe *because* every deploy reconstructs the whole site — the + one that replaces it gathers everything the cancelled one would have. This was unsafe + while publish was nested in CI, where cancelling marked an unrelated PR's run cancelled. + +## The release-layer cache + +Re-downloading and unzipping every release's `docs.zip` on every deploy was the one +recurring cost that scaled with the number of releases, and it stopped being theoretical +once a consumer reached 131 of them: 114 seconds per deploy, on every event, fetching the +same immutable bytes. -## Deferred: a release-layer cache +The engine now caches them (`actions/cache`, keyed on the exact set of asset ids it +intends to publish, files named by **asset id** so a re-cut release cannot be served from +a stale entry, and pruned to the published set so a capped site keeps a capped cache). +Steady state is a total hit; cutting a release downloads exactly one zip. -Re-downloading and unzipping every release's `docs.zip` on every deploy is the one -recurring cost that scales with the number of releases. A GitHub Actions cache of the -immutable released-tags layer could skip those re-downloads, but it is **deliberately -not built yet** — the dominant cost (N sequential `gh` round-trips) is already -addressed by one paginated releases call, and the benefit is zero at adoption. The -full analysis and an implementation sketch are tracked as future work in -[issue #6](https://github.com/DiamondLightSource/myst-version-switcher-plugin/issues/6). +Caches are written only when the **default branch** was what got built — one saved while +a PR is the trigger is scoped to that PR and unreadable by anything else, so saving there +would only churn the 10 GB repo quota. ## Key resolved decisions -- **No action wrapper — `publish.yml` runs `assemble/` directly** (self-checked-out +- **No action wrapper — the engine runs `assemble/` directly** (self-checked-out at `job.workflow_sha`, so the scripts match the workflow's own ref). The build half (`docs.yml`) computes the clean token inline and uploads the `docs` artifact. - **Direct Pages publish, no `gh-pages` branch** (`upload-pages-artifact` + `deploy-pages`), requiring the repo's Pages source set to "GitHub Actions". - **JS core + inline-bash glue.** Pure functions (and their node tests) live in `assemble.mjs`; the `gh`/`unzip`/`mv` IO lives as inline bash steps in - `publish.yml` — individually named so step timing and failures are visible in + `publish-gh-pages.yml` — individually named so step timing and failures are visible in the GH Actions UI. Python was a contender (the team is Python-heavy) but loses on a second toolchain in a JS-only repo. - **`release.yml` attaches `docs.zip`** (it downloads the run's artifacts and diff --git a/docs/how-to/choose-publish-events.md b/docs/how-to/choose-publish-events.md index 780217d..21bffcf 100644 --- a/docs/how-to/choose-publish-events.md +++ b/docs/how-to/choose-publish-events.md @@ -1,72 +1,75 @@ # How-to: choose which events publish the site -The [tutorial](../tutorials/adding-to-a-fresh-repo.md)'s `publish` job runs on **every** -event and lets `publish.yml` do the branching: +`publish.yml` fires on **every** completed CI run — pull requests, pushes to the default +branch, and tags alike: ```yaml - publish: - needs: [docs] - if: github.repository == 'ORG/REPO' - uses: ./.github/workflows/publish-dispatch.yml +on: + workflow_run: + workflows: [CI] + types: [completed] ``` -Every deploy rebuilds the *whole* site from durable sources, so this is also what makes -the site **self-healing**: any deploy that came out wrong is corrected by the next one, -whatever triggered it. Narrowing the gate trades that away, so narrow it deliberately. +Every deploy rebuilds the *whole* site from durable sources, so this is what makes the +site **self-healing**: any deploy that came out wrong is corrected by the next one, +whatever triggered it. Narrowing the trigger trades that away, so narrow it deliberately. + +## Check the cost first -## If per-PR deploys are too noisy +Publishing is no longer on any PR's critical path — it runs after CI, on its own run — so +"it slows my PRs down" is not a reason to narrow it any more. Deploys do still serialise +on the `pages` concurrency group, but each one supersedes the last (`cancel-in-progress: +true`), which is safe precisely because every deploy reconstructs everything. -Deploys serialise on the `pages` concurrency group, so a repo with a lot of PR churn may -not want one per push. +If a deploy takes minutes, the site itself is probably too big rather than the trigger too +broad — see [keep-the-site-small](keep-the-site-small.md). -Check the cost before narrowing, though. A deploy that reconstructs a large site used to -take many minutes, and most of that is gone: release assets are cached rather than -re-downloaded, the artifacts API is paginated once instead of once per open PR, and -downloads run in parallel. If it still takes minutes, the site itself is probably too -big — see [keep-the-site-small](keep-the-site-small.md) before trading away -self-healing. +## If you still want fewer deploys -To narrow it anyway, publish on **tags and the default branch**: +Narrow the job's `if:` in your `publish.yml`. To publish only on the default branch and +tags, skip the runs whose triggering event was a pull request: ```yaml +jobs: publish: - needs: [docs] if: >- - github.repository == 'ORG/REPO' && - (github.ref_type == 'tag' || github.ref == 'refs/heads/main') + github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.head_repository.full_name == github.repository && + github.event.workflow_run.event != 'pull_request') ``` -You keep PR previews — **any** deploy gathers every open PR's build artifact, so -`/pr-/` still appears; it just refreshes on the next merge to the default branch -rather than on every push to the PR. And you keep self-healing, at merge cadence. +You keep PR previews either way — **any** deploy gathers every open PR's build artifact, +so `/pr-/` still appears; it just refreshes on the next merge rather than on every +push. And you keep self-healing, at merge cadence. -:::{warning} -**Do not narrow it to tags alone.** With - -```yaml - if: github.repository == 'ORG/REPO' && github.ref_type == 'tag' # ← don't -``` - -the *only* deploys are release deploys, so nothing corrects a bad one until the next -release. A single missed source then persists for weeks: a real repo that did this -shipped a site that was permanently one release behind, because each release's deploy -was the only chance that release had to reach the site. +:::{danger} +**Never narrow it to tags alone.** With only release deploys, nothing corrects a bad one +until the next release, and a single missed source persists for weeks. A real repo did +exactly this and shipped a site that was permanently one release behind, because each +release's deploy was the only chance that release ever had to reach the site. ::: -## What you cannot narrow away +## What you must not remove + +The two guards in the shipped `if:` are not style: -The `publish` job must still be reached on a **tag** push, even if you only want release -deploys. That is the event that triggers `publish.yml`'s `re-dispatch` job — the -`workflow_dispatch` bounce that is the only way a release's same-SHA deploy reaches the -Pages origin at all (see [the architecture explanation](../explanations/architecture.md)). +- **`conclusion == 'success'`** — `workflow_run` fires on failed runs too, and a failed + run has no usable `docs` artifact. +- **`head_repository.full_name == github.repository`** — `workflow_run` runs with a + **write token** even when a fork's pull request triggered it. Dropping this hands + untrusted code a privileged path to your live site. Fork builds reach the site only + when a maintainer dispatches `publish.yml` with a `pr` number, which records approval + against that exact head SHA. -Fork PRs are already handled without a gate: they reach the read-only `warn` job, never a -deploy. +Both fail *open* if removed — the site keeps deploying and nothing looks wrong — so +`test/workflow-harness/test_shape.py` asserts them in this repo. Consider the same if you +edit your copy. -## Checking what your gate actually does +## Checking what your trigger actually does -On any run, `publish` showing as `skipped` in the job list means no deploy happened for -that event. When one does run, its `Gather release artifacts` step prints a decision for -every release the API listed — gathered, skipped and why — and the extract step reports -the assembled size and version count, so you can see whether a version made it in and -what kept it out. +`publish.yml` runs appear under Actions as their own runs, not as checks on the PR. A run +that skipped shows the `publish` job as skipped. When one does deploy, its +`Gather release artifacts` step prints a decision for every release the API listed — +gathered, skipped, and why — and the extract step reports the assembled size and version +count, so you can see whether a version made it in and what kept it out. diff --git a/docs/how-to/keep-the-site-small.md b/docs/how-to/keep-the-site-small.md index 82dd847..d9c01e1 100644 --- a/docs/how-to/keep-the-site-small.md +++ b/docs/how-to/keep-the-site-small.md @@ -36,26 +36,21 @@ gh api --paginate repos/ORG/REPO/releases \ ## Cap the number of published releases -Set `max-releases` in your `publish-dispatch.yml` shim, in the `with:` block: +Set `max-releases` in your `publish.yml`, in the `with:` block: ```yaml jobs: publish: - uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish.yml@ + uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish-gh-pages.yml@ with: - version-name: ${{ inputs.version-name }} max-releases: "20" # ← publish only the 20 most recent releases pr: ${{ inputs.pr }} - dispatch-workflow: publish-dispatch.yml - retry-until: ${{ inputs.retry-until }} ``` :::{important} -Set it in the **shim**, not in `ci.yml`. The shim is the single caller of the engine on -every path — inline publish, tag re-dispatch, fork preview and manual re-deploy alike — -so a literal there is the one place the policy cannot be bypassed. Threading it as an -input from `ci.yml` would leave manual dispatches publishing a different number of -versions from ordinary CI. +A **literal**, not an input threaded in from elsewhere. `publish.yml` is the single caller +of the engine on every path — the `workflow_run` deploy, a fork preview, a manual +re-deploy — so a literal here is the one place the policy cannot be bypassed. ::: The default is `0`, meaning unlimited, so upgrading the pinned tag never removes versions diff --git a/docs/how-to/migrate-from-gh-pages.md b/docs/how-to/migrate-from-gh-pages.md index 274c621..be7e195 100644 --- a/docs/how-to/migrate-from-gh-pages.md +++ b/docs/how-to/migrate-from-gh-pages.md @@ -92,7 +92,7 @@ stages `//` from the seed), and you can even rehearse the destructive f (`migrate.sh FORKORG/REPO --delete-gh-pages`) without risk — it only touches the fork. The `` pins still resolve to this project's upstream releases, so the reusable workflows behave identically. The only path it doesn't auto-cover is a release: push a -tag to the fork if you also want to rehearse the tag re-dispatch. +tag to the fork if you also want to rehearse a release. ::: ## Before you start diff --git a/docs/how-to/use-with-sphinx.md b/docs/how-to/use-with-sphinx.md index d0d2b01..1e29c70 100644 --- a/docs/how-to/use-with-sphinx.md +++ b/docs/how-to/use-with-sphinx.md @@ -62,25 +62,15 @@ jobs: uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/release.yml@__LATEST_TAG__ permissions: contents: write - - publish: - needs: [docs] - if: github.repository == 'ORG/REPO' - uses: ./.github/workflows/publish-dispatch.yml - with: - version-name: ${{ needs.docs.outputs.version-name }} - permissions: - contents: read - actions: write - pages: write - id-token: write - statuses: write ``` +`ci.yml` has no publish job: `publish.yml` picks the run up on `workflow_run` once CI +completes. + `uv` is preinstalled by `docs.yml` (honouring a committed `.python-version`), so -`build-command` can equally be `uv run sphinx-build …`. Add the -`publish-dispatch.yml` shim and set the Pages source / environment policy exactly as -the [tutorial](../tutorials/adding-to-a-fresh-repo.md) describes — none of that is +`build-command` can equally be `uv run sphinx-build …`. Add `publish.yml` and set the +Pages source / environment policy exactly as the +[tutorial](../tutorials/adding-to-a-fresh-repo.md) describes — none of that is MyST-specific. ## 3. Intersphinx via the `stable/` alias diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index b2becc1..323bada 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -1,31 +1,43 @@ # Reference: the reusable workflows + contracts -The public interface is two reusable workflows consumed by `uses:` at a `` — -`docs.yml` directly from `ci.yml`, and `publish.yml` via a local `publish-dispatch.yml` -shim (one per repo — see the [tutorial](../tutorials/adding-to-a-fresh-repo.md)): +The public interface is two reusable workflows consumed by `uses:` at a ``, from two +different files in your repo: ```yaml +# ci.yml — builds, never publishes jobs: docs: uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/docs.yml@ with: build-command: make docs +``` + +```yaml +# publish.yml — a separate workflow, triggered by ci.yml FINISHING +on: + workflow_run: { workflows: [CI], types: [completed] } + workflow_dispatch: { inputs: { pr: { required: false, default: "" } } } +jobs: publish: - needs: [docs] - if: github.repository == 'ORG/REPO' - uses: ./.github/workflows/publish-dispatch.yml + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.head_repository.full_name == github.repository) + uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish-gh-pages.yml@ with: - version-name: ${{ needs.docs.outputs.version-name }} + max-releases: "20" permissions: pages: write id-token: write contents: read - actions: write + actions: read statuses: write ``` -The site-reconstruction logic (`assemble/`) is **internal** — `publish.yml` runs it -directly; it is not a separately consumed action. +The full copy-paste versions are in the +[tutorial](../tutorials/adding-to-a-fresh-repo.md). The site-reconstruction logic +(`assemble/`) is **internal** — `publish-gh-pages.yml` runs it directly; it is not a +separately consumed action. ## `docs.yml` — build (unprivileged) @@ -36,70 +48,15 @@ preinstalled Node, so `build-command` can be `make` / `npx` / `tox` / `npm` driv | input | required | default | meaning | |---|---|---|---| -| `build-command` | **yes** | — | Command that builds the HTML site into `html-dir` at `$BASE_URL`. Fold any project setup (`cp CONFIG`, `npm ci`, apt deps) behind it. | -| `html-dir` | no | `docs/_build/html` | Directory the build writes the site to; staged into `docs.zip` as its single root directory, `html/`. | - -| output | meaning | -|---|---| -| `version-name` | The version this build was served at (`pr-` \| default-branch \| ``) — pass to `publish.yml`. | - -`build-command` runs with two env vars set: **`BASE_URL`** -(`/REPO/` — the sub-path the build is served at) and -**`VERSION_NAME`** (the bare version token — for builds that need it directly, -e.g. a Sphinx `conf.py` setting the pydata theme's switcher `version_match`; see -[how-to: use with Sphinx](../how-to/use-with-sphinx.md)). - -## `publish.yml` — the engine, owns the branching (privileged) - -Reconstructs the whole site and deploys it to Pages, routing each event to one of three -jobs. The `deploy` job carries the `github-pages` environment, `concurrency: pages`, and -`pages`/`id-token`/`statuses` write. Reached only via the `publish-dispatch.yml` shim -(below); the canonical-repo guard lives upstream (in `ci.yml`), so the engine stays -generic. - -- **`deploy`** — internal PR / default-branch push (inline), or any `workflow_dispatch`. - Self-checks-out `assemble.mjs` at `job.workflow_sha` / `job.workflow_repository` - (the `job` context resolves to the reusable file, not the caller), so the kernel - matches the `` the consumer pinned — no version bump; the consumer's repo stays - checked out at the root so `assemble.mjs`'s `git tag` lists *their* versions. The - inline path injects the in-run build via `version-name`; the dispatch paths pure-gather. -- **`re-dispatch`** — a tag push. Waits for the release, then re-fires the shim - (`dispatch-workflow`, a file in the consumer's repo — a `workflow_call` run executes with - the caller's token, so it can dispatch it) so the deploy lands as a `workflow_dispatch` - and re-serves. -- **`warn`** — a fork PR. Read-only, never deploys; just posts the manual-opt-in hint. - -**Tags must deploy via `workflow_dispatch`**: GitHub Pages silently drops a second deploy -of an already-deployed SHA (a release tag shares the merge commit's SHA) unless the event -is `workflow_dispatch` -([`actions/deploy-pages#383`](https://github.com/actions/deploy-pages/issues/383); see the -[architecture explanation](../explanations/architecture.md)). - -## `publish-dispatch.yml` — the thin shim (one per repo) - -The only thing that calls `publish.yml`, the single place you pin `publish.yml@`, and -the **dispatchable file** the re-dispatch job re-fires. It just forwards to the engine via -`workflow_call` (ci.yml's `publish`, every event) and `workflow_dispatch` (the tag -re-dispatch, a fork-PR preview, a manual re-deploy). A reusable workflow can't be -`workflow_dispatch`'d cross-repo, which is why this file lives in each repo. - -### `publish.yml` inputs (threaded through by the shim) - -| input | required | default | meaning | -|---|---|---|---| -| `version-name` | no | `""` | Version name of **this run's** `docs` artifact to stage directly, instead of gathering it from durable sources. Set by `ci.yml`'s inline publish (the run isn't a completed success yet, so the gather can't discover it — or would find a stale previous build). Empty → pure durable gather (the dispatch paths). | -| `pr` | no | `""` | Fork PR number to approve (pins its head SHA as `preview-approved`) and preview. Set on the shim's `workflow_dispatch` path. | -| `max-releases` | no | `"0"` | Publish only the N most recent releases, ranked by the tagged commit's date (`created_at`); `0` = all of them. The site deploys as **one** Pages artifact against a hard **1 GB** cap, which a long-lived project will eventually hit. Set it as a **literal in the shim's `with:` block**, not threaded from `ci.yml`, so it applies on every path. See [keep-the-site-small](../how-to/keep-the-site-small.md). | -| `dispatch-workflow` | no | `publish-dispatch.yml` | Filename of the shim in the consumer's repo, re-fired by the `re-dispatch` job. Override only if you renamed the shim. | -| `retry-until` | no | `""` | Internal — epoch-seconds deadline for auto-retrying a wedged Pages origin. Set only by the `re-dispatch` job (from the release's publish time) when it re-fires the shim; carried unchanged through any retries. **The shim must declare + forward this input** (`workflow_dispatch` + the `with:` block below) even though consumers never set it themselves — `re-dispatch` always passes it, and a shim missing the input rejects the dispatch. Don't set manually. | +| `pr` | no | `""` | Fork PR number to approve (pins its head SHA as `preview-approved`) and preview. Set on the caller's `workflow_dispatch` path. | +| `max-releases` | no | `"0"` | Publish only the N most recent releases, ranked by the tagged commit's date (`created_at`); `0` = all of them. The site deploys as **one** Pages artifact against a hard **1 GB** cap, which a long-lived project will eventually hit. Set it as a **literal** in `publish.yml`'s `with:` block so it applies on every path. See [keep-the-site-small](../how-to/keep-the-site-small.md). | -## What `publish.yml` gathers +## What the engine gathers Every deploy rebuilds the complete tree from authoritative inputs: | version kind | source | durability | |---|---|---| -| current build | this run's `docs` artifact, staged via `version-name` (highest priority — overwrites any same-name gathered source) | n/a (just built) | | default branch (e.g. `main`) | newest `docs` artifact built from that branch → the **Actions cache** copy (hard-fail if neither exists) | cached — re-saved on each default-branch deploy; evicted after 7 days unread | | released tags | the `docs.zip` asset attached to each **GitHub Release**, newest `max-releases` of them (the migration seed release, if present, seeds the default-branch zip and is never capped) | permanent | | open PRs (`pr-`) | each PR's `docs` artifact, keyed by current head SHA — internal always, fork PRs only when the SHA carries a `preview-approved` status | ephemeral — drops when the PR merges/closes | diff --git a/docs/tutorials/adding-to-a-fresh-repo.md b/docs/tutorials/adding-to-a-fresh-repo.md index 0fb6b72..f25d2a8 100644 --- a/docs/tutorials/adding-to-a-fresh-repo.md +++ b/docs/tutorials/adding-to-a-fresh-repo.md @@ -61,14 +61,16 @@ It is recommended that you use the structure below. ### `ci.yml` -This is the entry point, it defines three jobs: +This is the entry point, it defines two jobs: - **`docs`** — `docs.yml` builds your site at the versioned `BASE_URL` and uploads the `docs` artifact, for every event (fork PRs included). - **`release`** — a small tag-only job that attaches the built `docs.zip` to the GitHub Release. -- **`publish`** — calls your `publish-dispatch.yml` wrapper which publishes the complete - built site (with all versions) to GitHub Pages. + +It does *not* publish. `publish.yml` (below) picks this run up once it completes, which +keeps the site's reconstruction cost off your PRs and stops a Pages failure showing as a +red check against an author who cannot act on it. ```yaml # .github/workflows/ci.yml @@ -93,78 +95,61 @@ jobs: uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/release.yml@__LATEST_TAG__ permissions: contents: write # create the Release + attach assets - - publish: - needs: [docs] - # don't publish a pages site in the fork's org - if: github.repository == 'ORG/REPO' - uses: ./.github/workflows/publish-dispatch.yml # your workflow (below) - with: - version-name: ${{ needs.docs.outputs.version-name }} - permissions: - contents: read - actions: write - pages: write - id-token: write - statuses: write ``` -### `publish-dispatch.yml` +`ci.yml` has no publish job at all — publishing is a separate workflow that picks this +run up once it finishes. + +### `publish.yml` -`publish.yml` is the engine and owns all the branching; this is a thin **shim** — the -only thing that calls it, and the one place you pin `@__LATEST_TAG__`. It has to exist as a file -in your repo (not just be `uses:`'d) because the tag re-dispatch re-fires it as a -`workflow_dispatch`, and a reusable workflow can't be dispatched cross-repo. Copy it -verbatim; the only thing to keep current is the `publish.yml` pin, already set to -`__LATEST_TAG__`: +The one file you carry, and the one place you pin `@__LATEST_TAG__`. It calls the engine +(`publish-gh-pages.yml`) when CI completes. Copy it verbatim: ```yaml -# .github/workflows/publish-dispatch.yml -name: Publish (dispatch) +# .github/workflows/publish.yml +name: Publish on: - workflow_call: # ci.yml's `publish` job, for every event - inputs: - version-name: { required: false, default: "", type: string } - # tag re-dispatch + fork-PR preview + manual re-deploy + # Matches your CI workflow by its `name:`, NOT its filename. Rename that and you must + # change it here too, or publishing silently stops. + workflow_run: + workflows: [CI] + types: [completed] + # Fork-PR previews and manual re-deploys. workflow_dispatch: inputs: pr: - description: "Fork PR to approve + preview (empty = re-deploy)" - required: false - default: "" - retry-until: - description: "Internal — epoch-seconds retry deadline. Don't set manually." + description: "Fork PR to approve + preview (empty = plain re-deploy)" required: false default: "" + jobs: publish: - uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish.yml@__LATEST_TAG__ + # Both guards matter. `conclusion` because workflow_run fires on failed runs too, and + # a failed run has no usable docs artifact. `head_repository` because workflow_run + # gets a WRITE token even when a fork's PR triggered it — without this, untrusted + # code reaches your live site. Both fail OPEN if removed. + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.head_repository.full_name == github.repository) + uses: DiamondLightSource/myst-version-switcher-plugin/.github/workflows/publish-gh-pages.yml@__LATEST_TAG__ with: - # "" on dispatch → pure durable gather - version-name: ${{ inputs.version-name }} - # Publish only the N most recent releases. A LITERAL, not an input threaded from - # ci.yml, so it applies on every path — the site deploys as one Pages artifact - # against a hard 1 GB cap. 0 = unlimited. See docs/how-to/keep-the-site-small.md. - max-releases: "20" - # set (dispatch) → pin that fork head SHA pr: ${{ inputs.pr }} - # the file the tag re-dispatch re-fires - dispatch-workflow: publish-dispatch.yml - # forwarded so the re-dispatch job's auto-retry can re-fire this shim - retry-until: ${{ inputs.retry-until }} + # Publish only the N most recent releases; 0 = all. A LITERAL, so it holds on every + # path — the site deploys as one Pages artifact against a hard 1 GB cap. See + # docs/how-to/keep-the-site-small.md. + max-releases: "20" permissions: contents: read - actions: write + actions: read pages: write id-token: write statuses: write ``` -`publish.yml` then routes each event: **deploy** (internal PR / `main` push, or any -dispatch), **re-dispatch** (a tag → re-fires this shim so the deploy runs as a -`workflow_dispatch` — [why](../explanations/architecture.md)), or **warn** (a fork PR, -read-only, never deploys). A maintainer publishes a fork preview by running this workflow -from the Actions tab with the `pr` number. +That is the whole publish setup: no job in `ci.yml`, no shim, nothing to thread through. +A maintainer publishes a fork preview by running this workflow from the Actions tab with +the `pr` number. ## 4. Push your branch and open a PR @@ -173,27 +158,28 @@ see: - **`docs / build`** go green — it builds your docs at the versioned `BASE_URL` and uploads the `docs` artifact. This runs for every PR, forks included. -- **`publish / deploy`** runs for an *internal* PR (a branch in your own repo) and - deploys a preview of just this PR at `https://ORG.github.io/REPO/pr-/`, linked from - the PR's checks/Deployments. A **fork** PR instead gets **`publish / warn`** (a - read-only hint — forks never auto-publish); a maintainer dispatches the wrapper with - the `pr` number to preview one. +Then, once CI finishes, a **`Publish`** run appears under the **Actions** tab — not as a +check on the PR — and deploys a preview of this PR at +`https://ORG.github.io/REPO/pr-/`. A **fork** PR is skipped: forks never auto-publish, +because that run would hold a write token. A maintainer previews one by dispatching +`publish.yml` with the `pr` number. :::{note} First-time exception -On a brand-new repo the `publish` check is **red on this setup PR** — there's no -`main` build yet for the versioned site to anchor on, and the default-branch guard -refuses to publish a site missing it. It clears the moment you merge (step 6). Every -PR after that previews normally. +On a brand-new repo that first `Publish` run **fails** — there's no `main` build yet for +the versioned site to anchor on, and the default-branch guard refuses to publish a site +missing it. It clears the moment you merge (step 6). Every PR after that previews +normally. Because publishing is its own run, this failure does not mark your setup PR +red. ::: ## 5. Merge to main — your first deploy -Merging pushes to `main`, which builds `main` and runs `publish`: it assembles a +Merging pushes to `main`, which builds `main`; the `Publish` run that follows assembles a single-entry `switcher.json` and an `index.html` redirecting to `main/`, and deploys. Visit `https://ORG.github.io/REPO/` — the redirect lands you on `main/` with the switcher showing one entry. (The single-entry first deploy is graceful by design; no release required.) From here on, every push to `main` redeploys and every internal PR -gets its own `/pr-/` preview. +gets its own `/pr-/` preview — each one a separate `Publish` run in the Actions tab. ## 6. Cut your first release @@ -228,7 +214,7 @@ URL for cross-project `objects.inv` references. - The [architecture explanation](../explanations/architecture.md) — why it works this way. -- The [workflow reference](../reference/workflows.md) — the `docs.yml`/`publish.yml` +- The [workflow reference](../reference/workflows.md) — the `docs.yml`/`publish-gh-pages.yml` inputs and the `docs.zip` contract. - Migrating an existing site? See [how-to: migrate from `gh-pages`](../how-to/migrate-from-gh-pages.md).