migrate: open github-pages env policy + fix migration ordering docs

.github/workflows/_release.yml

@@ -16,7 +16,13 @@ jobs:
           sparse-checkout: plugins/version-switcher.mjs
       - run: cp .src/plugins/version-switcher.mjs .
 
-      # All other release artifacts from other stages (docs.zip).
+      # The two steps below are the generic release job a consumer needs (download the
+      # built docs.zip, attach it to the Release). They are literalinclude'd into
+      # docs/tutorials/adding-to-a-fresh-repo.md between these markers — keep the markers
+      # when editing. The checkout + cp above is plugin-specific (it stages
+      # version-switcher.mjs) and is deliberately left outside the markers.
+      # docs-literalinclude:release-steps:start
+      # Download every artifact produced earlier in the run (here: docs.zip).
       - name: Download artifacts
         uses: actions/download-artifact@v5
         with:
@@ -28,7 +34,8 @@ jobs:
         uses: softprops/action-gh-release@6da8fa9354ddfdc4aeace5fc48d7f679b5214090 # v2.4.1
         with:
           prerelease: ${{ contains(github.ref_name, 'a') || contains(github.ref_name, 'b') || contains(github.ref_name, 'rc') }}
-          files: "*"
+          files: "*"            # everything staged in the cwd — here, docs.zip
           generate_release_notes: true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      # docs-literalinclude:release-steps:end

.github/workflows/publish.yml

@@ -170,20 +170,30 @@ jobs:
           # comparison is robust to a curl/file trailing-newline difference.
           want=$(cat "$local_json")
           url="${PAGE_URL%/}/switcher.json"
-          deadline=$(( $(date +%s) + 60 ))
+          # DIAGNOSTIC: log the origin's response headers each poll. A frozen origin
+          # keeps the SAME etag/last-modified across the whole window while `age`
+          # climbs (the served artifact never changes), distinguishing a real wedge
+          # from slow CDN propagation. Window stretched 60→180s purely to make that
+          # signature unmistakable on this release; revert once we have the data.
+          hdr() { curl -sI "$url?cb=$(date +%s%N)" \
+            | grep -iE '^(HTTP/|etag|last-modified|age|x-served-by|x-cache|x-github-request-id)' \
+            | tr -d '\r' | paste -sd' ' -; }
+          echo "::group::origin baseline headers (before verify loop)"; hdr; echo "::endgroup::"
+          deadline=$(( $(date +%s) + 180 ))
           attempt=0
           while :; do
             attempt=$((attempt + 1))
+            echo "[attempt $attempt @ $(date -u +%H:%M:%S)] $(hdr)"
             got=$(curl -fsS "$url?cb=$(date +%s%N)" || true)   # cache-bust → hit the origin
             if [ -n "$got" ] && [ "$got" = "$want" ]; then
               echo "origin matches the assembled switcher.json (attempt $attempt)"
               exit 0
             fi
             if [ "$(date +%s)" -ge "$deadline" ]; then
-              echo "::error::live switcher.json still does not match the assembled one after 60s — Pages origin likely stale/wedged"
+              echo "::error::live switcher.json still does not match the assembled one after 180s — Pages origin likely stale/wedged"
               echo "--- assembled ($local_json) ---"; printf '%s\n' "$want"
               echo "--- live ($url) ---"; printf '%s\n' "$got"
               exit 1
             fi
-            sleep 1
+            sleep 3
           done

docs/explanations/architecture.md

@@ -53,6 +53,25 @@ whole site makes deletion self-healing: a merged PR or a deleted release simply
 isn't gathered next time, so it disappears — no `keep_files` drift, no branch to
 prune.
 
+### Migrating from gh-pages
+
+Two facts make the [cutover](../how-to/migrate-from-gh-pages.md) safe and fix its
+ordering:
+
+- **Flipping the Pages source from a branch to GitHub Actions is non-destructive.**
+  The last `gh-pages` deployment keeps serving until the first Actions deploy
+  supersedes it ([community
+  discussion #158055](https://github.com/orgs/community/discussions/158055)) — so the
+  source can be flipped up front, with no downtime and no blank window.
+- **A publish replaces the *whole* site, so the default branch must be durable before
+  any publish runs.** A publish that runs before `_sources/<default>.zip` exists would
+  drop `/<default>/`. The migration therefore *seeds* the default branch (a published
+  `pages-default-seed` release captured from the old gh-pages tree) before the first
+  publish — which is the pipeline PR's own CI. This makes that first publish safe even
+  when the repo already serves Pages from Actions (where a publish deploys live
+  immediately); with `guard-default-branch` at its default `true`, an un-seeded publish
+  fails loudly rather than silently dropping the branch.
+
 ## The `docs.zip` and version-token contracts
 
 Two small contracts let the build (in CI) and the reconstruction (in `assemble`)

docs/how-to/migrate-from-gh-pages.md

@@ -1,132 +1,148 @@
 # How-to: migrate from an existing `gh-pages` site
 
 If your docs already publish to a `gh-pages` branch (the `keep_files` model), this
-moves you onto the reconstruct-from-durable-sources model **in a single PR into your
-default branch**, **without losing any served version**, and with an instant
-rollback until the very last step.
+moves you onto the reconstruct-from-durable-sources model **in a single pipeline PR**,
+**without losing any served version**, and with an instant rollback until the very last
+step. It is **two local `migrate.sh` runs, with your pipeline PR doing the deploy in
+between**:
+
+1. **`migrate.sh ORG/REPO`** — backfill releases, seed the default branch, flip the
+   Pages source to Actions + open the environment policy. *Uploads and flips only — no
+   deploy.*
+2. **Open + merge your pipeline PR** — its CI runs the first publish, which reads the
+   seed and persists the default branch durably into the site.
+3. **`migrate.sh ORG/REPO --delete-gh-pages`** — verify the live site, then delete
+   `gh-pages` + the seed release.
 
 ## What you are migrating onto (read this first)
 
 Every deploy reconstructs the whole site from sources with **very different
-durability** — and the safety of the migration hinges entirely on that difference
-(see the [architecture explanation](../explanations/architecture.md)):
+durability** — and the safety of the migration hinges on that difference (see the
+[architecture explanation](../explanations/architecture.md)):
 
 | version | source under the new model | durable? |
 |---|---|---|
 | released tags | a `docs.zip` asset on each **GitHub Release** | **yes, permanent** — but only once attached |
-| default branch (`main`) | persisted each deploy into the site at `_sources/<branch>.zip` | **yes** — once one deploy has captured it (seeded at cutover) |
+| default branch (`main`) | persisted each deploy into the site at `_sources/<branch>.zip` | **yes** — once one deploy has captured it (seeded in run 1) |
 | open PRs (`pr-<n>`) | each PR's build artifact | no — drops on merge/close |
 
-Two consequences drive the whole procedure:
+Two consequences drive the procedure:
 
 1. **Your old releases are not durable yet.** Their docs exist only as directories on
-   `gh-pages`; the matching Releases have no `docs.zip`. So migration includes a
-   one-time **backfill** of `docs.zip` onto those Releases — not just a config flip.
-2. **The default branch has no permanent source until a deploy captures it.** Each
-   deploy persists the branch's `docs.zip` into the published site
-   (`_sources/<branch>.zip`) and restores it from there when the CI artifact expires —
-   but before the *first* such capture there is nothing in the site, and `gh-pages` is
-   the only recoverable copy of `/main/`. So the cutover **seeds** it: it captures the
-   gh-pages `<default>/` tree as a published seed release, which the first deploy reads
-   and persists to `_sources/<branch>.zip`. After that the in-site copy carries `/main/`
-   — *before* the default branch ever builds docs itself — which is what lets a repo cut
-   over to the reusable workflow in **one PR**, ahead of `docs→main`.
+   `gh-pages`; the matching Releases have no `docs.zip`. So run 1 includes a one-time
+   **backfill** of `docs.zip` onto those Releases — not just a config flip.
+2. **The default branch has no permanent source until a publish captures it.** Each
+   deploy persists the branch's `docs.zip` into the site (`_sources/<branch>.zip`) and
+   restores from there when the CI artifact expires — but before the *first* publish
+   there is nothing in the site. So run 1 **seeds** it: it captures the gh-pages
+   `<default>/` tree as a published seed release, which the first publish reads and
+   persists to `_sources/<branch>.zip`. After that the in-site copy carries `/<default>/`
+   — *before* the default branch ever builds docs itself.
+
+For *why* this ordering is safe — the non-destructive source flip and the
+seed-before-publish rule — see the [architecture explanation](../explanations/architecture.md#migrating-from-gh-pages).
 
 > **The load-bearing rule:** *keep `gh-pages` until `_sources/<default>.zip` is live in
-> the deployed site* (the cutover seed gets you there on the first deploy; the
-> `--delete-gh-pages` guard probes exactly this). Deleting `gh-pages` is a **separate,
-> gated step**, never part of the cutover — and it removes the seed release too.
+> the deployed site.* Run 1's seed gets you there on the first publish; the
+> `--delete-gh-pages` guard probes exactly this. Deleting `gh-pages` is the **separate,
+> gated final run** — never part of run 1 — and it removes the seed release too.
 
 ## Before you start
 
-- `gh` authenticated with **repo-admin** on the target repo (flipping the Pages
-  source needs admin; a CI token can't — which is why this is a local script).
-- Land the new pipeline + `myst.yml` changes from the
-  [tutorial](../tutorials/adding-to-a-fresh-repo.md) as **one PR, and merge it into
-  your default branch.** Merging is safe: `gh-pages` keeps serving until you flip the
-  Pages source, so the only effect is that the default branch's CI now produces a
-  `docs.zip` artifact — which is exactly the precondition the deletion later checks
-  for.
+- `gh` authenticated with **repo-admin** on the target repo (flipping the Pages source
+  and setting the environment policy need admin; a CI token can't — which is why this is
+  a local script).
 - Run the script from **inside a clone of the target repo**: it reads the repo's tags
-  and `gh-pages` tree from the working directory (it fetches `origin/gh-pages` for
-  you). Running it from anywhere else will find no tags/branches.
+  and `gh-pages` tree from the working directory (it fetches `origin/gh-pages` for you).
+  Running it from anywhere else will find no tags/branches.
 
-## Step 1 — always dry-run
-
-`scripts/migrate.sh` drives the sequence. Start with `--dry-run`: it prints the
-backfill plan and probes the current site, uploading nothing and skipping every
-destructive step.
+## Step 1 — dry-run (recommended)
 
 ```bash
 scripts/migrate.sh ORG/REPO --dry-run
 ```
 
-Compare the **backfill plan** against the **probe list**: any version the live site
-serves that is *not* in the backfill plan is a `gh-pages` directory with **no
-matching GitHub Release** — it will be **dropped** from the reconstructed site (it
-stays on `gh-pages`, so it is recoverable, but it won't be served). Decide whether
-that is acceptable, or cut real Releases for the versions you want to keep first.
-
-## Step 2 — cutover (reversible)
+It prints the backfill + seed plan and probes the current site, uploading nothing and
+flipping nothing. Compare the **backfill plan** against the **probe list**: any version
+the live site serves that is *not* in the backfill plan is a `gh-pages` directory with
+**no matching GitHub Release** — it will be **dropped** from the reconstructed site (it
+stays on `gh-pages`, so it is recoverable, but won't be served). Cut real Releases for
+any such versions you want to keep before proceeding.
 
-Run it for real:
+## Step 2 — prepare (uploads + flips; no deploy)
 
 ```bash
 scripts/migrate.sh ORG/REPO
 ```
 
-It executes, in order, then **stops with `gh-pages` intact**:
-
-1. **Backfill (non-destructive, idempotent).** For each release tag that is a
-   `gh-pages` directory and whose Release lacks a `docs.zip`, zip that directory as a
-   bare `html/` root and attach it as `docs.zip`. Tags containing `/` are skipped.
-   This is first: the reconstructed site is built from these assets.
-2. **Flip the Pages source → GitHub Actions** (`gh api PUT …/pages`). `deploy-pages`
-   refuses to publish unless the source is already "GitHub Actions", so this must
-   precede the deploy.
-3. **Trigger a deploy.** Pass `--deploy-workflow <file>` to dispatch one (your
-   `publish.yml` `workflow_dispatch` works: the default branch is *gathered* from its
-   latest CI artifact, so a pure dispatch still reconstructs `main` + the backfilled
-   releases), or push to the default branch / re-run its CI when prompted.
-4. **Verify.** The script fetches `switcher.json` and checks every listed version URL
-   returns `200`. Probes are cache-busted, so a too-early probe can't pin a cached
-   `404` at the CDN.
-
-It does **not** delete `gh-pages`. You are now live on the new model. The cutover
-deploy should have **seeded** the default branch — captured its gh-pages content as a
-seed release that this deploy persisted to `_sources/<default>.zip` — so `/main/` is
-now served from the in-site copy. `gh-pages` is retained only as the rollback.
-
-## Step 3 — finalize (irreversible), once `_sources/<default>.zip` is live
-
-When the deployed site serves `_sources/<default>.zip` (the cutover seed gets you there;
-or, later, the default branch building `docs.zip` itself keeps it fresh), delete
-`gh-pages`:
+It does the following, then **stops with `gh-pages` intact and still serving**:
+
+1. **Backfill (non-destructive, idempotent).** For each release tag that is a `gh-pages`
+   directory and whose Release lacks a `docs.zip`, zip that directory (bare `html/`
+   root) and attach it as `docs.zip`. Tags containing `/` are skipped.
+2. **Seed the default branch.** Capture the gh-pages `<default>/` tree as the published
+   `pages-default-seed` release, so the default branch is durable before any publish.
+3. **Flip the Pages source → GitHub Actions** and **open the `github-pages` environment's
+   `deployment_branch_policy`** to "no restriction" (so deploys from PR/tag refs — which
+   run under the nested-publish model — aren't rejected by the environment). The flip is
+   non-destructive; the site keeps serving the last `gh-pages` deployment until step 3
+   publishes.
+
+No deploy is triggered here — that is your pipeline PR's job.
+
+## Step 3 — publish, via your pipeline PR
+
+Prepare the new pipeline + `myst.yml` changes from the
+[tutorial](../tutorials/adding-to-a-fresh-repo.md) on a branch, and **open the PR only
+after run 1 has seeded the default branch** (so its first publish is safe). Then open
+and merge it: its CI runs the first **publish** — with the source on Actions, the seed
+present, and the env policy open, it reconstructs the whole site
+(default branch from the seed, the backfilled releases, any open PRs) and deploys it,
+**persisting `_sources/<default>.zip`** into the published site. Merging then has the
+default branch build its own docs, so `_sources/<default>.zip` refreshes with real
+content and the seed becomes redundant.
+
+Confirm the site is live on the new model (visit `https://ORG.github.io/REPO/` and the
+switcher) before finalizing.
+
+## Step 4 — finalize (irreversible)
+
+Once the publish has deployed and `_sources/<default>.zip` is live:
 
 ```bash
 scripts/migrate.sh ORG/REPO --delete-gh-pages
 ```
 
-This **guards** the deletion: it refuses unless `https://ORG.github.io/REPO/_sources/<default>.zip`
-returns `200` — i.e. the deployed site holds a durable copy of the default branch, so
-the new model can reconstruct `/main/` without `gh-pages`. It then re-verifies the live
-site, asks you to type the repo name, deletes `gh-pages`, **and deletes the seed
-release** (the in-site `_sources` copy supersedes it). After this, the rollback is gone.
+This **guards** the deletion: it refuses unless
+`https://ORG.github.io/REPO/_sources/<default>.zip` returns `200` — i.e. the deployed
+site holds a durable copy of the default branch, so the new model can reconstruct
+`/<default>/` without `gh-pages`. It then verifies the live site, asks you to type the
+repo name, deletes `gh-pages`, **and deletes the seed release** (the in-site `_sources`
+copy supersedes it). After this, the rollback is gone.
+
+> **Caveat — old pages that reference `gh-pages` at runtime.** Docs built under the old
+> model sometimes embed a hardcoded version switcher that reads `gh-pages` live — via
+> the GitHub *contents API* (`…/contents?ref=gh-pages`) or by loading assets from a
+> `gh-pages` URL. Those pages are reconstructed verbatim from their `docs.zip`, so the
+> references remain. After deletion, a switcher that only *queries the API* degrades
+> harmlessly: the request `404`s, its populate script throws an uncaught promise
+> (console-only), and the version list simply empties — the page itself is intact.
+> Anything that *loads assets* (CSS/JS/images) from `gh-pages`, though, will break.
+> `grep` your old release pages for `gh-pages` before finalizing and accept (the
+> switcher emptying is usually fine) or fix what you find.
 
 ## Rollback
 
-Between the cutover (step 2) and the deletion (step 3), `gh-pages` is no longer
-*served* but still *exists* — your rollback. If anything is wrong, flip the Pages
-source back to **Deploy from a branch / `gh-pages`** and serving is restored
-instantly, with nothing lost. That is exactly why the deletion is a separate, gated
-step.
+Between run 1 (step 2) and the deletion (step 4), `gh-pages` is no longer *served* but
+still *exists* — your rollback. If anything is wrong, flip the Pages source back to
+**Deploy from a branch / `gh-pages`** and serving is restored instantly, with nothing
+lost. That is exactly why the deletion is a separate, gated run.
 
 ## Flags
 
 | flag | effect |
 |---|---|
-| `--dry-run` | Print the backfill plan + probe the current site only; upload nothing; skip flip/deploy. |
-| `--delete-gh-pages` | **The only mode that deletes.** Guard that the default branch publishes `docs.zip`, re-verify, then delete `gh-pages`. |
+| `--dry-run` | Print the backfill + seed plan + probe the current site only; upload nothing; skip the flip. |
+| `--delete-gh-pages` | **The only mode that deletes.** Guard that `_sources/<default>.zip` is live (`200`), verify the live site, then delete `gh-pages` **and the seed release**. |
 | `--pages-ref <ref>` | `gh-pages` ref to read (default `origin/gh-pages`). |
-| `--deploy-workflow <file>` | `gh workflow run <file>` to trigger the cutover deploy; omit to be prompted to deploy by hand. |
 | `--yes` | Skip the typed confirmation before deleting `gh-pages` (with `--delete-gh-pages`; use with care). |

docs/myst.yml

@@ -42,4 +42,8 @@ site:
     # default flattened /directive.
     folders: true
     logo: images/dls-logo.svg
+    # Pin the favicon to the bundled DLS logo so `myst build` never fetches the
+    # default favicon from mystmd.org (a flaky remote fetch that intermittently
+    # fails the build).
+    favicon: images/dls-logo.svg
     github_url: https://github.com/DiamondLightSource/myst-version-switcher-plugin