From edccfd861523004d59d601aa123d2c80ed86eb2d Mon Sep 17 00:00:00 2001 From: Christopher Bartz Date: Mon, 31 Aug 2026 12:59:00 +0000 Subject: [PATCH 1/2] docs: describe the charm release and promotion process --- docs/changelog.md | 4 + .../charm-release-and-promotion.md | 139 ++++++++++++++++++ docs/explanation/index.rst | 8 + 3 files changed, 151 insertions(+) create mode 100644 docs/explanation/charm-release-and-promotion.md diff --git a/docs/changelog.md b/docs/changelog.md index f2485e09..fbbfe5bb 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -12,6 +12,10 @@ Each revision is versioned by the date of the revision. - Add a tutorial for deploying GARM. It walks through a first GARM deployment on Canonical Kubernetes with PostgreSQL and the `garm-configurator`, ending with a runner scale set registered on a GitHub repository. OpenStack is represented by placeholder configuration and a stand-in image provider so the tutorial runs without a cloud, and a closing section states what changes for a deployment that boots runners. +## 2026-08-31 + +- Document the charm release pipeline from `latest/edge` through `latest/candidate` to `latest/stable`, including the Renovate-gated production pin and the manual hotfix and rollback paths. + ## 2026-07-13 - `garm`: fix the SSH debug (tmate) connection details so the runner actually reads them. The `TMATE_SERVER_*` variables injected into the shared runner template were written to `/home/runner/.env`, but the GitHub Actions runner sources a `.env` file from its install directory (`/home/runner/actions-runner/.env`), so the debug-SSH server host, port, and fingerprints were silently ignored. They are now written to the `.env` file the runner reads, matching the per-scaleset runner options. diff --git a/docs/explanation/charm-release-and-promotion.md b/docs/explanation/charm-release-and-promotion.md new file mode 100644 index 00000000..aec74f27 --- /dev/null +++ b/docs/explanation/charm-release-and-promotion.md @@ -0,0 +1,139 @@ +--- +myst: + html_meta: + "description lang=en": "Explain how GitHub runner charms move through edge, candidate, and stable." +--- + +(release_process)= + +# Charm release and promotion process + +GitHub runner charm releases move through three Charmhub channels: +`latest/edge`, `latest/candidate`, and `latest/stable`. + +The release process is designed to keep the fast path automated while leaving +the two production decisions with a human reviewer: + +- whether to take a candidate revision into production, and +- whether a candidate revision has truly soaked in production long enough to + become stable. + +```{mermaid} +flowchart TD + PUSH[Push to main] --> EDGE[Publish all charms to latest/edge] + EDGE --> DAILY[Daily edge-to-candidate workflow] + DAILY -->|edge ahead of candidate| E2E[GARM end-to-end test] + E2E -->|pass| CAND[Release garm and garm-configurator to latest/candidate] + CAND --> RENOVATE[Renovate updates the production Terraform pin] + RENOVATE --> PROD[Human approves and applies the production pin] + PROD --> WEEKLY[Weekly candidate-to-stable workflow] + WEEKLY -->|7-day soak, approved| STABLE[Release latest/candidate to latest/stable] +``` + +## Channel flow + +Every push to `main` publishes all charms to `latest/edge` through +`.github/workflows/publish_charms.yml`. + +Once a day, `promote_edge_to_candidate.yaml` compares the edge revision with +the candidate revision. If edge is not ahead of candidate, the workflow skips +so the repository does not rerun the expensive end-to-end test for no change. +When edge is ahead, the workflow runs the GARM end-to-end test from +`garm_e2e.yaml` against the published edge revision. If that test passes, the +workflow releases the new revision to `latest/candidate`. + +Only two charms move through this automated edge-to-candidate promotion: +`garm` and `garm-configurator`. They always move together, and the workflow +releases them atomically so production never sees a mixed pair. + +Production does not follow candidate automatically. Instead, the internal +GitOps Terraform repository pins production to a specific candidate revision. +Renovate opens a revision-bump pull request when a newer candidate is available. +A human must approve and merge that pull request, then apply the Terraform +change. That approval is the production gate. + +Once a candidate revision has soaked for seven days, the weekly +`promote_candidate_to_stable.yaml` workflow promotes it to `latest/stable`. +The workflow uses a GitHub Environment named `charmhub-stable` with required +reviewers so a human approves the stable release at the end of the soak window. + +## Human review gates + +### Renovate pull request gate + +This gate decides whether a candidate revision should move into production. +The reviewer checks that the revision is the one they want to run in the live +environment, then merges the revision-bump pull request and applies the +Terraform change. + +### `charmhub-stable` environment gate + +This gate decides whether a candidate revision is ready to become stable. +The weekly workflow measures soak time using the Charmhub candidate release +timestamp. That timestamp proves when the revision was published to +`latest/candidate`; it does **not** prove that production ran that revision for +the full soak window. The reviewer at this gate must confirm that the revision +really has been running in production for the required time. + +That limitation is intentional in the current design, so the approval step is +the place where a human closes the gap between candidate publication and +production promotion. + +## One-time repository setup + +Create a GitHub Environment named `charmhub-stable` in the repository settings +and configure it with required reviewers. + +Also scope `CHARMHUB_TOKEN` to that environment so the weekly promotion +workflow can authenticate to Charmhub. + +If the environment does not exist, GitHub treats the approval as a no-op rather +than failing the workflow. That means the stable gate silently disappears until +the environment is created, so make sure the environment exists before you rely +on it. + +## Hotfixes + +For a hotfix, build the charm from the fix pull request and release it manually +to candidate: + +```bash +charmcraft release --revision= --channel=latest/candidate +``` + +Then take the same revision to production through the normal Terraform pin in +the internal GitOps repository. The hotfix still follows the production gate; +only the candidate release is done by hand. + +## Rollbacks + +Rollbacks are manual. + +First, clear any Juju units in error: + +```bash +juju resolved / +``` + +Then revert the production Terraform pin to the earlier revision and apply the +change. + +The candidate channel is not rolled back automatically. If you need candidate +to point at an older revision, you must update it separately. + +```{warning} +Before you point `latest/candidate` back at an older revision, disable +`promote_edge_to_candidate.yaml` first. Otherwise the next daily run sees edge +ahead of candidate and immediately repromotes the newer revision over your +rollback. +``` + +The rollback sequence therefore becomes: + +1. Disable the daily edge-to-candidate workflow. +2. Roll back the production Terraform pin. +3. Run `juju resolved` on any units in error. +4. Apply the Terraform change. + +When the older revision should stay on candidate again, re-enable the daily +workflow after you are satisfied that the rollback has settled. diff --git a/docs/explanation/index.rst b/docs/explanation/index.rst index a24e9a50..22f2946f 100644 --- a/docs/explanation/index.rst +++ b/docs/explanation/index.rst @@ -1,2 +1,10 @@ Explanation =========== + +The following document explains the release and promotion process for the +GitHub runner charms. + +.. toctree:: + :maxdepth: 1 + + Charm release and promotion process From a8a56e2243ff494210bc8fd45304cd49d98c12cb Mon Sep 17 00:00:00 2001 From: Christopher Bartz Date: Mon, 31 Aug 2026 13:00:35 +0000 Subject: [PATCH 2/2] docs: order the new changelog entry newest-first --- docs/changelog.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index fbbfe5bb..91c1040e 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -8,14 +8,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Each revision is versioned by the date of the revision. -## 2026-08-19 - -- Add a tutorial for deploying GARM. It walks through a first GARM deployment on Canonical Kubernetes with PostgreSQL and the `garm-configurator`, ending with a runner scale set registered on a GitHub repository. OpenStack is represented by placeholder configuration and a stand-in image provider so the tutorial runs without a cloud, and a closing section states what changes for a deployment that boots runners. - ## 2026-08-31 - Document the charm release pipeline from `latest/edge` through `latest/candidate` to `latest/stable`, including the Renovate-gated production pin and the manual hotfix and rollback paths. +## 2026-08-19 + +- Add a tutorial for deploying GARM. It walks through a first GARM deployment on Canonical Kubernetes with PostgreSQL and the `garm-configurator`, ending with a runner scale set registered on a GitHub repository. OpenStack is represented by placeholder configuration and a stand-in image provider so the tutorial runs without a cloud, and a closing section states what changes for a deployment that boots runners. + ## 2026-07-13 - `garm`: fix the SSH debug (tmate) connection details so the runner actually reads them. The `TMATE_SERVER_*` variables injected into the shared runner template were written to `/home/runner/.env`, but the GitHub Actions runner sources a `.env` file from its install directory (`/home/runner/actions-runner/.env`), so the debug-SSH server host, port, and fingerprints were silently ignored. They are now written to the `.env` file the runner reads, matching the per-scaleset runner options.