From c232f541b80f512f0c0f7fa85650e84144132ae9 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Fri, 25 Sep 2026 11:55:10 +0200 Subject: [PATCH] feat(release): publish a GitHub release, and verify what a chart pins Three things were missing from the chart release. It published without checking that the images the chart names exist. A chart version is a promise about images - appVersion pins every IDE image, versions.cloud the operator and service - and breaking that promise surfaces as an ImagePullBackOff in whichever environment installs it next. The IDE image list is read from EduIDE's build matrix at run time, because a copy kept here drifts the moment someone adds an image and a release then verifies a subset and passes. It left no record of what is current. The repository got a bare tag from the release train and nothing else, so answering "what is deployed?" meant reading a tag list or the registry. There is now a GitHub release naming the chart version, what it pins, and how to roll it out, ahead of the generated commit notes. Nothing checked that the two charts carry the same version, which AGENTS.md says they do. They had drifted to 2.2.1 and 2.2.2 across a pair of unrelated fixes - one tag and one release cannot name two versions. The release train keeps its lockstep contract and now says so: it is not the ordinary path, and failing its pre-check against a normal main is correct rather than a bug. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/cut-a-release.md | 127 +++++++++++++++++--------- .github/workflows/ci.yml | 16 ++++ .github/workflows/release-train.yml | 17 +++- .github/workflows/release.yml | 133 +++++++++++++++++++++++++++- 4 files changed, 249 insertions(+), 44 deletions(-) diff --git a/.claude/skills/cut-a-release.md b/.claude/skills/cut-a-release.md index fbddcc9..e37fade 100644 --- a/.claude/skills/cut-a-release.md +++ b/.claude/skills/cut-a-release.md @@ -1,76 +1,121 @@ --- name: cut-a-release -description: Cut an EduIDE platform release across all four repositories. Use when asked to release, cut a version, publish charts, or ship a version of EduIDE. +description: Release an EduIDE chart, which is what moves a component version into the environments. Use when asked to release, cut a version, publish charts, ship a version of EduIDE, or move an environment to a new IDE version. --- # Cutting a release -A release is one version across EduIDE-Cloud, EduIDE, EduIDE-Landing-Page and -EduIDE-Helm. +**Each repository releases on its own cadence.** EduIDE cuts `v1.3.0`, EduIDE-Cloud +cuts its own, the landing page cuts its own. A chart release then says which of +those versions belong together, and a deployment PR says which environments get +them. -**Do not bump the chart version from a workflow or by pushing to `main`.** The -release train deliberately does not do this, and neither should you. It checks -the charts are already at the requested version and fails otherwise. The bump is -a reviewed pull request; automation that pushes to `main` triggers the workflows -watching `main`. +So "release EduIDE 1.3.0 to production" is three merges in three repositories, +in this order. None of them is optional and the order is not a preference. -## 1. Dry run first, always +``` +EduIDE release v1.3.0 images published as 1.3.0 +EduIDE-Helm chart 2.3.0 appVersion 1.3.0 -> every IDE image +EduIDE-deployment chartVersion 2.3.0 merging this IS the deploy +``` + +## What a chart version pins + +One knob per source repository, and the chart version is the name for the set: + +| Value | Repository | Renders as | +|---|---|---| +| `appVersion` in `Chart.yaml` | EduIDE | every IDE image tag | +| `versions.cloud` in `values.yaml` | EduIDE-Cloud | operator, service, conversion webhook | +| `versions.landingPage` in `values.yaml` | EduIDE-Landing-Page | the landing page | + +`versions.ide` in an environment's values overrides `appVersion` for that +installation. It exists for pinning an unreleased image - a `pr-123` tag - and +every use of it is temporary. If one is in place, the comment beside it should +say what has to become true before it goes, and that condition should be checked +whenever a release moves past it. + +## 1. The component release exists first + +Whatever you are pinning must already be published. Check, do not assume: ```bash -gh workflow run release-train.yml --repo EduIDE/EduIDE-Helm \ - -f version=2.3.0 -f dry_run=true +gh release view v1.3.0 --repo EduIDE/EduIDE +gh run list --repo EduIDE/EduIDE --event release --limit 1 # did the build finish? +docker manifest inspect ghcr.io/eduide/eduide/java-17:1.3.0 # did it publish? ``` -Read the summary. It reports which images the version would need, and builds -nothing. +A GitHub release existing means somebody clicked release. It does **not** mean +the images exist: the build runs after the tag, takes the better part of an +hour, and can fail. `release.yml` now refuses to publish a chart whose pinned +images are missing, so getting this wrong costs a red build rather than an +`ImagePullBackOff` in production - but it is still the first thing to check. -## 2. Bump both charts in a pull request +## 2. Bump the chart in a pull request -Both charts, both fields — four values, all identical: +Both charts carry the same version - CI enforces it - and only `appVersion` +moves with the component: ```yaml -# charts/eduide/Chart.yaml AND charts/eduide-cluster/Chart.yaml -version: 2.3.0 -appVersion: "2.3.0" +# charts/eduide/Chart.yaml +version: 2.3.0 # and the same in charts/eduide-cluster/Chart.yaml +appVersion: "1.3.0" # the EduIDE release being pinned ``` -Then regenerate the READMEs, or the `docs-drift` job fails: +Chart version is semver about **the chart**: a values change that alters +rendered output is a minor, a fix is a patch. It does not track the component +version and never has. + +Then regenerate the READMEs, or `docs-drift` fails - the version badges come +from `Chart.yaml`: ```bash docker run --rm -v "$PWD/charts:/helm-docs" -u "$(id -u)" jnorwood/helm-docs:v1.14.2 ``` -Open the PR and let CI run. Do not merge it yourself unless asked to. +Open the PR, let CI run, and do not merge it yourself unless asked. -## 3. Run it for real +## 3. Merging publishes and releases it -```bash -gh workflow run release-train.yml --repo EduIDE/EduIDE-Helm \ - -f version=2.3.0 -f dry_run=false -``` +`release.yml` runs on push to `main` and does four things in order: + +1. verifies every image the chart pins exists and is multi-arch +2. packages and pushes any chart whose version is not already published +3. tags the repository `vX.Y.Z` +4. creates a GitHub release naming what the version pins, with generated notes + +Step 1 reads the IDE image list from EduIDE's build matrix at run time rather +than from a list here, because a hand-kept copy drifts the moment somebody adds +an image and then a release verifies a subset and passes. + +If `main` carries no version bump, steps 2 to 4 do nothing. That is every +ordinary merge. -Order: validate, build all 14 images, verify they exist and are multi-arch, -**then** tag, then publish. Building before tagging means a flaky image build -costs a re-run rather than stranding immutable tags on repositories whose -images were never published. +## 4. Roll it out -## 4. Roll it out separately +Nothing here deploys. In EduIDE-deployment, bump `spec.platform.chartVersion` in +the relevant `environments/*/env.yaml`, in a pull request - staging first, then +production, in separate PRs so production can be reverted without reverting the +environment that proved it. Merging the production one is the deploy. -The train deploys nothing. In EduIDE-deployment, bump -`spec.platform.chartVersion` in the relevant `environments/*/env.yaml`, in a -pull request. Production is never deployed automatically. +## The release train is a different thing -## Version forms +`release-train.yml` moves **all four repositories to one number** and requires +both charts to carry that number in both `version` and `appVersion`. That is a +deliberate lockstep release, not this process, and running it against an +ordinary `main` fails its pre-check by design. -- git tags `vX.Y.Z` -- chart `version`, chart `appVersion` and image tags all `X.Y.Z` -- release candidates `2.3.0-rc.1` throughout +Reach for it only when you actually want every component rebuilt and retagged +together. Almost nothing needs that. ## Common failures | Message | Meaning | |---|---| -| `version is 'X', expected 'Y'` | step 2 skipped, or only one chart/field bumped | -| `missing ghcr.io/...` | a component build failed; check that repository's Actions | -| `is not multi-arch` | one architecture failed; re-run the whole build, not just the merge job | -| `tag v2.3.0 already exists` | pick the next version, tags are immutable | +| `missing ghcr.io/...` in Release Charts | step 1 - the component release's build has not finished, or failed | +| `is not multi-arch` | one architecture failed in the component build; re-run that build, not this one | +| `eduide is X but eduide-cluster is Y` | the two charts drifted; they release together at one version | +| `changed but version is still X` | a chart changed with no bump; `release.yml` would publish nothing | +| `chart READMEs are out of date` | run helm-docs and commit the result | +| `tag vX.Y.Z already exists` | the charts were published under that version already; bump | +| Chart published but no release | the tag already existed - the charts are out, only the release page is missing | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ca27c9a..f86f411 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -46,6 +46,22 @@ jobs: - name: App definitions, landing page and preloading agree run: ./scripts/test-app-consistency.sh + # AGENTS.md says the two charts are released together at the same version + # and nothing checked it, so they drifted to 2.2.1 and 2.2.2 across a pair + # of unrelated fixes. One repository tag and one GitHub release name one + # version, so two versions leave the release ambiguous about half of what + # it published. + - name: Both charts carry the same version + run: | + set -euo pipefail + a="$(awk '/^version:/ {print $2; exit}' charts/eduide/Chart.yaml)" + b="$(awk '/^version:/ {print $2; exit}' charts/eduide-cluster/Chart.yaml)" + if [[ "$a" != "$b" ]]; then + echo "::error file=charts/eduide-cluster/Chart.yaml::eduide is $a but eduide-cluster is $b. They are released together at one version." + exit 1 + fi + echo "both charts are at $a" + - name: Chart version must be bumped when a chart changes if: github.event_name == 'pull_request' run: | diff --git a/.github/workflows/release-train.yml b/.github/workflows/release-train.yml index 37dbe19..c14ae48 100644 --- a/.github/workflows/release-train.yml +++ b/.github/workflows/release-train.yml @@ -1,4 +1,15 @@ -# Cut one EduIDE platform release across four repositories. +# Cut one EduIDE platform release across four repositories, moving all of them +# to a single version. +# +# THIS IS NOT THE ORDINARY RELEASE PATH. Each repository releases on its own +# cadence: EduIDE cuts vX.Y.Z, and a chart release pins it through appVersion. +# `.claude/skills/cut-a-release.md` is that process, and `release.yml` runs it. +# +# Use this workflow only for a deliberate lockstep release, where every +# component is rebuilt and tagged with the same number. It requires all four +# version fields across both charts to equal that number, which is exactly what +# an ordinary chart release does NOT do - so a run against a normal `main` will +# fail the pre-check, and that is correct rather than a bug. # # The order is deliberate: BUILD, VERIFY, then TAG. # @@ -234,7 +245,9 @@ jobs: fi done if [[ $fail -ne 0 ]]; then - echo "::error::Bump the charts in a reviewed pull request first, then re-run. See README: Cutting a release." + echo "::error::This workflow performs a LOCKSTEP release: both charts must carry $V in both version and appVersion." + echo "::error::An ordinary chart release does not do that - appVersion tracks the EduIDE release and the chart version moves on its own." + echo "::error::For a normal release see .claude/skills/cut-a-release.md; for a lockstep one, bump all four fields in a reviewed pull request first." exit 1 fi echo "both charts are at $V" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b1b0749..9260fef 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -16,7 +16,7 @@ jobs: if: github.event_name != 'pull_request' runs-on: ubuntu-latest permissions: - contents: read + contents: write packages: write env: OCI_REGISTRY: ghcr.io @@ -39,7 +39,74 @@ jobs: OWNER="$(echo "${GITHUB_REPOSITORY_OWNER}" | tr '[:upper:]' '[:lower:]')" echo "$GHCR_TOKEN" | helm registry login "$OCI_REGISTRY" -u "$OWNER" --password-stdin + # A chart version is a promise about images: appVersion pins every IDE + # image, versions.cloud the operator and service, versions.landingPage the + # landing page. Publishing a chart that names a tag nobody pushed produces + # an ImagePullBackOff in whichever environment installs it next, which is + # the slowest possible place to find out. Check first, publish second. + - name: Install crane and yq + run: | + set -euo pipefail + curl -fsSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz \ + | sudo tar -xz -C /usr/local/bin crane + echo '${{ secrets.GITHUB_TOKEN }}' | crane auth login ghcr.io -u '${{ github.actor }}' --password-stdin + sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 + sudo chmod +x /usr/local/bin/yq + + - name: Every image the chart pins exists + id: pins + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -uo pipefail + + ide=$(yq -r '.appVersion' charts/eduide/Chart.yaml) + cloud=$(yq -r '.versions.cloud' charts/eduide/values.yaml) + landing=$(yq -r '.versions.landingPage' charts/eduide/values.yaml) + + # The IDE image list is read from EduIDE's build matrix rather than + # kept here: a hand-written copy drifts the moment someone adds an + # image, and then a release verifies a subset and passes. + gh api repos/EduIDE/EduIDE/contents/.github/workflows/build.yml --jq '.content' | base64 -d > /tmp/build.yml + ide_images=$(yq -r '[.jobs.images.strategy.matrix.include[]["image-name"]] | .[]' /tmp/build.yml | sed 's|^eduide/||') + ide_base=$(yq -r '.jobs.base.with["image-name"]' /tmp/build.yml | sed 's|^eduide/||') + if [[ -z "${ide_images// /}" || -z "$ide_base" || "$ide_base" == "null" ]]; then + echo "::error::could not read EduIDE's build matrix; refusing to verify an incomplete image set" + exit 1 + fi + + fail=0 + check() { + local ref="$1" manifest arches + if ! manifest=$(crane manifest "$ref" 2>/dev/null); then + echo "::error::missing $ref" + fail=1 + return + fi + arches=$(jq -r '[.manifests[]?.platform.architecture] | sort | join(",")' <<<"$manifest") + if [[ "$arches" != *amd64* || "$arches" != *arm64* ]]; then + echo "::error::$ref is not multi-arch (has: ${arches:-none})" + fail=1 + return + fi + echo " ok $ref [$arches]" + } + + for img in $ide_base $ide_images; do check "ghcr.io/eduide/${img}:${ide}"; done + for img in eduide-cloud/operator eduide-cloud/service eduide-cloud/conversion-webhook; do + check "ghcr.io/eduide/${img}:${cloud}" + done + check "ghcr.io/eduide/eduide-landing-page:${landing}" + + { + echo "ide=$ide" + echo "cloud=$cloud" + echo "landing=$landing" + } >> "$GITHUB_OUTPUT" + exit $fail + - name: Package and publish charts + id: publish run: | set -euo pipefail @@ -52,6 +119,7 @@ jobs: eduide ) + published="" for chart in "${charts[@]}"; do version="$(awk '/^version:/ {print $2; exit}' "charts/${chart}/Chart.yaml")" artifact="dist/${chart}-${version}.tgz" @@ -66,8 +134,71 @@ jobs: helm package "charts/${chart}" --destination dist helm push "${artifact}" "${OCI_PREFIX}" helm show chart "${ref}" --version "${version}" >/dev/null + published="${published} ${chart}" done + # What the release step tags. Empty when main carried no version bump, + # which is every ordinary merge. + echo "charts=${published# }" >> "$GITHUB_OUTPUT" + echo "version=$(awk '/^version:/ {print $2; exit}' charts/eduide/Chart.yaml)" >> "$GITHUB_OUTPUT" + + # Without this the only record of what is current is a tag list, or the + # registry. A release names the version, says what it pins, and gives + # anyone asking "what is deployed?" one page to read. + - name: Tag and publish a GitHub release + if: steps.publish.outputs.charts != '' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + V: ${{ steps.publish.outputs.version }} + CHARTS: ${{ steps.publish.outputs.charts }} + IDE: ${{ steps.pins.outputs.ide }} + CLOUD: ${{ steps.pins.outputs.cloud }} + LANDING: ${{ steps.pins.outputs.landing }} + run: | + set -euo pipefail + + if git ls-remote --tags origin "refs/tags/v${V}" | grep -q .; then + echo "::notice::tag v${V} already exists; charts published, release left alone" + exit 0 + fi + + pre="" + [[ "$V" == *-* ]] && pre="--prerelease" + + { + echo "Charts published to \`oci://ghcr.io/eduide/charts\`:" + echo + for c in $CHARTS; do + echo "- \`${c}\` $(awk '/^version:/ {print $2; exit}' "charts/${c}/Chart.yaml")" + done + echo + echo "## What this chart version pins" + echo + echo "| Component | Version |" + echo "|---|---|" + echo "| IDE images (EduIDE) | ${IDE} |" + echo "| Operator and service (EduIDE-Cloud) | ${CLOUD} |" + echo "| Landing page | ${LANDING} |" + echo + echo "Every image above was verified to exist and be multi-arch before the charts were pushed." + echo + echo "## Deploying it" + echo + echo "Bump \`spec.platform.chartVersion\` to \`${V}\` in the relevant" + echo "\`environments/*/env.yaml\` in EduIDE-deployment, in a pull request." + echo "Merging that is the deploy; nothing here rolls anything out." + } > /tmp/notes.md + + git tag -a "v${V}" -m "EduIDE charts ${V}" + git push origin "v${V}" + + # --generate-notes adds the commit and contributor list; the body above + # is what a reader actually needs first, so it goes in --notes-file. + gh release create "v${V}" \ + --title "v${V}" \ + --notes-file /tmp/notes.md \ + --generate-notes $pre + release-pr-preview: if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository runs-on: ubuntu-latest