From 248435a1d1a697a32345f6304fcad28f6252dd7c Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Fri, 28 Aug 2026 03:14:37 +0200 Subject: [PATCH 1/2] docs: correct the known gaps, and record two traps that cost time The Known gaps table still said bootstrap passes no `gatewayClass` or `envoyProxy`. It does now, and tum-production depends on it - that entry would have sent the next person to create those by hand. Only `spec.loadBalancerIP` remains inert. Documents `spec.redirects`, which nothing described, and how to check one with headers rather than a browser. Adds a runbook for a certificate that sits unissued with no challenges outstanding: a transient ACME finalize failure puts cert-manager into an hour of backoff, and the fix is to clear it rather than to go looking at the config. Two traps in AGENTS.md: - a release tag is vX.Y.Z and the image is X.Y.Z, but an image-tag override is used verbatim, so a caller passing the raw tag publishes v1.2.0. The spelling that passes the tag-format check is the one that breaks the images. - helm-diff must stay pinned. Its current release cannot be parsed by the pinned helm, the plugin fails to load, and both preview steps end in `|| true` - so every rendered diff was silently empty. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019qeiQRFu8xAMRYWPdZewjG --- AGENTS.md | 14 +++++++++++++ docs/cluster-setup.md | 41 +++++++++++++++++++++++++++++-------- docs/envoy-gateway-setup.md | 26 +++++++++++++++++++++++ 3 files changed, 73 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6d4b3f3..c233cde 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -193,6 +193,20 @@ not ours to move. node; under `--wait` it would time out and `--atomic` would then roll back a healthy deploy. +**A release tag is `vX.Y.Z`; the image it publishes is `X.Y.Z`.** The shared +build workflow normalises with `${RELEASE_TAG#v}`, but an `image-tag` override +is used verbatim - so a caller passing `github.event.release.tag_name` defeats +the normalisation and publishes `v1.2.0`, which no chart can consume. Releasing +`1.2.0` without the `v` happens to produce the right image and fails the Tag +format check, which is how this stayed hidden. Fixed in EduIDE-Cloud and +EduIDE-Landing-Page; check any repo that has not been. + +**helm-diff must stay pinned.** Its current release declares `platformHooks`, +which the pinned helm v3.16.3 cannot parse, so the plugin fails to load. Both +the deploy and the bootstrap preview end in `|| true`, so the diff renders +nothing at all and the step still passes - every "Pending change" summary was +silently empty until it was pinned to v3.9.11. + ## Two `lookup` calls make rendering nondeterministic The `eduide` chart preserves live `minInstances`/`maxInstances` on diff --git a/docs/cluster-setup.md b/docs/cluster-setup.md index d130d47..9a010ea 100644 --- a/docs/cluster-setup.md +++ b/docs/cluster-setup.md @@ -14,6 +14,8 @@ that reports itself healthy and serves nothing. | Gateway API CRDs, Envoy Gateway, cert-manager, storage | **you**, once per cluster | | A GatewayClass whose load balancer address matches DNS | **you** | | The ACME `ClusterIssuer` | `Bootstrap cluster`, from `spec.acmeEmail` | +| The GatewayClass and its EnvoyProxy | `Bootstrap cluster`, from `spec.gatewayClass` and `spec.envoyProxy` | +| Redirects from hostnames you used to serve | `Bootstrap cluster`, from `spec.redirects` | | The webview wildcard certificate | **you**, see [tum-certificates.md](tum-certificates.md) | | DNS records | **you** (RBG) | | Keycloak client and redirect URIs | **you**, see [keycloak-setup.md](keycloak-setup.md) | @@ -112,9 +114,9 @@ own `EnvoyProxy` pinned to pool `lb3`, and set `gatewayClassName` in `clusters/tum-student.yaml` to match. EduIDE then has its own data plane on `131.159.88.14`, which is what DNS already says. -The `eduide-cluster` chart can create both objects (`gatewayClass.create` and -`envoyProxy.create`), but **`bootstrap-cluster.yml` does not pass either**, so -today option (b) means creating them by hand or extending the workflow. See +`bootstrap-cluster.yml` passes both from the cluster manifest, so option (b) is +`spec.gatewayClass.create: true` plus a `spec.envoyProxy` naming the pool - +which is exactly what `tum-production` does. See [envoy-gateway-setup.md](envoy-gateway-setup.md) for the MetalLB details. `clusters/tum-production.yaml` carries `spec.loadBalancerIP: 131.159.88.82`. @@ -317,12 +319,35 @@ EduIDE somewhere unexpected. `runner` is per cluster because deploying is not building: a deploy has to reach the API server, and the clusters may differ in how they are reachable. -## Known gaps +## When a certificate will not issue + +Two failures look identical from outside - the listener never programs - and +have different causes. + +**No challenges outstanding, order `errored`.** Let's Encrypt occasionally fails +to finalize an order that has already validated: + +``` +Failed to finalize Order: 404 urn:ietf:params:acme:error:malformed: +Certificate not found +``` + +cert-manager retries, but only after an exponential backoff that starts at an +hour. Clear the backoff to retry immediately: -Three things this repository describes but does not yet apply. All three are -listed above in context; collected here so they are not missed: +```bash +kubectl -n eduide-system patch certificate --type=merge --subresource=status \ + -p '{"status":{"lastFailureTime":null,"failedIssuanceAttempts":null}}' +``` + +It issued on the first retry when this happened during the `tum-production` +bootstrap. Nothing is wrong with the configuration; do not go looking. + +**Challenges pending, staying pending.** That is a real problem: the name has no +listener, or DNS does not reach the Gateway. See step 3. + +## Known gaps | Gap | Consequence | |---|---| -| `bootstrap-cluster.yml` passes no `gatewayClass` or `envoyProxy` | A cluster needing its own data plane address must have it created by hand (step 2) | -| `spec.loadBalancerIP` is in the schema and in `tum-production.yaml` | Nothing reads it. It records intent only (step 2) | +| `spec.loadBalancerIP` is in the schema and in `tum-production.yaml` | Nothing reads it. `spec.envoyProxy` is what actually pins the address (step 2) | diff --git a/docs/envoy-gateway-setup.md b/docs/envoy-gateway-setup.md index e4a00f3..371f65c 100644 --- a/docs/envoy-gateway-setup.md +++ b/docs/envoy-gateway-setup.md @@ -136,6 +136,32 @@ dig +short eduide.student.k8s.aet.cit.tum.de A The Gateway's address and the DNS answer must be the same. A service's `clusterIP` is internal and never what DNS should point at. +## Redirecting a hostname you used to serve + +A renamed installation leaves its old hostname in bookmarks, course pages and +Artemis exercise configurations. `spec.redirects` in the cluster manifest keeps +it answering and 301s it to the new name, path and query intact: + +```yaml +redirects: + - name: legacy-landing + from: theia.artemis.cit.tum.de + to: eduide.artemis.cit.tum.de + tlsSecretName: legacy-landing-cert-secret +``` + +Bootstrap gives it a listener and an ACME `:80` listener, and adds its hostname +to the certificate for that secret, so it renews like any other name. Check it +with headers, not a browser: + +```bash +curl -sI https://theia.artemis.cit.tum.de/foo?bar=1 | grep -i '^location' +# location: https://eduide.artemis.cit.tum.de/foo?bar=1 +``` + +A 302 would work but should not be used: it is not cached, so the old hostname +keeps taking traffic indefinitely. + ## TLS Each listener terminates with a Secret named per cluster in From 90389f3ad123ac617a36604453737727ff51e4b7 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Fri, 28 Aug 2026 18:53:04 +0200 Subject: [PATCH 2/2] docs: address review feedback on the cluster setup guides Six findings from the review, all valid. envoy-gateway-setup.md still claimed bootstrap-cluster.yml passes neither the GatewayClass nor the EnvoyProxy block and that both must be created by hand. It passes both, and creating them by hand as well would have two owners fighting over the same names. The redirect section presented the ACME :80 listener as unconditional. Bootstrap only emits it when the cluster sets acmeHttp, so a cluster issuing certificates some other way has no such listener. The 302 rationale was wrong: a 302 is not categorically uncacheable. The reason to use 301 is that it means permanently moved, which is what lets clients update what they have stored. cluster-setup.md described spec.envoyProxy as naming the MetalLB pool. It names the EnvoyProxy; the pool sits in its nested provider config. The certificate runbook jumped straight to patching away the backoff. It now says to confirm the finalize 404 down the CertificateRequest/Order/Challenge chain first, notes that --subresource needs kubectl v1.24, and says what to do when the cleared retry also fails - the previous 'do not go looking' was too absolute. Pending challenges are also split by cause: an HTTP 404 means something answered but did not route the solver path, while a DNS or connection failure means nothing answered at all. Also typed six fenced blocks that markdownlint flagged as MD040. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019qeiQRFu8xAMRYWPdZewjG --- docs/cluster-setup.md | 48 +++++++++++++++++++++++++++---------- docs/envoy-gateway-setup.md | 23 +++++++++++------- 2 files changed, 50 insertions(+), 21 deletions(-) diff --git a/docs/cluster-setup.md b/docs/cluster-setup.md index 9a010ea..1e60778 100644 --- a/docs/cluster-setup.md +++ b/docs/cluster-setup.md @@ -85,7 +85,7 @@ Envoy Gateway can be configured to **merge gateways**: every `Gateway` using a GatewayClass shares one Envoy deployment and therefore one external address. `tum-student` is configured that way today: -``` +```text GatewayClass envoy -> parametersRef: EnvoyProxy envoy-gateway-system/artemis-envoy-proxy mergeGateways: true @@ -94,7 +94,7 @@ GatewayClass envoy and the EduIDE hostnames point somewhere else: -``` +```text eduide.student.k8s.aet.cit.tum.de -> k8s-stud-lb3 -> 131.159.88.14 (pool lb3) ``` @@ -115,8 +115,10 @@ own `EnvoyProxy` pinned to pool `lb3`, and set `gatewayClassName` in `131.159.88.14`, which is what DNS already says. `bootstrap-cluster.yml` passes both from the cluster manifest, so option (b) is -`spec.gatewayClass.create: true` plus a `spec.envoyProxy` naming the pool - -which is exactly what `tum-production` does. See +`spec.gatewayClass.create: true` plus a `spec.envoyProxy` block. `envoyProxy.name` +is the name of the EnvoyProxy resource itself; the MetalLB pool goes inside it, +under `spec.provider.kubernetes.envoyService.annotations`. That is exactly what +`tum-production` does. See [envoy-gateway-setup.md](envoy-gateway-setup.md) for the MetalLB details. `clusters/tum-production.yaml` carries `spec.loadBalancerIP: 131.159.88.82`. @@ -176,7 +178,7 @@ certificate supplied through `wildcardTLSSecret`; see Four records per environment, all pointing at the address from step 2: -``` +```text the landing page service. the REST service instance. session ingress @@ -205,7 +207,7 @@ Full instructions, including what is missing today, are in ## Step 6: bootstrap -``` +```text Actions -> Bootstrap cluster cluster: tum-student chart_version: 2.1.0 @@ -276,7 +278,7 @@ invalid certificate. Nothing appeared in any log. ## Step 8: deploy an environment -``` +```text Actions -> Deploy (dispatch) -> environment: test1, dry_run: true ``` @@ -327,13 +329,22 @@ have different causes. **No challenges outstanding, order `errored`.** Let's Encrypt occasionally fails to finalize an order that has already validated: -``` +```text Failed to finalize Order: 404 urn:ietf:params:acme:error:malformed: Certificate not found ``` -cert-manager retries, but only after an exponential backoff that starts at an -hour. Clear the backoff to retry immediately: +Confirm it is that failure before doing anything, by reading down the chain the +`Certificate` owns - its conditions alone do not tell you: + +```bash +kubectl -n eduide-system describe certificate +kubectl -n eduide-system get certificaterequest,order,challenge +``` + +cert-manager retries on its own, but only after an exponential backoff that +starts at an hour. Clear the backoff to retry immediately (`--subresource` +needs kubectl v1.24 or newer): ```bash kubectl -n eduide-system patch certificate --type=merge --subresource=status \ @@ -341,10 +352,21 @@ kubectl -n eduide-system patch certificate --type=merge --subresource=sta ``` It issued on the first retry when this happened during the `tum-production` -bootstrap. Nothing is wrong with the configuration; do not go looking. +bootstrap, so one clean retry is the expected outcome and no configuration +change is called for. If the retry fails too, the transient explanation is +wrong: go back over the `CertificateRequest`, `Order`, `Challenge` and +`ClusterIssuer`, and the cert-manager controller logs, before changing anything. + +**Challenges pending, staying pending.** That is a real problem, and the error +on the `Challenge` says which: + +- **HTTP 404** - something answered, but nothing routes the solver path. The + hostname has no listener on the Gateway, or the listener exists on a Gateway + the solver's HTTPRoute does not attach to. +- **DNS, connection refused or timeout** - nothing answered at all. The name + does not resolve, or it resolves somewhere that is not this Gateway. -**Challenges pending, staying pending.** That is a real problem: the name has no -listener, or DNS does not reach the Gateway. See step 3. +See step 3. ## Known gaps diff --git a/docs/envoy-gateway-setup.md b/docs/envoy-gateway-setup.md index 371f65c..84cf301 100644 --- a/docs/envoy-gateway-setup.md +++ b/docs/envoy-gateway-setup.md @@ -58,7 +58,7 @@ If the GatewayClass is configured with `mergeGateways: true`, **every** Gateway using that class shares one Envoy deployment and one external address. On `tum-student` today: -``` +```text GatewayClass envoy -> EnvoyProxy envoy-gateway-system/artemis-envoy-proxy mergeGateways: true @@ -96,8 +96,10 @@ envoyProxy: type: LoadBalancer ``` -**`bootstrap-cluster.yml` does not pass either block.** Until it does, create -them by hand or add the values to the workflow. `spec.loadBalancerIP` in +**`bootstrap-cluster.yml` passes both blocks straight through** from +`clusters/.yaml`, so the manifest above is the whole configuration - do +not create a GatewayClass or an EnvoyProxy by hand as well, or bootstrap and +the hand-made copy will fight over the same names. `spec.loadBalancerIP` in `clusters/tum-production.yaml` records the intended address but is read by nothing. @@ -150,17 +152,22 @@ redirects: tlsSecretName: legacy-landing-cert-secret ``` -Bootstrap gives it a listener and an ACME `:80` listener, and adds its hostname -to the certificate for that secret, so it renews like any other name. Check it -with headers, not a browser: +Bootstrap gives it an HTTPS listener and adds its hostname to the certificate +for that secret, so it renews like any other name. It also gets an ACME `:80` +listener, but **only when the cluster sets `spec.acmeHttp: true`** - on a +cluster that issues its certificates some other way there is no such listener, +and nothing should be written that assumes one. Check it with headers, not a +browser: ```bash curl -sI https://theia.artemis.cit.tum.de/foo?bar=1 | grep -i '^location' # location: https://eduide.artemis.cit.tum.de/foo?bar=1 ``` -A 302 would work but should not be used: it is not cached, so the old hostname -keeps taking traffic indefinitely. +A 302 would work but should not be used. It means *temporarily* moved, so +clients and bookmarks keep the old hostname and keep sending traffic to it. A +301 says the move is permanent, which is both true here and what lets clients +and search engines update what they have stored. ## TLS