From 89c7719a4b1c2b9176a3f40720d3a17c0711115d Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Thu, 24 Sep 2026 19:44:02 +0200 Subject: [PATCH 1/4] docs: the eduide cluster is provisioned, and it has no load balancer Three places still said the `eduide` cluster does not exist. It does: parma, a single node k3s that has been serving Bonn and Mannheim since 2026-08-28. `clusters/eduide.yaml` was corrected when the cluster came up (#125), the README, AGENTS.md and docs/environments.md were not. Checked field by field against the live cluster rather than assumed: storage class `local-path`, GatewayClass `eduide`, the ACME issuer's contact address, the cert-manager ServiceMonitor, the Alertmanager config and the identity ConfigMap all match the manifest as written. `docs/cluster-setup.md` offered two ways to give EduIDE an address, both of them a load balancer - the decision step that the doc itself warns produces a cluster which reports itself healthy and serves nothing. parma has no load balancer at all, so that page had no option covering the only cluster of ours it applies to. Option (c) is now written down, with the hostNetwork dead end the cluster manifest records. The same step now also says to state the data plane's replica count. Leaving it out is what took bonn and mannheim down on 2026-09-23: Envoy Gateway stops reconciling the field when the EnvoyProxy omits it, so a single scale to zero was permanent. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 4 +++- README.md | 6 +++++- docs/cluster-setup.md | 36 +++++++++++++++++++++++++++++------- docs/environments.md | 8 ++++---- 4 files changed, 41 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3a1e1b6..a2f7d7a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,9 @@ An environment is one namespace on one cluster. `environments/`, the GitHub Environment and the landing host are the same string, so there is nothing to map and nothing to keep in sync. -The `eduide` cluster is not provisioned yet. +The `eduide` cluster is `parma.aet.cit.tum.de`, a single node k3s serving Bonn +and Mannheim since 2026-08-28. It has no load balancer, so its data plane +binds the node's ports with `hostPort`; `clusters/eduide.yaml` says why. The charts live in **EduIDE-Helm** and are pulled from `oci://ghcr.io/eduide/charts`. Chart templates are not edited here. diff --git a/README.md b/README.md index 9d9abee..8e6c6b7 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,11 @@ An environment is one namespace on one cluster. `environments/`, the GitHub Environment and the landing host are the same string, so there is nothing to map and nothing to keep in sync. -The `eduide` cluster is **not provisioned yet**. +The `eduide` cluster is `parma.aet.cit.tum.de` (131.159.88.106), a single +node k3s serving Bonn and Mannheim since 2026-08-28. It has **no load +balancer** - the Envoy data plane binds the node's own `:80` and `:443` with +`hostPort`. `clusters/eduide.yaml` records why, and why the obvious +alternative does not work. Namespaces stay short - `eduide-test1`, `eduide-tum-production` - because a Kubernetes namespace cannot contain dots. `spec.namespace` in each `env.yaml` diff --git a/docs/cluster-setup.md b/docs/cluster-setup.md index 1e60778..38c95e7 100644 --- a/docs/cluster-setup.md +++ b/docs/cluster-setup.md @@ -1,7 +1,9 @@ # Preparing a cluster -What has to exist on a TUM cluster before EduIDE can be deployed to it, which -parts `Bootstrap cluster` does for you, and which parts it does not. +What has to exist on a cluster before EduIDE can be deployed to it, which parts +`Bootstrap cluster` does for you, and which parts it does not. Written around +the TUM clusters, but the `eduide` cluster is not one of them and the places +that differ say so. Read this end to end before bootstrapping a cluster for the first time. Two of the manual steps below cannot be discovered by trying: they produce a cluster @@ -12,7 +14,7 @@ that reports itself healthy and serves nothing. | | Who does it | |---|---| | Gateway API CRDs, Envoy Gateway, cert-manager, storage | **you**, once per cluster | -| A GatewayClass whose load balancer address matches DNS | **you** | +| A GatewayClass whose address matches DNS - a load balancer's, or the node's | **you** | | The ACME `ClusterIssuer` | `Bootstrap cluster`, from `spec.acmeEmail` | | The GatewayClass and its EnvoyProxy | `Bootstrap cluster`, from `spec.gatewayClass` and `spec.envoyProxy` | | Redirects from hostnames you used to serve | `Bootstrap cluster`, from `spec.redirects` | @@ -114,13 +116,33 @@ own `EnvoyProxy` pinned to pool `lb3`, and set `gatewayClassName` in `clusters/tum-student.yaml` to match. EduIDE then has its own data plane on `131.159.88.14`, which is what DNS already says. -`bootstrap-cluster.yml` passes both from the cluster manifest, so option (b) is -`spec.gatewayClass.create: true` plus a `spec.envoyProxy` block. `envoyProxy.name` -is the name of the EnvoyProxy resource itself; the MetalLB pool goes inside it, -under `spec.provider.kubernetes.envoyService.annotations`. That is exactly what +**(c) There is no load balancer at all.** A single node cluster outside TUM may +have neither MetalLB nor servicelb, and then a Service of type `LoadBalancer` +sits `Pending` for ever while DNS publishes the node's own address. Bind the two +public ports on the node instead: `envoyService.type: ClusterIP`, and a +`StrategicMerge` patch on the Envoy Deployment giving its container `hostPort: 80` +and `hostPort: 443`. The `eduide` cluster does this, and +`clusters/eduide.yaml` carries the whole story - including why hostNetwork plus +`useListenerPortAsContainerPort` is the wrong answer: Envoy runs as non-root, +Kubernetes cannot grant `NET_BIND_SERVICE` effectively, and every listener then +fails with `cannot bind '0.0.0.0:80': Permission denied` while the pod reports +`Running`. + +`bootstrap-cluster.yml` passes both from the cluster manifest, so options (b) and +(c) are `spec.gatewayClass.create: true` plus a `spec.envoyProxy` block. +`envoyProxy.name` is the name of the EnvoyProxy resource itself; for (b) the +MetalLB pool goes inside it, under +`spec.provider.kubernetes.envoyService.annotations`. That is exactly what `tum-production` does. See [envoy-gateway-setup.md](envoy-gateway-setup.md) for the MetalLB details. +**State the data plane's replica count, whichever option you pick.** Envoy +Gateway reconciles the Envoy Deployment's replicas only while the EnvoyProxy +names them; left unset it writes the field once and never looks again, so one +`kubectl scale --replicas=0` takes every Gateway on the class down until +somebody scales it back by hand. The chart defaults it to 1 - do not override it +with nothing. + `clusters/tum-production.yaml` carries `spec.loadBalancerIP: 131.159.88.82`. **Nothing reads it.** It records the intent; it does not enforce it. diff --git a/docs/environments.md b/docs/environments.md index 9acf8b8..40b2f34 100644 --- a/docs/environments.md +++ b/docs/environments.md @@ -38,11 +38,11 @@ base sets. |---|---|---| | `tum-student` | `test1.…`, `test2.…`, `test3.…`, `e2e.…`, `staging.eduide.student.k8s.aet.cit.tum.de` | everything non-production | | `tum-production` | `eduide.artemis.cit.tum.de` | TUM's own installation | -| `eduide` | `bonn.eduide.aet.cit.tum.de`, `mannheim.eduide.aet.cit.tum.de` | other universities. **Not provisioned yet.** | +| `eduide` | `bonn.eduide.aet.cit.tum.de`, `mannheim.eduide.aet.cit.tum.de` | other universities. Single node k3s (`parma`), no load balancer | -The two `eduide` ones exist as reviewable configuration before the cluster does. -Deploying one stops at the cluster identity check until that cluster has been -bootstrapped and the GitHub Environment holds a `KUBECONFIG`. +An environment can be written and reviewed before its cluster exists. Deploying +one stops at the cluster identity check until that cluster has been bootstrapped +and the GitHub Environment holds a `KUBECONFIG`. ## Hostnames From 6f45792fd913eefb415a60dbcb36a25f20e2cdcc Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Thu, 24 Sep 2026 20:55:07 +0200 Subject: [PATCH 2/4] docs: name the data plane, not the GatewayClass, as what holds the address Review feedback on the step-2 rewrite. A GatewayClass has no address - the data plane does, whether that is a load balancer or the node itself - so the prerequisites table sent an operator looking for a field on the wrong resource. Step 2's heading still said "load balancer address" after option (c) was added, which framed the one cluster with no load balancer around infrastructure it does not have, and the traffic path in envoy-gateway-setup.md still described the load balancer as the only possibility. The section also still said there were two ways out. There are three. Co-Authored-By: Claude Opus 5 --- docs/cluster-setup.md | 6 +++--- docs/envoy-gateway-setup.md | 4 +++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/cluster-setup.md b/docs/cluster-setup.md index 38c95e7..2f568ba 100644 --- a/docs/cluster-setup.md +++ b/docs/cluster-setup.md @@ -14,7 +14,7 @@ that reports itself healthy and serves nothing. | | Who does it | |---|---| | Gateway API CRDs, Envoy Gateway, cert-manager, storage | **you**, once per cluster | -| A GatewayClass whose address matches DNS - a load balancer's, or the node's | **you** | +| A data plane answering on the address DNS publishes - a load balancer's, or the node's | **you** | | The ACME `ClusterIssuer` | `Bootstrap cluster`, from `spec.acmeEmail` | | The GatewayClass and its EnvoyProxy | `Bootstrap cluster`, from `spec.gatewayClass` and `spec.envoyProxy` | | Redirects from hostnames you used to serve | `Bootstrap cluster`, from `spec.redirects` | @@ -78,7 +78,7 @@ raising with whoever owns the cluster. The deploy sidesteps it by always naming every node, so the default set of eight is roughly 20 GB per node, once. Budget it before the first bootstrap rather than discovering it as disk pressure. -## Step 2: decide which load balancer address serves EduIDE +## Step 2: decide which address serves EduIDE This is the step that produces a healthy-looking cluster that serves nothing, so do it before anything else. @@ -104,7 +104,7 @@ So a Gateway created with `gatewayClassName: envoy` on that cluster comes up `Programmed=True`, is served on `131.159.88.15`, and is unreachable at every name DNS actually publishes. Nothing reports an error. -There are two ways out, and it is a decision, not a default: +There are three ways out, and it is a decision, not a default: **(a) Join the existing merged gateway.** Ask for the EduIDE DNS names to point at `131.159.88.15` instead. Nothing in this repository changes. EduIDE then diff --git a/docs/envoy-gateway-setup.md b/docs/envoy-gateway-setup.md index 84cf301..a22ad1d 100644 --- a/docs/envoy-gateway-setup.md +++ b/docs/envoy-gateway-setup.md @@ -9,7 +9,9 @@ step 2 of it. ## The traffic path -1. DNS points the EduIDE hostnames at Envoy Gateway's load balancer address. +1. DNS points the EduIDE hostnames at the address Envoy Gateway's data plane + answers on - a load balancer's, or the node's own where the cluster has no + load balancer at all. See step 2 of [cluster-setup.md](cluster-setup.md). 2. Envoy Gateway watches Gateway API resources and programs Envoy. 3. `Bootstrap cluster` creates one shared `Gateway` in `eduide-system`, with four listeners per environment. From 81550179e82d879b7470bd27f4fc0ead84df8599 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Thu, 24 Sep 2026 20:57:32 +0200 Subject: [PATCH 3/4] fix: pin the eduide data plane to one replica, and correct step 2 Review feedback, and one of the findings caught a false claim of mine. The guide said "the chart defaults it to 1" about the Envoy Deployment's replica count. No released chart does - that default is still in review as EduIDE-Helm#40 - so the advice to state the replica count pointed at a manifest that did not state it, and parma stayed one scale-to-zero away from the outage it had just had. `clusters/eduide.yaml` now pins `replicas: 1` explicitly, which protects the cluster on the chart it actually runs and still wins over the chart default once that ships. Options (b) and (c) also need `create: true` inside `spec.envoyProxy`, not just the block: the chart defaults `envoyProxy.create` to false, so a block without it renders no EnvoyProxy while `gatewayClass.create` still emits a `parametersRef` naming one. The verify snippet hard-coded the student cluster's hostname, and does not apply as written to a ClusterIP data plane, where the Gateway's address is never what DNS publishes. The DNS table still said Bonn and Mannheim did not resolve; both have pointed at parma since it came up. Co-Authored-By: Claude Opus 5 --- clusters/eduide.yaml | 6 ++++++ docs/cluster-setup.md | 18 ++++++++++++------ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/clusters/eduide.yaml b/clusters/eduide.yaml index 7b6f697..64283db 100644 --- a/clusters/eduide.yaml +++ b/clusters/eduide.yaml @@ -62,6 +62,12 @@ spec: # and the process never touches a privileged port. useListenerPortAsContainerPort: false envoyDeployment: + # Envoy Gateway reconciles this field only while it is stated here. + # Left out, the controller writes it once when it creates the + # Deployment and never looks again, so a manual scale to zero is + # permanent - which is how both installations went dark on + # 2026-09-23, with every pod healthy and nothing listening on :443. + replicas: 1 patch: type: StrategicMerge value: diff --git a/docs/cluster-setup.md b/docs/cluster-setup.md index 2f568ba..e4b8a1e 100644 --- a/docs/cluster-setup.md +++ b/docs/cluster-setup.md @@ -129,7 +129,10 @@ fails with `cannot bind '0.0.0.0:80': Permission denied` while the pod reports `Running`. `bootstrap-cluster.yml` passes both from the cluster manifest, so options (b) and -(c) are `spec.gatewayClass.create: true` plus a `spec.envoyProxy` block. +(c) are `spec.gatewayClass.create: true` plus a `spec.envoyProxy` block that +**sets `create: true` itself**. The chart defaults `envoyProxy.create` to false, +so a block without it renders no EnvoyProxy at all while the GatewayClass still +gets a `parametersRef` naming one - a dangling reference, and no data plane. `envoyProxy.name` is the name of the EnvoyProxy resource itself; for (b) the MetalLB pool goes inside it, under `spec.provider.kubernetes.envoyService.annotations`. That is exactly what @@ -140,8 +143,9 @@ MetalLB pool goes inside it, under Gateway reconciles the Envoy Deployment's replicas only while the EnvoyProxy names them; left unset it writes the field once and never looks again, so one `kubectl scale --replicas=0` takes every Gateway on the class down until -somebody scales it back by hand. The chart defaults it to 1 - do not override it -with nothing. +somebody scales it back by hand. That is what took Bonn and Mannheim off the air +on 2026-09-23. Put `replicas` in the `envoyDeployment` block, as +`clusters/eduide.yaml` now does. `clusters/tum-production.yaml` carries `spec.loadBalancerIP: 131.159.88.82`. **Nothing reads it.** It records the intent; it does not enforce it. @@ -151,10 +155,12 @@ Whichever option is chosen, verify it after bootstrap: ```bash kubectl -n eduide-system get gateway theia-shared-gateway \ -o jsonpath='{.status.addresses[*].value}{"\n"}' -dig +short eduide.student.k8s.aet.cit.tum.de A +dig +short A ``` -Those two must end at the same address. +Those two must end at the same address. On a cluster serving option (c) the +Gateway's address is the Service's ClusterIP, which DNS never publishes - check +the node's own address instead, and that something answers on `:443` there. ## Step 3: the ACME issuer @@ -215,7 +221,7 @@ Current state: |---|---| | `*.eduide.student.k8s.aet.cit.tum.de` | `131.159.88.14` | | `eduide.artemis.cit.tum.de` | `131.159.88.82` | -| `bonn.eduide.aet.cit.tum.de`, `mannheim.…` | **not yet** | +| `bonn.eduide.aet.cit.tum.de`, `mannheim.…` | `131.159.88.106` (parma itself) | ## Step 5: the GitHub Environment From 69fbd657de996b5ab4046b89dc195b9de06ea8bf Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Thu, 24 Sep 2026 21:10:48 +0200 Subject: [PATCH 4/4] docs: the replica requirement applies to options (b) and (c) only Option (a) joins a merged gateway whose EnvoyProxy belongs to another team, so there is no envoyDeployment block in this repository to put replicas in. The instruction said "whichever option you pick", which cannot be followed there. Scoped to (b) and (c), with a paragraph saying where the exposure goes under (a): it moves to the owner of the shared data plane, and scaling that to zero takes EduIDE down along with everything else on it. Co-Authored-By: Claude Opus 5 --- docs/cluster-setup.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/cluster-setup.md b/docs/cluster-setup.md index e4b8a1e..958f9e0 100644 --- a/docs/cluster-setup.md +++ b/docs/cluster-setup.md @@ -139,13 +139,18 @@ MetalLB pool goes inside it, under `tum-production` does. See [envoy-gateway-setup.md](envoy-gateway-setup.md) for the MetalLB details. -**State the data plane's replica count, whichever option you pick.** Envoy -Gateway reconciles the Envoy Deployment's replicas only while the EnvoyProxy -names them; left unset it writes the field once and never looks again, so one +**On (b) and (c), state the data plane's replica count.** Envoy Gateway +reconciles the Envoy Deployment's replicas only while the EnvoyProxy names them; +left unset it writes the field once and never looks again, so one `kubectl scale --replicas=0` takes every Gateway on the class down until somebody scales it back by hand. That is what took Bonn and Mannheim off the air on 2026-09-23. Put `replicas` in the `envoyDeployment` block, as -`clusters/eduide.yaml` now does. +`clusters/eduide.yaml` does. + +Option (a) has no such block here, because the EnvoyProxy belongs to whoever +owns the merged gateway. The exposure does not go away - it moves. Ask them +whether their EnvoyProxy states its replicas, and remember that scaling that +data plane to zero takes EduIDE down with everything else sharing it. `clusters/tum-production.yaml` carries `spec.loadBalancerIP: 131.159.88.82`. **Nothing reads it.** It records the intent; it does not enforce it.