Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ An environment is one namespace on one cluster.
`environments/`, the GitHub Environment and the landing host are the same
string, so there is nothing to map and nothing to keep in sync.

The `eduide` cluster is not provisioned yet.
The `eduide` cluster is `parma.aet.cit.tum.de`, a single node k3s serving Bonn
and Mannheim since 2026-08-28. It has no load balancer, so its data plane
binds the node's ports with `hostPort`; `clusters/eduide.yaml` says why.

The charts live in **EduIDE-Helm** and are pulled from
`oci://ghcr.io/eduide/charts`. Chart templates are not edited here.
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,11 @@ An environment is one namespace on one cluster.
`environments/`, the GitHub Environment and the landing host are the same
string, so there is nothing to map and nothing to keep in sync.

The `eduide` cluster is **not provisioned yet**.
The `eduide` cluster is `parma.aet.cit.tum.de` (131.159.88.106), a single
node k3s serving Bonn and Mannheim since 2026-08-28. It has **no load
balancer** - the Envoy data plane binds the node's own `:80` and `:443` with
`hostPort`. `clusters/eduide.yaml` records why, and why the obvious
alternative does not work.

Namespaces stay short - `eduide-test1`, `eduide-tum-production` - because a
Kubernetes namespace cannot contain dots. `spec.namespace` in each `env.yaml`
Expand Down
6 changes: 6 additions & 0 deletions clusters/eduide.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,12 @@ spec:
# and the process never touches a privileged port.
useListenerPortAsContainerPort: false
envoyDeployment:
# Envoy Gateway reconciles this field only while it is stated here.
# Left out, the controller writes it once when it creates the
# Deployment and never looks again, so a manual scale to zero is
# permanent - which is how both installations went dark on
# 2026-09-23, with every pod healthy and nothing listening on :443.
replicas: 1
Comment thread
Mtze marked this conversation as resolved.
patch:
type: StrategicMerge
value:
Expand Down
57 changes: 45 additions & 12 deletions docs/cluster-setup.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Preparing a cluster

What has to exist on a TUM cluster before EduIDE can be deployed to it, which
parts `Bootstrap cluster` does for you, and which parts it does not.
What has to exist on a cluster before EduIDE can be deployed to it, which parts
`Bootstrap cluster` does for you, and which parts it does not. Written around
the TUM clusters, but the `eduide` cluster is not one of them and the places
that differ say so.

Read this end to end before bootstrapping a cluster for the first time. Two of
the manual steps below cannot be discovered by trying: they produce a cluster
Expand All @@ -12,7 +14,7 @@ that reports itself healthy and serves nothing.
| | Who does it |
|---|---|
| Gateway API CRDs, Envoy Gateway, cert-manager, storage | **you**, once per cluster |
| A GatewayClass whose load balancer address matches DNS | **you** |
| A data plane answering on the address DNS publishes - a load balancer's, or the node's | **you** |
| The ACME `ClusterIssuer` | `Bootstrap cluster`, from `spec.acmeEmail` |
| The GatewayClass and its EnvoyProxy | `Bootstrap cluster`, from `spec.gatewayClass` and `spec.envoyProxy` |
| Redirects from hostnames you used to serve | `Bootstrap cluster`, from `spec.redirects` |
Expand Down Expand Up @@ -76,7 +78,7 @@ raising with whoever owns the cluster. The deploy sidesteps it by always naming
every node, so the default set of eight is roughly 20 GB per node, once. Budget
it before the first bootstrap rather than discovering it as disk pressure.

## Step 2: decide which load balancer address serves EduIDE
## Step 2: decide which address serves EduIDE

This is the step that produces a healthy-looking cluster that serves nothing,
so do it before anything else.
Expand All @@ -102,7 +104,7 @@ So a Gateway created with `gatewayClassName: envoy` on that cluster comes up
`Programmed=True`, is served on `131.159.88.15`, and is unreachable at every
name DNS actually publishes. Nothing reports an error.

There are two ways out, and it is a decision, not a default:
There are three ways out, and it is a decision, not a default:

**(a) Join the existing merged gateway.** Ask for the EduIDE DNS names to point
at `131.159.88.15` instead. Nothing in this repository changes. EduIDE then
Expand All @@ -114,13 +116,42 @@ own `EnvoyProxy` pinned to pool `lb3`, and set `gatewayClassName` in
`clusters/tum-student.yaml` to match. EduIDE then has its own data plane on
`131.159.88.14`, which is what DNS already says.

`bootstrap-cluster.yml` passes both from the cluster manifest, so option (b) is
`spec.gatewayClass.create: true` plus a `spec.envoyProxy` block. `envoyProxy.name`
is the name of the EnvoyProxy resource itself; the MetalLB pool goes inside it,
under `spec.provider.kubernetes.envoyService.annotations`. That is exactly what
**(c) There is no load balancer at all.** A single node cluster outside TUM may
have neither MetalLB nor servicelb, and then a Service of type `LoadBalancer`
sits `Pending` for ever while DNS publishes the node's own address. Bind the two
Comment thread
coderabbitai[bot] marked this conversation as resolved.
public ports on the node instead: `envoyService.type: ClusterIP`, and a
`StrategicMerge` patch on the Envoy Deployment giving its container `hostPort: 80`
and `hostPort: 443`. The `eduide` cluster does this, and
Comment thread
Copilot marked this conversation as resolved.
`clusters/eduide.yaml` carries the whole story - including why hostNetwork plus
`useListenerPortAsContainerPort` is the wrong answer: Envoy runs as non-root,
Kubernetes cannot grant `NET_BIND_SERVICE` effectively, and every listener then
fails with `cannot bind '0.0.0.0:80': Permission denied` while the pod reports
`Running`.

`bootstrap-cluster.yml` passes both from the cluster manifest, so options (b) and
(c) are `spec.gatewayClass.create: true` plus a `spec.envoyProxy` block that
**sets `create: true` itself**. The chart defaults `envoyProxy.create` to false,
so a block without it renders no EnvoyProxy at all while the GatewayClass still
gets a `parametersRef` naming one - a dangling reference, and no data plane.
`envoyProxy.name` is the name of the EnvoyProxy resource itself; for (b) the
MetalLB pool goes inside it, under
`spec.provider.kubernetes.envoyService.annotations`. That is exactly what
`tum-production` does. See
[envoy-gateway-setup.md](envoy-gateway-setup.md) for the MetalLB details.

**On (b) and (c), state the data plane's replica count.** Envoy Gateway
reconciles the Envoy Deployment's replicas only while the EnvoyProxy names them;
left unset it writes the field once and never looks again, so one
`kubectl scale --replicas=0` takes every Gateway on the class down until
somebody scales it back by hand. That is what took Bonn and Mannheim off the air
on 2026-09-23. Put `replicas` in the `envoyDeployment` block, as
`clusters/eduide.yaml` does.

Option (a) has no such block here, because the EnvoyProxy belongs to whoever
owns the merged gateway. The exposure does not go away - it moves. Ask them
whether their EnvoyProxy states its replicas, and remember that scaling that
data plane to zero takes EduIDE down with everything else sharing it.

`clusters/tum-production.yaml` carries `spec.loadBalancerIP: 131.159.88.82`.
**Nothing reads it.** It records the intent; it does not enforce it.

Expand All @@ -129,10 +160,12 @@ Whichever option is chosen, verify it after bootstrap:
```bash
kubectl -n eduide-system get gateway theia-shared-gateway \
-o jsonpath='{.status.addresses[*].value}{"\n"}'
dig +short eduide.student.k8s.aet.cit.tum.de A
dig +short <the landing host of an environment on this cluster> A
```

Those two must end at the same address.
Those two must end at the same address. On a cluster serving option (c) the
Gateway's address is the Service's ClusterIP, which DNS never publishes - check
the node's own address instead, and that something answers on `:443` there.

## Step 3: the ACME issuer

Expand Down Expand Up @@ -193,7 +226,7 @@ Current state:
|---|---|
| `*.eduide.student.k8s.aet.cit.tum.de` | `131.159.88.14` |
| `eduide.artemis.cit.tum.de` | `131.159.88.82` |
| `bonn.eduide.aet.cit.tum.de`, `mannheim.…` | **not yet** |
| `bonn.eduide.aet.cit.tum.de`, `mannheim.…` | `131.159.88.106` (parma itself) |

## Step 5: the GitHub Environment

Expand Down
8 changes: 4 additions & 4 deletions docs/environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ base sets.
|---|---|---|
| `tum-student` | `test1.…`, `test2.…`, `test3.…`, `e2e.…`, `staging.eduide.student.k8s.aet.cit.tum.de` | everything non-production |
| `tum-production` | `eduide.artemis.cit.tum.de` | TUM's own installation |
| `eduide` | `bonn.eduide.aet.cit.tum.de`, `mannheim.eduide.aet.cit.tum.de` | other universities. **Not provisioned yet.** |
| `eduide` | `bonn.eduide.aet.cit.tum.de`, `mannheim.eduide.aet.cit.tum.de` | other universities. Single node k3s (`parma`), no load balancer |

The two `eduide` ones exist as reviewable configuration before the cluster does.
Deploying one stops at the cluster identity check until that cluster has been
bootstrapped and the GitHub Environment holds a `KUBECONFIG`.
An environment can be written and reviewed before its cluster exists. Deploying
one stops at the cluster identity check until that cluster has been bootstrapped
and the GitHub Environment holds a `KUBECONFIG`.

## Hostnames

Expand Down
4 changes: 3 additions & 1 deletion docs/envoy-gateway-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ step 2 of it.

## The traffic path

1. DNS points the EduIDE hostnames at Envoy Gateway's load balancer address.
1. DNS points the EduIDE hostnames at the address Envoy Gateway's data plane
answers on - a load balancer's, or the node's own where the cluster has no
load balancer at all. See step 2 of [cluster-setup.md](cluster-setup.md).
2. Envoy Gateway watches Gateway API resources and programs Envoy.
3. `Bootstrap cluster` creates one shared `Gateway` in `eduide-system`, with
four listeners per environment.
Expand Down
Loading