From c2bfaf1fbe91583b9dd2cb90da4f1b09e59e78b2 Mon Sep 17 00:00:00 2001 From: Matthias Linhuber Date: Fri, 28 Aug 2026 21:31:33 +0200 Subject: [PATCH] fix: make EduIDEWarmPoolEmpty detect a warm pool that is gone The rule could only see a warm pool that was unhealthy, never one that had been deleted - which is the state that actually breaks session launches. kube-state-metrics emits nothing for a Deployment that does not exist, so when Mannheim's ten instances were removed, both sides of `desired > 0 and available == 0` evaluated to empty and the rule stayed silent. Sessions failed for six minutes with "Failed to complete session setup" and nothing alerted. Replayed against the recorded window, the old expression returns no series at any point during the outage. The left-hand side now uses max_over_time over a 6h window, so a namespace stays in the expression after its instance series disappear and a vanished pool is compared against a missing one. An environment that has never run a warm pool has no history and still cannot trigger it. `for` drops from 10m to 5m. The outage lasted six and a half minutes, so even with the corrected expression the old threshold would not have paged. Replayed with both changes, it is in alert state 21:19 to 21:25 continuously and would have fired. Both durations are values now. The description also says what to do, since the recovery is not obvious: the operator hands out warm instances by name and does not recreate missing ones, so the pool comes back with a rollout restart of the operator. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019qeiQRFu8xAMRYWPdZewjG --- charts/eduide-cluster/Chart.yaml | 2 +- charts/eduide-cluster/README.md | 6 ++- .../templates/monitoring/prometheusrule.yaml | 50 ++++++++++++++----- charts/eduide-cluster/values.yaml | 17 +++++++ 4 files changed, 60 insertions(+), 15 deletions(-) diff --git a/charts/eduide-cluster/Chart.yaml b/charts/eduide-cluster/Chart.yaml index 3eb5336..0ab333f 100644 --- a/charts/eduide-cluster/Chart.yaml +++ b/charts/eduide-cluster/Chart.yaml @@ -4,5 +4,5 @@ description: | Cluster-scoped half of EduIDE: CRDs, the conversion webhook, ClusterRoles and cert-manager issuers. Install once per cluster, before any eduide release. type: application -version: 2.2.1 +version: 2.2.2 appVersion: "1.2.0" diff --git a/charts/eduide-cluster/README.md b/charts/eduide-cluster/README.md index 6e4b6fd..1c1f450 100644 --- a/charts/eduide-cluster/README.md +++ b/charts/eduide-cluster/README.md @@ -1,6 +1,6 @@ # eduide-cluster -![Version: 2.2.1](https://img.shields.io/badge/Version-2.2.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.2.0](https://img.shields.io/badge/AppVersion-1.2.0-informational?style=flat-square) +![Version: 2.2.2](https://img.shields.io/badge/Version-2.2.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.2.0](https://img.shields.io/badge/AppVersion-1.2.0-informational?style=flat-square) Cluster-scoped half of EduIDE: CRDs, the conversion webhook, ClusterRoles and cert-manager issuers. Install once per cluster, before any eduide release. @@ -50,7 +50,7 @@ cert-manager issuers. Install once per cluster, before any eduide release. | managedCertificates.enabled | bool | `false` | | | managedCertificates.issuerRef.kind | string | `"ClusterIssuer"` | | | managedCertificates.issuerRef.name | string | `"letsencrypt-prod"` | | -| monitoring | object | `{"alerting":{"channels":[],"enabled":false,"grafanaUrl":"","groupInterval":"5m","groupWait":"30s","minSeverity":"warning","namespace":"eduide-system","repeatInterval":"4h","runbookUrl":"https://eduide.github.io/Docs/admins/operations","thresholds":{"certExpiryDays":21,"componentRestarts":3,"sessionCrashPerHour":5,"sessionOOMPerHour":3,"startupSeconds":90,"volumePercent":85},"webhookSecret":{"create":false,"data":{},"name":"eduide-alert-webhooks"}},"certManager":{"enabled":false,"namespace":"cert-manager","portName":"http-metrics","selectorLabels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/name":"cert-manager"}},"dashboardNamespace":"cattle-dashboards","enabled":false,"namespace":"cattle-monitoring-system","targetNamespaces":[]}` | ------------------------------------------------------------------------ | +| monitoring | object | `{"alerting":{"channels":[],"enabled":false,"grafanaUrl":"","groupInterval":"5m","groupWait":"30s","minSeverity":"warning","namespace":"eduide-system","repeatInterval":"4h","runbookUrl":"https://eduide.github.io/Docs/admins/operations","thresholds":{"certExpiryDays":21,"componentRestarts":3,"sessionCrashPerHour":5,"sessionOOMPerHour":3,"startupSeconds":90,"volumePercent":85,"warmPoolFor":"5m","warmPoolHistory":"6h"},"webhookSecret":{"create":false,"data":{},"name":"eduide-alert-webhooks"}},"certManager":{"enabled":false,"namespace":"cert-manager","portName":"http-metrics","selectorLabels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/name":"cert-manager"}},"dashboardNamespace":"cattle-dashboards","enabled":false,"namespace":"cattle-monitoring-system","targetNamespaces":[]}` | ------------------------------------------------------------------------ | | monitoring.alerting.channels | list | `[]` | Where to send alerts. Each entry needs `name`, `type` (`slack` or `discord`) and `secretKey`; Slack entries may also set `channel`. An empty list with alerting enabled means alerts fire but notify nobody, so the chart fails the render instead. A channel may set `environments: [ns, ...]` to receive only that installation's alerts. One cluster can host installations belonging to different people - Bonn and Mannheim share a cluster and each has its own Discord - and without scoping, both would see the other's incidents. Anything no scoped channel claims goes to **every** channel. That is how cluster-scoped alerts still get out: a certificate expiring or the conversion webhook failing is not about any one tenant's namespace, and dropping it for failing to match a tenant route would lose exactly the alerts that matter most. channels: - name: mannheim type: discord secretKey: discord-mannheim environments: [eduide-mannheim] - name: platform type: slack secretKey: slack-platform channel: "#eduide-alerts" | | monitoring.alerting.enabled | bool | `false` | Create the PrometheusRule and the AlertmanagerConfig. | | monitoring.alerting.grafanaUrl | string | `""` | Base URL of the Grafana that serves the EduIDE dashboards, with no trailing slash. Used to deep-link a notification straight to the affected session. Left empty, notifications carry no dashboard link. | @@ -66,6 +66,8 @@ cert-manager issuers. Install once per cluster, before any eduide release. | monitoring.alerting.thresholds.sessionOOMPerHour | int | `3` | Session OOM kills per hour in one environment before alerting. | | monitoring.alerting.thresholds.startupSeconds | int | `90` | p95 session startup, in seconds, that counts as too slow. | | monitoring.alerting.thresholds.volumePercent | int | `85` | Workspace volume fill percentage that counts as nearly full. | +| monitoring.alerting.thresholds.warmPoolFor | string | `"5m"` | How long an environment's warm pool must be empty before alerting. Shorter than the other component alerts on purpose. An empty warm pool is immediately visible to students - if the instance deployments are gone rather than unhealthy, session launches fail outright - so ten minutes is long enough for a real outage to pass unnoticed. Long enough, though, that recycling the pool during an image bump does not page. | +| monitoring.alerting.thresholds.warmPoolHistory | string | `"6h"` | How far back to look for evidence that an environment keeps a warm pool at all. kube-state-metrics emits nothing for a Deployment that does not exist, so a deleted warm pool leaves no series to compare against. Looking back over this window keeps the environment in the expression after its instances vanish. An environment that has genuinely stopped running a warm pool stops alerting once this window passes. | | monitoring.alerting.webhookSecret.create | bool | `false` | Create the Secret from `data` below. Off means the Secret already exists and is referenced by name only. | | monitoring.alerting.webhookSecret.data | object | `{}` | Base64-encoded webhook URLs, keyed by the `secretKey` a channel names. Supplied by the workflow, not committed. | | monitoring.alerting.webhookSecret.name | string | `"eduide-alert-webhooks"` | Name of the Secret channels read their URL from. | diff --git a/charts/eduide-cluster/templates/monitoring/prometheusrule.yaml b/charts/eduide-cluster/templates/monitoring/prometheusrule.yaml index 8f4b736..41c4285 100644 --- a/charts/eduide-cluster/templates/monitoring/prometheusrule.yaml +++ b/charts/eduide-cluster/templates/monitoring/prometheusrule.yaml @@ -134,26 +134,52 @@ spec: environment. Check the pod is running and that /q/metrics still answers. runbook_url: {{ $runbook }} + {{- /* + Anchored on what the environment *used to* run, not on what it runs now. + + The obvious form - desired > 0 AND available == 0 - cannot see the case + that matters most. kube-state-metrics emits nothing at all for a + Deployment that does not exist, so when a warm pool is deleted rather + than merely broken, both sides of the expression are empty and the alert + has nothing to fire on. That happened on Mannheim: the pool was gone for + six minutes, sessions failed with "Failed to complete session setup", + and this rule stayed silent throughout. + + `max_over_time` over the last {{ $t.warmPoolHistory }} keeps the namespace in the left-hand + vector even after its series disappear, so a vanished pool is still + compared against a missing one. An environment that never ran a warm + pool has no history and so can never trigger this. + */}} - alert: EduIDEWarmPoolEmpty expr: |- - (sum by (namespace) (kube_deployment_status_replicas{namespace=~"{{ $ns }}", deployment=~"instance-.*"}) > 0) - and - (sum by (namespace) (kube_deployment_status_replicas_available{namespace=~"{{ $ns }}", deployment=~"instance-.*"}) == 0) - for: 10m + ( + sum by (namespace) ( + max_over_time(kube_deployment_status_replicas{namespace=~"{{ $ns }}", deployment=~"instance-.*"}[{{ $t.warmPoolHistory }}]) + ) > 0 + ) + unless + ( + sum by (namespace) (kube_deployment_status_replicas_available{namespace=~"{{ $ns }}", deployment=~"instance-.*"}) > 0 + ) + for: {{ $t.warmPoolFor }} labels: severity: critical {{- include "eduide.alertRoutingLabels" . | nindent 12 }} annotations: summary: 'The EduIDE warm pool is empty in {{ "{{" }} $labels.namespace {{ "}}" }}' description: >- - Every pre-warmed instance in {{ "{{" }} $labels.namespace {{ "}}" }} is unavailable, while the - environment is configured to keep some. Sessions still start, but each - student now waits for a cold start - pulling the image and booting Theia - - instead of being handed a running IDE. - This is expressed against desired replicas, so an environment configured - with no warm pool cannot trigger it. Common causes are a node that cannot - pull the image, insufficient cluster capacity, or every instance having - just been claimed at once. + Every pre-warmed instance in {{ "{{" }} $labels.namespace {{ "}}" }} is gone or unavailable, + while this environment normally keeps some. + If the instances are merely unavailable, sessions still start and each + student waits for a cold start instead of being handed a running IDE. If + the instance deployments have been deleted outright, sessions fail + completely with "Failed to complete session setup", because the operator + hands out warm instances by name and does not recreate missing ones - a + `kubectl rollout restart deploy/operator-deployment` in that namespace + rebuilds the pool. + Common causes are a node that cannot pull the image, insufficient cluster + capacity, every instance having just been claimed at once, or an image + bump that recycled the pool. `kubectl -n {{ "{{" }} $labels.namespace {{ "}}" }} get pods | grep instance-` is the first look. runbook_url: {{ $runbook }} diff --git a/charts/eduide-cluster/values.yaml b/charts/eduide-cluster/values.yaml index 5676420..5844360 100644 --- a/charts/eduide-cluster/values.yaml +++ b/charts/eduide-cluster/values.yaml @@ -324,3 +324,20 @@ monitoring: volumePercent: 85 # -- Days before certificate expiry to start warning. certExpiryDays: 21 + # -- How long an environment's warm pool must be empty before alerting. + # + # Shorter than the other component alerts on purpose. An empty warm pool + # is immediately visible to students - if the instance deployments are + # gone rather than unhealthy, session launches fail outright - so ten + # minutes is long enough for a real outage to pass unnoticed. Long enough, + # though, that recycling the pool during an image bump does not page. + warmPoolFor: 5m + # -- How far back to look for evidence that an environment keeps a warm + # pool at all. + # + # kube-state-metrics emits nothing for a Deployment that does not exist, + # so a deleted warm pool leaves no series to compare against. Looking back + # over this window keeps the environment in the expression after its + # instances vanish. An environment that has genuinely stopped running a + # warm pool stops alerting once this window passes. + warmPoolHistory: 6h