From 33ca9c14ae35392d731caf49ca5dedbfa497d08f Mon Sep 17 00:00:00 2001 From: mdheller <21163552+mdheller@users.noreply.github.com> Date: Thu, 30 Jul 2026 02:03:47 -0400 Subject: [PATCH 1/3] observability: 28 alert rules in this estate cannot fire MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit KubePersistentVolumeFillingUp was loaded with health=ok, lastError=none, state=inactive, querying kubelet_volume_stats_available_bytes{job="kubelet"} against a Prometheus with no kubelet job. Zero series. It evaluated an empty set forever and read as "the volumes are fine" while the disk hit 100%. That generalises: a control that cannot fail is not a control. A first scan found 28 alerting rules in that state. Adds an hourly stdlib-Python guard that parses every alerting rule into its vector selectors and asks Prometheus whether each matched anything. "The rule returns nothing" is the wrong test — up == 0 correctly returns nothing when everything is up. The question is whether the INPUTS exist. Three causes, three verdicts, because collapsing them makes false positives: DEAD-NO-TARGET job="X" named but up{job="X"} has no series DEAD-LABEL-MISMATCH metric exists, label matchers exclude everything DORMANT target up, metric never created (lazy counters) Exemptions live in policy.json and must carry a reason, an expiry, and — where the hazard is still real — a compensating_control. Expired exemptions alert. Two entries record that they have NO compensating control yet (inode exhaustion, CPU throttling) and carry short expiries. The guard does not exempt itself. Also fixes two of this estate's own dead rules: HellGraphDown was up{job=~".*hellgraph.*"} == 0, matching zero series. Two causes: the ServiceMonitor selects `app: hellgraph` but the chart renders only app.kubernetes.io/* labels, AND hellgraph-service returns 404 on /metrics. Fixing only the selector would have produced a permanently-firing false alarm, which is not an improvement on a permanently-silent one. Now reads kube-state-metrics, which genuinely observes the deployment. CanaryGateBlind is new. The Argo Rollouts slo-gate AnalysisTemplate gates every canary on job:request_error_ratio:rate5m, which derives from http_server_request_duration_seconds_count — zero series. An SLO gate with no input is not a gate, and whether Rollouts treats the empty result as pass or error is not something to discover during a bad deploy. Built on absent(), so the missing input is itself loud. Proof it fires — a probe rule with two deliberately-dead rules and one healthy control: red checked=137 live=101 DEAD=9 both probes flagged, control not green checked=137 live=103 DEAD=7 probes cleared after fixing Two bugs the proof caught: 1. All nine findings were emitted as AlertRuleDead, and Alertmanager suppressed six of them — the default inhibit_rules drop warning when a critical shares (namespace, alertname). Six real findings swallowed by a control reporting success: this guard reproducing, in its own delivery path, the exact defect it detects. Fixed with distinct alertnames per severity. 2. The first in-cluster run exited 1 on findings, tripping backoffLimit into BackoffLimitExceeded — a permanently-red hourly Job. Findings now travel by alert; non-zero exit is reserved for the guard being broken. In-cluster run completes and delivers: checked=141 live=105 dead=7, 8 alerts delivered. It also caught its own RuleLivenessGuardFailing as DEAD-LABEL-MISMATCH before any CronJob-named Job existed, then cleared it once one did. --- deploy/argocd/rule-liveness-guard.yaml | 37 ++ .../base/prometheusrule-slos.yaml | 50 +- infra/k8s/rule-liveness-guard/README.md | 193 ++++++ .../k8s/rule-liveness-guard/base/cronjob.yaml | 75 +++ infra/k8s/rule-liveness-guard/base/guard.py | 563 ++++++++++++++++++ .../base/kustomization.yaml | 21 + .../k8s/rule-liveness-guard/base/policy.json | 123 ++++ .../base/prometheusrule-guard.yaml | 47 ++ 8 files changed, 1107 insertions(+), 2 deletions(-) create mode 100644 deploy/argocd/rule-liveness-guard.yaml create mode 100644 infra/k8s/rule-liveness-guard/README.md create mode 100644 infra/k8s/rule-liveness-guard/base/cronjob.yaml create mode 100644 infra/k8s/rule-liveness-guard/base/guard.py create mode 100644 infra/k8s/rule-liveness-guard/base/kustomization.yaml create mode 100644 infra/k8s/rule-liveness-guard/base/policy.json create mode 100644 infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml diff --git a/deploy/argocd/rule-liveness-guard.yaml b/deploy/argocd/rule-liveness-guard.yaml new file mode 100644 index 000000000..e4c6c1dd0 --- /dev/null +++ b/deploy/argocd/rule-liveness-guard.yaml @@ -0,0 +1,37 @@ +# Detects alert rules that are incapable of firing — the generalisation of the +# KubePersistentVolumeFillingUp failure, where a rule queried a metric with zero +# series and reported health=ok forever while the disk filled. +# +# Lives in deploy/argocd/, NOT infra/argocd/, for the reason documented in +# deploy/argocd/pvc-capacity-guard.yaml: the root app-of-apps only recurses +# `deploy/argocd`, so an Application under infra/argocd/ deploys nothing. For a +# guard against controls that silently do nothing, that would be the defect +# recursing one layer up. +# +# sync-wave "1": after the kube-prometheus-stack CRDs so PrometheusRule exists. +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: rule-liveness-guard + namespace: argocd + labels: + app.kubernetes.io/part-of: platform-ops + annotations: + argocd.argoproj.io/sync-wave: "1" + # Required by tools/preflight_deploy_contract.py check 5. reference: a + # replaceable meta-monitor on a digest-pinned stock python image. + socioprophet.io/tier: "reference" +spec: + project: default + destination: + server: https://kubernetes.default.svc + namespace: observability + source: + repoURL: https://github.com/SocioProphet/prophet-platform + targetRevision: main + path: infra/k8s/rule-liveness-guard/base + syncPolicy: + automated: { prune: true, selfHeal: true } + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/infra/k8s/observability/base/prometheusrule-slos.yaml b/infra/k8s/observability/base/prometheusrule-slos.yaml index f072f5fbb..ce381fa60 100644 --- a/infra/k8s/observability/base/prometheusrule-slos.yaml +++ b/infra/k8s/observability/base/prometheusrule-slos.yaml @@ -20,12 +20,28 @@ spec: labels: { severity: warning } annotations: summary: "Target {{ $labels.job }} / {{ $labels.instance }} is down" + # WAS `up{job=~".*hellgraph.*"} == 0`, which matched ZERO series and so + # could never fire while reporting health=ok — the same defect as + # KubePersistentVolumeFillingUp. Two independent causes: + # 1. the `hellgraph` ServiceMonitor selects `app: hellgraph`, but the + # Service is rendered by the shared socioprophet-service chart and + # carries only app.kubernetes.io/* labels, so it matched nothing; + # 2. hellgraph-service does not serve /metrics at all — probing + # http://hellgraph-service:8090/metrics returns HTTP 404. + # Fixing (1) alone would have produced a permanently-firing false alarm, + # which is not an improvement on a permanently-silent one. + # So this now reads kube-state-metrics, which genuinely observes the + # deployment (3 live series today, one per namespace). Giving hellgraph a + # real /metrics endpoint is app work and is NOT in this lane. - alert: HellGraphDown - expr: up{job=~".*hellgraph.*"} == 0 + expr: | + kube_deployment_status_replicas_available{deployment="hellgraph-service"} == 0 + and on (namespace, deployment) + kube_deployment_spec_replicas{deployment="hellgraph-service"} > 0 for: 2m labels: { severity: critical } annotations: - summary: "HellGraph managed graph service is down — person-graph reads/writes fail" + summary: "HellGraph managed graph service has no available replicas in {{ $labels.namespace }} — person-graph reads/writes fail" - name: socioprophet.request-slo rules: # 5xx error ratio per service — the canary gate metric. @@ -44,3 +60,33 @@ spec: labels: { severity: warning } annotations: summary: "{{ $labels.job }} 5xx ratio > 5% (SLO breach) — blocks canary promotion" + + # ── The canary gate must not be able to pass by knowing nothing. + # + # infra/k8s/rollouts/base/analysistemplate-slo.yaml gates every canary + # promotion on job:request_error_ratio:rate5m and job:request_latency_p99:5m. + # Both recording rules derive from http_server_request_duration_seconds_* + # (OTel HTTP semconv, via the collector). That source metric currently has + # ZERO series — no service emits it — so both recording rules produce + # nothing and the AnalysisTemplate queries an empty vector. + # + # An SLO gate with no input is not a gate. Whether Argo Rollouts treats the + # empty result as an error or as a pass is not something this estate should + # be discovering during a bad deploy. So the missing input is itself an + # alert, at critical, and it is deliberately built on absent() — the one + # construct whose whole purpose is to react to emptiness. + - alert: CanaryGateBlind + expr: absent(http_server_request_duration_seconds_count) + for: 15m + labels: { severity: critical } + annotations: + summary: "Canary SLO gate has no input — http_server_request_duration_seconds_count has zero series" + description: >- + job:request_error_ratio:rate5m and job:request_latency_p99:5m both + derive from this metric, and the Argo Rollouts slo-gate + AnalysisTemplate reads both. With no source series the gate cannot + evaluate the SLO it exists to enforce. Either services are not + exporting OTel HTTP metrics through the collector, or the collector + pipeline is broken. Until this clears, treat canary promotions as + ungated. + runbook: infra/k8s/rule-liveness-guard/README.md diff --git a/infra/k8s/rule-liveness-guard/README.md b/infra/k8s/rule-liveness-guard/README.md new file mode 100644 index 000000000..fcd82095d --- /dev/null +++ b/infra/k8s/rule-liveness-guard/README.md @@ -0,0 +1,193 @@ +# rule-liveness-guard — finds alert rules that cannot fire + +## The defect this generalises + +`KubePersistentVolumeFillingUp` was loaded in the `observability` Prometheus with +`health=ok`, `lastError=none`, `state=inactive`: + +``` +group: kubernetes-storage + KubePersistentVolumeFillingUp | state=inactive | health=ok | lastError=none +``` + +Its query was +`kubelet_volume_stats_available_bytes{job="kubelet"} / kubelet_volume_stats_capacity_bytes{...} < 0.03`. + +That Prometheus has no `kubelet` scrape job — GKE Autopilot's Warden denies +`nodes/proxy` cluster-wide. Both metrics resolve to **zero series**. The rule +evaluated an empty set forever and reported `inactive`, which on every dashboard +is indistinguishable from "the volumes are fine". The disk reached 100%. + +That is not one bad rule. **A control that cannot fail is not a control**, and +any rule whose inputs do not exist is in that category. The first scan of this +estate found **28 alerting rules** in that state. + +## The signature — and why the obvious test is wrong + +"The rule returns nothing" is the wrong test. `up == 0` returns nothing when +everything is up; that is a healthy rule doing its job. The distinguishing +question is whether the rule's **inputs** exist: + +``` +up == 0 -> `up` has 31 series -> LIVE +kubelet_volume_stats_available_bytes{job=…} -> selector has 0 series -> DEAD +``` + +So the guard parses every alerting rule into its **vector selectors** and asks +Prometheus (`/api/v1/series`) whether each matched anything over a lookback +window. A rule whose every selector is empty cannot fire, whatever the cluster +does. + +Rules built on `absent()` / `absent_over_time()` are exempt by construction — +reacting to emptiness is their whole purpose. + +## Three causes, three verdicts + +Collapsing them would produce false positives, so the guard separates them: + +| Verdict | Meaning | Severity | +|---|---|---| +| **DEAD-NO-TARGET** | The selector names `job="X"` and `up{job="X"}` has no series. Nothing will ever produce this metric. Structural. | critical | +| **DEAD-LABEL-MISMATCH** | The bare metric *does* exist, but the label matchers exclude every series. Almost always a real bug — a renamed label, or a selector copied from a different topology. | critical | +| **DORMANT** | The metric name has never appeared, but its target is up. Usually a counter created on first occurrence (`*_failures_total`), or a disabled collector. Needs a human decision, not an automatic verdict. | warning | + +`HellGraphDown` was caught as **DEAD-LABEL-MISMATCH**: `up{job=~".*hellgraph.*"}` +while `up` had 31 series and none matched. + +## Exemptions are declared, justified, and they expire + +Some selectors are legitimately absent — Autopilot nodes have no RAID arrays, so +`node_md_disks` will never exist. Those live in `policy.json`. Every entry needs: + +- `reason` — why it is absent +- `expires` — a date, after which it **alerts** (`AlertRuleExemptionExpired`) +- `compensating_control` — where the risk still exists, what covers it instead + +That last field is the point. `KubePersistentVolumeFillingUp` is exempted, but +only because `pvc-capacity-guard` (PR #1105) reads the same metrics from Google +Managed Prometheus, which *does* scrape kubelet. Two entries deliberately record +that they have **no** compensating control yet and carry short expiries: +`KubePersistentVolumeInodesFillingUp` (inode exhaustion is uncovered) and +`CPUThrottlingHigh` (throttling is unobserved in-cluster). + +Expiries are staggered so they do not all come due at once. An allowlist that +never expires just relocates the original problem. + +**Not exempted, on purpose:** `HellGraphDown` and `HighErrorRatio`. Both are the +estate's own rules, both are dead, both are fixable rather than acceptable. + +## It does not exempt itself + +The guard's own alerts (`RuleLivenessGuardNotRunning`, `RuleLivenessGuardFailing`) +are ordinary rules in the same Prometheus and are scanned like everything else. +A scanner that skips itself is how a ratchet ends up counting its own allowlist. + +## Failure is loud + +Every abnormal outcome raises or alerts; nothing is skipped quietly. + +- An expression it cannot parse → `AlertRuleUnscannable`, not a silent skip. +- Prometheus reporting **zero** rules → hard `RuntimeError`. An empty corpus is a + broken config, never an all-clear. +- A failed `/api/v1/series` lookup → counted as unscannable, not as "live". +- Alertmanager unreachable → `failed` receipt, exit 2. + +**Findings do not fail the Job.** The first in-cluster run exited 1 because it +found dead rules, tripped `backoffLimit`, and landed in `BackoffLimitExceeded` — +which would have meant a permanently-red hourly Job for as long as any dead rule +existed, the kind everyone learns to ignore, and would have made +`RuleLivenessGuardFailing` meaningless. Findings travel by **alert**, which is +the channel built for them and which is now actually delivered. A non-zero exit +is reserved for the guard itself being broken. + +## Receipts + +One `EvidenceReceipt` (`contracts/EvidenceReceipt.v0.1.json`) per scan, as a JSON +line on stdout → `fluentbit-gke` → **Google Cloud Logging**. + +```json +{"action":"rule-liveness-scan","status":"partial","service_ref":"infra/k8s/rule-liveness-guard", + "hash":"96326a03…","hash_algo":"sha256","subject_ref":"prometheus/observability", + "metrics":{"checked":134,"live":100,"dead":7,"exempted":21,"expired":0, + "unscannable":0,"absence_based":5,"selectors_tested":197}} +``` + +`status` is `succeeded` on a clean scan and `partial` when there are findings. +The `hash` covers the canonical finding set, so two scans with identical findings +produce an identical receipt id. + +Read them with: + +``` +gcloud logging read 'resource.labels.container_name="guard" + AND jsonPayload.action="rule-liveness-scan"' --limit 5 +``` + +## Proof it fires + +A probe `PrometheusRule` was applied with three rules: two deliberately dead by +different causes, and one healthy control that returns empty simply because +nothing is down. + +**Red:** + +``` +checked=137 live=101 DEAD=9 exempt=21 expired=0 unscannable=0 + DEAD-NO-TARGET wave1.probe/ProbeDeadNoTarget <- no scrape target with job='no-such-exporter' + DEAD-LABEL-MISMATCH wave1.probe/ProbeDeadLabelMismatch <- metric up has 31 series but the label matchers exclude all of them + DEAD-LABEL-MISMATCH socioprophet.availability/HellGraphDown + DORMANT socioprophet.request-slo/HighErrorRatio + … 5 more +``` + +`ProbeLive` was **not** flagged — the control against false positives held. + +**Green**, after pointing both probe rules at metrics that exist: + +``` +checked=137 live=103 DEAD=7 exempt=21 expired=0 unscannable=0 + (no wave1.probe findings) +``` + +`live` 101 → 103, findings 9 → 7. The probe rule was then deleted. + +### The proof caught a bug in this guard + +The first non-dry run emitted all nine findings as `AlertRuleDead`, and +Alertmanager **suppressed six of them**: + +``` +critical wave1.probe/ProbeDeadNoTarget state=active inhibitedBy=0 +warning kube-state-metrics/KubeStateMetr… state=suppressed inhibitedBy=1 +warning socioprophet.request-slo/HighErr… state=suppressed inhibitedBy=1 +… +``` + +The stack's default `inhibit_rules` drop `severity=warning` when a +`severity=critical` shares the same `(namespace, alertname)` — and every finding +carries `namespace=observability`. Six real findings were swallowed by a control +that reported success: this guard reproducing, in its own delivery path, exactly +the defect it exists to detect. + +Fixed by emitting **distinct alertnames per severity** — `AlertRuleDead` +(critical) and `AlertRuleDormant` (warning) — which cannot inhibit each other. +Re-run confirms all nine `state=active`. Do not merge those names back together. + +## Runbook + +**`AlertRuleDead` (critical)** — a rule is structurally incapable of firing. +Decide between three actions, in preference order: + +1. **Fix the input.** Wrong label, wrong job, a ServiceMonitor that matches + nothing. This is the usual answer for `DEAD-LABEL-MISMATCH`. +2. **Delete the rule.** If the hazard is not real here, a rule that cannot fire + is worse than no rule — it reads as coverage. +3. **Declare it** in `policy.json` with a reason, an expiry, and a compensating + control. Only if the hazard is real but is covered elsewhere. + +**`AlertRuleDormant` (warning)** — the target is up but the metric has never +appeared. Confirm whether the metric is created lazily (fine, leave it) or the +collector is disabled (fix or declare it). + +**`AlertRuleUnscannable`** — the guard could not parse an expression. That is a +gap in the extractor, not in the rule. File it; do not silence it. diff --git a/infra/k8s/rule-liveness-guard/base/cronjob.yaml b/infra/k8s/rule-liveness-guard/base/cronjob.yaml new file mode 100644 index 000000000..f6269fe7b --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/cronjob.yaml @@ -0,0 +1,75 @@ +apiVersion: batch/v1 +kind: CronJob +metadata: + name: rule-liveness-guard + namespace: observability +spec: + # Hourly. Rule liveness changes when someone edits a PrometheusRule, renames a + # label, or a scrape target disappears — none of which is sub-hourly. Hourly is + # also frequent enough that a regression is caught the same working day. + schedule: "17 * * * *" + concurrencyPolicy: Forbid + startingDeadlineSeconds: 600 + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 5 + jobTemplate: + spec: + backoffLimit: 2 + template: + metadata: + labels: + app: rule-liveness-guard + spec: + restartPolicy: OnFailure + securityContext: + runAsNonRoot: true + runAsUser: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: guard + # Pulled DIRECTLY from docker.io, not through + # registry.socioprophet.ai — same rule as pvc-capacity-guard and + # alert-delivery. A monitor whose own image comes from the thing it + # monitors goes blind exactly when it is needed. Stdlib only, so a + # bare python image needs no pip install. + image: python:3.12-slim@sha256:57cd7c3a7a273101a6485ba99423ee568157882804b1124b4dd04266317710de + command: ["python3", "/opt/guard/guard.py"] + env: + - name: PROM_URL + value: http://kube-prometheus-stack-prometheus.observability.svc:9090 + - name: ALERTMANAGER_URL + value: http://kube-prometheus-stack-alertmanager.observability.svc:9093 + - name: POLICY_PATH + value: /etc/guard/policy.json + # 7 days. Long enough that a metric which only appears under load + # is not called dead; short enough that a rule broken last week + # is still caught this week. + - name: LOOKBACK_DAYS + value: "7" + volumeMounts: + - name: guard + mountPath: /opt/guard/guard.py + subPath: guard.py + readOnly: true + - name: guard + mountPath: /etc/guard/policy.json + subPath: policy.json + readOnly: true + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: 500m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumes: + - name: guard + configMap: + name: rule-liveness-guard + defaultMode: 0555 diff --git a/infra/k8s/rule-liveness-guard/base/guard.py b/infra/k8s/rule-liveness-guard/base/guard.py new file mode 100644 index 000000000..5b0a31f60 --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/guard.py @@ -0,0 +1,563 @@ +#!/usr/bin/env python3 +"""Rule-liveness guard: finds alert rules that are incapable of firing. + +The defect +---------- +`KubePersistentVolumeFillingUp` was loaded in Prometheus with `health=ok`, +`lastError=none`, `state=inactive`. It queried +`kubelet_volume_stats_available_bytes{job="kubelet"}` against a Prometheus with +no kubelet scrape job. Both metrics resolved to zero series, so the rule +evaluated an empty set forever and reported `inactive` — which on every +dashboard is indistinguishable from "the volumes are fine". The disk hit 100%. + +This is not one bad rule. It is a whole class: any rule whose inputs do not +exist is a rule that cannot fail, and a control that cannot fail is not a +control. A first scan of this estate found 28 of them. + +The signature +------------- +"Rule returns nothing" is the WRONG test — `up == 0` correctly returns nothing +when everything is up. The distinguishing question is whether the rule's INPUTS +exist: + + up == 0 -> `up` has 14 series -> LIVE + kubelet_volume_stats_available_bytes{..} -> selector has 0 series -> DEAD + +So this guard parses each rule expression into its vector selectors and asks +Prometheus whether each selector matches any series over a lookback window. A +rule whose every selector is empty cannot fire, no matter what happens in the +cluster. + +Rules built on absent()/absent_over_time() are exempt by construction: reacting +to emptiness is their entire purpose. + +Exemptions are declared, justified, and they expire +--------------------------------------------------- +Some selectors are legitimately absent — this cluster has no RAID arrays, so +`node_md_disks` will never exist. Those are declared in policy.json. Every +exemption must carry a `reason`, an `expires` date, and — where the underlying +risk is still real — a `compensating_control` naming what covers it instead. +An expired exemption ALERTS. An undeclared dead rule ALERTS. That keeps the +allowlist from quietly becoming the place dead controls go to be forgotten. + +This guard does not exempt itself: its own alerts are ordinary rules and are +scanned like everything else. + +Failure is loud +--------------- +Every abnormal outcome raises or alerts — an unparseable expression, an empty +rule list, an unreachable Prometheus, an unreachable Alertmanager. Silently +skipping what it cannot evaluate is the exact defect it exists to find. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime, timezone + +PROM = os.environ.get("PROM_URL", "http://kube-prometheus-stack-prometheus.observability.svc:9090") +ALERTMANAGER = os.environ.get( + "ALERTMANAGER_URL", "http://kube-prometheus-stack-alertmanager.observability.svc:9093" +) +POLICY_PATH = os.environ.get("POLICY_PATH", "/etc/guard/policy.json") +LOOKBACK_DAYS = int(os.environ.get("LOOKBACK_DAYS", "7")) +DRY_RUN = os.environ.get("DRY_RUN", "").lower() in ("1", "true", "yes") +SERVICE_REF = "infra/k8s/rule-liveness-guard" +VERSION = "0.1" + +# --------------------------------------------------------------------------- # +# PromQL vector-selector extraction +# --------------------------------------------------------------------------- # +FUNCTIONS = { + "abs", "absent", "absent_over_time", "ceil", "changes", "clamp", "clamp_max", + "clamp_min", "day_of_month", "day_of_week", "day_of_year", "days_in_month", + "delta", "deriv", "exp", "floor", "histogram_avg", "histogram_count", + "histogram_fraction", "histogram_quantile", "histogram_stddev", + "histogram_stdvar", "histogram_sum", "holt_winters", + "double_exponential_smoothing", "hour", "idelta", "increase", "info", + "irate", "label_join", "label_replace", "ln", "log2", "log10", + "mad_over_time", "minute", "month", "pi", "predict_linear", "rate", + "resets", "round", "scalar", "sgn", "sort", "sort_desc", "sort_by_label", + "sort_by_label_desc", "sqrt", "time", "timestamp", "vector", "year", + "avg_over_time", "min_over_time", "max_over_time", "sum_over_time", + "count_over_time", "quantile_over_time", "stddev_over_time", + "stdvar_over_time", "last_over_time", "present_over_time", + "acos", "acosh", "asin", "asinh", "atan", "atanh", "cos", "cosh", "sin", + "sinh", "tan", "tanh", "deg", "rad", +} +AGGREGATIONS = { + "sum", "min", "max", "avg", "group", "stddev", "stdvar", "count", + "count_values", "bottomk", "topk", "quantile", "limitk", "limit_ratio", +} +KEYWORDS = { + "by", "without", "on", "ignoring", "group_left", "group_right", "offset", + "bool", "and", "or", "unless", "start", "end", "atan2", "inf", "nan", +} +LABEL_LIST_KEYWORDS = {"by", "without", "on", "ignoring", "group_left", "group_right"} +ABSENCE_FUNCS = {"absent", "absent_over_time"} + +IDENT_START = set("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_:") +IDENT_CHAR = IDENT_START | set("0123456789") + + +class ParseError(Exception): + pass + + +def _skip_string(expr, i): + quote = expr[i] + i += 1 + while i < len(expr): + c = expr[i] + if c == "\\" and quote != "`": + i += 2 + continue + if c == quote: + return i + 1 + i += 1 + raise ParseError("unterminated string literal") + + +def _match(expr, i, opener, closer): + depth = 0 + while i < len(expr): + c = expr[i] + if c in "\"'`": + i = _skip_string(expr, i) + continue + if c == opener: + depth += 1 + elif c == closer: + depth -= 1 + if depth == 0: + return i + 1 + i += 1 + raise ParseError("unbalanced %s %s" % (opener, closer)) + + +def _peek(expr, i): + while i < len(expr) and expr[i] in " \t\n\r": + i += 1 + return i + + +def extract_selectors(expr): + """Return (selectors, uses_absence_func). Raises ParseError on anything odd.""" + selectors = [] + uses_absence = False + i, n = 0, len(expr) + while i < n: + c = expr[i] + if c == "#": + while i < n and expr[i] != "\n": + i += 1 + continue + if c in "\"'`": + i = _skip_string(expr, i) + continue + if c == "[": # range / subquery: durations, not metrics + i = _match(expr, i, "[", "]") + continue + if c == "{": # bare {__name__="x"} selector + end = _match(expr, i, "{", "}") + selectors.append(expr[i:end]) + i = end + continue + if c in IDENT_START: + j = i + while j < n and expr[j] in IDENT_CHAR: + j += 1 + word = expr[i:j] + nxt = _peek(expr, j) + if word in ABSENCE_FUNCS: + uses_absence = True + if word in FUNCTIONS or word in AGGREGATIONS: + i = j + continue + if word in KEYWORDS: + if word in LABEL_LIST_KEYWORDS and nxt < n and expr[nxt] == "(": + i = _match(expr, nxt, "(", ")") + continue + i = j + continue + if nxt < n and expr[nxt] == "(": + # Unknown identifier used as a function. Do not guess. + raise ParseError("unknown function %r" % word) + sel, k = word, j + nk = _peek(expr, k) + if nk < n and expr[nk] == "{": + end = _match(expr, nk, "{", "}") + sel, k = word + expr[nk:end], end + selectors.append(sel) + i = k + continue + i += 1 + return selectors, uses_absence + + +# --------------------------------------------------------------------------- # +# Receipts (contracts/EvidenceReceipt.v0.1.json) +# --------------------------------------------------------------------------- # +def utc_now(): + return datetime.now(timezone.utc).replace(microsecond=0).isoformat() + + +def canonical(obj): + return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def sha256(text): + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def emit(obj): + sys.stdout.write(canonical(obj) + "\n") + sys.stdout.flush() + + +def receipt(action, status, subject_ref, body, **extra): + h = sha256(canonical(body)) + r = { + "version": VERSION, + "receipt_id": "evr-%s-%s" % (action, h[:32]), + "created_at": utc_now(), + "service_ref": SERVICE_REF, + "action": action, + "status": status, + "subject_ref": subject_ref, + "hash": h, + "hash_algo": "sha256", + } + r.update(extra) + return r + + +# --------------------------------------------------------------------------- # +# Prometheus / Alertmanager +# --------------------------------------------------------------------------- # +def http_json(url, timeout=45): + with urllib.request.urlopen(url, timeout=timeout) as resp: + return json.loads(resp.read().decode("utf-8")) + + +def prom_rules(): + d = http_json(PROM + "/api/v1/rules") + if d.get("status") != "success": + raise RuntimeError("prometheus /rules returned status=%s" % d.get("status")) + groups = d["data"]["groups"] + total = sum(len(g["rules"]) for g in groups) + # An empty rule corpus is a broken scrape/config, never an all-clear. + if total == 0: + raise RuntimeError("prometheus reports ZERO rules loaded — refusing to report all-clear") + return groups + + +def selector_series(selector, start, end): + """How many series this selector matched over the window. -1 on error.""" + url = PROM + "/api/v1/series?" + urllib.parse.urlencode( + {"match[]": selector, "start": start, "end": end} + ) + try: + d = http_json(url) + except (urllib.error.URLError, urllib.error.HTTPError, ValueError): + return -1 + if d.get("status") != "success": + return -1 + return len(d.get("data") or []) + + +def split_selector(selector): + """('metric_name', 'job="x"') -> bare name and the job matcher value if any.""" + name = selector.split("{", 1)[0].strip() + job = None + if "{" in selector: + inner = selector[selector.index("{") + 1: selector.rindex("}")] + for part in inner.split(","): + part = part.strip() + if part.startswith("job=") and "=~" not in part and "!=" not in part: + job = part[4:].strip().strip('"').strip("'") + return name, job + + +def diagnose(selector, start, end, cache): + """Why does this selector match nothing? The cause changes what to do. + + DEAD-NO-TARGET the job it names is not scraped at all, so no scraper will + ever produce this series. Structural. Critical. + DEAD-LABEL-MISMATCH the bare metric DOES exist but the label matchers exclude + everything. Almost always a real bug -- a renamed label or + a selector copied from a different topology. Critical. + DORMANT the metric name has never appeared, but its target is up. + Usually a counter that is only created on first occurrence + (e.g. *_failures_total), or a disabled collector. Warning: + needs a human decision, not an automatic verdict. + """ + name, job = split_selector(selector) + + if job is not None: + key = 'up{job="%s"}' % job + if key not in cache: + cache[key] = selector_series(key, start, end) + if cache[key] == 0: + return "DEAD-NO-TARGET", "no scrape target with job=%r" % job + + if "{" in selector: + if name not in cache: + cache[name] = selector_series(name, start, end) + if cache[name] > 0: + return "DEAD-LABEL-MISMATCH", ( + "metric %s has %d series but the label matchers exclude all of them" + % (name, cache[name]) + ) + + return "DORMANT", "metric %s has never been reported" % name + + +def post_alerts(alerts): + body = json.dumps(alerts).encode("utf-8") + req = urllib.request.Request( + ALERTMANAGER + "/api/v2/alerts", + data=body, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req, timeout=30) as resp: + return resp.status + + +def alert(name, severity, summary, description, **labels): + a = { + "labels": { + "alertname": name, + "severity": severity, + "namespace": "observability", + "service": "rule-liveness-guard", + }, + "annotations": {"summary": summary, "description": description}, + "startsAt": utc_now(), + } + a["labels"].update({k: str(v) for k, v in labels.items()}) + return a + + +# --------------------------------------------------------------------------- # +# Policy +# --------------------------------------------------------------------------- # +def load_policy(): + with open(POLICY_PATH, "r", encoding="utf-8") as fh: + p = json.load(fh) + exempt = {} + for e in p.get("expected_absent", []): + for field in ("rule", "reason", "expires"): + if field not in e: + raise RuntimeError("policy exemption missing %r: %r" % (field, e)) + exempt[e["rule"]] = e + return p, exempt + + +def main(): + started = time.time() + policy, exempt = load_policy() + now = int(time.time()) + start = now - LOOKBACK_DAYS * 86400 + + groups = prom_rules() + + live, dead, exempted, expired, unparseable, absence_based = [], [], [], [], [], [] + cache = {} + checked = 0 + + for g in groups: + for r in g["rules"]: + if r.get("type") != "alerting": + continue + checked += 1 + name = r["name"] + key = "%s/%s" % (g["name"], name) + try: + sels, uses_absence = extract_selectors(r["query"]) + except ParseError as exc: + unparseable.append((key, str(exc))) + continue + if uses_absence: + absence_based.append(key) + continue + if not sels: + continue + + counts = [] + for s in sels: + if s not in cache: + cache[s] = selector_series(s, start, end=now) + counts.append((s, cache[s])) + + errored = [s for s, c in counts if c < 0] + if errored: + unparseable.append((key, "series lookup failed for %s" % errored[0][:80])) + continue + + empty = [s for s, c in counts if c == 0] + if len(empty) != len(counts): + live.append(key) + continue + + # Every input is empty. WHY it is empty decides what to do about it. + kind, why = diagnose(empty[0], start, now, cache) + ex = exempt.get(name) or exempt.get(key) + if ex: + if ex["expires"] < utc_now()[:10]: + expired.append((key, ex)) + else: + exempted.append((key, ex)) + else: + dead.append((key, name, empty, kind, why)) + + # ---------------- report ---------------- + alerts = [] + # DORMANT is a warning, not a verdict: the target is up and the metric may + # simply not have been created yet (counters like *_failures_total appear on + # first occurrence). The other two kinds are structural and cannot self-heal. + severity_for = { + "DEAD-NO-TARGET": "critical", + "DEAD-LABEL-MISMATCH": "critical", + "DORMANT": "warning", + } + for key, name, empty, kind, why in dead: + alerts.append( + alert( + # Deliberately a DIFFERENT alertname per severity. + # + # The first live run of this guard emitted all nine findings as + # `AlertRuleDead` and Alertmanager suppressed six of them: the + # stack's default inhibit_rules drop severity=warning when a + # severity=critical shares the same (namespace, alertname), and + # every finding here carries namespace=observability. Six real + # findings were swallowed by a control that reported success — + # this guard reproducing, in its own delivery path, exactly the + # defect it was written to detect. + # + # Distinct alertnames make the two classes uninhibitable by each + # other. Do not merge them back into one name. + "AlertRuleDead" if severity_for.get(kind) == "critical" else "AlertRuleDormant", + severity_for.get(kind, "critical"), + "Alert rule %s cannot fire (%s)" % (name, kind), + "Rule %s evaluates against selectors that matched no series in the last %dd — %s. " + "It reports inactive forever regardless of cluster state, which on a dashboard is " + "indistinguishable from healthy. Fix the query or the scrape, delete the rule, or " + "declare it in infra/k8s/rule-liveness-guard/base/policy.json with a reason, an " + "expiry and a compensating control. Empty selectors: %s" + % (key, LOOKBACK_DAYS, why, ", ".join(e[:100] for e in empty[:3])), + rule=key, + kind=kind, + dead_selectors=str(len(empty)), + ) + ) + for key, exc in unparseable: + alerts.append( + alert( + "AlertRuleUnscannable", + "warning", + "Rule %s could not be checked for liveness" % key, + "The liveness guard could not evaluate %s (%s). An unscannable rule is treated as " + "a finding, not skipped — otherwise this guard would have the same blind spot it " + "exists to remove." % (key, exc), + rule=key, + ) + ) + for key, ex in expired: + alerts.append( + alert( + "AlertRuleExemptionExpired", + "warning", + "Dead-rule exemption for %s expired on %s" % (key, ex["expires"]), + "This rule is still dead and its declared exemption (%s) has expired. Re-justify " + "it or fix it." % ex["reason"], + rule=key, + ) + ) + + summary = { + "checked": checked, + "live": len(live), + "dead": len(dead), + "exempted": len(exempted), + "expired": len(expired), + "unscannable": len(unparseable), + "absence_based": len(absence_based), + "selectors_tested": len(cache), + "lookback_days": LOOKBACK_DAYS, + } + + body = { + "summary": summary, + "dead_rules": sorted("%s [%s]" % (k, kind) for k, _n, _e, kind, _w in dead), + "unscannable_rules": sorted(k for k, _ in unparseable), + "expired_exemptions": sorted(k for k, _ in expired), + "prometheus": PROM, + } + status = "succeeded" if not (dead or unparseable or expired) else "partial" + + delivered = None + if alerts and not DRY_RUN: + try: + post_alerts(alerts) + delivered = len(alerts) + except Exception as exc: # noqa: BLE001 - must never be silent + emit( + receipt( + "rule-liveness-scan", + "failed", + "prometheus/observability", + {"error": "alertmanager delivery failed: %s" % exc, **body}, + ) + ) + print("FATAL: could not deliver %d alerts: %s" % (len(alerts), exc), file=sys.stderr) + raise SystemExit(2) + + emit( + receipt( + "rule-liveness-scan", + status, + "prometheus/observability", + body, + metrics={**summary, "alerts_delivered": delivered or 0, + "duration_s": round(time.time() - started, 2)}, + policy_refs=["%s/policy.json" % SERVICE_REF], + evidence_refs=[PROM + "/api/v1/rules"], + ) + ) + + # Human-readable tail for `kubectl logs`. + print("checked=%(checked)d live=%(live)d DEAD=%(dead)d exempt=%(exempted)d " + "expired=%(expired)d unscannable=%(unscannable)d" % summary, file=sys.stderr) + for key, _name, empty, kind, why in dead: + print(" %-20s %s <- %s" % (kind, key, why), file=sys.stderr) + for key, exc in unparseable: + print(" UNSCANNABLE %s (%s)" % (key, exc), file=sys.stderr) + for key, ex in expired: + print(" EXPIRED-EXEMPTION %s (%s)" % (key, ex["expires"]), file=sys.stderr) + + # Exit 0 on a completed scan, EVEN WITH FINDINGS. + # + # The first in-cluster run exited 1 because it found dead rules, which + # tripped backoffLimit and left the Job in BackoffLimitExceeded. That would + # have meant the CronJob failed every hour for as long as any dead rule + # existed — a permanently-red Job that everyone learns to ignore, and + # RuleLivenessGuardFailing rendered meaningless. + # + # Findings travel by ALERT, which is the channel built for them and which is + # now actually delivered. A non-zero exit is reserved for the guard itself + # being broken: unreachable Prometheus, an empty rule corpus, or undelivered + # alerts (all of which raise above). "Found problems" is a successful scan. + return + + +if __name__ == "__main__": + main() diff --git a/infra/k8s/rule-liveness-guard/base/kustomization.yaml b/infra/k8s/rule-liveness-guard/base/kustomization.yaml new file mode 100644 index 000000000..38fceeb54 --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/kustomization.yaml @@ -0,0 +1,21 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: observability + +resources: + - cronjob.yaml + - prometheusrule-guard.yaml + +configMapGenerator: + - name: rule-liveness-guard + files: + - guard.py + - policy.json + +generatorOptions: + disableNameSuffixHash: true + +labels: + - includeSelectors: false + pairs: + bundle: observability diff --git a/infra/k8s/rule-liveness-guard/base/policy.json b/infra/k8s/rule-liveness-guard/base/policy.json new file mode 100644 index 000000000..185ccbbaa --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/policy.json @@ -0,0 +1,123 @@ +{ + "_comment": [ + "Declared exemptions for alert rules that legitimately have no series.", + "", + "Every entry needs `rule`, `reason` and `expires` (YYYY-MM-DD). Entries whose", + "underlying RISK still exists must also name a `compensating_control` -- the", + "thing that actually covers the hazard now that this rule cannot.", + "", + "An expired exemption raises AlertRuleExemptionExpired. That is deliberate:", + "the failure this whole guard exists to fix was a control everyone assumed", + "was working, so an allowlist that never expires would just relocate the", + "problem. Expiries are staggered so they do not all come due in one week.", + "", + "NOT exempted, on purpose: HellGraphDown and HighErrorRatio. Both are this", + "estate's own rules, both are dead, and both are fixable rather than", + "acceptable. They are left alerting until they are fixed." + ], + + "expected_absent": [ + { + "rule": "KubePersistentVolumeFillingUp", + "reason": "GKE Autopilot's Warden denies nodes/proxy cluster-wide, so the in-cluster Prometheus has no kubelet scrape job and kubelet_volume_stats_* can never exist here. Structural, not a misconfiguration.", + "compensating_control": "infra/k8s/pvc-capacity-guard (PR #1105) reads the same kubelet_volume_stats_* metrics from Google Managed Prometheus, which does scrape kubelet, and alerts PvcGuardExpanded/AtMaximum/CannotExpand/NoData.", + "expires": "2027-01-31" + }, + { + "rule": "KubePersistentVolumeInodesFillingUp", + "reason": "Same structural cause: no kubelet job in the in-cluster Prometheus.", + "compensating_control": "PENDING. pvc-capacity-guard tracks bytes, not inodes. Inode exhaustion on these volumes is currently UNCOVERED — tracked for Wave 0.", + "expires": "2026-10-31" + }, + { + "rule": "KubeletClientCertificateExpiration", + "reason": "No kubelet scrape job on Autopilot. Node certificates are managed by Google as part of the Autopilot control plane.", + "compensating_control": "GKE Autopilot node auto-repair and Google-managed node lifecycle.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletServerCertificateExpiration", + "reason": "No kubelet scrape job on Autopilot. Node certificates are Google-managed.", + "compensating_control": "GKE Autopilot node auto-repair and Google-managed node lifecycle.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletClientCertificateRenewalErrors", + "reason": "No kubelet scrape job on Autopilot.", + "compensating_control": "GKE Autopilot node auto-repair.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletServerCertificateRenewalErrors", + "reason": "No kubelet scrape job on Autopilot.", + "compensating_control": "GKE Autopilot node auto-repair.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletPlegDurationHigh", + "reason": "No kubelet scrape job on Autopilot. Kubelet internals are not our operational surface on a managed node pool.", + "compensating_control": "GKE Autopilot node auto-repair; pod-level symptoms are covered by KubePodNotReady and KubeContainerWaiting, which are live.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletPodStartUpLatencyHigh", + "reason": "No kubelet scrape job on Autopilot.", + "compensating_control": "KubePodNotReady (live) covers the user-visible symptom.", + "expires": "2027-01-31" + }, + { + "rule": "CPUThrottlingHigh", + "reason": "Reads container_cpu_cfs_throttled_periods_total from cAdvisor, which is exposed via the kubelet — unreachable on Autopilot for the same Warden reason.", + "compensating_control": "PENDING. CPU throttling is currently UNOBSERVED in-cluster. GMP does publish container/cpu metrics; wiring an equivalent is Wave 0 work.", + "expires": "2026-10-31" + }, + { + "rule": "NodeRAIDDegraded", + "reason": "Autopilot nodes are single-disk COS VMs with no md RAID arrays. node_md_* will never exist on this platform.", + "expires": "2027-06-30" + }, + { + "rule": "NodeRAIDDiskFailure", + "reason": "No md RAID arrays on Autopilot nodes.", + "expires": "2027-06-30" + }, + { + "rule": "NodeBondingDegraded", + "reason": "No bonded network interfaces on Autopilot nodes.", + "expires": "2027-06-30" + }, + { + "rule": "NodeSystemdServiceFailed", + "reason": "node-exporter runs unprivileged in a container on Autopilot and cannot read the host systemd D-Bus socket, so the systemd collector reports nothing.", + "compensating_control": "GKE Autopilot node auto-repair owns node-level service health.", + "expires": "2027-01-31" + }, + { + "rule": "PrometheusRemoteStorageFailures", + "reason": "No remote_write configured on this Prometheus, so prometheus_remote_storage_* is never registered.", + "compensating_control": "N/A — there is no remote storage to fail. If remote_write is ever configured this exemption must be removed.", + "expires": "2027-01-31" + }, + { + "rule": "PrometheusRemoteWriteBehind", + "reason": "No remote_write configured.", + "expires": "2027-01-31" + }, + { + "rule": "PrometheusRemoteWriteDesiredShards", + "reason": "No remote_write configured.", + "expires": "2027-01-31" + }, + { + "rule": "AlertmanagerClusterFailedToSendAlerts", + "reason": "Single-replica Alertmanager, so alertmanager_cluster_* gossip metrics do not exist.", + "compensating_control": "AlertmanagerFailedToSendAlerts (live, non-cluster variant) covers delivery failure for this topology, and AlertDeliveryDead in infra/k8s/alert-delivery covers the whole path.", + "expires": "2027-01-31" + }, + { + "rule": "AlertmanagerMembersInconsistent", + "reason": "Single-replica Alertmanager: no cluster members to be inconsistent.", + "expires": "2027-01-31" + } + ] +} diff --git a/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml b/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml new file mode 100644 index 000000000..f1f40fd66 --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml @@ -0,0 +1,47 @@ +# Meta-monitoring for the rule-liveness guard itself. +# +# The guard exists because a control that cannot fail is not a control. That +# applies to the guard: if its CronJob stops running, the estate quietly loses +# dead-rule detection and looks exactly the same. So the guard's own liveness is +# a rule — and, being an ordinary rule in this Prometheus, it is scanned by the +# guard on the next run. The check checks the checker. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: rule-liveness-guard + namespace: observability + labels: + bundle: observability + release: kube-prometheus-stack +spec: + groups: + - name: rule-liveness-guard.meta + rules: + - alert: RuleLivenessGuardNotRunning + # kube_state_metrics publishes job completion times; the guard runs + # hourly, so a gap over 3h means it is not running at all. + expr: | + time() - max(kube_job_status_completion_time{job_name=~"rule-liveness-guard-.*"}) > 10800 + or absent(kube_job_status_completion_time{job_name=~"rule-liveness-guard-.*"}) + for: 30m + labels: { severity: warning } + annotations: + summary: "rule-liveness-guard has not completed in over 3h" + description: >- + Dead-rule detection has stopped. Alert rules can silently become + incapable of firing again with nothing to notice. + runbook: infra/k8s/rule-liveness-guard/README.md + + - alert: RuleLivenessGuardFailing + expr: | + increase(kube_job_status_failed{job_name=~"rule-liveness-guard-.*"}[2h]) > 2 + for: 10m + labels: { severity: warning } + annotations: + summary: "rule-liveness-guard is failing repeatedly" + description: >- + The guard exits non-zero ONLY when it is itself broken — + unreachable Prometheus, an empty rule corpus, or alerts it could + not deliver. Findings do not fail the job (they travel by alert), + so a failing job means dead-rule detection is down, not that dead + rules exist. Check the pod logs for the EvidenceReceipt. From 3c88e885a90ccd1f6707122cdf1f809b25d27b9c Mon Sep 17 00:00:00 2001 From: mdheller <21163552+mdheller@users.noreply.github.com> Date: Thu, 30 Jul 2026 02:09:00 -0400 Subject: [PATCH 2/3] rule-liveness-guard: meta-alerts needed an explicit namespace label The absent() branch of RuleLivenessGuardNotRunning produces no labels of its own, so the alert would not match the AlertmanagerConfig sub-route (operator-scoped OnNamespace) and the alert reporting that dead-rule detection is down would itself be routed to null. Same defect found in the companion alert-delivery rules. --- .../rule-liveness-guard/base/prometheusrule-guard.yaml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml b/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml index f1f40fd66..02def2151 100644 --- a/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml +++ b/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml @@ -19,12 +19,15 @@ spec: rules: - alert: RuleLivenessGuardNotRunning # kube_state_metrics publishes job completion times; the guard runs - # hourly, so a gap over 3h means it is not running at all. + # hourly, so a gap over 3h means it is not running at all. Note the explicit + # namespace label below: the absent() branch produces no labels of its own, + # and without it this alert would not match the AlertmanagerConfig sub-route + # — the alert saying detection is down would go to null. expr: | time() - max(kube_job_status_completion_time{job_name=~"rule-liveness-guard-.*"}) > 10800 or absent(kube_job_status_completion_time{job_name=~"rule-liveness-guard-.*"}) for: 30m - labels: { severity: warning } + labels: { severity: warning, namespace: observability } annotations: summary: "rule-liveness-guard has not completed in over 3h" description: >- @@ -36,7 +39,7 @@ spec: expr: | increase(kube_job_status_failed{job_name=~"rule-liveness-guard-.*"}[2h]) > 2 for: 10m - labels: { severity: warning } + labels: { severity: warning, namespace: observability } annotations: summary: "rule-liveness-guard is failing repeatedly" description: >- From 29fec65b1564c4558d04535154506a8c8734c038 Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Thu, 30 Jul 2026 02:13:28 -0400 Subject: [PATCH 3/3] Revert the SLO-rule edits: fix/observability-delivery-plane already owns them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found a parallel branch, fix/observability-delivery-plane (pushed, no PR yet), that treats the same two dead rules more thoroughly than this PR did and would have conflicted on prometheusrule-slos.yaml. It reaches the same conclusion on HellGraphDown independently — repoint to kube_deployment_status_replicas_available, because the ServiceMonitor selected app: hellgraph against a chart that renders only app.kubernetes.io/* labels. It goes further by deleting the dead ServiceMonitor outright rather than leaving it selecting nothing. On the request-SLO group it is better evidenced than this PR was. It establishes that http_server_request_duration_seconds_* has no emitter anywhere in the estate — no OTel SDK, no prometheus_client, no promhttp in any requirements.txt or any of the four go.mod files, and /metrics 404s on every service probed — and therefore deletes the recording rules and HighErrorRatio rather than alerting on their absence. That matters, because the CanaryGateBlind alert added here would have fired permanently until roughly 47 services gained HTTP middleware. A permanently-firing critical is alert fatigue, which is the objection this wave raised against fixing HellGraphDown's selector alone. Deleting the rules and re-expressing the gate is the better remedy. So this PR keeps only what is genuinely additive: the guard that detects the whole CLASS of unfireable rules. That branch fixes specific rules; this one finds the next ones. The guard flagged both of those rules on its first live run, which is the evidence they are complementary rather than duplicative. --- .../base/prometheusrule-slos.yaml | 50 +------------------ 1 file changed, 2 insertions(+), 48 deletions(-) diff --git a/infra/k8s/observability/base/prometheusrule-slos.yaml b/infra/k8s/observability/base/prometheusrule-slos.yaml index ce381fa60..f072f5fbb 100644 --- a/infra/k8s/observability/base/prometheusrule-slos.yaml +++ b/infra/k8s/observability/base/prometheusrule-slos.yaml @@ -20,28 +20,12 @@ spec: labels: { severity: warning } annotations: summary: "Target {{ $labels.job }} / {{ $labels.instance }} is down" - # WAS `up{job=~".*hellgraph.*"} == 0`, which matched ZERO series and so - # could never fire while reporting health=ok — the same defect as - # KubePersistentVolumeFillingUp. Two independent causes: - # 1. the `hellgraph` ServiceMonitor selects `app: hellgraph`, but the - # Service is rendered by the shared socioprophet-service chart and - # carries only app.kubernetes.io/* labels, so it matched nothing; - # 2. hellgraph-service does not serve /metrics at all — probing - # http://hellgraph-service:8090/metrics returns HTTP 404. - # Fixing (1) alone would have produced a permanently-firing false alarm, - # which is not an improvement on a permanently-silent one. - # So this now reads kube-state-metrics, which genuinely observes the - # deployment (3 live series today, one per namespace). Giving hellgraph a - # real /metrics endpoint is app work and is NOT in this lane. - alert: HellGraphDown - expr: | - kube_deployment_status_replicas_available{deployment="hellgraph-service"} == 0 - and on (namespace, deployment) - kube_deployment_spec_replicas{deployment="hellgraph-service"} > 0 + expr: up{job=~".*hellgraph.*"} == 0 for: 2m labels: { severity: critical } annotations: - summary: "HellGraph managed graph service has no available replicas in {{ $labels.namespace }} — person-graph reads/writes fail" + summary: "HellGraph managed graph service is down — person-graph reads/writes fail" - name: socioprophet.request-slo rules: # 5xx error ratio per service — the canary gate metric. @@ -60,33 +44,3 @@ spec: labels: { severity: warning } annotations: summary: "{{ $labels.job }} 5xx ratio > 5% (SLO breach) — blocks canary promotion" - - # ── The canary gate must not be able to pass by knowing nothing. - # - # infra/k8s/rollouts/base/analysistemplate-slo.yaml gates every canary - # promotion on job:request_error_ratio:rate5m and job:request_latency_p99:5m. - # Both recording rules derive from http_server_request_duration_seconds_* - # (OTel HTTP semconv, via the collector). That source metric currently has - # ZERO series — no service emits it — so both recording rules produce - # nothing and the AnalysisTemplate queries an empty vector. - # - # An SLO gate with no input is not a gate. Whether Argo Rollouts treats the - # empty result as an error or as a pass is not something this estate should - # be discovering during a bad deploy. So the missing input is itself an - # alert, at critical, and it is deliberately built on absent() — the one - # construct whose whole purpose is to react to emptiness. - - alert: CanaryGateBlind - expr: absent(http_server_request_duration_seconds_count) - for: 15m - labels: { severity: critical } - annotations: - summary: "Canary SLO gate has no input — http_server_request_duration_seconds_count has zero series" - description: >- - job:request_error_ratio:rate5m and job:request_latency_p99:5m both - derive from this metric, and the Argo Rollouts slo-gate - AnalysisTemplate reads both. With no source series the gate cannot - evaluate the SLO it exists to enforce. Either services are not - exporting OTel HTTP metrics through the collector, or the collector - pipeline is broken. Until this clears, treat canary promotions as - ungated. - runbook: infra/k8s/rule-liveness-guard/README.md