diff --git a/deploy/argocd/rule-liveness-guard.yaml b/deploy/argocd/rule-liveness-guard.yaml new file mode 100644 index 000000000..e4c6c1dd0 --- /dev/null +++ b/deploy/argocd/rule-liveness-guard.yaml @@ -0,0 +1,37 @@ +# Detects alert rules that are incapable of firing — the generalisation of the +# KubePersistentVolumeFillingUp failure, where a rule queried a metric with zero +# series and reported health=ok forever while the disk filled. +# +# Lives in deploy/argocd/, NOT infra/argocd/, for the reason documented in +# deploy/argocd/pvc-capacity-guard.yaml: the root app-of-apps only recurses +# `deploy/argocd`, so an Application under infra/argocd/ deploys nothing. For a +# guard against controls that silently do nothing, that would be the defect +# recursing one layer up. +# +# sync-wave "1": after the kube-prometheus-stack CRDs so PrometheusRule exists. +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: rule-liveness-guard + namespace: argocd + labels: + app.kubernetes.io/part-of: platform-ops + annotations: + argocd.argoproj.io/sync-wave: "1" + # Required by tools/preflight_deploy_contract.py check 5. reference: a + # replaceable meta-monitor on a digest-pinned stock python image. + socioprophet.io/tier: "reference" +spec: + project: default + destination: + server: https://kubernetes.default.svc + namespace: observability + source: + repoURL: https://github.com/SocioProphet/prophet-platform + targetRevision: main + path: infra/k8s/rule-liveness-guard/base + syncPolicy: + automated: { prune: true, selfHeal: true } + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/infra/k8s/rule-liveness-guard/README.md b/infra/k8s/rule-liveness-guard/README.md new file mode 100644 index 000000000..fcd82095d --- /dev/null +++ b/infra/k8s/rule-liveness-guard/README.md @@ -0,0 +1,193 @@ +# rule-liveness-guard — finds alert rules that cannot fire + +## The defect this generalises + +`KubePersistentVolumeFillingUp` was loaded in the `observability` Prometheus with +`health=ok`, `lastError=none`, `state=inactive`: + +``` +group: kubernetes-storage + KubePersistentVolumeFillingUp | state=inactive | health=ok | lastError=none +``` + +Its query was +`kubelet_volume_stats_available_bytes{job="kubelet"} / kubelet_volume_stats_capacity_bytes{...} < 0.03`. + +That Prometheus has no `kubelet` scrape job — GKE Autopilot's Warden denies +`nodes/proxy` cluster-wide. Both metrics resolve to **zero series**. The rule +evaluated an empty set forever and reported `inactive`, which on every dashboard +is indistinguishable from "the volumes are fine". The disk reached 100%. + +That is not one bad rule. **A control that cannot fail is not a control**, and +any rule whose inputs do not exist is in that category. The first scan of this +estate found **28 alerting rules** in that state. + +## The signature — and why the obvious test is wrong + +"The rule returns nothing" is the wrong test. `up == 0` returns nothing when +everything is up; that is a healthy rule doing its job. The distinguishing +question is whether the rule's **inputs** exist: + +``` +up == 0 -> `up` has 31 series -> LIVE +kubelet_volume_stats_available_bytes{job=…} -> selector has 0 series -> DEAD +``` + +So the guard parses every alerting rule into its **vector selectors** and asks +Prometheus (`/api/v1/series`) whether each matched anything over a lookback +window. A rule whose every selector is empty cannot fire, whatever the cluster +does. + +Rules built on `absent()` / `absent_over_time()` are exempt by construction — +reacting to emptiness is their whole purpose. + +## Three causes, three verdicts + +Collapsing them would produce false positives, so the guard separates them: + +| Verdict | Meaning | Severity | +|---|---|---| +| **DEAD-NO-TARGET** | The selector names `job="X"` and `up{job="X"}` has no series. Nothing will ever produce this metric. Structural. | critical | +| **DEAD-LABEL-MISMATCH** | The bare metric *does* exist, but the label matchers exclude every series. Almost always a real bug — a renamed label, or a selector copied from a different topology. | critical | +| **DORMANT** | The metric name has never appeared, but its target is up. Usually a counter created on first occurrence (`*_failures_total`), or a disabled collector. Needs a human decision, not an automatic verdict. | warning | + +`HellGraphDown` was caught as **DEAD-LABEL-MISMATCH**: `up{job=~".*hellgraph.*"}` +while `up` had 31 series and none matched. + +## Exemptions are declared, justified, and they expire + +Some selectors are legitimately absent — Autopilot nodes have no RAID arrays, so +`node_md_disks` will never exist. Those live in `policy.json`. Every entry needs: + +- `reason` — why it is absent +- `expires` — a date, after which it **alerts** (`AlertRuleExemptionExpired`) +- `compensating_control` — where the risk still exists, what covers it instead + +That last field is the point. `KubePersistentVolumeFillingUp` is exempted, but +only because `pvc-capacity-guard` (PR #1105) reads the same metrics from Google +Managed Prometheus, which *does* scrape kubelet. Two entries deliberately record +that they have **no** compensating control yet and carry short expiries: +`KubePersistentVolumeInodesFillingUp` (inode exhaustion is uncovered) and +`CPUThrottlingHigh` (throttling is unobserved in-cluster). + +Expiries are staggered so they do not all come due at once. An allowlist that +never expires just relocates the original problem. + +**Not exempted, on purpose:** `HellGraphDown` and `HighErrorRatio`. Both are the +estate's own rules, both are dead, both are fixable rather than acceptable. + +## It does not exempt itself + +The guard's own alerts (`RuleLivenessGuardNotRunning`, `RuleLivenessGuardFailing`) +are ordinary rules in the same Prometheus and are scanned like everything else. +A scanner that skips itself is how a ratchet ends up counting its own allowlist. + +## Failure is loud + +Every abnormal outcome raises or alerts; nothing is skipped quietly. + +- An expression it cannot parse → `AlertRuleUnscannable`, not a silent skip. +- Prometheus reporting **zero** rules → hard `RuntimeError`. An empty corpus is a + broken config, never an all-clear. +- A failed `/api/v1/series` lookup → counted as unscannable, not as "live". +- Alertmanager unreachable → `failed` receipt, exit 2. + +**Findings do not fail the Job.** The first in-cluster run exited 1 because it +found dead rules, tripped `backoffLimit`, and landed in `BackoffLimitExceeded` — +which would have meant a permanently-red hourly Job for as long as any dead rule +existed, the kind everyone learns to ignore, and would have made +`RuleLivenessGuardFailing` meaningless. Findings travel by **alert**, which is +the channel built for them and which is now actually delivered. A non-zero exit +is reserved for the guard itself being broken. + +## Receipts + +One `EvidenceReceipt` (`contracts/EvidenceReceipt.v0.1.json`) per scan, as a JSON +line on stdout → `fluentbit-gke` → **Google Cloud Logging**. + +```json +{"action":"rule-liveness-scan","status":"partial","service_ref":"infra/k8s/rule-liveness-guard", + "hash":"96326a03…","hash_algo":"sha256","subject_ref":"prometheus/observability", + "metrics":{"checked":134,"live":100,"dead":7,"exempted":21,"expired":0, + "unscannable":0,"absence_based":5,"selectors_tested":197}} +``` + +`status` is `succeeded` on a clean scan and `partial` when there are findings. +The `hash` covers the canonical finding set, so two scans with identical findings +produce an identical receipt id. + +Read them with: + +``` +gcloud logging read 'resource.labels.container_name="guard" + AND jsonPayload.action="rule-liveness-scan"' --limit 5 +``` + +## Proof it fires + +A probe `PrometheusRule` was applied with three rules: two deliberately dead by +different causes, and one healthy control that returns empty simply because +nothing is down. + +**Red:** + +``` +checked=137 live=101 DEAD=9 exempt=21 expired=0 unscannable=0 + DEAD-NO-TARGET wave1.probe/ProbeDeadNoTarget <- no scrape target with job='no-such-exporter' + DEAD-LABEL-MISMATCH wave1.probe/ProbeDeadLabelMismatch <- metric up has 31 series but the label matchers exclude all of them + DEAD-LABEL-MISMATCH socioprophet.availability/HellGraphDown + DORMANT socioprophet.request-slo/HighErrorRatio + … 5 more +``` + +`ProbeLive` was **not** flagged — the control against false positives held. + +**Green**, after pointing both probe rules at metrics that exist: + +``` +checked=137 live=103 DEAD=7 exempt=21 expired=0 unscannable=0 + (no wave1.probe findings) +``` + +`live` 101 → 103, findings 9 → 7. The probe rule was then deleted. + +### The proof caught a bug in this guard + +The first non-dry run emitted all nine findings as `AlertRuleDead`, and +Alertmanager **suppressed six of them**: + +``` +critical wave1.probe/ProbeDeadNoTarget state=active inhibitedBy=0 +warning kube-state-metrics/KubeStateMetr… state=suppressed inhibitedBy=1 +warning socioprophet.request-slo/HighErr… state=suppressed inhibitedBy=1 +… +``` + +The stack's default `inhibit_rules` drop `severity=warning` when a +`severity=critical` shares the same `(namespace, alertname)` — and every finding +carries `namespace=observability`. Six real findings were swallowed by a control +that reported success: this guard reproducing, in its own delivery path, exactly +the defect it exists to detect. + +Fixed by emitting **distinct alertnames per severity** — `AlertRuleDead` +(critical) and `AlertRuleDormant` (warning) — which cannot inhibit each other. +Re-run confirms all nine `state=active`. Do not merge those names back together. + +## Runbook + +**`AlertRuleDead` (critical)** — a rule is structurally incapable of firing. +Decide between three actions, in preference order: + +1. **Fix the input.** Wrong label, wrong job, a ServiceMonitor that matches + nothing. This is the usual answer for `DEAD-LABEL-MISMATCH`. +2. **Delete the rule.** If the hazard is not real here, a rule that cannot fire + is worse than no rule — it reads as coverage. +3. **Declare it** in `policy.json` with a reason, an expiry, and a compensating + control. Only if the hazard is real but is covered elsewhere. + +**`AlertRuleDormant` (warning)** — the target is up but the metric has never +appeared. Confirm whether the metric is created lazily (fine, leave it) or the +collector is disabled (fix or declare it). + +**`AlertRuleUnscannable`** — the guard could not parse an expression. That is a +gap in the extractor, not in the rule. File it; do not silence it. diff --git a/infra/k8s/rule-liveness-guard/base/cronjob.yaml b/infra/k8s/rule-liveness-guard/base/cronjob.yaml new file mode 100644 index 000000000..f6269fe7b --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/cronjob.yaml @@ -0,0 +1,75 @@ +apiVersion: batch/v1 +kind: CronJob +metadata: + name: rule-liveness-guard + namespace: observability +spec: + # Hourly. Rule liveness changes when someone edits a PrometheusRule, renames a + # label, or a scrape target disappears — none of which is sub-hourly. Hourly is + # also frequent enough that a regression is caught the same working day. + schedule: "17 * * * *" + concurrencyPolicy: Forbid + startingDeadlineSeconds: 600 + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 5 + jobTemplate: + spec: + backoffLimit: 2 + template: + metadata: + labels: + app: rule-liveness-guard + spec: + restartPolicy: OnFailure + securityContext: + runAsNonRoot: true + runAsUser: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: guard + # Pulled DIRECTLY from docker.io, not through + # registry.socioprophet.ai — same rule as pvc-capacity-guard and + # alert-delivery. A monitor whose own image comes from the thing it + # monitors goes blind exactly when it is needed. Stdlib only, so a + # bare python image needs no pip install. + image: python:3.12-slim@sha256:57cd7c3a7a273101a6485ba99423ee568157882804b1124b4dd04266317710de + command: ["python3", "/opt/guard/guard.py"] + env: + - name: PROM_URL + value: http://kube-prometheus-stack-prometheus.observability.svc:9090 + - name: ALERTMANAGER_URL + value: http://kube-prometheus-stack-alertmanager.observability.svc:9093 + - name: POLICY_PATH + value: /etc/guard/policy.json + # 7 days. Long enough that a metric which only appears under load + # is not called dead; short enough that a rule broken last week + # is still caught this week. + - name: LOOKBACK_DAYS + value: "7" + volumeMounts: + - name: guard + mountPath: /opt/guard/guard.py + subPath: guard.py + readOnly: true + - name: guard + mountPath: /etc/guard/policy.json + subPath: policy.json + readOnly: true + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: 500m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumes: + - name: guard + configMap: + name: rule-liveness-guard + defaultMode: 0555 diff --git a/infra/k8s/rule-liveness-guard/base/guard.py b/infra/k8s/rule-liveness-guard/base/guard.py new file mode 100644 index 000000000..5b0a31f60 --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/guard.py @@ -0,0 +1,563 @@ +#!/usr/bin/env python3 +"""Rule-liveness guard: finds alert rules that are incapable of firing. + +The defect +---------- +`KubePersistentVolumeFillingUp` was loaded in Prometheus with `health=ok`, +`lastError=none`, `state=inactive`. It queried +`kubelet_volume_stats_available_bytes{job="kubelet"}` against a Prometheus with +no kubelet scrape job. Both metrics resolved to zero series, so the rule +evaluated an empty set forever and reported `inactive` — which on every +dashboard is indistinguishable from "the volumes are fine". The disk hit 100%. + +This is not one bad rule. It is a whole class: any rule whose inputs do not +exist is a rule that cannot fail, and a control that cannot fail is not a +control. A first scan of this estate found 28 of them. + +The signature +------------- +"Rule returns nothing" is the WRONG test — `up == 0` correctly returns nothing +when everything is up. The distinguishing question is whether the rule's INPUTS +exist: + + up == 0 -> `up` has 14 series -> LIVE + kubelet_volume_stats_available_bytes{..} -> selector has 0 series -> DEAD + +So this guard parses each rule expression into its vector selectors and asks +Prometheus whether each selector matches any series over a lookback window. A +rule whose every selector is empty cannot fire, no matter what happens in the +cluster. + +Rules built on absent()/absent_over_time() are exempt by construction: reacting +to emptiness is their entire purpose. + +Exemptions are declared, justified, and they expire +--------------------------------------------------- +Some selectors are legitimately absent — this cluster has no RAID arrays, so +`node_md_disks` will never exist. Those are declared in policy.json. Every +exemption must carry a `reason`, an `expires` date, and — where the underlying +risk is still real — a `compensating_control` naming what covers it instead. +An expired exemption ALERTS. An undeclared dead rule ALERTS. That keeps the +allowlist from quietly becoming the place dead controls go to be forgotten. + +This guard does not exempt itself: its own alerts are ordinary rules and are +scanned like everything else. + +Failure is loud +--------------- +Every abnormal outcome raises or alerts — an unparseable expression, an empty +rule list, an unreachable Prometheus, an unreachable Alertmanager. Silently +skipping what it cannot evaluate is the exact defect it exists to find. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime, timezone + +PROM = os.environ.get("PROM_URL", "http://kube-prometheus-stack-prometheus.observability.svc:9090") +ALERTMANAGER = os.environ.get( + "ALERTMANAGER_URL", "http://kube-prometheus-stack-alertmanager.observability.svc:9093" +) +POLICY_PATH = os.environ.get("POLICY_PATH", "/etc/guard/policy.json") +LOOKBACK_DAYS = int(os.environ.get("LOOKBACK_DAYS", "7")) +DRY_RUN = os.environ.get("DRY_RUN", "").lower() in ("1", "true", "yes") +SERVICE_REF = "infra/k8s/rule-liveness-guard" +VERSION = "0.1" + +# --------------------------------------------------------------------------- # +# PromQL vector-selector extraction +# --------------------------------------------------------------------------- # +FUNCTIONS = { + "abs", "absent", "absent_over_time", "ceil", "changes", "clamp", "clamp_max", + "clamp_min", "day_of_month", "day_of_week", "day_of_year", "days_in_month", + "delta", "deriv", "exp", "floor", "histogram_avg", "histogram_count", + "histogram_fraction", "histogram_quantile", "histogram_stddev", + "histogram_stdvar", "histogram_sum", "holt_winters", + "double_exponential_smoothing", "hour", "idelta", "increase", "info", + "irate", "label_join", "label_replace", "ln", "log2", "log10", + "mad_over_time", "minute", "month", "pi", "predict_linear", "rate", + "resets", "round", "scalar", "sgn", "sort", "sort_desc", "sort_by_label", + "sort_by_label_desc", "sqrt", "time", "timestamp", "vector", "year", + "avg_over_time", "min_over_time", "max_over_time", "sum_over_time", + "count_over_time", "quantile_over_time", "stddev_over_time", + "stdvar_over_time", "last_over_time", "present_over_time", + "acos", "acosh", "asin", "asinh", "atan", "atanh", "cos", "cosh", "sin", + "sinh", "tan", "tanh", "deg", "rad", +} +AGGREGATIONS = { + "sum", "min", "max", "avg", "group", "stddev", "stdvar", "count", + "count_values", "bottomk", "topk", "quantile", "limitk", "limit_ratio", +} +KEYWORDS = { + "by", "without", "on", "ignoring", "group_left", "group_right", "offset", + "bool", "and", "or", "unless", "start", "end", "atan2", "inf", "nan", +} +LABEL_LIST_KEYWORDS = {"by", "without", "on", "ignoring", "group_left", "group_right"} +ABSENCE_FUNCS = {"absent", "absent_over_time"} + +IDENT_START = set("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_:") +IDENT_CHAR = IDENT_START | set("0123456789") + + +class ParseError(Exception): + pass + + +def _skip_string(expr, i): + quote = expr[i] + i += 1 + while i < len(expr): + c = expr[i] + if c == "\\" and quote != "`": + i += 2 + continue + if c == quote: + return i + 1 + i += 1 + raise ParseError("unterminated string literal") + + +def _match(expr, i, opener, closer): + depth = 0 + while i < len(expr): + c = expr[i] + if c in "\"'`": + i = _skip_string(expr, i) + continue + if c == opener: + depth += 1 + elif c == closer: + depth -= 1 + if depth == 0: + return i + 1 + i += 1 + raise ParseError("unbalanced %s %s" % (opener, closer)) + + +def _peek(expr, i): + while i < len(expr) and expr[i] in " \t\n\r": + i += 1 + return i + + +def extract_selectors(expr): + """Return (selectors, uses_absence_func). Raises ParseError on anything odd.""" + selectors = [] + uses_absence = False + i, n = 0, len(expr) + while i < n: + c = expr[i] + if c == "#": + while i < n and expr[i] != "\n": + i += 1 + continue + if c in "\"'`": + i = _skip_string(expr, i) + continue + if c == "[": # range / subquery: durations, not metrics + i = _match(expr, i, "[", "]") + continue + if c == "{": # bare {__name__="x"} selector + end = _match(expr, i, "{", "}") + selectors.append(expr[i:end]) + i = end + continue + if c in IDENT_START: + j = i + while j < n and expr[j] in IDENT_CHAR: + j += 1 + word = expr[i:j] + nxt = _peek(expr, j) + if word in ABSENCE_FUNCS: + uses_absence = True + if word in FUNCTIONS or word in AGGREGATIONS: + i = j + continue + if word in KEYWORDS: + if word in LABEL_LIST_KEYWORDS and nxt < n and expr[nxt] == "(": + i = _match(expr, nxt, "(", ")") + continue + i = j + continue + if nxt < n and expr[nxt] == "(": + # Unknown identifier used as a function. Do not guess. + raise ParseError("unknown function %r" % word) + sel, k = word, j + nk = _peek(expr, k) + if nk < n and expr[nk] == "{": + end = _match(expr, nk, "{", "}") + sel, k = word + expr[nk:end], end + selectors.append(sel) + i = k + continue + i += 1 + return selectors, uses_absence + + +# --------------------------------------------------------------------------- # +# Receipts (contracts/EvidenceReceipt.v0.1.json) +# --------------------------------------------------------------------------- # +def utc_now(): + return datetime.now(timezone.utc).replace(microsecond=0).isoformat() + + +def canonical(obj): + return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def sha256(text): + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def emit(obj): + sys.stdout.write(canonical(obj) + "\n") + sys.stdout.flush() + + +def receipt(action, status, subject_ref, body, **extra): + h = sha256(canonical(body)) + r = { + "version": VERSION, + "receipt_id": "evr-%s-%s" % (action, h[:32]), + "created_at": utc_now(), + "service_ref": SERVICE_REF, + "action": action, + "status": status, + "subject_ref": subject_ref, + "hash": h, + "hash_algo": "sha256", + } + r.update(extra) + return r + + +# --------------------------------------------------------------------------- # +# Prometheus / Alertmanager +# --------------------------------------------------------------------------- # +def http_json(url, timeout=45): + with urllib.request.urlopen(url, timeout=timeout) as resp: + return json.loads(resp.read().decode("utf-8")) + + +def prom_rules(): + d = http_json(PROM + "/api/v1/rules") + if d.get("status") != "success": + raise RuntimeError("prometheus /rules returned status=%s" % d.get("status")) + groups = d["data"]["groups"] + total = sum(len(g["rules"]) for g in groups) + # An empty rule corpus is a broken scrape/config, never an all-clear. + if total == 0: + raise RuntimeError("prometheus reports ZERO rules loaded — refusing to report all-clear") + return groups + + +def selector_series(selector, start, end): + """How many series this selector matched over the window. -1 on error.""" + url = PROM + "/api/v1/series?" + urllib.parse.urlencode( + {"match[]": selector, "start": start, "end": end} + ) + try: + d = http_json(url) + except (urllib.error.URLError, urllib.error.HTTPError, ValueError): + return -1 + if d.get("status") != "success": + return -1 + return len(d.get("data") or []) + + +def split_selector(selector): + """('metric_name', 'job="x"') -> bare name and the job matcher value if any.""" + name = selector.split("{", 1)[0].strip() + job = None + if "{" in selector: + inner = selector[selector.index("{") + 1: selector.rindex("}")] + for part in inner.split(","): + part = part.strip() + if part.startswith("job=") and "=~" not in part and "!=" not in part: + job = part[4:].strip().strip('"').strip("'") + return name, job + + +def diagnose(selector, start, end, cache): + """Why does this selector match nothing? The cause changes what to do. + + DEAD-NO-TARGET the job it names is not scraped at all, so no scraper will + ever produce this series. Structural. Critical. + DEAD-LABEL-MISMATCH the bare metric DOES exist but the label matchers exclude + everything. Almost always a real bug -- a renamed label or + a selector copied from a different topology. Critical. + DORMANT the metric name has never appeared, but its target is up. + Usually a counter that is only created on first occurrence + (e.g. *_failures_total), or a disabled collector. Warning: + needs a human decision, not an automatic verdict. + """ + name, job = split_selector(selector) + + if job is not None: + key = 'up{job="%s"}' % job + if key not in cache: + cache[key] = selector_series(key, start, end) + if cache[key] == 0: + return "DEAD-NO-TARGET", "no scrape target with job=%r" % job + + if "{" in selector: + if name not in cache: + cache[name] = selector_series(name, start, end) + if cache[name] > 0: + return "DEAD-LABEL-MISMATCH", ( + "metric %s has %d series but the label matchers exclude all of them" + % (name, cache[name]) + ) + + return "DORMANT", "metric %s has never been reported" % name + + +def post_alerts(alerts): + body = json.dumps(alerts).encode("utf-8") + req = urllib.request.Request( + ALERTMANAGER + "/api/v2/alerts", + data=body, + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req, timeout=30) as resp: + return resp.status + + +def alert(name, severity, summary, description, **labels): + a = { + "labels": { + "alertname": name, + "severity": severity, + "namespace": "observability", + "service": "rule-liveness-guard", + }, + "annotations": {"summary": summary, "description": description}, + "startsAt": utc_now(), + } + a["labels"].update({k: str(v) for k, v in labels.items()}) + return a + + +# --------------------------------------------------------------------------- # +# Policy +# --------------------------------------------------------------------------- # +def load_policy(): + with open(POLICY_PATH, "r", encoding="utf-8") as fh: + p = json.load(fh) + exempt = {} + for e in p.get("expected_absent", []): + for field in ("rule", "reason", "expires"): + if field not in e: + raise RuntimeError("policy exemption missing %r: %r" % (field, e)) + exempt[e["rule"]] = e + return p, exempt + + +def main(): + started = time.time() + policy, exempt = load_policy() + now = int(time.time()) + start = now - LOOKBACK_DAYS * 86400 + + groups = prom_rules() + + live, dead, exempted, expired, unparseable, absence_based = [], [], [], [], [], [] + cache = {} + checked = 0 + + for g in groups: + for r in g["rules"]: + if r.get("type") != "alerting": + continue + checked += 1 + name = r["name"] + key = "%s/%s" % (g["name"], name) + try: + sels, uses_absence = extract_selectors(r["query"]) + except ParseError as exc: + unparseable.append((key, str(exc))) + continue + if uses_absence: + absence_based.append(key) + continue + if not sels: + continue + + counts = [] + for s in sels: + if s not in cache: + cache[s] = selector_series(s, start, end=now) + counts.append((s, cache[s])) + + errored = [s for s, c in counts if c < 0] + if errored: + unparseable.append((key, "series lookup failed for %s" % errored[0][:80])) + continue + + empty = [s for s, c in counts if c == 0] + if len(empty) != len(counts): + live.append(key) + continue + + # Every input is empty. WHY it is empty decides what to do about it. + kind, why = diagnose(empty[0], start, now, cache) + ex = exempt.get(name) or exempt.get(key) + if ex: + if ex["expires"] < utc_now()[:10]: + expired.append((key, ex)) + else: + exempted.append((key, ex)) + else: + dead.append((key, name, empty, kind, why)) + + # ---------------- report ---------------- + alerts = [] + # DORMANT is a warning, not a verdict: the target is up and the metric may + # simply not have been created yet (counters like *_failures_total appear on + # first occurrence). The other two kinds are structural and cannot self-heal. + severity_for = { + "DEAD-NO-TARGET": "critical", + "DEAD-LABEL-MISMATCH": "critical", + "DORMANT": "warning", + } + for key, name, empty, kind, why in dead: + alerts.append( + alert( + # Deliberately a DIFFERENT alertname per severity. + # + # The first live run of this guard emitted all nine findings as + # `AlertRuleDead` and Alertmanager suppressed six of them: the + # stack's default inhibit_rules drop severity=warning when a + # severity=critical shares the same (namespace, alertname), and + # every finding here carries namespace=observability. Six real + # findings were swallowed by a control that reported success — + # this guard reproducing, in its own delivery path, exactly the + # defect it was written to detect. + # + # Distinct alertnames make the two classes uninhibitable by each + # other. Do not merge them back into one name. + "AlertRuleDead" if severity_for.get(kind) == "critical" else "AlertRuleDormant", + severity_for.get(kind, "critical"), + "Alert rule %s cannot fire (%s)" % (name, kind), + "Rule %s evaluates against selectors that matched no series in the last %dd — %s. " + "It reports inactive forever regardless of cluster state, which on a dashboard is " + "indistinguishable from healthy. Fix the query or the scrape, delete the rule, or " + "declare it in infra/k8s/rule-liveness-guard/base/policy.json with a reason, an " + "expiry and a compensating control. Empty selectors: %s" + % (key, LOOKBACK_DAYS, why, ", ".join(e[:100] for e in empty[:3])), + rule=key, + kind=kind, + dead_selectors=str(len(empty)), + ) + ) + for key, exc in unparseable: + alerts.append( + alert( + "AlertRuleUnscannable", + "warning", + "Rule %s could not be checked for liveness" % key, + "The liveness guard could not evaluate %s (%s). An unscannable rule is treated as " + "a finding, not skipped — otherwise this guard would have the same blind spot it " + "exists to remove." % (key, exc), + rule=key, + ) + ) + for key, ex in expired: + alerts.append( + alert( + "AlertRuleExemptionExpired", + "warning", + "Dead-rule exemption for %s expired on %s" % (key, ex["expires"]), + "This rule is still dead and its declared exemption (%s) has expired. Re-justify " + "it or fix it." % ex["reason"], + rule=key, + ) + ) + + summary = { + "checked": checked, + "live": len(live), + "dead": len(dead), + "exempted": len(exempted), + "expired": len(expired), + "unscannable": len(unparseable), + "absence_based": len(absence_based), + "selectors_tested": len(cache), + "lookback_days": LOOKBACK_DAYS, + } + + body = { + "summary": summary, + "dead_rules": sorted("%s [%s]" % (k, kind) for k, _n, _e, kind, _w in dead), + "unscannable_rules": sorted(k for k, _ in unparseable), + "expired_exemptions": sorted(k for k, _ in expired), + "prometheus": PROM, + } + status = "succeeded" if not (dead or unparseable or expired) else "partial" + + delivered = None + if alerts and not DRY_RUN: + try: + post_alerts(alerts) + delivered = len(alerts) + except Exception as exc: # noqa: BLE001 - must never be silent + emit( + receipt( + "rule-liveness-scan", + "failed", + "prometheus/observability", + {"error": "alertmanager delivery failed: %s" % exc, **body}, + ) + ) + print("FATAL: could not deliver %d alerts: %s" % (len(alerts), exc), file=sys.stderr) + raise SystemExit(2) + + emit( + receipt( + "rule-liveness-scan", + status, + "prometheus/observability", + body, + metrics={**summary, "alerts_delivered": delivered or 0, + "duration_s": round(time.time() - started, 2)}, + policy_refs=["%s/policy.json" % SERVICE_REF], + evidence_refs=[PROM + "/api/v1/rules"], + ) + ) + + # Human-readable tail for `kubectl logs`. + print("checked=%(checked)d live=%(live)d DEAD=%(dead)d exempt=%(exempted)d " + "expired=%(expired)d unscannable=%(unscannable)d" % summary, file=sys.stderr) + for key, _name, empty, kind, why in dead: + print(" %-20s %s <- %s" % (kind, key, why), file=sys.stderr) + for key, exc in unparseable: + print(" UNSCANNABLE %s (%s)" % (key, exc), file=sys.stderr) + for key, ex in expired: + print(" EXPIRED-EXEMPTION %s (%s)" % (key, ex["expires"]), file=sys.stderr) + + # Exit 0 on a completed scan, EVEN WITH FINDINGS. + # + # The first in-cluster run exited 1 because it found dead rules, which + # tripped backoffLimit and left the Job in BackoffLimitExceeded. That would + # have meant the CronJob failed every hour for as long as any dead rule + # existed — a permanently-red Job that everyone learns to ignore, and + # RuleLivenessGuardFailing rendered meaningless. + # + # Findings travel by ALERT, which is the channel built for them and which is + # now actually delivered. A non-zero exit is reserved for the guard itself + # being broken: unreachable Prometheus, an empty rule corpus, or undelivered + # alerts (all of which raise above). "Found problems" is a successful scan. + return + + +if __name__ == "__main__": + main() diff --git a/infra/k8s/rule-liveness-guard/base/kustomization.yaml b/infra/k8s/rule-liveness-guard/base/kustomization.yaml new file mode 100644 index 000000000..38fceeb54 --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/kustomization.yaml @@ -0,0 +1,21 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: observability + +resources: + - cronjob.yaml + - prometheusrule-guard.yaml + +configMapGenerator: + - name: rule-liveness-guard + files: + - guard.py + - policy.json + +generatorOptions: + disableNameSuffixHash: true + +labels: + - includeSelectors: false + pairs: + bundle: observability diff --git a/infra/k8s/rule-liveness-guard/base/policy.json b/infra/k8s/rule-liveness-guard/base/policy.json new file mode 100644 index 000000000..185ccbbaa --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/policy.json @@ -0,0 +1,123 @@ +{ + "_comment": [ + "Declared exemptions for alert rules that legitimately have no series.", + "", + "Every entry needs `rule`, `reason` and `expires` (YYYY-MM-DD). Entries whose", + "underlying RISK still exists must also name a `compensating_control` -- the", + "thing that actually covers the hazard now that this rule cannot.", + "", + "An expired exemption raises AlertRuleExemptionExpired. That is deliberate:", + "the failure this whole guard exists to fix was a control everyone assumed", + "was working, so an allowlist that never expires would just relocate the", + "problem. Expiries are staggered so they do not all come due in one week.", + "", + "NOT exempted, on purpose: HellGraphDown and HighErrorRatio. Both are this", + "estate's own rules, both are dead, and both are fixable rather than", + "acceptable. They are left alerting until they are fixed." + ], + + "expected_absent": [ + { + "rule": "KubePersistentVolumeFillingUp", + "reason": "GKE Autopilot's Warden denies nodes/proxy cluster-wide, so the in-cluster Prometheus has no kubelet scrape job and kubelet_volume_stats_* can never exist here. Structural, not a misconfiguration.", + "compensating_control": "infra/k8s/pvc-capacity-guard (PR #1105) reads the same kubelet_volume_stats_* metrics from Google Managed Prometheus, which does scrape kubelet, and alerts PvcGuardExpanded/AtMaximum/CannotExpand/NoData.", + "expires": "2027-01-31" + }, + { + "rule": "KubePersistentVolumeInodesFillingUp", + "reason": "Same structural cause: no kubelet job in the in-cluster Prometheus.", + "compensating_control": "PENDING. pvc-capacity-guard tracks bytes, not inodes. Inode exhaustion on these volumes is currently UNCOVERED — tracked for Wave 0.", + "expires": "2026-10-31" + }, + { + "rule": "KubeletClientCertificateExpiration", + "reason": "No kubelet scrape job on Autopilot. Node certificates are managed by Google as part of the Autopilot control plane.", + "compensating_control": "GKE Autopilot node auto-repair and Google-managed node lifecycle.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletServerCertificateExpiration", + "reason": "No kubelet scrape job on Autopilot. Node certificates are Google-managed.", + "compensating_control": "GKE Autopilot node auto-repair and Google-managed node lifecycle.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletClientCertificateRenewalErrors", + "reason": "No kubelet scrape job on Autopilot.", + "compensating_control": "GKE Autopilot node auto-repair.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletServerCertificateRenewalErrors", + "reason": "No kubelet scrape job on Autopilot.", + "compensating_control": "GKE Autopilot node auto-repair.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletPlegDurationHigh", + "reason": "No kubelet scrape job on Autopilot. Kubelet internals are not our operational surface on a managed node pool.", + "compensating_control": "GKE Autopilot node auto-repair; pod-level symptoms are covered by KubePodNotReady and KubeContainerWaiting, which are live.", + "expires": "2027-01-31" + }, + { + "rule": "KubeletPodStartUpLatencyHigh", + "reason": "No kubelet scrape job on Autopilot.", + "compensating_control": "KubePodNotReady (live) covers the user-visible symptom.", + "expires": "2027-01-31" + }, + { + "rule": "CPUThrottlingHigh", + "reason": "Reads container_cpu_cfs_throttled_periods_total from cAdvisor, which is exposed via the kubelet — unreachable on Autopilot for the same Warden reason.", + "compensating_control": "PENDING. CPU throttling is currently UNOBSERVED in-cluster. GMP does publish container/cpu metrics; wiring an equivalent is Wave 0 work.", + "expires": "2026-10-31" + }, + { + "rule": "NodeRAIDDegraded", + "reason": "Autopilot nodes are single-disk COS VMs with no md RAID arrays. node_md_* will never exist on this platform.", + "expires": "2027-06-30" + }, + { + "rule": "NodeRAIDDiskFailure", + "reason": "No md RAID arrays on Autopilot nodes.", + "expires": "2027-06-30" + }, + { + "rule": "NodeBondingDegraded", + "reason": "No bonded network interfaces on Autopilot nodes.", + "expires": "2027-06-30" + }, + { + "rule": "NodeSystemdServiceFailed", + "reason": "node-exporter runs unprivileged in a container on Autopilot and cannot read the host systemd D-Bus socket, so the systemd collector reports nothing.", + "compensating_control": "GKE Autopilot node auto-repair owns node-level service health.", + "expires": "2027-01-31" + }, + { + "rule": "PrometheusRemoteStorageFailures", + "reason": "No remote_write configured on this Prometheus, so prometheus_remote_storage_* is never registered.", + "compensating_control": "N/A — there is no remote storage to fail. If remote_write is ever configured this exemption must be removed.", + "expires": "2027-01-31" + }, + { + "rule": "PrometheusRemoteWriteBehind", + "reason": "No remote_write configured.", + "expires": "2027-01-31" + }, + { + "rule": "PrometheusRemoteWriteDesiredShards", + "reason": "No remote_write configured.", + "expires": "2027-01-31" + }, + { + "rule": "AlertmanagerClusterFailedToSendAlerts", + "reason": "Single-replica Alertmanager, so alertmanager_cluster_* gossip metrics do not exist.", + "compensating_control": "AlertmanagerFailedToSendAlerts (live, non-cluster variant) covers delivery failure for this topology, and AlertDeliveryDead in infra/k8s/alert-delivery covers the whole path.", + "expires": "2027-01-31" + }, + { + "rule": "AlertmanagerMembersInconsistent", + "reason": "Single-replica Alertmanager: no cluster members to be inconsistent.", + "expires": "2027-01-31" + } + ] +} diff --git a/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml b/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml new file mode 100644 index 000000000..02def2151 --- /dev/null +++ b/infra/k8s/rule-liveness-guard/base/prometheusrule-guard.yaml @@ -0,0 +1,50 @@ +# Meta-monitoring for the rule-liveness guard itself. +# +# The guard exists because a control that cannot fail is not a control. That +# applies to the guard: if its CronJob stops running, the estate quietly loses +# dead-rule detection and looks exactly the same. So the guard's own liveness is +# a rule — and, being an ordinary rule in this Prometheus, it is scanned by the +# guard on the next run. The check checks the checker. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: rule-liveness-guard + namespace: observability + labels: + bundle: observability + release: kube-prometheus-stack +spec: + groups: + - name: rule-liveness-guard.meta + rules: + - alert: RuleLivenessGuardNotRunning + # kube_state_metrics publishes job completion times; the guard runs + # hourly, so a gap over 3h means it is not running at all. Note the explicit + # namespace label below: the absent() branch produces no labels of its own, + # and without it this alert would not match the AlertmanagerConfig sub-route + # — the alert saying detection is down would go to null. + expr: | + time() - max(kube_job_status_completion_time{job_name=~"rule-liveness-guard-.*"}) > 10800 + or absent(kube_job_status_completion_time{job_name=~"rule-liveness-guard-.*"}) + for: 30m + labels: { severity: warning, namespace: observability } + annotations: + summary: "rule-liveness-guard has not completed in over 3h" + description: >- + Dead-rule detection has stopped. Alert rules can silently become + incapable of firing again with nothing to notice. + runbook: infra/k8s/rule-liveness-guard/README.md + + - alert: RuleLivenessGuardFailing + expr: | + increase(kube_job_status_failed{job_name=~"rule-liveness-guard-.*"}[2h]) > 2 + for: 10m + labels: { severity: warning, namespace: observability } + annotations: + summary: "rule-liveness-guard is failing repeatedly" + description: >- + The guard exits non-zero ONLY when it is itself broken — + unreachable Prometheus, an empty rule corpus, or alerts it could + not deliver. Findings do not fail the job (they travel by alert), + so a failing job means dead-rule detection is down, not that dead + rules exist. Check the pod logs for the EvidenceReceipt.