Central, reusable validation for observability rules (Mimir + Loki) and dashboards

[QuentinBisson]

Problem

The giantswarm/prometheus-rules validation pipeline is repo-local bash (test/hack/bin/*: template-chart.sh, verify-rules.sh → promtool, run-pint.sh + shared pint-*.hcl, check-runbooks.sh, checkLabels/main.go, lokitool for Loki rules). Any other repo that ships alerting rules in its own chart — e.g. giantswarm/tunnelport (see giantswarm/tunnelport#71) — can't reuse it without copy-pasting the harness, which drifts. We already hit vendored pint 0.64.0 disagreeing with the checked-in config.

As more apps ship their own PrometheusRules (and dashboards), we want one validated, versioned gate that any repo can opt into — not N forks of the checks.

Proposal

A single, reusable validation surface, delivered via devctl (the same way check-values-schema already reaches every repo as a generated reusable workflow), covering:

1. Mimir / metric rules — promtool check rules + promtool test rules (unit tests).
2. Loki / log rules — lokitool (rules carrying application.giantswarm.io/prometheus-rule-kind: loki).
3. pint — the shared pint-*.hcl (mandatory description/area/severity/runbook_url, aggregation-label preservation, etc.).
4. runbook existence + inhibition-label validation.
5. Dashboards (phase 2) — Grafana JSON/schema + datasource lint (distinct tooling; sequence after the rule kinds land).

Delivery options (atlas to choose):

• devctl-generated reusable GitHub Actions workflow in giantswarm/github-workflows, pinned tool versions, consumed via uses: and regenerated fleet-wide by devctl gen. Lowest friction, matches the existing zz_generated.* pattern. Preferred.
• A small Go CLI (embedding the config) for local + CI use, wrapped by the generated workflow.

Single source of truth for pint config + valid inhibition-label set + tool versions + Loki/Mimir tooling. No forking the checks.

Open design question

How does the validator find the rules/dashboards to check across repos:

• (a) discover by convention/label (e.g. a well-known application.giantswarm.io/* label + path glob the tool selects on), or
• (b) a per-repo config file pointing at the globs?

(a) is zero-config but needs a label/path convention; (b) is explicit but per-repo. Atlas owns prometheus-rules and the check definitions, so this is their call.

Context / first consumer

giantswarm/tunnelport ships a PrometheusRule in its chart (giantswarm/tunnelport#71) and currently has no CI validating it — we deliberately removed a throwaway per-repo workflow in favor of this shared solution. It's the first concrete consumer.