[Hanson-Tsai]

Summary

Jira: OTTERSCALE-288. Any Pod in the cluster can read prometheus-stack directly today, a tenant's Pod included:

• Prometheus :9090: every namespace's metrics.
• Alertmanager :9093: including silence creation.
• kube-state-metrics :8080: resource names across namespaces.
• node-exporter :9100 on each node.

This PR puts an authenticating, authorizing guard in front of each of them, in the chart defaults, so Dashboard installs on external clusters and open-source users get it too.

prometheus-stack 1.1.0 → 1.2.0

component	before	after	who may read
Prometheus	http :9090	listenLocal, plus a kube-rbac-proxy sidecar on https :9091	get/create on monitoring.coreos.com prometheuses/api
Alertmanager	http :9093	listenLocal, plus a kube-rbac-proxy sidecar on https :9095	alertmanagers/api
node-exporter	:9100 on the host	subchart kubeRBACProxy on :9100; the exporter moves to 127.0.0.1:8100	get services/prometheus-stack-prometheus-node-exporter
kube-state-metrics	http :8080	KSM's own --auth-filter plus --tls-config	nonResourceURL /metrics get

• One cert-manager CA (templates/kube-rbac-proxy-tls.yaml) signs all four serving certs.
• Prometheus scrapes and sends alerts through the guards with its own ServiceAccount token, verifying the certificates (ca.crt + serverName).
• Prometheus and Alertmanager ServiceAccounts are bound to system:auth-delegator.
• Prometheus gets get on prometheuses/api (to scrape itself) and get/create on alertmanagers/api (to scrape and send alerts) via additionalRulesForClusterRole. It has no create on its own API, so its token cannot POST /-/reload or /-/quit through the guard.
• externalUrl is https, so alert links work.
• config-reloader moves to localhost along with listenLocal. Its metrics and the ConfigReloaderSidecarErrors alert are given up; the dead scrape endpoints are dropped by relabeling.
• validate.yaml refuses to render if any of these is changed, because the guard wiring hardcodes them:
  • a release name other than prometheus-stack;
  • a namespace other than monitoring;
  • any name override;
  • a namespaceOverride on kube-state-metrics or node-exporter;
  • a Prometheus or Alertmanager ServiceAccount other than the default name.
• Prometheus image pinned to quay.io/prometheus/prometheus:v3.14.0. With listenLocal, the operator's probes run sh -c, and the upstream default -distroless image has no shell, so Prometheus would never become Ready. A renovate rule keeps the tag off -distroless.

Why KSM is not behind kube-rbac-proxy. In the subchart's kubeRBACProxy mode, the KSM container declares no ports, but its liveness and readiness probes name http and metrics. kubelet resolves probe port names per container, so the pod never becomes Ready (strconv.Atoi: parsing "http"). This is the case in the bundled 8.4.2 and in the latest 8.6.0. KSM's built-in --auth-filter does the same TokenReview + SubjectAccessReview (controller-runtime WithAuthenticationAndAuthorization) on /metrics, and leaves /livez//readyz open.

otterscale-agent 1.1.2 → 1.2.0

• New ClusterRole <release>-metrics:
  • get/create on prometheuses/api for the Prometheus
  • get on the CA Secret, restricted by resourceNames
    It is a ClusterRole rather than a Role because monitoring may not exist yet when the agent is installed.
• New values metrics.{prometheusURL,prometheusName,caSecret}, passed to the agent as env. An empty metrics.prometheusURL fails rendering: the agent would treat an empty env as unset and use its default.
• New ClusterRole otterscale:metrics-cluster-view: get on prometheuses/api for that Prometheus. It is the same attribute the guard checks and the hub's cluster scope asks about (feat: guard metrics in the cluster and authorize them per scope at the hub otterscale#310). Bind it to give someone cluster-wide metrics without cluster-admin. Do not bind it to system:authenticated: that includes every ServiceAccount, so any Pod could then query Prometheus directly. Bind an oidc:-prefixed group instead.

kubevirt 1.1.1 → 1.2.0

• The PodMonitor sets honorLabels: true, so a VM metric's namespace is the VM's own, not kubevirt. The hub's namespace scope and the new dashboard select VM metrics by namespace.
• This changes label semantics (exported_namespace is gone), hence a minor bump. Upgrade it together with the new hub and dashboard: an old dashboard reads exported_namespace.

⚠️ Breaking for anyone reading Prometheus directly

• :9090 is gone. Use https :9091 with a token authorized as above, and the CA from Secret prometheus-stack-kube-prom-prometheus-tls.
• Alertmanager moved from :9093 to :9095.
• A bundled Grafana, if enabled, needs its datasource reconfigured.

The details are in templates/NOTES.txt. Should this be 2.0.0 instead of 1.2.0?

🚧 Draft: do not merge yet

release.yml publishes every new chart version on push to main. The agent chart's default image.tag is still v1.5.0, and that agent cannot use the https path, so the dashboards would be empty. No token leaks: v1.5.0 sends none.

1. Merge the otterscale PR and cut a release.
2. Bump otterscale-agent appVersion and image.tag here to that release.
3. Merge. CI then publishes prometheus-stack 1.2.0, otterscale-agent 1.2.0 and kubevirt 1.2.0.

Testing

• make test runs the render tests now committed under tests/ (also in CI, .github/workflows/test.yml): 26 checks.
  • prometheus-stack, 16 checks:
    • guards, TLS, ServiceMonitors and RBAC;
    • probe port names resolving in the same container;
    • the validate and override failures;
    • the Prometheus image has a shell while listenLocal is on;
    • kube-prometheus-stack.enabled=false;
    • names matching across the two charts and the agent defaults.
  • otterscale-agent, metrics-cluster-view, and the kubevirt PodMonitor.
  • make test builds the prometheus-stack subchart dependency when it is missing.
• helm lint passes on prometheus-stack, otterscale-agent and kubevirt.
• Deployed to a single-node lab cluster (RKE2 1.35), upgrading from 1.1.0 / 1.1.2 with the helm CLI; this update was re-applied there:
  • From a Workspace Pod: :9090/:9093 refuse connections. All four guards return 401 without a token and 403 with the Pod's own token.
  • Targets: Prometheus, Alertmanager, KSM and node-exporter are all up over https. Watchdog reaches Alertmanager over https :9095.
  • The 85 dashboard queries return the same results as before.
  • Re-rendering this update changed only the Prometheus ClusterRole; no Pod restarted.
• Not verified: a Flux-driven upgrade from 1.1.0, multi-node clusters, and externally managed clusters.

Related

• otterscale/otterscale (merge first): feat: guard metrics in the cluster and authorize them per scope at the hub otterscale#310
• otterscale/dashboard: feat(dashboard): show metrics at cluster, namespace and shared scope dashboard#744
• phison-ai/hardened-images: https://github.com/phison-ai/hardened-images/pull/9
• phison-ai/release: https://github.com/phison-ai/release/pull/48

🤖 Generated with Claude Code