Default-deny east-west networking for observability and network namespaces

[kelchm]

Summary

20 CiliumNetworkPolicies exist and are genuinely default-deny, but zero in observability, network, flux-system, or most of kube-system. Many currently unconfined pods can reach unauthenticated in-cluster APIs: Loki runs auth_enabled: false, and the Prometheus/VMSingle/VictoriaLogs query APIs have no auth layer. Grafana is the exception — it is Kanidm-OAuth-authenticated (kubernetes/apps/observability/grafana/app/helmrelease.yaml:105) — but still lacks egress hygiene. A compromised pod can read all logs and query all metrics.

The namespace-wide default-deny pattern already exists in-tree: kubernetes/apps/ai/networkpolicy.yaml.

Proposed work

• Apply the ai/ pattern to observability/: namespace-wide endpointSelector: {} default-deny on ingress and egress, plus per-workload allows (Alloy→both log backends, both metric stacks→scrape targets, Grafana→datasources and Kanidm, vmalert→Alertmanager).
• Then network/ (external-dns→Cloudflare, cloudflared→edge, k8s-gateway, traefik backends).
• kube-system/flux-system as a separate, careful follow-up phase — Cilium's own namespaces need conservative allowlists to avoid self-lockout.

Acceptance criteria

• observability and network have base default-deny ingress+egress CNPs with explicit per-workload allows.
• Positive tests: stack-to-stack scrape, Alloy dual-write, and Grafana datasource flows work post-rollout.
• Negative test: a pod in an unrelated namespace cannot reach Loki/Prometheus/VMSingle APIs.

Related

• Reduce unnecessary network exposure and verify perimeter policy #226 (north-south perimeter — this is the east-west counterpart)
• Wire actionable alert delivery and restore control-plane metric coverage #218
• Restrict AI and MCP workloads to approved LAN dependencies #295

[kelchm]

Status: the observability and network phases are live.

• feat(network): default-deny east-west traffic in observability and network namespaces #488 landed the baselines and per-workload policies; fix(observability,network): set ndots:1 so Cilium toFQDNs rules match #489 (ndots:1 for toFQDNs), fix(observability): grant kube-apiserver egress to config-reloader pods #490 (config-reloader API access), and fix(network): repair default-deny rollout regressions #491 (gateway hairpin identities, valid - {} deny form, Grafana plugin redirect, VM sync-job policy, CI validator) repaired the rollout regressions.
• fix(observability,network): close default-deny scrape and Loki gaps #492 closes what the rollout audits missed: reloader metrics ports on the four in-namespace scrape targets, Loki's rules sidecar and usage reporting, and the traefik ServiceMonitors that never had a metrics Service to select.

Acceptance criteria above are ticked: both baselines are Valid=True with per-workload allows, the positive flows (scrapes, Alloy dual-write, Grafana datasources) are healthy, and the negative test (unrelated-namespace pod → Loki/VMSingle/VictoriaLogs denied) passed during the #488 rollout.

Remaining scope for this issue: the kube-system/flux-system phase.