Investigate Cilium WireGuard encryption for inter-node pod traffic

[kelchm]

Context

PR #432 adds a parallel MCPHub deployment whose upstream MCP servers currently expose plain HTTP inside the cluster. The associated review comment claims MCPHub bearer credentials could be observed on those connections.

That credential-leak claim does not apply to the current configuration: MCPHub terminates the client bearer token, none of the upstream definitions specifies headers, OAuth, or passthroughHeaders, and MCPHub only forwards explicitly allowlisted request headers. The finding nevertheless highlights a broader question: MCP tool inputs and results can cross the physical network in plaintext when client and server pods run on different nodes.

This is not considered a blocker for PR #432. Investigate whether transparent Cilium WireGuard encryption is the appropriate cluster-level hardening instead of adding TLS sidecars and certificate management to every backend.

Cilium is currently pinned to 1.20.1 and uses native routing, direct node routes, endpoint routes, kube-proxy replacement, and Multus. The physical interfaces have an MTU of 1500. Cilium documents WireGuard as transparent encryption between Cilium-managed endpoints: https://docs.cilium.io/en/stable/security/network/encryption-wireguard/

Investigation

• Define the threat model and decide whether protecting inter-node pod payloads from LAN observation is required.
• Verify Talos kernel support and Cilium 1.20.1 compatibility with the cluster's native-routing, endpoint-route, kube-proxy-replacement, and Multus configuration.
• Confirm bidirectional UDP 51871 reachability between every node and identify any UniFi or host firewall changes.
• Determine the effective pod MTU and validate that WireGuard overhead does not cause fragmentation or PMTU failures, especially for long-lived SSE and streamable HTTP connections.
• Confirm which traffic is and is not covered, including same-node pod traffic, host-network traffic, and Multus secondary-network traffic such as Longhorn's storage network.
• Measure likely CPU and throughput impact on the three nodes.
• Compare Cilium WireGuard with per-workload TLS/mTLS proxies only if the threat model requires workload identity or same-node encryption.
• Design a staged rollout, observability checks, and a tested rollback path. Do not enable beta node encryption without a separate demonstrated need.

Proposed validation if enabled

• Render and validate the Flux/Helm changes before merge.
• Keep Cilium's rolling update at maxUnavailable: 1 and verify cluster health after each agent replacement.
• Verify Cilium status and encryption status on every node.
• Confirm WireGuard peers and handshakes between all nodes.
• Exercise cross-node DNS, ClusterIP, MCP streamable HTTP/SSE, ingress, egress, and Multus-backed storage traffic.
• Use a controlled packet capture to confirm cross-node Cilium pod payloads are encrypted while documenting expected exclusions.
• Record the rollback procedure and any temporary packet loss observed during device reconfiguration.

Completion criteria

Produce a documented go/no-go recommendation. If the result is go, implement it in a separate reviewed PR with pod-to-pod WireGuard encryption only (encryption.enabled=true, encryption.type=wireguard) unless the investigation provides evidence for a broader scope. Document residual risks, especially the intentional lack of same-node encryption.