Follow-up: in-transit mTLS for Ray channels (RAY_USE_TLS + cert-manager)

[brandonrc]

Context

The hardening PR makes NetworkPolicy the in-cluster authorization boundary for Ray's unauthenticated ports (10001 client, 6379 GCS, 8265 dashboard/job API). That restricts who can connect, but:

• traffic between Ray nodes and from clients is plaintext
• any allow-listed client is fully trusted by GCS — there is no per-client identity, no authn, no authz once a connection is admitted

Proposal

Support Ray's in-transit TLS (tls-auth docs) as an opt-in chart feature:

• tls.enabled values gate
• a cert-manager Certificate per cluster (chart-templated, cert-manager becomes a soft dependency)
• RAY_USE_TLS=1 + RAY_TLS_SERVER_CERT / RAY_TLS_SERVER_KEY / RAY_TLS_CA_CERT env on head and worker
• an initContainer (or cert-manager csi-driver) producing per-pod certs with correct SANs

Known blockers / open questions

• SAN generation vs RayService naming: RayService generates RayCluster names with a random suffix, so head/worker pod DNS names aren't statically known at template time. Options: wildcard SANs for the namespace (*.<ns>.svc), or per-pod cert issuance via csi-driver.
• Performance: Ray docs note TLS on gRPC channels costs meaningful GCS throughput (benchmarks ~10–20%).
• Client-side distribution: every client (checkmaite api, Jupyter notebooks via data-science-pack) needs the CA cert + a client cert mounted and the same RAY_USE_TLS env — needs a documented pattern per consumer pack.
• Interaction with the zero-downtime upgrade flow (two clusters live simultaneously, one Certificate?).

References

• Hardening PR: Harden Ray cluster isolation: NetworkPolicy authz boundary, securityContext, quotas #28
• README Security section documents this as the known residual risk.