Harden Ray cluster isolation: NetworkPolicy authz boundary, securityContext, quotas - #28
Open
brandonrc wants to merge 9 commits into
Open
Harden Ray cluster isolation: NetworkPolicy authz boundary, securityContext, quotas#28brandonrc wants to merge 9 commits into
brandonrc wants to merge 9 commits into
Conversation
…NetworkPolicy, quotas)
Ray has no native auth on the client (10001), GCS (6379), or dashboard/job
API (8265) ports — any pod that can reach them can execute arbitrary code.
This makes NetworkPolicy the in-cluster authorization boundary and asserts
pod-level hardening:
- Dedicated ServiceAccount with automountServiceAccountToken: false (Ray
never talks to the K8s API).
- Default pod/container securityContext matching the stock rayproject/ray
runtime user (1000:100), runAsNonRoot, seccomp RuntimeDefault, no
privilege escalation, all capabilities dropped — restricted-PSS
compatible; override with {} for root-running custom images. The
orgCABundle initContainer gets an explicit non-root securityContext in
the same change so runAsNonRoot doesn't reject it.
- Opt-in NetworkPolicy (networkPolicy.enabled): intra-Ray allow-all +
kuberay-operator control-plane ports on all Ray pods; gateway
namespaces and explicitly allow-listed cross-namespace clients on the
head only. Selectors use ray.io/is-ray-node / ray.io/node-type (never
ray.io/cluster — RayService generates random-suffixed names, and
zero-downtime upgrades run two clusters at once). Ingress-only: Ray
egress is legitimately broad.
- Opt-in ResourceQuota + LimitRange for one-RayService-per-namespace
isolation; quota must be sized for 2x a cluster (upgrade overlap).
- New head/worker nodeSelector, affinity, priorityClassName values for
dedicated node-pool pinning.
…g fields, NOTES - README Security section: threat model (unauthenticated Ray ports; Keycloak/NebariApp only covers external hostnames), NetworkPolicy design table, allowedClients recipes for checkmaite (10001 only) and Jupyter, pod-hardening defaults and how to disable them, quota 2x sizing rule, restricted-PSS note. - Extend the ArgoCD footgun: /spec/rayClusterConfig ignoreDifferences + RespectIgnoreDifferences silently drops all new pod-spec hardening fields (securityContext, serviceAccountName, automount, nodeSelector, affinity, priorityClassName), same as orgCABundle; verify against the running pod, not sync status. - NOTES.txt: print the effective netpol allow-list when enabled; warn that Ray ports are unauthenticated when disabled.
…kPolicy kind job - lint: template permutations for networkPolicy (+allowedClients render assertions), resourceQuota+limitRange, orgCABundle+scheduling (proves the initContainer renders non-root alongside the pod runAsNonRoot default), and a default-render guard pinning securityContext / serviceAccount wiring on both pod specs. - test: enable networkPolicy on the standalone job (apply-cleanly smoke only — kindnet doesn't enforce NetworkPolicy) and add a netpol-enforced job: kind with disableDefaultCNI + Calico, asserts both policies exist, an unlisted pod is denied at head:8265, an allow-listed namespace gets through, and RayService Ready structurally proves the kuberay-operator ingress rule. - test-integration: netpol smoke on the full nebari stack (kindnet, non-enforcing — commented as such).
…orward jobs Two CI fixes surfaced by this PR's first run: - The chart rename to nebari-rayserve-pack (#24) changed the stable service names to <release>-nebari-rayserve-pack-{head,serve}-svc, but the Test/Integration workflows, README, and dev Makefile still said <release>-nebari-rayserve-{head,serve}-svc — the standalone job's dashboard health check has been failing on main since 0.4.0. Fix all references. - Recent kind versions enforce NetworkPolicy by default (kindnet ships kube-network-policies), so port-forward-based health checks break the moment networkPolicy.enabled=true — that traffic matches no allow rule. Keep the standalone and integration jobs netpol-OFF baselines and concentrate enforcement coverage in the netpol-enforced job, which uses in-cluster client pods. Also fix its Calico install racing CRD establishment (custom-resources.yaml applied before the Installation CRD existed).
Same #24-rename fallout as the service names: the NebariApp resources are rayserve-nebari-rayserve-pack{,-dashboard} since 0.4.0, but the integration workflow (failing on main since then) and dev Makefile still waited on the old names.
…l matches The allow-listed-namespace curl ran once immediately after creating client-ns; Calico's dataplane can take a few seconds to propagate the namespaceSelector match for a brand-new namespace, making the one-shot check flaky (passed run 1, failed run 2). Retry with fresh pods via kubectl run --rm --attach, which propagates curl's exit code directly.
The upgrade step only re-set the nebariapp flags, silently reverting head/worker resources and worker.replicas to chart defaults. That spec change triggers a RayService zero-downtime cluster replacement (old + new clusters concurrently) that doesn't reliably fit the runner, so helm --wait timed out intermittently. The auth flip is not supposed to touch the Ray cluster; repeating the overrides makes rayClusterConfig a no-op on upgrade.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Hardens the Ray cluster this chart deploys. Ray has no native authentication on the client port (10001), GCS (6379), or dashboard/job API (8265) — any pod that can reach those ports can execute arbitrary code on the cluster. Keycloak OIDC (NebariApp) only protects the external hostnames; in-cluster traffic bypasses the gateway entirely. This PR makes NetworkPolicy the in-cluster authorization boundary and asserts pod-level hardening.
Changes
networkPolicy.enabled) — two Ingress-only policies:gatewayNamespaces→ 8265/8000, plus explicitly allow-listedallowedClients(namespace + optional podSelector + ports, default[10001, 8000, 8265])ray.io/is-ray-node/ray.io/node-type, neverray.io/cluster(random suffix; zero-downtime upgrades run two clusters)orgCABundleexists)RuntimeDefault+ no privilege escalation + drop ALL, matching what the stockrayproject/rayimage already does at runtime, so default installs behave identically but now satisfy the restricted PSS. TheorgCABundleinitContainer gets an explicit non-root securityContext in the same change (alpine would otherwise be rejected byrunAsNonRoot). Override with{}for root-running custom images.automountServiceAccountToken: false— Ray never talks to the K8s API.head/worker.nodeSelector,affinity,priorityClassNamefor dedicated node pools (GPU toleration auto-injection unchanged).ignoreDifferencesfootgun (all new pod-spec fields live under/spec/rayClusterConfigand are silently dropped by the documented sync policy), NOTES warnings.netpol-enforcedkind job — kindnet doesn't enforce NetworkPolicy, so this job installs Calico and proves an unlisted pod is denied at head:8265, an allow-listed namespace gets through, and RayService reaching Ready proves the operator ingress rule.0.4.1 → 0.5.0.Verified locally (kind + Calico, enforced)
curl head-svc:8265→ denied (timeout)/api/version)main: only the intended fields (SA, automount, securityContexts) — no behavioral change for stock imagesBreaking / follow-ups
podSecurityContext: {}+containerSecurityContext: {}(documented).RAY_USE_TLS) is a tracked follow-up — NetworkPolicy restricts who can connect, but traffic is plaintext and any allowed client is fully trusted by GCS.