From 9bb4881117b24f31371f6bf11c8d63f069001002 Mon Sep 17 00:00:00 2001 From: Elijah Rodriguez-Beltran Date: Tue, 18 Aug 2026 20:56:00 +0000 Subject: [PATCH] docs: add guide for CSI based external volumes TAG=agy CONV=872d25ed-2cb8-4575-a986-6a9b315f1fb6 --- docs/api-guide.md | 2 +- docs/csi-deployment.md | 316 +++++++++++++++++++++++++++++++++++++++++ docs/csi-volumes.md | 184 ++++++++++++++++++++++++ 3 files changed, 501 insertions(+), 1 deletion(-) create mode 100644 docs/csi-deployment.md create mode 100644 docs/csi-volumes.md diff --git a/docs/api-guide.md b/docs/api-guide.md index 4ef0bba220..cebb4bd93d 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -167,7 +167,7 @@ The `ActorTemplate` defines the code, environment, and state-management policies | `sandboxClass` | `string` | Optional. The sandbox runtime family this template's actors require: `gvisor` (default) or `microvm`. Only `WorkerPool`s whose `sandboxClass` matches are eligible. | | `workerSelector` | `*LabelSelector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels. If unset, all pools are eligible (subject to the actor's own `worker_selector`). | | `snapshotsConfig` | `SnapshotsConfig` | **Required.** The base object-storage location snapshots are written under, plus the pause/commit/resume scopes. See [Snapshot Storage Layout](#snapshot-storage-layout). | -| `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `externalVolumeTemplate`, or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `microvm` template may declare several `durableDir` volumes; a `gvisor` template is limited to one, and `externalVolumeTemplate` is `gvisor`-only. | +| `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `externalVolumeTemplate` (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `microvm` template may declare several `durableDir` volumes; a `gvisor` template is limited to one. | | `resources` | `*ResourceRequirements` | Optional. Declares each actor's compute size via `limits` — see [Sandbox Right-Sizing](#sandbox-right-sizing-specresources). Immutable, like the rest of the spec. | The sandbox itself — the binaries (e.g. the gVisor `runsc` binary) and the `pauseImage` holding the sandbox's namespaces — is **not configured on the `ActorTemplate`**. It is resolved from the referenced `WorkerPool`'s [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) — by name (`workerPool.spec.sandboxConfigName`) or, by default, the cluster default `SandboxConfig` for the pool's `sandboxClass`. diff --git a/docs/csi-deployment.md b/docs/csi-deployment.md new file mode 100644 index 0000000000..f65fe8452e --- /dev/null +++ b/docs/csi-deployment.md @@ -0,0 +1,316 @@ +# Deploying CSI Drivers for Agent Substrate + +This guide explains how to deploy and configure Container Storage Interface (CSI) drivers to work with Agent Substrate. + +In standard Kubernetes, CSI drivers are deployed to interact with asynchronous sidecars and kubelet. Substrate directly orchestrates storage operations from its control plane (`ateapi`) and node daemon (`atelet`), introducing specific deployment and networking requirements for both the **CSI Controller** and the **CSI Node DaemonSet**. + +--- + +## 1. Overview of Deployment Differences + +| Component | Standard Kubernetes CSI Deployment | Agent Substrate CSI Deployment | +| :--- | :--- | :--- | +| **CSI Controller Communication** | Controller listens strictly on an in-pod Unix domain socket (`/csi/csi.sock`). Sidecars (`csi-provisioner`, `csi-attacher`) watch `etcd` and talk to the driver locally. | The Substrate control plane (`ateapi`) communicates directly with the CSI Controller via gRPC over the cluster network. The controller must be exposed over TCP via a Kubernetes `Service`. | +| **Controller Authentication** | None (in-pod socket communication only). | Optional but recommended: Mutual TLS (mTLS) authenticated via Substrate's SPIFFE Pod Identity certificates and dynamic CA rotation. | +| **CSI Node Communication** | `kubelet` communicates with the node plugin Unix domain socket. | Substrate's node daemon (`atelet`) connects directly to the CSI node plugin Unix socket (discovered via `CSIDriverConfig`). | +| **Node Mount Propagation** | `kubelet` manages mounts under `/var/lib/kubelet/pods`. | The CSI Node plugin must mount Substrate target directories on the host (e.g. `/var/lib/ateom-gvisor`) with `mountPropagation: Bidirectional` so that `atelet` can bind-mount them into actor sandboxes. | + +--- + +## 2. CSI Controller Deployment Requirements + +Because Substrate's control plane (`ateapi`) invokes CSI Controller methods directly (`CreateVolume`, `ControllerPublishVolume`, `ControllerUnpublishVolume`, `DeleteVolume`), the CSI Controller's gRPC endpoint must be reachable across the cluster network. + +### Exposing the Controller Endpoint + +Standard CSI controller deployments can be made network-accessible and secured in one of two ways: + +1. **TLS Proxy Sidecar with Envoy (Recommended):** Add an Envoy reverse proxy sidecar container to the CSI Controller Deployment/StatefulSet. Envoy terminates incoming TLS/mTLS gRPC connections from `ateapi` and forwards the requests over HTTP/2 to the local Unix domain socket (`/csi/csi.sock`). +2. **Native TCP Endpoint:** If the CSI driver binary natively supports listening on a network socket (e.g. `--endpoint=tcp://0.0.0.0:`), configure the driver container to bind directly to a network port. + +### Exposing via a Kubernetes `Service` + +Create a Kubernetes `Service` targeting the CSI Controller pods. This provides a stable DNS name and load balancing across controller replicas. + +### Securing with mTLS and Pod Identity + +When exposing the CSI Controller over the network, communication should be secured with mutual TLS (mTLS). In the [`CSIDriverConfig`](csi-volumes.md#2-dynamic-csi-driver-discovery-csidriverconfig) resource, configure the following fields under `spec.tls`: + +* `enabled: true`: Enables TLS/mTLS for gRPC communication between `ateapi` and the CSI Controller service. If `false` (or omitted), communication falls back to unencrypted plaintext gRPC. +* `usePodIdentity: true`: Instructs `ateapi` to authenticate using Substrate's SPIFFE Pod Identity client certificate (`/run/podidentity.podcert.ate.dev/credential-bundle.pem`) and verify the controller's server certificate using Substrate's dynamic Service DNS CA trust bundle (`/run/servicedns.podcert.ate.dev/trust-bundle.pem`). + +> [!IMPORTANT] +> **What happens if `usePodIdentity` is `false`?** +> Currently, Substrate requires `usePodIdentity: true` whenever `spec.tls.enabled` is `true`. If `usePodIdentity` is set to `false` (or omitted when TLS is enabled), the `CSIDriverConfig` resource will be rejected by CRD validation (`tls.usePodIdentity must be true when tls.enabled is true; manual certificates are not yet supported`), and `ateapi` will return an error at runtime. + +### Restricting Controller Communication to `ateapi` + +Because the CSI Controller executes privileged storage operations (such as creating and deleting volumes), network access should be restricted to the Substrate control plane (`ateapi`). This can be enforced via the Envoy proxy sidecar. + +In the Envoy reverse proxy sidecar, enable client certificate validation (`require_client_certificate: true`) and configure the validation context with the Substrate Pod Identity CA (`signerName: podidentity.podcert.ate.dev/identity`). In `match_typed_subject_alt_names`, specify `ateapi`'s exact SPIFFE SAN: +```yaml +match_typed_subject_alt_names: +- san_type: URI + matcher: + exact: "spiffe://cluster.local/ns/ate-system/sa/ate-api-server" +``` +Envoy will reject any connection from clients that do not present a valid certificate issued to `ateapi`. + +### Example: `csi-nfs` Controller Deployment with Envoy TLS Proxy + +The following example demonstrates configuring the `csi-nfs-controller` Deployment with an **Envoy sidecar proxy** to terminate mTLS, authenticate with Service DNS certificates, validate `ateapi`'s SPIFFE identity via Pod Identity CA, and forward gRPC traffic to `/csi/csi.sock`: + +#### 1. Envoy Proxy ConfigMap + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: csi-nfs-envoy-config + namespace: kube-system +data: + envoy.yaml: | + static_resources: + listeners: + - name: grpc_listener + address: + socket_address: + address: 0.0.0.0 + port_value: 10000 + filter_chains: + - transport_socket: + name: envoy.transport_sockets.tls + typed_config: + "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext + require_client_certificate: true + common_tls_context: + alpn_protocols: + - h2 + tls_certificates: + - certificate_chain: + filename: /run/servicedns/credential-bundle.pem + private_key: + filename: /run/servicedns/credential-bundle.pem + watched_directory: + path: /run/servicedns + validation_context: + trusted_ca: + filename: /run/podidentity-ca/trust-bundle.pem + watched_directory: + path: /run/podidentity-ca + match_typed_subject_alt_names: + - san_type: URI + matcher: + exact: "spiffe://cluster.local/ns/ate-system/sa/ate-api-server" + filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + stat_prefix: grpc_csi + http2_protocol_options: {} + route_config: + name: local_route + virtual_hosts: + - name: csi_controller + domains: ["*"] + routes: + - match: + prefix: "/" + route: + cluster: csi_unix_socket + http_filters: + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + clusters: + - name: csi_unix_socket + connect_timeout: 0.25s + type: STATIC + typed_extension_protocol_options: + envoy.extensions.upstreams.http.v3.HttpProtocolOptions: + "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions + explicit_http2_config: {} + load_assignment: + cluster_name: csi_unix_socket + endpoints: + - lb_endpoints: + - endpoint: + address: + pipe: + path: /csi/csi.sock +``` + +#### 2. Envoy Sidecar in `csi-nfs-controller` Deployment + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: csi-nfs-controller + namespace: kube-system +spec: + replicas: 1 + selector: + matchLabels: + app: csi-nfs-controller + template: + metadata: + labels: + app: csi-nfs-controller + spec: + containers: + # Upstream CSI NFS driver container listening on local Unix socket + - name: nfs + image: registry.k8s.io/sig-storage/nfsplugin:v4.13.4 + args: + - "--nodeid=$(NODE_ID)" + - "--endpoint=unix:///csi/csi.sock" + volumeMounts: + - name: socket-dir + mountPath: /csi + # Envoy proxy sidecar terminating mTLS and forwarding to /csi/csi.sock + - name: envoy-proxy + image: envoyproxy/envoy:v1.34-latest + args: + - "-c" + - "/etc/envoy/envoy.yaml" + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: envoy-config + mountPath: /etc/envoy + - name: servicedns-certs + mountPath: /run/servicedns + readOnly: true + - name: podidentity-ca + mountPath: /run/podidentity-ca + readOnly: true + volumes: + - name: socket-dir + emptyDir: {} + - name: envoy-config + configMap: + name: csi-nfs-envoy-config + - name: servicedns-certs + projected: + sources: + - podCertificate: + signerName: servicedns.podcert.ate.dev/identity + keyType: ECDSAP256 + credentialBundlePath: credential-bundle.pem + - name: podidentity-ca + projected: + sources: + - clusterTrustBundle: + signerName: podidentity.podcert.ate.dev/identity + labelSelector: + matchLabels: + podcert.ate.dev/canarying: live + path: trust-bundle.pem +``` + +#### 3. Kubernetes `Service` for the Controller + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: csi-nfs-controller + namespace: kube-system +spec: + selector: + app: csi-nfs-controller + ports: + - name: grpc + port: 50052 + targetPort: 10000 +``` + +#### 4. Corresponding `CSIDriverConfig` + +```yaml +apiVersion: ate.dev/v1alpha1 +kind: CSIDriverConfig +metadata: + name: nfs.csi.k8s.io +spec: + driverName: nfs.csi.k8s.io + controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052 + nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock + tls: + enabled: true + usePodIdentity: true + serverName: csi-nfs-controller.kube-system.svc.cluster.local +``` + +--- + +## 3. CSI Node DaemonSet Mount Requirements + +The CSI Node plugin runs as a DaemonSet on each worker node and handles `NodeStageVolume` and `NodePublishVolume` operations when an actor is resumed. + +### Mount Propagation Requirements + +When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateom-gvisor`) on the host to see this filesystem mount, the following requirements must be met: + +1. **Bidirectional Mount Propagation (`mountPropagation: Bidirectional`):** The volume mount on the host Substrate target directory (e.g. `/var/lib/ateom-gvisor`) in the CSI Node plugin container must have `mountPropagation: Bidirectional`. This ensures that any mounts made by the CSI plugin inside the container propagate back to the host filesystem. +2. **Unix Domain Socket Accessibility:** The CSI Node plugin must place its Unix domain socket under `/var/lib/kubelet/plugins//` (or the path defined in `CSIDriverConfig.spec.nodeSocketOverride`), which is shared with `atelet`. + +### Example: `csi-nfs-node` DaemonSet Configuration + +In the `csi-nfs` driver deployment (see [`hack/third_party/csi-driver-nfs/deploy/csi-nfs-node.yaml`](../hack/third_party/csi-driver-nfs/deploy/csi-nfs-node.yaml) and [`hack/setup-csi-nfs-kind.sh`](../hack/setup-csi-nfs-kind.sh)), the DaemonSet is configured with bidirectional mount propagation to `/var/lib/ateom-gvisor`: + +```yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: csi-nfs-node + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-nfs-node + template: + metadata: + labels: + app: csi-nfs-node + spec: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + serviceAccountName: csi-nfs-node-sa + nodeSelector: + kubernetes.io/os: linux + containers: + - name: nfs + image: registry.k8s.io/sig-storage/nfsplugin:v4.13.4 + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + args: + - "--nodeid=$(NODE_ID)" + - "--endpoint=unix:///csi/csi.sock" + env: + - name: NODE_ID + valueFrom: + fieldRef: + fieldPath: spec.nodeName + volumeMounts: + # CSI communication socket shared with host/atelet + - name: socket-dir + mountPath: /csi + # Target directory with bidirectional mount propagation + - name: ateom-dir + mountPath: /var/lib/ateom-gvisor + mountPropagation: Bidirectional + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins/csi-nfsplugin + type: DirectoryOrCreate + - name: ateom-dir + hostPath: + path: /var/lib/ateom-gvisor + type: DirectoryOrCreate +``` diff --git a/docs/csi-volumes.md b/docs/csi-volumes.md new file mode 100644 index 0000000000..bfcff3890e --- /dev/null +++ b/docs/csi-volumes.md @@ -0,0 +1,184 @@ +# CSI Volumes for Actors in Agent Substrate + +Substrate integrates with the **Container Storage Interface (CSI)** to provide dynamically provisioned, per-actor external volumes that seamlessly attach and detach as actors transition through their lifecycle. + +--- + +## 1. CSI in Substrate vs. Standard Kubernetes + +In Kubernetes, volumes are reconciled asynchronously via standard Kubernetes objects (e.g. `PersistentVolumeClaim`, `PersistentVolume`). Agent Substrate takes a different approach tailored for actor lifecycle operations: + +* **No PV or PVC Objects:** External volumes are declaratively defined in the [`ActorTemplate`](api-guide.md#2-actortemplate-the-workload-blueprint) via `externalVolumeTemplate` and provisioned dynamically for each actor instance. Volume operations are coupled directly with the actor lifecycle. +* **Direct Network-Based CSI Controller:** The Substrate control plane (`ateapi`) communicates directly with the CSI Controller gRPC service over the network (via TCP or DNS endpoints, optionally secured with TLS/mTLS). + +--- + +## 2. Dynamic CSI Driver Discovery (`CSIDriverConfig`) + +To discover and communicate with CSI drivers, Substrate uses dynamic discovery driven by the cluster-scoped **`CSIDriverConfig`** Custom Resource Definition (CRD). + +### The `CSIDriverConfig` Resource + +`CSIDriverConfig` defines the gRPC connection parameters for a specific CSI driver. It bridges the Kubernetes `StorageClass` (referenced in the `ActorTemplate`) to the network endpoint of the CSI Controller service and the local socket path of the CSI Node plugin. + +```yaml +apiVersion: ate.dev/v1alpha1 +kind: CSIDriverConfig +metadata: + name: nfs.csi.k8s.io +spec: + driverName: nfs.csi.k8s.io + controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052 + nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock + tls: + enabled: true + usePodIdentity: true + serverName: csi-nfs-controller.kube-system.svc.cluster.local +``` + +### Specification (`CSIDriverConfigSpec`) + +| Field | Type | Description | +| :--- | :--- | :--- | +| `driverName` | `string` | **Required.** The standard CSI driver name (e.g. `nfs.csi.k8s.io`, `hostpath.csi.k8s.io`, `pd.csi.storage.gke.io`). Matches the `provisioner` field on the referenced Kubernetes `StorageClass`. | +| `controllerEndpoint` | `string` | **Required.** The gRPC endpoint for the CSI Controller service. Must be a valid URI starting with `tcp://`, `dns:///`, or `unix://` (e.g., `tcp://csi-controller.kube-system.svc:50051` or `dns:///csi-svc.default.svc:9000`). | +| `nodeSocketOverride` | `string` | **Optional.** Override for the CSI Node service Unix domain socket on worker nodes. Must begin with `unix://`. If omitted, Substrate defaults to `unix:///var/lib/kubelet/plugins//csi.sock`. | +| `tls` | `*CSIDriverTLSConfig` | **Optional.** Configures TLS or mTLS for the gRPC connection to the `controllerEndpoint`. | + +#### TLS / mTLS Configuration (`spec.tls`) + +| Field | Type | Description | +| :--- | :--- | :--- | +| `enabled` | `bool` | **Required.** Enables TLS/mTLS for the gRPC connection. | +| `usePodIdentity` | `bool` | **Optional.** When `true`, reuses Substrate's SPIFFE Pod Identity certificates for mutual TLS (mTLS) with dynamic CA trust bundle verification and rotation. Must be `true` when `enabled` is `true`. | +| `serverName` | `string` | **Optional.** Server name override for TLS certificate verification. | + +> [!NOTE] +> For details on exposing CSI controller endpoints over the network and configuring CSI node DaemonSets with required mount propagations, see the [CSI Driver Deployment Guide](csi-deployment.md). + +--- + +## 3. ActorTemplate: Configuring CSI Volumes + +External volumes are declared on the `ActorTemplate` resource. For complete details on actor templates, see the [ActorTemplate: The Workload Blueprint](api-guide.md#2-actortemplate-the-workload-blueprint) section in the Substrate API Guide. + +### Volume Configuration Fields + +To attach a CSI volume to an actor: + +1. Define the volume under `spec.volumes` with an `externalVolumeTemplate`. +2. Mount the volume inside one or more containers under `spec.containers[].volumeMounts`. + +#### `spec.volumes[]` + +```yaml +volumes: +- name: my-data-volume + externalVolumeTemplate: + capacity: 10Gi + storageClassName: standard-rwx +``` + +* `name`: Unique DNS-label-compliant volume name. +* `externalVolumeTemplate.capacity`: Quantity string representing the requested volume size (e.g. `1Gi`, `50Gi`). +* `externalVolumeTemplate.storageClassName`: Name of a Kubernetes `StorageClass` present in the cluster whose `provisioner` matches a registered `CSIDriverConfig`. + +#### `spec.containers[].volumeMounts[]` + +```yaml +volumeMounts: +- name: my-data-volume + mountPath: /var/data +``` + +* `name`: Must match the declared `spec.volumes[].name`. +* `mountPath`: Unix path inside the container sandbox where the volume will be mounted. + +> [!NOTE] +> All declared volumes in `spec.volumes` must be mounted by at least one container. + +--- + +## 4. End-to-End Example + +The following example demonstrates setting up an NFS CSI driver with Substrate and deploying an `ActorTemplate` that mounts an external NFS volume. + +### Step 1: Create the StorageClass + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: csi-nfs-sc +provisioner: nfs.csi.k8s.io +parameters: + server: nfs-server.default.svc.cluster.local + share: / +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: + - nfsvers=3 + - nolock +``` + +### Step 2: Register the CSIDriverConfig + +```yaml +apiVersion: ate.dev/v1alpha1 +kind: CSIDriverConfig +metadata: + name: nfs.csi.k8s.io +spec: + driverName: nfs.csi.k8s.io + controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052 + nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock + tls: + enabled: true + usePodIdentity: true + serverName: csi-nfs-controller.kube-system.svc.cluster.local +``` + +### Step 3: Define WorkerPool and ActorTemplate + +Refer to [ActorTemplate: The Workload Blueprint](api-guide.md#2-actortemplate-the-workload-blueprint) for general template options. + +```yaml +apiVersion: ate.dev/v1alpha1 +kind: WorkerPool +metadata: + name: agent-pool + namespace: ate-demo + labels: + workload: stateful-agent +spec: + replicas: 5 + ateomImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor +--- +apiVersion: ate.dev/v1alpha1 +kind: ActorTemplate +metadata: + name: stateful-agent-template + namespace: ate-demo +spec: + sandboxClass: gvisor + workerSelector: + matchLabels: + workload: stateful-agent + containers: + - name: agent + image: gcr.io/my-project/agent-app@sha256:7f28ab0... + volumeMounts: + - name: shared-storage + mountPath: /mnt/shared + readyz: + httpGet: + path: /readyz + port: 8080 + snapshotsConfig: + location: gs://my-snapshots-bucket/stateful-agent + volumes: + - name: shared-storage + externalVolumeTemplate: + capacity: 5Gi + storageClassName: csi-nfs-sc +```