From 740e5cb54929632ff539ac4de0ad1ce89fb141e9 Mon Sep 17 00:00:00 2001 From: ShifZhan <252984256+MioYuuIH@users.noreply.github.com> Date: Fri, 31 Jul 2026 14:50:32 +0800 Subject: [PATCH 1/4] Add Kubernetes deployment guide --- source/installation/existing-kubernetes.md | 486 +++++++++++++++++++++ 1 file changed, 486 insertions(+) create mode 100644 source/installation/existing-kubernetes.md diff --git a/source/installation/existing-kubernetes.md b/source/installation/existing-kubernetes.md new file mode 100644 index 00000000..58fd6747 --- /dev/null +++ b/source/installation/existing-kubernetes.md @@ -0,0 +1,486 @@ +# Deploy AUP Learning Cloud On Kubernetes + +This guide starts with an **existing, working Kubernetes cluster**. Cluster and node provisioning, cloud resource lifecycle, and provider-specific networking are outside its scope. + +Use this path when the cluster is already operated for you and you need to make it ready for AUP Learning Cloud, prepare deployment values, install the chart, and validate the result. If you still need to build a multi-node K3s cluster, follow [Multi-Node Cluster Deployment](multi-node.md) first, then return here when the cluster is ready. + +:::{seealso} +For AUP configuration details, see [Configuration Reference](../jupyterhub/configuration-reference.md) and [Authentication Guide](../jupyterhub/authentication-guide.md). +::: + +## Prerequisites + +- A Kubernetes cluster at version **1.28 or later**, as required by `runtime/chart/Chart.yaml`. +- A `kubectl` context with the cluster-scoped and namespaced permissions required by the chart. +- Helm 3. +- A checkout of the AUP Learning Cloud repository. +- AMD GPU nodes with a supported host operating system and driver stack. +- A CSI driver and StorageClass suitable for the cluster topology. +- Access to the configured container registries, including credentials for private images. + +The AUP chart's Kubernetes requirement does not replace AMD's requirements. Also honor the compatibility matrix for the AMD GPU management path and version you choose. + +Complete Steps 1 through 7 in order. Unless noted otherwise, run commands on the **operator machine** that holds the repository checkout, `kubectl` context, and Helm configuration. Run blocks marked **GPU node** through the site's approved node-access method. Replace ``, ``, ``, ``, ``, and other angle-bracketed placeholders with values from your environment. Stop and fix any failed gate before continuing. + +(existing-kubernetes-preflight)= + +## 1. Run Cluster Preflight Checks + +Confirm that `kubectl` points to the intended cluster and that its API and workers are ready: + +```bash +kubectl config current-context +kubectl cluster-info +kubectl version +kubectl get --raw='/readyz?verbose' +kubectl get nodes -o wide +``` + +The context and API endpoint must identify the intended cluster, the server version must be at least 1.28, `/readyz` must succeed, and every worker intended for AUP must report `Ready`. Stop here if any of those checks fails. + +Run representative permission checks next: + +```bash +kubectl auth can-i create namespaces +kubectl auth can-i create clusterroles.rbac.authorization.k8s.io +kubectl auth can-i create clusterrolebindings.rbac.authorization.k8s.io +kubectl auth can-i create deployments.apps --namespace jupyterhub +kubectl auth can-i create statefulsets.apps --namespace jupyterhub +kubectl auth can-i create persistentvolumeclaims --namespace jupyterhub +kubectl auth can-i create services --namespace jupyterhub +``` + +For the normal cluster-administrator flow, every command must print `yes`. These are smoke checks, not a complete permissions proof. Helm also needs the appropriate `get`, `list`, `watch`, `create`, `patch`, `update`, and `delete` lifecycle permissions for every rendered resource. Depending on enabled features, that includes namespaced RBAC, ServiceAccounts, Secrets, ConfigMaps, NetworkPolicies, Jobs, DaemonSets, and PodDisruptionBudgets. + +If cluster-scoped access cannot be delegated, use this namespace-only model: + +1. Ask a cluster administrator to pre-create the `jupyterhub` namespace. +2. Set `scheduling.userScheduler.enabled: false` so the release renders no user-scheduler cluster RBAC. +3. Keep `rbac.create: true` so the chart creates its required namespaced RBAC. +4. Omit `--create-namespace` from the Helm command in Step 5. + +The delegated identity still needs full lifecycle permissions for all rendered namespaced resources. A more restrictive `rbac.create: false` requires a cluster administrator to pre-provision **all** rendered namespaced and cluster-scoped RBAC, not only scheduler objects. Both models require approved values and review of the rendered manifest. + +Check Helm and discover the available StorageClasses: + +```bash +helm version +kubectl get storageclass +``` + +Helm must report version 3, and at least one candidate StorageClass must be available for Step 3. + +:::{warning} +AUP's pinned Z2JH 4.3.3 default is: + +```yaml +singleuser.cloudMetadata.blockWithIptables: true +``` + +It dynamically injects `block-cloud-metadata` into each user Pod at spawn time. This init container is privileged, runs as root (UID 0), and has the `NET_ADMIN` capability. Its iptables rule drops TCP traffic with destination port 80 to `169.254.169.254`; it does not block every port or protocol. Helm can install successfully even when admission policy rejects the first user spawn. + +Before installation, check Pod Security Admission, Gatekeeper, cloud-provider policy, and other site admission rules. Set the following only after a replacement control is enforced and empirically verified to block instance metadata access while preserving required DNS and Hub connectivity: + +```yaml +singleuser.cloudMetadata.blockWithIptables: false +``` + +A NetworkPolicy object, CNI selection, or other configuration alone is not proof of enforcement. +::: + +Continue only when the default init container is allowed or an approved replacement can meet the acceptance test in Step 7. + +## 2. Provide AMD GPU Support + +Use exactly one Kubernetes GPU management path. Do not install the GPU Operator and the standalone device plugin and node labeller together because they manage overlapping GPU discovery resources. + +### Standalone Device Plugin And Node Labeller + +This is the direct AUP-matching path when AMD host drivers are already installed and managed outside Kubernetes. The following block uses the same immutable upstream revision and manifest checksums pinned by AUP. It requires `curl` and `sha256sum` on the operator machine: + +```bash +( + set -euo pipefail + umask 077 + plugin_commit='dea1db13f05159e64d8114bca4c31f48c3cfcac6' + plugin_sha256='b751e467feecf6118bed1de8ba80b9abff01c1f52a6b0b8f31aca3609e6e9dbd' + labeller_sha256='c3e456967efdf14bcfeb97d8f87ca75a402cc6c7c8c6201a320efdd0370fa7aa' + work_dir="$(mktemp -d "${TMPDIR:-/tmp}/aup-amd-device-plugin.XXXXXX")" + trap 'rm -rf "$work_dir"' EXIT + base_url="https://raw.githubusercontent.com/ROCm/k8s-device-plugin/${plugin_commit}" + curl --fail --show-error --silent --location \ + "$base_url/k8s-ds-amdgpu-dp.yaml" \ + --output "$work_dir/k8s-ds-amdgpu-dp.yaml" + curl --fail --show-error --silent --location \ + "$base_url/k8s-ds-amdgpu-labeller.yaml" \ + --output "$work_dir/k8s-ds-amdgpu-labeller.yaml" + printf '%s %s\n' "$plugin_sha256" "$work_dir/k8s-ds-amdgpu-dp.yaml" | sha256sum --check - + printf '%s %s\n' "$labeller_sha256" "$work_dir/k8s-ds-amdgpu-labeller.yaml" | sha256sum --check - + kubectl apply -f "$work_dir/k8s-ds-amdgpu-dp.yaml" + kubectl apply -f "$work_dir/k8s-ds-amdgpu-labeller.yaml" + kubectl rollout status --namespace kube-system \ + daemonset/amdgpu-device-plugin-daemonset --timeout=5m + kubectl rollout status --namespace kube-system \ + daemonset/amdgpu-labeller-daemonset --timeout=5m +) +``` + +The block fails before applying anything if either download or checksum verification fails. `kubectl apply` is idempotent, both DaemonSets must roll out successfully, and the private temporary directory is removed on exit. + +### AMD GPU Operator Alternative + +Use the AMD GPU Operator instead when the site wants operator-owned GPU lifecycle and its Kubernetes version, node operating system, kernel, GPU, and ownership model are supported. Follow the official [AMD GPU Operator installation on Kubernetes](https://instinct.docs.amd.com/projects/gpu-operator/en/latest/installation/kubernetes-helm.html) and [GPU Operator compatibility matrix](https://instinct.docs.amd.com/projects/gpu-operator/en/latest/index.html#compatibility). + +Installing the Operator controller alone does not deliver working GPU resources. The site-specific DeviceConfig must deliberately define driver ownership, node selectors, device plugin rather than DRA-only operation, kubelet socket, and operand images. Review privileged workloads, RBAC, and other cluster-scoped resources before applying them. AUP requires the traditional device-plugin contract; a DRA-only configuration is not compatible. + +Whichever path you choose, verify the result: + +```bash +kubectl get nodes -L amd.com/gpu.product-name,amd.com/gpu.family,amd.com/gpu.vram +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.amd\.com/gpu}{"\n"}{end}' +kubectl get pods -A -o wide +``` + +At least one intended GPU node must have the expected `amd.com/gpu.product-name` label, a non-zero allocatable `amd.com/gpu` value, and healthy operands for the one selected path. Stop and repair GPU management if that contract is incomplete. + +The device plugin allocates `/dev/kfd` and selected `/dev/dri/renderD*` and `/dev/dri/card*` devices to a GPU Pod and establishes device-cgroup access. It does not set the Unix mode or ownership of host device nodes. + +On Ubuntu GPU nodes whose configured AMD package repository provides it, install the AUP-pinned permission policy: + +**GPU node:** + +```bash +sudo apt update +sudo apt install -y amdgpu-insecure-instinct-udev-rules +``` + +Other node operating systems or lifecycle systems need an approved equivalent host policy. It must set mode `0666` for `/dev/kfd` and `/dev/dri/renderD*`, keep `/dev/dri/card*` protected, and persist after node creation, replacement, reimage, and upgrade. + +**GPU node:** + +```bash +stat -c '%a %U %G %n' /dev/kfd /dev/dri/renderD* +stat -c '%a %U %G %n' /dev/dri/card* +``` + +Every listed KFD and render node must report mode `666`. Do not set card nodes to mode `666`; review them against the site's protected policy. + +## 3. Select Storage + +Use an existing named CSI StorageClass that supports the chart's default `ReadWriteOnce` claims. This guide does not prescribe a CSI driver or generic NFS installation. + +```bash +kubectl get storageclass +kubectl get storageclass -o yaml +``` + +Inspect the provisioner, `volumeBindingMode`, reclaim policy, allowed topologies, and mount options. Confirm from the CSI documentation and site policy that the class can dynamically provision RWO volumes and attach them where their Pods run. Stop if the class, provisioner health, topology, persistence, backup, or recovery ownership is unresolved. + +The chart defaults are: + +- Hub database: `sqlite-pvc`, `ReadWriteOnce`, 1 GiB, controlled by `hub.db.pvc.accessModes`. +- User homes: dynamic `ReadWriteOnce`, 10 GiB per user server. Configure their access modes with: + + ```yaml + singleuser.storage.dynamic.storageAccessModes + ``` + +RWO is suitable when the provisioner and topology permit reattachment to the scheduled node. RWX network storage is an alternative when homes must be accessible across nodes without reattachment or site policy requires it. See [dynamic volume provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) and [PersistentVolumes and access modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). + +`local-path` and direct `hostPath` bind data to a node and are not portable multi-node defaults. Use them only after accepting the node-loss, scheduling, mobility, backup, and recovery limits. + +:::{warning} +Changing a StorageClass value does not migrate existing PVC data. Plan and test a separate backup and migration before changing an installed deployment. +::: + +The default SQLite PVC is suitable for the base chart flow. External PostgreSQL or MySQL provisioning, migration, backup, and high availability are separate operator responsibilities. + +## 4. Prepare AUP Values + +Create a protected site file from the multi-node example without overwriting existing configuration: + +:::{warning} +Site values may contain OAuth, administrator, registry, or other credentials. Prefer non-secret settings plus references to pre-created Kubernetes Secrets where supported. Never commit credentials, place them on command lines, or publish them in logs or artifacts. +::: + +```bash +cd +( + set -eu + site_values='runtime/values-existing-cluster.yaml' + if test -e "$site_values"; then + printf 'Refusing to overwrite existing %s\n' "$site_values" >&2 + exit 1 + fi + cp --no-clobber runtime/values-multi-nodes.yaml.example "$site_values" + chmod 600 "$site_values" +) +ls -l runtime/values-existing-cluster.yaml +``` + +The file must show mode `-rw-------`. If it already existed, review it rather than overwriting it. + +The example is not production-ready. It currently contains site-specific samples such as `nfs-client`, authentication placeholders, mutable image tags, and an example supplemental group. Find those defaults, then open the file and replace or remove them: + +```bash +grep -nE 'nfs-client|:latest|supplementalGroups|TODO| -o name +kubectl get nodes -L amd.com/gpu.product-name,amd.com/gpu.family,amd.com/gpu.vram +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.amd\.com/gpu}{"\n"}{end}' +``` + +Map those observed and approved inputs to the values file: + +| Input | Values to review | +| --- | --- | +| Identity provider, callback URL, administrators | Authentication and administrator settings from the [Authentication Guide](../jupyterhub/authentication-guide.md); use Secret references where supported. | +| GPU product labels and enabled offerings | `custom.accelerators` and `custom.accelerators..nodeSelector`. | +| Approved CPU and ROCm images | `custom.resources.images`, requirements, `custom.resources.metadata`, `acceleratorOverrides`, and `custom.resources.metadata..acceleratorKeys`. | +| Teams and capacity | `custom.teams.mapping` and `custom.quota`; Helm replaces arrays rather than merging their entries. | +| Selected CSI class and modes | `hub.db.pvc.storageClassName`, `hub.db.pvc.accessModes`, `singleuser.storage.dynamic.storageClass`, and `singleuser.storage.dynamic.storageAccessModes`. | +| Volume ownership | `singleuser.extraPodConfig.securityContext`; use a storage `fsGid` only when required by the volume policy. | +| Metadata protection | `singleuser.cloudMetadata.blockWithIptables`; keep it `true` unless the replacement passed Step 1 and will be tested in Step 7. | +| Registry and artifacts | Pull Secret references, pull policy, resource images, accelerator overrides, and pre-puller images. | +| Access design | Proxy exposure, ingress, public scheme, TLS, DNS, callback URLs, and firewall or source restrictions. | + +Remove the example GPU `supplementalGroups`. AUP does not manage GPU-specific supplemental groups under this contract. Device allocation and device-cgroup access come from the AMD device plugin, while host modes come from the Step 2 udev policy. A storage `fsGid` is separate and must not grant GPU access. + +For every enabled GPU resource, align the actual `amd.com/gpu.product-name` label, its `nodeSelector`, the GPU architecture, and a ROCm image validated for that architecture. Pin reviewed image tags or digests instead of mutable tags such as `latest`, and align resource images, accelerator overrides, and pre-puller images. + +The chart creates NetworkPolicy objects by default, but isolation exists only when the cluster network implementation enforces them. Verify enforcement rather than treating the objects as proof. + +Keep the proxy private until production authentication is tested. Public exposure requires HTTPS with a valid certificate, correct callback URLs, working DNS, and approved firewall or source restrictions. Never expose development authentication to untrusted networks. + +Do not render until credentials, selectors, storage, metadata controls, images, NetworkPolicy enforcement, and exposure have all been reviewed. See [Customizing a Single-Node Deployment](customizing-deployment.md) for the AUP resource model and the [Configuration Reference](../jupyterhub/configuration-reference.md) for field details. + +## 5. Render And Install + +Lint and privately inspect the exact site configuration from the repository root: + +```bash +cd +helm lint runtime/chart \ + --namespace jupyterhub \ + -f runtime/values-existing-cluster.yaml +( + set -eu + umask 077 + rendered_manifest="$(mktemp "${TMPDIR:-/tmp}/aup-learning-cloud-rendered.XXXXXX.yaml")" + trap 'rm -f "$rendered_manifest"' EXIT + helm template jupyterhub runtime/chart \ + --namespace jupyterhub \ + -f runtime/values-existing-cluster.yaml > "$rendered_manifest" + less "$rendered_manifest" +) +``` + +The temporary manifest can contain Secrets or derived sensitive values. Do not publish or retain it; the trap removes it when the subshell exits. + +Before installation, confirm: + +- [ ] Images, digests or tags, pull Secrets, and pull policy match approved artifacts. +- [ ] GPU selectors and requests match the Step 2 labels and `amd.com/gpu` contract. +- [ ] Hub and user-home StorageClasses and access modes match Step 3. +- [ ] Metadata blocking and admission-sensitive Pod settings match Step 1. +- [ ] RBAC matches the cluster-admin or reviewed delegated model. +- [ ] Authentication, proxy, ingress, TLS, and exposure match the approved design. + +Stop if lint fails or the render contains an unexpected resource, secret, selector, storage class, or exposure setting. Otherwise install or upgrade: + +```bash +helm upgrade --install jupyterhub runtime/chart \ + --namespace jupyterhub \ + --create-namespace \ + -f runtime/values-existing-cluster.yaml \ + --wait \ + --timeout 10m +``` + +For the delegated namespace-only model from Step 1, omit `--create-namespace`. Keep `--wait` and the timeout. If Helm fails or times out, inspect Step 6 before retrying; do not begin user acceptance on a partial release. + +## 6. Validate Infrastructure + +Check the release and its cluster dependencies: + +```bash +helm status jupyterhub --namespace jupyterhub +kubectl get pods,services,ingress --namespace jupyterhub -o wide +kubectl get pvc --namespace jupyterhub +kubectl get events --namespace jupyterhub --sort-by=.metadata.creationTimestamp +kubectl get nodes -L amd.com/gpu.product-name +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.amd\.com/gpu}{"\n"}{end}' +``` + +Do not continue until: + +- [ ] Helm reports the release as deployed and required workloads are ready. +- [ ] No unresolved restart, scheduling, image-pull, or admission error remains. +- [ ] Hub and user PVCs created so far are `Bound`. +- [ ] The configured internal or external route resolves to the proxy. +- [ ] Intended GPU nodes retain their product label and non-zero `amd.com/gpu` value. +- [ ] Recent namespace events contain no unresolved AUP warning. + +(existing-kubernetes-end-to-end-acceptance)= + +## 7. Run End-To-End Acceptance + +Complete every case through the user-facing interface and retain the Pod inspection output. Shared clusters can create several user Pods at once, so always select the test Pod by exact name. Never auto-select the newest Pod. + +### Log In And Spawn A CPU Server + +Open ``, log in with a non-administrator test user, choose a CPU resource, and wait for the environment to become ready. Identify that user's exact Pod: + +```bash +kubectl get pods --namespace jupyterhub -l component=singleuser-server -o wide +CPU_USER_POD='' +kubectl get pod --namespace jupyterhub "$CPU_USER_POD" -o wide +kubectl describe pod --namespace jupyterhub "$CPU_USER_POD" +``` + +Resolve login, spawn, scheduling, or Pod identity problems before continuing. + +### Prove Metadata Protection + +When the default metadata blocker remains enabled, verify the selected CPU Pod's init result: + +```bash +kubectl get pod --namespace jupyterhub "$CPU_USER_POD" \ + -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state.terminated.reason}{"\t"}{.state.terminated.exitCode}{"\n"}{end}' +``` + +`block-cloud-metadata` must report `Completed` with exit code `0`. When the field is `false`, retain approved empirical evidence that the replacement blocks instance metadata access while DNS and Hub connectivity still work. Use a site-approved probe or test harness; do not assume every user image contains `curl`. + +### Prove Home Persistence + +Create a file in the user's home: + +```bash +kubectl exec --namespace jupyterhub "$CPU_USER_POD" -- \ + sh -c 'printf "%s\n" "AUP persistence acceptance" > "$HOME/aup-persistence-check.txt" && cat "$HOME/aup-persistence-check.txt"' +``` + +Stop the server through the interface, start the same CPU resource again, and manually set `CPU_USER_POD` to the exact restarted Pod: + +```bash +kubectl get pods --namespace jupyterhub -l component=singleuser-server -o wide +CPU_USER_POD='' +kubectl exec --namespace jupyterhub "$CPU_USER_POD" -- \ + sh -c 'cat "$HOME/aup-persistence-check.txt"' +``` + +The restarted server must print `AUP persistence acceptance`. Stop if the file is missing or the PVC was unexpectedly replaced. + +### Spawn And Select A GPU Server + +Stop the CPU server if quota requires it, choose an enabled GPU resource, and wait for it to become ready. Select the exact GPU Pod: + +```bash +kubectl get pods --namespace jupyterhub -l component=singleuser-server -o wide +GPU_USER_POD='' +kubectl get pod --namespace jupyterhub "$GPU_USER_POD" \ + -o jsonpath='{.spec.nodeName}{"\n"}{.spec.containers[0].resources}{"\n"}' +GPU_NODE="$(kubectl get pod --namespace jupyterhub "$GPU_USER_POD" -o jsonpath='{.spec.nodeName}')" +kubectl get node "$GPU_NODE" -L amd.com/gpu.product-name +``` + +The Pod must request `amd.com/gpu` and land on a node with the product label selected by its AUP resource. Resolve a missing request, selector mismatch, pending Pod, or wrong-node placement before continuing. + +### Verify Host Device Modes + +Connect to the exact `GPU_NODE` through the site's approved node-access method. There is no universal node-login command. + +**GPU node:** + +```bash +stat -c '%a %n' /dev/kfd /dev/dri/renderD* +stat -c '%a %n' /dev/dri/card* +``` + +KFD and every render node must report mode `666`; card nodes must remain protected. Repair the persistent host policy rather than weakening card-node permissions. + +### Verify GPU Access In The Pod + +```bash +kubectl exec --namespace jupyterhub "$GPU_USER_POD" -- rocminfo +``` + +`rocminfo` must exit successfully and report the expected AMD GPU. `amd-smi` or a basic operation in the installed GPU framework is a useful additional test, but it does not replace `rocminfo` here. + +Acceptance is cumulative. Login, CPU spawn, metadata evidence, persistent home storage, host mode `666` for KFD and render nodes, non-zero allocatable `amd.com/gpu`, GPU scheduling, and in-Pod `rocminfo` must all succeed. + +## Troubleshooting + +(existing-kubernetes-metadata-troubleshooting)= + +### User Spawn Fails At Admission Or During Metadata Blocking + +If no user Pod is created, inspect Hub logs, namespace events, and admission labels: + +```bash +kubectl logs --namespace jupyterhub deployment/hub --since=15m +kubectl get events --namespace jupyterhub --sort-by=.metadata.creationTimestamp +kubectl get namespace jupyterhub --show-labels +``` + +Look for `Forbidden`, `denied`, `PodSecurity`, `privileged`, `NET_ADMIN`, Gatekeeper, or another policy rejection. Helm success does not prove that user Pods pass admission. + +If the Pod exists but remains in init, inspect its state, blocker logs, and events: + +```bash +kubectl describe pod --namespace jupyterhub +kubectl get pod --namespace jupyterhub \ + -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}' +kubectl logs --namespace jupyterhub --container block-cloud-metadata +kubectl logs --namespace jupyterhub --container block-cloud-metadata --previous +kubectl get events --namespace jupyterhub \ + --field-selector involvedObject.name= \ + --sort-by=.metadata.creationTimestamp +``` + +Repair the reported image-pull, iptables, capability, runtime, or admission problem. Do not disable metadata blocking as an immediate fix. Return to the {ref}`preflight decision ` and disable it only after a replacement passes {ref}`end-to-end acceptance `. + +### GPU Resources Or Labels Are Missing + +```bash +kubectl get pods -A | grep -E 'amd|gpu' +kubectl describe node | grep -A12 'Allocatable' +kubectl get node --show-labels | grep 'amd.com/gpu' +``` + +A healthy GPU node needs both a matching product label and non-zero allocatable `amd.com/gpu`. Repair the one selected management path and its host compatibility. Do not install the second path as a workaround or change selectors to labels the node does not report. + +### PVCs Stay Pending + +```bash +kubectl get storageclass +kubectl get pvc --namespace jupyterhub +kubectl describe pvc --namespace jupyterhub +``` + +Check the class name, provisioner events, access mode, topology, capacity, attachment, and mount errors with the storage operator. Changing the class does not migrate data; back up and plan a separate migration when required. + +### Images Do Not Pull + +```bash +kubectl describe pod --namespace jupyterhub +kubectl get secrets --namespace jupyterhub +``` + +Events distinguish invalid references, architecture mismatch, registry reachability, missing pull Secrets, and rejected credentials. Fix the approved image or Secret while keeping resource, accelerator, and pre-puller images aligned. Do not expose credentials or replace an immutable reference with a mutable tag as a shortcut. + +### External Access Fails + +```bash +kubectl get service,ingress --namespace jupyterhub -o wide +kubectl describe service --namespace jupyterhub proxy-public +kubectl get events --namespace jupyterhub --sort-by=.metadata.creationTimestamp +``` + +Inspect the proxy Service, ingress status, controller events, TLS Secret, hostname, DNS, and callback URL. Repair the site's load balancer, ingress controller, certificates, firewall, or DNS through its owning procedure. Keep the proxy private and development authentication unexposed until the production path works. From eba73682a419ef055ca64b50378eccaeff3e7937 Mon Sep 17 00:00:00 2001 From: ShifZhan <252984256+MioYuuIH@users.noreply.github.com> Date: Fri, 31 Jul 2026 14:50:52 +0800 Subject: [PATCH 2/4] Refocus multi-node Kubernetes deployment guides --- source/installation/multi-node.md | 318 ++++++------------ .../multi-aipc-hardware-deployment.md | 153 ++++----- 2 files changed, 178 insertions(+), 293 deletions(-) diff --git a/source/installation/multi-node.md b/source/installation/multi-node.md index 46e8c5e4..4e11636d 100644 --- a/source/installation/multi-node.md +++ b/source/installation/multi-node.md @@ -1,30 +1,18 @@ -# Multi-Node Cluster Deployment +# Build A Multi-Node K3s Cluster -This guide covers the current Ansible + Helm workflow for deploying AUP Learning Cloud on a multi-node K3s cluster. +This guide builds and prepares a multi-node K3s cluster with the repository's Ansible playbooks. It stops at a readiness handoff. The canonical Kubernetes guide owns AUP Learning Cloud values, Helm installation, validation, access, and routine release operations. -Unlike the single-node path, multi-node deployment is not driven by `./auplc-installer install`. The main flow is: - -1. prepare SSH and inventory -2. build the cluster with Ansible -3. deploy the ROCm device plugin and node labeller -4. prepare storage and images -5. customize the multi-node values file -6. deploy the chart with Helm +Unlike the single-node path, this workflow isn't driven by `./auplc-installer install`. It prepares SSH access and inventory, builds K3s, establishes operator access, and makes cluster-specific storage and image choices. ## Overview -Multi-node deployment is the right path when you need: - -- multiple worker nodes for user workloads -- shared storage across the cluster -- explicit control over ingress, authentication, and network exposure -- a layout that is closer to a long-running lab or production environment - -Typical roles in a small cluster: +Use this path when you have machines with an operating system and SSH access and want Ansible to provision a new K3s cluster. A small cluster commonly has these roles: - **server node**: runs the K3s control plane - **agent nodes**: run Hub services and user notebook workloads -- **storage node**: optional, if you host NFS separately +- **storage node**: optional, when the site chooses a separate NFS backend + +Shared NFS is one storage example in this guide, not a K3s or AUP requirement. You can use another StorageClass that meets the site's access, topology, persistence, backup, and recovery needs. ## Prerequisites @@ -33,52 +21,46 @@ Typical roles in a small cluster: - Ansible available - SSH key access to all nodes - ability to connect as `root` or the configured `ansible_user` -- a checkout of this repository +- a checkout of the AUP Learning Cloud repository ### Cluster Nodes - Ubuntu 24.04 - consistent hostname resolution across the fleet -- AMD GPU-capable nodes if you want accelerator-backed resources +- AMD GPU-capable nodes for accelerator-backed resources Current inventory defaults are defined in `deploy/ansible/inventory.yml`, including the pinned `k3s_version`. -## 1. Prepare SSH Access +The repository-relative commands in this guide use `/path/to/aup-learning-cloud` as the deployment repository root. Replace that placeholder with the absolute path to your checkout. Each block that needs repository files establishes its own working directory, so you can run the guide sequentially without carrying a prior block's directory forward. + +## 1. Prepare SSH Access And Hostnames The Ansible flow assumes passwordless SSH to all nodes. In practice, the two most common issues are: -- the control node cannot reach every node by hostname -- the server node cannot SSH to agents with the same names used in `inventory.yml` +- the controller can't reach every node by hostname +- the server node can't reach agents with the names used in `inventory.yml` If needed, use the helper scripts in `deploy/scripts/`: ```bash +cd /path/to/aup-learning-cloud ./deploy/scripts/edit_sshd.sh ./deploy/scripts/setup_ssh_root_access.sh ./deploy/scripts/deploy-kubeconfig.sh ``` -These scripts help enable root SSH login, copy SSH access to cluster nodes, and distribute kubeconfig where needed. - -You should also make sure `/etc/hosts` entries are consistent across the nodes when you rely on hostnames instead of direct IPs. +These scripts help enable root SSH login, copy SSH access to cluster nodes, and distribute kubeconfig where needed. Make sure `/etc/hosts` entries are consistent across the nodes when you use hostnames instead of direct IP addresses. -## 2. Configure Inventory +## 2. Configure The Ansible Inventory -Edit the Ansible inventory: +Edit the inventory: ```bash -cd deploy/ansible +cd /path/to/aup-learning-cloud/deploy/ansible nano inventory.yml ``` -Key items to set: - -- server and agent hostnames -- `ansible_user` -- cluster token -- `api_endpoint` - -Minimal structure: +Set the server and agent hostnames, `ansible_user`, cluster token, and `api_endpoint`. A minimal structure is: ```yaml --- @@ -100,42 +82,40 @@ k3s_cluster: api_endpoint: "{{ hostvars[groups['server'][0]]['ansible_host'] | default(groups['server'][0]) }}" ``` +This is the normal inventory for machines that already have an operating system and SSH access. The diskless PXE topology uses a different inventory boundary, documented in the [3 Node Mini-Cluster Example](multi-node/multi-aipc-hardware-deployment.md). + ## 3. Build The Cluster +Run the provisioning playbooks from the Ansible directory: + ```bash -cd deploy/ansible +cd /path/to/aup-learning-cloud/deploy/ansible -# Base OS / package preparation +# Base OS and package preparation sudo ansible-playbook playbooks/pb-base.yml -# Deploy K3s cluster +# Deploy the K3s cluster sudo ansible-playbook playbooks/pb-k3s-site.yml -# Install ROCm on accelerator nodes +# Install host ROCm support on accelerator nodes sudo ansible-playbook playbooks/pb-rocm.yml ``` -Useful related playbooks: +After editing `inventory.yml`, run `pb-k3s-site.yml` again to add or reconcile nodes. Use the upgrade playbook for a planned K3s upgrade: ```bash -# Add or reconcile nodes after editing inventory.yml +cd /path/to/aup-learning-cloud/deploy/ansible sudo ansible-playbook playbooks/pb-k3s-site.yml - -# Upgrade cluster sudo ansible-playbook playbooks/pb-k3s-upgrade.yml - -# Reset cluster -sudo ansible-playbook playbooks/pb-k3s-reset.yml - -# Reset a single node -sudo ansible-playbook playbooks/pb-k3s-reset.yml --limit ``` -## 4. Install kubectl / Helm On The Operator Machine +`pb-rocm.yml` installs host support. It doesn't, by itself, choose or satisfy Kubernetes device-plugin ownership. The cluster must still use exactly one GPU management path and meet the label, allocatable-resource, and host-permission contract in the canonical [Kubernetes deployment guide](existing-kubernetes.md). Don't install both the GPU Operator and the standalone device plugin and node labeller. -You need a working `kubectl` and `helm` on the machine from which you will manage the cluster. +## 4. Prepare The Operator Machine -Example Helm install: +The operator machine needs the K3s kubeconfig, a working `kubectl`, and Helm 3. The `deploy-kubeconfig.sh` helper can distribute the kubeconfig. Before handoff, make sure the kubeconfig's server address is reachable from this machine and that its context identifies the intended cluster. + +Example Helm installation: ```bash wget https://get.helm.sh/helm-v3.17.2-linux-amd64.tar.gz -O /tmp/helm-linux-amd64.tar.gz @@ -144,7 +124,7 @@ sudo mv /tmp/linux-amd64/helm /usr/local/bin/helm rm /tmp/helm-linux-amd64.tar.gz ``` -Optional but useful for inspection: +K9s is optional inspection tooling: ```bash wget https://github.com/derailed/k9s/releases/latest/download/k9s_linux_amd64.deb @@ -152,44 +132,17 @@ sudo apt install ./k9s_linux_amd64.deb rm k9s_linux_amd64.deb ``` -## 5. GPU Device Plugin And Labels - -For manual cluster setup, deploy the ROCm device plugin and node labeller: - -```bash -kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml -kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml -``` - -Verify labels: - -```bash -kubectl describe node | grep amd.com/gpu -``` - -### About Accelerator Selectors - -The sample file `runtime/values-multi-nodes.yaml.example` now follows `runtime/values.yaml` and uses ROCm labeller keys such as `amd.com/gpu.product-name` directly. +Use the {ref}`canonical preflight checks ` to verify API access, the current context, node readiness, permissions, Helm, and StorageClasses. That section also owns the full GPU management and admission-policy contract for the AUP deployment. -That means multi-node deployments should rely on the device plugin plus labeller output, not on a separate manual `node-type` labelling convention. +## 5. Select Storage -Current examples in the values file include selectors like: +Choose a StorageClass for the cluster's persistence and topology needs. A named CSI StorageClass with suitable volume mobility can be enough. Network storage is another choice when the site needs shared access across nodes. -- `AMD_Radeon_780M_Graphics` -- `AMD_Radeon_890M_Graphics` -- `AMD_Radeon_8060S_Graphics` -- `AMD_Radeon_RX_9070_XT` -- `AMD_Radeon_AI_PRO_R9700` +The repository's multi-node values example uses `nfs-client`, so the following steps show one optional NFS backend. Skip them when the site has selected another StorageClass. -If your labeller normalizes a specific product name differently on your fleet, update the corresponding `custom.accelerators..nodeSelector` entry. +### Optional: Configure An NFS Server -## 6. Storage - -Multi-node deployments usually need a shared storage class. The example values file assumes `nfs-client`. - -### Configure An NFS Server - -On the controller node or a dedicated storage node: +Run these commands only on the host selected to serve NFS. Before changing it, confirm `/nfs` isn't an existing data directory, review the subnet boundary, and back up `/etc/exports`. The recursive ownership and mode changes affect every existing item below `/nfs`. The `no_root_squash` export option grants remote root broad access, so use it only when the site's security policy explicitly approves that risk. ```bash sudo apt install nfs-kernel-server @@ -198,22 +151,25 @@ sudo chown -R nobody:nogroup /nfs sudo chmod 777 /nfs ``` -Add an export for your subnet: +After checking that an equivalent export doesn't already exist, add the export for the intended cluster subnet and reload the NFS service: ```bash echo "/nfs (rw,sync,no_subtree_check,no_root_squash,insecure)" | sudo tee -a /etc/exports sudo systemctl restart nfs-kernel-server ``` -Install the NFS client on worker nodes if it is not already present: +Install the NFS client only on cluster nodes that will mount this backend: ```bash sudo apt install nfs-common ``` -### Deploy The NFS Provisioner +### Optional: Deploy The NFS Provisioner + +Confirm that `deploy/k8s/nfs-provisioner/values.yaml` points to the intended NFS server and export before installing the provisioner: ```bash +cd /path/to/aup-learning-cloud helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ helm repo update @@ -223,16 +179,16 @@ helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs -f deploy/k8s/nfs-provisioner/values.yaml ``` -Optionally make it the default storage class: +Making a StorageClass the default changes how future PVCs without an explicit class are provisioned cluster-wide. Check the current defaults and pending workloads first. If that site-wide change is intended, apply it to `nfs-client`: ```bash -kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' kubectl get storageclass +kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` -## 7. Prepare Images +## 6. Prepare Image Access -You can either push images to a registry or import them directly into cluster nodes. +The cluster can pull AUP images from a registry, or you can import images directly into K3s nodes. Pick the path that every node scheduled for the corresponding workload can use. ### Option A: Use A Registry @@ -245,131 +201,67 @@ docker push ghcr.io/amdresearch/auplc-default:latest docker push ghcr.io/amdresearch/auplc-cv:latest ``` -Then update `custom.resources.images` and, if needed, `prePuller.extraImages` to match your registry. +The site values must point `custom.resources.images` and, when used, `prePuller.extraImages` to accessible images. Private registries also need the appropriate pull credentials. -### Option B: Import Images Directly To Nodes +### Option B: Import Images Directly Into K3s Nodes ```bash -docker save ghcr.io/amdresearch/auplc-dl:latest -o auplc-dl.tar +cd /path/to/aup-learning-cloud/deploy/ansible +docker save ghcr.io/amdresearch/auplc-dl:latest -o /tmp/auplc-dl.tar -ansible agent -m copy -a "src=auplc-dl.tar dest=/tmp/" +ansible agent -m copy -a "src=/tmp/auplc-dl.tar dest=/tmp/" ansible agent -m shell -a "k3s ctr images import /tmp/auplc-dl.tar" ``` -## 8. Prepare The Multi-Node Values File +Repeat the import for each required image and every K3s node that may run it. A successful import on one node doesn't make that image available on the others. -The repository includes a standalone example file for multi-node deployments: +## 7. Create The Topology-Specific Values File -```bash -cd runtime -cp values-multi-nodes.yaml.example values-multi-nodes.yaml -nano values-multi-nodes.yaml -``` - -Review at least these sections: - -- `custom.authMode` -- `custom.githubOrgName` -- `custom.adminUser` -- `custom.accelerators` -- `custom.resources.images` -- `custom.resources.requirements` -- `custom.resources.metadata` -- `custom.teams.mapping` -- `custom.quota` -- `hub.config.GitHubOAuthenticator` -- `hub.db.pvc.storageClassName` -- `singleuser.storage.dynamic.storageClass` -- `proxy.service` -- `ingress` - -### What The Example Already Assumes - -The current example is not just a tiny patch file. It already includes: - -- accelerator definitions aligned with `runtime/values.yaml` -- course image placeholders using the current image set -- team-to-resource mappings -- quota configuration knobs -- Git clone settings -- storage, ingress, and Hub sections for a real deployment - -## 9. Deploy JupyterHub - -```bash -cd runtime -helm upgrade --install jupyterhub ./chart \ - -n jupyterhub --create-namespace \ - -f values-multi-nodes.yaml -``` - -## 10. Verify Deployment +`runtime/values-multi-nodes.yaml.example` is a standalone multi-node example, not a small override. Keep the generated site filename `runtime/values-multi-nodes.yaml`. From the repository root, create it only when it doesn't already exist: ```bash -kubectl get nodes -kubectl get pods -n jupyterhub -kubectl get pvc -n jupyterhub -kubectl get ingress -n jupyterhub -kubectl get storageclass -``` - -If you copied kubeconfig from the server node, verify the current context too: - -```bash -kubectl config current-context -``` - -## Access JupyterHub - -If you use ingress: - -```bash -kubectl get ingress -n jupyterhub -``` - -Then access the configured hostname, for example: - -```text -https://your-domain.com +cd /path/to/aup-learning-cloud +test ! -e runtime/values-multi-nodes.yaml +cp --no-clobber runtime/values-multi-nodes.yaml.example runtime/values-multi-nodes.yaml +chmod 600 runtime/values-multi-nodes.yaml ``` -If you expose the proxy with `NodePort`, use the node IP and configured port instead. +If the file exists, stop and review it instead of overwriting it. Maps are merged by Helm, but arrays are replaced. For example, changing `custom.teams.mapping.gpu` replaces that whole list. -## Operational Notes +The canonical Kubernetes guide owns the complete values review, GPU selector and image alignment, storage fields, exposure gate, manifest inspection, and Helm install. Keep using `runtime/values-multi-nodes.yaml` when following that workflow rather than changing to its example filename. -### Apply Later Configuration Changes +## High Availability Scope -Most routine changes after initial deployment are another Helm upgrade with the same values file: +This guide builds a basic multi-node cluster with one K3s server in the example inventory. Multiple agents don't make the control plane highly available. A highly available K3s server topology, external database, multiple Hub replicas, dedicated load balancers, production TLS, and certificate rotation are separate operator designs. -```bash -cd runtime -helm upgrade --install jupyterhub ./chart \ - -n jupyterhub \ - -f values-multi-nodes.yaml -``` +(multi-node-k3s-ready)= -### High Availability Scope +## K3s Readiness Handoff -This guide covers the base multi-node chart deployment. Choices such as: +Don't start the AUP Helm workflow until every applicable gate below passes: -- external database backends -- multiple Hub replicas -- dedicated load balancers -- production TLS and certificate rotation +- [ ] **Kubernetes API:** the operator kubeconfig reaches the intended K3s API and the current context names that cluster. +- [ ] **Ready nodes:** every server and agent intended for AUP reports `Ready`. +- [ ] **GPU ownership:** exactly one Kubernetes GPU management path owns the device plugin and node labeller. `pb-rocm.yml` host support isn't counted as that path. +- [ ] **GPU discovery:** intended GPU nodes expose the expected `amd.com/gpu.product-name` labels and a non-zero allocatable `amd.com/gpu` resource. +- [ ] **StorageClass:** the selected StorageClass exists and matches the site's access mode, topology, persistence, backup, and recovery needs. NFS is optional. +- [ ] **Image access:** each schedulable node can pull or has directly imported every image it may run, with private-registry credentials available where needed. +- [ ] **Operator tools:** the operator machine has the K3s kubeconfig, working `kubectl`, and Helm 3. +- [ ] **Site values:** `runtime/values-multi-nodes.yaml` exists as a reviewed site file and wasn't created by overwriting prior configuration. -should be treated as explicit operator decisions layered on top of this base flow. +When all eight gates pass, continue at the {ref}`canonical Kubernetes preflight `. Follow the canonical values, render, install, and infrastructure-validation steps, substituting `runtime/values-multi-nodes.yaml` as the explicit site values file. Finish with {ref}`canonical end-to-end acceptance `. If a user spawn fails at admission or during metadata blocking, use {ref}`canonical metadata troubleshooting `. -## Troubleshooting +## K3s Provisioning Troubleshooting ### kubectl Permission Denied On k3s.yaml -If you hit an error like: +If the operator sees this error: ```text error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied ``` -Set write permissions through the inventory before deployment: +Set the kubeconfig mode through the inventory before deployment: ```yaml k3s_cluster: @@ -377,7 +269,7 @@ k3s_cluster: extra_server_args: "--write-kubeconfig-mode=644" ``` -Or copy the config manually: +Or copy the config on the K3s server for the current operator account: ```bash mkdir -p ~/.kube @@ -385,8 +277,12 @@ sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $(id -u):$(id -g) ~/.kube/config ``` +When using that copy from another machine, make sure its server address is reachable and identifies the intended K3s server. + ### Agent Node Does Not Join The Cluster +On the affected agent only, inspect the service and its route to the server: + ```bash ssh sudo systemctl status k3s-agent.service @@ -394,44 +290,44 @@ journalctl -u k3s-agent -n 100 ping ``` -Most often this is a hostname resolution, token, or API endpoint mismatch in `inventory.yml`. +The usual causes are hostname resolution, token, or API endpoint mismatches in `inventory.yml`. -### GPU Labels Or Resources Missing +### Optional NFS Backend Fails -Check the daemonsets first: +Use this section only when the site selected the optional NFS provisioner. Check the provisioner against the intended server and export before changing either one: ```bash -kubectl get ds -A | grep amdgpu -kubectl describe node | grep amd.com/gpu +kubectl get pods -n nfs-provisioner +kubectl logs -n nfs-provisioner deployment/nfs-subdir-external-provisioner ``` -If the labels do not match your `custom.accelerators..nodeSelector`, the resource will not schedule onto that node. +For AUP PVC failures after the cluster handoff, use the PVC troubleshooting section in the canonical [Kubernetes deployment guide](existing-kubernetes.md). GPU labels, allocatable resources, scheduling, and user Pod GPU access are also owned by that guide and its {ref}`end-to-end acceptance `. -### Storage Provisioning Fails +### Reset The Cluster Or One Node -```bash -kubectl get pods -n nfs-provisioner -kubectl get pvc -n jupyterhub -kubectl logs -n nfs-provisioner deployment/nfs-subdir-external-provisioner -``` +:::{danger} +The reset playbook removes K3s state and interrupts workloads on its target. A full reset targets every host in the active Ansible inventory and can make the cluster unavailable. A limited reset removes one node's K3s state and any node-local workload data. Neither operation is a routine repair, and neither backs up AUP data, PVC contents, K3s state, or site configuration. -### Resetting The Cluster +Before running either command, verify the repository and inventory path, print and review the resolved target hosts, stop or evacuate affected workloads, and confirm tested backups for every persistent or node-local data set. Record a maintenance window and recovery plan. Don't continue if the target list, backup state, or storage impact is uncertain. +::: -To remove the cluster completely: +To reset the entire cluster, run this only after the full-inventory preflight above: ```bash -cd deploy/ansible +cd /path/to/aup-learning-cloud/deploy/ansible sudo ansible-playbook playbooks/pb-k3s-reset.yml ``` -To reset a single node only: +To reset one node, first confirm that `` resolves to exactly the intended inventory host and that its workloads and local data have been handled. Then limit the destructive playbook to that host: ```bash +cd /path/to/aup-learning-cloud/deploy/ansible sudo ansible-playbook playbooks/pb-k3s-reset.yml --limit ``` ## Notes On Scope -- The sample multi-node values file is a starting point, not a promise that every advanced topology is turnkey. -- The most important cluster-specific alignment is between real node labels and `custom.accelerators.*.nodeSelector`. -- If you want the simplest local install, use the single-node installer flow instead of this guide. +- This page owns normal Ansible and K3s provisioning for machines with an operating system and SSH access. +- The diskless PXE flow remains a specialized reference with different controller, inventory, and boot boundaries. +- The canonical Kubernetes guide owns AUP configuration, Helm operations, validation, access, GPU and PVC troubleshooting, and user-facing acceptance. +- For the simplest local installation, use the single-node installer instead. diff --git a/source/installation/multi-node/multi-aipc-hardware-deployment.md b/source/installation/multi-node/multi-aipc-hardware-deployment.md index 5ea70a91..a0a73e5a 100644 --- a/source/installation/multi-node/multi-aipc-hardware-deployment.md +++ b/source/installation/multi-node/multi-aipc-hardware-deployment.md @@ -49,6 +49,12 @@ Per-machine requirements: - **Agents**: a working **in-kernel** network driver (this role ships no vendor drivers — add the module to `pxe_initramfs_modules` if needed), and a local disk if you want persistent K3s state across reboots. - **All machines**: UEFI Secure Boot **disabled** in firmware (the UEFI path boots GRUB directly without a Microsoft-signed shim), and the ability to network-boot (PXE) from firmware. +::::::{danger} +Decide how each agent will handle local storage before you expose it to this netboot image. The generated `mount-local-disk` script selects the first existing whole-device candidate in this order: `/dev/sda`, `/dev/vda`, then `/dev/nvme0n1`. If that device isn't already ext4, the script runs `mkfs.ext4 -F` on the whole device. This can erase its partition table and all existing data. There is no interactive confirmation. + +For persistent K3s state, expose only a dedicated blank disk, or a disk whose contents have a verified, restorable backup and whose erasure you have explicitly approved. Check the agent's device mapping outside this boot flow before continuing. If persistence isn't wanted, make an explicit decision not to expose a local disk to the agent, for example by disconnecting it or removing it from the VM hardware. With none of the three candidates present, the script uses tmpfs for K3s data instead. +:::::: + ::::::{warning} No site values (IPs, subnet, SSH keys, passwords, tokens) ship in this repo. You set them in the inventory and the playbook, and the role **fails fast** if a required value is empty. Keep real secrets out of version control. :::::: @@ -134,7 +140,13 @@ Keep `k3s_version` here in sync with `pxe_k3s_version` in the PXE playbook (Step ## Step 3 — Configure The PXE Controller Playbook -Edit the `vars:` block in `deploy/ansible/playbooks/pb-pxe-controller.yml`. The required values are empty by default — set them all: +Edit the `vars:` block in this playbook: + +```text +deploy/ansible/playbooks/pb-pxe-controller.yml +``` + +The required values are empty by default — set them all: ```yaml pxe_rootfs_force_rebuild: true # true for the first build @@ -217,6 +229,12 @@ The netbooted agents are diskless and are intentionally **not** in the `agent` i At boot, each agent's `k3s-auto-join.sh` fetches `http://:8080/k3s/token` and `http://:8080/k3s/kubeconfig`. Publish both through Apache: +::::::{danger} +Use this design only on an isolated, trusted provisioning network. These files are served over plain HTTP and the Apache ACL makes them readable to reachable members of the configured `pxe_subnet`. The token allows a node to join the cluster, and the published server kubeconfig is an administrative kubeconfig. Anyone on that subnet who can reach the endpoint can read those credentials, and HTTP doesn't protect them in transit. + +Before publishing, restrict the provisioning subnet with network isolation and ACLs so only intended agents and operators can reach port 8080. Review the subnet before each run. After provisioning, rotate the node-join token and administrative kubeconfig, then update or withdraw the published copies according to the site's agent reboot requirements. +:::::: + ```bash sudo install -d -m 0755 /var/www/html/k3s @@ -257,7 +275,7 @@ The screenshots below use a virtual machine as the example agent (here a Proxmox ![Agent console login prompt after netboot (VM example)](../../_static/agent-login-prompt.png) -After boot, each agent mounts `/srv/nfs/rootfs`, sets its hostname to `agent-`, mounts its local K3s persistence disk, fetches the token, and joins the server. +After boot, each agent mounts `/srv/nfs/rootfs`, sets its hostname to `agent-`, runs `mount-local-disk`, fetches the token, and joins the server. `mount-local-disk` either mounts the first candidate device for K3s data or, when no candidate exists, creates a temporary in-memory mount as described in the storage warning above. Watch node registration from AIPC 1: @@ -267,15 +285,19 @@ watch kubectl get nodes -o wide Expected: AIPC 1 is `Ready`, and each netbooted agent shows up as an `agent-` node and becomes `Ready`. -## Step 9 — Validate Agent Persistence +## Step 9 — Validate Agent State Across Reboots + +Choose the validation branch that matches the local-storage decision made before netboot. -Reboot one agent and confirm it rejoins with the same node identity rather than as a new node: +### Dedicated-disk branch + +Use this branch only when the agent exposes an approved dedicated disk. Reboot one agent and confirm it rejoins with the same node identity rather than as a new node: ```bash kubectl get nodes -o wide ``` -On the agent, confirm the persistent K3s data mount and node password exist: +On the agent, confirm the dedicated K3s data mount and saved node password exist: ```bash mount | grep /var/lib/rancher/k3s @@ -284,6 +306,14 @@ systemctl status mount-local-disk --no-pager systemctl status k3s-agent --no-pager ``` +These reboot, mount, and saved-password checks are required acceptance checks for the dedicated-disk branch. + +### No-candidate-disk branch + +Use this branch when none of `/dev/sda`, `/dev/vda`, or `/dev/nvme0n1` is exposed to the agent. The script mounts tmpfs at the K3s data directory. Node-local K3s state, the saved node password, container images, and container runtime state are therefore ephemeral and are lost across a reboot. Don't claim local persistence or apply the dedicated-disk reboot and saved-password acceptance checks to this branch. After every reboot, validate the agent's boot and cluster registration as a fresh node-local state cycle. + +Notebook data has a separate persistence boundary. After Step 11 configures `nfs-client` and the AUP site values select it, notebook PVC data is stored on the dedicated NFS export and can persist across agent reboot or rescheduling. That NFS-backed PVC persistence does not make the agent's tmpfs-backed K3s or container state persistent. + If an agent reboots but cannot rejoin, inspect the boot services on the agent: ```bash @@ -347,6 +377,12 @@ Labels: amd.com/gpu.cu-count=40 The PXE NFS rootfs is not the notebook storage backend. Create a separate NFS export for Kubernetes PVCs; it can run on AIPC 1 for a small lab. +::::::{danger} +Choose a new, dedicated `` directory for notebook PVCs. Don't use `/srv/nfs/rootfs`, and don't continue if the chosen path already contains data unless you have a verified, restorable backup. Back up `/etc/exports`, review `` so the export isn't open to a broader network, and confirm that an equivalent export entry doesn't already exist before appending one. + +The recursive ownership and mode commands below change every existing item under ``, and mode `0777` permits all local users to write there. The `no_root_squash` option gives remote root broad access to the export. Continue only if the dedicated path, subnet boundary, permissions, and remote-root risk are approved by the site's storage and security policy. +:::::: + ```bash sudo mkdir -p sudo chown -R nobody:nogroup @@ -383,100 +419,53 @@ kubectl get storageclass kubectl get pods -n nfs-provisioner ``` -## Step 12 — Configure JupyterHub Values +## Step 12: Apply The PXE Topology Deltas -Create a deployment-specific values file from the multi-node example: +First, from the deployment repository root, create `runtime/values-basic-example.yaml`. The guarded sequence stops without overwriting an existing site file: ```bash -cd ~/aup-learning-cloud/runtime -cp values-multi-nodes.yaml.example values-basic-example.yaml -# edit values-basic-example.yaml +( + set -e + if test -e runtime/values-basic-example.yaml; then + printf '%s\n' 'Refusing to overwrite runtime/values-basic-example.yaml; review the existing site file.' >&2 + exit 1 + fi + umask 077 + cp --no-clobber runtime/values-multi-nodes.yaml.example runtime/values-basic-example.yaml + chmod 600 runtime/values-basic-example.yaml +) ``` -At minimum set the auth mode, the GPU node selector to match your real labels, the notebook images, and the storage class. A NodePort proxy keeps the example simple: +After creating the file, complete the {ref}`K3s readiness checklist `. For this PXE topology, `runtime/values-basic-example.yaml` substitutes for the checklist's normal `runtime/values-multi-nodes.yaml` site-values gate. Require the PXE file to exist as a reviewed site file that wasn't created by overwriting prior configuration. All other checklist gates remain unchanged. The Ready-node gate includes AIPC 1 and every netbooted `agent-` node intended for notebook workloads. The GPU discovery gate must use the labels and allocatable resources observed on those agents in Step 10. -```yaml -custom: - authMode: "auto-login" - accelerators: - strix-halo: - nodeSelector: - amd.com/gpu.product-name: "" - quotaRate: 3 - resources: - images: - cpu: "" - gpu: "" - -hub: - db: - pvc: - storageClassName: nfs-client - -singleuser: - storage: - dynamic: - storageClass: nfs-client - -proxy: - service: - type: NodePort - nodePorts: - http: 30890 -``` +Then follow the {ref}`canonical Kubernetes preflight and deployment flow ` for values review, manifest inspection, installation, and infrastructure validation. Use `runtime/values-basic-example.yaml` as this topology's site values file when following those canonical steps. -::::::{note} -Use `authMode: "auto-login"` for this single-machine example — it is the chart's intended single-node default (see the comments in `runtime/chart/values.yaml`) and drops you straight in as the `student` user. Avoid `authMode: "dummy"` here: its login form posts to `/hub/native/login`, which is not loaded in dummy mode, so the login returns `404` and you cannot sign in. -:::::: - -::::::{warning} -Do not reuse a site-specific values override as-is. It may contain real hostnames, OAuth settings, image tags, or registry credentials that must be replaced. For a private registry, create the image pull secret in the `jupyterhub` namespace before installing the chart. -:::::: +Keep these PXE-specific choices in the site values: -## Step 13 — Deploy AUP Learning Cloud +- Point both the Hub database PVC and dynamic notebook PVCs at the `nfs-client` StorageClass from Step 11. Confirm that its provisioner still targets ``, not the read-only PXE rootfs at `/srv/nfs/rootfs`. +- Set each GPU resource's node selector from labels observed on a netbooted agent. Do not copy a product label from another cluster. +- For this isolated lab example, expose the proxy with a NodePort and reserve HTTP port `30890`. Use the canonical exposure gate before making the service reachable from an untrusted network. +- Select authentication and notebook images through the canonical values review. Keep credentials and private-registry secrets out of the values file and version control. -```bash -cd ~/aup-learning-cloud -helm upgrade --install jupyterhub ./runtime/chart \ - --namespace jupyterhub --create-namespace \ - -f runtime/values.yaml \ - -f runtime/values-basic-example.yaml -``` - -Wait for the pods, then open the Hub. For the NodePort example, browse to `http://:30890`: - -```bash -kubectl get pods -n jupyterhub -o wide -kubectl get svc -n jupyterhub -``` +## Step 13: Install Through The Canonical Flow -## Step 14 — End-To-End Validation +Complete the canonical render, inspection, install, and infrastructure-validation steps linked in Step 12, substituting `runtime/values-basic-example.yaml` for the canonical guide's example site values filename. Do not omit the explicit site values file from any Helm operation. -Validate the infrastructure first: - -```bash -kubectl get nodes -o wide -kubectl get pods -A -kubectl get storageclass -kubectl describe node | grep amd.com/gpu -``` - -Expected: AIPC 1 and both netbooted agents are `Ready`; no platform pod is stuck in `CrashLoopBackOff`, `Pending`, or `ImagePullBackOff`; `nfs-client` exists; and AMD GPU resources/labels appear on the agent nodes. - -Then validate the user path. Open AUP Learning Cloud in a browser — for the NodePort example from Step 13 that is `http://:30890` (use your ingress host instead if you configured one): +After the canonical infrastructure checks pass, the lab NodePort URL is: ```text http://:30890 ``` -![AUP Learning Cloud home page after login](../../_static/aup-learning-cloud-home.png) +## Step 14: Validate The PXE Deployment Deltas -Log in, spawn a CPU notebook, create a file in the notebook home, restart the notebook and confirm the file persists, then spawn a GPU notebook and confirm its pod lands on a netbooted agent: +Run the complete {ref}`canonical end-to-end acceptance `. In addition to those checks, require the following PXE-specific results: -```bash -kubectl get pods -n jupyterhub -o wide -kubectl describe pod -n jupyterhub -``` +- AIPC 1 and every intended `agent-` node are `Ready` after a netboot cycle. +- The `nfs-client` provisioner uses the separate notebook export ``; it does not store notebook PVC data in `/srv/nfs/rootfs`. +- The CPU notebook's persisted test file survives a server restart because it is backed by its notebook PVC, not by the agent's volatile rootfs overlay. +- The GPU notebook pod lands on a netbooted `agent-` node with the selected GPU label and a non-zero `amd.com/gpu` allocation. +- The NodePort URL `http://:30890` reaches the Hub from the intended lab network. ## Troubleshooting @@ -505,6 +494,6 @@ This is a minimal teaching/lab example, not a production reference. To keep it t - **No high availability.** There is one K3s server with embedded SQLite, no HA control plane, and no external database. - **Shared resource contention.** PXE/NFS/Apache/K3s-server and the notebook storage compete for the same CPU, memory, disk, and network on one box. - **Storage durability.** The example NFS export lives on AIPC 1's local disk with no replication or backup; treat notebook data as disposable unless you add your own backups. -- **Agents are volatile.** Netboot agents run from a read-only NFS rootfs with a tmpfs overlay; only the local K3s data dir persists across reboots. +- **Agents are volatile.** Netboot agents run from a read-only NFS rootfs with a tmpfs overlay. With an approved dedicated disk, only the local K3s data directory persists across reboots. Without a candidate disk, that directory also uses tmpfs, so all node-local K3s and container state is ephemeral. Notebook PVC persistence remains separate and depends on the configured NFS StorageClass. For a longer-running or production deployment, split these roles onto separate hosts, use an HA K3s control plane with an external/replicated datastore, and back the storage with a dedicated, redundant NFS (or other) backend. From 1d3a375733344a2f6508423f5c29ac5ae5c64c72 Mon Sep 17 00:00:00 2001 From: ShifZhan <252984256+MioYuuIH@users.noreply.github.com> Date: Fri, 31 Jul 2026 14:51:11 +0800 Subject: [PATCH 3/4] Update Kubernetes installation navigation --- source/index.rst | 11 +++++++++++ source/introduction/overview.md | 8 ++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/source/index.rst b/source/index.rst index 49f6b898..51a074aa 100644 --- a/source/index.rst +++ b/source/index.rst @@ -9,6 +9,16 @@ AUP Learning Cloud is a tailored JupyterHub deployment designed to provide an in :alt: Software Architecture :align: center +Choose Your Installation Path +----------------------------- + +Start with the path that matches what you already have: + +* **Local workstation:** Use the :doc:`Quick Start ` for a supported workstation that will run AUP Learning Cloud locally. +* **Existing Kubernetes:** Use :doc:`Deploy on an existing Kubernetes cluster ` for a running cluster that is ready for operator access. +* **New SSH-managed multi-node K3s:** Use the :doc:`multi-node K3s guide ` when each machine already has an operating system and is reachable over SSH. +* **Diskless PXE:** Use the :doc:`specialized PXE reference topology ` when diskless agents must network-boot from a service machine. + .. toctree:: :maxdepth: 2 :caption: Introduction @@ -24,6 +34,7 @@ AUP Learning Cloud is a tailored JupyterHub deployment designed to provide an in installation/quick-start installation/single-node installation/customizing-deployment + installation/existing-kubernetes installation/multi-node installation/multi-node/multi-aipc-hardware-deployment diff --git a/source/introduction/overview.md b/source/introduction/overview.md index f2f1ef12..ff31d024 100644 --- a/source/introduction/overview.md +++ b/source/introduction/overview.md @@ -60,9 +60,13 @@ The checked-in default values in this repository currently describe a local depl - ingress disabled - prePuller disabled -### Multi-Node +### Existing Kubernetes -Cluster deployments use the Ansible playbooks in `deploy/ansible/` plus Helm deployment with `runtime/values-multi-nodes.yaml.example` as the starting point. +Deploy AUP Learning Cloud onto an existing Kubernetes cluster when the cluster, nodes, networking, and storage are already managed for you. This path uses Helm and does not provision the cluster or its nodes. + +### New Multi-Node K3s + +When each machine already has an operating system and SSH access, use the playbooks in `deploy/ansible/` to provision K3s and prepare the hosts. After the cluster is ready, deploy AUP Learning Cloud with Helm, using `runtime/values-multi-nodes.yaml.example` as the starting point. NFS storage, ingress, TLS, and other production-oriented components are deployment choices, not mandatory defaults. From 6ba0964e3dbd7b22a65bf8d24b1028ce68917b2a Mon Sep 17 00:00:00 2001 From: ShifZhan <252984256+MioYuuIH@users.noreply.github.com> Date: Mon, 3 Aug 2026 19:24:59 +0800 Subject: [PATCH 4/4] Simplify Kubernetes runbook commands --- source/installation/existing-kubernetes.md | 89 ++++++++++------------ 1 file changed, 39 insertions(+), 50 deletions(-) diff --git a/source/installation/existing-kubernetes.md b/source/installation/existing-kubernetes.md index 58fd6747..2e72b0d9 100644 --- a/source/installation/existing-kubernetes.md +++ b/source/installation/existing-kubernetes.md @@ -99,33 +99,35 @@ Use exactly one Kubernetes GPU management path. Do not install the GPU Operator This is the direct AUP-matching path when AMD host drivers are already installed and managed outside Kubernetes. The following block uses the same immutable upstream revision and manifest checksums pinned by AUP. It requires `curl` and `sha256sum` on the operator machine: ```bash -( - set -euo pipefail - umask 077 - plugin_commit='dea1db13f05159e64d8114bca4c31f48c3cfcac6' - plugin_sha256='b751e467feecf6118bed1de8ba80b9abff01c1f52a6b0b8f31aca3609e6e9dbd' - labeller_sha256='c3e456967efdf14bcfeb97d8f87ca75a402cc6c7c8c6201a320efdd0370fa7aa' - work_dir="$(mktemp -d "${TMPDIR:-/tmp}/aup-amd-device-plugin.XXXXXX")" - trap 'rm -rf "$work_dir"' EXIT - base_url="https://raw.githubusercontent.com/ROCm/k8s-device-plugin/${plugin_commit}" - curl --fail --show-error --silent --location \ - "$base_url/k8s-ds-amdgpu-dp.yaml" \ - --output "$work_dir/k8s-ds-amdgpu-dp.yaml" - curl --fail --show-error --silent --location \ - "$base_url/k8s-ds-amdgpu-labeller.yaml" \ - --output "$work_dir/k8s-ds-amdgpu-labeller.yaml" - printf '%s %s\n' "$plugin_sha256" "$work_dir/k8s-ds-amdgpu-dp.yaml" | sha256sum --check - - printf '%s %s\n' "$labeller_sha256" "$work_dir/k8s-ds-amdgpu-labeller.yaml" | sha256sum --check - - kubectl apply -f "$work_dir/k8s-ds-amdgpu-dp.yaml" - kubectl apply -f "$work_dir/k8s-ds-amdgpu-labeller.yaml" - kubectl rollout status --namespace kube-system \ - daemonset/amdgpu-device-plugin-daemonset --timeout=5m - kubectl rollout status --namespace kube-system \ - daemonset/amdgpu-labeller-daemonset --timeout=5m -) +curl -fsSL \ + https://raw.githubusercontent.com/ROCm/k8s-device-plugin/dea1db13f05159e64d8114bca4c31f48c3cfcac6/k8s-ds-amdgpu-dp.yaml \ + -o /tmp/k8s-ds-amdgpu-dp.yaml + +curl -fsSL \ + https://raw.githubusercontent.com/ROCm/k8s-device-plugin/dea1db13f05159e64d8114bca4c31f48c3cfcac6/k8s-ds-amdgpu-labeller.yaml \ + -o /tmp/k8s-ds-amdgpu-labeller.yaml + +echo 'b751e467feecf6118bed1de8ba80b9abff01c1f52a6b0b8f31aca3609e6e9dbd /tmp/k8s-ds-amdgpu-dp.yaml' \ + | sha256sum --check +echo 'c3e456967efdf14bcfeb97d8f87ca75a402cc6c7c8c6201a320efdd0370fa7aa /tmp/k8s-ds-amdgpu-labeller.yaml' \ + | sha256sum --check ``` -The block fails before applying anything if either download or checksum verification fails. `kubectl apply` is idempotent, both DaemonSets must roll out successfully, and the private temporary directory is removed on exit. +Both checksum commands must report `OK`. Stop and delete the downloaded files if either check fails. When both checks pass, apply the manifests: + +```bash +kubectl apply -f /tmp/k8s-ds-amdgpu-dp.yaml +kubectl apply -f /tmp/k8s-ds-amdgpu-labeller.yaml + +kubectl rollout status --namespace kube-system \ + daemonset/amdgpu-device-plugin-daemonset --timeout=5m +kubectl rollout status --namespace kube-system \ + daemonset/amdgpu-labeller-daemonset --timeout=5m + +rm -f /tmp/k8s-ds-amdgpu-dp.yaml /tmp/k8s-ds-amdgpu-labeller.yaml +``` + +The apply commands are idempotent, and both DaemonSets must roll out successfully. ### AMD GPU Operator Alternative @@ -205,16 +207,10 @@ Site values may contain OAuth, administrator, registry, or other credentials. Pr ```bash cd -( - set -eu - site_values='runtime/values-existing-cluster.yaml' - if test -e "$site_values"; then - printf 'Refusing to overwrite existing %s\n' "$site_values" >&2 - exit 1 - fi - cp --no-clobber runtime/values-multi-nodes.yaml.example "$site_values" - chmod 600 "$site_values" -) +cp --no-clobber \ + runtime/values-multi-nodes.yaml.example \ + runtime/values-existing-cluster.yaml +chmod 600 runtime/values-existing-cluster.yaml ls -l runtime/values-existing-cluster.yaml ``` @@ -225,7 +221,7 @@ The example is not production-ready. It currently contains site-specific samples ```bash grep -nE 'nfs-client|:latest|supplementalGroups|TODO| helm lint runtime/chart \ --namespace jupyterhub \ -f runtime/values-existing-cluster.yaml -( - set -eu - umask 077 - rendered_manifest="$(mktemp "${TMPDIR:-/tmp}/aup-learning-cloud-rendered.XXXXXX.yaml")" - trap 'rm -f "$rendered_manifest"' EXIT - helm template jupyterhub runtime/chart \ - --namespace jupyterhub \ - -f runtime/values-existing-cluster.yaml > "$rendered_manifest" - less "$rendered_manifest" -) + +helm template jupyterhub runtime/chart \ + --namespace jupyterhub \ + -f runtime/values-existing-cluster.yaml | less ``` -The temporary manifest can contain Secrets or derived sensitive values. Do not publish or retain it; the trap removes it when the subshell exits. +The rendered output can contain Secrets or derived sensitive values. Review it in the terminal; do not save, publish, or upload it. Before installation, confirm: @@ -386,15 +376,14 @@ kubectl get pods --namespace jupyterhub -l component=singleuser-server -o wide GPU_USER_POD='' kubectl get pod --namespace jupyterhub "$GPU_USER_POD" \ -o jsonpath='{.spec.nodeName}{"\n"}{.spec.containers[0].resources}{"\n"}' -GPU_NODE="$(kubectl get pod --namespace jupyterhub "$GPU_USER_POD" -o jsonpath='{.spec.nodeName}')" -kubectl get node "$GPU_NODE" -L amd.com/gpu.product-name +kubectl get nodes -L amd.com/gpu.product-name ``` The Pod must request `amd.com/gpu` and land on a node with the product label selected by its AUP resource. Resolve a missing request, selector mismatch, pending Pod, or wrong-node placement before continuing. ### Verify Host Device Modes -Connect to the exact `GPU_NODE` through the site's approved node-access method. There is no universal node-login command. +Connect to the node reported by the selected GPU Pod through the site's approved node-access method. There is no universal node-login command. **GPU node:**