diff --git a/crusoe-kserve-example/README.md b/crusoe-kserve-example/README.md index 1ab490d..72d6747 100644 --- a/crusoe-kserve-example/README.md +++ b/crusoe-kserve-example/README.md @@ -173,6 +173,8 @@ make deploy-amd-mi355x AUTOSCALE=1 # emit spec.scaling.wva.hpa (KServe-nativ 2. **EPP throughput ceiling — architectural.** The single leader-elected endpoint-picker (`llm-d-inference-scheduler`) serializes per-request routing and caps end-to-end throughput at **~1.6k tok/s (~12–17% of the ~9.7k tok/s the two replicas deliver when driven directly)**, independent of its CPU (3 vs 8: no change) or scoring config. Bumping it 256m→3 CPU (chart default; it scales *up*, not out) still roughly halved TPOT and lifted req/s ~25–46%. For max throughput, **load-balance clients across replica endpoints directly** (`bench-amd-mi355x-all` path); use the gateway for smart prefix/queue-aware routing at moderate load. 3. **Pod autoscaling is blocked three ways on a base CMK cluster.** `vllm:num_requests_waiting` stays ~0 under gateway load (the EPP gates admission, so congestion sits in the gateway, not the vLLM queue); a standalone KEDA ScaledObject can't actuate (KServe owns `spec.replicas` and reverts external scaling in <10s); and the native `spec.scaling.wva.hpa` path (`AUTOSCALE=1`) needs a **`VariantAutoscaling` CRD** (llm-d workload-variant-autoscaler operator) **and** a **metrics-server** — neither ships with base KServe or CMK, so the ISVC goes `Ready=False` (`ScalingCRDNotFound`) until they're installed. `install-vllm-hpa` installs the metric pipeline (on the more reliable `num_requests_running`) for observability; enabling `AUTOSCALE=1` requires installing those operators first. +**Scrape opt-in convention.** The Prometheus that `install-vllm-hpa` ships (`prometheus-vllm` in ns `vllm-metrics`) discovers vLLM pods in either of two ways — a pod is scraped if it carries the label `kserve.io/component=workload` (automatic for KServe LLMInferenceService workloads) **or** BOTH pod annotations `prometheus.io/scrape: "true"` and `prometheus.io/port: "8000"` (the hand-rolled recipes under `recipes/` carry them). A pod matching **both** paths is scraped twice (both jobs hit it), so annotate only non-KServe deployment types and let KServe-managed workloads rely on the label. Grafana integration lives in [grafana-cmk](../grafana-cmk/): apply `manifests/grafana-datasource-vllm-configmap.yaml` there to point Grafana at this Prometheus (optional — skip if you don't run that stack), and its **Inference / vLLM Overview** dashboard (TTFT/TPOT p50/p90/p99 with SLA thresholds, prefix-cache hit %, tokens/s per node, KV-cache usage, request outcomes, top endpoints table) populates off these series. + > **⚠️ Known issue — shared-disk topology-label lag.** New autoscaled nodes receive the `fs.csi.crusoe.ai/*` labels that the RWX weights PV's `nodeAffinity` requires ~4.5 min *after* the node is `Ready` (the driver works immediately, only the labels lag). During that gap the new replica can't schedule **and the autoscaler over-provisions** (adds a 2nd node for 1 replica) before reclaiming the extra. Scale-up to a serving replica is ~26 min (dominated by the 30 GB image pull + AITER compile, not node provisioning). ### 4. Chat diff --git a/crusoe-kserve-example/manifests/prometheus-vllm.yaml b/crusoe-kserve-example/manifests/prometheus-vllm.yaml index e18e047..0c4ec0f 100644 --- a/crusoe-kserve-example/manifests/prometheus-vllm.yaml +++ b/crusoe-kserve-example/manifests/prometheus-vllm.yaml @@ -1,7 +1,21 @@ # Lightweight in-cluster Prometheus that scrapes the vLLM workload pods' :8000/metrics. -# Needed because CMK's vmagent only remote-writes to an external endpoint (no queryable local API), -# and KEDA needs a Prometheus it can query for vllm:num_requests_waiting (the autoscaling signal). -# Applied by `make install-vllm-hpa`. +# Needed because CMK's vmagent only remote-writes to an external endpoint (no queryable local API). +# +# Consumers: +# - KEDA (vllm:num_requests_waiting autoscaling signal — applied by `make install-vllm-hpa`) +# - The grafana-cmk "Inference" dashboards (datasource points at svc prometheus-vllm:9090) +# +# Target detection (no namespace pinning — works cluster-wide): +# 1. KServe-managed pods: label kserve.io/component=workload (automatic on +# LLMInferenceService workloads). +# 2. Any other vLLM pod: annotate it with BOTH prometheus.io/scrape: "true" +# and prometheus.io/port: "8000" — e.g. the hand-rolled kimi-k3-atom +# Deployment. Pod annotations are picked up live (a pod-restart is not +# required), though the annotations belong in the pod template so new +# pods carry them from birth. +# The port match is deliberate: many infra images ship the generic +# prometheus.io/scrape annotation; without the port filter they would +# drown this job in permanently-down :8000 targets. --- apiVersion: v1 kind: Namespace @@ -35,12 +49,13 @@ data: scrape_interval: 5s evaluation_interval: 5s scrape_configs: + # Job 1 — KServe-managed vLLM pods (LLMInferenceService workloads in ANY + # namespace): they carry kserve.io/component=workload automatically. + # Excludes router-scheduler/EPP pods, which don't serve :8000/metrics. - job_name: vllm kubernetes_sd_configs: - role: pod - namespaces: { names: [kserve-test] } relabel_configs: - # keep only the KServe vLLM workload pods (exclude router/EPP) - source_labels: [__meta_kubernetes_pod_label_kserve_io_component] regex: workload action: keep @@ -49,6 +64,36 @@ data: regex: (.+) replacement: $1:8000 target_label: __address__ + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace + - source_labels: [__meta_kubernetes_pod_name] + target_label: pod + - source_labels: [__meta_kubernetes_pod_node_name] + target_label: node + # Job 2 — opt-in for non-KServe vLLM deployments (e.g. hand-rolled + # Deployments): annotate the pod with BOTH: + # prometheus.io/scrape: "true" + # prometheus.io/port: "8000" + # The port match is required to keep infra pods (cilium, cert-manager, + # envoy gateway, ...) that carry the generic prometheus.io/scrape + # annotation out of this vLLM-scoped job — they serve metrics on other + # ports and would appear as permanently-down targets. + - job_name: vllm-annotated + kubernetes_sd_configs: + - role: pod + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + regex: "true" + action: keep + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port] + regex: "8000" + action: keep + - source_labels: [__meta_kubernetes_pod_ip] + regex: (.+) + replacement: $1:8000 + target_label: __address__ + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace - source_labels: [__meta_kubernetes_pod_name] target_label: pod - source_labels: [__meta_kubernetes_pod_node_name] @@ -69,7 +114,7 @@ spec: image: prom/prometheus:v3.1.0 args: - --config.file=/etc/prometheus/prometheus.yml - - --storage.tsdb.retention.time=2h + - --storage.tsdb.retention.time=7d - --web.enable-lifecycle ports: [{ containerPort: 9090 }] volumeMounts: diff --git a/crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/README.md b/crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/README.md new file mode 100644 index 0000000..75af7f4 --- /dev/null +++ b/crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/README.md @@ -0,0 +1,86 @@ +# Kimi K3 on MI355X — AMD ATOM (vLLM plugin), fast + correct with CUDA graph capture + +Serve **Moonshot AI Kimi K3** (2.8T-param MoE, native MXFP4, 1M context) on a **single 8-GPU AMD +MI355X (gfx950/CDNA4)** node using **AMD's ATOM engine** (its vLLM out-of-tree plugin), tensor-parallel +across the 8 GPUs, OpenAI-compatible. + +## Why ATOM + +Kimi K3's MXFP4 latent MoE **corrupts under CUDA-graph capture** on the day-0 vLLM ROCm stack, and +even SGLang has to run **eager** (capture off) to stay correct — which costs a large amount of decode +throughput. ATOM ships an AITER MXFP4 MoE path that is correct **with** `FULL_AND_PIECEWISE` graph +capture (AMD validates GSM8K 0.9553 on 8×MI355 TP8), so you keep both correctness and speed. + +Measured here (8×MI355X TP8, random 512-in/200-out) vs the same model on SGLang eager: + +| Metric | ATOM (capture ON) | SGLang eager | +|---|---|---| +| TPOT @ 1 concurrent (per-user) | **~20 ms (~50 tok/s)** | ~143 ms (~7 tok/s) | +| TPOT @ 32 concurrent | ~29 ms | — | +| Aggregate output @ 32 concurrent | **~950 tok/s** | — | + +Correctness spot-checks: deterministic `42` smoke test, exact recall at ~97k-token context, coherent +tool calls with populated `content`, and a 32-way concurrency burst clean. + +## Prerequisites + +- A Crusoe Managed Kubernetes cluster with an **MI355X node pool** and a namespace holding a + HuggingFace secret named `hf-secret` (HF access to `moonshotai/Kimi-K3`). The repo root's + `make setup-amd` provisions the cluster and secret. +- The image `rocm/atom-dev:vllm-kimi-k3-20260807` is **public** on Docker Hub. If your nodes can't + reach Docker Hub (or you hit rate limits), mirror it into your own registry and update `image:` in + `kimi-k3-atom.yaml` (2 places). Mirror in-cluster with skopeo (no local Docker needed): + + ```bash + skopeo copy --authfile /path/to/dockerconfig.json \ + docker://docker.io/rocm/atom-dev:vllm-kimi-k3-20260807 \ + docker:///kimi-k3-atom:vllm-k3-20260807 + ``` + +## Deploy + +```bash +kubectl apply -n kserve-test -f kimi-k3-atom.yaml +``` + +This creates a ReadWriteMany weights disk (`kimi-k3-weights`, 2 TiB) and the `kimi-k3-atom` +Deployment + Service. **First boot is ~45–50 min**: a one-time ~1.5 TB weight download (init +container), then ATOM's staging loader reads the weights (network-disk bound), online-quantizes +non-MXFP4 tensors to `ptpc_fp8`, JIT-compiles the AITER kernels, and captures CUDA graphs. Watch it: + +```bash +kubectl -n kserve-test get pods -l app=kimi-k3-atom -w +kubectl -n kserve-test logs -l app=kimi-k3-atom -c main -f +``` + +## Use + +Served OpenAI-compatibly as `kimi-k3`. In-cluster: + +```bash +curl http://kimi-k3-atom-workload-svc.kserve-test.svc.cluster.local:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"kimi-k3","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}' +``` + +Tool calling and reasoning parsing are enabled: thinking is separated into `reasoning_content`, and +`content` carries the clean answer — so agent clients (e.g. opencode) that read `content` + `tool_calls` +work directly. + +## Notes / gotchas + +- **The critical flags** (baked into `kimi-k3-atom.yaml`, from the image's own recipe at + `/app/ATOM/recipes/atom_vllm/Kimi-K3.md`): + - `--additional-config '{"online_quant_config":{"global_quant_config":"ptpc_fp8",...}}'` — routes the + MoE through ATOM's AITER path. **Without it every token is garbage**: vLLM otherwise dispatches the + MXFP4 MoE to a generic Triton kernel that isn't present in the image. + - env `AITER_SITUV2_A4W4=1` (K3 MoE is A4W4, **not** A8W4) and `VLLM_USE_BREAKABLE_CUDAGRAPH=0`. + - `--kv-cache-dtype fp8` is validated on this build's aiter branch (it ships the fp8 MLA kernels). + - `--mamba-cache-mode align` is required for prefix caching on K3's hybrid KDA+MLA attention. +- **Single-node rolling update:** with one 8-GPU node the Deployment uses `strategy: Recreate`. If a + re-apply appears stuck, the old pod may be lingering in `Succeeded` (the server exits 0 on SIGTERM), + blocking Recreate — clear it with `kubectl delete pod --grace-period=0 --force`. +- **Load time** (~30 min of the boot) is the network-disk weight read; it dominates restart time and is + independent of serving speed. `--load-format fastsafetensors` is overridden by ATOM's staging loader. +- **Scaling:** K3 fits on one 8-GPU node, so add capacity with more `replicas` (independent full copies + behind a router), not multi-node tensor/expert parallel. diff --git a/crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/kimi-k3-atom.yaml b/crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/kimi-k3-atom.yaml new file mode 100644 index 0000000..1c17c18 --- /dev/null +++ b/crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/kimi-k3-atom.yaml @@ -0,0 +1,164 @@ +# Kimi K3 on AMD MI355X (gfx950) via AMD ATOM (vLLM plugin) — single-node TP=8, OpenAI-compatible. +# +# Why ATOM: Kimi K3's MXFP4 latent MoE corrupts under CUDA-graph capture on the day-0 vLLM ROCm +# stack, and even SGLang has to run eager (graph capture off) to stay correct — which costs decode +# throughput. AMD's ATOM engine ships an AITER MXFP4 MoE path that is correct WITH FULL_AND_PIECEWISE +# CUDA-graph capture (AMD validates GSM8K 0.9553 on 8xMI355 TP8), restoring fast + correct decode. +# The canonical launch below is from the image's own recipe: /app/ATOM/recipes/atom_vllm/Kimi-K3.md +# +# BEFORE APPLYING: +# 1. `rocm/atom-dev:vllm-kimi-k3-20260807` is public on Docker Hub. If your nodes can't reach +# Docker Hub (or you hit rate limits), mirror it into your own registry and set `image:` below +# (2 places), plus an imagePullSecret. Mirror in-cluster with skopeo: +# skopeo copy --authfile \ +# docker://docker.io/rocm/atom-dev:vllm-kimi-k3-20260807 \ +# docker:///kimi-k3-atom:vllm-k3-20260807 +# 2. Ensure an `hf-secret` with key HF_TOKEN exists in this namespace (HuggingFace access to +# moonshotai/Kimi-K3). The repo root's `make setup-amd` creates it. +# 3. Apply: kubectl apply -n kserve-test -f recipe-kimi-k3-atom.yaml +# +# METRICS: the pod carries prometheus.io/scrape + prometheus.io/port=8000 annotations, so the +# "vllm-annotated" job of manifests/prometheus-vllm.yaml picks it up in any namespace (no pod +# restart needed even if annotated later — pod annotations are watched live). +# +# First boot downloads the ~1.5 TB MXFP4 weights to the RWX disk (init container, one time). ATOM's +# staging loader then reads them (~30 min on network storage), online-quantizes non-MXFP4 tensors to +# ptpc_fp8, JIT-compiles AITER kernels, and captures CUDA graphs — allow ~45-50 min to Ready on first +# boot. Later restarts skip the download but still re-load + re-capture. +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: crusoe-fs +provisioner: fs.csi.crusoe.ai +volumeBindingMode: Immediate +reclaimPolicy: Delete +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: kimi-k3-weights +spec: + accessModes: [ReadWriteMany] # RWX so a rolling update can stage the new pod on another node + storageClassName: crusoe-fs + resources: + requests: + storage: 2Ti # min 1 TiB for fs.csi.crusoe.ai; 2Ti fits the 1.5 TB weights +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: kimi-k3-atom + labels: { app: kimi-k3-atom } +spec: + replicas: 1 + strategy: { type: Recreate } # single 8-GPU node: kill the old pod before starting the new one + selector: + matchLabels: { app: kimi-k3-atom } + template: + metadata: + labels: { app: kimi-k3-atom } + annotations: + # Opt-in for the prometheus-vllm "vllm-annotated" scrape job (manifests/prometheus-vllm.yaml) — + # picked up cluster-wide, in any namespace. Both annotations are required. + prometheus.io/scrape: "true" + prometheus.io/port: "8000" + spec: + nodeSelector: + crusoe.ai/accelerator: amd-mi355x-288gb-roce + securityContext: + fsGroup: 1000 + # imagePullSecrets: # uncomment + name your secret if the registry is private + # - name: + initContainers: + # One-time weight download to the RWX disk (skipped if already present). Not gated by the + # serving probes, so it can take as long as the 1.5 TB pull needs. + - name: fetch-weights + image: rocm/atom-dev:vllm-kimi-k3-20260807 # same image (has huggingface_hub) + command: ["/bin/sh", "-c"] + args: + - | + if [ -f /models/config.json ]; then echo "weights present, skipping download"; exit 0; fi + python3 -c "from huggingface_hub import snapshot_download; snapshot_download('moonshotai/Kimi-K3', local_dir='/models', local_dir_use_symlinks=False)" + env: + - name: HF_TOKEN + valueFrom: { secretKeyRef: { name: hf-secret, key: HF_TOKEN } } + volumeMounts: + - { name: models, mountPath: /models } + containers: + - name: main + image: rocm/atom-dev:vllm-kimi-k3-20260807 + workingDir: /app # /app/vllm shadows site-packages -> uses ATOM's cache_aware patch + command: ["/bin/sh", "-c"] + # Canonical vLLM-ATOM K3 launch (image's /app/ATOM/recipes/atom_vllm/Kimi-K3.md). + # --additional-config online_quant_config routes the MoE through ATOM's ptpc_fp8/AITER path + # (the generic vLLM Triton MXFP4 kernel isn't in this image and yields garbage without it). + args: + - | + set -e + cd /app + exec vllm serve /models \ + --host 0.0.0.0 --port 8000 \ + --served-model-name kimi-k3 \ + --tensor-parallel-size 8 \ + --trust-remote-code \ + --max-model-len 131072 \ + --enable-prefix-caching \ + --mamba-cache-mode align \ + --kv-cache-dtype fp8 \ + --max-num-seqs 64 \ + --max-num-batched-tokens 16384 \ + --gpu-memory-utilization 0.93 \ + --block-size 128 \ + --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ + --enable-auto-tool-choice \ + --tool-call-parser kimi_k3 \ + --reasoning-parser kimi_k3 \ + --additional-config '{"online_quant_config": {"global_quant_config": "ptpc_fp8", "exclude_layer": ["lm_head", "model.embed_tokens", "*self_attn.[qkv]_conv1d*", "*block_sparse_moe.experts*", "*block_sparse_moe.routed_expert_*", "*vision_tower*", "*mm_projector*"]}}' + env: + - name: HF_TOKEN + valueFrom: { secretKeyRef: { name: hf-secret, key: HF_TOKEN } } + - { name: HF_HUB_OFFLINE, value: "1" } # weights already on /models + - { name: VLLM_ROCM_USE_AITER, value: "1" } + - { name: AITER_SITUV2_A4W4, value: "1" } # K3 MoE is A4W4 (recipe); NOT A8W4 + - { name: AITER_SITUV2_A8W4, value: "0" } + - { name: VLLM_USE_BREAKABLE_CUDAGRAPH, value: "0" } # build auto-enables =1 which crashes at capture + ports: + - { containerPort: 8000, name: http } + resources: + limits: { amd.com/gpu: "8", cpu: "64", memory: 1Ti } + requests: { amd.com/gpu: "8", cpu: "16", memory: 128Gi } + securityContext: + runAsNonRoot: false + runAsUser: 0 + startupProbe: + httpGet: { path: /health, port: 8000 } + periodSeconds: 15 + failureThreshold: 240 # ~60 min: network-disk weight read + online quant + compile + capture + timeoutSeconds: 10 + readinessProbe: + httpGet: { path: /health, port: 8000 } + periodSeconds: 30 + timeoutSeconds: 8 + failureThreshold: 3 + volumeMounts: + - { name: models, mountPath: /models } + - { name: dshm, mountPath: /dev/shm } + volumes: + - name: models + persistentVolumeClaim: + claimName: kimi-k3-weights + - name: dshm + emptyDir: { medium: Memory, sizeLimit: 128Gi } # TP8 intra-node comms need large /dev/shm +--- +apiVersion: v1 +kind: Service +metadata: + name: kimi-k3-atom-workload-svc + labels: { app: kimi-k3-atom } +spec: + selector: { app: kimi-k3-atom } + ports: + - name: http + port: 8000 + targetPort: 8000