kserve-example: Kimi K3 MI355X ATOM recipe + generalized vLLM scraping & Grafana wiring

crusoe-kserve-example/README.md

@@ -173,6 +173,8 @@ make deploy-amd-mi355x AUTOSCALE=1     # emit spec.scaling.wva.hpa (KServe-nativ
 2. **EPP throughput ceiling — architectural.** The single leader-elected endpoint-picker (`llm-d-inference-scheduler`) serializes per-request routing and caps end-to-end throughput at **~1.6k tok/s (~12–17% of the ~9.7k tok/s the two replicas deliver when driven directly)**, independent of its CPU (3 vs 8: no change) or scoring config. Bumping it 256m→3 CPU (chart default; it scales *up*, not out) still roughly halved TPOT and lifted req/s ~25–46%. For max throughput, **load-balance clients across replica endpoints directly** (`bench-amd-mi355x-all` path); use the gateway for smart prefix/queue-aware routing at moderate load.
 3. **Pod autoscaling is blocked three ways on a base CMK cluster.** `vllm:num_requests_waiting` stays ~0 under gateway load (the EPP gates admission, so congestion sits in the gateway, not the vLLM queue); a standalone KEDA ScaledObject can't actuate (KServe owns `spec.replicas` and reverts external scaling in <10s); and the native `spec.scaling.wva.hpa` path (`AUTOSCALE=1`) needs a **`VariantAutoscaling` CRD** (llm-d workload-variant-autoscaler operator) **and** a **metrics-server** — neither ships with base KServe or CMK, so the ISVC goes `Ready=False` (`ScalingCRDNotFound`) until they're installed. `install-vllm-hpa` installs the metric pipeline (on the more reliable `num_requests_running`) for observability; enabling `AUTOSCALE=1` requires installing those operators first.
 
+**Scrape opt-in convention.** The Prometheus that `install-vllm-hpa` ships (`prometheus-vllm` in ns `vllm-metrics`) discovers vLLM pods in either of two ways — a pod is scraped if it carries the label `kserve.io/component=workload` (automatic for KServe LLMInferenceService workloads) **or** BOTH pod annotations `prometheus.io/scrape: "true"` and `prometheus.io/port: "8000"` (the hand-rolled recipes under `recipes/` carry them). A pod matching **both** paths is scraped twice (both jobs hit it), so annotate only non-KServe deployment types and let KServe-managed workloads rely on the label. Grafana integration lives in [grafana-cmk](../grafana-cmk/): apply `manifests/grafana-datasource-vllm-configmap.yaml` there to point Grafana at this Prometheus (optional — skip if you don't run that stack), and its **Inference / vLLM Overview** dashboard (TTFT/TPOT p50/p90/p99 with SLA thresholds, prefix-cache hit %, tokens/s per node, KV-cache usage, request outcomes, top endpoints table) populates off these series.
+
 > **⚠️ Known issue — shared-disk topology-label lag.** New autoscaled nodes receive the `fs.csi.crusoe.ai/*` labels that the RWX weights PV's `nodeAffinity` requires ~4.5 min *after* the node is `Ready` (the driver works immediately, only the labels lag). During that gap the new replica can't schedule **and the autoscaler over-provisions** (adds a 2nd node for 1 replica) before reclaiming the extra. Scale-up to a serving replica is ~26 min (dominated by the 30 GB image pull + AITER compile, not node provisioning).
 
 ### 4. Chat

crusoe-kserve-example/manifests/prometheus-vllm.yaml

@@ -1,7 +1,21 @@
 # Lightweight in-cluster Prometheus that scrapes the vLLM workload pods' :8000/metrics.
-# Needed because CMK's vmagent only remote-writes to an external endpoint (no queryable local API),
-# and KEDA needs a Prometheus it can query for vllm:num_requests_waiting (the autoscaling signal).
-# Applied by `make install-vllm-hpa`.
+# Needed because CMK's vmagent only remote-writes to an external endpoint (no queryable local API).
+#
+# Consumers:
+#   - KEDA (vllm:num_requests_waiting autoscaling signal — applied by `make install-vllm-hpa`)
+#   - The grafana-cmk "Inference" dashboards (datasource points at svc prometheus-vllm:9090)
+#
+# Target detection (no namespace pinning — works cluster-wide):
+#   1. KServe-managed pods: label kserve.io/component=workload (automatic on
+#      LLMInferenceService workloads).
+#   2. Any other vLLM pod: annotate it with BOTH prometheus.io/scrape: "true"
+#      and prometheus.io/port: "8000" — e.g. the hand-rolled kimi-k3-atom
+#      Deployment. Pod annotations are picked up live (a pod-restart is not
+#      required), though the annotations belong in the pod template so new
+#      pods carry them from birth.
+#      The port match is deliberate: many infra images ship the generic
+#      prometheus.io/scrape annotation; without the port filter they would
+#      drown this job in permanently-down :8000 targets.
 ---
 apiVersion: v1
 kind: Namespace
@@ -35,12 +49,13 @@ data:
       scrape_interval: 5s
       evaluation_interval: 5s
     scrape_configs:
+      # Job 1 — KServe-managed vLLM pods (LLMInferenceService workloads in ANY
+      # namespace): they carry kserve.io/component=workload automatically.
+      # Excludes router-scheduler/EPP pods, which don't serve :8000/metrics.
       - job_name: vllm
         kubernetes_sd_configs:
           - role: pod
-            namespaces: { names: [kserve-test] }
         relabel_configs:
-          # keep only the KServe vLLM workload pods (exclude router/EPP)
           - source_labels: [__meta_kubernetes_pod_label_kserve_io_component]
             regex: workload
             action: keep
@@ -49,6 +64,36 @@ data:
             regex: (.+)
             replacement: $1:8000
             target_label: __address__
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+      # Job 2 — opt-in for non-KServe vLLM deployments (e.g. hand-rolled
+      # Deployments): annotate the pod with BOTH:
+      #   prometheus.io/scrape: "true"
+      #   prometheus.io/port: "8000"
+      # The port match is required to keep infra pods (cilium, cert-manager,
+      # envoy gateway, ...) that carry the generic prometheus.io/scrape
+      # annotation out of this vLLM-scoped job — they serve metrics on other
+      # ports and would appear as permanently-down targets.
+      - job_name: vllm-annotated
+        kubernetes_sd_configs:
+          - role: pod
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+            regex: "true"
+            action: keep
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
+            regex: "8000"
+            action: keep
+          - source_labels: [__meta_kubernetes_pod_ip]
+            regex: (.+)
+            replacement: $1:8000
+            target_label: __address__
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
           - source_labels: [__meta_kubernetes_pod_name]
             target_label: pod
           - source_labels: [__meta_kubernetes_pod_node_name]
@@ -69,7 +114,7 @@ spec:
           image: prom/prometheus:v3.1.0
           args:
             - --config.file=/etc/prometheus/prometheus.yml
-            - --storage.tsdb.retention.time=2h
+            - --storage.tsdb.retention.time=7d
             - --web.enable-lifecycle
           ports: [{ containerPort: 9090 }]
           volumeMounts:

crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/README.md

@@ -0,0 +1,86 @@
+# Kimi K3 on MI355X — AMD ATOM (vLLM plugin), fast + correct with CUDA graph capture
+
+Serve **Moonshot AI Kimi K3** (2.8T-param MoE, native MXFP4, 1M context) on a **single 8-GPU AMD
+MI355X (gfx950/CDNA4)** node using **AMD's ATOM engine** (its vLLM out-of-tree plugin), tensor-parallel
+across the 8 GPUs, OpenAI-compatible.
+
+## Why ATOM
+
+Kimi K3's MXFP4 latent MoE **corrupts under CUDA-graph capture** on the day-0 vLLM ROCm stack, and
+even SGLang has to run **eager** (capture off) to stay correct — which costs a large amount of decode
+throughput. ATOM ships an AITER MXFP4 MoE path that is correct **with** `FULL_AND_PIECEWISE` graph
+capture (AMD validates GSM8K 0.9553 on 8×MI355 TP8), so you keep both correctness and speed.
+
+Measured here (8×MI355X TP8, random 512-in/200-out) vs the same model on SGLang eager:
+
+| Metric | ATOM (capture ON) | SGLang eager |
+|---|---|---|
+| TPOT @ 1 concurrent (per-user) | **~20 ms (~50 tok/s)** | ~143 ms (~7 tok/s) |
+| TPOT @ 32 concurrent | ~29 ms | — |
+| Aggregate output @ 32 concurrent | **~950 tok/s** | — |
+
+Correctness spot-checks: deterministic `42` smoke test, exact recall at ~97k-token context, coherent
+tool calls with populated `content`, and a 32-way concurrency burst clean.
+
+## Prerequisites
+
+- A Crusoe Managed Kubernetes cluster with an **MI355X node pool** and a namespace holding a
+  HuggingFace secret named `hf-secret` (HF access to `moonshotai/Kimi-K3`). The repo root's
+  `make setup-amd` provisions the cluster and secret.
+- The image `rocm/atom-dev:vllm-kimi-k3-20260807` is **public** on Docker Hub. If your nodes can't
+  reach Docker Hub (or you hit rate limits), mirror it into your own registry and update `image:` in
+  `kimi-k3-atom.yaml` (2 places). Mirror in-cluster with skopeo (no local Docker needed):
+
+  ```bash
+  skopeo copy --authfile /path/to/dockerconfig.json \
+    docker://docker.io/rocm/atom-dev:vllm-kimi-k3-20260807 \
+    docker://<your-registry>/kimi-k3-atom:vllm-k3-20260807
+  ```
+
+## Deploy
+
+```bash
+kubectl apply -n kserve-test -f kimi-k3-atom.yaml
+```
+
+This creates a ReadWriteMany weights disk (`kimi-k3-weights`, 2 TiB) and the `kimi-k3-atom`
+Deployment + Service. **First boot is ~45–50 min**: a one-time ~1.5 TB weight download (init
+container), then ATOM's staging loader reads the weights (network-disk bound), online-quantizes
+non-MXFP4 tensors to `ptpc_fp8`, JIT-compiles the AITER kernels, and captures CUDA graphs. Watch it:
+
+```bash
+kubectl -n kserve-test get pods -l app=kimi-k3-atom -w
+kubectl -n kserve-test logs -l app=kimi-k3-atom -c main -f
+```
+
+## Use
+
+Served OpenAI-compatibly as `kimi-k3`. In-cluster:
+
+```bash
+curl http://kimi-k3-atom-workload-svc.kserve-test.svc.cluster.local:8000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"kimi-k3","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'
+```
+
+Tool calling and reasoning parsing are enabled: thinking is separated into `reasoning_content`, and
+`content` carries the clean answer — so agent clients (e.g. opencode) that read `content` + `tool_calls`
+work directly.
+
+## Notes / gotchas
+
+- **The critical flags** (baked into `kimi-k3-atom.yaml`, from the image's own recipe at
+  `/app/ATOM/recipes/atom_vllm/Kimi-K3.md`):
+  - `--additional-config '{"online_quant_config":{"global_quant_config":"ptpc_fp8",...}}'` — routes the
+    MoE through ATOM's AITER path. **Without it every token is garbage**: vLLM otherwise dispatches the
+    MXFP4 MoE to a generic Triton kernel that isn't present in the image.
+  - env `AITER_SITUV2_A4W4=1` (K3 MoE is A4W4, **not** A8W4) and `VLLM_USE_BREAKABLE_CUDAGRAPH=0`.
+  - `--kv-cache-dtype fp8` is validated on this build's aiter branch (it ships the fp8 MLA kernels).
+  - `--mamba-cache-mode align` is required for prefix caching on K3's hybrid KDA+MLA attention.
+- **Single-node rolling update:** with one 8-GPU node the Deployment uses `strategy: Recreate`. If a
+  re-apply appears stuck, the old pod may be lingering in `Succeeded` (the server exits 0 on SIGTERM),
+  blocking Recreate — clear it with `kubectl delete pod <old-pod> --grace-period=0 --force`.
+- **Load time** (~30 min of the boot) is the network-disk weight read; it dominates restart time and is
+  independent of serving speed. `--load-format fastsafetensors` is overridden by ATOM's staging loader.
+- **Scaling:** K3 fits on one 8-GPU node, so add capacity with more `replicas` (independent full copies
+  behind a router), not multi-node tensor/expert parallel.

crusoe-kserve-example/recipes/kimi-k3-mi355x-atom/kimi-k3-atom.yaml

@@ -0,0 +1,164 @@
+# Kimi K3 on AMD MI355X (gfx950) via AMD ATOM (vLLM plugin) — single-node TP=8, OpenAI-compatible.
+#
+# Why ATOM: Kimi K3's MXFP4 latent MoE corrupts under CUDA-graph capture on the day-0 vLLM ROCm
+# stack, and even SGLang has to run eager (graph capture off) to stay correct — which costs decode
+# throughput. AMD's ATOM engine ships an AITER MXFP4 MoE path that is correct WITH FULL_AND_PIECEWISE
+# CUDA-graph capture (AMD validates GSM8K 0.9553 on 8xMI355 TP8), restoring fast + correct decode.
+# The canonical launch below is from the image's own recipe: /app/ATOM/recipes/atom_vllm/Kimi-K3.md
+#
+# BEFORE APPLYING:
+#   1. `rocm/atom-dev:vllm-kimi-k3-20260807` is public on Docker Hub. If your nodes can't reach
+#      Docker Hub (or you hit rate limits), mirror it into your own registry and set `image:` below
+#      (2 places), plus an imagePullSecret. Mirror in-cluster with skopeo:
+#        skopeo copy --authfile <dockerconfig> \
+#          docker://docker.io/rocm/atom-dev:vllm-kimi-k3-20260807 \
+#          docker://<your-registry>/kimi-k3-atom:vllm-k3-20260807
+#   2. Ensure an `hf-secret` with key HF_TOKEN exists in this namespace (HuggingFace access to
+#      moonshotai/Kimi-K3). The repo root's `make setup-amd` creates it.
+#   3. Apply:  kubectl apply -n kserve-test -f recipe-kimi-k3-atom.yaml
+#
+# METRICS: the pod carries prometheus.io/scrape + prometheus.io/port=8000 annotations, so the
+# "vllm-annotated" job of manifests/prometheus-vllm.yaml picks it up in any namespace (no pod
+# restart needed even if annotated later — pod annotations are watched live).
+#
+# First boot downloads the ~1.5 TB MXFP4 weights to the RWX disk (init container, one time). ATOM's
+# staging loader then reads them (~30 min on network storage), online-quantizes non-MXFP4 tensors to
+# ptpc_fp8, JIT-compiles AITER kernels, and captures CUDA graphs — allow ~45-50 min to Ready on first
+# boot. Later restarts skip the download but still re-load + re-capture.
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: crusoe-fs
+provisioner: fs.csi.crusoe.ai
+volumeBindingMode: Immediate
+reclaimPolicy: Delete
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: kimi-k3-weights
+spec:
+  accessModes: [ReadWriteMany]      # RWX so a rolling update can stage the new pod on another node
+  storageClassName: crusoe-fs
+  resources:
+    requests:
+      storage: 2Ti                  # min 1 TiB for fs.csi.crusoe.ai; 2Ti fits the 1.5 TB weights
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kimi-k3-atom
+  labels: { app: kimi-k3-atom }
+spec:
+  replicas: 1
+  strategy: { type: Recreate }      # single 8-GPU node: kill the old pod before starting the new one
+  selector:
+    matchLabels: { app: kimi-k3-atom }
+  template:
+    metadata:
+      labels: { app: kimi-k3-atom }
+      annotations:
+        # Opt-in for the prometheus-vllm "vllm-annotated" scrape job (manifests/prometheus-vllm.yaml) —
+        # picked up cluster-wide, in any namespace. Both annotations are required.
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8000"
+    spec:
+      nodeSelector:
+        crusoe.ai/accelerator: amd-mi355x-288gb-roce
+      securityContext:
+        fsGroup: 1000
+      # imagePullSecrets:            # uncomment + name your secret if the registry is private
+      #   - name: <your-pull-secret>
+      initContainers:
+        # One-time weight download to the RWX disk (skipped if already present). Not gated by the
+        # serving probes, so it can take as long as the 1.5 TB pull needs.
+        - name: fetch-weights
+          image: rocm/atom-dev:vllm-kimi-k3-20260807      # same image (has huggingface_hub)
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              if [ -f /models/config.json ]; then echo "weights present, skipping download"; exit 0; fi
+              python3 -c "from huggingface_hub import snapshot_download; snapshot_download('moonshotai/Kimi-K3', local_dir='/models', local_dir_use_symlinks=False)"
+          env:
+            - name: HF_TOKEN
+              valueFrom: { secretKeyRef: { name: hf-secret, key: HF_TOKEN } }
+          volumeMounts:
+            - { name: models, mountPath: /models }
+      containers:
+        - name: main
+          image: rocm/atom-dev:vllm-kimi-k3-20260807
+          workingDir: /app           # /app/vllm shadows site-packages -> uses ATOM's cache_aware patch
+          command: ["/bin/sh", "-c"]
+          # Canonical vLLM-ATOM K3 launch (image's /app/ATOM/recipes/atom_vllm/Kimi-K3.md).
+          # --additional-config online_quant_config routes the MoE through ATOM's ptpc_fp8/AITER path
+          # (the generic vLLM Triton MXFP4 kernel isn't in this image and yields garbage without it).
+          args:
+            - |
+              set -e
+              cd /app
+              exec vllm serve /models \
+                --host 0.0.0.0 --port 8000 \
+                --served-model-name kimi-k3 \
+                --tensor-parallel-size 8 \
+                --trust-remote-code \
+                --max-model-len 131072 \
+                --enable-prefix-caching \
+                --mamba-cache-mode align \
+                --kv-cache-dtype fp8 \
+                --max-num-seqs 64 \
+                --max-num-batched-tokens 16384 \
+                --gpu-memory-utilization 0.93 \
+                --block-size 128 \
+                --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
+                --enable-auto-tool-choice \
+                --tool-call-parser kimi_k3 \
+                --reasoning-parser kimi_k3 \
+                --additional-config '{"online_quant_config": {"global_quant_config": "ptpc_fp8", "exclude_layer": ["lm_head", "model.embed_tokens", "*self_attn.[qkv]_conv1d*", "*block_sparse_moe.experts*", "*block_sparse_moe.routed_expert_*", "*vision_tower*", "*mm_projector*"]}}'
+          env:
+            - name: HF_TOKEN
+              valueFrom: { secretKeyRef: { name: hf-secret, key: HF_TOKEN } }
+            - { name: HF_HUB_OFFLINE,             value: "1" }   # weights already on /models
+            - { name: VLLM_ROCM_USE_AITER,        value: "1" }
+            - { name: AITER_SITUV2_A4W4,          value: "1" }   # K3 MoE is A4W4 (recipe); NOT A8W4
+            - { name: AITER_SITUV2_A8W4,          value: "0" }
+            - { name: VLLM_USE_BREAKABLE_CUDAGRAPH, value: "0" } # build auto-enables =1 which crashes at capture
+          ports:
+            - { containerPort: 8000, name: http }
+          resources:
+            limits:   { amd.com/gpu: "8", cpu: "64", memory: 1Ti }
+            requests: { amd.com/gpu: "8", cpu: "16", memory: 128Gi }
+          securityContext:
+            runAsNonRoot: false
+            runAsUser: 0
+          startupProbe:
+            httpGet: { path: /health, port: 8000 }
+            periodSeconds: 15
+            failureThreshold: 240        # ~60 min: network-disk weight read + online quant + compile + capture
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet: { path: /health, port: 8000 }
+            periodSeconds: 30
+            timeoutSeconds: 8
+            failureThreshold: 3
+          volumeMounts:
+            - { name: models, mountPath: /models }
+            - { name: dshm,   mountPath: /dev/shm }
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: kimi-k3-weights
+        - name: dshm
+          emptyDir: { medium: Memory, sizeLimit: 128Gi }   # TP8 intra-node comms need large /dev/shm
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: kimi-k3-atom-workload-svc
+  labels: { app: kimi-k3-atom }
+spec:
+  selector: { app: kimi-k3-atom }
+  ports:
+    - name: http
+      port: 8000
+      targetPort: 8000