Add DeepSeek-V4-Pro, Qwen3.5, MiniMax-M3 (ATOM + single-node XGMI), and GLM-5.2 inference recipes (scripts/)

README.md

@@ -24,6 +24,11 @@ Below are blueprints of supported models along with their documentation.
 | [**vLLM disaggregated P/D inference**](scripts/vllm_dissag/README.MD) | Distributed Inference P/D disaggregation with vLLM (Default, MoRI EP, DeepEP) | DeepSeek-R1, DeepSeek-V3, DeepSeek-V3-5layer, amd-Llama-3.3-70B-Instruct-FP8-KV, Llama-3.1-405B-Instruct-FP8-KV, gpt-oss-120b |
 | [**SGLang disaggregated P/D inference**](scripts/sglang_disagg/README.MD) | Distributed Inference P/D disaggregation with SGLang (MoRI IO, Mooncake) | Llama-3.1-8B, Qwen3-32B, Llama-3.3-70B-FP8, Llama-3.1-405B-FP8, Mixtral-8x7B, DeepSeek-V3, DeepSeek-R1 |
 | [**SGLang disaggregated P/D inference with WideEP/LargeEP**](scripts/sglang_disagg/README.MD) | Distributed Inference P/D disaggregation with SGLang with WideEP/LargeEP | DeepSeek-V3, DeepSeek-R1 |
+| [**DeepSeek-V4-Pro disaggregated P/D inference**](scripts/DeepSeekV4/README.md) | Distributed Inference P/D disaggregation for DeepSeek-V4-Pro (ATOM+mooncake / SGLang+MoRI) | DeepSeek-V4-Pro |
+| [**Qwen3.5-35B-A3B aggregated inference**](scripts/Qwen3.5/README.md) | Single-node aggregated serving/benchmark (SGLang/vLLM); recommended default SGLang+MXFP4+TP1 (8 instances/node) | Qwen3.5-35B-A3B (MXFP4, FP8) |
+| [**MiniMax-M3 disaggregated P/D inference**](scripts/MiniMax-M3/README.md) | Distributed Inference P/D disaggregation for MiniMax-M3 on MI355X (ATOM+mooncake, MXFP4); top-throughput InferenceX config | MiniMax-M3 (MXFP4) |
+| [**MiniMax-M3 single-node P/D over XGMI**](scripts/MiniMax-M3-single-pd/README.md) | Single-node prefill/decode disaggregation for MiniMax-M3 on MI355X via vLLM + MoRIIO with GPU-to-GPU XGMI KV transfer (TP4/TP4) | MiniMax-M3 (MXFP4) |
+| [**GLM-5.2 disaggregated P/D inference**](scripts/GLM-5.2-disagg/README.md) | Two-node prefill/decode disaggregation for GLM-5.2 on MI355X (ATOM + atomesh + mooncake TCP KV transfer, TP8/TP8) | GLM-5.2 (MXFP4) |
 | [**KVCache Transfer Bench**](scripts/kvcache_transfer_bench/README.md) | Inter-node Transfer Benchmark | no specific models |
 
 ## Table of Contents

scripts/DeepSeekV4/README.md

@@ -0,0 +1,59 @@
+# DeepSeek-V4-Pro — disaggregated (P/D) serving/benchmark harness
+
+Multi-node **prefill/decode disaggregated** serving for DeepSeek-V4-Pro on AMD Instinct
+(gfx950), ported from InferenceX.
+
+## Why disagg
+DSv4-Pro is a large MoE (61 layers, 384 experts, hidden 7168, moe_intermediate 3072; FP8 block-quant
++ FP4 experts). It's served **disaggregated**: prefill workers (kv_producer) and decode workers
+(kv_consumer) on separate nodes, KV streamed over RDMA, fronted by ONE router. Bench drives the
+router → a TRUE aggregate number (no ×N extrapolation). TP8 is valid here (3072/8=384 ≥ 128).
+
+## Engines & proxy/router (see docs/proxy_and_disagg.md for detail)
+| Engine | Router | KV transfer | Status |
+|--------|--------|-------------|--------|
+| **atom-disagg** | **atomesh** | **mooncake** (RDMA) | PRIMARY — InferenceX-validated DSv4 config |
+| **sglang-disagg** | sglang_router | MoRI-IO | EXPERIMENTAL — adapted from DeepSeek-R1 recipe |
+
+(vLLM disagg uses MoRI-IO/Nixl; not scaffolded — its `server_vllm.sh` wasn't in our snapshot.)
+
+## Files
+| File | Role |
+|------|------|
+| `cluster.yaml` | nodes, gpus/node, reservation, router/prefill/decode/handshake ports, paths |
+| `model.yaml` | DSv4 def + shared `defaults{}` + per-engine image/router/kv + `topologies{}` |
+| `run_atom_disagg.sh` / `run_sglang_disagg.sh` | thin wrappers → `lib/run_disagg.sh` |
+| `lib/run_disagg.sh` | orchestrator: clean nodes → launch prefill/decode servers → start router → bench → teardown |
+| `lib/topo.py` | resolve a topology (1p1d / 2p1d_dpa) → node-role placement (prefill/decode, ports) |
+| `lib/cfg.py` | yaml reader (shared with qwen_v2) |
+| `lib/clean_node.sh` | pre-flight GPU clean (shared with qwen_v2) |
+| `utils/bench_serving/` | vendored InferenceX bench client |
+| `docs/proxy_and_disagg.md` | full per-engine proxy/router/KV mechanics + topologies |
+
+## Topologies (from dsv4-fp4-mi355x-atom-disagg)
+| name | nodes | layout | shapes | conc |
+|------|-------|--------|--------|------|
+| `1p1d` | 2 | 1 prefill TP8 + 1 decode TP8 | 1024/1024, 8192/1024 | 4–128 |
+| `2p1d_dpa` | 3 | 2 prefill + 1 decode, TP8 + DP-attn | 8192/1024 | 256–2048 |
+
+## Usage
+```bash
+# ATOM disagg, 1-prefill/1-decode (2 nodes), full bench sweep through the router:
+MODEL=dsv4-pro TOPO=1p1d   ./run_atom_disagg.sh
+MODEL=dsv4-pro TOPO=2p1d_dpa ./run_atom_disagg.sh      # 3 nodes, high-conc
+
+# Just serve (hold the endpoint up; drive it yourself):
+MODEL=dsv4-pro TOPO=1p1d ACTION=serve ./run_atom_disagg.sh
+
+# SGLang disagg (experimental, R1-template):
+MODEL=dsv4-pro TOPO=1p1d ./run_sglang_disagg.sh
+```
+Bench drives `http://<router_ip>:8000` (OpenAI-compatible). Results land in `results_dsv4/`.
+
+## Status / TODO
+- ✅ Weights: DSv4-Pro 64/64 shards integrity-verified on NFS; staging to local scratch (12 nodes).
+- ⏳ ATOM disagg: scaffolded faithfully from server_atom.sh; **needs a live dry-run** (RDMA/mooncake
+  handshake, atomesh router) once weights are staged.
+- ⏳ SGLang disagg: scaffolded from R1 template; DSv4 model support + sglang_router flags TBC.
+- ATOM DSv4 env applied by launcher: `ATOM_MOE_GU_ITLV=1`, `AITER_BF16_FP8_MOE_BOUND=0` (+ TBO path
+  `GPU_MAX_HW_QUEUES=5`, `ATOM_CPU_AFFINITY=1`).

scripts/DeepSeekV4/atom_disagg/accuracy_ctx.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+# accuracy_ctx.py — context-varied accuracy probe for the DSv4 ATOM disagg pipeline.
+#
+# Why: a PD-disaggregated pipeline can be correct at short context but corrupt the KV that
+# mooncake transfers prefill->decode at long context. A 5-short-prompt gate misses that.
+# This embeds a known "needle" fact inside filler of increasing size and checks the router
+# (which spans prefill+decode+mooncake) returns it verbatim — exercising the FULL pipeline
+# at each context length.
+#
+# Usage: ROUTER_URL, MODEL, OUT(json), [CTX_SIZES="512 2048 8192 16384"] via env.
+#        Exits 0 if all context sizes pass, 1 otherwise.
+import json, os, sys, time, urllib.request
+
+ROUTER = os.environ.get("ROUTER_URL", "http://0.0.0.0:8000")
+MODEL  = os.environ["MODEL"]
+OUT    = os.environ.get("OUT", "/out/accuracy_ctx.json")
+# approx tokens of filler context to test at (the needle adds a few more)
+CTX_SIZES = [int(x) for x in os.environ.get("CTX_SIZES", "512 2048 8192 16384").split()]
+
+# A deterministic needle: a unique code the model must echo back. Greedy (temp 0).
+SECRET = "PLATYPUS-7731-AZURE"
+FILLER_SENTENCE = ("The quarterly logistics report notes that warehouse throughput "
+                   "remained within nominal parameters across all regional hubs. ")
+
+def make_prompt(approx_tokens):
+    # ~1.3 tokens/word for English; build filler then plant the needle in the MIDDLE
+    words_needed = int(approx_tokens / 1.3)
+    filler_words = FILLER_SENTENCE.split()
+    reps = max(1, words_needed // len(filler_words))
+    block = (" ".join(filler_words) + " ") * reps
+    half = len(block) // 2
+    needle = f" IMPORTANT: the access code is {SECRET}. Remember it exactly. "
+    ctx = block[:half] + needle + block[half:]
+    q = ("\n\nQuestion: What is the exact access code mentioned above? "
+         "Reply with ONLY the code, nothing else.")
+    return ctx + q
+
+def ask(prompt):
+    body = json.dumps({
+        "model": MODEL,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": 32, "temperature": 0.0,
+    }).encode()
+    req = urllib.request.Request(f"{ROUTER}/v1/chat/completions",
+                                 data=body, headers={"Content-Type": "application/json"})
+    t0 = time.time()
+    with urllib.request.urlopen(req, timeout=300) as r:
+        d = json.load(r)
+    return d["choices"][0]["message"]["content"], time.time() - t0
+
+results = []; allpass = True
+for ctx in CTX_SIZES:
+    try:
+        prompt = make_prompt(ctx)
+        text, dt = ask(prompt)
+        found = SECRET in text.upper()
+        ok = found
+        results.append({"ctx_tokens": ctx, "ok": ok, "found_needle": found,
+                        "reply": text.strip()[:80], "latency_s": round(dt, 2),
+                        "prompt_chars": len(prompt)})
+        print(f"  ctx~{ctx:>6}tok: {'PASS' if ok else 'FAIL'} "
+              f"(needle={'yes' if found else 'NO'}, {dt:.1f}s) reply='{text.strip()[:40]}'")
+        allpass = allpass and ok
+    except Exception as e:
+        results.append({"ctx_tokens": ctx, "ok": False, "error": str(e)[:120]})
+        print(f"  ctx~{ctx:>6}tok: ERROR {str(e)[:80]}")
+        allpass = False
+
+os.makedirs(os.path.dirname(OUT), exist_ok=True)
+json.dump({"model": MODEL, "router": ROUTER, "all_pass": allpass, "results": results},
+          open(OUT, "w"), indent=2)
+print(f"[accuracy_ctx] {'ALL PASS' if allpass else 'FAILURES'} -> {OUT}")
+sys.exit(0 if allpass else 1)

scripts/DeepSeekV4/atom_disagg/bench.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+# bench.sh — driven by server_atom.sh on node 0 (router node), AFTER atomesh is /v1/models-ready.
+# Reproduces the InferenceX call signature exactly:
+#   bash bench.sh <xP> <yD> <PF_total_gpus> <DEC_total_gpus> <model_dir> <model_name> \
+#                 <log_dir> <isl> <osl> "<conc_list>" <req_rate> <range_ratio> <prompts_mult>
+#
+# Drives the OpenAI-compatible ATOM router at localhost:$ROUTER_PORT with the vendored
+# InferenceX benchmark_serving client (same one used for the Qwen sweep) — one result JSON
+# per concurrency. This is a TRUE aggregate (one endpoint, real PD + mooncake KV), no x N.
+set -uo pipefail
+xP="$1"; yD="$2"; PF_GPUS="$3"; DEC_GPUS="$4"
+MODEL_DIR="$5"; MODEL_NAME="$6"; LOG_DIR="$7"
+ISL="$8"; OSL="$9"; CONC_LIST="${10}"; REQ_RATE="${11}"; RANGE_RATIO="${12}"; PROMPTS_MULT="${13}"
+# server_atom.sh passes the conc list 'x'-separated (e.g. 4x8x16). Normalise to spaces.
+CONC_LIST=$(echo "$CONC_LIST" | tr 'x' ' ')
+
+ROUTER_PORT="${ROUTER_PORT:-8000}"
+BENCH_CLIENT="${BENCH_CLIENT:-/scripts/utils/bench_serving/benchmark_serving.py}"
+OUT="${LOG_DIR}"; mkdir -p "$OUT" 2>/dev/null || true
+
+echo "[bench] router=localhost:${ROUTER_PORT} model=${MODEL_NAME} isl=${ISL} osl=${OSL} conc='${CONC_LIST}' (xP=${xP} yD=${yD})"
+
+# --- ACCURACY GATE (context-varied) before perf -------------------------------------------
+# Validate the FULL PD pipeline (prefill+mooncake KV+decode) returns correct output at varied
+# context sizes. Runs ONCE per launch (guarded by a marker), not per-shape. Non-fatal: logs
+# result but does not abort the perf sweep (so we still get numbers even if a long-ctx probe fails).
+ACC_OUT="${OUT}/accuracy_ctx.json"
+if [[ ! -f "$ACC_OUT" ]]; then
+  echo "[accuracy] context-varied probe via router (sizes: ${CTX_SIZES:-512 2048 8192 16384})"
+  ROUTER_URL="http://localhost:${ROUTER_PORT}" MODEL="${MODEL_DIR}/${MODEL_NAME}" \
+    OUT="$ACC_OUT" CTX_SIZES="${CTX_SIZES:-512 2048 8192 16384}" \
+    python3 /ws/accuracy_ctx.py || echo "[accuracy] WARN: some context sizes failed (see $ACC_OUT)"
+fi
+
+for c in $CONC_LIST; do
+  echo "[bench] --- concurrency $c ---"
+  python3 "$BENCH_CLIENT" \
+    --model "${MODEL_DIR}/${MODEL_NAME}" --backend vllm \
+    --base-url "http://localhost:${ROUTER_PORT}" --dataset-name random \
+    --random-input-len "$ISL" --random-output-len "$OSL" --random-range-ratio "$RANGE_RATIO" \
+    --num-prompts "$(( c * PROMPTS_MULT ))" --num-warmups "$(( c * 2 ))" \
+    --max-concurrency "$c" --request-rate "$REQ_RATE" --ignore-eos --save-result \
+    --percentile-metrics ttft,tpot,itl,e2el --result-dir "$OUT" \
+    --result-filename "disagg_${xP}p${yD}d_isl${ISL}_osl${OSL}_c${c}.json" \
+    || echo "[bench] FAILED isl=$ISL osl=$OSL conc=$c"
+done
+echo "[bench] done -> $OUT"

scripts/DeepSeekV4/atom_disagg/env_atom.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+# ATOM/mooncake-specific environment setup for multi-node disaggregated serving.
+#
+# Sourced by server_atom.sh in place of env.sh (which is SGLang/MoRI-specific).
+#
+# REQUIRED ENVIRONMENT VARIABLES:
+#   IBDEVICES - RDMA/InfiniBand device names (e.g., ionic_0,ionic_1,...)
+#               Set by runner or auto-detected from hostname.
+
+set -x
+
+export PYTHONUNBUFFERED=1
+export PYTHONDONTWRITEBYTECODE=1
+
+# =============================================================================
+# IBDEVICES detection (same as env.sh)
+# =============================================================================
+
+if [[ -z "$IBDEVICES" ]]; then
+    DETECTED=$(ibv_devinfo 2>/dev/null | grep "hca_id:" | awk '{print $2}' | paste -sd',')
+    if [[ -n "$DETECTED" ]]; then
+        export IBDEVICES="$DETECTED"
+        echo "[INFO] Auto-detected IBDEVICES=$IBDEVICES via ibv_devinfo on $(hostname -s)"
+    else
+        # ATOM uses mooncake proxy_ip/handshake_port for KV transfer — IBDEVICES is
+        # not passed as a server argument (unlike SGLang --disaggregation-ib-device).
+        # Log a warning but do not fail; mooncake will use its own RDMA device selection.
+        echo "[WARN] Unable to detect RDMA devices via ibv_devinfo; IBDEVICES unset (non-fatal for ATOM/mooncake)" >&2
+    fi
+else
+    echo "[INFO] Using IBDEVICES=$IBDEVICES (set by runner or environment)"
+fi
+export IBDEVICES
+
+export SAFETENSORS_FAST_GPU=1
+export VLLM_LOG_LEVEL=WARNING
+export ATOM_LOG_LEVEL=WARNING
+export AITER_LOG_LEVEL=WARNING
+export LOG_LEVEL=WARNING
+export LOGLEVEL=WARNING
+
+# =============================================================================
+# ATOM/mooncake-specific environment
+# =============================================================================
+
+# mooncake RDMA KV transfer library path
+# FIX (verified 2026-06-28): image rocm/atom-dev:nightly_202606101403 ships Python 3.12, NOT 3.10.
+# Original InferenceX path (.../python3.10/...) does not exist here -> mooncake native libs fail to
+# load. Resolve the actual site-packages dir dynamically so this is image-version agnostic.
+_MOONCAKE_DIR=$(python3 -c "import os,mooncake;print(os.path.dirname(mooncake.__file__))" 2>/dev/null)
+export LD_LIBRARY_PATH=${_MOONCAKE_DIR:-/opt/venv/lib/python3.12/site-packages/mooncake}:/opt/rocm/lib:${LD_LIBRARY_PATH:-}
+
+
+# ATOM_HOST_IP is set per-node in server_atom.sh (= host_ip, used as handshake IP)
+
+# aiter logging (WARNING to reduce noise; use DEBUG for troubleshooting)
+export AITER_LOG_LEVEL=WARNING
+
+if [[ "$MODEL_NAME" == "DeepSeek-V4-Pro" ]]; then
+    # ATOM MoE gather/scatter interleave optimization
+    export ATOM_MOE_GU_ITLV=1
+    # Disable bf16->fp8 MoE bound (only for DeepSeek-V4-Pro)
+    export AITER_BF16_FP8_MOE_BOUND=0
+fi
+
+# Clear stale ATOM cache on startup (server_atom.sh handles this via rm -rf)
+# No env var needed; documented here for reference.
+
+set +x
+
+echo "[INFO] ATOM env: IBDEVICES=$IBDEVICES  LD_LIBRARY_PATH includes mooncake"