Skip to content

Add DeepSeek-V4-Pro, Qwen3.5, MiniMax-M3 (ATOM + single-node XGMI), and GLM-5.2 inference recipes (scripts/) - #173

Open
raviguptaamd wants to merge 12 commits into
ROCm:developfrom
raviguptaamd:add-deepseekv4-recipe
Open

Add DeepSeek-V4-Pro, Qwen3.5, MiniMax-M3 (ATOM + single-node XGMI), and GLM-5.2 inference recipes (scripts/)#173
raviguptaamd wants to merge 12 commits into
ROCm:developfrom
raviguptaamd:add-deepseekv4-recipe

Conversation

@raviguptaamd

@raviguptaamd raviguptaamd commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds five inference blueprints/recipes, linked from the top-level README Blueprints table:

  1. DeepSeek-V4-Pro disaggregated prefill/decode (P/D) inference on AMD Instinct MI355X, under scripts/DeepSeekV4/. Covers two transport backends:

    • ATOM + Mooncake (KV-cache transfer over RDMA)
    • SGLang + MoRI-IO

  2. Qwen3.5-35B-A3B single-node aggregated serving/benchmark (SGLang / vLLM), under scripts/Qwen3.5/. Recommended default is SGLang + MXFP4 + TP1 (8 single-GPU instances/node) — a 3B-active MoE fits one GPU, so filling the node with independent instances beats tensor-parallel. README includes a 4-step quickstart for the default.

  3. MiniMax-M3 disaggregated prefill/decode (P/D) inference on AMD Instinct MI355X, under scripts/MiniMax-M3/. ATOM MXFP4 config (minimaxm3-fp4-mi355x-atom-disagg, atomesh router + Mooncake RDMA) — the top-throughput MiniMax-M3 disagg config on InferenceX (~10,829 tok/s/GPU at 8k/1k, 2P1D+DPA). Engine code is copied verbatim from SemiAnalysisAI/InferenceX with a thin MAD wrapper (run.sh + cluster.yaml) so it runs standalone.

  4. MiniMax-M3 single-node P/D over XGMI on AMD Instinct MI355X, under scripts/MiniMax-M3-single-pd/. Simulates prefill/decode disaggregation on one node via vLLM + MoRIIO with GPU-to-GPU XGMI KV transfer (backend=xgmi): prefill TP4 (GPU0-3) + decode TP4 (GPU4-7), fronted by vllm-router. Complements the cross-node ATOM recipe above; co-location works here because vLLM's --disable-custom-all-reduce avoids the AITER custom-AR IPC collision and MoRIIO has a native XGMI backend. Validated end-to-end (router-mediated P/D, correct inference, 1k/1k + 8k/1k throughput).

  5. GLM-5.2 disaggregated prefill/decode (P/D) inference on AMD Instinct MI355X, under scripts/GLM-5.2-disagg/. Two-node ATOM + atomesh with KV-cache transfer over Mooncake TCP, TP8/TP8 (prefill + decode). GLM-5.2 (GlmMoeDsa) is served by the ATOM native engine; KV moves over Mooncake's TCP transport (nodes without amdgpu_peermem, so no GPU-direct RDMA). Two non-obvious settings make it work: the MXFP4 online_quant_config excludes *expert* (FP8 expert re-quant deadlocks the decode), and MC_FORCE_TCP=1 (else Mooncake attempts GPU-direct RDMA writes and every transfer returns chunk error -1). Validated end-to-end through the router (correct inference).

What's included

DeepSeekV4 (scripts/DeepSeekV4/)

  • README.md — usage and overview
  • cluster.yaml / model.yaml — cluster + model configuration
  • lib/ — launcher and helpers (run_disagg.sh, cfg.py, topo.py, check_accuracy.py, clean_node.sh, prompts.json, lib_inferencex.sh)
  • atom_disagg/ — ATOM+Mooncake env/server/launch/bench scripts
  • utils/bench_serving/ — serving benchmark client (throughput/latency)
  • docs/proxy_and_disagg.md — proxy + disaggregation notes
  • run_atom_disagg.sh / run_sglang_disagg.sh — entrypoints

Qwen3.5 (scripts/Qwen3.5/)

  • README.md — quickstart + recommended default (SGLang·MXFP4·TP1)
  • cluster.yaml / model.yaml — cluster + model config (MXFP4 + FP8), engine flags preset
  • lib/ — launcher and helpers (run_engine.sh, replica_entry.sh, placement.py, cfg.py, clean_node.sh, check_accuracy.py, prompts.json, lib_inferencex.sh)
  • run_sglang.sh / run_vllm.sh / run_atom.sh — entrypoints
  • utils/bench_serving/ — serving benchmark client

MiniMax-M3 (scripts/MiniMax-M3/)

  • README.md — nodes required, prerequisites (incl. amdgpu_peermem for GPUDirect RDMA), usage, topologies
  • run.sh / cluster.yaml — MAD launcher wrapper (supplies the per-node env + mounts InferenceX's job.slurm normally provides)
  • benchmarks/multi_node/amd_utils/ + benchmarks/benchmark_lib.sh — verbatim InferenceX ATOM code (server_atom.sh, models_atom.yaml, env_atom.sh, bench.sh, setup_deps.sh, server.sh, job.slurm, submit.sh, sync.py); models_atom.yaml drives all model flags
  • utils/bench_serving/ — serving benchmark client
  • configs/minimaxm3-fp4-mi355x-atom-disagg.yaml — config entry (all-TP4, STP; topologies 1p1d / 2p1d_dpa)

Topologies: 1p1d (2 nodes: 1 prefill + 1 decode) and 2p1d_dpa (3 nodes: 2 prefill + 1 decode + DP-attn).

MiniMax-M3 single-node P/D over XGMI (scripts/MiniMax-M3-single-pd/)

  • README.md — the 7 non-obvious fixes, exact config, verification steps, results, provenance
  • run_pd_singlenode.sh — launcher: vllm-router + prefill (GPU0-3) + decode (GPU4-7), MoRIIO backend=xgmi, waits for router registration, smoke inference
  • prep_model.sh — stage amd/MiniMax-M3-MXFP4 as a flat dir (weight symlinks + remote-code .py)
  • bench_pd.sh — vllm bench serve throughput sweep (shapes/concurrency via env)

Key requirements for single-node co-location: --disable-custom-all-reduce, per-container HIP_VISIBLE_DEVICES+CUDA_VISIBLE_DEVICES (0-3 / 4-7), distinct MoRIIO handshake/notify ports (6301/61005 vs 6311/61015), --max-model-len 32768. Related upstream fixes: ROCm/mori #481, vllm-project/router #181.

GLM-5.2 disaggregated P/D (scripts/GLM-5.2-disagg/)

  • README.md — stack rationale, the two non-obvious settings, prerequisites, usage
  • serve_atom.sh — per-node launcher (ROLE=prefill|decode|router): ATOM engine + atomesh router, Mooncake TCP KV transfer
  • bench_pd.sh — concurrency-sweep driver against the router (OpenAI-compatible)

Uses the public upstream image rocm/atom-dev (README notes the exact tested digest).

@raviguptaamd
Add DeepSeek-V4-Pro disaggregated P/D inference recipe (scripts/DeepS…
83e63d0 
…eekV4)

Multi-node prefill/decode disaggregated serving/benchmark harness for
DeepSeek-V4-Pro (ATOM+mooncake primary; SGLang+MoRI experimental), ported
from InferenceX. Adds a Blueprints row to the top-level README.
@raviguptaamd @claude
Add Qwen3.5-35B-A3B aggregated inference recipe (scripts/Qwen3.5)
41070cb 
Single-node aggregated serving/benchmark harness for Qwen3.5-35B-A3B (3B-active
hybrid-GDN MoE) on AMD Instinct MI350 (gfx950), engines SGLang + vLLM.

Recommended default: SGLang + MXFP4 + TP1 (8 single-GPU instances/node) — a 3B-active
MoE fits one GPU, so fill the node with independent instances rather than tensor-parallel.
Findings: TP1 > TP2 > TP4; SGLang > vLLM; MXFP4 > FP8 (and ~half the HBM). Both engines
serve MXFP4 correctly (no garbling). ATOM excluded — crashes at ISL>=8192.

Code/scripts + README only (no result data). Engine flags baked into model.yaml per
AMD Day-0 recipe (SGLang --disable-radix-cache required for the hybrid-GDN arch;
vLLM --enable-expert-parallel --reasoning-parser qwen3). cluster.yaml uses placeholders.

Co-Authored-By: Claude <noreply@anthropic.com>
raviguptaamd and others added 6 commits July 8, 2026 19:10
@raviguptaamd @claude
Qwen3.5: add lib/ files (force-add past .gitignore lib/ rule)
c44d29b 
The repo .gitignore has `lib/` (Python build-dir convention), which silently
excluded scripts/Qwen3.5/lib/* from the prior commit. Force-add the 8 harness
lib files — run_engine.sh (core launcher), replica_entry.sh, cfg.py, clean_node.sh,
placement.py, check_accuracy.py, lib_inferencex.sh, prompts.json. Without these the
recipe cannot run.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
Qwen3.5 README: add highlighted 4-step Quickstart for the default config
61beced 
Makes the recommended SGLang+MXFP4+TP1 default runnable cold: download weights ->
edit cluster.yaml -> one run_sglang.sh command -> query. Spells out the prerequisites
(weights path, cluster.yaml placeholders, Slurm alloc) that were previously implicit.

Co-Authored-By: Claude <noreply@anthropic.com>
Ravi Gupta @claude
Add MiniMax-M3 MXFP4 disaggregated P/D inference recipe (scripts/Mini…
f2239a4 
…Max-M3)

Multi-node prefill/decode disaggregated serving/benchmark harness for
MiniMax-M3 (MXFP4) on AMD MI355X (gfx950) via ATOM (atomesh + mooncake RDMA),
ported from the InferenceX minimaxm3-fp4-mi355x-atom-disagg config (top-throughput
MiniMax-M3 disagg config on MI355X: ~10,829 tok/s/GPU at 8k/1k, 2P1D+DPA).

All workers TP4 (no EP; TP4 beats TP8/TP4-EP on gfx950); STP only (DECODE_MTP_SIZE=0).
Topologies 1p1d (2 nodes) and 2p1d_dpa (3 nodes). Mirrors the DeepSeekV4 harness
structure (model.yaml/cluster.yaml/lib/atom_disagg/utils). Scaffolded from InferenceX;
needs a live gfx950 dry-run once amd/MiniMax-M3-MXFP4 weights are staged.

Also adds the MiniMax-M3 row to the root README models table.

Co-Authored-By: Claude <noreply@anthropic.com>
Ravi Gupta @claude
MiniMax-M3: vendor InferenceX ATOM MXFP4 disagg code verbatim
32856d6 
Replace the hand-adapted MiniMax-M3 harness with the actual InferenceX ATOM
MXFP4 disaggregated benchmark, copied unmodified from SemiAnalysisAI/InferenceX
@ a174f20 (benchmarks/multi_node/amd_utils + utils/bench_serving). Config key
minimaxm3-fp4-mi355x-atom-disagg (atomesh + mooncake, all-TP4). Includes
models_atom.yaml (authoritative MiniMax-M3-MXFP4 flags), server_atom.sh,
env_atom.sh, bench.sh, setup_deps.sh, job.slurm/submit.sh/sync.py, the
bench_serving client, and the extracted config block under configs/.

Co-Authored-By: Claude <noreply@anthropic.com>
Ravi Gupta @claude
MiniMax-M3: make ATOM MXFP4 disagg recipe runnable standalone in MAD
5b40488 
Keep the InferenceX engine code verbatim (benchmarks/multi_node/amd_utils/,
benchmarks/benchmark_lib.sh, utils/bench_serving/ @ a174f20 — byte-identical,
models_atom.yaml drives all model flags) but preserve InferenceX's relative
path layout so bench.sh's ../../benchmark_lib.sh and REPO_ROOT/utils/bench_serving
resolve. Add a thin MAD wrapper (run.sh + cluster.yaml) that supplies the per-node
env contract + container mounts InferenceX's job.slurm/submit.sh normally provide,
mounting the recipe root as /workspace and ATOM_WS_PATH at amd_utils/.

README documents prerequisites (amdgpu_peermem for GPUDirect RDMA, weight staging,
RDMA fabric), usage (TOPO=1p1d|2p1d_dpa, ACTION=dry), and topologies. Wrapper is
syntax-checked; not yet dry-run on live MI355X.

Co-Authored-By: Claude <noreply@anthropic.com>
Ravi Gupta
MiniMax-M3: trim README to plain runnable doc
da4c6c6
@raviguptaamd raviguptaamd changed the title Add DeepSeek-V4-Pro disaggregated P/D inference recipe (scripts/DeepSeekV4) Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 inference recipes (scripts/) Jul 16, 2026
Ravi Gupta and others added 2 commits July 23, 2026 07:28
Ravi Gupta @claude
Add MiniMax-M3 single-node P/D over XGMI recipe (scripts/MiniMax-M3-s…
478244f 
…ingle-pd)

Single-node prefill/decode disaggregation for MiniMax-M3-MXFP4 on AMD MI355X via
vLLM + MoRIIO, with GPU-to-GPU XGMI KV transfer (backend=xgmi): prefill TP4 on
GPU0-3 + decode TP4 on GPU4-7, co-located on one node, fronted by vllm-router.

Complements the cross-node ATOM recipe in scripts/MiniMax-M3/. Co-location works
here because vLLM exposes --disable-custom-all-reduce (avoiding the AITER custom
all-reduce IPC collision that blocks ATOM) and MoRIIO has a native XGMI backend.
README documents the 7 non-obvious fixes, exact config, verification steps,
validation results (1k/1k + 8k/1k throughput, GSM8K-style accuracy), and provenance
(ROCm/mori #481, vllm-project/router ROCm#181).

Files: run_pd_singlenode.sh (launcher), prep_model.sh (flat model + remote-code),
bench_pd.sh (vllm bench serve sweep), README.md. Adds a root README blueprint row.

Co-Authored-By: Claude <noreply@anthropic.com>
Ravi Gupta @claude
MiniMax-M3-single-pd: fix docker volume paths (absolute LOG, mount be…
00a7248 
…nch result-dir)

- LOG defaulted to a relative path but is bind-mounted (docker -v rejects relative
  paths); resolve it to absolute in run_pd_singlenode.sh and bench_pd.sh.
- bench_pd.sh mounted only $MOUNT, so --result-dir writes were lost inside the
  container; also mount $LOG so result JSONs persist.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd raviguptaamd changed the title Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 inference recipes (scripts/) Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 (ATOM + single-node XGMI) inference recipes (scripts/) Jul 23, 2026
Ravi Gupta and others added 2 commits July 23, 2026 07:54
Ravi Gupta @claude
MiniMax-M3-single-pd: retry smoke inference (engine warmup after regi…
48cfaaf 
…stration)

Dry-run showed the single curl right after worker registration can fail silently
(curl -sf returns empty during first-token warmup), leaving infer.json empty even
though the stack serves correctly. Replace with a retry loop (12×15s) that checks
for a valid "choices" response and prints the answer.
Verified on MI355X: manual request returns 391 (17x23).

Co-Authored-By: Claude <noreply@anthropic.com>
Ravi Gupta
Add GLM-5.2 disaggregated P/D recipe (scripts/GLM-5.2-disagg)
85905d5 
Two-node prefill/decode disaggregation for GLM-5.2-MXFP4 on MI355X:
ATOM native engine + atomesh router, KV cache over mooncake TCP, TP8/TP8.

Two non-obvious settings, both applied by serve_atom.sh:
- MXFP4 online_quant_config excludes "*expert*" so MoE experts stay MXFP4
  (FP8 per-block expert re-quant deadlocks the decode engine).
- MC_FORCE_TCP=1 so mooncake moves KV over TCP; without it it attempts
  GPU-direct RDMA writes and every transfer returns "chunk error -1"
  (these nodes have no amdgpu_peermem).

Uses public upstream image rocm/atom-dev; README notes the exact tested digest.
@raviguptaamd raviguptaamd changed the title Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 (ATOM + single-node XGMI) inference recipes (scripts/) Add DeepSeek-V4-Pro, Qwen3.5, MiniMax-M3 (ATOM + single-node XGMI), and GLM-5.2 inference recipes (scripts/) Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gargrahul gargrahul Awaiting requested review from gargrahul gargrahul is a code owner
@amathews-amd amathews-amd Awaiting requested review from amathews-amd amathews-amd is a code owner
@coketaste coketaste Awaiting requested review from coketaste coketaste is a code owner
@ppalaniappan-amd ppalaniappan-amd Awaiting requested review from ppalaniappan-amd ppalaniappan-amd is a code owner
@Rohan138 Rohan138 Awaiting requested review from Rohan138 Rohan138 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raviguptaamd