[ATOM SGL] runtime extraction

.claude/commands/atom-vllm-benchmark-guide.md

@@ -0,0 +1,283 @@
+# ATOM vLLM Benchmark Guide
+
+Use this guide when the user asks to benchmark ATOM vLLM Plugin performance, compare with the upstream vLLM (optional), or validate a performance claim.
+
+## What this guide should do
+
+1. Confirm benchmark goal and variables (throughput, TTFT, TPOT, E2EL, etc.).
+2. Use plugin-on as default setup; optionally benchmark upstream vLLM for performance comparison.
+3. Run repeatable measurements and summarize median results.
+4. Report reproducible commands, key metrics, and known risks.
+5. Prefer model-specific launch settings from `recipes/atom_vllm/` when available; otherwise use the standard workflow in this guide.
+
+## Inputs To Collect First
+
+Before running any benchmark, confirm:
+
+- Model path (HF id or local path)
+- Serving backend (`vllm`)
+- Hardware shape (GPU type/count)
+- `tensor_parallel_size`, KV cache dtype, and scheduler flags
+- Request mix (ISL/OSL/concurrency/request rate)
+- Comparison target:
+  - default candidate (plugin on)
+  - optional baseline/control (plugin off)
+  - optional runtime baseline (upstream vLLM nightly)
+
+If any of these are missing, ask before execution.
+
+## Environment Checklist
+
+For each benchmark round:
+
+- Kill old server processes and verify VRAM is released.
+- Clear cache when startup behavior is inconsistent:
+  - `rm -rf /root/.cache/atom/*`
+  - `rm -rf /root/.cache/vllm/*`
+- Set stable logging:
+  - `export AITER_LOG_LEVEL=WARNING`
+- Confirm server is actually ready:
+  - `curl -sf http://localhost:8000/v1/models`
+  - `rocm-smi --showmemuse` and verify allocated memory is greater than 0.
+
+## Benchmark Workflow (Plugin On by Default)
+
+Run plugin-on by default.
+
+### Recipe-first execution policy
+
+When running ATOM vLLM benchmark:
+
+1. First check whether the target model has a matching recipe under `recipes/atom_vllm/`.
+2. If a matching recipe exists, use its model-specific environment variables and `vllm serve` launch args as the default benchmark setup.
+3. If no matching recipe exists, follow the standard launch template and workflow in this guide.
+
+Keep benchmark methodology unchanged (smoke check, concurrency isolation, and fixed comparison variables) regardless of whether the launch config comes from a recipe or from the default template.
+
+- **Default (candidate):** plugin enabled
+- **Optional control (baseline):** plugin disabled (only when A/B comparison is requested)
+- **Default ATOM vLLM Plugin image:** `rocm/atom-dev:vllm-latest`
+
+If both conditions are run, only one variable is allowed to change between them.
+
+### 1) Launch server
+
+Use the same launch command template. Keep plugin-on as default, and only change plugin toggle env vars when running the optional plugin-off control.
+
+```bash
+vllm serve <model_path> \
+  --host localhost \
+  --port 8000 \
+  --tensor-parallel-size <tp> \
+  --attention-backend ROCM_AITER_FA \
+  --trust-remote-code \
+  --kv-cache-dtype fp8 \
+  --gpu_memory_utilization 0.9 \
+  --async-scheduling \
+  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
+  --no-enable-prefix-caching
+```
+
+Common plugin toggles:
+
+- Default plugin-on: do not set `ATOM_DISABLE_VLLM_PLUGIN` or `ATOM_DISABLE_VLLM_PLUGIN_ATTENTION` (or set them to `0`)
+- Optional plugin-off control: set `ATOM_DISABLE_VLLM_PLUGIN=1`
+- Optional attention-off control: set `ATOM_DISABLE_VLLM_PLUGIN_ATTENTION=1`
+- Always set `VLLM_ROCM_USE_AITER=1` in both ATOM plugin and upstream vLLM runs.
+
+### 2) Run smoke validation
+
+```bash
+curl -sf http://localhost:8000/v1/models
+vllm bench serve \
+  --backend vllm \
+  --base-url http://localhost:8000 \
+  --endpoint /v1/completions \
+  --model <model> \
+  --dataset-name random \
+  --random-input-len 32 \
+  --random-output-len 32 \
+  --temperature 0.0 \
+  --max-concurrency <smoke_conc> \
+  --num-prompts <smoke_conc_x10> \
+  --num-warmups <smoke_conc_x2> \
+  --request-rate 1 \
+  --disable-tqdm
+```
+
+Use `smoke_conc_x10 = smoke_conc * 10` and `smoke_conc_x2 = smoke_conc * 2`.
+
+If smoke test fails, do not continue with performance comparison.
+
+### 3) Run benchmark workload
+
+```bash
+vllm bench serve \
+  --backend vllm \
+  --base-url http://localhost:8000 \
+  --endpoint /v1/completions \
+  --model=<model> \
+  --dataset-name=random \
+  --random-input-len=<isl> \
+  --random-output-len=<osl> \
+  --random-range-ratio=0.8 \
+  --temperature=0.0 \
+  --num-prompts=<conc_x10> \
+  --num-warmups=<conc_x2> \
+  --max-concurrency=<conc> \
+  --trust-remote-code \
+  --request-rate=inf \
+  --ignore-eos \
+  --disable-tqdm \
+  --save-result \
+  --percentile-metrics="ttft,tpot,itl,e2el"
+```
+
+### 4) Concurrency sweep isolation (required)
+
+When sweeping multiple concurrency points, each point must run in a fresh container lifecycle:
+
+1. Start a fresh container.
+2. Start vLLM server in that container.
+3. Run smoke + benchmark for one `--max-concurrency=<conc>`.
+4. Stop vLLM server.
+5. Exit and stop/remove the container.
+6. Start a new container for the next concurrency point.
+
+Do not reuse the same running server/container across different concurrency points.
+
+Example pattern:
+
+```bash
+for conc in 4 8 16 32 64 128; do
+  docker run -dt --name "atom_bench_c${conc}" <image>
+  docker exec "atom_bench_c${conc}" bash -lc 'vllm serve <model_path> ...'
+  docker exec "atom_bench_c${conc}" bash -lc "vllm bench serve ... --max-concurrency=${conc} ..."
+  docker exec "atom_bench_c${conc}" bash -lc 'pkill -f "vllm serve" || true'
+  docker stop "atom_bench_c${conc}" && docker rm "atom_bench_c${conc}"
+done
+```
+
+### 5) Minimum benchmark matrix
+
+| Scenario | ISL | OSL | Concurrency |
+|---|---:|---:|---:|
+| Short in / short out | 1024 | 1024 | 4, 8, 16, 32, 64, 128 |
+| Short in / long out | 1024 | 8192 | 4, 8, 16, 32, 64, 128 |
+| Long in / short out | 8192 | 1024 | 4, 8, 16, 32, 64, 128 |
+
+### 6) Repetition and statistics
+
+- Run each scenario at least 1 times.
+- Use median values for comparison.
+- If variance is large, increase repeat count.
+
+## Optional Upstream vLLM Benchmark
+
+Use this optional path when you need ATOM vs upstream runtime comparison.
+
+Reference images:
+
+- ATOM vLLM plugin default image: `rocm/atom-dev:vllm-lastest`
+- Upstream vLLM comparison image: `vllm/vllm-openai-rocm:nightly`
+
+### 1) Pull latest upstream ROCm image
+
+```bash
+docker pull vllm/vllm-openai-rocm:nightly
+```
+
+### 2) Run the same workflow and config
+
+For upstream runs, keep everything identical to ATOM runs:
+
+- Same model path
+- Same GPU set and `tensor_parallel_size`
+- Same KV cache dtype and scheduler flags
+- Same request mix (ISL/OSL/concurrency/request rate)
+- Same concurrency sweep isolation policy (restart container per concurrency point)
+
+Only runtime image is allowed to change.
+
+### 3) Upstream execution note
+
+- Use the same smoke and benchmark commands from this guide.
+- Do not set ATOM plugin toggle env vars for upstream runs.
+- Always set `VLLM_ROCM_USE_AITER=1` for upstream runs.
+- Keep result naming explicit (for example, include `upstream-nightly` in filenames).
+
+## Regression Profiling Workflow
+
+When regression is observed:
+
+```bash
+# 1) Start (or restart) server with profiler enabled
+vllm serve <model_path> \
+  --host localhost \
+  --port 8000 \
+  --tensor-parallel-size <tp> \
+  --attention-backend ROCM_AITER_FA \
+  --trust-remote-code \
+  --kv-cache-dtype fp8 \
+  --profiler-config '<profiler_config_json>'
+
+# 2) Run profiled benchmark workload
+vllm bench serve \
+  --backend vllm \
+  --base-url http://localhost:8000 \
+  --endpoint /v1/completions \
+  --model <model> \
+  --dataset-name random \
+  --random-input-len <isl> \
+  --random-output-len 20 \
+  --temperature 0.0 \
+  --max-concurrency <conc> \
+  --num-prompts <conc_x2> \
+  --num-warmups <conc_x2> \
+  --request-rate inf \
+  --profile \
+  --save-result
+```
+
+- `--profile` requires server-side `--profiler-config`.
+- Keep profiler config identical between plugin-off and plugin-on runs.
+- For reproducibility, keep benchmark client semantics fixed to completion mode:
+  `--backend vllm --endpoint /v1/completions` (this path uses `max_tokens`).
+
+## Reporting Template
+
+Use this format in final response:
+
+```markdown
+### ATOM vLLM Benchmark Result
+- Goal: <what was measured and why>
+- Setup:
+  - Model: <model>
+  - Hardware: <gpu and count>
+  - Fixed args: <tp, kv dtype, scheduler flags>
+  - Isolation: <restart container per concurrency point: yes/no>
+- Validation: <smoke test pass/fail>
+- Benchmark:
+  - Default (plugin on): <throughput/latency median>
+  - Optional control (plugin off): <throughput/latency median, if run>
+  - Optional runtime baseline (upstream nightly): <throughput/latency median, if run>
+  - Delta: <percent change, if both runs exist>
+- Confidence:
+  - Runs: <repeat count>
+  - Variance: <low/medium/high>
+- Risk:
+  - <known risks or unknowns>
+- Repro:
+  - <exact default launch command (plugin on)>
+  - <exact optional control launch command (plugin off), if run>
+  - <exact benchmark command>
+```
+
+## Guardrails
+
+- Never compare runs with different model, TP, KV dtype, or request mix.
+- For concurrency sweep, stop server and restart container between concurrency points.
+- For ATOM vs upstream comparison, only runtime image may differ.
+- Keep `VLLM_ROCM_USE_AITER=1` fixed for both ATOM and upstream runs.
+- If GPU memory is not allocated, treat benchmark results as invalid.
+- If plugin-on and plugin-off outputs differ semantically, run accuracy checks before claiming performance gains.

.claude/commands/ci-pr-guide.md

@@ -37,8 +37,8 @@ PR/Push → pre-checks (black + ruff)
 | DeepSeek-R1-0528 MTP | `--kv_cache_dtype fp8 -tp 8 --method mtp` | 0.94 | Yes | 8gpu |
 | DeepSeek-R1-0528 MXFP4 | `--kv_cache_dtype fp8 -tp 8` | 0.93 | Yes | 8gpu |
 | DeepSeek-R1-0528 MXFP4 MTP | `--kv_cache_dtype fp8 -tp 8 --method mtp` | 0.93 | Yes | 8gpu |
-| gpt-oss-120b | `--kv_cache_dtype fp8 --gpu-memory-utilization 0.3` | 0.38 | Yes | mi355-1 |
-| gpt-oss-120b (2 GPU) | `-tp 2 --enable-dp-attention --enable-expert-parallel` | 0.38 | No | mi355-4 |
+| gpt-oss-120b | `--kv_cache_dtype fp8 --gpu-memory-utilization 0.5` | 0.88 | Yes | mi355-1 |
+| gpt-oss-120b (2 GPU) | `-tp 2 --enable-dp-attention --enable-expert-parallel` | 0.88 | No | mi355-4 |
 | Qwen3-235B FP8 | `-tp 8 --enable-expert-parallel` + `ATOM_ENABLE_QK_NORM_ROPE_CACHE_QUANT_FUSION=1` | 0.87 | Yes | 8gpu |
 | Qwen3-235B MXFP4 | `-tp 8 --enable-expert-parallel` + `ATOM_ENABLE_QK_NORM_ROPE_CACHE_QUANT_FUSION=1` | 0.87 | No | 8gpu |
 | Qwen3-Next-80B-A3B | `-tp 8` | 0.65 | Yes | 8gpu |

.claude/commands/inferencex-sync.md

@@ -0,0 +1,84 @@
+---
+name: inferencex-sync
+description: Compare the performance of the latest successful run from ATOM upstream benchmark(https://github.com/ROCm/ATOM/actions/workflows/atom-benchmark.yaml) and that of InferenceX(https://inferencex.semianalysis.com/api/v1/benchmarks?model=$MODEL) and report tput_per_gpu (Total token throughput / number of GPUs), report performance regression of each model on atom framework, create a PR if the performance of ATOM upstream benchamrk is better than that of InferenceX by updating docker image and atom serve arguments (python3 -m atom.entrypoints.openai_server)
+memory: project
+model: opus
+---
+
+# Compare ATOM upstream benchmark and InferneceX, and create a PR to InferenceX Instructions
+
+Use this skill to compare the througput of ATOM upstream benchamrk and InferenceX and create a PR to InferenceX with the udpated docker image and atom serve arguments.
+
+## Workflow
+
+```text
+- [ ] 1) Create a throughput table from InferenceX benchmark
+- [ ] 2) Create a throughput table from ATOM upstream benchmark
+- [ ] 3) Chek performance regression of ATOM upstream benchmark from InferenceX benchmark
+- [ ] 4) Ask users what model needs a new PR
+```
+
+## 1) Create a throughput table from InferenceX benchmark
+
+Check the InferenceX benchmark that is from these URLs and create a throughput performance table from ATOM upstream benchmark. 
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=Kimi-K2.5
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=DeepSeek-V4-Pro
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=DeepSeek-R1-0528
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=Qwen-3.5-397B-A17B
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=GLM-5
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=MiniMax-M2.5
+- https://inferencex.semianalysis.com/api/v1/benchmarks?model=gpt-oss-120b
+
+Rules:
+- 1. Use curl to fetch data from URLs
+- 2. Use only "hardware":"mi355x","framework":"atom","disagg":false,"is_multinode":false,"spec_method":"none"
+- 3. If muliple data of "hardware":"mi355x","framework":"atom","precision":"fp4","disagg":false,"is_multinode":false exist for a same "model", use only the latest "date"
+- 4. Create a throughput performance table of each "tput_per_gpu" of each "model","isl", "osl","conc","num_decode_gpus"(number of GPUs),"image","precision"
+
+## 2) Create a throughput table from ATOM upstream benchmark
+
+Check the ATOM upstream benchmark that is from the latest successful gh action runs at https://github.com/ROCm/ATOM/actions/workflows/atom-benchmark.yaml, which as all jobs passed. Append the throughput to the InferenceX performance table in Step 1) 
+
+Rules:
+- 1. Use main branch. Find the latest job of https://github.com/ROCm/ATOM/actions/workflows/atom-benchmark.yaml where all jobs passed    
+- 2. "tput_per_gpu" (tput/GPU (tok/s)) is "Total token throughput" divided by TP(number of GPUs). TP(number of GPUs) is the value followed by -tp in https://github.com/ROCm/ATOM/blob/main/.github/benchmark/models.json
+- 3. "conc" is "Max Concurrency". Compare same "conc" or "Max Concurrency" between InferenceX benchmark and Atom benchmark
+- 4. Model name follows this mapping 
+
+    ```python
+    # inferenceX_model -> (atom_model, precision)
+    MODEL_MAP = {
+        "dsv4":        [("DeepSeek-V4-Pro",                   "fp4")],
+        "dsr1":        [("DeepSeek-R1-0528",                  "fp8"),
+                        ("DeepSeek-R1-0528-MXFP4",            "fp4")],
+        "kimik2.5":    [("Kimi-K2.5-MXFP4",                   "fp4")],
+        "qwen3.5":     [("Qwen3.5-397B-A17B-FP8",             "fp8"),
+                        ("Qwen3.5-397B-A17B-MXFP4",           "fp4")],
+        "glm5":        [("GLM-5-FP8",                         "fp8"),
+                        ("GLM-5.1-MXFP4",                     "fp4")],
+        "minimaxm2.5": [("MiniMax-M2.7",                      "fp8"),
+                        ("MiniMax-M2.7-MXFP4",                "fp4")],
+        "gptoss120b":  [("gpt-oss-120b",                      "fp4")],
+    }
+    ```
+
+If the data is not available, stop and ask for any workaround
+
+## 3) Chek performance regression of ATOM upstream benchmark from InferenceX benchmark
+
+Update the performance table to show regression in % of ATOM upstream benchmark from InferenceX benchmark
+
+## 4) Chek performance regression of ATOM upstream benchmark from InferenceX benchmark
+
+Based on Step 3) results, ask user what models need a PR to https://github.com/SemiAnalysisAI/InferenceX/pulls and create a PR to each model.
+
+Rules:
+- 1. Find the docker image from the step 2) ATOM upstream benchmark gh action runs. for eample, "Docker:
+  rocm/atom-dev:nightly_202605141645" from https://github.com/ROCm/ATOM/actions/runs/25875375446/job/76041129934#step:6:26 
+- 2. Create a PR and check https://github.com/ROCm/ATOM/blob/main/.github/benchmark/models.json options
+
+PR example: 
+- 1. https://github.com/SemiAnalysisAI/InferenceX/pull/1311/changes
+- 2. https://github.com/SemiAnalysisAI/InferenceX/blob/main/benchmarks/single_node/dsv4_fp4_mi355x_atom.sh
+- 3. https://github.com/SemiAnalysisAI/InferenceX/blob/main/.github/configs/amd-master.yaml#L1646
+- 4. https://github.com/SemiAnalysisAI/InferenceX/blob/main/perf-changelog.yaml#L1824-L1833