Skip to content

Cuda Graph Capture WA for HIP Runtime #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sammysun0711 wants to merge 5 commits into
base: dev/perf
Choose a base branch
from
Open

Cuda Graph Capture WA for HIP Runtime #147

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
02bde47
Fix fix pa metadata args type mismatch
sammysun0711 Jan 8, 2026
b8384c3
Increase --cuda-graph-max-bs 16 to match concurrency used in mmmu acc…
sammysun0711 Jan 8, 2026
a31d159
Pass q/k/v descale=None to flash_attn_varlen_fp8_pertensor_func to im…
sammysun0711 Jan 8, 2026
fc6abff
Add temp wa for hip runtime bug during cuda graph capture in doc, upd…
sammysun0711 Jan 9, 2026
2a0000b
Merge branch 'dev/perf' into add_hipgraph_capture_wa
sammysun0711 Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions evaluation/Qwen3_Qmni_README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
echo "launching ${model}"
echo "TP=${TP}"
# AITER_MOE_PADDING_SIZE=128 \ only for fp8 tp8
TORCH_NCCL_BLOCKING_WAIT=1 \
AITER_MOE_SMALL_BATCH=1 \
SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=0 \
ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
Expand All @@ -77,15 +78,18 @@
--mm-attention-backend "aiter_attn" \
--max-prefill-tokens 32768 \
--disable-radix-cache \
--page-size 32 \
--page-size 1024 \
--kv-cache-dtype fp8_e4m3 \
--mm-enable-dp-encoder \
--enable-aiter-allreduce-fusion \
--max-running-requests 128 \
--cuda-graph-max-bs 8
--cuda-graph-max-bs 16
2>&1 | tee log.server.log &

```
Please add **AITER_MOE_PADDING_SIZE=128** when launch server for fp8 model with tp8.
Please note
- add **AITER_MOE_PADDING_SIZE=128** when launch server for fp8 model with tp8.
- add **TORCH_NCCL_BLOCKING_WAIT=1** as temporal workaround due to a known HIP runtime bug during cuda graph caputure, which that is tracked here: [ROCm/hip#3876](https://github.com/ROCm/hip/issues/3876).

# Curl request

Expand Down
3 changes: 3 additions & 0 deletions scripts/ci/sglang_benchmark_workflow.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ if [[ "${TYPE}" == "launch" ]]; then
echo "========== LAUNCHING SERVER ========"
if [[ "${model_name}" == "Qwen3-VL-235B" ]]; then
export SGLANG_USE_AITER=1
export TORCH_NCCL_BLOCKING_WAIT=1
echo "********** AOT Prebuild aiter kernel start ... **********"
cd /aiter
python3 op_tests/test_rope.py
Expand Down Expand Up @@ -86,6 +87,7 @@ if [[ "${TYPE}" == "launch" ]]; then
export SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1
export SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=0
export ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export TORCH_NCCL_BLOCKING_WAIT=1
echo "********** AOT Prebuild aiter kernel start ... **********"
cd /aiter
python3 op_tests/test_rope.py
Expand Down Expand Up @@ -119,6 +121,7 @@ if [[ "${TYPE}" == "launch" ]]; then
export SGLANG_ROCM_USE_AITER_PA_ASM_PRESHUFFLE_LAYOUT=0
export SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1
export SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1
export TORCH_NCCL_BLOCKING_WAIT=1
python3 -m sglang.launch_server \
--model-path "${model_path}" \
--host localhost \
Expand Down
Loading