Problem trying to get vLLM + Step3 + AFD running

[orionorin]

Hi StepMesh team,

We are recently experimenting with AFD models, but failed to get the Step3 AFD PR in the vLLM repo (draft AFD implementation for step3 by Oliver-ss · Pull Request #25162 · vllm-project/vllm) running, and we would like to get some suggestions on configuration and environment setup.

Environment

Hardware

We are using 2x8 H20 servers. Each server has 9 dual-port RNICs with bond, among which bond2~bond9 supports RDMA.

FFN Server:

bond8            UP             1.2.3.176/25
bond2            UP             1.2.3.219/25
bond7            UP             1.2.3.87/25
bond9            UP             1.2.3.93/25
bond3            UP             1.2.3.9/25
bond4            UP             1.2.3.162/25
bond5            UP             1.2.3.67/25
bond1            UP             1.2.4.41/26
bond6            UP             1.2.3.205/25

Attn Server:

bond8            UP             1.2.3.211/25
bond2            UP             1.2.3.180/25
bond7            UP             1.2.3.84/25
bond9            UP             1.2.3.66/25
bond3            UP             1.2.3.6/25
bond4            UP             1.2.3.217/25
bond1            UP             1.2.4.18/26
bond5            UP             1.2.3.63/25
bond6            UP             1.2.3.202/25

Software

• vLLM commit is draft AFD implementation for step3 by Oliver-ss · Pull Request #25162 · vllm-project/vllm
• StepMesh commit is bf55db9
• Both are built from source with minor modifications -- vLLM is modified to set DMLC_INTERFACE to bond2 in our environment, and some extra logging statements are inserted into StepMesh.
• CUDA version is 12.8
• PyTorch version is 2.8.0

ATTN DP + FFN TP

When we ran Step3 (FP8) with Attn DP8 + FFN TP8 disaggregation, vLLM would crash during StepMesh scheduler initialization. The steps are:

Start FFN server:

vllm fserver /data1/step3-fp8 \
        -tp 8 \
        --afd-config '{"afd_connector": "stepmesh", "afd_role": "ffn", "afd_host": "1.2.3.219"}' \
        --max-num-batched-tokens 384 --max-num-seqs 384 \
        --compilation-config '{"cudagraph_mode": "PIECEWISE", "cudagraph_capture_sizes": [1, 8]}'

Start Attn server:

vllm serve /data1/step3-fp8 \
        -dp 8 \
        --served-model-name step3 \
        --afd-config '{"afd_connector": "stepmesh", "afd_role": "attention", "afd_host": "1.2.3.219"}' \
        --max-num-batched-tokens 384 --max-num-seqs 384 \
        --compilation-config '{"cudagraph_mode": "PIECEWISE", "cudagraph_capture_sizes": [1, 8]}'

What we see is each of the 8 Attention servers would send an ADD_NODE with aux_id=0, resulting in a collision in the StepMesh scheduler:

[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.63, port=48821, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
bind to DMLC_NODE_HOST: 1.2.3.217
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:557: automatic detect interface and ip from gpu(0): bond3 (1.2.3.6)
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: 1.2.3.6
BindToCpuCore: gpu 0 -> cpu 11
BindToCpuCore: gpu 0 -> cpu 12
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:198: qp created: pd=0x7f3730000fc0 , cq=0x7f37300010d0, qp=37482
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.84, port=53355, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.84, port=53355, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
BindToCpuCore: gpu 0 -> cpu 11
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.66, port=45585, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.66, port=45585, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:173:  bound to port 56187
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:601: Bind to [role=worker, ip=1.2.3.6, port=56187, is_recovery=0, aux_id=-1, num_ports=1]
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:180: Connecting to Node 1, My_Node=32767
BindToCpuCore: gpu 0 -> cpu 12
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:173:  bound to port 40665
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:601: Bind to [role=worker, ip=1.2.3.217, port=40665, is_recovery=0, aux_id=-1, num_ports=1]
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:180: Connecting to Node 1, My_Node=32767
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.211, port=57059, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.211, port=57059, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:198: qp created: pd=0x7fb72c001030 , cq=0x7fb72c001140, qp=19012
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:198: qp created: pd=0x7fdc1c001030 , cq=0x7fdc1c001140, qp=16317
BindToCpuCore: gpu 0 -> cpu 11
BindToCpuCore: gpu 0 -> cpu 12
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.202, port=60397, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.202, port=60397, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
BindToCpuCore: gpu 0 -> cpu 11
BindToCpuCore: gpu 0 -> cpu 12
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:198: qp created: pd=0x7fb72c001030 , cq=0x7fb72c001140, qp=19013
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:198: qp created: pd=0x7fdc1c001030 , cq=0x7fdc1c001140, qp=16318
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
BindToCpuCore: gpu 0 -> cpu 11
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.180, port=42787, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.180, port=42787, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
BindToCpuCore: gpu 0 -> cpu 12
BindToCpuCore: gpu 0 -> cpu 11
BindToCpuCore: gpu 0 -> cpu 12
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.6, port=56187, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.6, port=56187, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
BindToCpuCore: gpu 0 -> cpu 11
BindToCpuCore: gpu 0 -> cpu 12
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/./rdma_van.h:1099: 32767 OnConnected to 1
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:628: rdma 32767     gonna notify scheduler of myself: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.217, port=40665, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/././rdma_transport.h:413: SendRendezvousBegin 
[14:11:52] worker 0 /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:703: rdma 32767     sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=1.2.3.217, port=40665, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0

[14:11:52] scheduler 0 /nfs_disk/XXXXXX/afd/StepMesh/include/dmlc/logging.h:301: [14:11:52] /nfs_disk/XXXXXX/afd/StepMesh/src/van.cc:226: Check failed: (worker_ranks.find(node.aux_id)) == (worker_ranks.end()) rank must be unique: [role=worker, ip=1.2.3.63, port=48821, is_recovery=0, aux_id=0, num_ports=1]

It seems to me this aux_id ultimately comes form gpu_model_runner.py in vLLM, and for some reason vLLM obtained an incorrect local_rank.

        # init AFD config
        self.afd_config = vllm_config.afd_config
        if self.afd_config and self.afd_config.afd_role == "attention":
            self.afd_connector = AFDConnectorFactory.create_connector(
                get_world_group().rank,
                get_world_group().local_rank, vllm_config)
            self.afd_connector.init_afd_connector()
            self.num_stages = self.afd_config.num_afd_stages

ATTN TP + FFN TP (If this is a supported config)

Dynamo Issue

vLLM could not start with default configuration. The error messages below seem to indicate the call to time.time in the AFD metadata creation code cannot be traced. So we use --enforce_eager to disable compilation in later runs.

(Worker_TP1 pid=25649) INFO 11-11 15:19:14 [gpu_model_runner.py:2470] Starting to load model /data1/step3-fp8...
(Worker_TP1 pid=25649) INFO 11-11 15:19:14 [gpu_model_runner.py:2502] Loading model from scratch...
(Worker_TP1 pid=25649) INFO 11-11 15:19:14 [layer.py:437] MultiHeadAttention attn_backend: _Backend.XFORMERS, use_upstream_fa: False
(Worker_TP1 pid=25649) INFO 11-11 15:19:15 [cuda.py:368] Using Flash Attention backend on V1 engine.
(Worker_TP1 pid=25649) WARNING 11-11 15:19:15 [fp8.py:574] CutlassBlockScaledGroupedGemm not supported on the current platform.
(Worker_TP1 pid=25649) INFO 11-11 15:19:38 [default_loader.py:268] Loading weights took 21.74 seconds
(Worker_TP1 pid=25649) INFO 11-11 15:19:39 [gpu_model_runner.py:2524] Model loading took 39.8007 GiB and 23.545146 seconds
(Worker_TP1 pid=25649) INFO 11-11 15:19:39 [gpu_model_runner.py:3186] Encoder cache will be initialized with a budget of 3164 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP1 pid=25649) WARNING 11-11 15:19:39 [__init__.py:2179] The following intended overrides are not keyword args and will be dropped: {'truncation'}
(Worker_TP1 pid=25649) WARNING 11-11 15:19:39 [__init__.py:2179] The following intended overrides are not keyword args and will be dropped: {'truncation'}
(Worker_TP1 pid=25649) /usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py:1481: UserWarning: Dynamo does not know how to trace the builtin `time.time.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).
(Worker_TP1 pid=25649) If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.
(Worker_TP1 pid=25649) If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.
(Worker_TP1 pid=25649)   torch._dynamo.utils.warn_once(explanation + "\n" + "\n".join(hints))
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] WorkerProc hit an exception.
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_worker.py", line 270, in determine_available_memory
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     self.model_runner.profile_run()
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 3217, in profile_run
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     = self._dummy_run(self.max_num_tokens, is_profile=True)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 2995, in _dummy_run
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     outputs = self.model(
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]               ^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_vl.py", line 1065, in forward
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     hidden_states = self.language_model(input_ids,
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 609, in forward
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/compilation/decorators.py", line 305, in __call__
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     output = self.compiled_callable(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 745, in compile_wrapper
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     raise e.with_traceback(None) from e.__cause__  # User compiler error
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Explanation: Dynamo does not know how to trace the builtin `time.time.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Hint: If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Hint: If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Developer debug context: module: time, qualname: time, skip reason: <missing reason>
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] from user code:
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]    File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 523, in forward
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     metadata = AFDConnectorMetadata.create_attention_metadata(
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/distributed/afd_transfer/afd_connector/metadata.py", line 72, in create_attention_metadata
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     timestamp=time.time())
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_worker.py", line 270, in determine_available_memory
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     self.model_runner.profile_run()
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 3217, in profile_run
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     = self._dummy_run(self.max_num_tokens, is_profile=True)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 2995, in _dummy_run
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     outputs = self.model(
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]               ^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_vl.py", line 1065, in forward
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     hidden_states = self.language_model(input_ids,
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 609, in forward
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/compilation/decorators.py", line 305, in __call__
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     output = self.compiled_callable(*args, **kwargs)
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 745, in compile_wrapper
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     raise e.with_traceback(None) from e.__cause__  # User compiler error
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Explanation: Dynamo does not know how to trace the builtin `time.time.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Hint: If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Hint: If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   Developer debug context: module: time, qualname: time, skip reason: <missing reason>
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] from user code:
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]    File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 523, in forward
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     metadata = AFDConnectorMetadata.create_attention_metadata(
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/distributed/afd_transfer/afd_connector/metadata.py", line 72, in create_attention_metadata
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]     timestamp=time.time())
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671] 
(Worker_TP1 pid=25649) ERROR 11-11 15:19:45 [multiproc_executor.py:671]

Inference Issue

With --enforce-eager we are able to bring up the servers:

vllm fserver /data1/step3-fp8 \
        -tp 8 \
        --afd-config '{"afd_connector": "stepmesh", "afd_role": "ffn", "afd_host": "1.2.3.219"}' \
        --max-num-batched-tokens 384 --max-num-seqs 384 \
        --enforce-eager

vllm serve /data1/step3-fp8 \
        -tp 8 \
        --served-model-name step3 \
        --afd-config '{"afd_connector": "stepmesh", "afd_role": "attention", "afd_host": "1.2.3.219"}' --max-num-batched-tokens 384 --max-num-seqs 384 \
        --enforce-eager

But after sending a request, we got an error:

(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] WorkerProc hit an exception.
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_worker.py", line 462, in execute_model
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     output = self.model_runner.execute_model(scheduler_output,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 2181, in execute_model
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     model_output = self.model(
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                    ^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_vl.py", line 1065, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     hidden_states = self.language_model(input_ids,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 609, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/compilation/decorators.py", line 223, in __call__
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self.forward(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 519, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     layer.compute_attn_output(
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 383, in compute_attn_output
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._compute_attn_output(hidden_states, residual,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 346, in _compute_attn_output
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     hidden_states = self.self_attn(positions=positions,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 196, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     qkv, _ = self.qkv_proj(hidden_states)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/linear.py", line 375, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     output = self.quant_method.apply(self, x, bias)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/quantization/fp8.py", line 493, in apply
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1243, in __call__
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._op(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 167, in apply_w8a8_block_fp8_linear
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     input_2d = torch.nn.functional.pad(input_2d,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py", line 5290, in pad
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return torch._C._nn.pad(input, pad, mode, value)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] torch.AcceleratorError: CUDA error: invalid configuration argument
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] 
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_worker.py", line 462, in execute_model
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     output = self.model_runner.execute_model(scheduler_output,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 2181, in execute_model
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     model_output = self.model(
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                    ^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_vl.py", line 1065, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     hidden_states = self.language_model(input_ids,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 609, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/compilation/decorators.py", line 223, in __call__
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self.forward(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 519, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     layer.compute_attn_output(
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 383, in compute_attn_output
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._compute_attn_output(hidden_states, residual,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 346, in _compute_attn_output
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     hidden_states = self.self_attn(positions=positions,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 196, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     qkv, _ = self.qkv_proj(hidden_states)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/linear.py", line 375, in forward
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     output = self.quant_method.apply(self, x, bias)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/quantization/fp8.py", line 493, in apply
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1243, in __call__
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return self._op(*args, **kwargs)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 167, in apply_w8a8_block_fp8_linear
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     input_2d = torch.nn.functional.pad(input_2d,
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py", line 5290, in pad
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]     return torch._C._nn.pad(input, pad, mode, value)
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] torch.AcceleratorError: CUDA error: invalid configuration argument
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671] 
(Worker_TP1 pid=26589) ERROR 11-11 15:41:15 [multiproc_executor.py:671]

[niehao100]

Hi @orionorin

Our test scrips is as bellow and we only set enforce-eager for attention.

In addition, set DMLC_INTERFACE will force all 8 process use bond0, please just set it to "auto", or set this ENV based on GPU id.

vllm fserver /mnt/step3-flash-tp 8 --afd-config '{"afd_connector": "stepmesh", "afd_role": "ffn", "afd_host": "10.203.8.x"}' --max-num-batched-tokens 384 --max-num-seqs 384 --compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8, 16]}' --load-format dummy


vllm serve /mnt/step3 -dp 8 --afd-config '{"afd_connector": "stepmesh", "afd_role": "attention", "afd_host": "10.203.8.x"}' --enforce-eager --max-num-batched-tokens 256 --max-num-seqs 256 --port 9999

[orionorin]

Hi,

In addition, set DMLC_INTERFACE will force all 8 process use bond0, please just set it to "auto", or set this ENV based on GPU id.

We only modified DMLC_INTERFACE = bond2 in vLLM's _start_scheduler_process as it was originally hard-coded as os.environ["DMLC_INTERFACE"] = "brainpf_bond0";. I think other places still used "auto". Does this sound correct?

(In addition, the result didn't change if I set DMLC_INTERFACE = auto in that line.)

Our test scrips is as bellow and we only set enforce-eager for attention.

With Attn DP, I tried using --enforce-eager only at the Attn side (not at the FFN side), and still got the same error as the "ATTN DP + FFN TP" section. I think the real problem is all DP ranks are initializing StepMesh with gpu=0?

FFN side:

INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=5, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=6, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=4, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:107] AFD connector init: self.afd_config.afd_role='ffn', rank=7, local_rank=7
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=7, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=2, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=1, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
INFO 11-11 18:35:08 [stepmesh_connector.py:150] StepMesh environment setup: role=server, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=3, group_size=8

Attn side:

(EngineCore_DP6 pid=28039) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 6 in world size 8 is assigned as DP rank 6, PP rank 0, TP rank 0, EP rank 6
(EngineCore_DP0 pid=28033) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP5 pid=28038) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 5 in world size 8 is assigned as DP rank 5, PP rank 0, TP rank 0, EP rank 5
(EngineCore_DP2 pid=28035) INFO 11-11 17:40:36 [cuda_communicator.py:89] Using naive all2all manager.
(EngineCore_DP3 pid=28036) INFO 11-11 17:40:36 [cuda_communicator.py:89] Using naive all2all manager.
(EngineCore_DP4 pid=28037) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 4 in world size 8 is assigned as DP rank 4, PP rank 0, TP rank 0, EP rank 4
(EngineCore_DP1 pid=28034) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 1 in world size 8 is assigned as DP rank 1, PP rank 0, TP rank 0, EP rank 1
(EngineCore_DP2 pid=28035) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 2 in world size 8 is assigned as DP rank 2, PP rank 0, TP rank 0, EP rank 2
(EngineCore_DP3 pid=28036) INFO 11-11 17:40:36 [parallel_state.py:1206] rank 3 in world size 8 is assigned as DP rank 3, PP rank 0, TP rank 0, EP rank 3
...
(EngineCore_DP3 pid=28036) INFO 11-11 17:40:37 [stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
[(EngineCore_DP7 pid=28040) INFO 11-11 17:40:37 [stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
[stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
(EngineCore_DP4 pid=28037) INFO 11-11 17:40:37 [stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
(EngineCore_DP0 pid=28033) INFO 11-11 17:40:37 [stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
(EngineCore_DP2 pid=28035) INFO 11-11 17:40:37 [stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8
(EngineCore_DP5 pid=28038) INFO 11-11 17:40:37 [stepmesh_connector.py:150] StepMesh environment setup: role=worker, num_worker=1, num_server=1, port=1239, host=1.2.3.219, node_rank=0, gpu=0, group_size=8

[niehao100]

Hi，

_start_scheduler_process can use any one of RNICs, it is not the root cause.

You are right, all Attn worker uses gpu 0, that's wrong. It seems this problem is caused by Stepmesh_connector.py in vllm, as the local rank send to StepMeshConnector is wrong.

Local rank is set in get_world_group().local_rank at https://github.com/vllm-project/vllm/blob/d9df2089a0fce26cbdbe72d639736627983d9c3d/vllm/v1/worker/gpu_model_runner.py#L384.

Maybe print them for trace is a good idea.

[orionorin]

Hi,

I traced the code and found the local_rank for the world_group is ultimately set in the function parallel_state.py:init_distributed_environment: https://github.com/vllm-project/vllm/blob/d9df2089a0fce26cbdbe72d639736627983d9c3d/vllm/distributed/parallel_state.py#L1023C5-L1023C33, at line https://github.com/vllm-project/vllm/blob/d9df2089a0fce26cbdbe72d639736627983d9c3d/vllm/distributed/parallel_state.py#L1079.

I see in every the Attn DP processes, this function is invoked with world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://xxx:yyy backend=nccl. Then it adjusts these values to world_size=8 rank=K in lines https://github.com/vllm-project/vllm/blob/d9df2089a0fce26cbdbe72d639736627983d9c3d/vllm/distributed/parallel_state.py#L1035-L1047, but does not adjust local_rank. Since local_rank is passed as 0 instead of -1, it will be used for all processes, causing errors.

Do the arguments world_size=1 rank=0 local_rank=0 to init_distributed_environment sound right? Or is there extra logic for setting local_rank that is missing from the public PR? I have tried setting local_rank manually there but am still blocked by some other errors yet. Thanks!

[orionorin]

Hi, I could successfully get the Attn + DP server running by setting DMLC_RANK_OFFSET to the actual rank manually in _setup_stepmesh_env. However, I still hit the CUDA configuration error when I tried to send a request.

Since the error is thrown from PyTorch, I wonder if there is a CUDA/PyTorch version mismatch. Could you share the relevant software versions you used for the test?

(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/v1/worker/gpu_model_runner.py", line 3004, in _dummy_run
(EngineCore_DP6 pid=51223)     outputs = self.model(
(EngineCore_DP6 pid=51223)               ^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP6 pid=51223)     return self._call_impl(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP6 pid=51223)     return forward_call(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_vl.py", line 1065, in forward
(EngineCore_DP6 pid=51223)     hidden_states = self.language_model(input_ids,
(EngineCore_DP6 pid=51223)                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP6 pid=51223)     return self._call_impl(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP6 pid=51223)     return forward_call(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 609, in forward
(EngineCore_DP6 pid=51223)     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(EngineCore_DP6 pid=51223)                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/compilation/decorators.py", line 223, in __call__
(EngineCore_DP6 pid=51223)     return self.forward(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 519, in forward
(EngineCore_DP6 pid=51223)     layer.compute_attn_output(
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 383, in compute_attn_output
(EngineCore_DP6 pid=51223)     return self._compute_attn_output(hidden_states, residual,
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 346, in _compute_attn_output
(EngineCore_DP6 pid=51223)     hidden_states = self.self_attn(positions=positions,
(EngineCore_DP6 pid=51223)                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP6 pid=51223)     return self._call_impl(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP6 pid=51223)     return forward_call(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/models/step3_text.py", line 196, in forward
(EngineCore_DP6 pid=51223)     qkv, _ = self.qkv_proj(hidden_states)
(EngineCore_DP6 pid=51223)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP6 pid=51223)     return self._call_impl(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP6 pid=51223)     return forward_call(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/linear.py", line 375, in forward
(EngineCore_DP6 pid=51223)     output = self.quant_method.apply(self, x, bias)
(EngineCore_DP6 pid=51223)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/quantization/fp8.py", line 493, in apply
(EngineCore_DP6 pid=51223)     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1243, in __call__
(EngineCore_DP6 pid=51223)     return self._op(*args, **kwargs)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/nfs_disk/XXXXXX/afd/vllm-afd/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 167, in apply_w8a8_block_fp8_linear
(EngineCore_DP6 pid=51223)     input_2d = torch.nn.functional.pad(input_2d,
(EngineCore_DP6 pid=51223)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py", line 5290, in pad
(EngineCore_DP6 pid=51223)     return torch._C._nn.pad(input, pad, mode, value)
(EngineCore_DP6 pid=51223)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP6 pid=51223) torch.AcceleratorError: CUDA error: invalid configuration argument
(EngineCore_DP6 pid=51223) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP6 pid=51223) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP6 pid=51223) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[niehao100]

Hi @orionorin,

Thanks for your test, the behaviour of get_local_rank is also confusing for us, it's great to hear that manual configuration can solve the first problem.
Our testbed is built with torch 2.7.0 on CUDA 12.8, and NVIDAI driver 570.124.06, hope it will work for you too.

[orionorin]

Hi, it seems the CUDA configuration error we encountered in our environment was triggered when executing a dummy batch. We have a minimal reproducing script containing the computation steps to trigger the error below. Can you confirm if it also errors in your environment? Thanks!

The Minimal Example

min.py:

import torch
from vllm.model_executor.layers.layernorm import RMSNorm

torch.set_default_device('cuda')
norm = RMSNorm(7168, eps=1e-5)

x = torch.empty((0, 7168))
print(0, x.shape)

n = norm(x)
print(1, n.shape)

y = torch.nn.functional.pad(n, (0, 0, 0, 4), "constant", 0)
print(2, y.shape)

Cause of Error

When executing a dummy batch, a hidden_states of shape (0, 7168) is passed in to _compute_attn_output, causing the latter to invoke a torch C op which in turn launches a CUDA kernel with gridDim = (0,1,1), which is illegal. The error is not caught until the later call to torch.nn.functional.pad, but the pad itself is not the culprit. To be specific, the error is triggered during hidden_states = self.input_layernorm(hidden_states) of step3_text.py:Step3TextDecoderLayer::_compute_attn_output (https://github.com/vllm-project/vllm/pull/25162/files#diff-040c89e6819d540a73cefdd9b8388acb7d7ff541c584e52f51e6d94589285e31L268).

When tracing this using an internal library (see [DEBUG] lines), we see this script invokes a kernel with gridDim = (0,1,1), and cudaLaunchKernel returned an error code, but the error was not caught until the call to pad.

# LD_PRELOAD=/root/cuda_tracer.so python3 min.py
INFO 11-27 23:16:12 [__init__.py:216] Automatically detected platform cuda.
WARNING 11-27 23:16:13 [__init__.py:3732] Current vLLM config is not set.
[DLTRACER] Process 66952:66952 launching kernel, func = 0x7f75422cfbc0 ((NULL):(NULL)), gridDim = (7,1,1), blockDim = (128,1,1), args = 0x7fffe99c4cb0, sharedMem = 0, stream = (nil) => returned 0
0 torch.Size([0, 7168])
[DEBUG] Process 66952:66952 launching kernel, func = 0x7f736c9c78a0 ((NULL):(NULL)), gridDim = (0,1,1), blockDim = (1024,1,1), args = 0x7fffe99c3fd0, sharedMem = 0, stream = (nil) => returned 9
1 torch.Size([0, 7168])
[DEBUG] Process 66952:66952 launching kernel, func = 0x7f75422cfbc0 ((NULL):(NULL)), gridDim = (28,1,1), blockDim = (128,1,1), args = 0x7fffe99c5370, sharedMem = 0, stream = (nil) => returned 0
Traceback (most recent call last):
  File "/nfs_disk/XXXXXX/afd/scripts/min.py", line 13, in <module>
    y = torch.nn.functional.pad(n, (0, 0, 0, 4), "constant", 0)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py", line 5195, in pad
    return handle_torch_function(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/overrides.py", line 1721, in handle_torch_function
    result = mode.__torch_function__(public_api, types, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_device.py", line 104, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/functional.py", line 5209, in pad
    return torch._C._nn.pad(input, pad, mode, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[niehao100]

Hi, we have the same issus:

INFO 12-01 19:32:49 [__init__.py:244] Automatically detected platform cuda.
WARNING 12-01 19:33:11 [config.py:4801] Current vLLM config is not set.
WARNING 12-01 19:33:11 [config.py:4801] Current vLLM config is not set.
WARNING 12-01 19:33:11 [config.py:4801] Current vLLM config is not set.
0 torch.Size([0, 7168])
1 torch.Size([0, 7168])
Traceback (most recent call last):
  File "/tmp/test.py", line 13, in <module>
    y = torch.nn.functional.pad(n, (0, 0, 0, 4), "constant", 0)
  File "/data/torch_venv/2.7.0/lib/python3.10/site-packages/torch/nn/functional.py", line 5195, in pad
    return handle_torch_function(
  File "/data/torch_venv/2.7.0/lib/python3.10/site-packages/torch/overrides.py", line 1721, in handle_torch_function
    result = mode.__torch_function__(public_api, types, args, kwargs)
  File "/data/torch_venv/2.7.0/lib/python3.10/site-packages/torch/utils/_device.py", line 104, in __torch_function__
    return func(*args, **kwargs)
  File "/data/torch_venv/2.7.0/lib/python3.10/site-packages/torch/nn/functional.py", line 5209, in pad
    return torch._C._nn.pad(input, pad, mode, value)
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[niehao100]

It seems the root cause is that the first dim of x = torch.empty((0, 7168)) is 0. This can work,

import torch
from vllm.model_executor.layers.layernorm import RMSNorm

torch.set_default_device('cuda')
norm = RMSNorm(7168, eps=1e-5)

x = torch.empty((1, 7168))
print(0, x.shape)

n = norm(x)
print(1, n.shape)

y = torch.nn.functional.pad(n, (0, 0, 0, 4), "constant", 0)
print(2, y.shape)