Skip to content

[Bugfix][Qwen4Exp] Reduce the PLE offload config to DP1 before it is built - #1237

Open
Peuqui wants to merge 1 commit into
1CatAI:mainfrom
Peuqui:pr-ple-offload-dp1-config
Open

Peuqui wants to merge 1 commit into
1CatAI:mainfrom
Peuqui:pr-ple-offload-dp1-config

Conversation

@Peuqui

@Peuqui Peuqui commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fixes #739: its first test was fixed on main by 9170150, this fixes the second. _init_offload_distributed (vllm/v1/ple_offload/worker.py) builds the VllmConfig for the PLE offload process's isolated Gloo world as VllmConfig() and forces DP1/TP1/PP1 only afterwards. ParallelConfig takes the DP variables the process inherits from its GPU worker (VLLM_DP_SIZE, VLLM_DP_RANK, VLLM_DP_RANK_LOCAL), and VllmConfig.__post_init__ derives the participating devices from them (apply_runtime_policy_defaults -> _participating_cuda_device_ids). With an inherited local DP rank of 1 and one visible device, that lookup asks for device 1 and fails before DP1 is set:

vllm/v1/ple_offload/worker.py:374: in _init_offload_distributed
vllm/config/vllm.py:1425: in __post_init__
vllm/platforms/runtime_defaults.py:259: in apply_runtime_policy_defaults
vllm/platforms/runtime_defaults.py:79: in _any_participating_device_is_capability
...
vllm/platforms/cuda.py:700: IndexError
E   IndexError: list index out of range

This PR builds the ParallelConfig first, reduces it to DP1/TP1/PP1 with the same assignments as before, and passes it to VllmConfig(parallel_config=...), so __post_init__ already sees DP1. From the source only, not run: outside the test this would hit the offload process of a DP rank whose GPU worker sees only its own devices (set_device_control_env_var restricts the visible devices per DP rank), if it inherits the DP variables as the test assumes.

The regression test, test_offload_distributed_sets_config_only_for_model_parallel, set the DP variables with monkeypatch.setattr(envs, ...). VLLM_DP_RANK_LOCAL defaults to the current VLLM_DP_RANK, and the test patched VLLM_DP_RANK first, so monkeypatch recorded 1 as the old value of VLLM_DP_RANK_LOCAL and restored it as a module attribute. Every later test in the process then ran as local DP rank 1; with one visible device, test_offload_world_drops_the_pipeline_layer_partition failed with the same IndexError (it passes alone). The test now sets the three variables through set_lazy_env, which exists for exactly this shadowing.

Not a duplicate: gh pr list --repo 1CatAI/1Cat-vLLM --state all --search with "_init_offload_distributed", "PLE offload DP", "VLLM_DP_RANK_LOCAL", "offload DP1", "test_offload_distributed_sets_config_only_for_model_parallel", "ple_offload worker IndexError" and "739 in:body" finds no PR that changes this construction or this test (#1005 mentions #739 only as not covered). #1214 (ours) repairs three other stale fakes in the same test file and merges with this PR without conflicts. git merge-tree against all 124 open PR heads adds one conflict: #1023 (draft) also edits the from vllm.config import ... line of worker.py.

AI assistance was used for this change. I reviewed every line and ran the tests below.

Test Plan

CUDA_VISIBLE_DEVICES=<one GPU> pytest tests/v1/worker/test_ple_offload_worker.py
CUDA_VISIBLE_DEVICES=<two GPUs> pytest tests/v1/worker/test_ple_offload_worker.py
pre-commit run --from-ref origin/main --to-ref HEAD
pre-commit run mypy-3.10 --hook-stage manual --files vllm/v1/ple_offload/worker.py tests/v1/worker/test_ple_offload_worker.py

Path coverage

Path / provider Format Key conditions (shape, TP/EP, graph, capability) Expected selection / fallback Evidence (source / static / executed / measured) Unverified scope
PLE offload process world setup any PLE table process inherits DP variables of its GPU worker, one or more visible devices VllmConfig built from a DP1/TP1/PP1 ParallelConfig executed: unit test on a Tesla V100 and a Quadro RTX 8000, one and two visible devices DP > 1 serving with PLE offload (no such setup here)
PLE offload process world setup any PLE table no inherited DP variables unchanged: the same DP1/TP1/PP1 values as before executed: same test file

Acceleration and benchmark contract (required for performance changes)

Not a performance change. Python only; no new switches, fields or environment variables.

Test Result

  • tests/v1/worker/test_ple_offload_worker.py on main 6f3557c: one visible device (a Tesla V100, and separately a Quadro RTX 8000) 5 failed, 50 passed, 1 skipped; two visible Quadro RTX 8000 3 failed, 52 passed, 1 skipped.
  • The same file with this PR: 3 failed, 52 passed, 1 skipped with one V100, with one RTX 8000 and with two V100. The three remaining failures (test_ple_offload_uses_event_per_inflight_mrv2_batch, test_ple_offload_requires_ple_layers[ple_layer_ids0-True], test_ple_offload_runner_groups_registrations_by_dp_rank) are the stale fakes [Test] Repair stale fakes in PLE offload and SM70 decode-graph tests #1214 repairs; with the test part of [Test] Repair stale fakes in PLE offload and SM70 decode-graph tests #1214 applied as well: 55 passed, 1 skipped with one V100 and with one RTX 8000.
  • Both parts are needed (the regression test and test_offload_world_drops_the_pipeline_layer_partition, one visible V100): with only the runtime change, the second test fails after the first (1 failed, 1 passed); with only the test change, the regression test fails with the IndexError above (1 failed, 1 passed).
  • pre-commit run --from-ref origin/main --to-ref HEAD and mypy-3.10: all hooks pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YaG8VjXn5eN3Aq7eZrA3dp

…built

The PLE offload process builds its own VllmConfig for the isolated Gloo
world and forced DP1 only after construction. ParallelConfig takes the DP
variables the process inherits from its GPU worker, and
VllmConfig.__post_init__ derives the participating devices from them. For
a GPU worker of DP rank 1 that sees only its own device, that lookup asks
for device 1 and raises IndexError before DP1 is set. Build the parallel
config first, reduce it to DP1/TP1/PP1, and pass it to VllmConfig.

The regression test set the DP variables as envs attributes. Because
VLLM_DP_RANK_LOCAL defaults to VLLM_DP_RANK, monkeypatch recorded the
patched rank as its old value and restored local DP rank 1 as a module
attribute, which broke the next test on a host with one visible GPU. Set
them through set_lazy_env.

Co-authored-by: Claude
Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Test] Two tests fail on main: stale QSA E4M3 expectation, PLE offload config with one visible GPU

1 participant