fix(hcu): disable CustomAllreduce by default on PCIe (no XGMI), including TP=2

[kabishou11]

Summary

Fixes #54

On PCIe-only HCU topologies (no XGMI), CustomAllreduce previously warned (We are using PCIe's custom allreduce) and then stayed enabled. should_custom_ar still selected the custom kernel for world_size == 2 when fully_connected was false.

On BW100 (gfx936) PCIe, TP=2 + CustomAllreduce can hard-lock the host on the first chat request after health=200. --disable-custom-all-reduce (HCCL) is the stable path.

This change:

• Fail closed to HCCL when is_fully_connected is false, including TP=2.
• Leaves the fully-connected / XGMI CustomAllreduce fast path unchanged.
• Adds an explicit opt-in: VLLM_HCU_ENABLE_PCIE_CUSTOM_ALLREDUCE=1.
• Logs that HCCL is used by default and names the opt-in env in that warning.

Validation

python3 -m pytest tests/patch/test_custom_all_reduce_pcie_gate.py \
  tests/patch/test_custom_all_reduce_native_lifecycle.py \
  tests/runtime_patch/test_shutdown_lifecycle.py -v

20 passed (no real DCU required for the topology gate):

• is_fully_connected=False, world_size=2 → CustomAllreduce disabled
• fully-connected topology still enables CustomAllreduce
• VLLM_HCU_ENABLE_PCIE_CUSTOM_ALLREDUCE=1 re-enables the PCIe path

[alexanderbin123]

Code review summary: one high-priority safety gap was found; please address the inline finding before merge.

Verification performed against ffd3822: the targeted PCIe-gate/native-lifecycle/shutdown suite passes locally (20 passed), and git diff --check is clean. The repository's selected HCU hardware jobs are currently skipped pending trusted ready-hcu authorization, so the aggregate ci-gate remains failing.

[alexanderbin123]

[High] Gate the AITER custom-allreduce path too

This check is bypassed when VLLM_ROCM_USE_AITER=1. In CudaCommunicator.__init__, use_aiter_allreduce then becomes true because VLLM_ROCM_USE_AITER_CUSTOM_AR defaults to true; it constructs AiterCustomAllreduce, and this HCU CustomAllreduce is instantiated only when aiter_ar_comm is None. The AITER wrapper delegates directly to AITER and never evaluates allow_custom_allreduce_for_topology, so a PCIe TP=2 deployment can still select a custom-allreduce backend without VLLM_HCU_ENABLE_PCIE_CUSTOM_ALLREDUCE=1.

--disable-custom-all-reduce disables both paths, whereas this patch gates only one. That leaves the advertised PCIe fail-closed policy incomplete for a supported configuration; given that the reported failure mode is a host hard lock, this path needs an explicit decision before merge. Please enforce the topology/opt-in decision before choosing either custom-AR backend (or explicitly disable AITER custom AR on non-XGMI unless the new opt-in is set), and add coverage for VLLM_ROCM_USE_AITER=1.

[kabishou11]

@alexanderbin123 Thanks for the catch. AITER custom-allreduce is now behind the same PCIe/no-XGMI fail-closed opt-in as the native path (VLLM_HCU_ENABLE_PCIE_CUSTOM_ALLREDUCE=1). Coverage includes VLLM_ROCM_USE_AITER=1.