fix(rccl): HIPFIRE_RCCL_LIB points the loader at a librccl outside the ROCm root (nixpkgs ships it in its own store path) - #728
Open
alpineQ wants to merge 1 commit into
Conversation
…e ROCm root (nixpkgs ships it in its own store path)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On NixOS every multi-GPU EP load dies with
failed to dlopen librccl.so, because nixpkgs ships RCCL in its own store path whilerocmtoolkit-merged(the ROCm root hipfire resolves) carries HIP and HSA only; a newHIPFIRE_RCCL_LIBlets the operator point the loader at that path, and nothing changes where RCCL lives under the root.What was wrong
RcclComms::init_all(crates/hip-bridge/src/rccl.rs) askshipfire_config::rocm::library_candidatesforlibrccl.so*. With a selected ROCm root that function returns paths inside the root only, by design (rocm.rs: "An explicit environment root is authoritative. Its split-tree children are considered, but unrelated installs and bare loader sonames are not"). The comment inrccl.rsstill promised "bare sonames last", which the code stopped doing when that policy landed.On nixpkgs the two never meet:
…-rocmtoolkit-merged(selected root, viaHIPFIRE_ROCM_PATHorhipcconPATH)…-rccl-7.2.3So the loader tries six paths under the merged prefix and gives up. Reachable from
hipfire_runtime::ep::all_reduce(ep_batch.rs), i.e. any EP/MoE model on more than one card (DeepSeek-V4, MiniMax, ornith A3B with--tp). Single-card serve and dense TP (rooted reduce-add, #662) never touch RCCL and are unaffected. Distributions that keep RCCL under the ROCm prefix (AMD apt/rpm repos, Arch/opt/rocm, Fedora and Gentoo/usr) are unaffected too, which is why nobody hit it upstream.What changes
HIPFIRE_RCCL_LIB(read throughdeveloper_var, row indocs/env-vars.md): an explicitlibrccl.sopath tried before the ROCm-root candidates. The root policy is untouched; no bare-soname search is added./opt/rocmhints.crates/hip-bridge/map.mdregenerated (scripts/check-crate-maps.py hip-bridge).Which surface(s) does this touch?
crates/hip-bridge(dlopen candidate list inrccl.rsonly; no kernel, dispatch or FFI signature change)crates/hipfire-quantizeTest plan
./scripts/no-gpu-ci.sh: Rust check and no-GPU unit tests pass (732 tests, 0 failures); the Python stage did not run in this shell (pytest/numpy missing and uv unavailable), and this change has no Python.scripts/check-env-docs.pyandscripts/check-crate-maps.py --checkprint byte-identical output to a cleanmastercheckout: the crate-map drift they report (saddle-core, rdna-compute, hipfire-dispatch, hipfire-generate, hipfire-runtime) is onmastertoday and is not touched herecargo build --release -p hip-bridge --features lab --example rccl_smokecleanhip-bridge/examples/rccl_smokeon 2× RX 7900 XTX (gfx1100), nixpkgs ROCm 7.2,HIPFIRE_ROCM_PATH=rocmtoolkit-merged,HIP_VISIBLE_DEVICES=0,1, binary md5f5b44f51bd462f6a5f6e014bac6df32d— A/B belowrccl_smoke A/B (same binary; only the variable differs)
A —
HIPFIRE_RCCL_LIBunset,LD_LIBRARY_PATH= root only (whatmasterdoes today, error text from this branch):B —
HIPFIRE_RCCL_LIB=/nix/store/c8dc…-rccl-7.2.3/lib/librccl.so:Hardware validation request (optional)
{ "routes": [ {"mode": "battery", "tag": "qwen3.6:27b"} ], "claim": "single-card and dense-TP serve never reach this code; with RCCL under the ROCm root the candidate list is unchanged, so no hardware route changes behaviour. The fix is reachable only where the resolved root lacks librccl (nixpkgs)." }Related: #162, #645 (NixOS packaging).
Architecture-trait change?
No.