Skip to content

fix(rccl): HIPFIRE_RCCL_LIB points the loader at a librccl outside the ROCm root (nixpkgs ships it in its own store path) #728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alpineQ wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix(rccl): HIPFIRE_RCCL_LIB points the loader at a librccl outside the ROCm root (nixpkgs ships it in its own store path) #728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions crates/hip-bridge/map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,21 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside

| File | Lines | Public items | Tests |
|---|---:|---:|---:|
| [`src/error.rs`](src/error.rs) | 70 | 8 | 0 |
| [`src/ffi.rs`](src/ffi.rs) | 1,772 | 96 | 0 |
| [`src/error.rs`](src/error.rs) | 132 | 10 | 2 |
| [`src/ffi.rs`](src/ffi.rs) | 1,799 | 98 | 0 |
| [`src/kernarg.rs`](src/kernarg.rs) | 178 | 14 | 3 |
| [`src/lib.rs`](src/lib.rs) | 181 | 18 | 2 |
| [`src/rccl.rs`](src/rccl.rs) | 470 | 18 | 0 |
| [`src/lib.rs`](src/lib.rs) | 222 | 19 | 2 |
| [`src/rccl.rs`](src/rccl.rs) | 475 | 18 | 0 |
| [`src/rocblas.rs`](src/rocblas.rs) | 744 | 17 | 3 |
| [`src/rocsolver.rs`](src/rocsolver.rs) | 374 | 12 | 3 |
| [`src/vmm.rs`](src/vmm.rs) | 566 | 12 | 4 |
| [`src/vmm.rs`](src/vmm.rs) | 567 | 12 | 4 |

### Public API surface

- [`src/error.rs`](src/error.rs): `HipErrorCode`, `HipResult`, `HIP_ERROR_INVALID_IMAGE`, `HIP_ERROR_PEER_ACCESS_UNSUPPORTED`, `HIP_ERROR_PEER_ACCESS_ALREADY_ENABLED`, `HIP_ERROR_PEER_ACCESS_NOT_ENABLED`, `HipError`, `new`
- [`src/ffi.rs`](src/ffi.rs): `launch_counters`, `record`, `record_bytes`, `time_ns`, `count`, `bytes`, `reset`, `HipMemGenericAllocationHandle`, `HIP_MEM_LOCATION_TYPE_DEVICE`, `HIP_MEM_ALLOCATION_TYPE_PINNED`, `HIP_MEM_ACCESS_FLAGS_PROT_READ_WRITE`, `HIP_MEM_ALLOCATION_GRANULARITY_MINIMUM`, +84 more
- [`src/error.rs`](src/error.rs): `HipErrorCode`, `HipResult`, `HIP_ERROR_INVALID_IMAGE`, `HIP_ERROR_PEER_ACCESS_UNSUPPORTED`, `HIP_ERROR_PEER_ACCESS_ALREADY_ENABLED`, `HIP_ERROR_PEER_ACCESS_NOT_ENABLED`, `LaunchContext`, `HipError`, `new`, `with_kernel`
- [`src/ffi.rs`](src/ffi.rs): `launch_counters`, `record`, `record_bytes`, `time_ns`, `count`, `bytes`, `reset`, `HipMemGenericAllocationHandle`, `HIP_ERROR_NOT_READY`, `HIP_MEM_LOCATION_TYPE_DEVICE`, `HIP_MEM_ALLOCATION_TYPE_PINNED`, `HIP_MEM_ACCESS_FLAGS_PROT_READ_WRITE`, +86 more
- [`src/kernarg.rs`](src/kernarg.rs): `KernargBlob`, `new`, `with_capacity`, `len`, `is_empty`, `push_ptr`, `push_u32`, `push_i32`, `push_f32`, `push_u64`, `pad_to`, `as_mut_slice`, +2 more
- [`src/lib.rs`](src/lib.rs): `error`, `ffi`, `kernarg`, `rccl`, `rocblas`, `rocsolver`, `vmm`, `MemcpyKind`, `MemoryType`, `from_raw`, `DeviceBuffer`, `as_ptr`, +6 more
- [`src/lib.rs`](src/lib.rs): `error`, `ffi`, `kernarg`, `rccl`, `rocblas`, `rocsolver`, `vmm`, `MemcpyKind`, `MemoryType`, `from_raw`, `DeviceBuffer`, `as_ptr`, +7 more
- [`src/rccl.rs`](src/rccl.rs): `NCCL_SUCCESS`, `RcclError`, `RcclResult`, `RcclDataType`, `RcclRedOp`, `RcclComms`, `init_all`, `len`, `is_empty`, `version`, `group_start`, `group_end`, +6 more
- [`src/rocblas.rs`](src/rocblas.rs): `RocblasError`, `RocblasResult`, `ROCBLAS_STATUS_SUCCESS`, `ROCBLAS_STATUS_INVALID_VALUE`, `RocblasOperation`, `RocblasDatatype`, `RocblasGemmAlgo`, `Rocblas`, `load`, `set_stream`, `handle`, `has_dgemm`, +5 more
- [`src/rocsolver.rs`](src/rocsolver.rs): `ROCSOLVER_STATUS_SUCCESS`, `RocblasFill`, `RocblasDiagonal`, `RocsolverError`, `RocsolverResult`, `Rocsolver`, `load`, `load_from_handle`, `has_dpotri`, `dpotrf`, `dtrtri`, `dpotri`
Expand All @@ -56,6 +56,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside

### Totals

- 8 modules · 4,355 lines · 195 public items · 15 tests · 8 examples
- 8 modules · 4,491 lines · 200 public items · 17 tests · 8 examples

<!-- crate-map:generated:end -->
21 changes: 13 additions & 8 deletions crates/hip-bridge/src/rccl.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,13 +101,18 @@ impl RcclComms {
/// `ncclCommInitAll`. Each comm[i] binds to `device_ids[i]`.
pub fn init_all(device_ids: &[i32]) -> RcclResult<Self> {
let lib = unsafe {
// Resolved ROCm roots first, bare sonames last, so RCCL is found on
// side-by-side and /opt/rocm/core-<ver> installs too.
let candidates = hipfire_config::rocm::library_candidates(&[
"librccl.so",
"librccl.so.1",
"librccl.so.1.0",
]);
// `HIPFIRE_RCCL_LIB` first, then the resolved ROCm root.
// `library_candidates` stops at the selected root on purpose (an
// explicit root is authoritative; see `hipfire_config::rocm`), and
// RCCL follows that policy. The override is for distributions whose
// ROCm prefix does not carry RCCL: nixpkgs ships librccl in its own
// store path next to `rocmtoolkit-merged`.
const SONAMES: [&str; 3] = ["librccl.so", "librccl.so.1", "librccl.so.1.0"];
let mut candidates = Vec::new();
if let Ok(explicit) = hipfire_config::developer_var("HIPFIRE_RCCL_LIB") {
candidates.push(explicit);
}
candidates.extend(hipfire_config::rocm::library_candidates(&SONAMES));
let mut loaded = None;
for name in &candidates {
if let Ok(l) = Library::new(name) {
Expand All @@ -118,7 +123,7 @@ impl RcclComms {
loaded.ok_or_else(|| RcclError {
status: 0,
context: format!(
"failed to dlopen librccl.so. Tried: {:?}. Is RCCL installed (apt install rccl, or /opt/rocm/lib/librccl.so.1)?",
"failed to dlopen librccl.so. Tried: {:?}. Is RCCL installed (apt install rccl, or /opt/rocm/lib/librccl.so.1)? If it lives outside the ROCm root, set HIPFIRE_RCCL_LIB=/path/to/librccl.so.",
candidates
),
})?
Expand Down
4 changes: 3 additions & 1 deletion docs/env-vars.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ diagnostic and developer harness exports pending their cleanup.
| `HIPFIRE_MAX_REQUEST_BYTES` | Body cap |
| `HIPFIRE_SERVE_MAX_QUEUE` / `HIPFIRE_SERVE_QUEUE_TIMEOUT_MS` | Admission queue |
| `HIPFIRE_EXPERIMENTAL_BUDGET_ALERT` | Research budget nudge |
| `HIPFIRE_RCCL_LIB` | Explicit `librccl.so` path, tried before the ROCm root. For distributions whose ROCm prefix does not carry RCCL (nixpkgs: `rocmtoolkit-merged` has HIP/HSA, `librccl` is a separate store path). |
| `HIPFIRE_DEVICES` / `HIPFIRE_TP` / `HIPFIRE_TP_USE_RCCL` | Multi-GPU / TP. `HIPFIRE_DEVICES` is the compatibility alias for `hardware.devices`; startup lowers its physical list to ROCr selectors plus matching HIP logical selectors. |
| `HIPFIRE_ALLOW_MIXED_ARCH=1` | Mixed arch pairs |
| `HIPFIRE_PP_LAYERS` / `HIPFIRE_PP_PFLASH` | Pipeline parallel |
Expand Down Expand Up @@ -286,7 +287,7 @@ Copyable user, developer, and retained-PM4 TOML profiles are in
**Do not hand-edit rows below** except by re-running the source scan.
**Generation method:** token scan over visible `*.rs`, `*.py`, and `*.sh`, excluding ignored/generated files.
**Columns:** variable; up to two lexical source paths.
**Count:** 715
**Count:** 735

| Variable | Example source path(s) |
|---|---|
Expand Down Expand Up @@ -895,6 +896,7 @@ Copyable user, developer, and retained-PM4 TOML profiles are in
| `HIPFIRE_QWEN_MOE_FINAL_NORM_RAW` | scripts/test_pr228_spiral_check.sh |
| `HIPFIRE_QWEN_MTP` | crates/hipfire-daemon/src/main.rs, scripts/serve_harness.py |
| `HIPFIRE_QWEN_PROMPT_CACHE` | crates/hipfire-daemon/src/main.rs |
| `HIPFIRE_RCCL_LIB` | crates/hip-bridge/src/rccl.rs |
| `HIPFIRE_RDNA2_VARIANT` | crates/hipfire-cli/src/main.rs, crates/rdna-compute/src/feature_flags.rs |
| `HIPFIRE_RDNA3_HFQ4_LM_HEAD_K2048` | crates/rdna-compute/src/feature_flags.rs |
| `HIPFIRE_RDNA3_HFQ4_MOE_GATE_UP_K2048` | crates/rdna-compute/src/feature_flags.rs |
Expand Down
Loading