feat(rocm): add RDNA3 and RDNA4 runtime foundation - #132
Conversation
- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents - Update pinned_tensor.cpp to compile under both nvcc and hipcc - Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family) - Guard NVIDIA arch checks to return None on ROCm - Skip nvcc version check in _toolchain.py when on ROCm - Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch) - Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm - Add is_rocm() and driver_hip_version() in backend.py - Add rocm-smi fallback in __main__.py for clangd generation - Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds, Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache - Add AMD ROCm classifier in pyproject.toml
|
Draft follow-ups are now available:
All four are opened as Draft PRs against |
Independent test report: RDNA3 / gfx1100 (RX 7900 XT)Ran this branch on an RDNA3 card, since the validation above is gfx1201. Short version: the foundation works on gfx1100, I was able to serve models end to end, and the offload architecture measures very well against llama.cpp on this hardware. Environment
Install
Test suite
|
| engine / backend | decode tok/s |
|---|---|
llama.cpp (Vulkan) -ngl 99 -fa on |
193.5 |
llama.cpp (Vulkan) --cpu-moe |
25.5 |
FreeToken --moe-backend fused |
100.8 |
FreeToken --moe-backend offload |
91.2 |
FreeToken --moe-backend hybrid (forced) |
71.8 |
Two things stand out:
- The offload penalty is the story. llama.cpp loses 86.8% moving experts off the card (193.5 → 25.5). FreeToken loses 9.5% (100.8 → 91.2). In the offloaded regime FreeToken is 3.58× faster than llama.cpp on the same model and card.
ft bench bw's hybrid heuristic is correct here, and I verified it by overriding it. It recommendedoffload(1.65×, under the 2.0 threshold); forcinghybridcosts 21%. Also worth recording: hybrid runs on gfx1100, so that path is functional on ROCm.
Qwen/Qwen3.6-35B-A3B-FP8 (35 GB, does NOT fit 20 GB) — moe_backend='offload', cache_type='hybrid_radix', VRAM 19.90 → 2.02 GiB, ~3 min load: 37.8 tok/s. For scale, my own earlier llama.cpp measurements of Qwen3.6-35B-A3B on this card were 115.9 tok/s for Q4_K_M resident and 28.25 tok/s for Q4_K_M with --n-cpu-moe 32. So FreeToken serves a 75% larger, higher-precision checkpoint 34% faster than llama.cpp manages on the smaller Q4 once it has to offload. Different quantisations, so treat that one as indicative; the gpt-oss-20b table above is exact.
Since "full model-serving validation" is an open follow-up here and #136 says serving is out of scope, I hope this is useful.
One design observation
auto selected offload for gpt-oss-20b even though it fits the card, costing 9.5% for nothing. Per docs/models.md that is intended (dense → fused, MoE → offload always), and fused is documented as "never auto-selected" — but on a card where the MoE fits, auto picks the slower path. A fit check before defaulting to offload might be worth considering.
Host-RAM ceiling, for anyone sizing a box
Offload pins expert memory, so host RAM is the real limit. On 62 GiB total / 45 GiB available, a 35 GB model consumed ~31 GB pinned and dropped available RAM to 14 GB — at which point sshd could no longer fork, while the box stayed up and port 22 stayed open. Extrapolating, ~50 GB models need ~46 GB pinned and are out of reach on this machine. Not a bug, just a sizing note.
Not tested (deliberately)
The GGUF path. On #132 alone it fails in kernel/gguf.py: --expt-relaxed-constexpr and -ccbin are passed unconditionally in extra_cuda_cflags and reach hipcc/clang++, which reject them. After locally gating those on torch.version.hip the next failure is thrust/complex.h not found — the modular ROCm SDK wheels in that image ship no rocThrust. Both look like exactly what #136 covers, and #136 depends on #133 which I did not apply, so I did not pursue it and am not filing anything for it.
Happy to re-run any of this, or to test the follow-up PRs on gfx1100 if a second architecture is useful to you. Thanks for doing this work — the RDNA3 scope in #132 is what made it worth trying. (and I Hope i did this correctly! its my first comment!)
|
Hi @Ewwgoblins Thank you for your detailed report, it was very helpful. |
4 participants
Summary
gfx1100-gfx1103) and RDNA4 (gfx1200,gfx1201);/opt/rocmlayouts and the modular ROCm SDK shipped by the official ROCm 7.14 PyTorch image;gcnArchName, with explicit environment overrides for cross compilation.Why this targets
maindirectlyThis PR is a standalone, current-
mainintegration. It incorporates the useful work from #23 and the RDNA3 runtime follow-up at bouclem#1, then adds the RDNA4 and ROCm 7.14 changes. It therefore does not require either external feature branch to merge first.This intentionally overlaps those open PRs so maintainers can review and merge a complete, hardware-tested ROCm foundation without being blocked by a cross-fork base chain.
Attribution
7caa62d(upstream source commit27c0977).af67560through62bb962(upstream follow-up head4e11ed0).Thank you to both contributors for establishing and validating the earlier ROCm paths.
Validation
Hardware and software:
gfx1201)2.11.0+rocm7.14.07.14.608503.7.1Results:
python -m pip install -e . --no-build-isolation --no-deps: PASS; both native extensions compiled and loaded;48 passed;python -m pip check: PASS;git diff --check: PASS.Compatibility
ROCm-specific code is guarded by HIP/ROCm detection and existing CUDA paths are retained. Physical NVIDIA regression testing is not claimed; upstream CUDA CI is requested before merge.
Follow-ups
Kept out of this foundation PR for separate review: