diff --git a/CHANGELOG.md b/CHANGELOG.md index 84461c999..754c46f3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Changed + +- AMD ROCm is documented as an officially supported backend since 0.8, on Linux and on Windows x86_64, rather than experimental. It remains a source build, and the validated configuration and the open gaps are unchanged; see the platform matrix and the Linux and Windows ROCm sections in [Installation](docs/installation.md). + ## [v0.8.1] - 2026-11-03 ### Added @@ -73,7 +77,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - CUDA server prompt lookup pauses rows whose recent verify rounds emit fewer than four tokens, resumes after seven shadow tokens confirm copying, and re-primes proposal-free ticks sooner (#2264). An isolated zero-accept miss after strong copy evidence receives one round of grace; partial low-yield matches do not. The CLI governor and non-CUDA governor policy are unchanged. The paged context cap is 4608 total tokens, allowing the existing 4096-target copy workload and its 256-token continuation while retaining the long-context guard. The final quiet GB10 measurement passes every criterion: chat −0.275%, short-copy +149–159%, and long-copy 1.52–2.82×. - `mlxcel serve` and `mlxcel-server` accept `--draft-kind prompt-lookup` with no drafter checkpoint: each eligible request runs prompt-lookup verify rounds inside the batch scheduler, at most one per tick, while proposal-free ticks stay on the lookahead pipeline (#2255). `--draft-block-size` sets the largest proposal and the new `--prompt-lookup-max-batch` (default 2) is the largest active decode batch at which verify rounds run; a paged sequence stops verifying past 4608 total tokens of context (#2264 adjusts the original 4096 cutoff for tokenization and continuation headroom). On GB10 the flag gains 53 to 67 % at B=1 on replies that copy the prompt and costs 3.5 % on chat replies that copy little (#2264 tracks gating those rounds off). Finished requests report the rounds in `timings` with `draft_kind: "prompt-lookup"`. `scripts/bench_serving_concurrency.py` gains `--prompt-style plain|copy`. `mlxcel generate --prompt-lookup` runs the same `Engine::verify_round`. -- **Experimental AMD GPU (ROCm) backend on Linux** (#1818). `--features rocm` builds against an MLX ROCm backend kept as whole-file overlays in mlxcelverse, copied only when `MLX_BUILD_ROCM` is on, so Metal and CUDA builds compile no ROCm-modified file. Affine, mxfp4 and mxfp8 checkpoints are native; ModelOpt NVFP4 converts to affine 4-bit at load through the new backend quantization capability table (#2030). HIP ports cover FFT through hipFFT, Hadamard, SearchSorted, BitLinear, the Gumbel-max and rejection samplers, the SSM update step, the Mamba1 scan, the fused MoE decode kernels, xIELU, the add3 norm, paged attention, and fused add-RMSNorm and RoPE-append (#2098, #2099, #2100, #2101, #2103, #2107, #2189). A self-hosted gfx1151 runner builds, links and generates in CI (#1989), and `make verify-rocm` runs the gate locally (#2008). See `docs/installation.md#linux-with-amd-rocm-experimental`. +- **Experimental AMD GPU (ROCm) backend on Linux** (#1818). `--features rocm` builds against an MLX ROCm backend kept as whole-file overlays in mlxcelverse, copied only when `MLX_BUILD_ROCM` is on, so Metal and CUDA builds compile no ROCm-modified file. Affine, mxfp4 and mxfp8 checkpoints are native; ModelOpt NVFP4 converts to affine 4-bit at load through the new backend quantization capability table (#2030). HIP ports cover FFT through hipFFT, Hadamard, SearchSorted, BitLinear, the Gumbel-max and rejection samplers, the SSM update step, the Mamba1 scan, the fused MoE decode kernels, xIELU, the add3 norm, paged attention, and fused add-RMSNorm and RoPE-append (#2098, #2099, #2100, #2101, #2103, #2107, #2189). A self-hosted gfx1151 runner builds, links and generates in CI (#1989), and `make verify-rocm` runs the gate locally (#2008). See `docs/installation.md#linux-with-amd-rocm`. - **Bundled WebUI** (`--webui` on `mlxcel serve` and `mlxcel-server`, compiled in by default through the `webui` Cargo feature) (#1851, #1857, #1860, #1865, #1871, #1891, #1894, #2324). It serves a model library with download consent and progress, load, unload and managed-cache delete; bounded streaming local chat; a runtime Activity page; and scoped Settings with next-load profiles. Model-free startup lets the server start with no model loaded. User guide in `docs/webui/README.md` and `docs/webui/user-guide.ko.md`. - **`mlxcel generate-image` and `/v1/images/generations`, `/v1/images/edits` behind `--image-model`** (#2497). This is the shared runtime: checkpoint family probe, flow-match schedulers, FLUX.2 VAE, Z-Image DiT blocks, Qwen3 hidden-state conditioner and PNG writer. No model family is registered yet, so every image checkpoint is recognized and then refused (see Known Issues in the release notes and `docs/image-generation.md`). - **Nemotron VoiceChat** offline duplex inference (#2037), a cache-aware online session with frame events (#2039), and a `/v1/realtime` WebSocket on both server entry points with base64 PCM16 audio in both directions (#2041). The upgrade enforces the CORS origin policy (#2047), and streaming frame time is under 80 ms (#2054). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3a9245b41..9a1d7dcae 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -196,7 +196,8 @@ Minimum: - Rust **1.97+** (project uses edition 2024) - CMake available on `PATH` on both platforms (required by the `mlxcel-core` and `sentencepiece-sys` build scripts) - macOS: Apple Silicon Mac on macOS Sonoma+; Xcode Command Line Tools -- Linux: CUDA 13+ toolchain, OpenBLAS, LAPACK, `pkg-config` (see [`docs/installation.md`](docs/installation.md) for the package list); for the experimental `rocm` feature, a ROCm installation instead of CUDA (see [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental)) +- Linux: CUDA 13+ toolchain, OpenBLAS, LAPACK, `pkg-config` (see [`docs/installation.md`](docs/installation.md) for the package list); for the `rocm` feature, a ROCm installation instead of CUDA (see [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm)) +- Windows: Visual Studio 2022+ with the "Desktop development with C++" workload and the `x86_64-pc-windows-msvc` Rust target; for `cuda`, a CUDA 13 toolkit with cuDNN 9 (see [Windows with NVIDIA CUDA](docs/installation.md#windows-with-nvidia-cuda-experimental)); for `rocm`, a ROCm 10.x SDK (see [Windows with AMD ROCm](docs/installation.md#windows-with-amd-rocm)) Recommended local tooling: diff --git a/README.md b/README.md index 3489bf7db..19de1c716 100644 --- a/README.md +++ b/README.md @@ -4,19 +4,19 @@ [![Latest Release](https://img.shields.io/github/v/release/lablup/mlxcel)](https://github.com/lablup/mlxcel/releases/latest) [![CI](https://github.com/lablup/mlxcel/actions/workflows/ci.yml/badge.svg)](https://github.com/lablup/mlxcel/actions/workflows/ci.yml) -High-performance LLM, VLM, embedding, reranking, image generation, and audio inference for Apple Silicon and NVIDIA CUDA systems, with experimental AMD ROCm support on Linux. The CLI and server are implemented in Rust and execute MLX SafeTensors checkpoints through native MLX C++ bindings, without Python in the request path or a checkpoint-conversion step. +High-performance LLM, VLM, embedding, reranking, image generation, and audio inference for Apple Silicon and NVIDIA CUDA systems, with AMD ROCm support on Linux and Windows. The CLI and server are implemented in Rust and execute MLX SafeTensors checkpoints through native MLX C++ bindings, without Python in the request path or a checkpoint-conversion step. ## Overview `mlxcel` is both a local inference CLI and a production-oriented model server. It runs text generation, multimodal input, embeddings, reranking, token classification, image generation and editing, speech transcription and synthesis, continuous batching, prompt caching, speculative decoding, and distributed inference in one native runtime. -Apple Silicon is the primary target. Linux/CUDA is supported as a secondary target. AMD GPUs on Linux (ROCm) and native Windows with NVIDIA CUDA are experimental, source-build-only targets, and an opt-in OpenXLA/IREE backend is available as an alpha development path. +Apple Silicon is the primary target. Linux/CUDA is supported as a secondary target. AMD GPUs on Linux and Windows (ROCm) are supported, source-build-only targets; native Windows with NVIDIA CUDA is experimental; and an opt-in OpenXLA/IREE backend is available as an alpha development path. ## Current main highlights The current `main` branch is v0.8.1 plus unreleased work. Install from source to use features that have not reached the latest tagged release; see the [changelog](CHANGELOG.md) for the release-by-release record. -- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the samplers, the SSM update step, the Mamba1 scan, the decode-MoE kernels, xIELU, the add3 LayerNorm, and the fused add-RMSNorm and RoPE + append, which are on by default on ROCm) run natively and the rest fall back to MLX graphs. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps. +- **AMD GPU (ROCm) support, official since 0.8.** `--features rocm` builds mlxcel on Linux and on Windows x86_64 against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the samplers, the SSM update step, the Mamba1 scan, the decode-MoE kernels, xIELU, the add3 LayerNorm, and the fused add-RMSNorm and RoPE + append, which are on by default on ROCm) run natively and the rest fall back to MLX graphs. The backend is a source build; see [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm) and [Windows with AMD ROCm](docs/installation.md#windows-with-amd-rocm) for the validated configurations and the open gaps. - **Bundled WebUI.** `--webui` serves a browser interface from `mlxcel serve` or `mlxcel-server`: a model library with download, load, unload, and delete; local streaming chat; runtime activity; and scoped settings with next-load profiles. See the [WebUI guide](docs/webui/README.md). - **Verified `llama-server` compatibility.** A frozen b10621 manifest classifies all 376 pinned options, routes, and native request fields, with no deferred entries. Native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router, and LoRA surfaces are implemented or explicitly classified instead of being silently ignored. - **OpenAI, Anthropic, and Vertex-compatible serving.** Chat Completions, Completions, Responses, Embeddings, Reranking, Audio, and Anthropic Messages are available alongside the native `llama-server` routes. Optional Vertex AI custom-container routing is supported through the standard `AIP_*` variables. @@ -267,7 +267,7 @@ Linux/NVIDIA builds require the CUDA toolkit and MLX's CUDA system dependencies: cargo build --release --features cuda ``` -Linux/AMD builds are experimental and need a ROCm installation with its HIP, rocBLAS, rocWMMA, hipBLASLt, and hipRTC CMake packages: +Linux/AMD builds are supported and need a ROCm installation with its HIP, rocBLAS, rocWMMA, hipBLASLt, and hipRTC CMake packages: ```bash cargo build --release --features rocm @@ -275,7 +275,9 @@ cargo build --release --features rocm Native Windows builds with NVIDIA CUDA are experimental and single-node: run `cargo build --release --features cuda` from a Visual Studio developer prompt with `CUDA_PATH` pointing at a CUDA 13 toolkit that has cuDNN 9. See [Windows with NVIDIA CUDA](docs/installation.md#windows-with-nvidia-cuda-experimental). -A plain Linux build has no GPU feature and runs on the CPU, which is not a validated release target and is much slower. See [Installation](docs/installation.md) for the complete prerequisite, CUDA and ROCm architecture, runtime-header, and packaging matrix, and [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the current ROCm status. +Native Windows builds with AMD ROCm are supported since 0.8 and single-node: run `cargo build --release --features rocm` from the x64 Visual Studio developer prompt with `ROCM_PATH` pointing at a ROCm 10.x SDK. See [Windows with AMD ROCm](docs/installation.md#windows-with-amd-rocm) for the SDK layout and the validated configuration. + +A plain Linux build has no GPU feature and runs on the CPU, which is not a validated release target and is much slower. See [Installation](docs/installation.md) for the complete prerequisite, CUDA and ROCm architecture, runtime-header, and packaging matrix, and [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm) for the ROCm build and validated configuration. ## Performance @@ -315,7 +317,9 @@ The split in that table is architectural rather than arbitrary. Across the whole The VLM figures also moved between mlxcel releases, and unevenly. Re-sweeping that table at one commit left the median row unchanged at 1.00x against its previous reading, with 9 of 71 rows moving more than 10%: `mistral-small-4-119b-2603-4bit` 5.30x, `moondream2` 4.19x, `qwen3-vl-30b-a3b` 2.61x, `qwen3-omni-30b-a3b` 2.57x, then five between 1.16x and 1.46x. The gains come from dtype fixes that reach specific paths, so the aggregate moved from 98% to 105% while most individual rows stayed where they were. Read the per-model tables rather than the median if a particular checkpoint matters. -Per-host sweeps are more current than the combined report. M5 Max and M1 Ultra are both measured on the v0.7.0 commits (2026-09-06 through 2026-09-09) at the same MLX pin and the same pp512/tg128 shape, so cross-host ratios between those two are taken on one version. The per-host records label those runs `0.7.0-beta.1`, which is the crate version string the tree carried until the release commit. Coverage differs by host rather than version: M5 Max carries text, VLM, speculative decoding, batched serving and the embedding and rerank ladder, while M1 Ultra carries text, VLM, speculative decoding and embeddings. GB10 is still on v0.4.0-rc.1 and the older measurement shape, so any ratio involving it mixes versions. Focused reports cover paged attention, MoE, KV compression, embeddings, and speculative decoding. Read [Benchmark results](docs/benchmark_results/model_tests.md), the [benchmark report](docs/benchmark_results/benchmark-report.md), and the [methodology](docs/benchmarks.md) before comparing hosts or planning capacity. +GB10 has no same-host `mlx-lm` reference, so its record is version-over-version. The 0.8.1 sweep (text on 2026-11-03, VLM on 2026-11-04, pp512/tg128) paired 111 text checkpoints with the 0.4.0-rc.1 record at a **1.026 median decode ratio**, with the 10th percentile at 0.928 and the 90th at 1.364. The 52 VLM pairs read **1.081 median** and 1.012 at the 10th percentile, with no row below the previous record. Prefill is not comparable across those two sweeps because the harness moved to the 512-token synthetic prompt. The same pass fixed and re-measured the three checkpoints that had started failing: the BitNet pair now reads 845.05 and 843.60 tok/s prefill at 140.08 and 191.67 tok/s decode, and DiffusionGemma runs its model-owned diffusion loop at 797.61 prefill and 6.86 decode tok/s. It also added the first embedding and rerank ladder on this host (102 cells, no failures; 9 of 11 text embedders faster than their 2026-08-26 cells) and the image generation ladder (21 text-to-image checkpoints plus 3 edit-only; 4.61 s/image median for generation at 512x512 and 8 steps, 1.83 s fastest). + +Per-host sweeps are more current than the combined report. M5 Max and M1 Ultra are both measured on the v0.7.0 commits (2026-09-06 through 2026-09-09) at the same MLX pin and the same pp512/tg128 shape, so cross-host ratios between those two are taken on one version. The per-host records label those runs `0.7.0-beta.1`, which is the crate version string the tree carried until the release commit. GB10 is measured at 0.8.1 on the same pp512/tg128 shape, so a ratio between GB10 and either Apple Silicon host still mixes versions; within GB10, the decode deltas above are version-over-version. Coverage differs by host rather than version: M5 Max carries text, VLM, speculative decoding, batched serving and the embedding and rerank ladder, M1 Ultra carries text, VLM, speculative decoding and embeddings, and GB10 carries text, VLM, embeddings and rerank, and image generation and editing. Focused reports cover paged attention, MoE, KV compression, embeddings, and speculative decoding. Read [Benchmark results](docs/benchmark_results/model_tests.md), the [benchmark report](docs/benchmark_results/benchmark-report.md), and the [methodology](docs/benchmarks.md) before comparing hosts or planning capacity. MTP performance depends on the target/drafter pair, prompt, verify width, and hardware generation. With the v0.6.0 exactness policy, Gemma 4 12B plus its 4-bit assistant measured 93.2 tok/s against 43.8 tok/s classic decode on M5 Max (**2.13x**, greedy output byte-identical). The faster inexact kernel measured 120.4 tok/s but is not the default. How much that is worth depends heavily on the prompt: on the same host and release, the `speculative_bench --sweep` default 14-token prompt measures the same pairing at 70.9 tok/s against a 45.3 tok/s baseline (**1.57x** at K=4, 35.0% acceptance), against the 2.13x above on a code prompt. The runtime measures exactness and profitability instead of treating either speedup as universal; see [Speculative-decoding acceptance](docs/speculative-acceptance.md), [MTP policy API](docs/mtp-policy-api.md), and the benchmark records under `docs/benchmark_results/`. diff --git a/docs/README.md b/docs/README.md index 3b407b348..805793f1a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,7 +14,7 @@ MkDocs pages are added later. Current GitHub-facing docs: -1. `installation.md` - platform prerequisites and build flags for macOS, Linux with CUDA, Linux with AMD ROCm (experimental), and Windows with NVIDIA CUDA (experimental). +1. `installation.md` - platform prerequisites and build flags for macOS, Linux with CUDA, Linux with AMD ROCm, Windows with AMD ROCm, and Windows with NVIDIA CUDA (experimental). 2. `environment-variables.md` - `MLXCEL_*` runtime, build, downloader, cache, and diagnostic knobs. 3. `benchmarks.md` - benchmark methodology and the requirements for future raw result tables. 4. `supported-models.md` - maintained architecture/checkpoint support matrix. diff --git a/docs/architecture.md b/docs/architecture.md index 51130099e..dcf65fa79 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -204,15 +204,18 @@ Release builds use `panic = "unwind"` (issue #375), so the deliberate audio work `MLXCEL_PAGED_ATTENTION_V2=1`), and Gumbel-max sampling. - CUDA kernel behavior is mostly inherited from MLX; `mlxcel` passes the CUDA architecture list through `MLX_CUDA_ARCHITECTURES` at build time. -- ROCm (experimental, `--features rocm`) runs the vendored ROCm backend from - `src/lib/mlxcelverse/overlays/rocm/`. The build passes the HIP `gfx` targets - through `MLX_ROCM_ARCHITECTURES` (default: the targets `rocminfo` reports), - links the backend's kernel archive and the ROCm libraries, and embeds - `$ROCM_PATH/lib` as an rpath. The fused launchers above choose CUDA whenever +- ROCm (supported since 0.8, `--features rocm`) runs the vendored ROCm backend from + `src/lib/mlxcelverse/overlays/rocm/` on Linux and on Windows x86_64 MSVC. On + Linux the build passes the HIP `gfx` targets through `MLX_ROCM_ARCHITECTURES` + (default: the targets `rocminfo` reports), links the backend's kernel archive + and the ROCm libraries, and embeds `$ROCM_PATH/lib` as an rpath. The Windows + build links the SDK import libraries and delay-loads the ROCm DLLs from one + relocatable runtime directory. The fused launchers above choose CUDA whenever Metal is unavailable, so on ROCm they take their MLX graph fallbacks where one exists; routing them by backend kind is lablup/mlxcel#1803. See - [Linux with AMD ROCm](installation.md#linux-with-amd-rocm-experimental) for - status and known gaps. + [Linux with AMD ROCm](installation.md#linux-with-amd-rocm) and + [Windows with AMD ROCm](installation.md#windows-with-amd-rocm) for status and + known gaps. ## Distributed and multi-device diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 84e1a755e..e31d83a15 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -62,21 +62,22 @@ with its conclusions in plain text next to it. ## Current result snapshot Keep public result summaries in a single place so aggregate numbers do not drift -between documents. The current Apple Silicon benchmark report is: +between documents. The current benchmark reports are: - [Benchmark Report - 2026-05-19](benchmark_results/benchmark-report.md) +- [Model compatibility and performance on GB10 at 0.8.1](benchmark_results/model_tests_gb10.md): the full text and VLM sweep at pp512/tg128 (2026-11-03 and 2026-11-04), with the modality ladders in the reports below. - [Native batched decode on M3 Ultra — 2026-10-27](benchmark_results/native-batched-decode-m3ultra-2026-10-27.md): 21 checkpoints, 23 configurations, with raw traces and serving evidence. - [CUDA quantized matmul batch fold on RTX Spark N1X, 2026-10-31](benchmark_results/cuda-qmm-fold-batch-n1x-2026-10-31.md) - [MoE expert overlap in batched decode on RTX Spark N1X, 2026-10-31](benchmark_results/moe-distinct-experts-n1x-2026-10-31.md) - [Few-row quantized matmul for batched decode on M1 Ultra, 2026-10-30](benchmark_results/few-row-qmm-batched-decode-m1ultra-2026-10-30.md): Llama 3.1 8B 4-bit 8-row decode step 56.5 ms to 39 ms, B=8/B=1 serving ratio 1.35x to 1.95x. That combined report is the cross-hardware surface, but it is older than the -per-host tables it summarizes. The newest full sweep is M5 Max on mlxcel 0.6.0 -(2026-09-03/04), covering text, VLM, speculative and batched serving, in -[model_tests_m5max.md](benchmark_results/model_tests_m5max.md); M1 Ultra and -GB10 are still at 0.4.0-rc.1, so cross-host ratios currently mix versions. Take -per-host numbers from the per-hardware tables and the combined report only for -the comparison shape, until the other hosts are re-measured. +per-host tables it summarizes. The newest full sweeps are M5 Max on mlxcel 0.6.0 +(text, VLM, speculative and batched serving) and GB10 on 0.8.1 (text, VLM, +embeddings and rerank, and image generation and editing); M1 Ultra is still at +0.4.0-rc.1, so cross-host ratios currently mix versions. Take per-host numbers +from the per-hardware tables and the combined report only for the comparison +shape, until the remaining host is re-measured. Use those tables for release notes, README updates, or capacity planning. This page should stay focused on methodology, required metadata, and caveats. @@ -210,6 +211,11 @@ python3 scripts/bench_serving_concurrency.py --port \ python3 scripts/bench_embeddings.py --bin target/release/mlxcel-server \ --out benchmarks/{backend}_{hw}_embeddings_{date}.csv --repeats 5 --port 18091 +# Image generation and edit ladder. One image-only server per checkpoint; +# writes its own CSV. Add --task edit --image for the edit route. +python3 scripts/bench_imagegen.py --bin target/release/mlxcel-server \ + --out benchmarks/{backend}_{hw}_imagegen_{date}.csv --repeats 3 --port 18092 + # Continuous-batching sweep over every model dir. Takes ONE positional # argument, the output log path, and no flags. ./scripts/bench_all_models.sh diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 97fe2b209..5d780615d 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -212,13 +212,13 @@ CUDA builds also use non-`MLXCEL_*` variables such as `CUDA_HOME` and Native Windows CUDA builds read `CUDA_PATH` (`CUDA_HOME` is honored ahead of it) and `CUDNN_PATH`, the cuDNN root, only when cuDNN is not unpacked into the CUDA toolkit directory; see [Installation](installation.md#windows-with-nvidia-cuda-experimental). On Windows these are build-time inputs: the binaries load the CUDA, cuDNN and NVRTC DLLs from one runtime directory (`MLXCEL_CUDA_RUNTIME_DIR`, else the executable's directory when it holds the CUDA runtime DLL the binary imports, else the toolkit and cuDNN `bin\` recorded at build time), and NVRTC takes its CUDA headers from `CUDA_HOME`, else a runtime bundle's `include` (which outranks `CUDA_PATH`, set machine-wide by the CUDA installer), else `CUDA_PATH`, else the toolkit `include` recorded at build time; see [Installation](installation.md#cuda-runtime-directory-and-bundle). ROCm builds (`--features rocm`) read `ROCM_PATH` (default `/opt/rocm`) and `MLX_ROCM_ARCHITECTURES` (the `gfx` targets, `;`-separated HIP target IDs such as `gfx1151`, `gfx11-generic` or `gfx90a:xnack+`, anything else fails the configure step; default detected with `rocminfo`); see -[Installation](installation.md#linux-with-amd-rocm-experimental). -Native Windows ROCm builds read `ROCM_PATH`, then `HIP_PATH` when `ROCM_PATH` is unset or empty (there is no default; the build fails naming both), and detect the `gfx` targets with the SDK's `amdgpu-arch.exe` (`bin\` or `lib\llvm\bin\`) instead of `rocminfo`; `MLX_ROCM_ARCHITECTURES` still overrides detection. On Windows `ROCM_PATH` and `HIP_PATH` are needed only at build time: the binaries load the ROCm DLLs from one runtime directory (`MLXCEL_ROCM_RUNTIME_DIR`, else the executable's directory when it holds the HIP runtime, else the SDK `bin` recorded at build time), and the backend's kernel compiler takes its headers from `ROCM_HOME` or `ROCM_PATH`, else a runtime bundle's `include` (which outranks `HIP_PATH`), else `HIP_PATH`, else the SDK `include` beside or recorded with the build; distributable Windows builds set `MLX_ROCM_ARCHITECTURES=gfx1100;gfx1101;gfx1102;gfx1150;gfx1151;gfx1200;gfx1201`; see [Installation](installation.md#windows-with-amd-rocm-experimental). The variables the ROCm +[Installation](installation.md#linux-with-amd-rocm). +Native Windows ROCm builds read `ROCM_PATH`, then `HIP_PATH` when `ROCM_PATH` is unset or empty (there is no default; the build fails naming both), and detect the `gfx` targets with the SDK's `amdgpu-arch.exe` (`bin\` or `lib\llvm\bin\`) instead of `rocminfo`; `MLX_ROCM_ARCHITECTURES` still overrides detection. On Windows `ROCM_PATH` and `HIP_PATH` are needed only at build time: the binaries load the ROCm DLLs from one runtime directory (`MLXCEL_ROCM_RUNTIME_DIR`, else the executable's directory when it holds the HIP runtime, else the SDK `bin` recorded at build time), and the backend's kernel compiler takes its headers from `ROCM_HOME` or `ROCM_PATH`, else a runtime bundle's `include` (which outranks `HIP_PATH`), else `HIP_PATH`, else the SDK `include` beside or recorded with the build; distributable Windows builds set `MLX_ROCM_ARCHITECTURES=gfx1100;gfx1101;gfx1102;gfx1150;gfx1151;gfx1200;gfx1201`; see [Installation](installation.md#windows-with-amd-rocm). The variables the ROCm backend reads at run time are in [ROCm backend variables](#rocm-backend-variables). ## ROCm backend variables -These variables are read by the ROCm backend of a `--features rocm` build. The code that reads them is under `src/lib/mlxcelverse/overlays/rocm/mlx/backend/rocm/` (the file is named where a row is not obvious). Unless a row says "every call", a variable is read once per process, the first time the code path that uses it runs, so it cannot be changed afterwards. "Presence" variables take effect for any value, including `0` and the empty string. Parsers that look at the first character accept exactly the characters listed. The variables are tuning, A/B and bisecting switches, not a stable public API, and none is recommended for deployment. How to build for ROCm and what the backend supports is in [Installation](installation.md#linux-with-amd-rocm-experimental). +These variables are read by the ROCm backend of a `--features rocm` build. The code that reads them is under `src/lib/mlxcelverse/overlays/rocm/mlx/backend/rocm/` (the file is named where a row is not obvious). Unless a row says "every call", a variable is read once per process, the first time the code path that uses it runs, so it cannot be changed afterwards. "Presence" variables take effect for any value, including `0` and the empty string. Parsers that look at the first character accept exactly the characters listed. The variables are tuning, A/B and bisecting switches, not a stable public API, and none is recommended for deployment. How to build for ROCm and what the backend supports is in [Installation](installation.md#linux-with-amd-rocm). The paragraphs just below give the background for the first of these variables: `MLX_ROCM_FFT_CACHE_SIZE`, `MLX_ROCM_MAX_INFLIGHT_MB`, `MLX_ROCM_GPU_WATCHDOG_SECS`, `MLX_ROCM_WMMA_QMM`, `MLX_ROCM_WMMA_QMM_MAX_M`, `MLX_ROCM_QMM_DEQUANT_GEMM` and the integer knobs. After them, every variable has a row in the tables, grouped by the code it controls. @@ -783,7 +783,7 @@ recommended as normal deployment settings. | `MLXCEL_SDPA_FALLBACK_MAX_QUERIES` | non-negative integer | `32` | **CUDA only.** A masked SDPA call with 2 to N query rows over a longer key sequence (the speculative verify shape: a block appended to a KV cache) bypasses cuDNN and takes MLX's own ops fallback (issue #1799). cuDNN caches its execution plan by the exact shapes, and a verify round's key length changes every round, so with cuDNN every such layer class rebuilt a plan on the host each round: about 22 ms per build on GB10, 67 to 76 ms per round on the Laguna DFlash pairing, the whole fixed floor of that round, and the plan cache's lifetime miss counter then aborted the process. The one-row decode step is unaffected (below 256 cache positions it takes the vector kernel; from 256 it takes cuDNN through MLX's own decode canonicalization, or the vector kernel with `MLXCEL_SDPA_DETERMINISTIC=1`); prefill keeps cuDNN (its key length equals its query length, or it has more rows than the bound), except the trailing short chunk of a chunked prefill and a short incremental prefill over a reused prefix-cache prefix, which take the fallback. The fallback's cost grows with the key length (a `[B, heads, q_len, k_len]` score matrix, about 10 ms more per round than cuDNN at block 2 with 350 keys) while the plan build it replaces is constant, so on very long contexts (tens of thousands of keys at block 16, unmeasured) cuDNN could be the cheaper side again. Issue #1820 left this gate's shipped behaviour unchanged: bucketing is off by default, so this gate still claims every array-masked verify block as it did before. It is narrowed only when `MLXCEL_SDPA_PLAN_BUCKET_MAX_QUERIES` is set non-zero, in which case it stops claiming calls whose plan-cache key can be bucketed and covers only a causal-mode block with no array mask, where there is no mask to widen. `0` restores upstream dispatch without a rebuild, which is the kill switch used for the A/B in `docs/benchmark_results/laguna-dflash-verify-cost-gb10-2026-09-11.md`. | | `MLXCEL_PIPELINE_GRANULARITY` | `off`, `layer`, `block:N` | `off` | Inserts layer-boundary async-eval hints for pipeline experiments. | | `MLXCEL_FUSED_ADD_NORM` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fuses a parallel-residual block's `(attn + mlp) + x` with the LayerNorm that consumes it (the next block's input norm, or the final norm) into one kernel (Metal, ROCm), saving one dispatch and one barrier level per layer during decode. Each port copies its own backend's `layer_norm` reduction (MLX's single-row Metal kernel; on ROCm, lablup/mlxcel#2069, the overlay's `layer_norm_kernel`), so its outputs are byte-identical to that backend's unfused add and norm (pinned by `residual_add3_layer_norm_matches_the_unfused_pair`, which on gfx1151 covers f16, bf16 and f32 up to the 6656-wide limit); it falls back to the unfused pair on CUDA and CPU, above a 6656-wide row, or on mixed dtypes. No Cohere2 checkpoint was available on the ROCm host to measure decode. Used by Cohere2: command-r7b 4-bit decode on M1 Ultra +1.1% (eight ABBA pairs, 112.45 to 113.72 tok/s). | -| `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, granitemoehybrid, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On CUDA the kernel pair is NVRTC-compiled once per machine and kept in `MLX_PTX_CACHE_DIR`; before lablup/mlxcel#2492 every process recompiled it inside its first decode step, which on GB10 cost about 1.2 s per run and made short `mlxcel generate` runs read slower with the kernel than without it, although it is faster per token ([results](benchmark_results/fused-moe-jit-cache-gb10-2026-10-22.md)). On the experimental ROCm build the kernel runs through `mx.fast.hip_kernel` since lablup/mlxcel#2065, with the CUDA default for `MLXCEL_FUSED_MOE_MAX_DFF`. It is verified on gfx1151 (wave32) only; on a wave64 part (CDNA) it is untested, and `0` here keeps such a host on `gather_qmm`. | +| `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, granitemoehybrid, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On CUDA the kernel pair is NVRTC-compiled once per machine and kept in `MLX_PTX_CACHE_DIR`; before lablup/mlxcel#2492 every process recompiled it inside its first decode step, which on GB10 cost about 1.2 s per run and made short `mlxcel generate` runs read slower with the kernel than without it, although it is faster per token ([results](benchmark_results/fused-moe-jit-cache-gb10-2026-10-22.md)). On the ROCm build the kernel runs through `mx.fast.hip_kernel` since lablup/mlxcel#2065, with the CUDA default for `MLXCEL_FUSED_MOE_MAX_DFF`. It is verified on gfx1151 (wave32) only; on a wave64 part (CDNA) it is untested, and `0` here keeps such a host on `gather_qmm`. | | `MLXCEL_FUSED_MOE_SGY` | `1`-`32` | `8` (Metal, CUDA) / `2` (ROCm) | Simdgroups (Metal) / warps (CUDA) / wavefronts (ROCm) per threadgroup for the fused decode-MoE kernel; tune per hardware. The output does not depend on it. On gfx1151 `8` made Qwen3-30B-A3B decode slower than `gather_qmm` and `2` faster (lablup/mlxcel#2065), hence the ROCm default. | | `MLXCEL_FUSED_MOE_MAX_DFF` | positive int | `4096` (Metal) / `8192` (CUDA) / `14336` (ROCm) | Expert-intermediate (Dff) upper bound for the fused path; above it the caller falls back to `gather_qmm`. The fused path wins only while `gather_qmm` underutilizes the GPU (small experts), so the break-even is backend-dependent and the default is chosen from the live backend: `4096` on Metal (M1 Ultra tuning) `8192` on CUDA (GB10 re-measured under MLX pin e9463bb, #626; fused wins through Dff 6400 and is break-even at 8192) and `14336` on ROCm (gfx1151 Mixtral-8x7B: 24.9 tok/s fused, 8.2 through `gather_qmm`). An explicit value overrides the default on every backend: lower it to force `gather_qmm` sooner, raise it (e.g. `20000`) to force the fused kernel on larger experts such as mixtral on CUDA (Dff 14336, where it is a slight net loss). Read by every family on the shared `SwitchGLU` fused path, qwen3_moe and qwen3_vl_moe (and the qwen3_omni_moe thinker through it) included since #1884; qwen3_next (and Qwen3.5 and the qwen3_omni_moe talker through it) and gemma4 dispatch their own kernel and do not read it yet. | | `MLXCEL_FUSED_MOE_RELU2` | presence enables | off | Enables the squared-ReLU fused routed-expert path for nemotron-class experts. It is performance-neutral on nemotron-h even though MoE is its largest block because it replaces only the already-efficient routed fc1/fc2 GEMVs; the router, always-on shared expert, and combine remain outside this kernel. Kept for a future squared-ReLU model whose routed-expert slice dominates. The fc1 kernel has Metal and ROCm ports (ROCm since lablup/mlxcel#2069); CUDA has none, so there the flag leaves the default `gather_qmm` routed path in place (lablup/mlxcel#2065). ROCm builds default the branch's rows per block to 2, as `MLXCEL_FUSED_MOE_SGY` describes. On gfx1151 it is performance-neutral too (Nemotron-3-Nano-30B-A3B-4bit decode 75.15 to 75.24 tok/s, medians of three) and, unlike Metal, not byte-identical to the default path: its fc1, relu² and fc2 stay in f32, so its logits sit closer to an f32 reference and greedy text can part from the default at a near-tie ([results](benchmark_results/rocm-metal-only-ports-gfx1151-2026-10-05.md)). | diff --git a/docs/installation.md b/docs/installation.md index ce8bd4e0c..f8ab4a5c8 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -14,10 +14,10 @@ static binaries: platform GPU/runtime libraries are still required. |----------|--------|-----------------------|-------| | macOS on Apple Silicon | primary | `metal,accelerate` | Main development and validation target. | | Linux with NVIDIA CUDA | secondary | `cuda` | Release builds currently target CUDA 13-era systems; other versions depend on MLX/CUDA compatibility. | -| Linux with AMD ROCm | experimental | `rocm` | Source build only. Validated on RDNA 3.5 (`gfx1151`, Strix Halo) with ROCm 10.0 / HIP 7.15; tracked in lablup/mlxcel#1801. See [Linux with AMD ROCm](#linux-with-amd-rocm-experimental). | +| Linux with AMD ROCm | supported (since 0.8) | `rocm` | Source build only. Validated on RDNA 3.5 (`gfx1151`, Strix Halo) with ROCm 10.0 / HIP 7.15; tracked in lablup/mlxcel#1801. See [Linux with AMD ROCm](#linux-with-amd-rocm). | | Linux CPU-only | not a release target | none | May compile in limited configurations, but it is not a useful or validated inference target for this project. | | Windows with NVIDIA CUDA | experimental | `cuda` | Source build only, single node (no tensor or pipeline parallelism). Validated natively on `aarch64-pc-windows-msvc` (NVIDIA RTX Spark N1X); the build glue also covers `x86_64-pc-windows-msvc`. See [Windows with NVIDIA CUDA](#windows-with-nvidia-cuda-experimental). | -| Windows with AMD ROCm | experimental | `rocm` | Source build only, single node. Validated natively on `x86_64-pc-windows-msvc` with a Radeon 8060S (`gfx1151`) and ROCm 10.1.0; runs without the ROCm `bin` directory on `PATH` or `ROCM_PATH` set. See [Windows with AMD ROCm](#windows-with-amd-rocm-experimental). | +| Windows with AMD ROCm | supported (since 0.8) | `rocm` | Source build only, single node. Validated natively on `x86_64-pc-windows-msvc` with a Radeon 8060S (`gfx1151`) and ROCm 10.1.0; runs without the ROCm `bin` directory on `PATH` or `ROCM_PATH` set. See [Windows with AMD ROCm](#windows-with-amd-rocm). | ## Cargo feature flags @@ -33,7 +33,7 @@ opt-in seams, diagnostics harnesses, or test scaffolding. | `metal` | off | Apple Silicon Metal GPU backend (delegates to `mlxcel-core/metal`). Standard on macOS. | | `accelerate` | off | Apple Accelerate CPU BLAS backend (delegates to `mlxcel-core/accelerate`). Standard on macOS. | | `cuda` | off | NVIDIA CUDA GPU backend (delegates to `mlxcel-core/cuda`). Required on NVIDIA hosts; a plain build is CPU-only (see the footgun note below). | -| `rocm` | off | AMD GPU backend on Linux and on Windows x86_64 MSVC (delegates to `mlxcel-core/rocm`), built from the ROCm overlay in `src/lib/mlxcelverse/overlays/rocm/`. Cannot be combined with `cuda` or `metal`. Experimental; see [Linux with AMD ROCm](#linux-with-amd-rocm-experimental) and [Windows with AMD ROCm](#windows-with-amd-rocm-experimental). | +| `rocm` | off | AMD GPU backend on Linux and on Windows x86_64 MSVC (delegates to `mlxcel-core/rocm`), built from the ROCm overlay in `src/lib/mlxcelverse/overlays/rocm/`. Cannot be combined with `cuda` or `metal`. Supported since 0.8; see [Linux with AMD ROCm](#linux-with-amd-rocm) and [Windows with AMD ROCm](#windows-with-amd-rocm). | | `experimental-backend` | off | Reserves the non-MLX compute-backend seam slot (issue #338). Ships no kernels and adds no runtime dispatch; it only compiles the plug-in boundary where a future non-MLX engine (e.g. FuriosaAI RNGD) would implement `ComputeBackend`. `select_backend()` still folds to MLX. | | `xla-backend` | off | OpenXLA / StableHLO backend seam (issue #449, [ADR 0004](adr/0004-compute-backend-session-seam-and-stablehlo-family.md)). Pulls in `mlxcel-xla` and compiles the `Backend::Xla` / `Session::Xla` arms and the `MLXCEL_BACKEND=xla` selector, but no native execution engine: the crate is pure-Rust stubs plus the StableHLO graph emitter, so CI builds it unchanged. | | `xla-iree` | off | `xla-backend` plus real IREE execution (`mlxcel-xla/iree`). Compiles a C shim against a prebuilt IREE runtime and drives the bundled prefill / decode_step graphs. Needs `IREE_DIST` (or the source-build vars below) at build time, so it is a local / opt-in build, not a CI or release default. | @@ -650,7 +650,7 @@ Notes: DLLs run in the mlxcel process, and the headers are compiled into GPU kernels. `C:\Program Files` has that protection; for a folder created directly under `C:\`, remove the inherited access as described for the - [ROCm SDK](#windows-with-amd-rocm-experimental). + [ROCm SDK](#windows-with-amd-rocm). - The bundle does not carry the Visual C++ runtime. `MSVCP140.dll` and `VCRUNTIME140.dll` loaded from System32 on the validated host, which also has Visual Studio, so a machine without them needs the Visual C++ Redistributable. @@ -681,12 +681,12 @@ reports a path it cannot open, enable Win32 long paths or set a short | Prebuilt release archives | None yet. | | CI | Compile-only, no GPU. The hosted `Windows build (compile-only)` job in `ci.yml` runs on `windows-latest` when the Windows build scripts, `build_support/`, the overlays' runtime-directory sources, the bundle packagers or the MLX pin change (or the PR carries `ci:full`, a label the repository does not define yet): the `mlxcel-mlx-pin` unit tests, the `tests/` Python suites, and `cargo check -p mlxcel-core --features rocm` against the ROCm 10.1.0 `gfx1151` TheRock SDK, which builds MLX and its HIP kernels. No CI job builds the `cuda` feature on Windows. The host tests of both overlays' runtime-directory choice run on Linux in `overlay-host-tests`. | -## Windows with AMD ROCm (experimental) +## Windows with AMD ROCm A native `mlxcel.exe` / `mlxcel-server.exe` with the `rocm` feature, built with -MSVC (lablup/mlxcel#2601). It uses the same ROCm overlay as the -[Linux build](#linux-with-amd-rocm-experimental) and the same single-node -limits as the [Windows CUDA build](#current-limits-on-windows). +MSVC (lablup/mlxcel#2601) and supported since 0.8. It uses the same ROCm overlay +as the [Linux build](#linux-with-amd-rocm) and the same single-node limits as +the [Windows CUDA build](#current-limits-on-windows). Validated configuration: `x86_64-pc-windows-msvc` on an AMD Ryzen AI MAX+ 395 (Radeon 8060S, `gfx1151`, with a BIOS UMA carve-out), Windows 11, Adrenalin @@ -1019,11 +1019,9 @@ that would fail at the first matrix multiply on that GPU. feature on Windows without a GPU and runs nothing on AMD hardware; see the CI row in [Current limits on Windows](#current-limits-on-windows). -## Linux with AMD ROCm (experimental) +## Linux with AMD ROCm -**Experimental.** AMD GPU support on Linux was added in lablup/mlxcel#1802 and -is tracked by lablup/mlxcel#1801. It is a source build only: there is no -release artifact and no ROCm CI job running yet, and the gaps listed below are open. +**Supported since 0.8.** AMD GPU support on Linux was added in lablup/mlxcel#1802 and is tracked by lablup/mlxcel#1801. It is a source build only: there is no release artifact, and the optional ROCm CI job is disabled unless the repository sets `ROCM_CI_ENABLED`. The gaps listed below are open. The backend is not part of upstream MLX. mlxcel vendors the ROCm backend from the `rocm-support` branch of @@ -1295,7 +1293,7 @@ the 1.20 headroom factor alone fell short for the dense models. | `mlxcel-server` chat completions | Work for dense and affine MoE checkpoints, streaming and non-streaming; verified with `scripts/server_chat_smoke.sh`. | | Audio (speech to text, text to speech) | Works. The FFT primitive runs on hipFFT; plans are cached up to `MLX_ROCM_FFT_CACHE_SIZE` (default 128, as on CUDA; lablup/mlxcel#1825, #1876); a value that is not a positive integer is ignored with a warning (#2051). | | Wave64 devices (CDNA: MI200 `gfx90a`, MI300 `gfx942`) | Not validated; the only wave64 run so far was one MI300X bring-up (`overlays/rocm/LOCAL_FIXES.md` item 52), and lablup/mlxcel#2699 tracks validation. The fused HIP kernels that reduce across lanes (BitLinear, the two fused MoE kernels and the squared-ReLU one, the Mamba2 SSM update, the Mamba1 scan, the add3 LayerNorm, the fused residual-add RMSNorm, and the paged-attention v1 decode and v2 partial) have only run on 32-lane wavefronts, and their compile-time wavefront guards do nothing with AMD clang 23. So mlxcel selects them only when the device reports a 32-lane wavefront (the hardware value, which `MLX_ROCM_FORCE_WARP_SIZE` does not change); on a 64-lane device those paths use their MLX graph fallbacks, and stderr says so once per process (lablup/mlxcel#2147). The kernels with no cross-lane operation (xIELU, RoPE + KV append, the paged-attention merge, both samplers) are selected at any width. BitLinear has no graph fallback, so BitNet checkpoints are refused at load on a wave64 device. The rocWMMA quantized GEMMs (`qmm_wmma_dense_kernel` and the grouped-prefill gather) assume 32-lane warps too and are likewise held to wave32 devices; before that, every bf16 quantized prefill on gfx942 produced garbage (`!!!!`) unless `MLX_ROCM_WMMA_QMM=0` was set (`overlays/rocm/LOCAL_FIXES.md` item 52). | -| Windows | Experimental; see [Windows with AMD ROCm](#windows-with-amd-rocm-experimental). | +| Windows | Supported since 0.8; see [Windows with AMD ROCm](#windows-with-amd-rocm). | | Multiple GPUs, distributed inference | Not supported. | Decode throughput measured on the tested configuration, for orientation only: @@ -1450,7 +1448,7 @@ make test-fast # Linux / CUDA make test-fast-cuda -# Linux / ROCm (experimental): no make target yet +# Linux / ROCm (supported since 0.8): no make target yet cargo test --profile test-fast --features rocm -- --test-threads=1 # Narrow to a subset while iterating diff --git a/src/lib/mlxcelverse/overlays/rocm/README.md b/src/lib/mlxcelverse/overlays/rocm/README.md index 5ee122a0c..0065bd2bf 100644 --- a/src/lib/mlxcelverse/overlays/rocm/README.md +++ b/src/lib/mlxcelverse/overlays/rocm/README.md @@ -1,6 +1,6 @@ # overlays/rocm: the ROCm part of mlxcelverse -mlxcelverse is the name for everything mlxcel builds on top of upstream MLX: per-backend source overlays and mlxcel's own kernels. This directory is its ROCm overlay. It gives MLX an AMD GPU (ROCm/HIP) backend so that `cargo build --features rocm` produces a GPU build on Linux AMD hosts, and on Windows x86_64 with a ROCm 10 SDK and the MSVC toolchain (experimental, lablup/mlxcel#2601). +mlxcelverse is the name for everything mlxcel builds on top of upstream MLX: per-backend source overlays and mlxcel's own kernels. This directory is its ROCm overlay. It gives MLX an AMD GPU (ROCm/HIP) backend so that `cargo build --features rocm` produces a GPU build on Linux AMD hosts, and on Windows x86_64 with a ROCm 10 SDK and the MSVC toolchain (supported since 0.8; lablup/mlxcel#2601). ## What is in here