feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal - #11137
feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal#11137localai-bot wants to merge 12 commits into
Conversation
f1c213b to
927206e
Compare
|
@localai-org-maint-bot pick this up and verify against vllm.cpp and current state. rebase, fix CI |
64cddf8 to
9fdb13d
Compare
|
Rebased onto current |
|
@localai-org-maint-bot rebase and fix CI issues |
9fdb13d to
b6ad37d
Compare
|
Rebased onto current |
|
@localai-org-maint-bot fix ci |
b6ad37d to
d480f9f
Compare
|
Rebased onto current |
|
Resumed after the fresh Darwin run exposed that the first suppression was ineffective. Pushed |
ffe14a7 to
19ad1db
Compare
|
Resumed the unfinished Darwin failure, rebased onto current |
|
Resumed the remaining Darwin failure and moved the fix to the component that owns the warning policy. Opened mudler/vllm.cpp#23 with an AppleClang-only, source-scoped exception for |
|
Resumed the new Darwin failure. The source-level |
|
Resumed the completed Darwin failure and fixed the warning at its owning layer. The prior CMake source option was ordered before vllm.cpp’s target-level Objective-C++ |
|
@localai-org-maint-bot that was merged |
The darwin vllm-cpp image built the Metal backend with vllm.cpp's native MSL
GEMM only. vllm.cpp also ships an optional MLX provider for the dense GEMM,
kept OFF upstream because it costs a ~19 MB libmlx.dylib plus a ~105 MB
mlx.metallib, on the stated position that it must earn that cost by
measurement.
Measured on an Apple M4 (16 GiB, macOS 26.5.2) it does. One binary, arms
toggled with VT_OP_PROVIDER_DISABLE=mlx so there is no build-difference
confound, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order alternated per rep:
B=1 5.79 vs 3.08 agg tok/s (1.88x) TTFT 3.32 s vs 7.68 s
B=8 25.70 vs 13.69 (1.88x) TTFT 13.95 s vs 34.38 s
B=16 38.65 vs 17.69 (2.19x) TTFT 18.33 s vs 54.48 s
Peak RSS is unchanged (6.65 to 7.50 GB in both arms) and the output is
bit-identical: vllm.cpp's three-way parity test measures mlx-vs-msl NMSE of 0
on all six shapes, and mlx-vs-cpu equal to msl-vs-cpu, against a 5e-4 bar. MLX
serves the dense GEMM alone; paged attention stays vllm.cpp's own kernel
because MLX has no paged-KV primitive. Full disposition, including the
INDICATIVE status and the isolation actually achieved, is in vllm.cpp
docs/BENCHMARKS.md "MLX GEMM provider A/B on Apple M4".
Build: MLX comes from the pinned prebuilt pip wheel (MLX_VERSION, default
0.29.3) into a venv under the backend dir. Building MLX from source needs
`xcrun metal`, i.e. a full Xcode the macOS runners do not have, while the wheel
ships include/, lib/libmlx.dylib and the compiled metallib ready to link. The
install is a stamp FILE rather than a phony target, because a phony
prerequisite is always newer than libvllm and would re-link it every
invocation. VLLM_CPP_MLX=off restores the previous Metal build.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/. Three things this had to get right, each verified on the M4
before it was written rather than after:
1. libvllm.dylib links @rpath/libmlx.dylib and its build-time LC_RPATH points
inside the build venv, a path no user has. Every build rpath is deleted
and replaced with @loader_path/lib.
2. MLX loads its metallib from beside its OWN dylib, so both files must land
in the same directory or every Metal op fails with "Failed to load the
default metallib".
3. install_name_tool invalidates the code signature and macOS refuses to load
an arm64 image with a stale one, so the patched library is re-signed
ad-hoc.
Verified end to end on the M4 by building through this Makefile and running the
packaged artifact: `DYLD_PRINT_LIBRARIES` resolves libmlx from package/lib/,
`codesign -v` passes, no build-venv path survives in the load commands, and a
real generation runs with the provider selected (op=65 selected=mlx) and zero
metallib failures. A missing rpath now fails the build instead of the user's
first inference.
Cost: the darwin vllm-cpp image grows by about 124 MB.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was correct when taken and is now stale: vllm.cpp's own Metal kernels have improved several-fold since, through mma prefill attention, a vectorised decode V accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a simdgroup-per-row softmax. The native path MLX was compared against no longer exists. Re-measured on the same Apple M4, in the same binary, with the arms toggled by VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128: MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s Shipping the previous default would have halved Apple Silicon throughput. MLX's steel GEMM is still about 20% faster than ours in isolation, but the provider pays a per-op mx::eval synchronisation plus an output memcpy, because it cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead leaves a 2% gain; on decode, where the same synchronisation is paid once per matmul per token, it costs 46%. The option is kept for prefill-dominated workloads, where the margin is small but real. The README section is rewritten rather than patched: it previously presented the stale table as the reason for the default, so leaving it in place would have made the new default look arbitrary. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on. These two must move together, which is why they are one commit. Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against 602 for the native path, but loses decode badly because the provider pays an mx::eval synchronisation and an output memcpy on every call while decode makes about 112 calls per token. Ungated it does both; gated it does only the good half. Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128: MLX gated to prefill (pin >= 89c46aeb) TTFT 524.5 ms 24.40 tok/s, 99.1% of MLX-LM MLX ungated (older pins) TTFT 537 ms 12.7 tok/s MLX off TTFT 602 ms 23.9 tok/s This branch briefly defaulted the provider off, which was the correct call for an ungated provider at the old pin. The gate is what makes on correct again, so the pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while leaving MLX on would select the middle row and roughly halve throughput. Both the Makefile comment and the README state that dependency explicitly. The bump also brings six Metal kernels landed upstream since the old pin — mma prefill attention, a vectorised decode V accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to 96.4% of MLX-LM on their own. One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native kernel, so an MLX build produces a different greedy sequence than a non-MLX build. That is a property of the provider rather than of the gate and predates this packaging. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks, MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was an outlier and averaging it in overstated us by roughly 1.5 points. Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms against MLX-LM's 532.6, so we remain about 1.5% faster there. Nothing else changes. MLX still wins prefill and loses decode, the shape gate is still the right disposition, and the pin and the flag are still coupled. The gate is worth about 1.7 points over the MLX-off build rather than 2.7. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The previous pin was a merge commit from the experimental C ABI v9 branch. Pin the same MLX prefill gate on upstream main so the backend build does not pull unrelated ABI v9 work into every platform variant. Assisted-by: Codex:gpt-5 [systematic-debugging]
Keep the current master pin when enabling MLX so every backend variant builds against the known-good vllm.cpp revision. Suppress Apple clang’s GNU constant-folding diagnostic for Objective-C++ Metal compilation only, since upstream treats warnings as errors. Assisted-by: Codex:gpt-5 [systematic-debugging]
MLX 0.29.3 headers trigger Apple clang's gnu-folding-constant diagnostic in the Objective-C++ provider. Keep the diagnostic visible while exempting only it from vllm.cpp's global warnings-as-errors policy. Assisted-by: Codex:gpt-5 [systematic-debugging]
Target-level Objective-C++ -Werror is appended after the directory flags, so a no-error demotion is re-promoted. Disable this single warning for the MLX header while keeping every other warning fatal. Assisted-by: Codex:gpt-5 [systematic-debugging]
Move the AppleClang warning exception into vllm.cpp where its target warning policy is defined, and pin LocalAI to that source-scoped fix. Assisted-by: Codex:gpt-5
The source-scoped no-error flag was overridden by the target warning policy. Pin the companion vllm.cpp change that disables only the MLX header diagnostic for its Objective-C++ translation unit. Assisted-by: Codex:gpt-5
Pin the companion vllm.cpp correction that scopes the AppleClang folding warning suppression inside the MLX translation unit, after command-line warning policy. Assisted-by: Codex:gpt-5 [systematic-debugging]
Advance the MLX-enabled backend to the vllm.cpp revision already validated by the dependency update branch. This includes the feature guards and AppleClang pragma boundary needed by the Darwin build. Assisted-by: Codex:gpt-5 [systematic-debugging]
ca26d72 to
5f9f886
Compare
|
Resumed after the upstream merge and pushed |
3 participants
What
Builds the darwin
vllm-cppimage with vllm.cpp's optional MLX GEMM provider and vendors its runtime into the backend package.vllm.cpp keeps
VLLM_CPP_MLXOFF by default because it costs a ~19 MBlibmlx.dylibplus a ~105 MBmlx.metallib, on the explicit position that it must earn that cost by measurement.Why: it earns it
Measured on an Apple M4 (16 GiB, macOS 26.5.2). One binary, arms toggled with
VT_OP_PROVIDER_DISABLE=mlxso there is no build-difference confound. Qwen3-1.7B-bf16, p=512 g=128, 2 reps, arm order alternated per rep to cancel thermal drift.Peak RSS is unchanged (6.65 to 7.50 GB in both arms), and the output is bit-identical: vllm.cpp's three-way parity test reports
mlx-vs-mslNMSE of0on all six shapes andmlx-vs-cpuequal tomsl-vs-cpu, against a 5e-4 bar.MLX serves the dense GEMM only. Paged attention stays vllm.cpp's own kernel, because MLX has no paged-KV primitive at all.
Honest limits: the numbers are INDICATIVE, not binding. The three
actions.runneragents on the box were verified idle, booted out for the sweep and restored, and the whole A/B ran inside onelockfso nothing could interleave between arms, butcom.localai.workerand the aerial wallpaper stayed up (no passwordless sudo). Rep spread reached 9.4% on 2 reps, so read the multipliers as +/-10%; the gap far exceeds the noise. Full disposition in vllm.cppdocs/BENCHMARKS.md, "MLX GEMM provider A/B on Apple M4" (mudler/vllm.cpp@41d7f8d7).How
Build. MLX comes from the pinned prebuilt pip wheel (
MLX_VERSION, default0.29.3) into a venv under the backend dir. Building MLX from source needsxcrun metal, a full Xcode the macOS runners do not have; the wheel shipsinclude/,lib/libmlx.dyliband the compiled metallib ready to link. The install is a stamp file rather than a phony target, since a phony prerequisite is always newer thanlibvllmand would re-link it on every invocation.VLLM_CPP_MLX=offrestores the previous Metal build.Packaging. Vendors
libmlx.dylib,mlx.metalliband MLX's MIT license intopackage/lib/. Three things this had to get right, each verified on the M4 before it was written:libvllm.dyliblinks@rpath/libmlx.dyliband its build-timeLC_RPATHpoints inside the build venv, a path no user has. Every build rpath is deleted and replaced with@loader_path/lib.install_name_toolinvalidates the code signature and macOS refuses to load an arm64 image with a stale one, so the patched library is re-signed ad-hoc.Verification
Built through this Makefile on the M4 and ran the packaged artifact:
DYLD_PRINT_LIBRARIESresolves libmlx frompackage/lib/libmlx.dylib, not the build venvcodesign -vpasses on the patched libraryop=65 selected=mlx priority=100) and zero metallib failuresA missing rpath now fails the build rather than the user's first inference request.
Also caught and fixed during verification: make expands a whole recipe before running its first line, so the original
$(shell)glob forMLX_ROOTresolved before the venv existed. It now resolves in the shell inside the stamp recipe.Cost
The darwin
vllm-cppimage grows by about 124 MB.Notes
-metal-darwin-arm64-vllm-cpp), so no CI matrix change is needed.MLX_VERSIONinto the existing dep-bump automation, as done forvllm-metal.