docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc - #20
Open
Geekgineer wants to merge 3 commits into
Open
docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc#20Geekgineer wants to merge 3 commits into
Geekgineer wants to merge 3 commits into
Conversation
Measured with vla-bench on an Apple M5 Max (18-core CPU, 40-core GPU, 64 GB unified memory), macOS 26.6.1, High Power Mode, llama.cpp b10331. Ten of the eleven released checkpoints, each at the same view count and input size as the existing RTX 5090 table so the two are comparable: 20 reps after 3 warmups, three sweeps per model, the lowest-p50 sweep reported. Verified backend = Metal in the load banner on all 30 runs; none fell back to CPU. Sweep-to-sweep p50 agreement is within 0.6% per model. BitVLA is omitted rather than reported: the shipped GGUFs are int2-packed and model_load rejects them outside a CUDA build (VLA_BITVLA_CUDA_KERNELS), and its ggml graph is CPU-only by design, so there is no Metal path to time. Also corrects the note under the CUDA table, which claimed Apple targets had not been measured with vla-bench. Docs only, no code touched, so predict() numerics cannot have moved; the roadmap support matrix is deliberately left alone.
The GPU offload section told readers to confirm a Metal build by looking for 'clip_ctx: CLIP using GPU backend' next to the vla banner. Nothing in the VLA path prints that line: each arch calls backend_init once and runs the vision tower on the same backend, so there is no separate CLIP context to log. Anyone following the old steps concludes a working Metal build failed. The banner is also arch-tagged, e.g. vla(pi0), and only SmolVLA logs under a bare vla tag. Verified by running vla-bench on this machine: SmolVLA prints 'vla: backend = Metal' and pi0 prints 'vla(pi0): backend = Metal', with zero clip_ctx or CLIP lines on stdout or stderr. Drop the '--n-gpu-layers 0' instruction: that flag is parsed only by vlm-server on the VLM path, and neither vla-cli, vla-server nor vla-bench accepts it. VLA_DEVICE does not help either, since it only picks a CUDA or SYCL ordinal. Configuring with -DGGML_METAL=OFF is the only way onto the CPU; confirmed that it leaves GGML_USE_METAL out of vla_core's compile flags, so backend_init falls through to ggml_backend_cpu_init. Note that BitVLA reports a CPU backend even on a Metal build, by design, and that its int2-packed GGUFs fail to load outside a CUDA build, so the banner is not mistaken for a broken setup. Point the Results section at the README's vla-bench table for current numbers and mark the existing M4 figures as older server-path measurements rather than leaving the two sets to be compared directly. Record that protobuf, zeromq and cppzmq are hard configure-time requirements, and that -hf needs the hf CLI.
Flips π0.5, GR00T N1.5, GR00T N1.6, Evo-1, VLA-Adapter, OpenVLA-OFT and VLA-JEPA from ~ to Y for Metal. Each loads at stock defaults on an M5 Max, appears in the new Apple Silicon latency table, and was diffed against the CPU backend with vla_predict_check on fixed images, tokens, state and noise: the same tree built twice, once as configured and once with -DGGML_METAL=OFF. Worst deviation among the seven is 6.9e-3 absolute on actions peaking near 1.0, RMS 1.7e-3 - the BF16/F32 kernel rounding that docs/backend/sycl.md already documents for SYCL, where the same comparison against CPU is what backs its Y cells. For scale, π0 and GR00T N1.7 were already Y and deviate further, 1.9e-2 and 1.4e-2, from compounding per-step rounding in their denoise loops; none of the cells moved here is looser than those. Metal output is bit-identical across repeat runs, so the deviation is rounding, not instability. BitVLA keeps ~ for Metal. It is not untested but structurally excluded: the graph is CPU-only by design and the published GGUFs are int2-packed for CUDA. Records the verification in docs/backend/metal.md alongside the latency pointer, matching how the SYCL doc reports it, and notes under the matrix what the Metal column now rests on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three docs commits, all from actually running the thing on an Apple M5 Max.
1. Add the Apple Silicon (Metal) latency table
The note under the CUDA table said Apple targets "have not been re-measured with
vla-bench". This measures them.vla-benchon an Apple M5 Max (18-core CPU, 40-core GPU, 64 GB unified memory),macOS 26.6.1, High Power Mode, llama.cpp
b10331, weights as shipped. Eachmodel runs at the same view count and input size as its RTX 5090 row, so the
two tables compare cell-for-cell. 20 reps after 3 warmups, three sweeps per
model, lowest-p50 sweep reported. VLA-JEPA gets its 32
<embodied>tokens via--extra-token 151697 --extra-count 32, matchingnum_futurein its own banner.What was verified:
backend = Metalin the load banner on all 30 successful runs; none fellback to CPU. A markdown row looks identical either way, so stdout and stderr
were captured per run and checked individually rather than once.
script that re-derives the sweep selection independently of how the table was
written, plus checks on view/input parity with the CUDA table and ascending
sort by
min ms.millisecond" spread claim (Evo-1's worst span is exactly 2.00 ms) and an
"under 1%" agreement figure (real worst case 0.533%).
An earlier pass of this sweep was discarded: the machine had Low Power Mode
enabled, which inflated π0 from 221 ms to 396 ms and turned a sub-millisecond
spread into a 70-140 ms scatter. Worth knowing if these get re-measured.
BitVLA is omitted, not measured. Its shipped GGUFs are int2-packed and
model_loadrejects them outside a CUDA build (VLA_BITVLA_CUDA_KERNELS), andsrc/models/bitvla.cpp:568callsggml_backend_cpu_init()by design, so thereis no Metal path to time. The repo ships no bf16 variant. Stated in the README
rather than left as a silent gap.
2. Fix the Metal confirmation steps in
docs/backend/metal.mdFound while following that doc to validate this build. It was actively
misleading:
clip_ctx: CLIP using GPU backend. Nothing in the VLA path prints that line - each arch callsbackend_initonce and runs the vision tower on the same backend, so there isno separate CLIP context to log. Following the old steps, a working Metal
build reads as a failed one. The banner is also arch-tagged (
vla(pi0)), withSmolVLA the only one under a bare
vlatag.--n-gpu-layers 0for disabling GPU inference. That flag isparsed only by
vlm-serveron the VLM path;vla-cli,vla-serverandvla-benchdo not accept it, andVLA_DEVICEonly picks a CUDA/SYCL ordinal.-DGGML_METAL=OFFis the only route to CPU.Both fixes are verified, not inferred:
vla-benchon this machine printsvla: backend = Metalfor SmolVLA andvla(pi0): backend = Metalfor π0 with zeroclip_ctx/CLIPlines on either stream, and a-DGGML_METAL=OFFconfigureleaves
GGML_USE_METALout ofvla_core's compile flags sobackend_initfalls through to
ggml_backend_cpu_init.Also: BitVLA's CPU banner is documented as expected rather than a broken build;
the M4 figures are marked as older server-path measurements and point at the new
vla-benchtable instead of inviting a direct comparison; and the hardconfigure-time deps (protobuf, zeromq, cppzmq) plus the
hfCLI needed by-hfare recorded, since a clean box fails on both.
3. Mark the Metal column supported for the archs that pass
Flips π0.5, GR00T N1.5, GR00T N1.6, Evo-1, VLA-Adapter, OpenVLA-OFT and
VLA-JEPA from
~toYfor Metal.Latency alone did not seem like enough to move a support cell, so I went looking
for what
Yhas actually meant.docs/backend/sycl.mdanswers it: itsYcellsrest on latency plus "outputs checked against the CPU backend ... max
absolute deviation 2.9e-3 ... not a numerical regression", and VLA-Adapter sits
at
~for SYCL despite appearing in that doc's results table, because it neededa non-stock
VLA_ADAPTER_F32_WEIGHTS=1. So the bar is: benchmarked, numericallyverified against CPU, at stock defaults.
I ran that same comparison for Metal -
vla_predict_checkon fixed images,tokens, state and noise, the same tree built twice, once as configured and once
with
-DGGML_METAL=OFF:The two loosest, π0 and GR00T N1.7, are already
Y- they run multi-stepdenoise loops where per-step rounding compounds, and metal.md's own M4 LIBERO
table shows them at 0.8 and 1.0 success rate, so that magnitude is evidently
tolerable in practice. Every cell this moves is tighter than both. Metal output
is bit-identical across repeat runs, so this is BF16/F32 kernel rounding, not
instability.
BitVLA keeps
~. It is not untested but structurally excluded: CPU-onlygraph by design, int2-packed GGUFs that need CUDA. I left it at
~rather than-because "planned" would assert an intent I have no basis for.A note under the matrix records what the Metal column now rests on, and the
verification is written up in
docs/backend/metal.mdnext to the latencypointer, matching how the SYCL doc reports the same thing.
Scope
Docs only, no code touched, so
predict()numerics cannot have moved - that iswhy there is no
vla_predict_checkbefore/after here, rather than an oversight.The Roadmap matrix change is scoped to the Metal column and backed by the
CPU-comparison above; CPU, CUDA, SYCL and OpenVINO cells are untouched.