From a12f1e2c2eec83677b9633064f2b65397b62495b Mon Sep 17 00:00:00 2001 From: Geekgineer Date: Thu, 13 Aug 2026 10:56:41 +0200 Subject: [PATCH 1/3] add Apple Silicon (Metal) latency table to the benchmarks section Measured with vla-bench on an Apple M5 Max (18-core CPU, 40-core GPU, 64 GB unified memory), macOS 26.6.1, High Power Mode, llama.cpp b10331. Ten of the eleven released checkpoints, each at the same view count and input size as the existing RTX 5090 table so the two are comparable: 20 reps after 3 warmups, three sweeps per model, the lowest-p50 sweep reported. Verified backend = Metal in the load banner on all 30 runs; none fell back to CPU. Sweep-to-sweep p50 agreement is within 0.6% per model. BitVLA is omitted rather than reported: the shipped GGUFs are int2-packed and model_load rejects them outside a CUDA build (VLA_BITVLA_CUDA_KERNELS), and its ggml graph is CPU-only by design, so there is no Metal path to time. Also corrects the note under the CUDA table, which claimed Apple targets had not been measured with vla-bench. Docs only, no code touched, so predict() numerics cannot have moved; the roadmap support matrix is deliberately left alone. --- README.md | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 17fb5e1..5e261b2 100644 --- a/README.md +++ b/README.md @@ -269,8 +269,40 @@ view count. | Evo-1 | 1 | 448 | 52.2 | 55.2 | 57.3 | 17.8 | | pi0.5 | 2 | 224 | 53.4 | 56.1 | 59.3 | 11.4 | -Jetson and Apple targets are absent: they have not been re-measured with -`vla-bench`. +Jetson targets are absent: they have not been re-measured with `vla-bench`. + +### Apple Silicon (Metal) + +The same harness on the Metal backend. Apple M5 Max (18-core CPU, +40-core GPU, 64 GB unified memory), macOS 26.6.1, High Power Mode, llama.cpp +`b10331`, weights as shipped, 20 reps after 3 warmups, best of three sweeps +(the sweep with the lowest p50), each model at its native input size and view +count. + +| Model | Views | Input | min ms | p50 ms | p90 ms | vision ms | +|---|--:|--:|--:|--:|--:|--:| +| VLA-Adapter | 1 | 224 | 64.5 | 64.8 | 65.1 | 33.0 | +| VLA-JEPA | 1 | 256 | 74.9 | 75.1 | 75.4 | 17.2 | +| GR00T N1.5 | 1 | 224 | 104.0 | 104.3 | 104.6 | 20.7 | +| SmolVLA | 2 | 512 | 114.8 | 115.2 | 115.9 | 29.0 | +| GR00T N1.7 | 1 | 256 | 128.0 | 128.4 | 128.9 | 17.3 | +| GR00T N1.6 | 1 | 224 | 133.0 | 133.3 | 134.4 | 21.3 | +| OpenVLA-OFT | 1 | 224 | 183.4 | 184.2 | 184.6 | 33.7 | +| Evo-1 | 1 | 448 | 214.5 | 215.0 | 216.2 | 50.3 | +| pi0 | 2 | 224 | 220.2 | 220.8 | 221.3 | 39.3 | +| pi0.5 | 2 | 224 | 237.1 | 237.5 | 237.7 | 39.1 | + +Every run came up on Metal (`backend = Metal` in the load banner); none fell +back to CPU. The three sweeps agree to within 0.6% per model, and `min` to `p90` +spans no more than 2 ms, so these settle rather than scatter. + +BitVLA has no row. Its shipped GGUFs are int2-packed and the loader rejects them +outside a CUDA build (`VLA_BITVLA_CUDA_KERNELS`), and its ggml graph is CPU-only +by design with the LM offloading to CUDA, so there is nothing to measure on +Metal. + +These are latency numbers on the Metal backend, not a support claim: see +[Roadmap](#roadmap) for which pairings are released and benchmarked. ### Task success From 11798b925154bde9aab585e5cba513870c133de0 Mon Sep 17 00:00:00 2001 From: Geekgineer Date: Thu, 13 Aug 2026 11:22:56 +0200 Subject: [PATCH 2/3] correct the Metal backend confirmation steps in the macOS doc The GPU offload section told readers to confirm a Metal build by looking for 'clip_ctx: CLIP using GPU backend' next to the vla banner. Nothing in the VLA path prints that line: each arch calls backend_init once and runs the vision tower on the same backend, so there is no separate CLIP context to log. Anyone following the old steps concludes a working Metal build failed. The banner is also arch-tagged, e.g. vla(pi0), and only SmolVLA logs under a bare vla tag. Verified by running vla-bench on this machine: SmolVLA prints 'vla: backend = Metal' and pi0 prints 'vla(pi0): backend = Metal', with zero clip_ctx or CLIP lines on stdout or stderr. Drop the '--n-gpu-layers 0' instruction: that flag is parsed only by vlm-server on the VLM path, and neither vla-cli, vla-server nor vla-bench accepts it. VLA_DEVICE does not help either, since it only picks a CUDA or SYCL ordinal. Configuring with -DGGML_METAL=OFF is the only way onto the CPU; confirmed that it leaves GGML_USE_METAL out of vla_core's compile flags, so backend_init falls through to ggml_backend_cpu_init. Note that BitVLA reports a CPU backend even on a Metal build, by design, and that its int2-packed GGUFs fail to load outside a CUDA build, so the banner is not mistaken for a broken setup. Point the Results section at the README's vla-bench table for current numbers and mark the existing M4 figures as older server-path measurements rather than leaving the two sets to be compared directly. Record that protobuf, zeromq and cppzmq are hard configure-time requirements, and that -hf needs the hf CLI. --- docs/backend/metal.md | 65 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 55 insertions(+), 10 deletions(-) diff --git a/docs/backend/metal.md b/docs/backend/metal.md index e9adc61..fe505b9 100644 --- a/docs/backend/metal.md +++ b/docs/backend/metal.md @@ -9,13 +9,24 @@ CMake flag. ```bash brew install protobuf zeromq cppzmq pkg-config ``` + +All four are required at configure time, not optional: `find_package(Protobuf)` +and `pkg_check_modules(libzmq)` are unconditional, so a missing `zeromq` or +`cppzmq` fails CMake before anything builds. + +To let the binaries fetch checkpoints with `-hf`, also install the Hugging Face +CLI - the fetch shells out to `hf` and stops with `hf: command not found` +without it: + +```bash +pip install -U "huggingface_hub[cli]" # or: uv tool install huggingface_hub +``` + ## Configure & build On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU. To disable the Metal build at compile time use the `-DGGML_METAL=OFF` cmake option. -When built with Metal support, you can explicitly disable GPU inference with the `--n-gpu-layers 0` command-line argument. - ```bash # cmake fetches llama.cpp at the pinned tag; no patch step. # On MacOS, Metal is enabled by default @@ -23,28 +34,62 @@ cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -j$(sysctl -n hw.ncpu) ``` +The VLA binaries pick their backend at load time and take no flag for it: +`-DGGML_METAL=OFF` at configure time is the only way to put `vla-cli`, +`vla-server` and `vla-bench` on the CPU. `VLA_DEVICE` only chooses an ordinal +for CUDA and SYCL, so it does nothing here, and `--n-gpu-layers` belongs to +`vlm-server` on the VLM path, not to the VLA binaries. + ## GPU offload -The VLA core selects its compute backend at load time. On macOS it picks Metal -(`ggml_backend_metal_init`); the CLIP/vision encoder mirrors that choice, so both -the transformer and the vision tower run on the GPU. Confirm it from the startup -banner: +Each arch calls `backend_init` (`src/backend.h`) exactly once at load time and +runs everything on what it returns, vision tower included. On macOS that is +Metal (`ggml_backend_metal_init`). Confirm it from the startup banner, which is +tagged with the arch: + +``` +vla(pi0): backend = Metal +``` + +SmolVLA is the one that logs under a bare `vla` tag, so for it the line reads: ``` vla: backend = Metal -clip_ctx: CLIP using GPU backend ``` -If you instead see `vla: backend = CPU (4 threads)` / `CLIP using CPU backend`, -the build didn't pick up Metal - rebuild from a clean `build/` and check -`GGML_METAL` is `ON` in the CMake cache. +There is no second banner to look for: the VLA path prints no `clip_ctx: CLIP +using GPU backend` line, because there is no separate CLIP context to bring up - +the single backend already covers both towers. A Metal build that reports only +the line above is working. + +If you instead see `vla(): backend = CPU (N threads)`, the build didn't +pick up Metal - rebuild from a clean `build/` and check `GGML_METAL` is `ON` in +the CMake cache (`grep GGML_METAL build/CMakeCache.txt`). > Single-backend, no per-op CPU fallback: the core uses one backend + `gallocr`, > not a scheduler. SmolVLA's ops are all Metal-supported; an arch that hits an > unimplemented op would assert at predict time rather than silently fall back. +BitVLA is the exception and does not run on Metal at all. It calls +`ggml_backend_cpu_init()` directly (`src/models/bitvla.cpp:568`) because its +graph stays on CPU and the LM offloads through CUDA, so it reports `vla(bitvla): +ggml backend = CPU (N threads)` even on a Metal build - that banner is expected, +not a broken build. The published GGUFs are also int2-packed, which `model_load` +rejects outside a CUDA build (`VLA_BITVLA_CUDA_KERNELS`), so on macOS it fails +to load rather than running slowly. + ## Results +For current per-model latency on Apple Silicon, see the Apple Silicon (Metal) +table in [the README](../../README.md#benchmarks): that one is measured with +`vla-bench`, in-process on synthetic inputs, and is directly comparable to the +CUDA table above it. + +The measurements below predate it and are not the same experiment - they were +taken end-to-end through `vla-server` on an M4, so they include transport and +preprocessing that `vla-bench` excludes, and they come from an older revision. +Read them as evidence that GPU offload is worth having, not as current numbers. + SmolVLA (libero, `mmproj` + 878 MiB BF16 weights), **Apple M4**, steady state: | Stage | CPU (before) | Metal GPU (after) | From ed4d6fdfd9818b7504eea17a186728da17d5e10a Mon Sep 17 00:00:00 2001 From: Geekgineer Date: Thu, 13 Aug 2026 12:12:24 +0200 Subject: [PATCH 3/3] mark the Metal column supported for the ten archs that pass on Metal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flips π0.5, GR00T N1.5, GR00T N1.6, Evo-1, VLA-Adapter, OpenVLA-OFT and VLA-JEPA from ~ to Y for Metal. Each loads at stock defaults on an M5 Max, appears in the new Apple Silicon latency table, and was diffed against the CPU backend with vla_predict_check on fixed images, tokens, state and noise: the same tree built twice, once as configured and once with -DGGML_METAL=OFF. Worst deviation among the seven is 6.9e-3 absolute on actions peaking near 1.0, RMS 1.7e-3 - the BF16/F32 kernel rounding that docs/backend/sycl.md already documents for SYCL, where the same comparison against CPU is what backs its Y cells. For scale, π0 and GR00T N1.7 were already Y and deviate further, 1.9e-2 and 1.4e-2, from compounding per-step rounding in their denoise loops; none of the cells moved here is looser than those. Metal output is bit-identical across repeat runs, so the deviation is rounding, not instability. BitVLA keeps ~ for Metal. It is not untested but structurally excluded: the graph is CPU-only by design and the published GGUFs are int2-packed for CUDA. Records the verification in docs/backend/metal.md alongside the latency pointer, matching how the SYCL doc reports it, and notes under the matrix what the Metal column now rests on. --- README.md | 30 +++++++++++++++++++++++------- docs/backend/metal.md | 10 ++++++++++ 2 files changed, 33 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 5e261b2..8ebfee7 100644 --- a/README.md +++ b/README.md @@ -337,15 +337,31 @@ supported (released and benchmarked), `~` = in progress, `-` = planned. |---|:--:|:--:|:--:|:--:|:--:| | [SmolVLA](https://hf.co/vrfai/smolvla-libero-gguf) | Y | Y | Y | Y | - | | [π0](https://hf.co/vrfai/pi0-libero-finetuned-v044-gguf) | Y | Y | - | Y | - | -| [π0.5](https://hf.co/vrfai/pi05-libero-gguf) | Y | Y | - | ~ | - | -| [GR00T N1.5](https://hf.co/vrfai/gr00tn1d5-libero-object-gguf) | Y | Y | - | ~ | - | -| [GR00T N1.6](https://hf.co/vrfai/gr00tn1d6-libero-gguf) | Y | Y | - | ~ | - | +| [π0.5](https://hf.co/vrfai/pi05-libero-gguf) | Y | Y | - | Y | - | +| [GR00T N1.5](https://hf.co/vrfai/gr00tn1d5-libero-object-gguf) | Y | Y | - | Y | - | +| [GR00T N1.6](https://hf.co/vrfai/gr00tn1d6-libero-gguf) | Y | Y | - | Y | - | | [GR00T N1.7](https://hf.co/vrfai/gr00tn1d7-libero-gguf) | Y | Y | - | Y | - | | [BitVLA](https://hf.co/vrfai/bitvla-libero-gguf) | Y | Y | - | ~ | - | -| [Evo-1](https://hf.co/vrfai/evo1-libero-gguf) | Y | Y | Y | ~ | - | -| [VLA-Adapter](https://hf.co/vrfai/vla-adapter-libero-gguf) | Y | Y | ~ | ~ | - | -| [OpenVLA-OFT](https://hf.co/vrfai/openvla-oft-libero-gguf) | Y | Y | - | ~ | - | -| [VLA-JEPA](https://hf.co/vrfai/vla-jepa-libero) | Y | Y | - | ~ | - | +| [Evo-1](https://hf.co/vrfai/evo1-libero-gguf) | Y | Y | Y | Y | - | +| [VLA-Adapter](https://hf.co/vrfai/vla-adapter-libero-gguf) | Y | Y | ~ | Y | - | +| [OpenVLA-OFT](https://hf.co/vrfai/openvla-oft-libero-gguf) | Y | Y | - | Y | - | +| [VLA-JEPA](https://hf.co/vrfai/vla-jepa-libero) | Y | Y | - | Y | - | + +The Metal column was filled from a sweep on an Apple M5 Max: every model marked +`Y` there loads on Metal at stock defaults, times as shown in +[Benchmarks](#apple-silicon-metal), and was diffed against the CPU backend with +`vla_predict_check` on fixed images, tokens, state and noise. Worst case per +model is 6.9e-3 absolute on actions peaking near 1.0 (RMS 1.7e-3) - BF16/F32 +kernel rounding, the same conclusion +[docs/backend/sycl.md](docs/backend/sycl.md) reaches for SYCL. π0 and GR00T +N1.7, both already `Y`, deviate most at 1.9e-2 and 1.4e-2, which is what +iterating a denoise loop in a different rounding regime costs; the seven cells +this sweep moved are all tighter than that. + +BitVLA stays short of `Y` on Metal for a structural reason rather than an +untested one: its graph is CPU-only by design and its published GGUFs are +int2-packed for CUDA, so there is nothing to run. See +[Benchmarks](#apple-silicon-metal). --- diff --git a/docs/backend/metal.md b/docs/backend/metal.md index fe505b9..689e214 100644 --- a/docs/backend/metal.md +++ b/docs/backend/metal.md @@ -85,6 +85,16 @@ table in [the README](../../README.md#benchmarks): that one is measured with `vla-bench`, in-process on synthetic inputs, and is directly comparable to the CUDA table above it. +Outputs were checked against the CPU backend on all ten models in that table, on +an M5 Max, using `vla_predict_check` on fixed images, tokens, state and noise - +the same build twice, once as configured and once with `-DGGML_METAL=OFF`. The +loosest model is π0: 1.9e-2 absolute on actions peaking near 1.0, RMS 5.2e-4 for +that model, with a worst per-model RMS of 1.7e-3 across the set. Eight of the ten +stay under 7e-3 absolute. The two outliers are π0 and GR00T N1.7 (1.4e-2), which +run multi-step denoise loops where per-step rounding compounds. +Metal output is deterministic run to run - repeated runs are bit-identical - so +these are BF16/F32 kernel rounding, not instability. + The measurements below predate it and are not the same experiment - they were taken end-to-end through `vla-server` on an M4, so they include transport and preprocessing that `vla-bench` excludes, and they come from an older revision.