Skip to content

docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Geekgineer wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a12f1e2
add Apple Silicon (Metal) latency table to the benchmarks section
Geekgineer Aug 13, 2026
11798b9
correct the Metal backend confirmation steps in the macOS doc
Geekgineer Aug 13, 2026
ed4d6fd
mark the Metal column supported for the ten archs that pass on Metal
Geekgineer Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 57 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,8 +269,40 @@ view count.
| Evo-1 | 1 | 448 | 52.2 | 55.2 | 57.3 | 17.8 |
| pi0.5 | 2 | 224 | 53.4 | 56.1 | 59.3 | 11.4 |

Jetson and Apple targets are absent: they have not been re-measured with
`vla-bench`.
Jetson targets are absent: they have not been re-measured with `vla-bench`.

### Apple Silicon (Metal)

The same harness on the Metal backend. Apple M5 Max (18-core CPU,
40-core GPU, 64 GB unified memory), macOS 26.6.1, High Power Mode, llama.cpp
`b10331`, weights as shipped, 20 reps after 3 warmups, best of three sweeps
(the sweep with the lowest p50), each model at its native input size and view
count.

| Model | Views | Input | min ms | p50 ms | p90 ms | vision ms |
|---|--:|--:|--:|--:|--:|--:|
| VLA-Adapter | 1 | 224 | 64.5 | 64.8 | 65.1 | 33.0 |
| VLA-JEPA | 1 | 256 | 74.9 | 75.1 | 75.4 | 17.2 |
| GR00T N1.5 | 1 | 224 | 104.0 | 104.3 | 104.6 | 20.7 |
| SmolVLA | 2 | 512 | 114.8 | 115.2 | 115.9 | 29.0 |
| GR00T N1.7 | 1 | 256 | 128.0 | 128.4 | 128.9 | 17.3 |
| GR00T N1.6 | 1 | 224 | 133.0 | 133.3 | 134.4 | 21.3 |
| OpenVLA-OFT | 1 | 224 | 183.4 | 184.2 | 184.6 | 33.7 |
| Evo-1 | 1 | 448 | 214.5 | 215.0 | 216.2 | 50.3 |
| pi0 | 2 | 224 | 220.2 | 220.8 | 221.3 | 39.3 |
| pi0.5 | 2 | 224 | 237.1 | 237.5 | 237.7 | 39.1 |

Every run came up on Metal (`backend = Metal` in the load banner); none fell
back to CPU. The three sweeps agree to within 0.6% per model, and `min` to `p90`
spans no more than 2 ms, so these settle rather than scatter.

BitVLA has no row. Its shipped GGUFs are int2-packed and the loader rejects them
outside a CUDA build (`VLA_BITVLA_CUDA_KERNELS`), and its ggml graph is CPU-only
by design with the LM offloading to CUDA, so there is nothing to measure on
Metal.

These are latency numbers on the Metal backend, not a support claim: see
[Roadmap](#roadmap) for which pairings are released and benchmarked.

### Task success

Expand Down Expand Up @@ -305,15 +337,31 @@ supported (released and benchmarked), `~` = in progress, `-` = planned.
|---|:--:|:--:|:--:|:--:|:--:|
| [SmolVLA](https://hf.co/vrfai/smolvla-libero-gguf) | Y | Y | Y | Y | - |
| [π0](https://hf.co/vrfai/pi0-libero-finetuned-v044-gguf) | Y | Y | - | Y | - |
| [π0.5](https://hf.co/vrfai/pi05-libero-gguf) | Y | Y | - | ~ | - |
| [GR00T N1.5](https://hf.co/vrfai/gr00tn1d5-libero-object-gguf) | Y | Y | - | ~ | - |
| [GR00T N1.6](https://hf.co/vrfai/gr00tn1d6-libero-gguf) | Y | Y | - | ~ | - |
| [π0.5](https://hf.co/vrfai/pi05-libero-gguf) | Y | Y | - | Y | - |
| [GR00T N1.5](https://hf.co/vrfai/gr00tn1d5-libero-object-gguf) | Y | Y | - | Y | - |
| [GR00T N1.6](https://hf.co/vrfai/gr00tn1d6-libero-gguf) | Y | Y | - | Y | - |
| [GR00T N1.7](https://hf.co/vrfai/gr00tn1d7-libero-gguf) | Y | Y | - | Y | - |
| [BitVLA](https://hf.co/vrfai/bitvla-libero-gguf) | Y | Y | - | ~ | - |
| [Evo-1](https://hf.co/vrfai/evo1-libero-gguf) | Y | Y | Y | ~ | - |
| [VLA-Adapter](https://hf.co/vrfai/vla-adapter-libero-gguf) | Y | Y | ~ | ~ | - |
| [OpenVLA-OFT](https://hf.co/vrfai/openvla-oft-libero-gguf) | Y | Y | - | ~ | - |
| [VLA-JEPA](https://hf.co/vrfai/vla-jepa-libero) | Y | Y | - | ~ | - |
| [Evo-1](https://hf.co/vrfai/evo1-libero-gguf) | Y | Y | Y | Y | - |
| [VLA-Adapter](https://hf.co/vrfai/vla-adapter-libero-gguf) | Y | Y | ~ | Y | - |
| [OpenVLA-OFT](https://hf.co/vrfai/openvla-oft-libero-gguf) | Y | Y | - | Y | - |
| [VLA-JEPA](https://hf.co/vrfai/vla-jepa-libero) | Y | Y | - | Y | - |

The Metal column was filled from a sweep on an Apple M5 Max: every model marked
`Y` there loads on Metal at stock defaults, times as shown in
[Benchmarks](#apple-silicon-metal), and was diffed against the CPU backend with
`vla_predict_check` on fixed images, tokens, state and noise. Worst case per
model is 6.9e-3 absolute on actions peaking near 1.0 (RMS 1.7e-3) - BF16/F32
kernel rounding, the same conclusion
[docs/backend/sycl.md](docs/backend/sycl.md) reaches for SYCL. π0 and GR00T
N1.7, both already `Y`, deviate most at 1.9e-2 and 1.4e-2, which is what
iterating a denoise loop in a different rounding regime costs; the seven cells
this sweep moved are all tighter than that.

BitVLA stays short of `Y` on Metal for a structural reason rather than an
untested one: its graph is CPU-only by design and its published GGUFs are
int2-packed for CUDA, so there is nothing to run. See
[Benchmarks](#apple-silicon-metal).

---

Expand Down
75 changes: 65 additions & 10 deletions docs/backend/metal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,42 +9,97 @@ CMake flag.
```bash
brew install protobuf zeromq cppzmq pkg-config
```

All four are required at configure time, not optional: `find_package(Protobuf)`
and `pkg_check_modules(libzmq)` are unconditional, so a missing `zeromq` or
`cppzmq` fails CMake before anything builds.

To let the binaries fetch checkpoints with `-hf`, also install the Hugging Face
CLI - the fetch shells out to `hf` and stops with `hf: command not found`
without it:

```bash
pip install -U "huggingface_hub[cli]" # or: uv tool install huggingface_hub
```

## Configure & build

On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU.
To disable the Metal build at compile time use the `-DGGML_METAL=OFF` cmake option.

When built with Metal support, you can explicitly disable GPU inference with the `--n-gpu-layers 0` command-line argument.

```bash
# cmake fetches llama.cpp at the pinned tag; no patch step.
# On MacOS, Metal is enabled by default
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(sysctl -n hw.ncpu)
```

The VLA binaries pick their backend at load time and take no flag for it:
`-DGGML_METAL=OFF` at configure time is the only way to put `vla-cli`,
`vla-server` and `vla-bench` on the CPU. `VLA_DEVICE` only chooses an ordinal
for CUDA and SYCL, so it does nothing here, and `--n-gpu-layers` belongs to
`vlm-server` on the VLM path, not to the VLA binaries.

## GPU offload

The VLA core selects its compute backend at load time. On macOS it picks Metal
(`ggml_backend_metal_init`); the CLIP/vision encoder mirrors that choice, so both
the transformer and the vision tower run on the GPU. Confirm it from the startup
banner:
Each arch calls `backend_init` (`src/backend.h`) exactly once at load time and
runs everything on what it returns, vision tower included. On macOS that is
Metal (`ggml_backend_metal_init`). Confirm it from the startup banner, which is
tagged with the arch:

```
vla(pi0): backend = Metal
```

SmolVLA is the one that logs under a bare `vla` tag, so for it the line reads:

```
vla: backend = Metal
clip_ctx: CLIP using GPU backend
```

If you instead see `vla: backend = CPU (4 threads)` / `CLIP using CPU backend`,
the build didn't pick up Metal - rebuild from a clean `build/` and check
`GGML_METAL` is `ON` in the CMake cache.
There is no second banner to look for: the VLA path prints no `clip_ctx: CLIP
using GPU backend` line, because there is no separate CLIP context to bring up -
the single backend already covers both towers. A Metal build that reports only
the line above is working.

If you instead see `vla(<arch>): backend = CPU (N threads)`, the build didn't
pick up Metal - rebuild from a clean `build/` and check `GGML_METAL` is `ON` in
the CMake cache (`grep GGML_METAL build/CMakeCache.txt`).

> Single-backend, no per-op CPU fallback: the core uses one backend + `gallocr`,
> not a scheduler. SmolVLA's ops are all Metal-supported; an arch that hits an
> unimplemented op would assert at predict time rather than silently fall back.

BitVLA is the exception and does not run on Metal at all. It calls
`ggml_backend_cpu_init()` directly (`src/models/bitvla.cpp:568`) because its
graph stays on CPU and the LM offloads through CUDA, so it reports `vla(bitvla):
ggml backend = CPU (N threads)` even on a Metal build - that banner is expected,
not a broken build. The published GGUFs are also int2-packed, which `model_load`
rejects outside a CUDA build (`VLA_BITVLA_CUDA_KERNELS`), so on macOS it fails
to load rather than running slowly.

## Results

For current per-model latency on Apple Silicon, see the Apple Silicon (Metal)
table in [the README](../../README.md#benchmarks): that one is measured with
`vla-bench`, in-process on synthetic inputs, and is directly comparable to the
CUDA table above it.

Outputs were checked against the CPU backend on all ten models in that table, on
an M5 Max, using `vla_predict_check` on fixed images, tokens, state and noise -
the same build twice, once as configured and once with `-DGGML_METAL=OFF`. The
loosest model is π0: 1.9e-2 absolute on actions peaking near 1.0, RMS 5.2e-4 for
that model, with a worst per-model RMS of 1.7e-3 across the set. Eight of the ten
stay under 7e-3 absolute. The two outliers are π0 and GR00T N1.7 (1.4e-2), which
run multi-step denoise loops where per-step rounding compounds.
Metal output is deterministic run to run - repeated runs are bit-identical - so
these are BF16/F32 kernel rounding, not instability.

The measurements below predate it and are not the same experiment - they were
taken end-to-end through `vla-server` on an M4, so they include transport and
preprocessing that `vla-bench` excludes, and they come from an older revision.
Read them as evidence that GPU offload is worth having, not as current numbers.

SmolVLA (libero, `mmproj` + 878 MiB BF16 weights), **Apple M4**, steady state:

| Stage | CPU (before) | Metal GPU (after) |
Expand Down