Skip to content

docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc - #20

Open
Geekgineer wants to merge 3 commits into
VinRobotics:mainfrom
Geekgineer:bench-apple-silicon-metal
Open

docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc#20
Geekgineer wants to merge 3 commits into
VinRobotics:mainfrom
Geekgineer:bench-apple-silicon-metal

Conversation

@Geekgineer

@Geekgineer Geekgineer commented Aug 13, 2026

Copy link
Copy Markdown

Three docs commits, all from actually running the thing on an Apple M5 Max.

1. Add the Apple Silicon (Metal) latency table

The note under the CUDA table said Apple targets "have not been re-measured with
vla-bench". This measures them.

vla-bench on an Apple M5 Max (18-core CPU, 40-core GPU, 64 GB unified memory),
macOS 26.6.1, High Power Mode, llama.cpp b10331, weights as shipped. Each
model runs at the same view count and input size as its RTX 5090 row, so the
two tables compare cell-for-cell. 20 reps after 3 warmups, three sweeps per
model, lowest-p50 sweep reported. VLA-JEPA gets its 32 <embodied> tokens via
--extra-token 151697 --extra-count 32, matching num_future in its own banner.

What was verified:

  • backend = Metal in the load banner on all 30 successful runs; none fell
    back to CPU.
    A markdown row looks identical either way, so stdout and stderr
    were captured per run and checked individually rather than once.
  • Every published value was cross-checked against the raw per-run output by a
    script that re-derives the sweep selection independently of how the table was
    written, plus checks on view/input parity with the CUDA table and ascending
    sort by min ms.
  • Two draft claims were corrected after recomputing them: a "within a
    millisecond" spread claim (Evo-1's worst span is exactly 2.00 ms) and an
    "under 1%" agreement figure (real worst case 0.533%).

An earlier pass of this sweep was discarded: the machine had Low Power Mode
enabled, which inflated π0 from 221 ms to 396 ms and turned a sub-millisecond
spread into a 70-140 ms scatter. Worth knowing if these get re-measured.

BitVLA is omitted, not measured. Its shipped GGUFs are int2-packed and
model_load rejects them outside a CUDA build (VLA_BITVLA_CUDA_KERNELS), and
src/models/bitvla.cpp:568 calls ggml_backend_cpu_init() by design, so there
is no Metal path to time. The repo ships no bf16 variant. Stated in the README
rather than left as a silent gap.

2. Fix the Metal confirmation steps in docs/backend/metal.md

Found while following that doc to validate this build. It was actively
misleading:

  • It told readers to confirm Metal by looking for clip_ctx: CLIP using GPU backend. Nothing in the VLA path prints that line - each arch calls
    backend_init once and runs the vision tower on the same backend, so there is
    no separate CLIP context to log. Following the old steps, a working Metal
    build reads as a failed one. The banner is also arch-tagged (vla(pi0)), with
    SmolVLA the only one under a bare vla tag.
  • It documented --n-gpu-layers 0 for disabling GPU inference. That flag is
    parsed only by vlm-server on the VLM path; vla-cli, vla-server and
    vla-bench do not accept it, and VLA_DEVICE only picks a CUDA/SYCL ordinal.
    -DGGML_METAL=OFF is the only route to CPU.

Both fixes are verified, not inferred: vla-bench on this machine prints vla: backend = Metal for SmolVLA and vla(pi0): backend = Metal for π0 with zero
clip_ctx/CLIP lines on either stream, and a -DGGML_METAL=OFF configure
leaves GGML_USE_METAL out of vla_core's compile flags so backend_init
falls through to ggml_backend_cpu_init.

Also: BitVLA's CPU banner is documented as expected rather than a broken build;
the M4 figures are marked as older server-path measurements and point at the new
vla-bench table instead of inviting a direct comparison; and the hard
configure-time deps (protobuf, zeromq, cppzmq) plus the hf CLI needed by -hf
are recorded, since a clean box fails on both.

3. Mark the Metal column supported for the archs that pass

Flips π0.5, GR00T N1.5, GR00T N1.6, Evo-1, VLA-Adapter, OpenVLA-OFT and
VLA-JEPA
from ~ to Y for Metal.

Latency alone did not seem like enough to move a support cell, so I went looking
for what Y has actually meant. docs/backend/sycl.md answers it: its Y cells
rest on latency plus "outputs checked against the CPU backend ... max
absolute deviation 2.9e-3 ... not a numerical regression", and VLA-Adapter sits
at ~ for SYCL despite appearing in that doc's results table, because it needed
a non-stock VLA_ADAPTER_F32_WEIGHTS=1. So the bar is: benchmarked, numerically
verified against CPU, at stock defaults.

I ran that same comparison for Metal - vla_predict_check on fixed images,
tokens, state and noise, the same tree built twice, once as configured and once
with -DGGML_METAL=OFF:

Model max abs dev RMS vs peak |a|
GR00T N1.6 1.3e-03 3.1e-04 0.15%
OpenVLA-OFT 1.2e-03 4.0e-04 0.14%
pi0.5 1.2e-03 1.6e-04 0.13%
SmolVLA 1.7e-03 2.0e-04 0.17%
VLA-JEPA 2.7e-03 1.1e-03 0.23%
VLA-Adapter 2.8e-03 9.0e-04 0.42%
Evo-1 2.9e-03 4.0e-04 0.33%
GR00T N1.5 6.9e-03 1.7e-03 0.80%
GR00T N1.7 1.4e-02 4.1e-04 1.50%
pi0 1.9e-02 5.2e-04 1.82%

The two loosest, π0 and GR00T N1.7, are already Y - they run multi-step
denoise loops where per-step rounding compounds, and metal.md's own M4 LIBERO
table shows them at 0.8 and 1.0 success rate, so that magnitude is evidently
tolerable in practice. Every cell this moves is tighter than both. Metal output
is bit-identical across repeat runs, so this is BF16/F32 kernel rounding, not
instability.

BitVLA keeps ~. It is not untested but structurally excluded: CPU-only
graph by design, int2-packed GGUFs that need CUDA. I left it at ~ rather than
- because "planned" would assert an intent I have no basis for.

A note under the matrix records what the Metal column now rests on, and the
verification is written up in docs/backend/metal.md next to the latency
pointer, matching how the SYCL doc reports the same thing.

Scope

Docs only, no code touched, so predict() numerics cannot have moved - that is
why there is no vla_predict_check before/after here, rather than an oversight.

The Roadmap matrix change is scoped to the Metal column and backed by the
CPU-comparison above; CPU, CUDA, SYCL and OpenVINO cells are untouched.

Measured with vla-bench on an Apple M5 Max (18-core CPU, 40-core GPU, 64 GB
unified memory), macOS 26.6.1, High Power Mode, llama.cpp b10331. Ten of the
eleven released checkpoints, each at the same view count and input size as the
existing RTX 5090 table so the two are comparable: 20 reps after 3 warmups,
three sweeps per model, the lowest-p50 sweep reported.

Verified backend = Metal in the load banner on all 30 runs; none fell back to
CPU. Sweep-to-sweep p50 agreement is within 0.6% per model.

BitVLA is omitted rather than reported: the shipped GGUFs are int2-packed and
model_load rejects them outside a CUDA build (VLA_BITVLA_CUDA_KERNELS), and its
ggml graph is CPU-only by design, so there is no Metal path to time.

Also corrects the note under the CUDA table, which claimed Apple targets had
not been measured with vla-bench.

Docs only, no code touched, so predict() numerics cannot have moved; the
roadmap support matrix is deliberately left alone.
The GPU offload section told readers to confirm a Metal build by looking for
'clip_ctx: CLIP using GPU backend' next to the vla banner. Nothing in the VLA
path prints that line: each arch calls backend_init once and runs the vision
tower on the same backend, so there is no separate CLIP context to log. Anyone
following the old steps concludes a working Metal build failed. The banner is
also arch-tagged, e.g. vla(pi0), and only SmolVLA logs under a bare vla tag.

Verified by running vla-bench on this machine: SmolVLA prints 'vla: backend =
Metal' and pi0 prints 'vla(pi0): backend = Metal', with zero clip_ctx or CLIP
lines on stdout or stderr.

Drop the '--n-gpu-layers 0' instruction: that flag is parsed only by vlm-server
on the VLM path, and neither vla-cli, vla-server nor vla-bench accepts it.
VLA_DEVICE does not help either, since it only picks a CUDA or SYCL ordinal.
Configuring with -DGGML_METAL=OFF is the only way onto the CPU; confirmed that
it leaves GGML_USE_METAL out of vla_core's compile flags, so backend_init falls
through to ggml_backend_cpu_init.

Note that BitVLA reports a CPU backend even on a Metal build, by design, and
that its int2-packed GGUFs fail to load outside a CUDA build, so the banner is
not mistaken for a broken setup.

Point the Results section at the README's vla-bench table for current numbers
and mark the existing M4 figures as older server-path measurements rather than
leaving the two sets to be compared directly. Record that protobuf, zeromq and
cppzmq are hard configure-time requirements, and that -hf needs the hf CLI.
@Geekgineer Geekgineer changed the title add Apple Silicon (Metal) latency table to the benchmarks section docs: add Apple Silicon (Metal) benchmarks and fix the macOS Metal doc Aug 13, 2026
Flips π0.5, GR00T N1.5, GR00T N1.6, Evo-1, VLA-Adapter, OpenVLA-OFT and VLA-JEPA
from ~ to Y for Metal. Each loads at stock defaults on an M5 Max, appears in the
new Apple Silicon latency table, and was diffed against the CPU backend with
vla_predict_check on fixed images, tokens, state and noise: the same tree built
twice, once as configured and once with -DGGML_METAL=OFF.

Worst deviation among the seven is 6.9e-3 absolute on actions peaking near 1.0,
RMS 1.7e-3 - the BF16/F32 kernel rounding that docs/backend/sycl.md already
documents for SYCL, where the same comparison against CPU is what backs its Y
cells. For scale, π0 and GR00T N1.7 were already Y and deviate further, 1.9e-2
and 1.4e-2, from compounding per-step rounding in their denoise loops; none of
the cells moved here is looser than those. Metal output is bit-identical across
repeat runs, so the deviation is rounding, not instability.

BitVLA keeps ~ for Metal. It is not untested but structurally excluded: the graph
is CPU-only by design and the published GGUFs are int2-packed for CUDA.

Records the verification in docs/backend/metal.md alongside the latency pointer,
matching how the SYCL doc reports it, and notes under the matrix what the Metal
column now rests on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant