WIP: wire MLX engine into mesh-llm serve (safetensors routing) by michaelneale · Pull Request #1010 · Mesh-LLM/mesh-llm

michaelneale · 2026-07-16T21:17:47Z

What this is

WIP integration that wires the MLX (Metal) serving engine into mesh-llm serve,
stacked on top of the standalone engine crate from #1009. On a Mac,
mesh-llm serve --model <hf-safetensors> routes the model to the MLX engine and
it shows up in /v1/models — with non-macOS / no-feature builds unaffected.

Status: functionally complete and verified locally, but blocked on one repo
publish invariant that needs a maintainer decision. Opening as a draft to
capture the working integration and the reasoning so it isn't lost. Not for
merge as-is.

Full detail: crates/skippy-engine-mlx/SERVE_INTEGRATION_STATUS.md.

What works (verified on Apple Silicon)

skippy-engine-mlx promoted to a real workspace member (out of
default-members; both CI crate-list scripts updated; xtask repo-consistency ci-crate-lists passes).
mlx feature on mesh-llm-host-runtime + mesh-llm, wired as a
macOS-target-gated optional dep.
inference/mlx.rs: MlxModelHandle + MlxHttpHandle serving over the real
openai-frontend::router_for + axum.
LocalRuntimeBackendHandle::Mlx variant + all match arms;
start_runtime_mlx_model + is_safetensors_model_path route safetensors
models to MLX before the GGUF planning path.
cargo build -p mesh-llm --features mlx → exit 0. No-feature/default builds
clean; cargo tree confirms safemlx is absent unless --features mlx.
fmt + clippy clean with and without the feature.

Architecture

Two-native-stack link collision (found and resolved): statically linking MLX
next to the patched llama.cpp fails with duplicate symbol: gguf_get_key — MLX
vendors antirez's gguflib while the patched llama.cpp exports the same GGUF
symbols. Fixed by making the mlx feature imply dynamic-native-runtime, so the
llama.cpp runtime loads as a dylib (as release builds already do) and the two
GGUF parsers live in separate link units.

Protocol / publishing

Blocker: xtask repo-consistency release-targets fails —

mesh-llm-host-runtime: publishable crate depends on non-publishable
skippy-engine-mlx

skippy-engine-mlx git-pins safemlx (crates.io's published safemlx produces
gibberish for dense models; only the git commit serves correctly). crates.io
forbids git deps → the crate is publish = false. But mesh-llm-host-runtime is
a published SDK crate, and the repo forbids published crates depending on
unpublishable ones (a real cargo publish constraint). This is not solved
elsewhere in the repo — the other publish = false crates are only consumed by
the non-published mesh-llm binary.

Options (see the status doc):

A. [patch.crates-io] redirect safemlx→git (the existing hf-hub
precedent) + make the crate publishable + add to publish-crates.sh.
B. Hold this wiring; ship the standalone crate (long term effort: Serve safetensors models on Metal via MLX (Apple Silicon) #1009) until safemlx cuts a
working crates.io release, then flip git-pin → version-pin and merge (trivial).
C. Loosen the xtask invariant (not recommended; would allow a manifest that
can't cargo publish).

The clean long-term unlock is a working safemlx crates.io release, after which
this becomes a git-pin → version-pin swap and the invariant is satisfied
automatically.

Validation

export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
cargo build -p mesh-llm --features mlx                     # exit 0
cargo check -p mesh-llm-host-runtime                       # no-feature: clean
cargo run -p xtask -- repo-consistency ci-crate-lists      # PASS
cargo run -p xtask -- repo-consistency release-targets     # FAILS (the blocker)

Design proposal evaluating Apple MLX (via safemlx/safemlx-lm) as an additive second engine behind a StageEngine trait. Covers the staged execution boundary, safemlx-lm layer-split seam, JIT-quant-from-safetensors artifact strategy, selective download, platform/dependency footprint, and a solo-first phased plan gated on partial-load and boundary-fence spikes.

Standalone spikes/mlx-solo crate (own workspace, NOT in the mesh-llm workspace) that loads an HF safetensors model via safemlx-lm and generates tokens in Rust. Results (Qwen3-0.6B, Apple Silicon CPU-only, no Metal compiler on this box): - source precision: 18.1 tok/s decode, coherent — no GGUF, no ahead-of-time quant - JIT 4/8-bit affine: correct but ~0.4 tok/s (MLX quant matmul is Metal-optimized; no fast CPU kernel) -> JIT quant must be gated behind Metal/CUDA Also validates MLX C++ builds under cmake 4.4 CPU-only (the #1 feared build risk). Findings folded into the plan (status, Phase 2, risks). Full writeup and the required safemlx fork edits are in spikes/mlx-solo/FINDINGS.md.

Redo of the solo spike the goose way (metal backend, Device::Gpu) after installing the Metal toolchain (Xcode 26.6 + MetalToolchain component). Metal results (Qwen3-0.6B, Apple Silicon): - source precision bf16: 321 tok/s decode, coherent, ZERO fork patches - pre-quantized 4-bit mlx-community repo: 603 tok/s - JIT 4-bit quant on load: 604 tok/s Headline: JIT-quantize-on-load (604) == pre-quantized artifact (603), so quantizing on load is free at inference time. Supersedes the earlier CPU-only run (18/0.4 tok/s), which was an unrepresentative CPU-kernel artifact. The goose baseline (source precision) needs no fork changes. Two small safemlx-lm fixes are only needed to go beyond it (JIT quant of a tied-embedding checkpoint; loading published quant repos that omit config mode) and are upstream-PR candidates for jbg/safemlx, not mesh-llm drift. Details + repro in spikes/mlx-solo/FINDINGS.md.

…le yet - ROCm is a real but unmerged upstream MLX experiment (PR #2300), not 'no signal'; Vulkan is wishlist-only; hardware coverage has two gates (upstream mlx -> safemlx). - safemlx supply chain: published crates.io 0.4.1 collides version strings with a different, older codebase than fork HEAD 0.4.1 (851 vs 2221 lines qwen3.rs). Fork-free build ran but produced gibberish for Qwen3 + crashed on pre-quant repo; working dense-model + JIT-quant code is fork-HEAD-only. Must pin a git rev. - Adds spikes/mlx-solo-published (crates.io-only) demonstrating the breakage.

Adds crates/skippy-engine-mlx: a working MLX serving engine that serves HF safetensors models over mesh-llm's REAL OpenAI frontend (openai-frontend router_for), on Apple Silicon. Verified against Qwen3-0.6B: - GET /v1/models lists the model - POST /v1/chat/completions returns a real generation with usage - streaming returns proper SSE (role + content deltas + final finish_reason) Design: a dedicated OS worker thread owns the non-Send MLX objects (model, streams, arrays) and communicates via Send channels; this also serializes GPU access. MlxBackend implements OpenAiBackend; incremental detokenization via decode-prefix-diff. Reuses safemlx-lm (pinned fork checkout) and ports goose's generation-loop patterns rather than depending on goose-local-inference (which force-compiles a second llama.cpp and is pinned to a safemlx version proven broken for Qwen3). Standalone by design: own [workspace], not a main-workspace member, so it does not perturb the main build or CI (verified: cargo metadata on the main workspace still resolves and excludes this crate). All MLX code is gated behind both the mlx feature and target_os=macos. WIRING.md documents the promotion path into the shipped binary: git-pin safemlx, add an Mlx variant to LocalRuntimeBackendHandle, route ModelFormat::Safetensors to the MLX engine at launch, and auto-enable on macOS. Model discovery/listing already handles MLX safetensors; the missing piece was the serving engine. Refs branch micn/mlx-redux.

Pins safemlx/safemlx-lm to a public commit of jbg/safemlx (4e53c5e) instead of a path-dep on a local fork, and switches to goose-style plain source-precision serving (drops JIT quantization). Why the git-rev pin (not crates.io): safemlx's published crates (0.1.5 and 0.4.1) emit repeated-token gibberish for dense models (Qwen3 AND Llama) with this exact crate code, while the pinned upstream commit serves them correctly. Verified to be a library-version issue, not prompting: the Qwen3 chat template is confirmed applied and greedy sampling (temp=0 -> argmax) is used; swapping only the safemlx version flips output coherent<->gibberish. No private patches (plain LoadedModel::load avoids the quant-path loader quirks entirely). Will swap to a version pin once safemlx cuts a working dense-model release. Verified on Apple Silicon (Metal) over the real openai-frontend router: - Qwen3-0.6B: coherent non-stream + streaming (19 SSE chunks) - SmolLM2 (Llama arch): coherent WIRING.md updated with the git-pin rationale, the published-is-broken finding, and a Linux+NVIDIA (CUDA) future note. Crate stays a standalone workspace so the heavy MLX native build never runs in unrelated builds/CI.

Stacks the serve-integration on top of the standalone MLX engine crate. On a Mac, routes safetensors models to the MLX (Metal) engine over the real openai-frontend; non-macOS / no-feature builds unaffected. Done & verified locally (Apple Silicon): - skippy-engine-mlx is a workspace member (out of default-members); CI crate lists updated; xtask ci-crate-lists passes. - `mlx` feature on mesh-llm-host-runtime + mesh-llm as a macOS-target-gated optional dep; implies dynamic-native-runtime. - inference/mlx.rs (MlxModelHandle + MlxHttpHandle over openai-frontend router_for); LocalRuntimeBackendHandle::Mlx variant + arms; start_runtime_mlx_model + is_safetensors_model_path route safetensors before the GGUF path. - cargo build -p mesh-llm --features mlx -> exit 0. no-feature/default clean. fmt + clippy clean with and without the feature. Link gate resolved: static MLX + patched llama.cpp collide on gguf_get_key (MLX vendors antirez gguflib; llama.cpp exports the same). Fixed by making the `mlx` feature imply dynamic-native-runtime so llama.cpp loads as a dylib. BLOCKED: xtask release-targets forbids the published mesh-llm-host-runtime from depending on non-publishable skippy-engine-mlx (git-pinned safemlx -> publish=false; crates.io safemlx is broken for dense models). See crates/skippy-engine-mlx/SERVE_INTEGRATION_STATUS.md for options and reasoning. Not for merge as-is; captures the working integration + the publish-invariant decision for a maintainer.

coderabbitai · 2026-07-16T21:17:56Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fd916d4c-61a5-4c40-80a0-dbda53d98d44

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch micn/mlx-serve-wiring

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-16T21:17:58Z

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

michaelneale · 2026-07-16T21:34:43Z

Consolidating into #1009 — did not mean to open two PRs. The serve-integration wiring is being folded into the single branch.

michaelneale added 7 commits July 16, 2026 12:16

michaelneale closed this Jul 16, 2026

michaelneale deleted the micn/mlx-serve-wiring branch July 16, 2026 21:35

Navigation Menu

Uh oh!

Uh oh!

WIP: wire MLX engine into mesh-llm serve (safetensors routing) - #1010

WIP: wire MLX engine into mesh-llm serve (safetensors routing)#1010
michaelneale wants to merge 7 commits into
mainfrom
micn/mlx-serve-wiring

michaelneale commented Jul 16, 2026

Uh oh!

coderabbitai Bot commented Jul 16, 2026

Review skipped

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

michaelneale commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

michaelneale commented Jul 16, 2026

What this is

What works (verified on Apple Silicon)

Architecture

Protocol / publishing

Validation

Uh oh!

coderabbitai Bot commented Jul 16, 2026

Review skipped

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

michaelneale commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant