Adaptive inference for Apple Silicon. A local inference framework with MLX execution and DeepInfer-inspired scheduling.
中文 · Website · Recorded demo · Compatibility · Contribute
Run a model from your terminal, connect a streamed chat API, or use the bundled browser workspace. Explore execution that measures its own draft, verification and recovery cost. Qwen3.5-9B is our primary demonstration model; the framework is built around model and cache capabilities.
Actual Qwen3.5-9B-8bit request on M4 Pro / 24 GiB. Replay the recorded SSE response; the website runs no inference.
- One local workflow. CLI, Python, a supported OpenAI-compatible chat API and local UI share the same engine.
- State that carries forward. Bounded context reuse, native MLX-LM LRU and exact hybrid-state snapshots preserve the consumed prefix.
- Scheduling you can inspect. The adaptive path connects empirical acceptance and full-cycle cost to a DeepInfer-derived budget decision, checked against the original C++ reference.
The 0.1.0a3 local release candidate defaults serve to native MLX-LM batching, with serial capability fallback. CLI, streamed API and the bundled browser UI are ready to run. Release entry and verification · Execution details and measurements.
Use an Apple Silicon Mac with Metal, Python 3.12/3.13 and uv. Model weights download separately at locked revisions. Allow memory for weights, active state, snapshots and macOS.
git clone --branch main --single-branch https://github.com/ZEVIN17/deepinfer-mac.git
cd deepinfer-mac
uv sync --frozen --extra mlx
uv run deepinfer-mac doctor
uv run deepinfer-mac models --download qwen3.5-9b-8bit --target-only
uv run deepinfer-mac run --profile qwen3.5-9b-8bit --mode baseline 'Explain unified memory.'
uv run deepinfer-mac serveOpen http://127.0.0.1:8765 and select qwen3.5-9b-8bit. The default profile is the smaller qwen3-4b regression model. Baseline supports temperature, top-p, top-k, presence penalty and a seed. Run uv run deepinfer-mac models --json to inspect profile metadata.
These commands use the current main branch. Existing checkout: git switch main, git pull --ff-only, then repeat uv sync --frozen --extra mlx. Preserve local changes before updating. v0.1.0a1 is the historical published release; 0.1.0a3 is the current local candidate, with no assumed PyPI publication.
from deepinfer_mac.engine import InferenceEngine
from deepinfer_mac.state import GenerationRequest
engine = InferenceEngine()
try:
result = engine.generate(GenerationRequest(
profile="qwen3.5-9b-8bit", prompt="Explain unified memory.",
mode="baseline", max_tokens=128,
))
print(result.text)
finally:
engine.close()With serve running:
curl http://127.0.0.1:8765/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"qwen3.5-9b-8bit","messages":[{"role":"user","content":"Hello"}],"max_tokens":128,"temperature":0,"stream":true}'The service implements a documented chat API subset, bounded queuing, cancellation and streaming. Default service uses baseline; explicit configurations connect bounded multi-request execution and joint target verification with per-request cancellation and recovery.
The capability adapter accepts local model directories or exact repository revisions. SmolLM2-135M-Instruct-8bit has passed real baseline/native token matching, dense cache reuse, CLI generation, API and SSE acceptance. This verifies a new model family through the shared interface; other custom models require their own acceptance. For an existing complete MLX model directory:
uv run deepinfer-mac run --model /path/to/complete-mlx-model \
--model-id my-local-model --max-tokens 32 'Hello'Registration is explicit and per process; custom models start with unverified baseline compatibility, with unsupported optimizations rejected. A JSON configuration can register the same ID for CLI, Python and API. Adapter reference explains identity, resource guards and explicit fallback.
SmolLM2 acceptance / 验收报告 · Weight identity / 权重核验
“Verified” describes the listed path, not every model or optimization supported by MLX-LM.
| Model / path | Local baseline | Adaptive path | Scope |
|---|---|---|---|
| Qwen3.5-9B-8bit | Verified: CLI / API / UI, sampling | Separate configuration, explicit opt-in | Primary text demonstration; hybrid state |
| Qwen3.5-9B-4bit | Verified: baseline / native controls | Not enabled publicly | Same-model quality / memory control |
| Qwen3-14B-4bit | Verified: baseline / native controls | Validation pending | Dense engineering control |
| Qwen3-4B | Verified regression path | Fixed / adaptive | Smaller dense regression model |
| Configurable model / cache adapters | SmolLM2-135M-8bit baseline / cache / CLI / API / SSE verified | Capability-gated baseline | Model path/repo registration and transparent baseline fallback |
| Joint verification service | 360M target + 135M draft pass 35/35 service checks | Explicit joint-budget / variable-length verification opt-in | Dense research pair; actual token SSE, cancellation isolation and recovery |
Public 9B API/UI modes remain baseline. Hybrid speculation is greedy and uses the explicit configuration in DS method mapping. Model loading is distinct from task quality and optimization validation. See model selection, compatibility and immutable model locks.
| Real workload | Result | Evidence |
|---|---|---|
| Small-model joint API service | 35/35 real-model service checks; 330 portable + 48 Metal tests pass | Service report |
| 9B adaptive development trace | 28/28 mature decisions match original C++; 128 output tokens equal baseline | Supplemental verification |
| 9B context reuse | 20 requests; 16/16 complete-token checks pass across two sampler settings | Cache report |
| 9B baseline on M4 Pro / 24 GiB | 4,204 input + 256 output tokens; no measured swap increase for this run | Resource check |
Each milestone retains its model, hardware and workload provenance. Benchmark protocol · Complete measurements and technical scope.
The default service builds on MLX-LM native batching with bounded admission, streaming, cancellation isolation and recovery. This version passes 364 portable + 95 Metal tests, alongside 46/46 actual-model service checks. A separately installed wheel has completed real-model CLI, API, SSE and bundled UI acceptance.
The recorded three-process native paired-task median is 124.09 ms on the audited SmolLM2 pair development workload. Service report · Complete measurement · Installed-wheel checks.
CLI / Python / chat API / local UI
│
model + cache capabilities
│
MLX / MLX-LM execution
│
draft → target verify → exact recover
↑ observed acceptance + full cost │
└──── DeepInfer-derived budget ──────┘
The scheduling path now connects joint request budgets and real variable-length target verification to a bounded API service. Per-request state, committed token events, cancellation isolation and recovery have independent acceptance; complete-cost calibration and trace-driven overlap continue under the research protocol. Build the local showcase to explore its interactive budget diagram: explicitly a mechanism illustration, separate from measured traces.
MLX / MLX-LM provide model execution, sampling and native LRU. DeepInfer supplies the reference scheduling methods; hybrid recovery and service integration are Mac adaptations. This independent community project is not a full DSpark / DeepSeek V4.1 port. Architecture · Source mapping · NOTICE · Apache-2.0.
uv sync --frozen --extra mlx --extra dev
uv run pytest -q
uv run pytest -q --run-metal -m metal
uv run python site/build.py
uv run python site/check.pyContribution guide · Release notes · Evidence index · Phase 3 execution brief
Contribute a model adapter, a Mac hardware measurement or a real application. Bring a reproducible command and help make local inference easier to build on.