anemll-qwen35

Qwen 3.5 on Apple Neural Engine, with research findings on heterogeneous compute patterns across ANE + GPU on Apple Silicon.

Companion project to ANEMLL.

Related repos:

• Anemll/Anemll — ANE LLM toolchain we build alongside
• shipstuff/anemll-profile — profiling toolkit, includes our ane-costplan (MLComputePlan-based compiler cost analyzer)

What this project contains:

1. Qwen3.5 hybrid-attention port (GatedDeltaNet + full attention in DDD A pattern). New architecture not in ANEMLL's stock converters. Works for 0.8B and 9B dense variants.
2. Model-agnostic optimizations validated across sizes:
   • Super-block fusion (+39% on 0.8B, +21% on 9B)
   • Context length right-sizing (+32% at ctx=256)
   • Combined: 49 → 67 tok/s end-to-end on 0.8B (+36%)
3. Cross-compute research (Exp A-E):
   • ANE + MLX parallel pattern for foreground/background workloads
   • 35B MoE on GPU + 0.8B on ANE = 125 tok/s combined, 86% foreground preserved
   • Power, handoff cost, and split strategy measurements
4. Documented negative results: speculative decoding (cross-model and expert-skip), heterogeneous single-model offload, pre-expanded KV cache.

Companion tool: anemll-profile provides both measured per-op runtime (anemll-profile) and compiler cost model analysis (ane-costplan).

Tested on: Mac mini M4 Pro, 64 GB (mini-02).

Results

Model	Config	tok/s	Notes
Qwen3.5-0.8B ANE	full fusion + ctx=256	66.7	recommended: best speed + quality
Qwen3.5-0.8B ANE	baseline (unfused, ctx=1024)	49.2
Qwen3.5-0.8B MLX 4bit	GPU	342.7
Qwen3.5-9B ANE	unfused baseline	11.4
Qwen3.5-9B ANE	2-sb fusion (projected)	~13.6	+19% from fusion
Qwen3.5-9B MLX 4bit	GPU	44.7
Qwen3.5-35B-A3B MoE MLX 4bit	GPU	91.6	stock mlx-lm, no custom code
35B GPU + 0.8B ANE parallel	ANE+GPU	79 + 46 = 125	target use case: 86% foreground preserved

Key insight: 0.8B on ANE (67 tok/s) matches 9B on MLX GPU (45 tok/s) in absolute throughput while keeping the GPU completely free. For background tool-call agents this is the sweet spot.

Quality note: full fusion uses LUT4 for the lmhead (not LUT6). Counterintuitively this produces much better generation — correct fibonacci, correct Pythagorean theorem, coherent text. LUT6 trapped greedy decode in repetition loops.

Quick start

# ── 0.8B recommended (full fusion + ctx=256, best quality) ──────
# First run: converts all components (~5 min)
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/test_full_fusion.py --ctx 256

# Subsequent runs: skip conversion
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/test_full_fusion.py --ctx 256 --skip-convert

# ── 0.8B alternative (SB-only fusion, separate LUT6 lmhead) ────
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/test_fused_chain.py --ctx 256 --n-fuse 6

# ── 0.8B baseline chain (unfused, ctx=1024) ─────────────────────
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/chain_runner_small.py \
  --ckpt /Users/carl/models/qwen3.5-0.8b-textonly \
  --models-dir /Users/carl/models/anemll-qwen3.5-0.8b \
  --prompt "The capital of France is" --max-new-tokens 50

# ── 9B chain (unfused, ctx=1024) ────────────────────────────────
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/chain_runner_kv.py \
  --prompt "The capital of France is" --max-new-tokens 20

# ── MLX GPU reference (0.8B) ────────────────────────────────────
/Users/carl/models/mlx-venv/bin/python -m mlx_lm generate \
  --model /Users/carl/models/qwen3.5-0.8b-mlx-4bit \
  --prompt "The capital of France is" --max-tokens 50

# ── MLX GPU reference (9B) ──────────────────────────────────────
/Users/carl/models/mlx-venv/bin/python -m mlx_lm generate \
  --model /Users/carl/models/qwen3.5-9b-mlx-4bit \
  --prompt "The capital of France is" --max-tokens 50 --temp 0

Conversion (from scratch)

# 1. Strip VL wrapper from HF checkpoint
/Users/carl/projects/anemll/env-anemll/bin/python scripts/strip_vl.py \
  --src /Users/Shared/models/huggingface/hub/models--Qwen--Qwen3.5-0.8B \
  --dst /Users/carl/models/qwen3.5-0.8b-textonly --dtype bfloat16

# 2. Convert all components (embed + super-blocks + lmhead)
#    Auto-detects num_super_blocks from config.json
#    Auto-tunes lmhead split factor via profile sweep
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/convert_small_qwen35.py \
  --ckpt /Users/carl/models/qwen3.5-0.8b-textonly \
  --out-dir /Users/carl/models/anemll-qwen3.5-0.8b

# 3. Run generation
/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/chain_runner_small.py \
  --ckpt /Users/carl/models/qwen3.5-0.8b-textonly \
  --models-dir /Users/carl/models/anemll-qwen3.5-0.8b \
  --prompt "The capital of France is" --max-new-tokens 50

Same process works for 9B — just change the paths.

Optimizations

Two model-agnostic optimizations that scale across sizes:

Super-block fusion

Fuse N super-blocks into one CoreML model. Eliminates predict() dispatch boundaries so ANE sees the full graph and can pipeline across super-block boundaries + keep intermediates in on-chip buffers.

# Test fusion sweep (profiles 1/2/3/6-sb configurations)
/Users/carl/projects/anemll/env-anemll/bin/python scripts/test_sb_fusion.py

Model	1-sb (baseline)	Best fusion	Gain
0.8B	3.60 ms/sb	2.11 ms/sb (2-sb)	39%
9B	10.54 ms/sb	8.26 ms/sb (2-sb)	21%

Gain is larger on smaller models where dispatch overhead is a bigger fraction.

Context length tuning

Smaller CONTEXT_LEN = less KV cache data movement per token. For tool-call use cases, 256 tokens is plenty.

# Test context length sweep
/Users/carl/projects/anemll/env-anemll/bin/python scripts/sweep_context_len.py

ctx_len	ms/sb	vs 1024
128	1.54	2.06×
256	2.41	1.32×
512	3.19	1.00× (plateaus)
1024	3.19	baseline

Combined (validated end-to-end)

/Users/carl/projects/anemll/env-anemll/bin/python \
  scripts/test_fused_chain.py --ctx 256 --n-fuse 6

Result: 66.7 tok/s (from 49.2 baseline, +36%)

Profiling tools

# anemll-profile: measured runtime, bandwidth, per-op timing
/Users/carl/projects/anemll-profile/anemll-profile -j report.json model.mlmodelc

# ane-costplan: Apple's compiler cost weights, per-op device placement
/Users/carl/projects/anemll-profile/ane-costplan -j report.json model.mlmodelc

Use both together: anemll-profile shows what actually runs slow, ane-costplan shows what the compiler thinks is expensive. Disagreements reveal optimization opportunities.

Experiments

Exp	Result	Write-up
A — handoff cost	Zero overhead between CoreML models	notes/exp_a_handoff_cost.md
B — ANE/MLX split	18.75 tok/s (+65% vs pure ANE)	notes/exp_b_ane_mlx_split.md
D — power	ANE ~4.9W, MLX ~14.2W sustained	notes/exp_d_power_results.md
E — parallel (4bit/8bit/bf16)	ANE preserves foreground at 89% of solo vs 82% for 2×MLX	notes/exp_e_parallel_results.md
0.8B port	49→67 tok/s with optimizations	notes/small_model_0_8b.md
Cost analysis	ane-costplan tool + per-op findings	notes/ane_cost_analysis.md
35B MoE parallel	35B GPU + 0.8B ANE = 125 tok/s combined	notes/35b_moe_parallel.md

Ruled-out directions (saved research time)

• Speculative decoding with 0.8B as draft for 9B: 12% acceptance rate. The models generate completely different text despite same architecture family. Would need a purpose-built distilled draft model.
• Expert-skip speculative decoding on 35B MoE: 4.2% acceptance. Selected experts are load-bearing, not refinement — shared-expert-only produces degenerate output.
• Heterogeneous ANE+GPU for single-model speedup: GPU is ~11× faster per layer than ANE at hidden=2048+. Splitting makes it slower.
• Pre-expanded KV cache: 25% per-op, -33% end-to-end (bigger cache costs more than transposes save).
• Mixed precision FP16ComputePrecision(op_selector): no placement retains ANE while recovering precision.

Docs

File	Purpose
STATUS.md	Current numbers, component breakdown, speed table
CLAUDE.md	Project memory for Claude Code (loaded automatically)
PORT_PLAN.md	Architecture deep-dive of Qwen3.5 hybrid attention
notes/	Per-experiment writeups, cost analysis, probe results

Environments

Venv	Python	Use for
~/projects/anemll/env-anemll/	3.9.6	Everything CoreML (conversion, chain runners, profiling)
~/models/mlx-venv/	3.11	MLX reference (mlx-lm ≥0.30 needs py≥3.10)

Layout

scripts/       # conversion, chain runners, optimizations, experiments
model/         # Qwen35ForCausalLM PyTorch scaffold
converter/     # (stub) ANEMLL-style converter
notes/         # experiment writeups + probe results