Add Qwen 3.6 35B-A3B as a second supported architecture by NeelM0906 · Pull Request #29 · drumih/turbo-fieldfare

NeelM0906 · 2026-07-31T11:26:12Z

Add Qwen 3.6 35B-A3B as a second supported architecture

Adds text-only support for
Qwen 3.6 35B-A3B from the pinned
mlx-community/Qwen3.6-35B-A3B-4bit checkpoint, alongside Gemma 4 26B-A4B and
under the same bounded-memory contract. The runtime supports the two
architectures by explicit enumeration — each with its own pinned checkpoint,
compile-time baseline, and manifest contract.

Gemma 4 behavior is unchanged. The architecture is selected from
manifest.json -> arch.family; manifests without that field are Gemma 4, so
existing installations load exactly as before.

Why this model fits the runtime

Qwen 3.6 is a 40-layer hybrid: 30 gated-DeltaNet linear-attention layers and 10
gated full-attention layers (every 4th), with 256 routed experts per layer
(top-8) plus a sigmoid-gated shared expert. About 3B parameters activate per
token, the same class of per-token work as Gemma 4's 3.88B.

Its resident footprint is smaller than Gemma 4's, because its experts are
half the size and only a quarter of its layers keep a KV cache:

Component	Qwen 3.6 35B-A3B	Gemma 4 26B-A4B
Common weights (mapped)	1.39 GB	1.35 GB
Routed-expert slots, 16/layer	1.13 GB	1.61 GB
KV cache at 4K	84 MB	320 MB
Gated-DeltaNet recurrent state	64 MB	—
Expert files on disk	18.1 GB	12.9 GB

Only the on-disk install is larger: about 19.6 GB against 14.3 GB.

Measured results

M5, 24 GB, macOS 26.5, Swift 6.2, following the
community benchmark protocol: the three frozen
real-generation-v1 prompts with their fixed seeds, app sampling defaults
(temperature 0.2, Top-K 64, Top-P 0.95), 4K context, 16 expert-cache
slots, one discarded warmup, then one measured run per case in a fresh process.
Every measured footer reported stop=endOfTurn.

Case	Prompt / generated tokens	Prefill	Decode	Peak RSS / footprint
short-explanation	62 / 493	7.74 s	23.05 tok/s	1,139 / 1,447 MiB
medium-review	426 / 697	12.71 s	21.20 tok/s	1,142 / 1,448 MiB
long-synthesis	2,940 / 700	59.16 s	18.84 tok/s	1,093 / 1,464 MiB

Bounded memory under an 8 GB working set

16 GB of the host was pinned resident by a separate process, leaving about
8 GB for the OS, page cache, and the model. All three cases were rerun
unchanged in fresh processes with the same seeds:

Case	Decode, 24 GB	Decode, ~8 GB	Footprint, 24 GB	Footprint, ~8 GB	Output
short-explanation	23.05 tok/s	22.95 tok/s	1,447 MiB	1,464 MiB	byte-identical
medium-review	21.20 tok/s	21.35 tok/s	1,448 MiB	1,448 MiB	byte-identical
long-synthesis	18.84 tok/s	18.62 tok/s	1,464 MiB	1,388 MiB	byte-identical

Every case still reported stop=endOfTurn.

Unchanged within noise. Repeated runs at full memory do not speed up either
(21.4, 22.1, 20.8 tok/s back to back), so Qwen's throughput does not depend on
the OS page cache on this host — its expert reads already come from SSD, and
removing memory does not change what it reads.

This is emulated pressure on M5 hardware, not a physical 8 GB Mac. A real
8 GB machine has a slower SSD and GPU and should be expected to decode more
slowly, as the existing 8 GB M2 rows show for Gemma 4.

What changed

Architecture plumbing. ArchConfig gains a ModelFamily discriminator and
family-dependent fields (attention output gate, attention scale, embedding
scaling, router scaling, FFN sandwich norms, gated shared expert, partial-RoPE
convention, gated-DeltaNet dimensions). fullAttentionLayerMask gains a third
value for linear-attention layers. ManifestReader decodes the new fields as
optional so existing Gemma manifests validate unchanged, and Model.load now
detects the family from the manifest.

Installer. TurboFieldfareRepack --model qwen36 streams the pinned
checkpoint into the .gturbo layout — per-family tensor classification, the
stacked switch_mlp expert tensors sliced per expert, the vision tower
omitted, and the new arch fields written to manifest.json.

Kernels. A new gdn.metal module implements the gated delta rule: causal
depthwise conv with SiLU, per-head q/k normalization with the delta-rule scales
folded in, the recurrence (decode step and chunked prefill), and the gated
output norm. Also a partial-RoPE variant confined to the first rotary_dim
elements, an attention output gate, SiLU selection for the expert FFNs via
function constants, and a fused four-way input projection.

Runtime. RealForwardRunner dispatches three layer kinds. Linear layers
carry a fixed recurrent state (GDNStateManager) instead of per-token K/V, so
only the 10 full-attention layers grow with context.

Tokenizer. A ChatML dialect resolved from the loaded tokenizer's special
tokens, Qwen's <tool_call> XML function-call parser, and think-tag handling.
The Gemma dialect is untouched.

Products. The CLI works with either model; the server derives its model id
and validation dialect from the loaded model; the Mac app installs Qwen when
selected.

Correctness

The gated-DeltaNet kernels are validated against a straight-line CPU
reference of the mlx-vlm math, including that a chunked prefill of T rows
matches T sequential decode steps through the same kernels (state and
conv-tail carry).
The fused input projection is asserted bit-identical to the four separate
GEMVs it replaces.
Real-model greedy output was verified identical across the optimization
changes.
Scripts/test.sh: 570 tests in 114 suites pass, including the pre-existing
Gemma suites unchanged.
swift build -c release and ruby Scripts/check_markdown_links.rb pass.

Limitations

Vision input is omitted, as it is for Gemma 4.
Acceptance evidence covers 4K context. Longer contexts are untested.
The 8 GB figure is emulated pressure on M5 hardware, not a physical 8 GB Mac.
The server's tool-call loop is covered by unit tests but has not been
exercised against a live client.
Decode is expert-I/O bound (53% of the token) on this host, and slower than
Gemma 4's published rows on comparable hardware. The likely cause is stated
as a hypothesis, not a finding: Gemma 4 was not run under the same protocol
on this host, so the comparison is not instrumented. Profiling, including the
negative results, is in
docs/QWEN36_PERFORMANCE.md.

- ArchConfig: ModelFamily (gemma4/qwen36), LinearAttentionConfig, qwen36_35B_A3B baseline, layer-mask value 2 for gated-DeltaNet linear attention layers - ManifestReader: optional family-extension arch fields, family peek for auto-detection, arch-registry production sniff - Model: family-aware tensor accessors (router, shared expert, scalar gate, linear_attn.*), untied lm_head, auto-detecting load overload - KVCacheManager: LayerKind.linear (no per-token KV); GDNStateManager for fp32 delta state + conv tails - Sampler/logit path: finalLogitSoftcap plumbed through, softcap <= 0 disables - New gdn.metal module: causal depthwise conv + SiLU (decode/prefill/tail), per-head q/k norm with folded delta scales, gated delta rule recurrence (decode + chunked prefill), gated output norm; validated against a CPU reference incl. prefill == sequential decode equivalence - rope_neox_subdim (decode + prefill): Qwen partial-rotary convention - silu activation function constants for routed MoE, prefill MoE, INT8 shared expert; silu_mul_fp16; sigmoid gate/scalar and residual-add elementwise kernels; MoE/SharedExpert/PrefillGroupedRoutedMoE wrappers parameterized

…source) - SupportedModelSource: selectable gemma4/qwen36 pinned sources with per-source modelID, download/installed byte estimates - SourceFingerprint: qwen index sha -> qwen3.6-35b-a3b-4bit - CLI: --model {gemma4|qwen36} selector (default gemma4) - ArchInfo: qwen3_5_moe parse path (layer mask 2=linear/1=full, GDN dims, attn output gate, silu, untied head) + production baseline cross-check - RepackPlanner: per-family tensor classification (.mlp.switch_mlp routed experts) and resident ordering (embed first, lm_head last) - GTurboJSON/RemoteStreamingRepacker: qwen36 family arch fields in manifest.json (gemma output unchanged), bit-width sniffing for qwen names - SyntheticSnapshot.buildQwen + planner/CLI/end-to-end install tests

…ecoder support - Rename GemmaToolCallParserError to neutral ToolCallParserError with a back-compat typealias - Resolve ChatDialect (gemma/chatml) at GFTokenizer init by sniffing the <|im_end|> special token; chatml requires the nine ChatML specials by string, gemma keeps its ten-token contract - ChatML: no BOS (encode addBOS is a no-op), stop tokens {im_end, endoftext}, padded logits vocab 248320, hand-rolled ChatML chat template with enable_thinking=false generation prompt, text continuation bridge - encodeToolResultContinuation throws unsupportedForDialect for chatml so the server prompt cache falls back to prefix matching - New QwenToolCallParser for the <function=/<parameter= XML-ish body with JSON/string value inference, fail-closed tool allow-list, 256 KiB guard - StructuredAssistantDecoder chatml mode: <think> suppression and <tool_call> buffering via special-token IDs - Synthetic ChatML tokenizer fixture + template/parser/decoder test suites

… tests)

Gemma's tokenizer maps unknown strings to <unk> instead of nil, so probing for <|im_end|> misclassified Gemma as ChatML. Verify resolution by round-tripping the ID back to the token string.

…t + model-id) - AppModelInstallDescriptor.qwen36 (pinned repo/revision/index sha, byte estimates from the range plan) + TURBO_FIELDFARE_MODEL=qwen36 selection; install dirs gemma4.gturbo / qwen36.gturbo; badge shows descriptor name - AppModelInstallationProbe validates against the manifest's own family - Server: ChatDialect threaded into request validation (Gemma DSL key/args checks relaxed for ChatML), --model-id default derived from the loaded model family, backend exposes chatDialect/defaultModelID - CLI usage banner covers both models

…refill Decode: three-way per-layer dispatch (gated-DeltaNet linear / gated full attention with NeoX sub-dim RoPE / existing Gemma path), plain pre-norm post-attention block with single moeX norm, sigmoid-gated shared expert, phase-2 residual fusing the shared branch, residual-add tail, untied lm_head. Prefill: chunked GDN path (conv + tail carry + delta recurrence), packed q_proj split, prefill_rope_neox_subdim_block Swift encode, no V norm, gate multiply, ones router scale buffers, residual-add tail; linear layers skip KV staging/blit/attention. Init: family-neutral kernel construction (SiLU MoE/shared/grouped, specialized dims from cfg), GDN + state manager + Elementwise + scalar-gate GEMV gated on data flags, ones effective/per-expert scale buffers when routerScaled is false, embed outScale and attention scale from config. Gemma path byte-identical; all 134 existing tests pass.

QwenToySynthetic writes a tiny runnable .gturbo with the qwen36 tensor contract (linear_attn.* GDN tensors, packed [query;gate] q_proj, mlp.gate router, gated shared expert, untied lm_head, arch-v2 manifest, no sandwich tensors). Tests: runner init without touching sandwich tensors, hybrid linear/full decode smoke with reset determinism, chunked prefill smoke + decode continuation, prefill-vs-decode argmax consistency (KV + GDN state carry), KV/GDN-state reset interplay, prefill scratch layout sizing for qwen vs unchanged gemma.

…l dispatch)

…B peak)

…phase counters Specialization measured on M5: unspecialized 4096x2048 INT4 GEMV runs at 101.6 GB/s, the constant-folded variant at 141.0 GB/s. Shapes now come from ArchConfig rather than a hardcoded Gemma list, and the LM head specializes on the loaded model's D/vocab instead of 2816/262144. Also adds --expert-cache-slots, --rdadvise, and TURBO_FIELDFARE_PHASES=1 decode phase reporting so the I/O-vs-GPU split is measurable.

The probe now derives the expected checkpoint from the family the manifest declares when the caller does not pin a descriptor, so a Gemma install is still recognized while Qwen is the selected model (and vice versa). The post-install verification still pins to the descriptor that was downloaded. Also covers the two new CLI options in the help-options test.

Decode issued four separate INT4 GEMVs per gated-DeltaNet layer, all reading the same hidden vector: in_proj_qkv (8192 rows), in_proj_z (4096), in_proj_a (32) and in_proj_b (32). The last two were four threadgroups apiece — near-empty launches. gdn_in_proj_gemv_simd dispatches over the concatenated 12352-row space (1544 threadgroups at 8 rows/TG) and routes each row to its own weight/scale/bias base and output buffer with a 4-way compare, reusing dequant_int4_gemv_simd_body verbatim. The per-row math and operand order are unchanged, so the result is bit-identical to the four dispatches; greedy decode output on scratch/qwen36.gturbo is byte-for-byte the same. The body's ushort-pair weight loads are kept: packed sub-tensor offsets are only 2-byte aligned. Qwen-only (the GDN path); Gemma 4 is untouched. No new buffers. Measured (M5, 30 layers = 1 decode token, median GPU time): separate x4 3.66 ms -> fused x1 3.42 ms (-6.6%) That stage is ~3.4 ms of a ~59 ms/token decode, so end-to-end tok/s does not move outside run-to-run noise (13.8-18.2 tok/s on an idle baseline). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…hat stage)

…che residency + slot coverage)

CONTRIBUTING requires public runtime controls to appear in docs/RUNTIME_CONTROLS.md; --expert-cache-slots and --rdadvise now do, along with the TURBO_FIELDFARE_PHASES diagnostic. SYSTEM_DESIGN's scope section described a Gemma-only runtime while the branch also ships Qwen 3.6, and now states what that support covers and what it has not been validated against.

Pinning 16 GB of a 24 GB host to leave an ~8 GB working set changes neither throughput (19.7 vs 19.8 tok/s) nor footprint (1.52 vs 1.49 GB peak). The 18.1 GB expert pool never fit the page cache on either configuration, so decode already streams from SSD and constraining memory does not change what it reads. Every resident component is equal to or smaller than the Gemma path's: expert slots 1.13 GB vs 1.61 GB, KV at 4K 84 MB vs 320 MB. Only the on-disk install is larger (19.6 GB vs 14.3 GB). Replaces the earlier claim that the 8 GB configuration was unvalidated.

Ran the three frozen real-generation-v1 prompts with their fixed seeds, app sampling defaults, 4K context and 16 slots — one discarded warmup then one measured run per case in a fresh process. Every measured footer reported stop=endOfTurn. short-explanation 62/493 23.05 tok/s 1,447 MiB footprint medium-review 426/697 21.20 tok/s 1,448 MiB long-synthesis 2,940/700 18.84 tok/s 1,464 MiB Rerunning the short case with 16 GB pinned elsewhere (~8 GB working set) gave 23.36 tok/s at 1,448 MiB with byte-identical output. The CLI footer now also reports prefill seconds, which the protocol's report format asks for.

Qwen 3.6 is presented alongside Gemma 4 rather than as an experiment: the README at-a-glance covers both models' memory, storage and measured decode, and SYSTEM_DESIGN's scope section describes two explicitly enumerated architectures, each with its own pinned checkpoint, compile-time baseline and manifest contract. Completes the 8 GB validation across all three community benchmark cases with 16 GB pinned elsewhere. Every case reported stop=endOfTurn and produced output byte-identical to its unconstrained counterpart: short-explanation 23.05 -> 22.95 tok/s 1,447 -> 1,464 MiB medium-review 21.20 -> 21.35 tok/s 1,448 -> 1,448 MiB long-synthesis 18.84 -> 18.62 tok/s 1,464 -> 1,388 MiB Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The comparison quoted 41.5 tok/s for Gemma 4 from an app HUD reading rather than the repository's own published 31-35 tok/s protocol rows, and derived per-model read bandwidths from an assumed 100% expert cache-miss rate that was never instrumented. Replaced with what is measured — Qwen reads fewer expert bytes per token than Gemma, has half the slot coverage, and its throughput is page-cache independent (repeated runs do not warm up: 21.4, 22.1, 20.8 tok/s; an ~8 GB working set does not slow it down) — and labels the page-cache explanation for the gap as an untested hypothesis, since Gemma was never run under the same protocol on this host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cenote-dev · 2026-08-01T02:04:52Z

Benchmarks: Apple M4 Pro 24GB Tahoe 26.6
Low power mode: Disabled
Swift version 6.3.3

Case	Prompt / generated tokens	Prefill	Decode
short-explanation	62 / 536	7.65 s	27.88 tok/s
medium-review	426 / 667	13.23 s	27.35 tok/s
long-synthesis	2,940 / 564	101.68 s	21.44 tok/s

for case_seed in \
  short-explanation:20260721 \
  medium-review:20260722 \
  long-synthesis:20260723; do
  case_id="${case_seed%%:*}"; seed="${case_seed##*:}"
  .build/release/TurboFieldfareCLI \
    --model scratch/qwen36.gturbo \
    --messages-file "docs/benchmark-prompts/real-generation-v1/${case_id}.json" \
    --max-new 1024 --max-context 4096 \
    --temperature 0.2 --top-k 64 --top-p 0.95 \
    --seed "$seed" \
    > "benchmark-results/measured/${case_id}.stdout" \
    2> "benchmark-results/measured/${case_id}.stderr"
done
grep -h '^\[stop=' benchmark-results/measured/*.stderr | tee benchmark-results/summary.txt

AbhinavPanwarDev · 2026-08-01T12:29:00Z

We need to somehow make this also work for Deepseek V4 flash

jasongoldsmith · 2026-08-03T23:19:49Z

We need to somehow make this also work for Deepseek V4 flash

I was coming to see if that had already been attempted!

Pummelchen · 2026-08-03T23:30:30Z

Forked and merged with enhancedments into:

https://github.com/Pummelchen/NVMAI

Pummelchen · 2026-08-03T23:31:20Z

We need to somehow make this also work for Deepseek V4 flash

I was coming to see if that had already been attempted!

Yes but of course way slower than streaming Qwen3.6 or Gemma 4 models:

https://github.com/antirez/ds4

Pummelchen · 2026-08-03T23:32:39Z

We need to somehow make this also work for Deepseek V4 flash

#29 (comment)

Merges upstream PR #29 (NeelM0906) onto current main. The branch predated the .gturbo v1 format contract, the Gemma-specific decoding pipeline, and the CLI runtime-control flags, so the conflict resolution keeps main's newer machinery and re-lands the Qwen work on top of it: - Format: `GTurboManifestArchV1` gains the optional family extension fields, so Gemma manifests stay byte-identical while Qwen carries its arch facts. Layer masks accept 2 (linear attention), and `slidingWindow` may be 0 when no layer uses a sliding window. - Decoding: main replaced library decode with `GemmaDecoding`, which is wrong for Qwen's ByteLevel tokenizer. Adds `ByteLevelDecoding` + `ByteLevelRun` and routes `GFDetokenizer` on the decoder declared by tokenizer.json. Lossy decoding follows from_utf8_lossy's maximal-subpart rule (verified against Python's decoder for nine byte sequences). - Runtime schema: `validateRuntimeSchema` hardcoded the Gemma tensor contract. Adds the Qwen 3.6 per-layer contract (hybrid linear/full graph, packed [query ; gate] q_proj, gated shared expert, no sandwich norms) and validates an untied lm_head. - Server: Gemma tool-schema adaptation now applies only to the Gemma dialect; ChatML passes schemas through as JSON. - CLI/server flags keep main's richer runtime controls; the Qwen-specific bits (model-id derivation, phase diagnostics) are preserved. 758 tests in 133 suites pass.

ulises-c · 2026-08-25T22:21:41Z

This would be really nice to have. In general support for MoE not just Gemma 4

Pummelchen · 2026-08-25T23:40:37Z

This would be really nice to have. In general support for MoE not just Gemma 4

PR29 merged into a fork:

https://github.com/Pummelchen/NVMAI

NeelM0906 and others added 27 commits July 30, 2026 23:37

Merge branch 'qwen36-support' into worktree-agent-a74ae3c5cca73a8ea

abb2d3c

Merge branch 'qwen36-support' into worktree-agent-a99262c50d80b1547

11027dd

feat: split_q_gate kernel for Qwen attention output gate

ea9cf9a

merge: Qwen repacker support (source, planner, manifest v2, synthetic…

60f18a1

… tests)

fix: reject unk-token fallback when resolving dialect special tokens

7815cc1

Gemma's tokenizer maps unknown strings to <unk> instead of nil, so probing for <|im_end|> misclassified Gemma as ChatML. Verify resolution by round-tripping the ID back to the token string.

merge: ChatML tokenizer dialect, Qwen tool parser, structured decoder

ad4d706

merge: forward-runner integration for Qwen3.6 (hybrid decode + prefil…

15066be

…l dispatch)

docs: document experimental Qwen3.6 35B-A3B support

33e00dc

fix: raise layout.json metadata cap to 64 MiB for 40x256-expert layouts

3f24c2a

docs: first measured Qwen3.6 reference point (M5, 17-18 tok/s, ~1.5 G…

5b31eea

…B peak)

docs: Qwen 3.6 decode profiling findings and negative results

0d9ff00

merge: fused GDN four-way input projection (bit-identical, -6.6% on t…

ca47049

…hat stage)

docs: record why Qwen decodes slower than Gemma on this host (page-ca…

e60e8a6

…che residency + slot coverage)

Pummelchen mentioned this pull request Aug 11, 2026

NVMAI: Qwen 3.6 35B-A3B port with measured decode optimizations #105

Open

ulises-c mentioned this pull request Aug 28, 2026

feat: Qwen 3.6 35B-A3B as a second model family (Stage 1: correctness) ulises-c/turbo-fieldfare#3

Merged

Navigation Menu

Add Qwen 3.6 35B-A3B as a second supported architecture - #29

Add Qwen 3.6 35B-A3B as a second supported architecture#29
NeelM0906 wants to merge 27 commits into
drumih:mainfrom
NeelM0906:qwen36-support

NeelM0906 commented Jul 31, 2026 •

edited

Loading

Uh oh!

cenote-dev commented Aug 1, 2026 •

edited

Loading

Uh oh!

AbhinavPanwarDev commented Aug 1, 2026

Uh oh!

jasongoldsmith commented Aug 3, 2026

Uh oh!

Pummelchen commented Aug 3, 2026

Uh oh!

Pummelchen commented Aug 3, 2026 •

edited

Loading

Uh oh!

Pummelchen commented Aug 3, 2026

Uh oh!

ulises-c commented Aug 25, 2026

Uh oh!

Pummelchen commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

NeelM0906 commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add Qwen 3.6 35B-A3B as a second supported architecture

Why this model fits the runtime

Measured results

Bounded memory under an 8 GB working set

What changed

Correctness

Limitations

Uh oh!

cenote-dev commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbhinavPanwarDev commented Aug 1, 2026

Uh oh!

jasongoldsmith commented Aug 3, 2026

Uh oh!

Pummelchen commented Aug 3, 2026

Uh oh!

Pummelchen commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pummelchen commented Aug 3, 2026

Uh oh!

ulises-c commented Aug 25, 2026

Uh oh!

Pummelchen commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

NeelM0906 commented Jul 31, 2026 •

edited

Loading

cenote-dev commented Aug 1, 2026 •

edited

Loading

Pummelchen commented Aug 3, 2026 •

edited

Loading