Repository navigation
EXL3 checkpoints on the native engine (tracker) #832
Description
EXL3 served on CUDA in the Python engine from 0.3.6 to 0.6.6 through the module @vcruz305 wrote (#42): every codebook, widths 1 to 8 mixed per tensor, a row-invariant linear, and grouped routed experts. The native engine has no EXL3 model yet. This tracker is the port, with @vcruz305 invited to lead it, and his authorship kept on everything that follows his design.
In the tree or in review
- Format and a scalar reference decoder in core:
zig/src/core/exl3_format.zig(@akol1, core: backend-neutral EXL3 trellis reference decoder with an oracle fixture #496), with an oracle fixture. - A standalone Metal projection:
zig/src/core/exl3_projection.zig(@akol1, Add standalone resident Metal EXL3 projections #681). - CUDA projections behind
qlinear(cuda: EXL3 projections behind qlinear (#593) #740), routed experts behindexperts.Grouped(cuda: EXL3 routed experts behind experts.Grouped (#593, stacked on #740) #741) and fp32 outputs for split layers (cuda: EXL3 projections write fp32 too (qlinear decodeF32 and promptF32) #806), by @BHCC2025, on the arithmetic ofdocs/recipes/exl3.md. They wait for a first model to call them.
What is left
- A loader: read a checkpoint's EXL3 groups from the safetensors headers (trellis, suh/su, svh/sv, bias, the codebook marker), each tensor's own width, and the plain tensors beside them; refuse what does not parse before a download, as the Python engine did.
- The first model: Qwen3.8-27B 3.00bpw + DFlash2 on one GB10, through the 27B CUDA engine. That engine lands first (planned for the next release); the EXL3 work stacks on it and on cuda: EXL3 projections behind qlinear (#593) #740.
- The stop token some Qwen EXL3 packs keep only in
generation_config.json(fix: Qwen3.8 Flash Next and 27B on EXL3 checkpoints stop at the end of the chat turn #69 in the Python engine): check the native server reads it, and add a test with such a pack. - Then the MoE packs through cuda: EXL3 routed experts behind experts.Grouped (#593, stacked on #740) #741: Flash Next once it has a native CUDA engine, GLM-5.3-Flash (cuda: EXL3 projections write fp32 too (qlinear decodeF32 and promptF32) #806's receipts used turboderp's 4.05 bpw pack), MiMo-V2.6-Flash (Proposal: MiMo-V2.6-Flash on the CUDA native engine across two GPUs (EXL3, DFlash) #725, MiMo-V2.6-Flash on CUDA: EXL3, DFlash, two GPUs #728).
The gate
- Exactness: drafted replies equal
"draft": false, a resumed prompt equals a fresh one, concurrent streams equal their solo runs, and decode and prompt rows keep the recipe's arithmetic at every row count. - Speed on the same box against the 0.3.6 measurements (27B 3.00bpw on one Spark: 83.4 tok/s code sampled) and against the native engine's MLX 4-bit 27B.
- Kernel bandwidth against ExLlamaV3's
LinearEXL3on the same tensors, asdocs/recipes/exl3.md(Numbers) did.
Credit
Commits that carry #42's design carry Co-authored-by for @vcruz305 when someone else writes them; @BHCC2025 and @akol1 keep theirs. Picks land under each author's name.
Activity
ashhart commentedon Oct 14, 2026
An update for everyone working on EXL3 in the native engine, with thanks to each of you:
qlinear, grouped experts, fp32 outputs).The plan: one GLM family on CUDA, #804's, with EXL3 weights coming through it. One EXL3 module behind
qlinear, chosen by running #837 and #740/#741/#806 on the same tensors (bits against the float64 oracle, bandwidth againstLinearEXL3) and keeping the faster at equal bits, with both authors credited. And @vcruz305 credited on everything that follows #42.@vcruz305, #837 carries your own kernels from #42 into the native engine. If you have time, a review from you would mean a lot, and the invitation to lead the port stands.
ling: weights in the checkpoint's own format (bf16, FP8 blocks, Model…
ashhart commentedon Oct 17, 2026
Thank you, @justincasey, for the same-tensor comparison plan and mixed-width layer case.
We will use #837 as the shared consolidation base, carrying #806's FP32 qlinear contract, #741's Grouped integration and #729's Metal path. Both CUDA contributors and @vcruz305's #42 work keep their credit; per-expert widths stay in the representation.
Please retain the proposed alternating equal-bit comparisons against FP64 and LinearEXL3, including every result where the other implementation is faster. We will use that evidence while composing the single module, preserving prompt/decode and resumed/fresh exactness and the existing output API. This is integration work, not a second family-specific EXL3 path.
Posted by Ash's AI Assistant Synthena
paragontasx commentedon Oct 17, 2026
EXL3 on the native engine: Flash Next 5.05 passes the gate (12/12, drafts 24/24)
This is a result, not a proposal. #837 is the EXL3 module; the code below predates it and I am not
asking for a second one. If it is useful, I will rebase the family wiring onto #837's
qlinear/Groupedsurface and #869's family, and re-run this gate.Setup.
turboderp/Qwen3.8-Flash-Next-exl3,5.05bpw_h6_ng6(7cef615f), one GB10, TP=1, textonly, bf16 KV, drafts off for the gate. Vehicle: MiaAI-Lab's Flash Next CUDA family on a local branch
(never pushed), with EXL3 kernels ported from #42's
decode.cuh,linear.cuandexperts.cubefore#837 existed. Oracle: a v0.6.6 capture of the same pack.
gate --no-drafts, 6 prompts greedy + 6 sampledlong101 at EOS), 34.8–38.6 ms/tokencheck-weightsprefill --dumpteacher --dump, 48 stepsgatedrafts on, MTP head byte-exactAcceptance matches the reference closely: on
code, 212/254 against 211/253.Capture. The v0.6.6 oracle is public at https://github.com/paragontasx/TensorFold/releases/tag/exl3-flashnext-capture-p3-1011-0955: 81 files, 64 MB unarchived. It holds
results.json(36 entries: the token ids and shas this gate compares),kernels_exl3.json(65specializations in 22 families, with Triton and cubin hashes),
weights.json(digests and shapesonly, no weights), the prompts, and the launch records. It is the piece that can no longer be
regenerated from
mainnow that the Python engine is gone.Offered, if wanted. The seam map for EXL3 inside a Flash Next CUDA family (21 hook sites), the
P5/P6 receipts, and the rebase above followed by a re-run on the same box.
Credit. @vcruz305: #42's kernels and arithmetic, every bit of which is reproduced here. @akol1:
core/exl3_format.zigand the Metal projection the checks lean on. @BHCC2025: #740, #741, #806.@justincasey: #837. MiaAI-Lab: the vehicle's family and its capture harness.
Limits. The oracle is v0.6.6, not
main; the pack is 5.05 bpw Flash Next, not the 27B; one GB10;speed is parity with Python (1.04x drafted), not a headline. To my knowledge no public artifact shows
another EXL3 model on the native engine. If someone is further along, I will compare against it rather
than claim this.
ashhart commentedon Oct 17, 2026
Thanks @paragontasx for the pinned 12/12 plain and 24/24 drafted results, and for preserving the oracle and contributor credits. Yes, the seam map and reusable fixtures would help the integration.
Keep one EXL3 implementation through #837's common surface and the selected #952 Flash Next family, with #869's tests and credit. We will carry the maintained integration; there is no need to create another family-specific EXL3 path. The old capture remains reference evidence, not a production dependency. The combined source still needs its own chunk, cache, draft and concurrency checks; these 5.05 bpw results do not qualify other packs.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 18, 2026
Thank you, @vcruz305, @akol1, @BHCC2025 and @justincasey, for the EXL3 operators, fixtures and integration receipts.
1.1.0 carries #837's common CUDA EXL3 module with #806's tiles, #741's grouped experts and #729's Metal prefill work, retaining the original credits.
This closes the operator-port tracker for those released integrations. Please reopen with the pinned checkpoint, layout and device if a released integration still fails.
CUDA operator checks and Apple M3/M5 Ultra row-parity checks passed; whole-model/two-rank oracle coverage remains incomplete.
Posted by Ash's AI Assistant Synthena