Repository navigation
Conversation
Adds qwen35_bf16_rows.cu and its launcher (cuda_bf16_rows.zig). tf_q35_embed_bf16 copies the rows of a bf16 embedding table, with the engine's padding-row convention (rows at or past params[2] are left untouched). tf_q35_dense_bf16 is a bf16-in, bf16-out dense projection for small matrices such as the GDN in_proj_a and in_proj_b: one block reduces one output in a fixed order (fp32 fused multiply-adds per thread, a fixed shuffle tree, then the eight warp sums in order), so a row's bits do not depend on how many rows share the launch. Nothing calls them yet; the EXL3 loader in a later commit of this series does. Both launches are listed as pending contract records beside the other Qwen3.5 launches.
Dense.reserve grows the split-K sums, the counters and the fp16 input staging to what 128 rows of a given weight need. The buffers otherwise grow on first use, which allocates inside a CUDA graph capture when a weight is first called there. The EXL3 loader in the next commit calls it for every projection it loads.
Reads a Hugging Face-layout Qwen3.5-MoE checkpoint whose config.json says quantization_config.quant_method "exl3" (ExLlamaV3's trellis format, mul1 codebook, with each tensor's width taken from its trellis shape). Projections and every routed and shared expert stay in their stored layout and run on the existing cuda.exl3 Dense and exl3_experts grouped kernels; embeddings stay bf16, the f16 router and GDN in_proj_a/in_proj_b become bf16 at load, and the zero-centred RMSNorm weights are shifted by one into the engine's form. - cuda_geometry.zig accepts quant_method "exl3" in place of the MLX `quantization` table. Only the routed (MoE) model without a learning pocket is admitted; per-tensor widths come from the trellis shapes at load. - cuda_exl3_weights.zig loads the checkpoint twice over one code path: a dry run sizes the arena, then the real pass uploads. The kit holds the EXL3 dense and grouped-expert kernels and the bf16 rows; their scratch is reserved before any graph capture. - cuda_engine.zig routes `.x3` and `.bf` projections to those kernels at any row count and embeds from the bf16 table. - An EXL3 checkpoint refuses --drafter for now (Exl3DrafterUnsupported): its MTP layer and DFlash2 drafter are not read by this change. The format and the kernel design are turboderp's ExLlamaV3 (https://github.com/turboderp-org/exllamav3); this repository already carries its Dense and grouped-expert kernels in cuda/exl3.zig. The checkpoint this was exercised on is Mia-AiLab/Ornith-1.5-122B-EXL3-EAQv1-4-bpw. Prompt and decode speed use the existing generic EXL3 kernels and are not tuned for this model.
|
Thanks @MiaAI-Lab for the EXL3 Qwen3.5-MoE loader, for reusing the shared EXL3 kernels, and for being clear about what the GB10 smoke run does and does not show. We are carrying your three commits onto current main ourselves, kept as yours, porting the routed MoE calls to main's - Synthema, Ash's AI Assistant. |
|
Landed on main, with your three records under your authorship and four maintainer children. Thank you, @MiaAI-Lab. What the children do:
Measured on a GB10:
- Synthema, Ash's AI Assistant. |
What this changes
Serves ExLlamaV3 (EXL3) Qwen3.5-MoE checkpoints on the native CUDA engine (tracking issue: #1149). The format and kernel design are turboderp's ExLlamaV3 (https://github.com/turboderp-org/exllamav3); the checkpoint exercised is https://huggingface.co/Mia-AiLab/Ornith-1.5-122B-EXL3-EAQv1-4-bpw. Three commits, each building:
qwen35_bf16_rows.cuandcuda_bf16_rows.zig: a bf16 embedding-row copy and a small bf16 dense projection (GDN in_proj_a / in_proj_b) with a fixed summation order. Pending contract records only.exl3.Dense.reserve: grows the dense scratch before graph capture so no allocation happens inside a capture.cuda_geometry.zigacceptsquant_method "exl3",cuda_exl3_weights.zigreads the HF tensor names (trellis projections and every expert kept in their stored layout, f16 router and in_proj_a/b converted to bf16, zero-centred RMSNorm weights shifted by one), andcuda_engine.zigroutes the new projection kinds to the existingcuda.exl3kernels. The weights load runs twice over one code path: a dry run sizes the arena, then the real pass.--drafteris refused for an EXL3 checkpoint (Exl3DrafterUnsupported): its MTP layer and a DFlash2 drafter are not read by this change.Related open PRs, not duplicated: #1137 (BF16 scoring) also adds a
cuda_bf16file and edits geometry, engine andProj, so mine are named*_bf16_rowswith.x3/.bfprojection kinds; whichever lands second will need a small textual merge. #1145 (ModelOpt NVFP4) edits the same geometry and engine regions. #1113 (EXL3 Flash Next) is a different family. #1047 touchescuda_routed.zig.Receipt
--no-drafts --no-thinking --parallel 1 --context 8192 --kv-cache fp8 --vision, temperature 0, 64 tokens, five requests: three text prompts, one image, one tool call (get_weather, not forced). The reply text, finish reason and tool call arguments of this branch's build equal those of a build of the same engine carrying my local, unsubmitted speed patches. That is a consistency check between two builds, not a quality evaluation. Drafted-versus-plain and concurrent-versus-solo checks were not run.zig build teston this branch: 211/211 steps succeeded, 1835 of 1847 tests passed, 12 skipped; upstream main with the same command: 211/211, 2102 of 2110, 8 skipped (the totals differ because cached unchanged steps are not rerun).zig fmt --check build.zig zigclean.tools/zig/lean_check.py: 444 problems before and after. New host tests (cuda_exl3_admission_test.zig): an EXL3 config is admitted byquant_methodalone, and configs with neither an MLX table norexl3are refused; a deliberately wrong expectation madetest-cuda-hostfail, so they run.Limits
Serving works with the generic EXL3 kernels; prompt and decode speed are modest and not tuned for this model, and there is no drafter. Only the MoE model without a learning pocket is admitted.
Checklist
--no-drafts.)"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Not run: no drafter, and one stream only.)