feat(windows): integrate upstream Qwen3.8 DFlash2 support - #8
Merged
Conversation
Run the five-layer draft backbone, top-16 conditional selector and sparse rejection through the shared Program for K=1..15. Keep local context in StateImage, separate proposal and verify extents, and commit GDN state and token counts after the Frontend chooses the final output prefix. Share dynamic-convolution workspace recipes and use one backend selector for round storage. Support eager/Graph execution, both proposal heads, Text/Vision, partial terminal commits and Device/Host context reuse. Unify product benchmark options with --spec/--draft-tokens and report the backend explicitly in schema v14. Promote the algorithm reference and remove the completed support plan. Validation: real DFlash2 K15/B8 Graph and K7 Vision/Host restore; NVFP4 K2 eager benchmark; legacy 35B DFlash and MTP Graph; eight focused C++ tests, Python benchmark summary test, and affected documentation link checks.
…n place of eight dependent rounds
The small-T MoE router runs S2 on a single CTA. dim3(1) is written into launch_s2 for both of
its branches, and the trace agrees: every one of the 1240 launches of a batch-1 round and every
one of the 1200 launches of a batch-8 round has gridX=gridY=gridZ=1, while the S1 kernel of the
same operation runs on 1028 blocks and the D3 kernel on 6144.
Nothing in the work asks for that. S2 is T independent top-8 selections over 256 router logits,
one warp per token, plus a fold of the four partial sums S1 left behind. There is no dependency
between tokens; the single CTA is a consequence of scores[Tokens][257] living in one shared
array, and the two_batch branch at T >= 45 exists only to cap that array at 32 tokens.
Two changes, in two files, independent of each other:
* S2 takes one CTA per token, 128 threads. The shared array becomes scores[257] plus eight
selected logits: 1060 B per block instead of up to 46 640 B, at every T. The kernel stops
being a template - nothing in it depended on Tokens except the shared size - and the
two_batch kernel goes away with the reason for its existence.
* sparse_moe_select_top8_warp reaches the warp-wide top-8 with a merge network. Each lane
already holds its eight candidates sorted; five xor exchanges merge the runs pairwise, and
each exchange is eight independent shuffles followed by three compare-exchange stages over
static indices. That is five dependent shuffle levels instead of the eight dependent
warp_best rounds - about 48 levels - that the loop walked before.
Output is bit-identical, and not by luck: sparse_moe_ranked_better (greater value, lower id on a
tie) is a total order over 256 distinct expert ids, so the top-8 set and its order are uniquely
determined and cannot depend on the network that finds them. The tail - softmax over the eight
selected logits, warp_reduce_sum, the shared-expert sigmoid - is untouched. The origin field of
SparseMoeRankedValue was only ever read by the cursor of the old loop and is deleted with it.
The selector is shared with the decode D2 kernel and the prefill route kernel, so both get the
shorter chain. Registers drop where it matters and rise where it does not: S2 goes 48 -> 38,
d2_warp goes 40 -> 38, and the prefill select_count kernel goes 40 -> 54 - which the prefill
measurement below says costs nothing. STACK and LOCAL are 0 on every instantiation, before and
after.
Measured on RTX 5090 sm_120a, driver 580.159.03, CUDA 13.1, 525 W cap, Release, base ad0f3d3,
qwen3_6_35b_a3b.ninfer (groupwise-int), KV bf16. Every cell is its own process under an
exclusive card lock with its own witness window; 472 measurement cells, none rejected.
Product benchmark, ninfer_bench -pg 2048,384 -r 6 --warmup 2 --prefill-chunk 8192
--mtp-draft-tokens 3, eight passes, arm order rotated, deltas paired inside a pass, median over
passes:
decode 658.99 -> 669.51 tok/s +1.598 %
prefill 22634.6 -> 22640.6 tok/s +0.026 % (i.e. nothing)
total time 0.6747 -> 0.6649 s -1.384 %
acceptance rate and round count identical to four decimals
Decode round bench (dflash, --context 1024 --proposal-head full, 200 warmup rounds, 40 measured,
eight passes): -1.271 % at batch 1 (T=4) and -1.599 % at batch 8 (T=32), against a control of
+0.047 % and +0.071 % measured between two builds of the unpatched code. Cells separate
completely in both points.
Kernel duration from nsys with --cuda-graph-trace=node, medians over 1240 and 1200 launches:
T=4 5568 -> 4000 ns -28.2 % grid 1 -> 4 blocks, 128 threads
T=32 12608 -> 8608 ns -31.7 % grid 1 x 1024 threads -> 32 x 128
Neither ceiling of this card is what the kernel is short of, which is the point. S2 reads
T x 257 x 4 x 4 B; against the measured DRAM read ceiling of this part, 1689.4 GB/s, that is
0.175 % -> 0.243 % of the ceiling at T=4 and 0.618 % -> 0.905 % at T=32. The kernel issues no
MMA instruction at all, and its arithmetic - the fold, T x 257 x 3 additions - is 0.0012 % of
the measured 66.1 TFLOP/s scalar tier. What the kernel was short of was a dependent chain and
a grid, and that is what changed.
Swept over the token window so that the shared path is not taken on trust. Round level, T from
2 to 45 at twenty points reachable as batch x (draft+1), three passes each: every point improves,
-0.91 % to -2.74 %, best at T=45 where the two_batch branch used to run. Operator level (weaker
instrument, but it reaches the eleven T that the round bench cannot express), all 45 values of
T from 2 to 46: kernel duration -22.4 % to -51.5 %, no point worse.
ctest is 104/104 on the base and 104/104 on the change, run from the same worktree with the same
cmake flags, with NINFER_QWEN3_6_35B_A3B_WEIGHTS set so the real-model tests run rather than
skip. Two tests skip on both arms for want of a 27B bf16 artifact on this machine.
ninfer_sparse_moe_test checks the op against a double-precision CPU reference at T = 1, 2, 19,
20, 46, 47, 768 and 4097, which covers T=46 - the case the deleted branch used to serve.
Bit-identity gate: ninfer --print-token-ids --greedy --no-thinking --spec dflash --draft-tokens 3,
192 tokens. Stock against itself, both single changes, both together and the built product binary
all give 8a13dc80376529e6685cc33bee37432b. The gate is shown to be able to tell the difference
rather than assumed to be: two sabotage arms that keep the kernel legal and the experts valid -
one shifting the eighth expert id by one, one swapping the logits of ranks 6 and 7 without
swapping their ids - both change the hash.
Owner
|
This looks good, I'll test on a second machine and get it merged/built this evening |
natpate
added a commit
that referenced
this pull request
Sep 6, 2026
Integrates bigwario's qwen3.8-27b nvfp4full target profile + converter on top of the PR #8 DFlash2 sync. Single conflict in bindings.cpp: both PRs inserted a new bind_* function into the same gap; kept both (nvfp4full first, bind_dflash2 last) and added the one function-closing brace git dropped, so both bundles bind. Resolves the bindings.cpp collision between the nvfp4full text-layer binder and the DFlash2 weight-bundle binder; variant.cpp/package.cpp/package.h and the 7 Python converter tools auto-merged clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This brings the canonical Qwen3.8-27B DFlash2 implementation from Neroued/ninfer into the Windows fork. Current groupwise-int and NVFP4 artifacts can run
--spec dflash2 --draft-tokens 7 --lm-head-draftthrough the existing Engine, CLI and server.The first commit merges upstream through
487f89773f07cb18a2fb841fe0971ec9634d409b, preserving upstream authorship and history. This includes the associated runtime, operator, artifact, conversion and test changes, which account for most of the diff. The second commit contains the Windows integration fixes and documentation:constexpr std::sqrtinitializers in numerical tests withconst, preserving the oracle formulas.Related to #4, using the current canonical upstream implementation. The OpenAI test update overlaps the area covered by #5; this branch carries the newer upstream suite with the Windows fallback adaptation.
Validation
Tested on Windows with an RTX 5090, driver 616.56, CUDA 13.1 and MSVC 14.44. The Qwen3.8-27B NVFP4 artifact was checked against the model-card size and full SHA-256.
--helppassed for all three applications.--dflash2-onlysoftmax-attention and input-projection subsets.[DONE], and Anthropic Messages. DFlash2 started with CUDA Graphs at 131,072-token context/KV capacity, reporting 3.04 GiB free after startup.Reproduction commands and the selected coverage are in docs/windows.md. This is focused validation: the full repository suite, groupwise-int real-model execution, eight concurrent requests and Linux execution were not run.
Measured decode throughput
Same expanded NVFP4 artifact, one active request, INT8 KV, 131,072-token context/KV capacity, prefill chunk 1024, CUDA Graphs and optimized draft head. Prefix reuse and thinking were off; sampling was greedy with seed 42. Each prompt generated 1,024 tokens, with two measured repeats after warmup. Values are mean server-reported decode tokens/s.
These compare fixed prompts and output budgets. Continuations differed between backends, and every response reached the token limit; this is a small throughput comparison, not a quality or completed-task evaluation.