TEMP: refresh ROCm qwen35moe integration to upstream #131 bb432e8 - #40
Merged
nekomario28 merged 2 commits intoAug 26, 2026
Conversation
Three defects with one root: "gguf" is a container tag, not a weight layout. The checkpoint picks a ggml type per tensor and the concrete CPU format has to be recovered from the bank types, but two call sites tested the tag directly. _cpu_moe_executor_viable compared expert_quant against _WFMT_IDS, which answers False for EVERY GGUF checkpoint. That silently disabled the automatic residency split on hosts where CUDA pinning is quota-capped -- WSL caps it near 40% of RAM (measured: 81.78 GiB of 204). The symptom was not a clear refusal but cudaHostRegister failing partway through the banks, which reads as a memory shortage rather than a dispatch gap. gemm1_dot handled bf16 and q4_0 and then FELL THROUGH to the NVFP4 path, which dereferences scale/global pointers that are null for GGUF banks. An unhandled format therefore segfaulted inside a worker thread with no Python traceback. It now raises through TORCH_CHECK naming the format, same reasoning as the kernel default: guards in FlashML-org#138: an unhandled case that reads null or uninitialised memory is far worse than one that errors. GGUFEmbedding dequantized unconditionally, but the unquantized types are raw value bytes with no dequant kernel at all (ggml_dequantize rejects type 1 outright). DeepSeek-V4 ships token_embd as F16 and died on the first lookup. The gathered rows are now reinterpreted for those types, matching the fix already applied to fused_mul_mat_gguf.
Five fixes that stood between a reconciled adapter and a generated token, found
by loading the real 164GB antirez/deepseek-v4-gguf Q4KExperts checkpoint.
convert_deepseek4_to_gguf was never called from the model's __init__, so the
fp8 Linears were never swapped and load_state_dict demanded a wq_a.scale no
GGUF tensor can fill. The reconciliation test passed because it called the
converter explicitly; the engine does not. Third time this hook has been missed
in this package (gemma4 and qwen35moe before it).
GGUFLinearNN passed its input straight to fused_mul_mat_gguf, which takes
[tokens, in_features] and treats dim 0 as the batch. F.linear -- which it
replaces -- accepts any number of leading dims, and deepseek_v4 relies on that:
its attention passes 3-D tensors. Collapsing one silently reshaped q so the
sparse-attention kernel's `b, m, h, d = q.shape` unpack failed. Leading dims are
now folded and restored.
GGUFEmbeddingNN dequantized unconditionally; token_embd is F16 here, which has
no dequant kernel. Reinterpreted instead.
The DSV4 sparse-attention kernel exceeded Turing's 64KB shared-memory block
limit. BLOCK_T was tuned for sm_120's ~99KB budget, but the dominant cost is not
the KV tile: q and acc are [BLOCK_H, D] in fp32, which at BLOCK_H=16 and
head_dim 512 is 65536 B on its own -- an entire Turing block before a single KV
byte, which is why shrinking BLOCK_T alone left the requirement stuck at 66624.
BLOCK_H, BLOCK_T and num_stages are now chosen from the device's opt-in shared
memory; sm_80 and above are unchanged.
_TOKENIZER_ARCH mapped deepseek4 to "llama". The file says
tokenizer.ggml.model = gpt2: the llama converter is sentencepiece-shaped and
encodes a space as U+2581, so against a GPT2-BPE vocab every space was silently
DROPPED on detokenization ("ThecapitalcityofFranceisParis"). The model was
correct throughout; only the detokenizer was wrong, which presents as model
damage and is not. Mapped to the qwen2 (GPT2-BPE) entry.
Verified on a Quadro RTX 6000 (Turing, sm_75, 24GB) with 204 GiB of WSL RAM:
145 GiB of expert banks split 24 layers GPU-pinned / 19 OS-locked for CPU
decode, CPU executor on the Q4_K kernels, ~18GB VRAM. At temperature 0:
"The capital city of France is" -> " Paris. The capital city of England is
London..."
"The largest planet in our solar system is" -> " Jupiter. It is so big that
more than 1,300 Earths"
Known limitation: CUDA graph capture crashes the worker on this configuration,
so this runs with --cuda-graph-max-bs 0. Decode is slower than it should be as
a result. Not yet diagnosed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Temporary internal history-preserving refresh carrier. Base is exact combined ROCm×qwen35moe head 2195d02. Head is exact upstream FlashML-org#131 donor bb432e8 imported into the fork. Purpose: detect/resolve only real overlap with the already validated ROCm/DSV4 integration before spending another native GGUF compile. Close after the refreshed exact head is validated and promoted.