TEMP baseline: integrate #131 GGUF CPU/hybrid fix - #39
Merged
nekomario28 merged 1 commit intoAug 26, 2026
Merged
Conversation
Three defects with one root: "gguf" is a container tag, not a weight layout. The checkpoint picks a ggml type per tensor and the concrete CPU format has to be recovered from the bank types, but two call sites tested the tag directly. _cpu_moe_executor_viable compared expert_quant against _WFMT_IDS, which answers False for EVERY GGUF checkpoint. That silently disabled the automatic residency split on hosts where CUDA pinning is quota-capped -- WSL caps it near 40% of RAM (measured: 81.78 GiB of 204). The symptom was not a clear refusal but cudaHostRegister failing partway through the banks, which reads as a memory shortage rather than a dispatch gap. gemm1_dot handled bf16 and q4_0 and then FELL THROUGH to the NVFP4 path, which dereferences scale/global pointers that are null for GGUF banks. An unhandled format therefore segfaulted inside a worker thread with no Python traceback. It now raises through TORCH_CHECK naming the format, same reasoning as the kernel default: guards in FlashML-org#138: an unhandled case that reads null or uninitialised memory is far worse than one that errors. GGUFEmbedding dequantized unconditionally, but the unquantized types are raw value bytes with no dequant kernel at all (ggml_dequantize rejects type 1 outright). DeepSeek-V4 ships token_embd as F16 and died on the first lookup. The gathered rows are now reinterpreted for those types, matching the fix already applied to fused_mul_mat_gguf.
nekomario28
marked this pull request as ready for review
August 26, 2026 09:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Synthetic baseline only. Adds upstream FlashML-org#131 commit 3178a59 to the existing ROCm+qwen35moe integration branch so the split-JIT review PR can exclude unrelated CPU/hybrid reachability changes. Preserve donor history; do not use as a product PR.