Skip to content

GGUF kernels: raise on unsupported quant type instead of returning uninitialized memory - #138

Open
vcruz305 wants to merge 1 commit into
FlashML-org:mainfrom
vcruz305:fix/gguf-kernel-default-guards
Open

GGUF kernels: raise on unsupported quant type instead of returning uninitialized memory#138
vcruz305 wants to merge 1 commit into
FlashML-org:mainfrom
vcruz305:fix/gguf-kernel-default-guards

Conversation

@vcruz305

Copy link
Copy Markdown

Splitting this out of #131 because it stands on its own and is quick to review. It is a correctness fix with no feature attached.

None of the five switch (type) blocks in gguf_kernel.cu has a default: case, and the output tensor is allocated with torch::empty. So passing a quant type the kernel does not implement returns uninitialized memory instead of raising:

at::Tensor Y = torch::empty({batch, row}, options);   // uninitialised
switch (type) {
  case 2:  ...
  case 14: ...
  // no default -> Y returned as-is
}
return Y;

ggml_moe_get_block_size has the same shape and falls through to return 0, which then divides or sizes a launch by zero downstream.

The gap is real rather than theoretical. ggml_mul_mat_vec_a8 and ggml_moe_a8_vec handle 19 types, while ggml_mul_mat_a8 and ggml_moe_a8 handle 10: the I-quants have no MMQ kernel. So any caller that routes an I-quant to the MMQ path today gets silent garbage, and the only symptom is a model that loads fine, runs at full speed and emits fluent nonsense. I lost a fair amount of time to exactly that before adding these guards.

Each default: now raises through TORCH_CHECK and names the function plus which type families that entry point actually supports. I also null-check the ggml_get_to_cuda function pointer inside ggml_dequantize, since it returns nullptr for an unknown type (dequantize.cuh) and the result was being called unconditionally.

No kernel math is touched. The only behaviour change is that an unsupported type now fails loudly at the call instead of producing wrong numbers. There is a comment above the first guard noting these kernels are vendored from sgl-kernel and that the guards are a local addition, so a future re-vendor does not quietly drop them.

Found while working on #131 (GGUF quant types and qwen35moe), but it is independent of that branch.

None of the five switch (type) blocks in gguf_kernel.cu had a default: case, and the
output tensor is allocated with torch::empty, so an unsupported quant type returned
uninitialized memory rather than raising. ggml_moe_get_block_size fell through to
return 0, which then divides or sizes a launch by zero downstream.

The gap is reachable: ggml_mul_mat_vec_a8 and ggml_moe_a8_vec handle 19 types while
ggml_mul_mat_a8 and ggml_moe_a8 handle 10, because the I-quants have no MMQ kernel. Any
caller routing an I-quant to the MMQ path gets silent garbage, and the only symptom is a
model that loads, runs at full speed, and produces fluent nonsense.

Each default: now raises via TORCH_CHECK naming the function and the type families that
entry point supports. ggml_dequantize also null-checks the ggml_get_to_cuda function
pointer, which returns nullptr for an unknown type and was being called unconditionally.

No kernel math changed. A comment above the first guard notes these kernels are vendored
from sgl-kernel so a future re-vendor does not drop them.
@paralin

paralin commented Aug 26, 2026

Copy link
Copy Markdown

Failure exists on any backend where an unsupported quant path falls through, +1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants