tq2_0: MMVQ unreachable for MoE (get_mmvq_mmid_max_batch returns -1) + scalar vec_dot — 7.2x generation speedup

[PascalAI2024]

Summary

Two changes to the tq2_0 CUDA kernels take Maple-Preview from 52 → 377 tok/s generation on an RTX 4080 SUPER (7.2×), with output verified unchanged. Patches and full measurements: https://github.com/PascalAI2024/maple-preview-windows-cuda

Thank you for the prism fork — it is the only working CUDA path for this model, and the compute graph matched the MLX reference on the first build.

1. get_mmvq_mmid_max_batch returns -1, making the ternary MMVQ kernel unreachable

ggml/src/ggml-cuda/mmvq.cu:

if (type == GGML_TYPE_TQ2_0) {
    return -1;
}

The call site in ggml_cuda_mul_mat_id tests:

const int mmvq_mmid_max = get_mmvq_mmid_max_batch(src0->type, cc);
if (ne2 <= mmvq_mmid_max) {
    ggml_cuda_mul_mat_vec_q(ctx, src0, src1, ids, dst);
    return;
}

ne2 is a batch size, so it is always ≥ 1. With mmvq_mmid_max == -1 the condition is never true at any batch size — including batch-1 generation, which is exactly what MMVQ handles. Execution then falls through should_use_mmq (no ternary MMQ kernels) and should_use_mmf (float only) to the generic dequantize-to-F32 + GEMM path, so every MoE expert matmul dequantizes.

The comment reads "route large batches to dequant+gemm", which suggests the intent was a small cap rather than disabling it outright. Returning 1 keeps large batches on the existing path while letting batch-1 use vec_dot_tq2_0_q8_1:

 if (type == GGML_TYPE_TQ2_0) {
-    return -1;
+    return 1;
 }

52.1 → 254.8 t/s.

2. vec_dot_tq2_0_q8_1 is scalar

The kernel loops 32 times, one byte load / shift / mask / multiply-add per element, while comparable quant types use packed integer SIMD. Four elements per step:

const int q4    = get_int_b2(bq->qs, (byte_base + j) / 4);
const int codes = (q4 >> shift) & 0x03030303;
const int syms  = __vsub4(codes, 0x01010101);
sumi = ggml_cuda_dp4a(syms, get_int_b4(bq8_1_chunk->qs, j / 4), sumi);

Two constraints worth noting: __vsub4 is needed because a plain integer subtract lets code 0 borrow into the neighbouring byte, and get_int_b2 is needed because block_tq2_0 is 66 bytes, so blocks are only 2-byte aligned. (Same __vsub4 idiom already used in mmq-hopper-q1.cu:234.)

254.8 → 376.7 t/s.

Measurements

RTX 4080 SUPER (16 GB), sm_89, CUDA 12.8, MSVC 14.44, fork rev 9ee03ee, -fa off -r 5:

build	pp512	tg128
stock	1255.7 ± 59.6	52.1 ± 1.1
+ fix 1	1173.4 ± 45.3	254.8 ± 2.9
+ fix 2	1635.8 ± 25.1	376.7 ± 1.1
maple-q4_k_m.gguf (control)	8130.5 ± 88.2	298.2 ± 3.0

The q4_k_m control on the same model is what convinced me this was a defect rather than just a slow path: ternary generation went from 17% to 85% of the mature-kernel baseline, at 44% of the memory. Patched tq2_0 now generates faster than q4_k_m (377 vs 298) while using 5.07 vs 11.48 GiB.

Validation

tq2_0 is commented out of tests/test-backend-ops.cpp upstream (// TODO: implement for all backends), so these kernels have no automated coverage — which is plausibly why #1 went unnoticed. Adding GGML_TYPE_TQ2_0 to base_types gives 103 tests against the CPU reference, and both fixes pass:

MUL_MAT_ID (tq2_0):  72/72 tests passed
MUL_MAT    (tq2_0):  31/31 tests passed

Generation also verified unchanged at --temp 0 on arithmetic, multi-step time reasoning, and factual recall.

Caveats

• One GPU only (sm_89). Ada takes a specific branch in get_mmvq_mmid_max_batch, and I cannot rule out that -1 guards something real on hardware I do not have.
• No perplexity comparison — validation is test-backend-ops plus behavioural checks.
• Prompt processing still trails q4_k_m ~5×, which is the absent ternary MMQ kernels. Q2_0's MMQ traits reuse the generic vec_dot_q8_0_q8_1_* functions with only load_tiles custom, so that looks like the template if you want to close it.

Patches (.patch files) are attached to the release here: https://github.com/PascalAI2024/maple-preview-windows-cuda/releases/tag/v1.0.0

Happy to test further configurations if useful.

---

Disclosure: this investigation and the measurements were done with AI assistance, under my direction and on my hardware. Filing as an issue rather than a PR deliberately — the fix is yours to make if you agree with it.