fix: harden Lunar Lake Gemma 4 attention and quantized CPU copies

README.md

@@ -26,6 +26,8 @@ Every entry below carries a **Why** — the reason it exists downstream. That is
 | DFlash speculative decoding | Block-diffusion drafter integration (`LLM_ARCH_DFLASH`, `--spec-type dflash`, `llama_set_dflash`, CUDA kernels for partial-accept feature extraction). Designed against the [z-lab DFlash](https://github.com/z-lab/dflash) reference for Gemma4 31B targets. | Active product bet: diffusion drafting to lift Gemma4 decode throughput on the 3090/P5200 fleet. Upstream has no diffusion-drafter framework to extend. | No |
 | Gemma4 MTP speculative | Vendored upstream PR [#23398](https://github.com/ggml-org/llama.cpp/pull/23398) (`gemma4-assistant` arch + `--spec-type draft-mtp`) ahead of upstream merge so the gemma-4-12b-qat-mtp preset can ship on titan. Retires when #23398 merges upstream and flows through a normal master sync. | Ships a measured 1.66× sampled / 3.02× greedy decode speedup on titan months before upstream review completes. Explicitly temporary. | [#23398](https://github.com/ggml-org/llama.cpp/pull/23398) |
 | D=512 FA vec kernels | CUDA flash-attention vec-kernel instances for head size 512 with matched quantized KV (`q4_0`/`q8_0`), dispatch-gated to `gqa_ratio <= 4`; deployment-shape (Gemma4 MQA-16) correctness + perf cases in `test-backend-ops`. | Low-GQA D=512 shapes skip the per-step F16 dequant staging (up to 2× per-op). The gate keeps Gemma4's MQA-16 global layers on the faster TILE/MMA path — measured on both sm_61 and sm_86. | No |
+| Lunar Lake Gemma 4 tuning | Opt-in hybrid F16 attention for single-sequence contexts up to 2K and Q4_0 MMVQ dispatch, restricted to Intel `8086:64a0` with Mesa Vulkan. [Notes](docs/backend/VULKAN-GEMMA4-INTEL-XE2.md), [validation](scripts/xe2/README.md). | Keep device-specific speed experiments bounded to supported cache types and measured shapes; preserve stock selection elsewhere. | No |
+| CPU quantized copy safety | Reject unsupported quantized transposes before direct/planned execution; correct row-byte counts for valid reshaped copies. | Hybrid attention exposed a fallback copy that wrote beyond its tensor allocation. | No |
 | Router-mode robustness | `llama-server` router detects worker crashes via `subprocess_alive` polling; fixes hardcoded proxy timeout | Router mode is the fleet's deployment shape (multi-model boxes); a hung worker must surface as an error, not a stuck request. | [#22003](https://github.com/ggml-org/llama.cpp/pull/22003) |
 | Tool-calling resilience | Fallback tool-call parser and skip non-`function` tool types so non-conforming models still work | heierchat exposes tools to arbitrary local models; strict parsing turned every malformed call into a hard failure. | No |
 | Developer-role remap | `--remap-developer-role` flag merges `developer` messages into the system prompt for templates that reject duplicates | OpenAI-client compatibility: clients that emit `developer` roles hit template errors on Gemma-family chat templates. | No |

docs/backend/VULKAN-GEMMA4-INTEL-XE2-VALIDATION.md

@@ -0,0 +1,120 @@
+# Lunar Lake acceptance record — 2026-09-05
+
+Status: acceptance run in progress. Hybrid remains experimental and opt-in.
+No fresh speedup or deployment-readiness claim is made until the results below
+are complete.
+
+## Scope and reproducibility
+
+Device: Core Ultra 5 238V, 32 GB UMA, Intel PCI `8086:64a0`, Linux `xe`,
+Mesa/vulkan-intel 26.1.5, GCC 16.1.1. Runs use AC power and the performance
+platform profile. The [model manifest](../../scripts/xe2/models.json) pins the
+12B target, matching MTP assistant, and 26B A4B target by revision and SHA256.
+These establish a fresh baseline; historical Xe2 artifact hashes were absent.
+
+Use the [validation commands](../../scripts/xe2/README.md) to reproduce the
+checks. Raw outputs include commands, Git revision/diff, binary version,
+model hashes, device information, power/temperature samples, and process/DRM
+memory accounting. Server presets set a 1 GiB prompt-cache budget and a 512 MiB checkpoint
+budget per slot. Cache policies can exceed those nominal budgets by a retained
+or newly appended state; measured memory use is reported separately.
+
+## Safety and local tests
+
+- CPU quantized-copy regression tests and ASan/UBSan pass. Unsupported block
+  transposes are rejected before direct/planned execution writes output.
+  An oversized-buffer reproduction changed from writing 17,268,736 bytes beyond
+  the logical destination to returning failure with zero writes beyond it.
+- Vulkan comparisons pass: 280 CPY/CONT, 29 Q4_0 MUL_MAT, and 247 Gemma attention
+  cases with 256/512 heads and F16/Q8_0/Q4_0 K/V.
+- Local CPU CI passed Debug 45/45 and Release 47/47 using `GG_BUILD_LOW_PERF=1`.
+  This selection excludes the large-model/high-performance jobs.
+  `LLAMA_FATAL_WARNINGS=OFF` was required for a GCC 16 warning in unchanged
+  vocabulary construction code. The later model-free logit-metric test passes in Debug and Release, and all
+  three telemetry/process infrastructure regression tests pass.
+
+## Corrections to the validation method
+
+Initial tests fed untemplated text to instruction models. Some completions were
+repetitive or nonsensical even while structured chat answered correctly. The
+corrected lifecycle and soak tests apply the GGUF chat template, check capital
+and arithmetic answers, verify the requested tool city, and compare MTP against
+target-only capital and code responses. Stream cancellation waits for an actual
+generated token before closing the connection.
+
+The native test now prefills a templated conversation, then evaluates continuation
+batches of 1, 2, 16, 31, 32, 63, 64, 128, and 1152 tokens. This exercises the
+attention thresholds and sliding-window reuse with a valid instruction prefix.
+
+A single-operation NMSE limit was also an unsuitable full-model sampling test.
+Logits have an arbitrary common offset; negligible-probability vocabulary tails
+can dominate their squared error. Full-model comparisons now require KL divergence
+below 0.005 nats and total variation below 0.05. Raw NMSE and top-token agreement
+remain in every report. KV-reuse checks additionally retain NMSE below `5e-4`.
+Model-free checks cover shift invariance, changed predictions, near ties, and
+non-finite rejection. Declared suppressed tokens must remain negative infinity;
+other logits must be finite.
+
+The original raw-text failures are retained as diagnostic evidence. An isolated
+build of the original `ht` baseline (`06d9d42`) produced byte-identical full
+logits at all nine lengths for both models with FA-off and hybrid. This establishes
+that those numerical differences predate the safety hardening; it does not
+establish universal CPU/GPU equivalence. A dequantized F16 CPU reference and
+higher-precision Vulkan trial explained much of the short-prompt difference,
+but the original long, repetitive raw prompt remained numerically sensitive.
+
+## End-to-end results
+
+| Check | 12B | 26B A4B |
+|---|---|---|
+| FA-off/hybrid distributions and KV reuse | Pass | Pass |
+| Plain-FA-on comparison | Pass | **Fail**: KL at 16 tokens |
+| Templated lifecycle, fallback, tools, cancellation | Pass (6 profiles) | Pass (5 profiles) |
+| Baseline and hybrid soak, 15 minutes each | Pass | Pass (hybrid switch with four-slot fallback) |
+| Five interleaved runs per mode at 2K/8K/16K/32K | Pending | Pending |
+
+The complete 26B parity sweep returns failure: plain FA-on at 16 continuation
+tokens has KL `0.00991865776` against the CPU reference, exceeding `0.005`.
+Its top token still agrees. This is not waived or converted into a passing test.
+Both offered presets pass their comparisons; plain FA-on is a benchmark/control
+configuration, not a recommended serving preset.
+
+| Model/mode | Maximum CPU-reference KL | Maximum total variation | Top-token agreement |
+|---|---:|---:|---:|
+| 12B FA-off | 0.00115015 | 0.0186292 | 9/9 |
+| 12B hybrid | 0.000639504 | 0.0116276 | 9/9 |
+| 26B FA-off | 0.00312896 | 0.0339058 | 9/9 |
+| 26B hybrid | 0.00457187 | 0.0359762 | 9/9 |
+| 26B plain FA-on control | **0.00991866** | 0.0359762 | 9/9 |
+
+Automatic FA with hybrid enabled produces byte-identical logits to explicit
+FA-on with hybrid enabled on both models. Worst hybrid KV-reuse NMSE is
+`0.000430299` for 12B and `0.0000268791` for 26B. F16 FA-off/FA-on and Q8/Q4
+reuse comparisons are exact. Quantized-cache checks establish finite outputs
+and correct reuse, not quantization-quality equivalence to the F16 reference.
+
+Above 2K, and with multiple configured sequences or quantized caches, the hybrid
+switch intentionally retains stock attention. Benchmarks at those settings
+measure fallback behavior. The four-slot 26B soak likewise exercises fallback,
+not eligibility for hybrid attention.
+
+## Sustained-load results
+
+All four phases completed on the final serving cache budgets, totaling 60.47
+minutes and 887 requests. No answer, process, finite-output, or telemetry check
+failed. Both 12B phases used MTP; the 26B hybrid-switch phase used four slots.
+
+| Model/profile | Minutes | Requests | MTP accepted/drafted | Warm GPU resident GTT (MiB) | Warm CPU RSS cycle peaks (MiB) |
+|---|---:|---:|---:|---:|---:|
+| 12B baseline | 15.21 | 21 | 448/728 | 8304.6 | 1830.1–2105.6 |
+| 12B hybrid | 15.01 | 270 | 5670/8561 | 7548.9 | 1756.5–1761.5 |
+| 26B baseline | 15.07 | 84 | — | 13985.1 | 1549.2–1578.9 |
+| 26B four-slot fallback | 15.18 | 512 | — | 14653.9 | 1700.0–1714.5 |
+
+Warm measurements exclude the first five minutes of each phase. GPU residency
+was flat to the displayed precision; CPU cycle-peak slopes ranged from -38.89
+to +0.95 MiB/min. CPU peaks use complete three-round prompt cycles to account
+for cache churn. RSS and DRM GTT are separate views of UMA memory and must not
+be added as disjoint allocations. This finite soak does not prove the absence
+of every leak. Request counts are not comparable throughput benchmarks: context
+lengths, prompt lengths, and slot counts differ across profiles.

docs/backend/VULKAN-GEMMA4-INTEL-XE2.md

@@ -1,10 +1,38 @@
 # Gemma 4 on Intel Xe2 Vulkan: architecture, Flash Attention, and tuning
 
-Status: engineering notes and local measurements, 2026-08-29.  The target used for
+Status: historical measurements from 2026-08-29; safety hardening 2026-09-05.
+The target used for
 measurements is a Core Ultra 5 238V / Lunar Lake Arc 130V/140V iGPU, Linux `xe`,
 Mesa 26.1.5 ANV, Vulkan 1.4, 32-wide subgroups, 48 KiB shared memory, FP16/BF16,
 integer dot product, KHR cooperative matrix, and unified system memory.
 
+## Safety and validation update (2026-09-05)
+
+The hybrid experiment now requires the exact value
+`LLAMA_VK_GEMMA4_HYBRID_FA=1`; unset, `0`, and other values disable it.
+Eligibility is resolved once per context and requires all model layers on the
+validated Lunar Lake PCI device `8086:64a0` with Mesa Vulkan and KV offload.
+Only F16 K **and** V, a single sequence, and configured context at most 2048
+tokens use the experimental policy. Longer contexts, multiple slots, partial
+offload, other backends/devices, and quantized caches retain stock selection.
+Flash Attention's `auto` capability probe always runs without hybrid rewriting.
+
+This restriction fixes a real unsafe combination: hybrid decomposition with
+quantized V created a transposed quantized copy unsupported by Vulkan. The CPU
+fallback treated blocks as scalar elements and wrote beyond the output tensor.
+CPU scheduling and direct/planned graph execution now reject unsupported
+quantized copy layouts before writing. Valid reshaped quantized copies also
+have a corrected row-byte calculation. The Q4_0 MMVQ exception is restricted
+to this Lunar Lake/Mesa device; prior dispatch is preserved elsewhere.
+
+The benchmark tables below are historical, not fresh acceptance evidence.
+Their original Xe2 model hashes were not recorded. The pinned, checksummed
+artifacts and repeatable validation commands in [`scripts/xe2`](../../scripts/xe2/README.md)
+establish a new baseline without claiming byte-identical reproduction.
+The [acceptance record](VULKAN-GEMMA4-INTEL-XE2-VALIDATION.md) separates passing
+preset checks from failed controls and unfinished measurements. Keep hybrid
+opt-in until its parity, lifecycle tests, depth benchmarks, and soak pass. Do not infer long-context gains from the short runs.
+
 This document is about making each usable Gemma 4 member perform well.  It does
 not recommend replacing one family member with another: the dense, unified,
 MoE, PLE, multimodal, and MTP variants serve different purposes.
@@ -190,7 +218,8 @@ should remain in a deployment A/B; the 26B prompt result clearly favored adaptiv
 ### Adaptive attention retained in this branch
 
 When `LLAMA_VK_GEMMA4_HYBRID_FA=1` is set and `--flash-attn on` is requested,
-the graph builder applies the policy to both `gemma4` and `gemma4-assistant`:
+the graph builder applies the policy to eligible `gemma4` and
+`gemma4-assistant` contexts described in the safety update above:
 
 - fewer than 32 query tokens: decomposed attention for decode and speculative
   verification;
@@ -199,9 +228,9 @@ the graph builder applies the policy to both `gemma4` and `gemma4-assistant`:
 - 64 or more query tokens: FA for D=256 local layers and decomposed attention
   for D=512 global layers.
 
-The environment gate is intentional: graph construction does not know which
-backend will ultimately execute each node, so making this Xe2 result the default
-would risk regressing CUDA, Metal, or a future faster Vulkan driver.  This is a
+The environment gate is intentional: the context checks backend/device
+eligibility, but making this experimental result the default would still risk
+regression with future drivers or unmeasured workloads. This is a
 shape-aware scheduler win rather than a new mathematical attention algorithm.
 
 Absolute iGPU rates move with package power, temperature, display activity, and
@@ -288,7 +317,8 @@ prefill; compare with all-off for 12B):
 ```bash
 LLAMA_VK_GEMMA4_HYBRID_FA=1 ./build-vulkan/bin/llama-server \
   --model MODEL.gguf --n-gpu-layers all \
-  --flash-attn on --ubatch-size 512
+  --flash-attn on --ubatch-size 512 --ctx-size 2048 --parallel 1 \
+  --cache-type-k f16 --cache-type-v f16
 ```
 
 All-off baseline:

ggml/src/ggml-cpu/copy.h

@@ -0,0 +1,48 @@
+#pragma once
+
+#include "ggml.h"
+
+// Quantized blocks cannot be transposed as individual scalar elements. Keep
+// this check shared by scheduling and direct/planned CPU graph execution.
+static inline bool ggml_cpu_quantized_rows_supported(const struct ggml_tensor * tensor) {
+    if (!ggml_is_quantized(tensor->type)) {
+        return true;
+    }
+    if (tensor->nb[0] != ggml_type_size(tensor->type)) {
+        return false;
+    }
+    // A one-block row can have nb[0] == nb[1], so strides alone cannot
+    // distinguish its invalid scalar transpose. Follow view provenance too.
+    int axis = 0;
+    for (const struct ggml_tensor * cur = tensor; cur != NULL; cur = cur->src[0]) {
+        switch (cur->op) {
+            case GGML_OP_TRANSPOSE:
+                if (axis < 2) { axis = 1 - axis; }
+                break;
+            case GGML_OP_PERMUTE:
+                for (int i = 0; i < GGML_MAX_DIMS; ++i) {
+                    if (cur->op_params[i] == axis) {
+                        axis = i;
+                        break;
+                    }
+                }
+                break;
+            case GGML_OP_VIEW:
+            case GGML_OP_RESHAPE:
+                if (axis != 0) { return false; }
+                break;
+            default:
+                return axis == 0;
+        }
+    }
+    return axis == 0;
+}
+
+static inline bool ggml_cpu_copy_layout_supported(const struct ggml_tensor * op) {
+    if (op->op != GGML_OP_CPY && op->op != GGML_OP_CONT && op->op != GGML_OP_DUP) {
+        return true;
+    }
+    const struct ggml_tensor * src = op->src[0];
+    const struct ggml_tensor * dst = op->op == GGML_OP_CPY ? op->src[1] : op;
+    return ggml_cpu_quantized_rows_supported(src) && ggml_cpu_quantized_rows_supported(dst);
+}

ggml/src/ggml-cpu/ggml-cpu.c

@@ -12,6 +12,7 @@
 #include "binary-ops.h"
 #include "vec.h"
 #include "ops.h"
+#include "copy.h"
 #include "ggml.h"
 #include "common.h"
 
@@ -3320,6 +3321,16 @@ struct ggml_threadpool * ggml_threadpool_new(struct ggml_threadpool_params * tpp
 enum ggml_status ggml_graph_compute(struct ggml_cgraph * cgraph, struct ggml_cplan * cplan) {
     ggml_cpu_init();
 
+    // Check before any worker can write output, including callers that bypass
+    // backend scheduling or execute a previously created graph plan.
+    for (int i = 0; i < cgraph->n_nodes; ++i) {
+        if (!ggml_cpu_copy_layout_supported(cgraph->nodes[i])) {
+            GGML_LOG_ERROR("%s: unsupported quantized copy layout for %s\n",
+                           __func__, cgraph->nodes[i]->name);
+            return GGML_STATUS_FAILED;
+        }
+    }
+
     GGML_ASSERT(cplan);
     GGML_ASSERT(cplan->n_threads > 0);
     GGML_ASSERT(cplan->work_size == 0 || cplan->work_data != NULL);

ggml/src/ggml-cpu/ggml-cpu.cpp

@@ -1,6 +1,7 @@
 #include "ggml-backend.h"
 #include "ggml-backend-impl.h"
 #include "ggml-cpu.h"
+#include "copy.h"
 #include "repack.h"
 #include "traits.h"
 #include "ggml-impl.h"
@@ -421,6 +422,9 @@ static ggml_backend_buffer_t ggml_backend_cpu_device_buffer_from_host_ptr(ggml_b
 }
 
 static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
+    if (!ggml_cpu_copy_layout_supported(op)) {
+        return false;
+    }
     const struct ggml_tensor * src0 = op->src[0];
     const struct ggml_tensor * src1 = op->src[1];
 

ggml/src/ggml-cpu/ops.cpp

@@ -1,4 +1,5 @@
 #include "ops.h"
+#include "copy.h"
 
 #include "ggml-cpu.h"
 #include "ggml-impl.h"
@@ -373,7 +374,7 @@ static void ggml_compute_forward_dup_bytes(
     if (ggml_is_contiguous(dst)) {
         size_t id = 0;
         char * dst_ptr = (char *) dst->data;
-        const size_t rs = ne00 * type_size;
+        const size_t rs = ggml_row_size(src0->type, ne00);
 
         if (nb00 == type_size) {
             // src0 is contiguous on first dimension, copy by rows
@@ -559,6 +560,8 @@ void ggml_compute_forward_dup(
         const ggml_compute_params * params,
         ggml_tensor * dst) {
 
+    GGML_ASSERT(ggml_cpu_copy_layout_supported(dst));
+
     const ggml_tensor * src0 = dst->src[0];
 
     if (src0->type == dst->type) {