models(transformer): wire Gemma 3 per-layer sliding window + attn/final softcaps + QPAS

[jamesburton]

Summary

Wires four Gemma-family ModelConfig fields through the dense CPU forward path in TransformerModel (both RunLayersAndFinalNormCore and the batched RunLayersAndFinalNormBatched), completing the CPU-side Gemma 3 forward implementation on top of the architecture detection / config plumbing landed in #208 and the CPU Attention.Execute softCap parameter landed in #255.

What gets wired

1. PerLayerSlidingWindow[layer] — per-layer override for the sliding-window mask. Gemma 3's sliding_window_pattern produces an interleaved local/global pattern (Gemma 3 ships a 5:1 local:global ratio; e.g. with pattern=2, layers 0/2 are sliding, 1/3 are full). A new GetLayerSlidingWindow(int) helper resolves per-layer-override (with null for full attention) and falls back to the model-wide SlidingWindowSize. No-op on every architecture without the per-layer field set.

2. AttnLogitSoftcap — passed as the new softCap argument to Attention.Execute so the post-scale soft-cap fires in the CPU kernel exactly as in the Vulkan FA shader (attention_flash_f32.comp). Gemma 2 ships 50.0; Gemma 3 leaves the field null but the plumbing is wired regardless.

3. FinalLogitSoftcap — applied in-place via a new ApplyFinalLogitSoftcap helper after RunLmHead's GEMM and before the result-tensor copy; mirrored in the batched lm_head path inside ForwardBatch. Uses TensorPrimitives.Multiply + TensorPrimitives.Tanh for the SIMD-accelerated kernel.

4. QueryPreAttnScalar (QPAS) — overrides the default 1/sqrt(headDim) attention scale with 1/sqrt(QueryPreAttnScalar) when non-null. Routed via the existing scale-providing Attention.Execute overloads. Gemma 3 ships 256.

All four mechanisms are no-ops on every existing architecture (default field values are null), so this is purely additive plumbing.

Dependencies

• Stacks on models(safetensors): SmolLM3 + Gemma3 architecture detection + NoPE layers #208 — provides the Architecture.Gemma3 enum value and the four ModelConfig fields (PerLayerSlidingWindow, AttnLogitSoftcap, FinalLogitSoftcap, QueryPreAttnScalar).
• Stacks on kernels(cpu)(attention): softCap parameter for Gemma 2/3 attn_logit_softcapping #255 — provides the softCap parameter on every Attention.Execute overload (span / pointer / parallel / scalar reference paths).

Test plan

• dotnet build src/DotLLM.Models/DotLLM.Models.csproj is clean.
• dotnet build tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj is clean.
• Existing Gemma / Transformer tests still pass.
• End-to-end synthetic Gemma 3 forward-pass coverage follows in a separate PR stacked on this one.

Files touched

• src/DotLLM.Models/Architectures/TransformerModel.cs (+74/-8)

[jamesburton]

Update: precursors are now extracted/ready and this can move forward.

Full dependency chain (bottom-up):

• models(transformer): Phase 5a — extract RunLayersAndFinalNormCore + RunLmHead for ForwardBatch fusion #276 — TransformerModel ForwardBatch CPU lm_head fusion (Phase 5a)
• models(transformer): Phase 5b — fuse intra-block matmuls across simple-subgroup ForwardBatch sequences #277 — TransformerModel ForwardBatch CPU intra-block matmul fusion (Phase 5b)
• models(safetensors): SmolLM3 + Gemma3 architecture detection + NoPE layers #208 — SafeTensors SmolLM3 + Gemma3 architecture detection and ModelConfig fields (PerLayerSlidingWindow, AttnLogitSoftcap, FinalLogitSoftcap, QueryPreAttnScalar)
• kernels(cpu)(attention): softCap parameter for Gemma 2/3 attn_logit_softcapping #255 — CPU Attention.Execute softCap parameter
• models(transformer): wire Gemma 3 per-layer sliding window + attn/final softcaps + QPAS #274 — this PR (wires the four Gemma 3 mechanisms through the dense CPU forward path)

End-to-end synthetic Gemma 3 forward-pass coverage will follow as a separate PR stacked on this one.

[jamesburton]

Loader-interface migration PR is up: #314 (issue #313).

The PR ports the MLA / Mixtral / Qwen-MoE loaders on the #277 stack onto a new ISafetensorsTensorSource interface. Pure type substitution — no behaviour change, 73 existing safetensors/MLA/MoE unit tests pass byte-identical.

Verified the unblock: git cherry-pick --no-commit d6a7e6c (the Gemma 3 forward-wiring commit) auto-merges cleanly on top of the migration with zero conflict markers. The remaining build errors are about #208's ModelConfig fields (PerLayerSlidingWindow, AttnLogitSoftcap, FinalLogitSoftcap, QueryPreAttnScalar) and the Attention.Execute softCap parameter from #255 — i.e. the dependencies already listed in #274's stack. The loader-vs-interface mismatch is gone.

Once #313 lands on main and #208 / #255 advance, this PR can be re-extracted as a clean diff on top.

[jamesburton]

Gemma 3 transformer wiring has been re-extracted onto the just-landed loader-interface migration (PR #314) as PR #318 (issue/274-gemma3-transformer-wiring-rebased). It stacks on PR #317 (rebased SmolLM3 + Gemma 3 architecture detection, the rebased form of held PR #208).

ROADMAP Phase 8 Step 57 (Gemma 4) follows immediately as PR #320 — issue #319 — built on top of #318. Track X: Google publicly released Gemma 4 on 2026-04-02 (Apache 2.0); the HF Transformers gemma4/ model directory carries Gemma4Config / Gemma4ForCausalLM and friends. PR #320 reuses every Gemma 2/3 mechanism wired here and ships only the new enum value + HF detection + ModelLoader dispatch + tests.

Stack summary: #320 → #318 (this) → #317 → #314 → #277 → #276 → #243 → #222 → main.