diff --git a/.continue-here.md b/.continue-here.md
new file mode 100644
index 00000000..55cf5dd6
--- /dev/null
+++ b/.continue-here.md
@@ -0,0 +1,225 @@
+---
+branch: feature/mamba-3
+workstream: Phase 1-4 + Phase 8 of the post-Mamba-3-merge roadmap (K-quants Vulkan + real-weight Vulkan parity + long-context YaRN verification + LoRA foundation/Vulkan/server/quant + F16/BF16 native Vulkan). Branch and main both at 9864bc6.
+stage: this session landed 45+ commits across 8 logical phases plus a CUDA bring-up cycle merged from a sibling machine. Vulkan unit suite 115 → 367 (+252). LoRA fully wired CPU + Vulkan + server API; F16/BF16 + K-quants all in tree; real-weight Vulkan parity for 6 architectures + DeepSeek-V2-Lite-Q4_K_M. 4d.2 actual MLA/MoE delta wiring + macro-bench deferred to follow-up.
+status: feature/mamba-3 == origin/main == origin/feature/mamba-3 == 9864bc6, working tree clean, no agents running.
+last_commit: 9864bc6
+last_updated: 2026-04-28
+
+handoff_intent: continue plan execution. Next batch is THREE LIGHT PHASES IN
+ PARALLEL (6 / 9 / 12), then the heavier batch sequentially (5 / 7 / 10 / 11).
+ Plus three LoRA follow-ups that should land before paged attention.
+---
+
+
+Branch state at `9864bc6`. main and origin both at the same SHA. Working
+tree clean. No agents running. Phase 4 (LoRA) is functionally complete with
+two documented deferrals (MLA-LoRA / per-expert MoE-LoRA delta wiring +
+real-checkpoint macro-bench).
+
+Vulkan unit suite: **367 passed / 0 failed / 1 skipped**. Phase 2 Vulkan
+integration suite: **10 passed / 1 skip (DeepSeek-V2-Lite SafeTensors,
+budget-bound)**. CPU/Models unit suite: **1767 passed / 158 skipped / 0
+failed**.
+
+
+## Session ledger (this turn — 45+ commits across phases)
+
+**Phase 1 — K-quants Vulkan (6 commits)**
+- `afb2272` + `b1ee6bc` — Q4_K_M kernels + upload/dispatcher (+16 tests)
+- `15099b9` + `83e0732` — Q5_K_M kernels + upload/dispatcher (+16 tests)
+- `29a1459` + `39b7646` — Q6_K_M kernels + upload/dispatcher (+16 tests)
+
+**Phase 2 — Real-weight Vulkan parity (3 commits)**
+- `0b3137c` — `RealHfSafetensorsEndToEndVulkanTests` 6 architectures (TinyLlama / Qwen2.5 / Phi-3.5 / Granite-3-MoE / DeepSeek-V2-Lite + Mamba-3 ib-ssm 370M)
+- `5b12f90` — `RealGgufVulkanParityTests` 3 GGUF cases (Llama-3.2-1B-Q8_0 / Bielik-1.5B-Q4_K_M / DeepSeek-V2-Lite-Q4_K_M)
+- `4724397` — `GgufModelConfigExtractor` defaults DeepSeek-V2/V3 to Phase C latent cache (paired with the HF-side default at `4b54a72`)
+
+**Phase 3 — Long-context YaRN reference (1 commit)**
+- `2b00a0f` — Reusable `--prompt-file` + `--last-n-rows` script flags + `logits_offset` JSON schema field + 7.4 MB `deepseek-v2-lite-longctx-reference.json`. `DeepSeekV2Lite_LongContext_LogitsMatchPyTorchReference` now actually runs in 36m 55s (was a 50ms self-skip) at positions 4912-4920 within `DriftTolerances.Tight` against PyTorch BF16 oracle.
+
+**Phase 8 (brought forward) — F16/BF16 native Vulkan (2 commits)**
+- `c9c08c5` + `6787492` — F16 GEMV/GEMM/coopmat + BF16 GEMV/GEMM kernels + upload paths + dispatcher routing (+48 tests, Vulkan 312 → 360)
+
+**Phase 4 — LoRA full stack (10 commits)**
+- 4a foundation (CPU): `8737008` + `baeb240` + `27f52ec` + `3b28fa6`
+- 4b Vulkan dispatch: `cf9dfc2` + `731c53c` + `c578046`
+- 4c server API + multi-adapter batcher seam: `43f0733` + `a7e7883`
+- 4d quant LoRA + MLA/MoE acceptance + bench: `9864bc6` (single commit, reduced scope)
+
+**Documentation sync (1 commit)**
+- `17820df` — README/SUPPORTED_MODELS/QUANTIZATION/VULKAN/KV_CACHE updates reflecting all of the above
+
+**Plus** a `~107-commit CUDA bring-up cycle` merged from a sibling machine in the middle of this session (covers Q2_K through Q6_K MMQ kernels, MLA Phase B, MoE forward, DeepSeek-V2-Lite-Q4_K_M end-to-end on GPU). All CUDA work was untouched by this session's edits.
+
+## Coverage matrix (post-batch)
+
+| Architecture | CPU FWD | CPU real-weight | Vulkan FWD | Vulkan real-weight |
+|---|:-:|---|:-:|---|
+| Llama | ✓ | ✓ TinyLlama-1.1B + Llama-3.2-1B-Q8_0 + Bielik-1.5B-Q4_K_M | ✓ | ✓ all three |
+| Mistral | ✓ | ⚠ tiny-random | ✓ (same path) | — |
+| Phi | ✓ | ✓ Phi-3.5-mini | ✓ | ✓ Phi-3.5-mini |
+| Qwen | ✓ | ✓ Qwen2.5-0.5B | ✓ | ✓ Qwen2.5-0.5B |
+| DeepSeek-V2 | ✓ | ✓ V2-Lite (incl. long-context YaRN) | ✓ MLA + MoE | ✓ V2-Lite-Q4_K_M (19m 27s) |
+| DeepSeek-V3 | ✓ | ⚠ tiny-random | ✓ (same MLA path) | — |
+| NemotronH | ✓ | ⚠ env-var gated | ✓ | — |
+| Mamba-3 | ✓ | ✓ ib-ssm 370M | ✓ SISO + MIMO + streaming-chunk + Q8_0 | ✓ ib-ssm 370M |
+| Mixtral | ✓ | ⚠ tiny-random fetch fail | ✓ MoE | — |
+| QwenMoe | ✓ | ⚠ tiny-random | ✓ MoE | — |
+| GraniteMoe | ✓ | ✓ Granite-3-MoE | ✓ MoE | ✓ Granite-3-MoE |
+
+Quant on Vulkan: F32, F16, BF16, Q8_0, Q4_K_M, Q5_K_M, Q6_K_M (all native, no F32 expansion). Q4_0/Q4_1/Q5_0/Q5_1/Q2_K/Q3_K/IQ-family fall back to F32 dequant on upload (CUDA has them; Vulkan tracked as follow-up).
+
+## Next batches
+
+### Batch 1 — parallel (light, independent file scopes)
+
+These three can run concurrently as separate background agents — touch
+disjoint file scopes and have no shared dependencies.
+
+#### Phase 6 — Sliding-window real-weight integration test
+- File scope: `tests/DotLLM.Tests.Integration/Models/Loaders/` (new test class) + possibly `tests/DotLLM.Tests.Unit/Cpu/Kernels/SlidingWindowAttentionTests.cs` for additional cases.
+- Cheapest implementation: synthetic small-config Qwen2-style model with `sliding_window=4`, 16-token prompt, brute-force masked reference vs `TransformerModel.Forward` output. 1 commit, ~300 LOC.
+- Optional stretch: feed Qwen2.5-0.5B (cached) a >32K-token prompt, verify finite logits + correct sliding pattern. Slow (10+ min CPU forward) but real.
+- Scope: integration test only. No kernel changes. CPU only — Vulkan attention kernels handle sliding via the same config path.
+
+#### Phase 9 — ALiBi position encoding
+- File scope: `src/DotLLM.Core/PositionEncoding/AlibiPositionEncoding.cs` (new), `src/DotLLM.Cpu/Kernels/Attention.cs` (extend), `src/DotLLM.Vulkan/Kernels/AttentionF32Kernel.cs` (or new ALiBi variant), tests.
+- ALiBi math: additive linear bias `score[i, j] += -slope[h] * (i - j)` per head. Slopes are a fixed geometric sequence depending on numHeads. No frequency tables — much simpler than RoPE.
+- Architectures: BLOOM, MPT, Falcon-legacy use ALiBi. None are in the supported-arches list today, so this lands as foundation for adding them later.
+- Estimated 2-3 commits, ~600 LOC.
+
+#### Phase 12 — Cosmetic cleanup (the "we don't really want cosmetic gaps either" item)
+- File scope: 6 src TODOs (`MatMul.cs:1368/1611`, `SchemaTracker.cs:249/538`, `TextGenerator.cs:129`, `RequestValidator.cs:56`), `Architecture.cs` enum (mark `Architecture.DeepSeek` legacy as `[Obsolete]` or delete), README/docs sweep for stale claims.
+- Mostly non-overlapping with batch-1 siblings since the TODOs span unrelated subsystems.
+- 1-2 commits, ~300 LOC.
+
+### Batch 2 — sequential (heavier, possibly conflicting file scopes)
+
+#### Phase 5 — Real-weight checkpoints for the four `verified: tiny-random` archs
+- Mistral 7B (~14 GB FP16 / ~4 GB Q4_K_M)
+- DeepSeek-V3 — likely a small derivative (V3-tiny enriched) since the full V3 is ~660 GB
+- Mixtral-8×7B (~26 GB Q4_K_M; full FP16 is ~94 GB — too big)
+- Qwen1.5-MoE-A2.7B (~8 GB Q4_K_M)
+- OLMoE-1B-7B (~4 GB Q4_K_M)
+- Disk planning: 47 GB free now, 30 GB DeepSeek-V2-Lite SafeTensors is redundant (Q4_K_M GGUF works) — delete it to free 30 GB. Then ~46 GB Q4_K_M downloads fit cleanly.
+- Each: download + add gated test + update SUPPORTED_MODELS row. ~5 commits, +5 real-weight tests.
+
+#### Phase 7 — Q8_0 MoE indexed-expert kernel
+- New SPIR-V shader: per-row Q8_0 indexed weight lookup + dequant in inner loop.
+- New `MoeIndexedMatmulQ8_0F32Kernel` C# wrapper.
+- Wire `RecordMoeLayer` to dispatch to this kernel when source weights are Q8_0 indexed (currently F32 indexed used).
+- 1-2 commits, ~500 LOC. File scope: `src/DotLLM.Vulkan/` (Vulkan-only, conflicts with Phase 11).
+
+#### Phase 10 — Paged attention KV-cache
+- Multi-commit milestone: new `PagedKvCache` (block-table + per-block storage, vLLM-style), engine integration, block-aware attention kernels (CPU + Vulkan), benchmark vs `SimpleKvCache`.
+- 5000-8000 LOC. Touches `src/DotLLM.Engine/`, `src/DotLLM.Cpu/Kernels/Attention.cs`, `src/DotLLM.Vulkan/Kernels/AttentionF32Kernel.cs`, `IModel`, `TransformerModel.Forward`, etc.
+- High risk of conflict with anything else touching attention kernels. Best done alone on its own sub-branch or carefully sequenced.
+
+#### Phase 11 — Coopmat MoE Strategy C (multi-commit redesign)
+- Per the prior agent's architectural finding (the MoE-coopmat investigation): coopmat's shared-A-tile semantics fundamentally conflict with MoE's per-row indexed weights. Real speedup needs:
+ 1. New `MoeExpandAndGroupByExpertF32Kernel` (replaces broadcast for the coopmat path; emits packed input + permutation + per-expert offsets).
+ 2. New `MatMulCoopmatF16F32Kernel` (regular dense F16 GEMM coopmat, reusable beyond MoE).
+ 3. New `MoeUngroupAndScatterF32Kernel`.
+ 4. F16 expert-bank overlay buffers in `VulkanWeights.MoeLayerBuffers`.
+ 5. Parallel `RecordMoeLayerCoopmat` path gated on `device.HasCooperativeMatrix && shape-compatible`.
+- 6-8 commits, multi-week. Best as its own sub-branch / milestone.
+
+## LoRA follow-ups (track as a small batch before Phase 10)
+
+- **4d.2 actual delta wiring** for MLA-specific projections (`q_a_proj` / `q_b_proj` / `kv_a_proj_with_mqa` / `kv_b_proj`) and per-expert MoE projections (`mlp.experts.{j}.{gate,up,down}_proj`). Acceptance lift is in (validation no longer rejects); the runtime delta application at MlaAttention.Execute / MoeSwiGluMlp.Dispatch sites is what's missing. Blocked on a public PEFT release using these names — none observed as of 2026-04. Stub a synthetic-fixture parity test to land alongside, even without a real adapter.
+- **4d.3 macro-bench** vs a real TinyLlama checkpoint: kernel-level bench reports decode F32 LoRA at +4% (under 5% target), prefill F32 at +9% (over). Real-checkpoint forward measurement to confirm whether the prefill regression matters in production.
+- **Fused `RmsNormQuantizeAndKeepF32` shader** to recover the LoRA-active fused-rmsnorm bypass cost in the Vulkan decode path. Not blocking but caps perf headroom.
+
+## Remaining CUDA-side work (sibling machine)
+
+Carried forward from the prior handoff after the CUDA bring-up rebase
+landed at `b041866`. CUDA work happens on a separate Windows + RTX 3060
+host; this branch is the integration target. The 107 commits merged in
+this session covered through **Spec 1 Phase 1 (Q2_K full L3)**.
+
+**Spec docs** (in tree):
+- `docs/superpowers/specs/2026-04-27-q2k-iquant-coverage-design.md` — Spec 1 master
+- `docs/superpowers/plans/2026-04-27-q2k-phase1-plan.md` — Phase 1 (Q2_K) plan, **complete**
+
+### Spec 1 — IQ-family quantization coverage (Phases 2-5 outstanding)
+
+| Phase | Quants | Effort | Plan doc | Status |
+|---|---|---|---|---|
+| 1 — Q2_K | Q2_K | ~1 week | `docs/superpowers/plans/2026-04-27-q2k-phase1-plan.md` | **Complete** (merged into this branch) |
+| 2 — IQ4 | IQ4_NL, IQ4_XS | ~2 weeks | TBD | Pending |
+| 3 — IQ3 | IQ3_S, IQ3_XXS | ~2-3 weeks | TBD | Pending |
+| 4 — IQ2 family | IQ2_S, IQ2_XS, IQ2_XXS | ~2-3 weeks | TBD | Pending |
+| 5 — IQ1 | IQ1_S | ~1-2 weeks | TBD | Pending |
+
+Each remaining phase produces: CPU dequant + GPU dequant + per-call
+quantized GEMV + MMQ (legacy + pre-Q8_1) + MMVQ-large (legacy +
+pre-Q8_1) + grouped-MoE-GEMV variants for the phase's quant types,
+plus a real-GGUF end-to-end smoke + (recommended) llama.cpp logits
+parity sidecar + ~25-75 new tests per phase. Phase ordering per the
+spec, value-per-effort: **IQ4 first** (highest production
+relevance — Llama-3.1 / Qwen2.5 IQ4_XS), IQ3, IQ2, IQ1. Each phase
+independently shippable.
+
+### Other CUDA-side follow-ups (not Spec-1)
+
+- **Real V2-Lite Q4_K_M decode benchmark** — Round 15 unlocked
+ down_proj's GEMV fast path on real Q8_0 weights via the
+ `MinKAlignmentFor` gate fix; the Q3_K_M 27-layer smoke saw a
+ measurable 26% wall-clock improvement, but the Q4_K_M end-to-end
+ decode hasn't been benchmarked since. `profile-cuda-decode --compare`
+ against the cached 10.4 GB Q4_K_M would quantify the real-V2-Lite
+ improvement.
+- **CUDA HF-parity gates** — 4 `CudaLogitsMatchPyTorchReferenceTests`
+ remain skipped on missing safetensors fixtures (TinyLlama,
+ Phi-3.5, Qwen2.5-0.5B, DeepSeek-V2-Lite). Loader
+ (`CudaTransformerModel.LoadFromSafetensors`) is wired; fixtures are
+ on the Ryzen+iGPU box. Gates auto-activate when fixtures land on the
+ CUDA host.
+- **Q2_K real-GGUF smoke fixture** — `RealGguf_Q2K_FullModel_27LayerSmoke`
+ is gated `Skip.If(!File.Exists)` for
+ `bartowski/DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf` (~5 GB).
+ Activates automatically once the fixture is downloaded to
+ `~/.dotllm/models/bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF/`.
+- **Larger-model scaling** — Qwen3-30B-A3B / Mixtral-8×7B /
+ DeepSeek-V3-MTP all need >12 GB GPU memory and would benefit from
+ grouped-GEMM raw-quant compaction (kernels would hold quantized
+ weights in shared memory rather than fully resident; 1-2 weeks of
+ focused kernel work). Out of scope for Spec 1 phases.
+
+### Vulkan parity for IQ-family quants (cross-backend follow-up)
+
+Once CUDA Spec 1 Phase 2+ lands the IQ4 / IQ3 / IQ2 / IQ1 kernels,
+mirror them on Vulkan to keep K-quant + IQ-quant coverage aligned
+across both backends. Today Vulkan covers F32 / F16 / BF16 / Q8_0 /
+Q4_K_M / Q5_K_M / Q6_K_M; CUDA additionally covers Q2_K + (post-Spec-1)
+the IQ family. Both should converge.
+
+## Open questions / decisions for next session
+
+1. Do we want Phase 10 (paged attention) before or after Phase 5 (real-weight downloads)? Paged attention enables genuinely-long context; real-weight tests benefit from it. Lean: Phase 10 first.
+2. Coopmat MoE Strategy C — sub-branch or main-line? Lean: sub-branch given the multi-week scope.
+3. Phase 12 cleanup — should `Architecture.DeepSeek` legacy be `[Obsolete]` (back-compat) or removed (clean break)? Lean: `[Obsolete]` with a 1-version deprecation window since it's public API surface.
+
+## DO NOT do early
+
+- Don't attempt Phase 11 (coopmat Strategy C) before Phase 10 (paged attention) lands — paged attention reshapes the engine surface that the MoE redesign would need to coordinate against.
+- Don't merge anything to `main` before checking `feature/mamba-3` is clean and ahead-only — the user has been running `git push origin feature/mamba-3:main` after each phase batch, which is FF-only.
+- Don't delete `.perf-runs/` artifacts — verified earlier as real user benchmark work, untracked but valuable.
+- Don't restructure existing LoRA code from Phase 4a/4b/4c. Additive only.
+
+## Environment snapshot (unchanged)
+
+- **Box:** Windows 11, Ryzen AI Max+ 395 (Zen 5, 32T), AMD Radeon 8060S iGPU (Strix Halo, gfx1151, RDNA3.5).
+- **Vulkan SDK:** scoop-installed, `glslc` on PATH.
+- **CUDA / ROCm:** CUDA not installed here (handled on a sibling machine; rebased + merged this session); ROCm 7.1 installed but iGPU runtime blocks `hipModuleLoadData`.
+- **Real weights cached** under `C:/Users/james/.dotllm/test-cache/` and `C:/temp/dotllm-*` — see SUPPORTED_MODELS.md for the inventory. DeepSeek-V2-Lite-Q4_K_M GGUF (10.4 GB) just landed in this session at `C:/Users/james/.dotllm/test-cache/mradermacher/DeepSeek-V2-Lite-GGUF/`.
+- **Disk:** ~37 GB free of 1.9 TB. Clearing `C:/temp/dotllm-deepseek-v2-lite/` (30 GB BF16 SafeTensors) before Phase 5 frees enough for ~50 GB of Q4_K_M downloads.
+
+## Suggested first actions for the next session
+
+1. `git pull origin feature/mamba-3` — should be no-op if branch is current.
+2. Confirm baseline: `dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Debug --filter "FullyQualifiedName~Vulkan" --logger "console;verbosity=minimal" --nologo` should report 367 / 0 / 1.
+3. **Dispatch Batch 1 — three agents in parallel** (Phase 6 + Phase 9 + Phase 12). Each in its own background agent. They touch disjoint file scopes so no race risk.
+4. After Batch 1 lands, optionally land the LoRA follow-ups (4d.2 wiring + macro-bench + fused-rmsnorm) as a small bundle.
+5. Then sequence Batch 2: Phase 5 (parallel-ish — downloads + tests) → Phase 7 (Vulkan-only) → Phase 10 (paged attention, large) → Phase 11 (coopmat Strategy C, multi-week, possibly own sub-branch).
diff --git a/.gitignore b/.gitignore
index 694a7ad3..624b545d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,9 +3,10 @@
##
## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore
-# Claude Code local config (machine-specific paths, agent memory)
+# Claude Code local config (machine-specific paths, agent memory, session lock)
.claude/local-paths.md
.claude/agent-memory/
+.claude/scheduled_tasks.lock
# User-specific files
*.rsuser
@@ -425,3 +426,12 @@ FodyWeavers.xsd
# dotLLM test model cache
tests/.models/
+
+# Scratch files used during nemotron_h research (do not commit)
+_*.cpp
+_*.txt
+.mamba3-reference/
+.mamba3-reference-canonical/
+.mamba3-python-venv/
+.venv-pyref/
+.gpu-lock/
diff --git a/.perf-runs/cross-lib-20260424-expA/README.md b/.perf-runs/cross-lib-20260424-expA/README.md
new file mode 100644
index 00000000..1cbc717d
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424-expA/README.md
@@ -0,0 +1,61 @@
+---
+date: 2026-04-24
+experiment: A — clamp _decodeThreadCount to min(8, threadCount) when no NumaTopology
+baseline: .perf-runs/cross-lib-20260424/ (dotLLM at 8.66 tok/s, 22.17 s wall)
+---
+
+## Result
+
+Same harness, same model, same prompt, same host — only change is the
+one-line decode-thread-count clamp in `ComputeThreadPool`:
+
+| Backend | Run | tok/s | wall (s) |
+|---|---|---:|---:|
+| llama.cpp CPU (`-ngl 0`, from baseline) | 1 | 34.73 | 5.53 |
+| dotLLM CPU (pre-experiment, from baseline) | 1 | **8.66** | 22.17 |
+| dotLLM CPU + experiment A (cold) | 1 | **33.76** | 5.69 |
+| dotLLM CPU + experiment A (warm prefix cache) | 2 | 39.03 | 4.92 |
+| dotLLM CPU + experiment A (warm prefix cache) | 3 | 39.64 | 4.84 |
+
+First-run / cold comparison: **dotLLM 33.76 vs llama.cpp 34.73 — within 3%**.
+The 4× gap reported in the previous cross-lib comparison collapses to noise.
+
+## What changed
+
+One default, in `src/DotLLM.Cpu/Threading/ComputeThreadPool.cs`:
+
+```diff
+- : threadCount;
++ : Math.Clamp(DefaultDecodeThreadCountCap /* = 8 */, 2, threadCount);
+```
+
+When the pool is built without a `NumaTopology`, `_decodeThreadCount` used
+to fall back to `threadCount`, so SpinWait mode ran all 31 workers on a
+32-thread host. The `ThreadPoolDispatchBenchmarks` microbench (see
+`.perf-runs/cross-lib-20260424/dispatch-microbench.md`) showed that path
+collapses: Dispatch(no-op) costs 333 µs at 32T SpinWait vs 1 µs at 8T,
+driven by 31 threads hammering one cache line on `_dispatchGeneration`
+and a contended `_completion.Signal()` decrement.
+
+Capping to 8 keeps SpinWait in the regime the microbench shows scales
+well (2–16 threads, sub-µs dispatch cost). Prefill still uses all 32
+threads via EventBased mode — unaffected.
+
+## Why 8
+
+Matches the heuristic already applied when a `NumaTopology` *is*
+provided: `topology.MemoryChannelEstimate` (typically 2 channels × 4 on
+Zen 5 desktop/workstation = 8). Decode is memory-bandwidth bound, not
+compute bound, so extra concurrent loads past a small multiple of memory
+channels add cache-line contention without adding useful bandwidth.
+
+## Next
+
+The 3% residual gap vs llama.cpp is within benchmark noise. If we want
+to push past llama.cpp on this workload, the remaining levers in
+priority order are the ones already listed in
+`.perf-runs/cross-lib-20260424/dispatch-microbench.md` §Concrete next-step
+experiments B and C (size-gated single-threaded fast path, cache-line
+padding in `ComputeThreadPool`). Neither is currently justified by the
+numbers — come back to them if we regress or if we see a similar issue
+on a host with different topology.
diff --git a/.perf-runs/cross-lib-20260424-expA/dotllm-throughput-2.json b/.perf-runs/cross-lib-20260424-expA/dotllm-throughput-2.json
new file mode 100644
index 00000000..f37e8490
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424-expA/dotllm-throughput-2.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T16:48:19.500009+00:00",
+ "run_finished_at": "2026-04-24T16:48:25.691817+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424-expA\\dotllm-throughput-2.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T16:48:20.557529+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 39.03,
+ "total_s": 4.92,
+ "cpu_avg_pct": 0,
+ "cpu_peak_pct": 0,
+ "ram_peak_gb": 0.71,
+ "gpu_util_avg": 25.5,
+ "gpu_util_peak": 25.5,
+ "gpu_mem_peak_gb": 3.33,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n## 1.1.1. Introduction\n\nDependency injection is a technique that allows you to write a single unit of code that can "
+ }
+ ]
+}
diff --git a/.perf-runs/cross-lib-20260424-expA/dotllm-throughput-3.json b/.perf-runs/cross-lib-20260424-expA/dotllm-throughput-3.json
new file mode 100644
index 00000000..7d1b97aa
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424-expA/dotllm-throughput-3.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T16:48:31.857488+00:00",
+ "run_finished_at": "2026-04-24T16:48:37.834574+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424-expA\\dotllm-throughput-3.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T16:48:32.801487+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 39.64,
+ "total_s": 4.844,
+ "cpu_avg_pct": 0,
+ "cpu_peak_pct": 0,
+ "ram_peak_gb": 0.71,
+ "gpu_util_avg": 21.5,
+ "gpu_util_peak": 21.5,
+ "gpu_mem_peak_gb": 3.33,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n## 1.1.1. Introduction\n\nDependency injection is a technique that allows you to write a single unit of code that can "
+ }
+ ]
+}
diff --git a/.perf-runs/cross-lib-20260424-expA/dotllm-throughput.json b/.perf-runs/cross-lib-20260424-expA/dotllm-throughput.json
new file mode 100644
index 00000000..c526d4ec
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424-expA/dotllm-throughput.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T16:48:01.079631+00:00",
+ "run_finished_at": "2026-04-24T16:48:09.358730+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424-expA\\dotllm-throughput.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T16:48:02.358822+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 33.76,
+ "total_s": 5.686,
+ "cpu_avg_pct": 0,
+ "cpu_peak_pct": 0,
+ "ram_peak_gb": 0.7,
+ "gpu_util_avg": 27.9,
+ "gpu_util_peak": 27.9,
+ "gpu_mem_peak_gb": 3.33,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n## 1.1.1. Introduction\n\nDependency injection is a technique that allows you to write a single unit of code that can "
+ }
+ ]
+}
diff --git a/.perf-runs/cross-lib-20260424-expA/throughput-resource-SmolLM-135M.json b/.perf-runs/cross-lib-20260424-expA/throughput-resource-SmolLM-135M.json
new file mode 100644
index 00000000..41270175
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424-expA/throughput-resource-SmolLM-135M.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T16:48:31.857488+00:00",
+ "run_finished_at": "2026-04-24T16:48:37.835576+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424-expA\\throughput-resource-SmolLM-135M.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T16:48:32.801487+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 39.64,
+ "total_s": 4.844,
+ "cpu_avg_pct": 0,
+ "cpu_peak_pct": 0,
+ "ram_peak_gb": 0.71,
+ "gpu_util_avg": 21.5,
+ "gpu_util_peak": 21.5,
+ "gpu_mem_peak_gb": 3.33,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n## 1.1.1. Introduction\n\nDependency injection is a technique that allows you to write a single unit of code that can "
+ }
+ ]
+}
diff --git a/.perf-runs/cross-lib-20260424/README.md b/.perf-runs/cross-lib-20260424/README.md
new file mode 100644
index 00000000..d9b5c4cc
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/README.md
@@ -0,0 +1,149 @@
+---
+date: 2026-04-24
+model: SmolLM-135M Q8_0 (~145 MB)
+host: Strix Halo (Ryzen AI Max+ 395 Zen5 32T), AMD Radeon 8060S iGPU
+harness: OllamaBenchmarks/scripts/benchmark_throughput_openai.py
+prompt: "Write a concise explanation of dependency injection with one short Python example."
+max_tokens: 192
+---
+
+## Cross-library throughput (end-to-end, same harness)
+
+| Backend | toks/s | total (s) | RAM peak | GPU util | Notes |
+|------------------|-------:|----------:|---------:|---------:|-------|
+| llama.cpp CPU | 34.73 | 5.53 | 0.25 GB | 14.6 | `llama-server -ngl 0 -c 2048` |
+| dotLLM CPU | **8.66** | **22.17** | 1.27 GB | 18.7 | Sample.Server, `ThreadingConfig.Auto` → 32 threads |
+
+**Gap: dotLLM is ~4.0× slower end-to-end** on this workload.
+
+Numbers captured with the fix below. Without it, dotLLM appeared to run at ~2.6 toks/s because generation was silently truncated to 12 tokens.
+
+## Correctness bug fixed during the run
+
+Found and patched in `src/DotLLM.Engine/TextGenerator.cs:850`:
+
+```csharp
+// before
+if (entry.KvCache.MaxLength >= requiredSize || entry.KvCache.MaxLength >= promptLen)
+ return (entry.KvCache, matchedTokens, false);
+// after
+if (entry.KvCache.MaxLength >= requiredSize)
+ return (entry.KvCache, matchedTokens, false);
+```
+
+Symptom: every `/v1/chat/completions` request after the first returned `finish_reason=length` with ~12 generated tokens, regardless of `max_tokens`. Cause: the warmup request primed the prefix cache with a small cache (`promptLen + warmup_max_tokens`); subsequent chat requests matched the template prefix and reused that cache. The `|| MaxLength >= promptLen` branch accepted a cache with zero room for generation, and the decode loop silently broke at `pos >= cacheSize` after exhausting the undersized cache.
+
+The fix reuses a prefix-cache entry only when it can fit the full `promptLen + maxTokens`, otherwise falls through to allocate fresh. This matches the intent of the comment one line above.
+
+Introduced in 17bc4383 (Step 36: Paged KV-cache); `/v1/completions` was unaffected in the benchmark because the harness did not issue a pre-warmup with matching tokens. Production chat users and the harness were affected.
+
+## Profile summary (dotnet-trace, `dotnet-sampled-thread-time`)
+
+30s trace while generating 192+100 tokens. Hot path on the request thread (inclusive time under `TextGenerator.Generate`, summed over the trace):
+
+| Frame | of request time |
+|-------|---:|
+| `TransformerModel.Forward` | 100% |
+| ├─ `GemmInterleaved` (prefill/lm_head/O/Down) | 41% |
+| ├─ `FusedGateUpDecode` → `FusedDecodeGemv2` | 19% |
+| ├─ `FusedQkvDecode` → `FusedDecodeGemv3` | 19% |
+| ├─ `Attention.Execute` (two overloads) | 35% |
+| └─ `ComputeThreadPool.Dispatch` | 19% |
+
+100% of leaf self-time is `UNMANAGED_CODE_TIME` because the SIMD inner loops (`VecDotQ8_0Avx512_4Rows`, `VecDotQ8_0Avx2_4RowsR4`) are JIT-emitted intrinsics that managed-stack sampling cannot walk. Kernel ETW CPU sampling (PerfView) would be needed for leaf attribution.
+
+Artefacts:
+- `dotllm-profile.nettrace` (3.7 MB) — raw trace, can be loaded in PerfView
+- `dotllm-profile.speedscope.json` (32 MB) — viewable at https://speedscope.app
+- `profile-summary.txt` — flat top-N text summary
+
+## PerfView kernel CPU profile
+
+`PerfView /ThreadTime collect` with kernel stack sampling, 25 s window, same
+150-token chat request as above. Parsed with a small TraceEvent-based
+analyzer in `.perf-runs/tools/EtlStackDump/`; full output in
+`kernel-profile.txt`. CPU samples attributed to `dotnet`: 100,059
+(no-stack: 1,665).
+
+### Where every CPU sample went (self-time, `dotnet` process only)
+
+| Bucket | Samples | % of dotnet CPU |
+|---|---:|---:|
+| `ComputeThreadPool.WorkerLoop` spin (JIT'd) — sum of all `coreclr!0x7FFF17B42B2x-6x` plus `WorkerLoop` self plus `SpinWait.SpinOnce` | ~76,000 | **~76%** |
+| kernel32/ntoskrnl/ntdll (thread wake / sched) | ~7,000 | ~7% |
+| `VecDotQ8_0Avx512_4Rows` (inner matmul kernel) | 827 | 0.84% |
+| `ComputeRowsQ8_0R4Worker` (R4-layout matmul worker) | 122 | 0.12% |
+| `AttentionWorker` | 7 | 0.01% |
+| `QuantizeF32ToQ8_0Avx512` | 1 | <0.01% |
+| everything else (JIT, GC, HTTP, ancillary) | remainder | ~15% |
+
+Attribution trick: for every `coreclr!0x7FFF17B42B??` unresolved leaf, the
+analyzer walks up the stack to the nearest managed frame. In every case
+>99% of those samples resolve to `ComputeThreadPool.WorkerLoop` or
+`SpinWait.SpinOnce`. So the "66% in unresolved coreclr" is the JIT-emitted
+body of the pool's spin loop — it's not JIT compilation, not GC, not
+unrelated runtime work.
+
+### Signal
+
+- **The 4× gap vs llama.cpp is a worker-pool coordination gap, not a
+ kernel-quality gap.** Real matmul work is ~1% of dotnet CPU time across
+ 32 threads. llama.cpp on the same box does the same compute in ~1/4
+ the wall time because its OMP-style barriers don't burn three-quarters
+ of thread-time on spin.
+- **Kernel choice is fine.** The non-R4 AVX-512 path (`VecDotQ8_0Avx512_4Rows`)
+ dominates the kernel work at 827 samples vs R4's 122 — MatMul is picking
+ the best available variant for this workload.
+- **AVX-512 R4 kernel would help only modestly.** Even if we halved the
+ 122 R4 samples to 0, we'd save ~0.1% of dotnet CPU. It's not where the
+ gap lives on this CPU for this model.
+
+## Recommended next work (in ROI order)
+
+1. **`ComputeThreadPool` rework — the actual target.** Some combination of:
+ (a) shrink the spin horizon when no work is queued (bound spin by
+ elapsed ns, not by iteration count); (b) for decode-path matmuls under
+ some threshold (say, lm_head / QKV / Gate-Up-Down at 576×576 for
+ SmolLM-135M) run single-threaded and skip dispatch entirely; (c) batch
+ Q/K/V dispatches across layers when possible. Pre-work: write a
+ microbench in `benchmarks/DotLLM.Benchmarks/ParallelBenchmarks.cs`
+ measuring `Dispatch(work=0)` round-trip vs `Dispatch(work=small)` vs
+ single-thread at realistic decode matmul sizes. ~1-2 days for the
+ microbench + first cut.
+2. **Tiled GEMM with cache blocking** (prefill-side). Still relevant for
+ the 7× prefill gap in `llama-bench` numbers, which the PerfView trace
+ didn't exercise (we only decoded 150 tokens from a short prompt).
+3. **Vulkan LLM kernels.** Largest absolute headroom on this iGPU
+ (llama.cpp Vulkan = 392 tok/s decode), but only worth pursuing after
+ the CPU path stops blocking on its own coordination.
+4. **Push 53cee5d + d7599c4 to origin**, then open an issue that tracks
+ the ComputeThreadPool rework with this profile linked.
+
+Artefacts:
+- `dotllm-kernel.etl.zip` (127 MB) — PerfView ETL archive, openable in
+ PerfView (CPU Stacks → dotnet)
+- `etl-extract/dotllm-kernel.etl` (176 MB) — unzipped for TraceEvent
+- `kernel-profile.txt` — text summary from the analyzer
+- `kernel-profile.txt` includes per-`coreclr!0x…` caller attribution
+- `tools/EtlStackDump/` — the TraceEvent-based analyzer used to produce
+ the above; build with `dotnet build -c Release`
+
+## Repro
+
+```bash
+# llama.cpp
+C:/Development/llama.cpp/llama-server.exe \
+ -m C:/Users/james/.dotllm/test-cache/QuantFactory/SmolLM-135M-GGUF/SmolLM-135M.Q8_0.gguf \
+ --port 8000 --host 127.0.0.1 -ngl 0 -c 2048 --alias SmolLM-135M --no-webui
+
+# dotLLM
+dotnet samples/DotLLM.Sample.Server/bin/Release/net10.0/DotLLM.Sample.Server.dll \
+ C:/Users/james/.dotllm/test-cache/QuantFactory/SmolLM-135M-GGUF/SmolLM-135M.Q8_0.gguf --port 8000
+
+# harness (in either case)
+cd C:/Development/OllamaBenchmarks/scripts
+python benchmark_throughput_openai.py --model SmolLM-135M \
+ --base-url http://127.0.0.1:8000 --num-predict 192 \
+ --process-name {llama-server|dotnet} \
+ --output .../throughput.json
+```
diff --git a/.perf-runs/cross-lib-20260424/analyze.py b/.perf-runs/cross-lib-20260424/analyze.py
new file mode 100644
index 00000000..550572f8
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/analyze.py
@@ -0,0 +1,59 @@
+import json, sys, re
+from collections import defaultdict
+
+path = sys.argv[1] if len(sys.argv) > 1 else '.perf-runs/cross-lib-20260424/dotllm-profile.speedscope.json'
+with open(path, 'r', encoding='utf-8') as f:
+ data = json.load(f)
+
+frames = data['shared']['frames']
+profiles = data['profiles']
+
+self_time = defaultdict(float)
+inclusive_time = defaultdict(float)
+total_time = 0.0
+
+for p in profiles:
+ events = p.get('events') or []
+ stack = []
+ last_at = None
+ for ev in events:
+ at = ev['at']
+ # Attribute time since last event to the current leaf
+ if last_at is not None and stack:
+ dt = at - last_at
+ self_time[stack[-1]] += dt
+ for fr in set(stack):
+ inclusive_time[fr] += dt
+ total_time += dt
+ if ev['type'] == 'O':
+ stack.append(ev['frame'])
+ elif ev['type'] == 'C':
+ if stack and stack[-1] == ev['frame']:
+ stack.pop()
+ else:
+ # Unmatched close — pop best match
+ try:
+ idx = len(stack) - 1 - stack[::-1].index(ev['frame'])
+ stack = stack[:idx]
+ except ValueError:
+ pass
+ last_at = at
+
+def top(label, d, n=40, filter_re=None):
+ print(f"\n=== top {n} by {label}" + (f" (filter={filter_re})" if filter_re else "") + " ===")
+ items = sorted(d.items(), key=lambda kv: -kv[1])
+ pat = re.compile(filter_re) if filter_re else None
+ count = 0
+ for idx, w in items:
+ name = frames[idx].get('name', '?')
+ if pat and not pat.search(name):
+ continue
+ pct = 100 * w / (total_time or 1)
+ print(f" {pct:5.2f}% {w:14.1f} {name}")
+ count += 1
+ if count >= n:
+ break
+
+print(f"profiles: {len(profiles)} frames: {len(frames)} total time units: {total_time:.1f}")
+top('self', self_time, 50)
+top('inclusive (dotLLM only)', inclusive_time, 40, r'DotLLM')
diff --git a/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report-full-compressed.json b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report-full-compressed.json
new file mode 100644
index 00000000..ccf59fd7
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report-full-compressed.json
@@ -0,0 +1 @@
+{"Title":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-20260424-170416","HostEnvironmentInfo":{"BenchmarkDotNetCaption":"BenchmarkDotNet","BenchmarkDotNetVersion":"0.14.0","OsVersion":"Windows 11 (10.0.26200.7019)","ProcessorName":"Unknown processor","PhysicalProcessorCount":null,"PhysicalCoreCount":null,"LogicalCoreCount":null,"RuntimeVersion":".NET 10.0.3 (10.0.326.7603)","Architecture":"X64","HasAttachedDebugger":false,"HasRyuJit":true,"Configuration":"RELEASE","DotNetCliVersion":"10.0.103","ChronometerFrequency":{"Hertz":10000000},"HardwareTimerKind":"Unknown"},"Benchmarks":[{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=2&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 2, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[333.4307909011841,298.3829975128174,315.2637481689453,234.67695713043213,291.55704975128174,308.73260498046875,284.4216823577881,299.515438079834,235.2696418762207],"N":9,"Min":234.67695713043213,"LowerFence":247.9552984237671,"Q1":284.4216823577881,"Median":298.3829975128174,"Mean":289.0278789732191,"Q3":308.73260498046875,"UpperFence":345.19898891448975,"Max":333.4307909011841,"InterquartileRange":24.310922622680664,"LowerOutliers":[234.67695713043213,235.2696418762207],"UpperOutliers":[],"AllOutliers":[234.67695713043213,235.2696418762207],"StandardError":11.252967077302678,"Variance":1139.6634123857218,"StandardDeviation":33.758901231908034,"Skewness":-0.5798074559396119,"Kurtosis":1.8486789414407394,"ConfidenceInterval":{"N":9,"Mean":289.0278789732191,"StandardError":11.252967077302678,"Level":12,"Margin":56.729644037511235,"Lower":232.29823493570788,"Upper":345.7575230107303},"Percentiles":{"P0":234.67695713043213,"P25":284.4216823577881,"P50":298.3829975128174,"P67":302.8336181640625,"P80":311.3450622558594,"P85":313.95751953125,"P90":318.89715671539307,"P95":326.1639738082886,"P100":333.4307909011841}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":4194304,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":294900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":432500},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":250400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":245700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":29200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":61300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":66200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":116600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":244700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":442100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":909700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":1705400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":3869000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":10240100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":4852100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":26810400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":20808100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":43213400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":168141200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":80635200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":253025300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":493199200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":19,"Operations":4194304,"Nanoseconds":1409431800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":9157100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":9556500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":9058100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":9380300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":9266400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":9221500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":8926800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":8769800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":8966600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":8849600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":9161000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":9239700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":9439300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":9515400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":9502300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":4194304,"Nanoseconds":9696600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":4194304,"Nanoseconds":9486600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":4194304,"Nanoseconds":9045700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":4194304,"Nanoseconds":9245000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":4194304,"Nanoseconds":9402300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":1309550000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":1283341700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":1453966200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":1407749800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":1260748700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":1331551700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":993546200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":1232118600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":1304158100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":1202190700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":1265498500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":996032100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":1501654200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":1398510100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":1251509000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":1322312000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":984306500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":1222878900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":1294918400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":1192951000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":1256258800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":986792400}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=2&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 2, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[516.8079853057861,383.68687629699707,442.47164726257324,444.39024925231934,439.080286026001,427.5961399078369,444.80137825012207,393.38183403015137,483.65540504455566],"N":9,"Min":383.68687629699707,"LowerFence":401.7882823944092,"Q1":427.5961399078369,"Median":442.47164726257324,"Mean":441.7635334862603,"Q3":444.80137825012207,"UpperFence":470.6092357635498,"Max":516.8079853057861,"InterquartileRange":17.205238342285156,"LowerOutliers":[383.68687629699707,393.38183403015137],"UpperOutliers":[483.65540504455566,516.8079853057861],"AllOutliers":[383.68687629699707,393.38183403015137,483.65540504455566,516.8079853057861],"StandardError":13.603936329583696,"Variance":1665.603752934304,"StandardDeviation":40.81180898875109,"Skewness":0.3010663930799336,"Kurtosis":2.07030723021925,"ConfidenceInterval":{"N":9,"Mean":441.7635334862603,"StandardError":13.603936329583696,"Level":12,"Margin":68.58159809628066,"Lower":373.1819353899797,"Upper":510.34513158254094},"Percentiles":{"P0":383.68687629699707,"P25":427.5961399078369,"P50":442.47164726257324,"P67":444.5382556915283,"P80":460.3429889678955,"P85":475.88459968566895,"P90":490.28592109680176,"P95":503.54695320129395,"P100":516.8079853057861}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1048576,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":274200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":627600},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":248200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":316800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":250600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":300300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":238100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":729300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":736400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1618400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":2261400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":4772000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":9206200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":21054700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":22576100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":56400300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":89345100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":66233900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":126837800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":304779200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":572461200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2130900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2146100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2056800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2194400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2126300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2171900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2127400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2106900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2087900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":1683000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":1708100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2009800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2229400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2282500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2312100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1048576,"Nanoseconds":2349900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1048576,"Nanoseconds":2102200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1048576,"Nanoseconds":2203800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1048576,"Nanoseconds":2193700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1048576,"Nanoseconds":2307200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":1048576,"Nanoseconds":2242100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":2225500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":1048576,"Nanoseconds":2257400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":1048576,"Nanoseconds":2286400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":1048576,"Nanoseconds":2287600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":529129200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":574572800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":419376700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":544127100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":404539500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":466179800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":468191600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":462623700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":450581700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":610165500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":468622700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":414705400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":509364100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":541912450},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":402324850},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":463965150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":465976950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":460409050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":448367050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":466408050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":412490750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":507149450}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=2&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 2, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[258.6294412612915,256.8997621536255,258.83777141571045,259.2512369155884,257.33630657196045,256.9296598434448,252.64384746551514,258.56592655181885],"N":8,"Min":252.64384746551514,"LowerFence":254.28317785263062,"Q1":256.92218542099,"Median":257.95111656188965,"Mean":257.3867440223694,"Q3":258.68152379989624,"UpperFence":261.3205313682556,"Max":259.2512369155884,"InterquartileRange":1.75933837890625,"LowerOutliers":[252.64384746551514],"UpperOutliers":[],"AllOutliers":[252.64384746551514],"StandardError":0.7495268874788122,"Variance":4.494324440429409,"StandardDeviation":2.119982179271658,"Skewness":-1.2306960576218822,"Kurtosis":3.261086097781749,"ConfidenceInterval":{"N":8,"Mean":257.3867440223694,"StandardError":0.7495268874788122,"Level":12,"Margin":4.05335335151043,"Lower":253.33339067085896,"Upper":261.44009737387984},"Percentiles":{"P0":252.64384746551514,"P25":256.92218542099,"P50":257.95111656188965,"P67":258.609751701355,"P80":258.75443935394287,"P85":258.8273549079895,"P90":258.9618110656738,"P95":259.1065239906311,"P100":259.2512369155884}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":358700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":361900},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":389000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":482800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":73100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":97900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":179100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":263600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":557000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":922500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1970800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3782000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7269200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":14696800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":29176800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":63188000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":94991500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":32841500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":66663500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":139685100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":269117000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":551707400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":5791000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":5313600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4999500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":5316300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4431000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4516500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4828700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4845100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4902800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":5330800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4524200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":5379300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":3497300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":3312900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":3309900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":5030700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":5471900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":3923200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4735700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4339800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4464300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":5118100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":5863800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4899000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":6080000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2097152,"Nanoseconds":5321200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2097152,"Nanoseconds":4655200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":4530500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":2097152,"Nanoseconds":4622100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":2097152,"Nanoseconds":4523400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":545903500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":544542700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":564814600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":563795400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":547023900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":543396500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":547460800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":548327900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":562214200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":544312000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":543459200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":534471200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":546890700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":542385250},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":538757850},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":542822150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":543689250},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":539673350},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":538820550},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":529832550},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":542252050}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=2&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 2, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[14339.507293701172,14359.288787841797,13143.95980834961,11852.237701416016,16072.03140258789,16553.556060791016,15938.25912475586,16475.249481201172,14497.56088256836,15260.945892333984],"N":10,"Min":11852.237701416016,"LowerFence":11803.249168395996,"Q1":14344.452667236328,"Median":14879.253387451172,"Mean":14849.259643554688,"Q3":16038.588333129883,"UpperFence":18579.791831970215,"Max":16553.556060791016,"InterquartileRange":1694.1356658935547,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":482.0424647941461,"Variance":2323649.3786481563,"StandardDeviation":1524.3521176710308,"Skewness":-0.5571236707248705,"Kurtosis":2.0071590350051163,"ConfidenceInterval":{"N":10,"Mean":14849.259643554688,"StandardError":482.0424647941461,"Level":12,"Margin":2304.602886254312,"Lower":12544.656757300376,"Upper":17153.862529809},"Percentiles":{"P0":11852.237701416016,"P25":14344.452667236328,"P50":14879.253387451172,"P67":15942.27229309082,"P80":16152.675018310547,"P85":16334.123153686523,"P90":16483.080139160156,"P95":16518.318099975586,"P100":16553.556060791016}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":65536,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":236400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":647400},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":202000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1195000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":754700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":1489100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":2368700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":5547700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":17818600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":18448400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":56629800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":38379600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":51281800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":136748400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":204036300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":406092600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":889514000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":139200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":135600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":135000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":145400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":134500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":134200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":134500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":112100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":112100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":110100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":111500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":110800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":109900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":109600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":133200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":133200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":136300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":140700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":65536,"Nanoseconds":133000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":65536,"Nanoseconds":132800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":133600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":65536,"Nanoseconds":132800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":65536,"Nanoseconds":132900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":65536,"Nanoseconds":139800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":65536,"Nanoseconds":133700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":65536,"Nanoseconds":133600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":65536,"Nanoseconds":132600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":65536,"Nanoseconds":145200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":1081894200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":935452300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":874764900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":939886900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":941183300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":861535500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":776881200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":1053429600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":1084986800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":1044662700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":1079854900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":950245100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":1000274300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":939753950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":941050350},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":861402550},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":776748250},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":1053296650},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":1084853850},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":1044529750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":1079721950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":950112150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":1000141350}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=2&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 2, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[183.42821598052979,202.55274772644043,191.98384284973145,195.7854986190796,268.2229280471802,176.9813060760498,260.5644941329956,262.75150775909424,215.18874168395996,203.56011390686035],"N":10,"Min":176.9813060760498,"LowerFence":108.50480794906616,"Q1":192.93425679206848,"Median":203.0564308166504,"Mean":216.10193967819214,"Q3":249.2205560207367,"UpperFence":333.650004863739,"Max":268.2229280471802,"InterquartileRange":56.28629922866821,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":10.956401919416512,"Variance":1200.4274301979383,"StandardDeviation":34.64718502559679,"Skewness":0.5034211747063564,"Kurtosis":1.3929627018433597,"ConfidenceInterval":{"N":10,"Mean":216.10193967819214,"StandardError":10.956401919416512,"Level":12,"Margin":52.38159981866439,"Lower":163.72033985952774,"Upper":268.4835394968565},"Percentiles":{"P0":176.9813060760498,"P25":192.93425679206848,"P50":203.0564308166504,"P67":216.55001425743103,"P80":261.00189685821533,"P85":261.9860529899597,"P90":263.29864978790283,"P95":265.7607889175415,"P100":268.2229280471802}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":4194304,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":183200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":210300},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":165700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":168800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":22700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":23700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":39600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":74100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":145500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":294000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":595900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":1158300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":2382400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":4001200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":9604600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":14309700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":9363000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":19012700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":37996700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":118647500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":211729600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":373598600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":19,"Operations":4194304,"Nanoseconds":913358100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":9151800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":8839400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":9247400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":9464300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":9988800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":9639300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":9881700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":9595100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":9390500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":10144700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":9015100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":9091200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":9183300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":9252500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":8786500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":9116600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":9147900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":9286700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":4194304,"Nanoseconds":9803900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":4194304,"Nanoseconds":9481300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":4194304,"Nanoseconds":9407300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":4194304,"Nanoseconds":9115800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":4194304,"Nanoseconds":9139000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":963803700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":932134600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":818829200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":778537000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":858751100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":814421900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":830367200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":1134191800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":751496700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":1102070000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":1111243000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":911750300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":862976300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":769353700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":849567800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":805238600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":821183900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":1125008500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":742313400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":1092886700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":1102059700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":902567000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":853793000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=2&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 2, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[328.3986568450928,367.7133083343506,337.61563301086426,261.09209060668945,429.0351390838623,466.5832042694092,419.81215476989746,412.0417594909668,286.6504669189453,341.3318634033203],"N":10,"Min":261.09209060668945,"LowerFence":199.95291829109192,"Q1":330.70290088653564,"Median":354.52258586883545,"Mean":365.02742767333984,"Q3":417.8695559501648,"UpperFence":548.6195385456085,"Max":466.5832042694092,"InterquartileRange":87.16665506362915,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":20.877831392174386,"Variance":4358.838436400623,"StandardDeviation":66.02149980423516,"Skewness":-0.037970338411685026,"Kurtosis":1.5482533312042654,"ConfidenceInterval":{"N":10,"Mean":365.02742767333984,"StandardError":20.877831392174386,"Level":12,"Margin":99.81508684236626,"Lower":265.2123408309736,"Upper":464.8425145157061},"Percentiles":{"P0":261.09209060668945,"P25":330.70290088653564,"P50":354.52258586883545,"P67":412.2748713493347,"P80":421.65675163269043,"P85":425.8070945739746,"P90":432.789945602417,"P95":449.68657493591303,"P100":466.5832042694092}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":264800},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":424000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":221400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":305900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":107400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":131300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":171100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":261200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":395700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":720200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1301900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":2473100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":4972300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":9112900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":24587400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":43087600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":98148400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":70010800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":111739000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":175590000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":471995700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":994859900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4437600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4402800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4511300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4584000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4512700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4500300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4639600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4510000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4653000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4549900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4560000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4614100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4563100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4824300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4568900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4494900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4573700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4584400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4611700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":4570200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4400200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4837900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":4605800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":734995400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":797440100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":833331700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":693275600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":775724400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":712605000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":552123500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":904325600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":983069600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":884983600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":868687900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":605723300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":720398500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":688701900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":771150700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":708031300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":547549800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":899751900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":978495900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":880409900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":864114200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":601149600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":715824800}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=2&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 2, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[275.8315086364746,273.16789627075195,274.4237422943115,258.9895248413086,258.3646297454834,262.55221366882324,265.0533676147461,257.7023506164551,261.327600479126,256.9928169250488],"N":10,"Min":256.9928169250488,"LowerFence":239.5932376384735,"Q1":258.5208535194397,"Median":261.9399070739746,"Mean":264.44056510925293,"Q3":271.1392641067505,"UpperFence":290.0668799877167,"Max":275.8315086364746,"InterquartileRange":12.618410587310791,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":2.325508296472348,"Variance":54.07988836961724,"StandardDeviation":7.353902934470732,"Skewness":0.5018131579537864,"Kurtosis":1.369778276042977,"ConfidenceInterval":{"N":10,"Mean":264.44056510925293,"StandardError":2.325508296472348,"Level":12,"Margin":11.118051880236768,"Lower":253.32251322901615,"Upper":275.5586169894897},"Percentiles":{"P0":256.9928169250488,"P25":258.5208535194397,"P50":261.9399070739746,"P67":265.29680347442627,"P80":273.41906547546387,"P85":273.9841961860657,"P90":274.56451892852783,"P95":275.1980137825012,"P100":275.8315086364746}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":275600},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":261600},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":257200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":279800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":67100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":76900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":115900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":201900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":391300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":777500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1546300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3485500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7460100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":14624800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":30332300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":60563100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":86374500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":33990400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":67951500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":137762100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":276099800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":563841400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4996700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":5119500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":5271700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":5086900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":5074500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":5334700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":5243700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4674100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4812300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4859300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":5084400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4302500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4768000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4831900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":5000500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":5113300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":5168100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":5264600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":5254100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4896200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4946400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":5085400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2097152,"Nanoseconds":5166300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2097152,"Nanoseconds":4969100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":570846400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":582615000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":565142800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":583429700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":577843700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":580477400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":548109500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":546799000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":555581000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":560826300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":545410100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":553012800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":543922100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":578460600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":572874600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":575508300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":543140400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":541829900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":550611900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":555857200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":540441000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":548043700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":538953000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=2, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=2&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 2, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[10434.36279296875,11934.097290039062,11362.071228027344,8731.352233886719,10948.187255859375,13212.701416015625,12859.849548339844,11359.689331054688,10469.664001464844,14215.435791015625],"N":10,"Min":8731.352233886719,"LowerFence":7530.619812011719,"Q1":10589.294815063477,"Median":11360.880279541016,"Mean":11552.741088867188,"Q3":12628.411483764648,"UpperFence":15687.086486816406,"Max":14215.435791015625,"InterquartileRange":2039.1166687011719,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":500.2504574553473,"Variance":2502505.2018428426,"StandardDeviation":1581.9308461000571,"Skewness":0.00954018805319687,"Kurtosis":2.0318019144317745,"ConfidenceInterval":{"N":10,"Mean":11552.741088867188,"StandardError":500.2504574553473,"Level":12,"Margin":2391.65370750887,"Lower":9161.087381358317,"Upper":13944.394796376058},"Percentiles":{"P0":8731.352233886719,"P25":10589.294815063477,"P50":11360.880279541016,"P67":11961.869857788086,"P80":12930.419921875,"P85":13089.203262329102,"P90":13312.974853515625,"P95":13764.205322265623,"P100":14215.435791015625}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":65536,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":196500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":1199800},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":138000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1750100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":1484400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":2952800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":4961000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":8042300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":21436900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":19454400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":72118000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":126974100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":97531800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":55859800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":108885800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":267134900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":792753100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":129500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":130800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":130800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":127900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":130700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":146700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":171100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":143500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":135300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":134800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":136500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":133100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":133200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":137000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":135800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":65536,"Nanoseconds":133600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":65536,"Nanoseconds":135100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":134700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":65536,"Nanoseconds":133700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":65536,"Nanoseconds":143900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":790856900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":785153900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":671579500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":683961700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":782248300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":744760000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":572353200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":717635700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":866042900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":842918400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":744603900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":686275200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":931758100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":683826400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":782113000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":744624700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":572217900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":717500400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":865907600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":842783100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":744468600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":686139900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":931622800}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=4&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 4, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[839.9788856506348,781.1376571655273,733.3870887756348,732.6624870300293,833.839225769043,814.4565582275391,797.2434997558594,689.9422645568848,501.92108154296875,507.5516700744629],"N":10,"Min":501.92108154296875,"LowerFence":536.3258600234985,"Q1":700.6223201751709,"Median":757.262372970581,"Mean":723.2120418548584,"Q3":810.1532936096191,"UpperFence":974.4497537612915,"Max":839.9788856506348,"InterquartileRange":109.53097343444824,"LowerOutliers":[501.92108154296875,507.5516700744629],"UpperOutliers":[],"AllOutliers":[501.92108154296875,507.5516700744629],"StandardError":39.41422060267703,"Variance":15534.807857164902,"StandardDeviation":124.63870930479384,"Skewness":-0.8571254581079737,"Kurtosis":2.075449112909329,"ConfidenceInterval":{"N":10,"Mean":723.2120418548584,"StandardError":39.41422060267703,"Level":12,"Margin":188.43594329222412,"Lower":534.7760985626343,"Upper":911.6479851470825},"Percentiles":{"P0":501.92108154296875,"P25":700.6223201751709,"P50":757.262372970581,"P67":797.7598915100098,"P80":818.3330917358398,"P85":827.0552921295166,"P90":834.4531917572021,"P95":837.2160387039185,"P100":839.9788856506348}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1048576,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":253200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":276800},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":205100},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":606900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":50900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":30300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":36400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":906900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":134800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":4848300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":676600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":8663300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":2248700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":4147000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":8848000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":17023100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":43510400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":99288100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":151395400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":345631300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":616202000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2490900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2153800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2252300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2387700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2698100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2571100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2270400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2209300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2608300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2173600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2180600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2304000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2279900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2178000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2179200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2117300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2158800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2188000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2185200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2323300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1048576,"Nanoseconds":2218500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1048576,"Nanoseconds":2238600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1048576,"Nanoseconds":2247700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1048576,"Nanoseconds":2263300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1048576,"Nanoseconds":2255500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":574349500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":570441300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":625183300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":883000200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":821300700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":771230600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":770470800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":876562300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":856238100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":838188900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":725675400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":528520900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":534425000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":880781700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":819082200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":769012100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":768252300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":874343800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":854019600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":835970400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":723456900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":526302400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":532206500}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=4&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 4, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[788.8387680053711,807.8622817993164,853.7059783935547,743.7116622924805,736.6607666015625,698.5282897949219,825.3198623657227,824.9269485473633,903.5202026367188,543.5943603515625],"N":10,"Min":543.5943603515625,"LowerFence":608.2262754440308,"Q1":738.423490524292,"Median":798.3505249023438,"Mean":772.6669120788574,"Q3":825.2216339111328,"UpperFence":955.418848991394,"Max":903.5202026367188,"InterquartileRange":86.79814338684082,"LowerOutliers":[543.5943603515625],"UpperOutliers":[],"AllOutliers":[543.5943603515625],"StandardError":31.76287486110234,"Variance":10088.802194420472,"StandardDeviation":100.44302959598775,"Skewness":-0.9265887481533911,"Kurtosis":3.0841814963190584,"ConfidenceInterval":{"N":10,"Mean":772.6669120788574,"StandardError":31.76287486110234,"Level":12,"Margin":151.8555281470711,"Lower":620.8113839317863,"Upper":924.5224402259286},"Percentiles":{"P0":543.5943603515625,"P25":738.423490524292,"P50":798.3505249023438,"P67":824.9387359619141,"P80":830.9970855712891,"P85":843.7708377838135,"P90":858.6874008178711,"P95":881.1038017272949,"P100":903.5202026367188}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":524288,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":253700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":502900},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":183900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":289500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":135500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":187800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":218800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":320200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":563300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":756700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1422000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":10686600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":4497000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":8291800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":19541100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":48706800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":70138300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":96976700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":287249600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":568994900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1188000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1157200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1152200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1177500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1120900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1148600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1277200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":1271500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":835000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1109800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1245100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1134600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1123700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1185900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1032000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":833500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":831500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":872400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":524288,"Nanoseconds":1716700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":524288,"Nanoseconds":825500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":524288,"Nanoseconds":834400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":524288,"Nanoseconds":831400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":524288,"Nanoseconds":830500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":837400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":524288,"Nanoseconds":1137100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":524288,"Nanoseconds":1307500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":524288,"Nanoseconds":1143800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":524288,"Nanoseconds":1273300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":364573500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":476797800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":397175900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":414649600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":424623400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":448658700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":390990000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":387293300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":367300900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":433776200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":433570200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":474775700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":286070900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":413578700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":423552500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":447587800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":389919100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":386222400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":366230000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":432705300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":432499300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":473704800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":285000000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=4&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 4, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[261.77074909210205,257.77480602264404,259.3189001083374,254.97610569000244,255.01692295074463,266.00935459136963,271.1118459701538,259.94675159454346,263.1373167037964,265.9312963485718],"N":10,"Min":254.97610569000244,"LowerFence":247.55287170410156,"Q1":258.1608295440674,"Median":260.85875034332275,"Mean":261.49940490722656,"Q3":265.23280143737793,"UpperFence":275.84075927734375,"Max":271.1118459701538,"InterquartileRange":7.071971893310547,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1.6355450642304667,"Variance":26.750076571286424,"StandardDeviation":5.172047618814663,"Skewness":0.32929971090030824,"Kurtosis":1.836563830570838,"ConfidenceInterval":{"N":10,"Mean":261.49940490722656,"StandardError":1.6355450642304667,"Level":12,"Margin":7.819397980288276,"Lower":253.6800069269383,"Upper":269.31880288751483},"Percentiles":{"P0":254.97610569000244,"P25":258.1608295440674,"P50":260.85875034332275,"P67":263.22113609313965,"P80":265.94690799713135,"P85":265.9820342063904,"P90":266.51960372924805,"P95":268.8157248497009,"P100":271.1118459701538}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":355000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":390000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":364600},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":496800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":72600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":104300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":148400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":276300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":466700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":925800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1876500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3827200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7851900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":15969300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":32293700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":64644600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":84206500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":34283100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":68567400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":137536300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":277265700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":543663000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4348000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":3345600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":3228600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":3424600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4496700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4522900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4459400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4677600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4858000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4807100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4610200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4816300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":3701400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":3805900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":3869100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4636500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4259300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4369500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4312000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4569400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4881800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":4651000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4773900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4708800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":4636200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2097152,"Nanoseconds":5901500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2097152,"Nanoseconds":4608300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":4622000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":2097152,"Nanoseconds":4528600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":2097152,"Nanoseconds":4589900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":555937000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":546227400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":563219700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":553582300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":545202200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":548440400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":539332900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":539418500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":562471300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":573172000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":549757100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":556448200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":562307600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":548973050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":540592950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":543831150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":534723650},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":534809250},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":557862050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":568562750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":545147850},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":551838950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":557698350}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=4&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 4, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[24935.842895507812,16486.517333984375,21934.185791015625,30249.728393554688,29927.566528320312,31416.046142578125,28976.028442382812,31022.259521484375,30931.100463867188,29625.057983398438],"N":10,"Min":16486.517333984375,"LowerFence":18723.587036132812,"Q1":25945.889282226562,"Median":29776.312255859375,"Mean":27550.433349609375,"Q3":30760.757446289062,"UpperFence":37983.05969238281,"Max":31416.046142578125,"InterquartileRange":4814.8681640625,"LowerOutliers":[16486.517333984375],"UpperOutliers":[],"AllOutliers":[16486.517333984375],"StandardError":1558.087686437223,"Variance":24276372.386272985,"StandardDeviation":4927.105883403866,"Skewness":-1.1421767505249993,"Kurtosis":2.822262735068446,"ConfidenceInterval":{"N":10,"Mean":27550.433349609375,"StandardError":1558.087686437223,"Level":12,"Margin":7449.081028025094,"Lower":20101.35232158428,"Upper":34999.51437763447},"Percentiles":{"P0":16486.517333984375,"P25":25945.889282226562,"P50":29776.312255859375,"P67":30270.169555664062,"P80":30949.332275390625,"P85":30990.35385131836,"P90":31061.63818359375,"P95":31238.842163085938,"P100":31416.046142578125}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":32768,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":196600},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":973300},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":812900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":2312500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":2250100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":4166500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":6410500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":18636600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":18769000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":26745900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":39207100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":121146600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":184972100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":216398900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":352710500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":857129800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":68500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":72800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":74400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":65700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":65100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":66800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":65000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":68300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":65100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":63800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":110600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":63500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":64000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":63000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":32768,"Nanoseconds":63200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":32768,"Nanoseconds":63100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":32768,"Nanoseconds":69300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":32768,"Nanoseconds":70900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":64000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":32768,"Nanoseconds":63400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":32768,"Nanoseconds":63200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":32768,"Nanoseconds":90900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":848380100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":810548000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":799060300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":817161700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":540294200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":718803400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":991287100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":980730500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":1029505000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":949550500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":32768,"Nanoseconds":1016601400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":32768,"Nanoseconds":1013614300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":32768,"Nanoseconds":970817900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":817097700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":540230200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":718739400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":991223100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":980666500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":1029441000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":949486500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":32768,"Nanoseconds":1016537400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":32768,"Nanoseconds":1013550300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":32768,"Nanoseconds":970753900}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=4&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 4, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[691.0709381103516,606.1327934265137,712.2993469238281,397.56622314453125,727.8974533081055,533.3039283752441,418.1546211242676,310.20164489746094,629.1257858276367,450.97455978393555],"N":10,"Min":310.20164489746094,"LowerFence":52.52203941345215,"Q1":426.35960578918457,"Median":569.7183609008789,"Mean":547.6727294921875,"Q3":675.5846500396729,"UpperFence":1049.4222164154053,"Max":727.8974533081055,"InterquartileRange":249.22504425048828,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":46.60168309671046,"Variance":21717.1686744623,"StandardDeviation":147.3674613829739,"Skewness":-0.18276187458188822,"Kurtosis":1.3410235669120232,"ConfidenceInterval":{"N":10,"Mean":547.6727294921875,"StandardError":46.60168309671046,"Level":12,"Margin":222.79857318141396,"Lower":324.8741563107735,"Upper":770.4713026736015},"Percentiles":{"P0":310.20164489746094,"P25":426.35960578918457,"P50":569.7183609008789,"P67":630.9841403961182,"P80":695.3166198730469,"P85":704.8694038391113,"P90":713.8591575622559,"P95":720.8783054351807,"P100":727.8974533081055}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1048576,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":244700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":312400},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":189800},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":185100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":25900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":13400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":25300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":54200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":110400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":200400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":403600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":810400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":3278800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":8023000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":15502800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":31880000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":24914800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":49230700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":147830600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":338880600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":671484300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2565800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2602900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2608300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2654800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2589200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2571000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2559000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2593200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2596800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2599400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2609100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2575700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2544900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2481600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2475500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2466800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2435600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2441900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2442800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2419500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1048576,"Nanoseconds":2401400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1048576,"Nanoseconds":2453400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1048576,"Nanoseconds":2468400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1048576,"Nanoseconds":2559600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1048576,"Nanoseconds":2558900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":490202300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":671057200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":738198500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":727108800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":638044700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":749368400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":419346800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":765724200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":561678100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":440935300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":327738400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":662154600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":475349500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":724640400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":635576300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":746900000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":416878400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":763255800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":559209700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":438466900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":325270000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":659686200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":472881100}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=4&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 4, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[557.4668407440186,440.3604030609131,496.9412326812744,430.5764675140381,532.250452041626,479.3210506439209,404.14280891418457,561.3917827606201,612.5102519989014,458.26077461242676],"N":10,"Min":404.14280891418457,"LowerFence":285.34462451934814,"Q1":444.8354959487915,"Median":488.13114166259766,"Mean":497.3222064971924,"Q3":551.1627435684204,"UpperFence":710.6536149978638,"Max":612.5102519989014,"InterquartileRange":106.3272476196289,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":21.1981600002398,"Variance":4493.619873957667,"StandardDeviation":67.03446780543325,"Skewness":0.23068913251081108,"Kurtosis":1.563260785186765,"ConfidenceInterval":{"N":10,"Mean":497.3222064971924,"StandardError":21.1981600002398,"Level":12,"Margin":101.34654991587917,"Lower":395.9756565813132,"Upper":598.6687564130716},"Percentiles":{"P0":404.14280891418457,"P25":444.8354959487915,"P50":488.13114166259766,"P67":533.0069437026978,"P80":558.2518291473389,"P85":560.0180530548096,"P90":566.5036296844482,"P95":589.5069408416748,"P100":612.5102519989014}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1048576,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":382200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":548900},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":201400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":352600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":147300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":200000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":353000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":451500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":665800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1560600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1199100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3809900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":6118900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":7042500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":24035300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":52379200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":101229100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":187461400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":164509200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":305809300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":534667500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2361000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2343300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2369400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2399000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2739400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2645100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2915800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2725800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2602800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2936200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2479800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2458200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2075400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":1746300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":1746000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":1746900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":1741200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":1737600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1048576,"Nanoseconds":1735400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1048576,"Nanoseconds":1738500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1048576,"Nanoseconds":2177200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1048576,"Nanoseconds":2431100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1048576,"Nanoseconds":2432000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":1048576,"Nanoseconds":2505400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":2542200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":1048576,"Nanoseconds":2588000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":1048576,"Nanoseconds":2579400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":1048576,"Nanoseconds":2582400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":483915900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":578929600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":554709100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":586977900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":464182900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":523512200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":453923700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":560536600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":505036100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":426206000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":591093500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":644695100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":482952800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":584546350},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":461751350},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":521080650},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":451492150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":558105050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":502604550},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":423774450},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":588661950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":642263550},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":480521250}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=4&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 4, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[257.9475164413452,255.18882274627686,261.91275119781494,262.18931674957275,257.6523542404175,263.3981943130493,264.0465497970581,260.44085025787354,256.1342477798462],"N":9,"Min":255.18882274627686,"LowerFence":250.84691047668457,"Q1":257.6523542404175,"Median":260.44085025787354,"Mean":259.87895594702826,"Q3":262.18931674957275,"UpperFence":268.99476051330566,"Max":264.0465497970581,"InterquartileRange":4.536962509155273,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1.0817430312954017,"Variance":10.53151187180548,"StandardDeviation":3.245229093886205,"Skewness":-0.12039182916855347,"Kurtosis":1.221967003951804,"ConfidenceInterval":{"N":9,"Mean":259.87895594702826,"StandardError":1.0817430312954017,"Level":12,"Margin":5.453397018216113,"Lower":254.42555892881214,"Upper":265.33235296524435},"Percentiles":{"P0":255.18882274627686,"P25":257.6523542404175,"P50":260.44085025787354,"P67":262.01231479644775,"P80":262.6728677749634,"P85":263.156418800354,"P90":263.5278654098511,"P95":263.7872076034546,"P100":264.0465497970581}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":263400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":241800},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":259900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":291400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":64300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":80300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":119300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":213400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":414400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":824800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1669600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3149800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":6598000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":15262500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":29795000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":60192400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":80902600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":33658500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":70025400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":136665800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":275723800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":549423600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4874500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4703500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4969900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4659500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4636200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4560200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4875100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4742300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4716600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":5542700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":3821800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":5520500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4674500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":5318200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4642300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4819100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":5390400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4914400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":5000200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":5044800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4838200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4957000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":5055100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2097152,"Nanoseconds":3557800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2097152,"Nanoseconds":4052600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":4509000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":2097152,"Nanoseconds":4506400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":2097152,"Nanoseconds":4566700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":548029400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":546267500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":551693300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":545783800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":585549800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":539998400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":554099500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":554679500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":545164800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":557214700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":558574400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":551012700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":541981100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":540955150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":535169750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":549270850},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":549850850},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":540336150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":552386050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":553745750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":546184050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":537152450}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=4, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=4&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 4, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[14712.722778320312,20124.267578125,25145.285034179688,18935.269165039062,14834.359741210938,14857.693481445312,20504.705810546875,14128.69873046875,19778.582763671875,13787.042236328125],"N":10,"Min":13787.042236328125,"LowerFence":6801.060485839844,"Q1":14743.132019042969,"Median":16896.481323242188,"Mean":17680.862731933594,"Q3":20037.84637451172,"UpperFence":27979.917907714844,"Max":25145.285034179688,"InterquartileRange":5294.71435546875,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1193.958439323655,"Variance":14255367.548321785,"StandardDeviation":3775.6280998426982,"Skewness":0.5362712679256124,"Kurtosis":1.8804683428145657,"ConfidenceInterval":{"N":10,"Mean":17680.862731933594,"StandardError":1193.958439323655,"Level":12,"Margin":5708.210928072586,"Lower":11972.651803861008,"Upper":23389.07366000618},"Percentiles":{"P0":13787.042236328125,"P25":14743.132019042969,"P50":16896.481323242188,"P67":19788.95330810547,"P80":20200.355224609375,"P85":20371.55242919922,"P90":20968.763732910156,"P95":23057.024383544915,"P100":25145.285034179688}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":32768,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":220100},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":758100},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":169700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1884300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":1146000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":2099200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":1839800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":4104200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":6473400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":15595600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":46686900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":80662400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":168733300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":245719700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":237105300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":642212400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":94500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":91900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":93500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":56400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":58200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":56300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":84600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":95400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":86800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":90800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":87100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":84200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":79700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":32768,"Nanoseconds":80700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":32768,"Nanoseconds":85500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":32768,"Nanoseconds":80200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":32768,"Nanoseconds":80600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":81400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":32768,"Nanoseconds":82800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":32768,"Nanoseconds":80000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":32768,"Nanoseconds":86000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":452048800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":558997000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":600288500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":482190700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":659516200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":824044900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":620555100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":486176500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":486941100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":671982400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":32768,"Nanoseconds":463053400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":32768,"Nanoseconds":648188800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":32768,"Nanoseconds":451858000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":32768,"Nanoseconds":482106500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":32768,"Nanoseconds":659432000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":32768,"Nanoseconds":823960700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":32768,"Nanoseconds":620470900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":32768,"Nanoseconds":486092300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":32768,"Nanoseconds":486856900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":32768,"Nanoseconds":671898200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":32768,"Nanoseconds":462969200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":32768,"Nanoseconds":648104600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":32768,"Nanoseconds":451773800}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=8&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 8, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[1483.1609725952148,1666.4451599121094,1569.7877883911133,1506.3692092895508,1523.2378005981445,1414.1128540039062,1569.255256652832,1420.0603485107422,1467.7038192749023,1594.3729400634766],"N":10,"Min":1414.1128540039062,"LowerFence":1324.4382858276367,"Q1":1471.5681076049805,"Median":1514.8035049438477,"Mean":1521.4506149291992,"Q3":1569.654655456543,"UpperFence":1716.7844772338867,"Max":1666.4451599121094,"InterquartileRange":98.0865478515625,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":25.243192330500076,"Variance":6372.1875903461805,"StandardDeviation":79.82598317807417,"Skewness":0.2289226749334427,"Kurtosis":1.7975961730991339,"ConfidenceInterval":{"N":10,"Mean":1521.4506149291992,"StandardError":25.243192330500076,"Level":12,"Margin":120.68549588880468,"Lower":1400.7651190403944,"Upper":1642.136110818004},"Percentiles":{"P0":1414.1128540039062,"P25":1471.5681076049805,"P50":1514.8035049438477,"P67":1569.2712326049805,"P80":1574.704818725586,"P85":1585.7681369781494,"P90":1601.5801620483396,"P95":1634.0126609802246,"P100":1666.4451599121094}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":524288,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":225500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":308000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":172300},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":236600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":96600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":70600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":94000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":92600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":267900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":431600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":8253700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":1367100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":2008000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":4400300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":16229200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":16390700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":34646100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":79421400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":144773900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":680170000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1071600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":860200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":845900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":845000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":826500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1062700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1059300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":1069500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":1085300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":1101900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1123000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1105200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1107500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1104900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1111700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1113700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1118800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":1129700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":1140200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":1149900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":524288,"Nanoseconds":1159800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":524288,"Nanoseconds":1161200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":524288,"Nanoseconds":1160200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":524288,"Nanoseconds":1164200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":524288,"Nanoseconds":1251900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":923476500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1079213700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1039931700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":778733200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":874826900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":824150600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":790901000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":799745000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":742532100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":823871400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":745650300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":770629200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":837040300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":777603500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":873697200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":823020900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":789771300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":798615300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":741402400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":822741700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":744520600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":769499500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":835910600}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=8&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 8, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[677.6828765869141,1374.8237609863281,1491.3679122924805,1577.9926300048828,1502.323341369629,1486.5150451660156,1540.8843994140625,1501.7898559570312,1553.520393371582,1544.5137023925781],"N":10,"Min":677.6828765869141,"LowerFence":1403.9110898971558,"Q1":1487.7282619476318,"Median":1502.05659866333,"Mean":1425.1413917541504,"Q3":1543.6063766479492,"UpperFence":1627.4235486984253,"Max":1577.9926300048828,"InterquartileRange":55.87811470031738,"LowerOutliers":[677.6828765869141,1374.8237609863281],"UpperOutliers":[],"AllOutliers":[677.6828765869141,1374.8237609863281],"StandardError":84.89874198055828,"Variance":72077.96389881411,"StandardDeviation":268.47339514151884,"Skewness":-2.1050981986767177,"Kurtosis":6.033398140478717,"ConfidenceInterval":{"N":10,"Mean":1425.1413917541504,"StandardError":84.89874198055828,"Level":12,"Margin":405.89346395303477,"Lower":1019.2479278011156,"Upper":1831.0348557071852},"Percentiles":{"P0":677.6828765869141,"P25":1487.7282619476318,"P50":1502.05659866333,"P67":1540.993278503418,"P80":1546.315040588379,"P85":1550.3680515289307,"P90":1555.967617034912,"P95":1566.9801235198975,"P100":1577.9926300048828}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":524288,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":317100},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":810300},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":239300},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":432400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":355700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":278700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":343500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":528200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":841300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1582900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":10254200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":4143000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7387300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":16469800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":27404000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":54215200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":114813300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":200592900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":330468000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":762248900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1099900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1098400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1093900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1154500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1097900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1098000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1106500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":1131900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":1235800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":1093100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1091200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1168100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1147500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1096600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1091900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1091500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1092500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":1090800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":1241500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":1156100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":524288,"Nanoseconds":1089700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":524288,"Nanoseconds":1094100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":524288,"Nanoseconds":1107200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":524288,"Nanoseconds":1104900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":524288,"Nanoseconds":1111300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":727911900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":772473300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":699256600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":356397600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":721900200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":783002900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":828419200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":788746700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":780458600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":808963800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":788467000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":815588700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":810866600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":355301000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":720803600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":781906300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":827322600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":787650100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":779362000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":807867200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":787370400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":814492100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":809770000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=8&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 8, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[263.19870948791504,260.512638092041,264.3120288848877,266.0970211029053,259.2947483062744,261.0729694366455,266.6475296020508,240.91863632202148,249.21422004699707,263.62500190734863],"N":10,"Min":240.91863632202148,"LowerFence":252.78764367103577,"Q1":259.59922075271606,"Median":262.1358394622803,"Mean":259.4893503189087,"Q3":264.14027214050293,"UpperFence":270.9518492221832,"Max":266.6475296020508,"InterquartileRange":4.541051387786865,"LowerOutliers":[240.91863632202148,249.21422004699707],"UpperOutliers":[],"AllOutliers":[240.91863632202148,249.21422004699707],"StandardError":2.5885744461096283,"Variance":67.00717663051768,"StandardDeviation":8.185791142615214,"Skewness":-1.2023703269279604,"Kurtosis":3.0210738286482193,"ConfidenceInterval":{"N":10,"Mean":259.4893503189087,"StandardError":2.5885744461096283,"Level":12,"Margin":12.37574814562448,"Lower":247.11360217328422,"Upper":271.86509846453316},"Percentiles":{"P0":240.91863632202148,"P25":259.59922075271606,"P50":262.1358394622803,"P67":263.6456127166748,"P80":264.6690273284912,"P85":265.4722738265991,"P90":266.1520719528198,"P95":266.3998007774353,"P100":266.6475296020508}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":393900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":468700},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":522500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":510100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":89700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":125300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":185600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":318900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":590600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1135800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":2078800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":4261200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7434000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":14443000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":28342200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":57517300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":77557800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":33762700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":70186300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":139606600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":280337400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":565938800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4644100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4573300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4610000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4568200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4608400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4639600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":5036800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4810400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4815100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4693400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4745600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4436200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4417700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4645200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4743200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4659200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4850800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4824400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":5008300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":5150800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4989800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":5010400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":4831200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":563182200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":576638700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":567880800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":556782800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":551149700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":559117600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":562861000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":548595600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":552324800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":564015500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":510058100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":527455200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":557676800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":551967700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":546334600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":554302500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":558045900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":543780500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":547509700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":559200400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":505243000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":522640100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":552861700}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=8&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 8, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[39454.693603515625,50749.8779296875,48860.577392578125,44989.544677734375,43006.93359375,43705.46875,39459.967041015625,41611.181640625,42588.458251953125,40004.57763671875],"N":10,"Min":39454.693603515625,"LowerFence":34012.78305053711,"Q1":40406.22863769531,"Median":42797.69592285156,"Mean":43443.12805175781,"Q3":44668.52569580078,"UpperFence":51061.971282958984,"Max":50749.8779296875,"InterquartileRange":4262.297058105469,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1215.963914986219,"Variance":14785682.42548613,"StandardDeviation":3845.215523931803,"Skewness":0.6660123840755143,"Kurtosis":2.000679846209282,"ConfidenceInterval":{"N":10,"Mean":43443.12805175781,"StandardError":1215.963914986219,"Level":12,"Margin":5813.417183598229,"Lower":37629.710868159586,"Upper":49256.54523535604},"Percentiles":{"P0":39454.693603515625,"P25":40406.22863769531,"P50":42797.69592285156,"P67":43743.99102783204,"P80":45763.75122070312,"P85":47505.715942382805,"P90":49049.50744628906,"P95":49899.69268798828,"P100":50749.8779296875}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":16384,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":217200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":1045000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":202700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":2773100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":1076600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":1104800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":7712300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":5482100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":14459800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":28930900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":63347100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":111002200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":216297500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":476308800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":842097700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":37200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":36300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":34700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":33400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":36800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":36600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":33600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":33600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":36500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":34200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":35900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":34900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":35400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":35900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":33900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":33800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":38200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":37000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":36200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":34100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":16384,"Nanoseconds":36800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":16384,"Nanoseconds":37100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":16384,"Nanoseconds":35600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":16384,"Nanoseconds":35800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":944544200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":680858100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":702693300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":646461500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":831521800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":800567500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":737144500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":704661400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":716106200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":646547900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":681793400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":697805100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":655470800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":646425700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":831486000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":800531700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":737108700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":704625600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":716070400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":646512100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":681757600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":697769300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":655435000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=8&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 8, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[1105.8454513549805,1039.7074699401855,1095.0838088989258,1048.9280700683594,1098.8545417785645,1052.5751113891602,1028.2953262329102,996.9865798950195,810.9001159667969,1146.5316772460938],"N":10,"Min":810.9001159667969,"LowerFence":931.0031175613403,"Q1":1031.148362159729,"Median":1050.7515907287598,"Mean":1042.3708152770996,"Q3":1097.9118585586548,"UpperFence":1198.0571031570435,"Max":1146.5316772460938,"InterquartileRange":66.76349639892578,"LowerOutliers":[810.9001159667969],"UpperOutliers":[],"AllOutliers":[810.9001159667969],"StandardError":29.219159839922334,"Variance":8537.593017509304,"StandardDeviation":92.39909641067548,"Skewness":-1.3670672563004718,"Kurtosis":4.152549689359779,"ConfidenceInterval":{"N":10,"Mean":1042.3708152770996,"StandardError":29.219159839922334,"Level":12,"Margin":139.69424899063137,"Lower":902.6765662864682,"Upper":1182.065064267731},"Percentiles":{"P0":810.9001159667969,"P25":1031.148362159729,"P50":1050.7515907287598,"P67":1095.196930885315,"P80":1100.2527236938477,"P85":1103.3986330032349,"P90":1109.9140739440918,"P95":1128.2228755950928,"P100":1146.5316772460938}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1048576,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":229700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":207000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":185200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":196700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":41000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":21500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":48000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":83500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":147100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":316700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":573200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":1308600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":2364600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":4662900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":9528200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":18388000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":39943300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":105980900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":296855400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":344541400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":953543000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2489900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2514600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2485600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2432900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2465500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2405200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2439600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2496100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2508500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2581900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2588900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2603100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2623600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2624900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2636800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2641500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1048576,"Nanoseconds":2635900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1048576,"Nanoseconds":2639000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1048576,"Nanoseconds":2630100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1048576,"Nanoseconds":2611100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1048576,"Nanoseconds":2570000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":1251575400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":1148280000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":1162885300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":1162174100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":1092823400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":1150889700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":1102491900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":1154843600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":1106316100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":1080856900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":1048027300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":852901500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":1204836700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":1159563000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":1090212300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":1148278600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":1099880800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":1152232500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":1103705000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":1078245800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":1045416200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":850290400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":1202225600}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=8&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 8, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[1136.4351272583008,1203.3628463745117,1224.427318572998,1020.0942993164062,1142.1899795532227,1319.352626800537,1331.3069343566895,1313.8486862182617,857.0784568786621,1128.102970123291],"N":10,"Min":857.0784568786621,"LowerFence":888.2250070571899,"Q1":1130.1860094070435,"Median":1172.7764129638672,"Mean":1167.619924545288,"Q3":1291.4933443069458,"UpperFence":1533.4543466567993,"Max":1331.3069343566895,"InterquartileRange":161.30733489990234,"LowerOutliers":[857.0784568786621],"UpperOutliers":[],"AllOutliers":[857.0784568786621],"StandardError":46.79557362054585,"Variance":21898.25710475927,"StandardDeviation":147.9805970550169,"Skewness":-0.6798838961607865,"Kurtosis":2.397032822159229,"ConfidenceInterval":{"N":10,"Mean":1167.619924545288,"StandardError":46.79557362054585,"Level":12,"Margin":223.72554682685663,"Lower":943.8943777184314,"Upper":1391.3454713721446},"Percentiles":{"P0":857.0784568786621,"P25":1130.1860094070435,"P50":1172.7764129638672,"P67":1227.109959602356,"P80":1314.9494743347168,"P85":1317.4262475967407,"P90":1320.5480575561523,"P95":1325.927495956421,"P100":1331.3069343566895}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1048576,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":260000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":486100},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":195400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":348200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":130000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":152800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":195100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":313700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":453800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":745700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1235100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":2160200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":4066700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":7353400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":13682700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":29136700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":97237000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":218450800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":236143500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":422724800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":637684700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2228000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2153500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2152200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2207000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2227700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2212300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2261800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2291200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2253800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":2274900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":2298700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":2302400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":2320700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":2359600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":2371200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":2386500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":2389200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":2397000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":2408900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1048576,"Nanoseconds":2411300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1048576,"Nanoseconds":2413800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1048576,"Nanoseconds":2408600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1048576,"Nanoseconds":2393700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1048576,"Nanoseconds":2397800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":903486000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":1278071500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":1135482300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":1194027800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":1264206600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":1286294300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":1072035600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":1200062200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":1385830700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":1398365700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":1380059400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":901101100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":1185290900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1048576,"Nanoseconds":1191638600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1048576,"Nanoseconds":1261817400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1048576,"Nanoseconds":1283905100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1048576,"Nanoseconds":1069646400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1048576,"Nanoseconds":1197673000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1048576,"Nanoseconds":1383441500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1048576,"Nanoseconds":1395976500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1048576,"Nanoseconds":1377670200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1048576,"Nanoseconds":898711900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":1048576,"Nanoseconds":1182901700}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=8&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 8, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[262.63816356658936,268.5833930969238,262.8746509552002,268.35758686065674,273.09417724609375,264.0660762786865,264.5477771759033,259.44788455963135,255.23204803466797,267.3555374145508],"N":10,"Min":255.23204803466797,"LowerFence":254.5826017856598,"Q1":262.69728541374207,"Median":264.3069267272949,"Mean":264.6197295188904,"Q3":268.10707449913025,"UpperFence":276.2217581272125,"Max":273.09417724609375,"InterquartileRange":5.409789085388184,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1.602578725363629,"Variance":25.682585709881145,"StandardDeviation":5.067798901878521,"Skewness":-0.18082350384101195,"Kurtosis":2.1471515572224695,"ConfidenceInterval":{"N":10,"Mean":264.6197295188904,"StandardError":1.602578725363629,"Level":12,"Margin":7.66178879593105,"Lower":256.95794072295934,"Upper":272.2815183148214},"Percentiles":{"P0":255.23204803466797,"P25":262.69728541374207,"P50":264.3069267272949,"P67":267.38559889793396,"P80":268.40274810791016,"P85":268.50436091423035,"P90":269.0344715118408,"P95":271.0643243789673,"P100":273.09417724609375}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":4194304,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":241700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":252500},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":313100},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":348700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":71200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":91100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":142600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":243800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":428400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":840300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1678600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3416900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":6978600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":13727800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":28830100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":56833400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":87683100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":36135200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":65758400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":133415200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":251572900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":463028500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":19,"Operations":4194304,"Nanoseconds":1106291300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":9164000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":9914200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":9847000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":9858300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":9539600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":10221500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":9903900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":9802500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":10173000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":9197700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":7488200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":9346200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":6874000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":10080300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":9399300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":4194304,"Nanoseconds":8755100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":4194304,"Nanoseconds":9840300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":4194304,"Nanoseconds":9367400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":4194304,"Nanoseconds":9971400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":4194304,"Nanoseconds":9490900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":4194304,"Nanoseconds":9099800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":4194304,"Nanoseconds":7399900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":4194304,"Nanoseconds":6687000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":4194304,"Nanoseconds":5439300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":4194304,"Nanoseconds":6062700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":1111421600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":1124557900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":1133597400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":1110941100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":1135877200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":1111933000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":1134930100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":1154796800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":1116930200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":1118950600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":1097560100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":1079877600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":1130727200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":4194304,"Nanoseconds":1101584300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":4194304,"Nanoseconds":1126520400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":4194304,"Nanoseconds":1102576200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":4194304,"Nanoseconds":1125573300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":4194304,"Nanoseconds":1145440000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":4194304,"Nanoseconds":1107573400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":4194304,"Nanoseconds":1109593800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":4194304,"Nanoseconds":1088203300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":4194304,"Nanoseconds":1070520800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":4194304,"Nanoseconds":1121370400}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=8, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=8&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 8, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[38829.913330078125,29584.283447265625,29192.840576171875,39892.767333984375,33424.853515625,38524.072265625,30042.938232421875,29081.878662109375,27197.967529296875,31280.419921875],"N":10,"Min":27197.967529296875,"LowerFence":17352.85186767578,"Q1":29290.701293945312,"Median":30661.679077148438,"Mean":32705.193481445312,"Q3":37249.267578125,"UpperFence":49187.11700439453,"Max":39892.767333984375,"InterquartileRange":7958.5662841796875,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1483.9994947115727,"Variance":22022545.003042035,"StandardDeviation":4692.81844982757,"Skewness":0.4725566548816489,"Kurtosis":1.3642319058921382,"ConfidenceInterval":{"N":10,"Mean":32705.193481445312,"StandardError":1483.9994947115727,"Level":12,"Margin":7094.871859832387,"Lower":25610.321621612926,"Upper":39800.0653412777},"Percentiles":{"P0":27197.967529296875,"P25":29290.701293945312,"P50":30661.679077148438,"P67":33577.830078125,"P80":38585.240478515625,"P85":38722.86895751953,"P90":38936.19873046875,"P95":39414.48303222656,"P100":39892.767333984375}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":16384,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":182300},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":823600},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":164800},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1134800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":633800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":1529800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":3866800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":7176000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":11972400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":23382900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":47208600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":92423200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":180373900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":306607800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":513338300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":43900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":31000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":106400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":35400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":36900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":41200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":37200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":36100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":44300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":36000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":35400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":28600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":26700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":25200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":25200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":33200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":35300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":34600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":16384,"Nanoseconds":47300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":16384,"Nanoseconds":35600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":16384,"Nanoseconds":38700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":16384,"Nanoseconds":50700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":16384,"Nanoseconds":31700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":16384,"Nanoseconds":32600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":16384,"Nanoseconds":31000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":16384,"Nanoseconds":30100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":16384,"Nanoseconds":29500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":558938200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":486707900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":525041100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":636223200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":484742800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":478329400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":653637000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":547666700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":631212300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":492257400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":476511400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":445645400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":512532300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":636189300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":484708900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":478295500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":653603100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":547632800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":631178400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":492223500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":476477500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":445611500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":512498400}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=16&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 16, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[9877.204895019531,9153.697204589844,8030.030822753906,10618.026733398438,4154.991149902344,2505.2398681640625,4127.3773193359375,14487.019348144531,13581.149291992188,14727.951049804688],"N":10,"Min":2505.2398681640625,"LowerFence":-6451.175308227539,"Q1":5123.751068115234,"Median":9515.451049804688,"Mean":9126.268768310547,"Q3":12840.36865234375,"UpperFence":24415.295028686523,"Max":14727.951049804688,"InterquartileRange":7716.617584228516,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1404.3374438659732,"Variance":19721636.562440157,"StandardDeviation":4440.904926075333,"Skewness":-0.13400147690909744,"Kurtosis":1.3801615389267998,"ConfidenceInterval":{"N":10,"Mean":9126.268768310547,"StandardError":1404.3374438659732,"Level":12,"Margin":6714.014558428231,"Lower":2412.254209882316,"Upper":15840.283326738778},"Percentiles":{"P0":2505.2398681640625,"P25":5123.751068115234,"P50":9515.451049804688,"P67":10706.920410156252,"P80":13762.323303222654,"P85":14169.964828491211,"P90":14511.112518310547,"P95":14619.531784057617,"P100":14727.951049804688}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":65536,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":267300},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":378000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":265200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":911800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":1089700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":1993800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":2194700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":1964200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":8644800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":38097900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":6873900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":57330900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":30049400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":163545600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":211403200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":241629600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":662696100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":163200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":155800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":156200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":163800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":156600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":156600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":160700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":155300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":155600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":161800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":156000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":158700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":162300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":158900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":158000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":161900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":161800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":65536,"Nanoseconds":156300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":65536,"Nanoseconds":161500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":158100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":65536,"Nanoseconds":157600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":65536,"Nanoseconds":155900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":600875500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":586779600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":500390400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":647470600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":600054800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":526414200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":696021100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":272459600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":164341500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":270649900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":949579400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":890212300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":965369100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":647312500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":599896700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":526256100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":695863000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":272301500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":164183400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":270491800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":949421300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":890054200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":965211000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=16&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 16, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[49932.71484375,16894.17724609375,29169.189453125,13690.85693359375,23334.9853515625,17215.283203125,10379.26025390625,8709.8876953125,7982.11669921875],"N":9,"Min":7982.11669921875,"LowerFence":-9054.327392578125,"Q1":10379.26025390625,"Median":16894.17724609375,"Mean":19700.941297743055,"Q3":23334.9853515625,"UpperFence":42768.572998046875,"Max":49932.71484375,"InterquartileRange":12955.72509765625,"LowerOutliers":[],"UpperOutliers":[49932.71484375],"AllOutliers":[49932.71484375],"StandardError":4428.474016816736,"Variance":176502439.05858862,"StandardDeviation":13285.422050450208,"Skewness":1.1621281905162486,"Kurtosis":3.159816132334599,"ConfidenceInterval":{"N":9,"Mean":19700.941297743055,"StandardError":4428.474016816736,"Level":12,"Margin":22325.29011038389,"Lower":-2624.348812640834,"Upper":42026.23140812694},"Percentiles":{"P0":7982.11669921875,"P25":10379.26025390625,"P50":16894.17724609375,"P67":19418.3759765625,"P80":25668.6669921875,"P85":28002.3486328125,"P90":33321.894531249985,"P95":41627.30468749999,"P100":49932.71484375}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":8192,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":243000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":740000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":222300},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1735900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":407500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":2212700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":4050800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":8774800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":17745700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":43073600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":70021400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":87152400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":294060600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":616962600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":20500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":17400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":16900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":16400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":15200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":14000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":13900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":13800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":14500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":13900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":14300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":18000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":18700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":17500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":16900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":16600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":18300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":16500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":16600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":16200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":8192,"Nanoseconds":17000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":8192,"Nanoseconds":16700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":8192,"Nanoseconds":16600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":8192,"Nanoseconds":17200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":8192,"Nanoseconds":18000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":8192,"Nanoseconds":16700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":8192,"Nanoseconds":20400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":8192,"Nanoseconds":18400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":8192,"Nanoseconds":16700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":8192,"Nanoseconds":16600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":610508800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":610964900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":584225500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":580668800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":409065600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":138413900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":238970800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":112172300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":191177000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":141044400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":85043700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":71368200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":65406300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":409048800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":138397100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":238954000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":112155500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":191160200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":141027600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":85026900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":71351400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":65389500}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=16&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 16, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[262.80150413513184,264.33305740356445,258.19687843322754,258.09998512268066,265.0515079498291,263.56754302978516,266.02020263671875,245.46899795532227,217.40188598632812,234.72461700439453],"N":10,"Min":217.40188598632812,"LowerFence":225.35434365272522,"Q1":248.62674474716187,"Median":260.4991912841797,"Mean":253.56661796569824,"Q3":264.14167881011963,"UpperFence":287.4140799045563,"Max":266.02020263671875,"InterquartileRange":15.514934062957764,"LowerOutliers":[217.40188598632812],"UpperOutliers":[],"AllOutliers":[217.40188598632812],"StandardError":5.119335168370165,"Variance":262.07592566111595,"StandardDeviation":16.188759237851304,"Skewness":-1.127707261523176,"Kurtosis":2.7866216588772406,"ConfidenceInterval":{"N":10,"Mean":253.56661796569824,"StandardError":5.119335168370165,"Level":12,"Margin":24.475093931335245,"Lower":229.091524034363,"Upper":278.0417118970335},"Percentiles":{"P0":217.40188598632812,"P25":248.62674474716187,"P50":260.4991912841797,"P67":263.59050846099854,"P80":264.4767475128174,"P85":264.8000502586365,"P90":265.14837741851807,"P95":265.5842900276184,"P100":266.02020263671875}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":257900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":322700},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":427800},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":477300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":65600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":92600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":129500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":221700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":461000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":895700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1866100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3600600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7252900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":14315300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":28949100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":57621400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":93776900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":37980400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":74082800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":142178700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":284366400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":563335700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4751600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4861000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4902500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4642300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4922600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4798000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":5019300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4786700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4875000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4798600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4910000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4887100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4833800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4856400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":5137500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":5055800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4911200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":4908500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4837500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4828200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":4856200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":557003100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":569493300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":555782000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":556009700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":559221600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":546353100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":546149900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":560728300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":557616200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":562759800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":519660800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":460799800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":497128200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":551134700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":554346600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":541478100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":541274900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":555853300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":552741200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":557884800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":514785800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":455924800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":492253200}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=16&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 16, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[785689.55078125,520301.3671875,565309.86328125,655850.09765625,758923.14453125,697091.30859375,674964.16015625,472083.88671875,506168.06640625,253988.8671875],"N":10,"Min":253988.8671875,"LowerFence":236914.19677734375,"Q1":509701.3916015625,"Median":610579.98046875,"Mean":589037.03125,"Q3":691559.521484375,"UpperFence":964346.7163085938,"Max":785689.55078125,"InterquartileRange":181858.1298828125,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":50458.655760749076,"Variance":25460759411.817764,"StandardDeviation":159564.2798743433,"Skewness":-0.6256060603645833,"Kurtosis":2.375295824996286,"ConfidenceInterval":{"N":10,"Mean":589037.03125,"StandardError":50458.655760749076,"Level":12,"Margin":241238.42232944138,"Lower":347798.6089205586,"Upper":830275.4535794414},"Percentiles":{"P0":253988.8671875,"P25":509701.3916015625,"P50":610579.98046875,"P67":675627.974609375,"P80":709457.6757812499,"P85":737282.0019531249,"P90":761599.78515625,"P95":773644.66796875,"P100":785689.55078125}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":1024,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":224200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":1701300},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":289700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":5099100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":7936400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":19205700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":25618500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":99671500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":146633400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":272950400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":660835200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1024,"Nanoseconds":4900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1024,"Nanoseconds":2300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1024,"Nanoseconds":2200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":1024,"Nanoseconds":3100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":1024,"Nanoseconds":3200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":1024,"Nanoseconds":3000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":2900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":1024,"Nanoseconds":2900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":1024,"Nanoseconds":3000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1024,"Nanoseconds":3400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1024,"Nanoseconds":3500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1024,"Nanoseconds":3300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1024,"Nanoseconds":2900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1024,"Nanoseconds":3300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1024,"Nanoseconds":2900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":3000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1024,"Nanoseconds":2800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1024,"Nanoseconds":3000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1024,"Nanoseconds":2900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":1024,"Nanoseconds":2900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":1024,"Nanoseconds":3700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":1024,"Nanoseconds":2800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":1024,"Nanoseconds":2300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":1024,"Nanoseconds":2400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":1024,"Nanoseconds":2300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":1024,"Nanoseconds":2200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":1024,"Nanoseconds":5800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":1024,"Nanoseconds":2700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":1024,"Nanoseconds":2200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":1024,"Nanoseconds":576513100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":1024,"Nanoseconds":665676500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":1024,"Nanoseconds":681831900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":1024,"Nanoseconds":804549000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":1024,"Nanoseconds":532791500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":1024,"Nanoseconds":578880200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":1024,"Nanoseconds":671593400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":1024,"Nanoseconds":777140200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":1024,"Nanoseconds":713824400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":691166200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":1024,"Nanoseconds":483416800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":1024,"Nanoseconds":518319000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":1024,"Nanoseconds":260087500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":1024,"Nanoseconds":804546100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":1024,"Nanoseconds":532788600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":1024,"Nanoseconds":578877300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":1024,"Nanoseconds":671590500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":1024,"Nanoseconds":777137300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":1024,"Nanoseconds":713821500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":691163300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":1024,"Nanoseconds":483413900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":1024,"Nanoseconds":518316100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":1024,"Nanoseconds":260084600}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=16&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 16, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[1992.6807403564453,2252.694320678711,2521.810531616211,2005.5549621582031,2085.013198852539,2306.364059448242,1840.7833099365234,2150.8529663085938,1865.2030944824219],"N":9,"Min":1840.7833099365234,"LowerFence":1602.6603698730469,"Q1":1992.6807403564453,"Median":2085.013198852539,"Mean":2113.439687093099,"Q3":2252.694320678711,"UpperFence":2642.7146911621094,"Max":2521.810531616211,"InterquartileRange":260.0135803222656,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":73.38738299654635,"Variance":48471.371847736016,"StandardDeviation":220.16214898963904,"Skewness":0.41054121295470164,"Kurtosis":1.856070360125659,"ConfidenceInterval":{"N":9,"Mean":2113.439687093099,"StandardError":73.38738299654635,"Level":12,"Margin":369.9682124402431,"Lower":1743.471474652856,"Upper":2483.407899533342},"Percentiles":{"P0":1840.7833099365234,"P25":1992.6807403564453,"P50":2085.013198852539,"P67":2187.515853881836,"P80":2274.1622161865234,"P85":2295.630111694336,"P90":2349.453353881836,"P95":2435.6319427490234,"P100":2521.810531616211}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":262144,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":244000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":674700},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":271000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1061200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":84900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":253200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":142500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":1915000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":508600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1117000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":2644500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":4448100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":9387800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":18038800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":58313800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":73245100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":163429800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":263893300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":520805400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":262144,"Nanoseconds":538200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":262144,"Nanoseconds":532000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":262144,"Nanoseconds":528000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":262144,"Nanoseconds":536900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":262144,"Nanoseconds":567700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":262144,"Nanoseconds":524900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":262144,"Nanoseconds":528100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":262144,"Nanoseconds":510200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":262144,"Nanoseconds":530000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":262144,"Nanoseconds":523600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":262144,"Nanoseconds":543100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":262144,"Nanoseconds":579100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":262144,"Nanoseconds":544400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":262144,"Nanoseconds":547800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":262144,"Nanoseconds":554500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":262144,"Nanoseconds":597000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":262144,"Nanoseconds":562100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":262144,"Nanoseconds":570900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":262144,"Nanoseconds":570000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":262144,"Nanoseconds":587000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":262144,"Nanoseconds":621700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":262144,"Nanoseconds":595000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":628900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":262144,"Nanoseconds":592300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":262144,"Nanoseconds":589300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":262144,"Nanoseconds":623800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":262144,"Nanoseconds":578400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":262144,"Nanoseconds":579214400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":262144,"Nanoseconds":516574600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":262144,"Nanoseconds":494856200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":262144,"Nanoseconds":876458600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":262144,"Nanoseconds":522947700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":262144,"Nanoseconds":591108700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":262144,"Nanoseconds":661655900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":262144,"Nanoseconds":526322600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":262144,"Nanoseconds":547152100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":262144,"Nanoseconds":605177900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":262144,"Nanoseconds":483128700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":262144,"Nanoseconds":564411600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":262144,"Nanoseconds":489530200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":262144,"Nanoseconds":522369300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":262144,"Nanoseconds":590530300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":262144,"Nanoseconds":661077500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":262144,"Nanoseconds":525744200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":262144,"Nanoseconds":546573700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":262144,"Nanoseconds":604599500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":262144,"Nanoseconds":482550300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":262144,"Nanoseconds":563833200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":262144,"Nanoseconds":488951800}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=16&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 16, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[1451.307487487793,1438.8799667358398,1684.7532272338867,1673.040771484375,1697.3825454711914,1745.9161758422852,1652.91748046875,1517.9672241210938,1557.7253341674805,1673.5008239746094],"N":10,"Min":1438.8799667358398,"LowerFence":1296.856689453125,"Q1":1527.9067516326904,"Median":1662.9791259765625,"Mean":1609.3391036987305,"Q3":1681.9401264190674,"UpperFence":1912.9901885986328,"Max":1745.9161758422852,"InterquartileRange":154.03337478637695,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":34.51393417986978,"Variance":11912.116525723832,"StandardDeviation":109.14264302152405,"Skewness":-0.42577259876964313,"Kurtosis":1.4250809070975252,"ConfidenceInterval":{"N":10,"Mean":1609.3391036987305,"StandardError":34.51393417986978,"Level":12,"Margin":165.00810226519528,"Lower":1444.3310014335352,"Upper":1774.3472059639257},"Percentiles":{"P0":1438.8799667358398,"P25":1527.9067516326904,"P50":1662.9791259765625,"P67":1673.8383960723877,"P80":1687.2790908813477,"P85":1692.9622840881348,"P90":1702.2359085083008,"P95":1724.076042175293,"P100":1745.9161758422852}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":524288,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":281200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":563500},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":247000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":536900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":1646000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":307600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":406100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":453500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":777100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1259700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":2394800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3507800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":6411800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":12827200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":23493200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":51598100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":98836100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":198355000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":368469200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":761728000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1111400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1147500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1108900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1150600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1122500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":1126400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":1147700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":1104100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":1091800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":1094900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":1153600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":1203900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":1171300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":1204900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":1164800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":524288,"Nanoseconds":1167000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":524288,"Nanoseconds":1269900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":524288,"Nanoseconds":1337900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":524288,"Nanoseconds":1204000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":524288,"Nanoseconds":1169300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":662523500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":746124800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":807251200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":762070100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":755554500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":884462900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":878322200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":891084300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":916529900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":867771800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":797019000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":817863700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":878563400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":524288,"Nanoseconds":760903100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":524288,"Nanoseconds":754387500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":524288,"Nanoseconds":883295900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":524288,"Nanoseconds":877155200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":524288,"Nanoseconds":889917300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":524288,"Nanoseconds":915362900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":524288,"Nanoseconds":866604800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":524288,"Nanoseconds":795852000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":524288,"Nanoseconds":816696700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":524288,"Nanoseconds":877396400}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=16&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 16, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[265.75188636779785,267.15989112854004,274.5347023010254,270.41592597961426,273.766565322876,277.82459259033203,259.8881244659424,266.58711433410645,257.3215961456299,265.45610427856445],"N":10,"Min":257.3215961456299,"LowerFence":254.4317662715912,"Q1":265.5300498008728,"Median":266.87350273132324,"Mean":267.87065029144287,"Q3":272.92890548706055,"UpperFence":284.02718901634216,"Max":277.82459259033203,"InterquartileRange":7.398855686187744,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":2.033622648659447,"Variance":41.356210771406644,"StandardDeviation":6.430879471068218,"Skewness":-0.07708158375296688,"Kurtosis":1.7271954828598837,"ConfidenceInterval":{"N":10,"Mean":267.87065029144287,"StandardError":2.033622648659447,"Level":12,"Margin":9.722572113338874,"Lower":258.148078178104,"Upper":277.5932224047817},"Percentiles":{"P0":257.3215961456299,"P25":265.5300498008728,"P50":266.87350273132324,"P67":270.5164451599121,"P80":273.92019271850586,"P85":274.2658543586731,"P90":274.86369132995605,"P95":276.34414196014404,"P100":277.82459259033203}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":611400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":331700},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":685400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":386700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":75200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":92500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":139300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":321000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":457400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1009200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1882300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3794400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7619600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":15600500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":31416900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":61961000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":87738700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":34016500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":67528300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":136662400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":271462000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":546818300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4364600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4666200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4726500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4991500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4953700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":5179200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":5174700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":5175700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4930800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4382300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4517400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4625100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4583500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4715700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4835800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4720500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4627000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4912200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":5126200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":5153100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":5180900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":5205900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2097152,"Nanoseconds":5271800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2097152,"Nanoseconds":5404700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":5153700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":2097152,"Nanoseconds":5189000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":2097152,"Nanoseconds":5178000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":553295000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":555796300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":571792700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":562350600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":565303400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":580769500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":572131800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":579158600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":587668900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":550053400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":564102200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":544671000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":561730300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":557322100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":560274900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":575741000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":567103300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":574130100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":582640400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":545024900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":559073700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":539642500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":556701800}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=16, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=16&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 16, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[43270.318603515625,53018.499755859375,52414.520263671875,53464.520263671875,51939.361572265625,52322.406005859375,54747.918701171875,55841.888427734375,53374.896240234375],"N":9,"Min":43270.318603515625,"LowerFence":50609.234619140625,"Q1":52322.406005859375,"Median":53018.499755859375,"Mean":52266.036648220484,"Q3":53464.520263671875,"UpperFence":55177.691650390625,"Max":55841.888427734375,"InterquartileRange":1142.1142578125,"LowerOutliers":[43270.318603515625],"UpperOutliers":[55841.888427734375],"AllOutliers":[43270.318603515625,55841.888427734375],"StandardError":1197.4131400132258,"Variance":12904184.050887,"StandardDeviation":3592.2394200396775,"Skewness":-1.5903126479228578,"Kurtosis":4.506602580155625,"ConfidenceInterval":{"N":9,"Mean":52266.036648220484,"StandardError":1197.4131400132258,"Level":12,"Margin":6036.52536545689,"Lower":46229.5112827636,"Upper":58302.56201367737},"Percentiles":{"P0":43270.318603515625,"P25":52322.406005859375,"P50":53018.499755859375,"P67":53407.160888671875,"P80":53977.879638671875,"P85":54491.239013671875,"P90":54966.712646484375,"P95":55404.300537109375,"P100":55841.888427734375}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":8192,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":203800},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":875200},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":183500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":2085100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":1353700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":2266300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":3679800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":6793600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":12655000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":23699100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":45756500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":101465100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":206030600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":515854200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":40000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":17100},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":16500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":17500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":17900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":16400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":16500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":17600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":18300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":16500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":16500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":17300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":17600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":16300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":16700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":17200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":17600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":16300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":16300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":16300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":8192,"Nanoseconds":20600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":8192,"Nanoseconds":16600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":8192,"Nanoseconds":17200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":8192,"Nanoseconds":16300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":8192,"Nanoseconds":18900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":8192,"Nanoseconds":16300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":447422500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":403956500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":317115700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":354487100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":434344200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":429396400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":437998000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":478368000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":425503900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":428641800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":448511600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":457473400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":437263800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":8192,"Nanoseconds":354470450},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":8192,"Nanoseconds":434327550},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":8192,"Nanoseconds":429379750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":8192,"Nanoseconds":437981350},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":8192,"Nanoseconds":425487250},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":8192,"Nanoseconds":428625150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":8192,"Nanoseconds":448494950},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":8192,"Nanoseconds":457456750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":8192,"Nanoseconds":437247150}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=32&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 32, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[45766.943359375,37661.053466796875,33303.8330078125,44697.747802734375,10736.29150390625,14714.74609375,9281.121826171875,17416.986083984375,16135.504150390625,9756.622314453125],"N":10,"Min":9281.121826171875,"LowerFence":-25530.359649658203,"Q1":11730.905151367188,"Median":16776.2451171875,"Mean":23947.0849609375,"Q3":36571.74835205078,"UpperFence":73833.01315307617,"Max":45766.943359375,"InterquartileRange":24840.843200683594,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":4669.2870389756945,"Variance":218022414.52346408,"StandardDeviation":14765.582092266599,"Skewness":0.39954596623444183,"Kurtosis":1.2311260347084803,"ConfidenceInterval":{"N":10,"Mean":23947.0849609375,"StandardError":4669.2870389756945,"Level":12,"Margin":22323.45316582971,"Lower":1623.6317951077908,"Upper":46270.538126767206},"Percentiles":{"P0":9281.121826171875,"P25":11730.905151367188,"P50":16776.2451171875,"P67":33434.54962158203,"P80":39068.39233398437,"P85":42234.90478515624,"P90":44804.66735839844,"P95":45285.80535888672,"P100":45766.943359375}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":16384,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":271800},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":574300},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":351000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":1068500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":662100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":3638300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":7873400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":11196500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":31631700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":66879700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":27997100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":42512700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":75815300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":181063500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":682000300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":39300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":36600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":34000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":32600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":34300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":33300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":33300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":33100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":37400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":34400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":34500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":32700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":34600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":33200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":33400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":29600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":31700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":29400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":29000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":16384,"Nanoseconds":33200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":16384,"Nanoseconds":34800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":16384,"Nanoseconds":32600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":16384,"Nanoseconds":32500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":16384,"Nanoseconds":32900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":16384,"Nanoseconds":34300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":16384,"Nanoseconds":36400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":16384,"Nanoseconds":33200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":16384,"Nanoseconds":33400},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":691571300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":557889200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":895862800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":749878800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":617071900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":545683200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":732361100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":175936600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":241119600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":152095100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":285393100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":264397300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":159885700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":16384,"Nanoseconds":749845600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":16384,"Nanoseconds":617038700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":16384,"Nanoseconds":545650000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":16384,"Nanoseconds":732327900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":16384,"Nanoseconds":175903400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":16384,"Nanoseconds":241086400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":16384,"Nanoseconds":152061900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":16384,"Nanoseconds":285359900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":16384,"Nanoseconds":264364100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":16384,"Nanoseconds":159852500}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=32&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 32, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[13297.30224609375,17385.16082763672,11707.208251953125,15229.18701171875,12114.450073242188,16047.377014160156,9657.156372070312,17584.962463378906],"N":8,"Min":9657.156372070312,"LowerFence":5458.864593505859,"Q1":12012.639617919922,"Median":14263.24462890625,"Mean":14127.850532531738,"Q3":16381.822967529297,"UpperFence":22935.59799194336,"Max":17584.962463378906,"InterquartileRange":4369.183349609375,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":1018.1340110321634,"Variance":8292774.9153635325,"StandardDeviation":2879.717853430008,"Skewness":-0.14649771248718782,"Kurtosis":1.3108700863893756,"ConfidenceInterval":{"N":8,"Mean":14127.850532531738,"StandardError":1018.1340110321634,"Level":12,"Margin":5505.949119164367,"Lower":8621.90141336737,"Upper":19633.799651696107},"Percentiles":{"P0":9657.156372070312,"P25":12012.639617919922,"P50":14263.24462890625,"P67":15793.73811340332,"P80":16850.047302246094,"P85":17318.27163696289,"P90":17445.101318359375,"P95":17515.03189086914,"P100":17584.962463378906}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":65536,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":353500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":948000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":232600},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":499000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":436200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":530000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":698400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":1432400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":2946300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":13231000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":9235500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":18021400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":38430000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":78805900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":335312200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":478796900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":936710900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":141500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":144000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":140900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":135000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":141300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":141300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":141900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":147400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":142400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":145800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":143300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":144300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":143100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":144600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":152000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":143800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":65536,"Nanoseconds":142600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":65536,"Nanoseconds":144100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":141300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":65536,"Nanoseconds":138900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":65536,"Nanoseconds":145300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":877025200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":1560317100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":806317100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":871595800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":1139497700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":767387400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":998203800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":794076400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":1051824700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":633035200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":1152591900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":65536,"Nanoseconds":2033974200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":65536,"Nanoseconds":3151293000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":65536,"Nanoseconds":871452000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":65536,"Nanoseconds":1139353900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":65536,"Nanoseconds":767243600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":65536,"Nanoseconds":998060000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":65536,"Nanoseconds":793932600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":65536,"Nanoseconds":1051680900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":65536,"Nanoseconds":632891400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":65536,"Nanoseconds":1152448100}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=32&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 32, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[265.6013011932373,265.34624099731445,267.9187774658203,264.6777153015137,261.66625022888184,261.7133617401123,263.5772228240967,260.8123302459717],"N":8,"Min":260.8123302459717,"LowerFence":256.13895058631897,"Q1":261.7015838623047,"Median":264.1274690628052,"Mean":263.91414999961853,"Q3":265.41000604629517,"UpperFence":270.9726393222809,"Max":267.9187774658203,"InterquartileRange":3.7084221839904785,"LowerOutliers":[],"UpperOutliers":[],"AllOutliers":[],"StandardError":0.857204394459358,"Variance":5.8783949910434785,"StandardDeviation":2.4245401607404813,"Skewness":0.17985651803768274,"Kurtosis":1.4882197614157147,"ConfidenceInterval":{"N":8,"Mean":263.91414999961853,"StandardError":0.857204394459358,"Level":12,"Margin":4.63566065908413,"Lower":259.2784893405344,"Upper":268.54981065870265},"Percentiles":{"P0":260.8123302459717,"P25":261.7015838623047,"P50":264.1274690628052,"P67":265.1389980316162,"P80":265.49927711486816,"P85":265.58854818344116,"P90":266.2965440750122,"P95":267.10766077041626,"P100":267.9187774658203}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":350400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":321400},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":483500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":467900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":66800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":88500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":160700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":272300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":504800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":968600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1792800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3795500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7774500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":15379200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":29308700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":59278700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":77882700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":34007400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":68252000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":142208100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":278439000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":553381500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4493500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4711200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4730300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4961900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4789800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4360500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":3426600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":3205800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":3207300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":3315300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":3905500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4662300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4609300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4789000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4855600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4876500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":5067200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":5246200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":5198600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4996300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4781200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":4660700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4732800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4665100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":4828000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2097152,"Nanoseconds":3664700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2097152,"Nanoseconds":5394600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":5178400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":2097152,"Nanoseconds":5335300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":2097152,"Nanoseconds":5317300},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":553209900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":558897700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":582886900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":561848100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":561313200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":566708200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":585841400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":584921500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":559911200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":553595700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":553694500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":557603300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":551804900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":557006300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":556471400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":561866400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":555069400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":548753900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":548852700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":552761500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":546963100}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=EventBased]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=32&Mode=EventBased","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 32, Mode: EventBased)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[1907840.33203125,1767222.94921875,1490432.91015625,465191.69921875,1217783.49609375,1349570.41015625,1198138.18359375,2057877.83203125,1493467.48046875,1203097.55859375],"N":10,"Min":465191.69921875,"LowerFence":468746.484375,"Q1":1206769.04296875,"Median":1420001.66015625,"Mean":1415062.28515625,"Q3":1698784.08203125,"UpperFence":2436806.640625,"Max":2057877.83203125,"InterquartileRange":492015.0390625,"LowerOutliers":[465191.69921875],"UpperOutliers":[],"AllOutliers":[465191.69921875],"StandardError":142569.07165548496,"Variance":203259401927.06805,"StandardDeviation":450842.990327085,"Skewness":-0.49635473110387246,"Kurtosis":2.5777688157172465,"ConfidenceInterval":{"N":10,"Mean":1415062.28515625,"StandardError":142569.07165548496,"Level":12,"Margin":681610.2688549247,"Lower":733452.0163013253,"Upper":2096672.5540111747},"Percentiles":{"P0":465191.69921875,"P25":1206769.04296875,"P50":1420001.66015625,"P67":1501680.14453125,"P80":1795346.4257812498,"P85":1858624.2480468748,"P90":1922844.08203125,"P95":1990360.95703125,"P100":2057877.83203125}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":512,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":253500},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":1524400},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":291700},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":5144500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":5114500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":21138600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":66548100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":98988100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":120443700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":906588300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":512,"Nanoseconds":19300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":512,"Nanoseconds":1900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":512,"Nanoseconds":2000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":512,"Nanoseconds":1700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":512,"Nanoseconds":2000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":512,"Nanoseconds":2100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":512,"Nanoseconds":2800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":512,"Nanoseconds":3200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":512,"Nanoseconds":1700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":512,"Nanoseconds":2700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":2300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":512,"Nanoseconds":4000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":512,"Nanoseconds":1400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":512,"Nanoseconds":1200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":512,"Nanoseconds":1700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":512,"Nanoseconds":1400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":512,"Nanoseconds":1300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":512,"Nanoseconds":1200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":512,"Nanoseconds":1400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":512,"Nanoseconds":1200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":512,"Nanoseconds":1900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":512,"Nanoseconds":1300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":512,"Nanoseconds":1800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":512,"Nanoseconds":1300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":512,"Nanoseconds":1200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":512,"Nanoseconds":844599500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":512,"Nanoseconds":497866600},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":512,"Nanoseconds":537370100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":512,"Nanoseconds":976815800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":512,"Nanoseconds":904819700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":512,"Nanoseconds":763103200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":512,"Nanoseconds":238179700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":512,"Nanoseconds":623506700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":690981600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":512,"Nanoseconds":613448300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":512,"Nanoseconds":1053635000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":512,"Nanoseconds":764656900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":512,"Nanoseconds":615987500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":512,"Nanoseconds":976814250},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":512,"Nanoseconds":904818150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":512,"Nanoseconds":763101650},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":512,"Nanoseconds":238178150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":512,"Nanoseconds":623505150},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":690980050},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":512,"Nanoseconds":613446750},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":512,"Nanoseconds":1053633450},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":512,"Nanoseconds":764655350},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":512,"Nanoseconds":615985950}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.Dispatch(no-op): Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_NoWork","MethodTitle":"Dispatch(no-op)","Parameters":"Threads=32&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_NoWork(Threads: 32, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[373059.521484375,353004.98046875,373340.33203125,287170.458984375,365955.615234375,351247.9736328125,329092.28515625,201680.2490234375,353193.798828125,346309.08203125],"N":10,"Min":201680.2490234375,"LowerFence":289343.46923828125,"Q1":333396.484375,"Median":352126.47705078125,"Mean":333405.4296875,"Q3":362765.1611328125,"UpperFence":406818.17626953125,"Max":373340.33203125,"InterquartileRange":29368.6767578125,"LowerOutliers":[201680.2490234375,287170.458984375],"UpperOutliers":[],"AllOutliers":[201680.2490234375,287170.458984375],"StandardError":16665.04642642572,"Variance":2777237723.949247,"StandardDeviation":52699.50401995494,"Skewness":-1.5037758539040256,"Kurtosis":4.047865422156034,"ConfidenceInterval":{"N":10,"Mean":333405.4296875,"StandardError":16665.04642642572,"Level":12,"Margin":79674.13018333158,"Lower":253731.29950416842,"Upper":413079.55987083155},"Percentiles":{"P0":201680.2490234375,"P25":333396.484375,"P50":352126.47705078125,"P67":353576.6533203125,"P80":367376.396484375,"P85":370573.154296875,"P90":373087.6025390625,"P95":373213.96728515625,"P100":373340.33203125}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":4096,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":309900},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":436000},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":175400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":3570200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":3912100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":7928400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":14066400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":26340400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":62433200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":125256100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":224795100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":66488600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":894987800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4096,"Nanoseconds":11900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4096,"Nanoseconds":8900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4096,"Nanoseconds":8800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":4096,"Nanoseconds":8500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":4096,"Nanoseconds":8900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":4096,"Nanoseconds":8500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":4096,"Nanoseconds":8600},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":4096,"Nanoseconds":8100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4096,"Nanoseconds":9000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4096,"Nanoseconds":9000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4096,"Nanoseconds":9200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4096,"Nanoseconds":8800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4096,"Nanoseconds":9500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4096,"Nanoseconds":8600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4096,"Nanoseconds":8400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4096,"Nanoseconds":8700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":11200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4096,"Nanoseconds":8300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":4096,"Nanoseconds":8600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":4096,"Nanoseconds":8300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":4096,"Nanoseconds":8400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":4096,"Nanoseconds":8300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":4096,"Nanoseconds":8500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":4096,"Nanoseconds":894234000},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":4096,"Nanoseconds":857878800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":4096,"Nanoseconds":900495000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":4096,"Nanoseconds":1528060400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":4096,"Nanoseconds":1445917000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":4096,"Nanoseconds":1529210600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":4096,"Nanoseconds":1176258800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":4096,"Nanoseconds":1498962800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":4096,"Nanoseconds":1438720300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":4096,"Nanoseconds":1347970600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":4096,"Nanoseconds":826090900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":1446690400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":4096,"Nanoseconds":1418490600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":4096,"Nanoseconds":1528051800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":4096,"Nanoseconds":1445908400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":4096,"Nanoseconds":1529202000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":4096,"Nanoseconds":1176250200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":4096,"Nanoseconds":1498954200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":4096,"Nanoseconds":1438711700},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":4096,"Nanoseconds":1347962000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":4096,"Nanoseconds":826082300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":1446681800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":4096,"Nanoseconds":1418482000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Dispatch(~1µs work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"Dispatch_SmallWork","MethodTitle":"'Dispatch(~1µs work)'","Parameters":"Threads=32&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.Dispatch_SmallWork(Threads: 32, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[367125.439453125,357914.16015625,316744.140625,369009.326171875,203978.80859375,371728.955078125,381776.416015625,378208.0078125,266854.150390625,375104.00390625],"N":10,"Min":203978.80859375,"LowerFence":256201.25122070312,"Q1":327036.6455078125,"Median":368067.3828125,"Mean":338844.3408203125,"Q3":374260.24169921875,"UpperFence":445095.6359863281,"Max":381776.416015625,"InterquartileRange":47223.59619140625,"LowerOutliers":[203978.80859375],"UpperOutliers":[],"AllOutliers":[203978.80859375],"StandardError":18774.07947789608,"Variance":3524660602.423588,"StandardDeviation":59368.85212317641,"Skewness":-1.2217426914717808,"Kurtosis":2.963988247887173,"ConfidenceInterval":{"N":10,"Mean":338844.3408203125,"StandardError":18774.07947789608,"Level":12,"Margin":89757.23284048021,"Lower":249087.10797983228,"Upper":428601.5736607927},"Percentiles":{"P0":203978.80859375,"P25":327036.6455078125,"P50":368067.3828125,"P67":371830.20654296875,"P80":375724.8046875,"P85":377121.6064453125,"P90":378564.8486328125,"P95":380170.63232421875,"P100":381776.416015625}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2048,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":296400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":1105100},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":258200},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":7560400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":6419700},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":13282800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":27267400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":51431900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":99043000},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":190036600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":352694300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":570194000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2048,"Nanoseconds":7200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2048,"Nanoseconds":5000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2048,"Nanoseconds":4800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2048,"Nanoseconds":4200},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2048,"Nanoseconds":4900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2048,"Nanoseconds":4500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2048,"Nanoseconds":4300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":4300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":2048,"Nanoseconds":4300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2048,"Nanoseconds":4800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2048,"Nanoseconds":4700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2048,"Nanoseconds":5000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2048,"Nanoseconds":4500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2048,"Nanoseconds":4800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2048,"Nanoseconds":4500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2048,"Nanoseconds":4400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":4400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2048,"Nanoseconds":4300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2048,"Nanoseconds":4600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2048,"Nanoseconds":4300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2048,"Nanoseconds":4600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2048,"Nanoseconds":4800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2048,"Nanoseconds":5000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2048,"Nanoseconds":4400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":2048,"Nanoseconds":4400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":2048,"Nanoseconds":5500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":2048,"Nanoseconds":5400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":2048,"Nanoseconds":4600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":2048,"Nanoseconds":5500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2048,"Nanoseconds":770047100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2048,"Nanoseconds":769421900},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2048,"Nanoseconds":460846000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2048,"Nanoseconds":751877500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2048,"Nanoseconds":733012800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2048,"Nanoseconds":648696600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2048,"Nanoseconds":755735700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2048,"Nanoseconds":417753200},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2048,"Nanoseconds":761305500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2048,"Nanoseconds":781882700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":774574600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2048,"Nanoseconds":546521900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2048,"Nanoseconds":768217600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2048,"Nanoseconds":751872900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2048,"Nanoseconds":733008200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2048,"Nanoseconds":648692000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2048,"Nanoseconds":755731100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2048,"Nanoseconds":417748600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2048,"Nanoseconds":761300900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2048,"Nanoseconds":781878100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":774570000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2048,"Nanoseconds":546517300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2048,"Nanoseconds":768213000}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'Single-thread same work': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"SingleThreaded_SmallWork","MethodTitle":"'Single-thread same work'","Parameters":"Threads=32&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.SingleThreaded_SmallWork(Threads: 32, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[261.1793518066406,254.3170928955078,262.73131370544434,259.5396041870117,266.62755012512207,244.4730281829834,260.4879856109619,263.72218132019043,262.8903865814209,262.78843879699707],"N":10,"Min":244.4730281829834,"LowerFence":255.14439940452576,"Q1":259.77669954299927,"Median":261.9553327560425,"Mean":259.875693321228,"Q3":262.86489963531494,"UpperFence":267.49719977378845,"Max":266.62755012512207,"InterquartileRange":3.088200092315674,"LowerOutliers":[244.4730281829834,254.3170928955078],"UpperOutliers":[],"AllOutliers":[244.4730281829834,254.3170928955078],"StandardError":1.9900797999616235,"Variance":39.604176102152955,"StandardDeviation":6.2931848933709995,"Skewness":-1.357504283137284,"Kurtosis":3.810016001196418,"ConfidenceInterval":{"N":10,"Mean":259.875693321228,"StandardError":1.9900797999616235,"Level":12,"Margin":9.514397560029359,"Lower":250.36129576119868,"Upper":269.3900908812574},"Percentiles":{"P0":244.4730281829834,"P25":259.77669954299927,"P50":261.9553327560425,"P67":262.7914972305298,"P80":263.0567455291748,"P85":263.4310531616211,"P90":264.0127182006836,"P95":265.32013416290283,"P100":266.62755012512207}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":2097152,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":324000},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":299200},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":351400},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":392500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":76100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":107500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":161800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":286300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":5,"Operations":256,"Nanoseconds":530800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":6,"Operations":512,"Nanoseconds":1013100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":7,"Operations":1024,"Nanoseconds":1955100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":8,"Operations":2048,"Nanoseconds":3731300},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":9,"Operations":4096,"Nanoseconds":7217800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":10,"Operations":8192,"Nanoseconds":14082600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":11,"Operations":16384,"Nanoseconds":29730500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":12,"Operations":32768,"Nanoseconds":55901400},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":13,"Operations":65536,"Nanoseconds":72982200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":14,"Operations":131072,"Nanoseconds":35975800},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":15,"Operations":262144,"Nanoseconds":72946200},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":16,"Operations":524288,"Nanoseconds":147361600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":17,"Operations":1048576,"Nanoseconds":300567500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":18,"Operations":2097152,"Nanoseconds":580096500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4471900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4392500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4410000},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4684300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4505700},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4411400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4471800},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4493300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4406700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":4332300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":4504300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":4487800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":4555300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":4577000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":4734100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":4455100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":4490700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":4689900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":4610600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":2097152,"Nanoseconds":4856100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":2097152,"Nanoseconds":4537800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":2097152,"Nanoseconds":4595500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":2097152,"Nanoseconds":4521500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":2097152,"Nanoseconds":4399800},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":545168500},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":550867700},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":553773700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":552270600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":537879400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":555525300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":548831800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":563696300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":517234900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":550820700},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":557603300},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":555858900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":555645100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":2097152,"Nanoseconds":547732800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":2097152,"Nanoseconds":533341600},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":2097152,"Nanoseconds":550987500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":2097152,"Nanoseconds":544294000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":2097152,"Nanoseconds":559158500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":2097152,"Nanoseconds":512697100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":2097152,"Nanoseconds":546282900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":2097152,"Nanoseconds":553065500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":2097152,"Nanoseconds":551321100},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":2097152,"Nanoseconds":551107300}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]},{"DisplayInfo":"ThreadPoolDispatchBenchmarks.'30× Dispatch(small work)': Job-TVTRHL(IterationCount=10, WarmupCount=3) [Threads=32, Mode=SpinWait]","Namespace":"DotLLM.Benchmarks","Type":"ThreadPoolDispatchBenchmarks","Method":"DispatchBurst_DecodePattern","MethodTitle":"'30× Dispatch(small work)'","Parameters":"Threads=32&Mode=SpinWait","FullName":"DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks.DispatchBurst_DecodePattern(Threads: 32, Mode: SpinWait)","HardwareIntrinsics":"AVX-512F+CD+BW+DQ+VL+VBMI,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT,AvxVnni VectorSize=256","Statistics":{"OriginalValues":[10687452.34375,10845549.21875,10457521.875,11079045.3125,10700389.84375,10729714.0625,10816628.90625,10472906.25,10670048.4375,9818729.6875],"N":10,"Min":9818729.6875,"LowerFence":10113129.19921875,"Q1":10522191.796875,"Median":10693921.09375,"Mean":10627798.59375,"Q3":10794900.1953125,"UpperFence":11203962.79296875,"Max":11079045.3125,"InterquartileRange":272708.3984375,"LowerOutliers":[9818729.6875],"UpperOutliers":[],"AllOutliers":[9818729.6875],"StandardError":106211.21454094524,"Variance":112808220942.62695,"StandardDeviation":335869.35100218205,"Skewness":-1.1285451712370402,"Kurtosis":3.7329301550178915,"ConfidenceInterval":{"N":10,"Mean":10627798.59375,"StandardError":106211.21454094524,"Level":12,"Margin":507786.53222630126,"Lower":10120012.061523698,"Upper":11135585.125976302},"Percentiles":{"P0":9818729.6875,"P25":10522191.796875,"P50":10693921.09375,"P67":10732321.5078125,"P80":10822412.96875,"P85":10835427.109375,"P90":10868898.828125,"P95":10973972.0703125,"P100":11079045.3125}},"Memory":{"Gen0Collections":0,"Gen1Collections":0,"Gen2Collections":0,"TotalOperations":128,"BytesAllocatedPerOperation":0},"Measurements":[{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":365600},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":1,"Operations":1,"Nanoseconds":8832300},{"IterationMode":"Overhead","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":349600},{"IterationMode":"Workload","IterationStage":"Jitting","LaunchIndex":1,"IterationIndex":2,"Operations":16,"Nanoseconds":68532100},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":1,"Operations":16,"Nanoseconds":106561900},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":2,"Operations":32,"Nanoseconds":231328500},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":3,"Operations":64,"Nanoseconds":427378600},{"IterationMode":"Workload","IterationStage":"Pilot","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":646017900},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":128,"Nanoseconds":3400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":128,"Nanoseconds":300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":5,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":6,"Operations":128,"Nanoseconds":300},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":7,"Operations":128,"Nanoseconds":400},{"IterationMode":"Overhead","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":8,"Operations":128,"Nanoseconds":400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":128,"Nanoseconds":800},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":128,"Nanoseconds":1000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":128,"Nanoseconds":1900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":1000},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":128,"Nanoseconds":1200},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":128,"Nanoseconds":600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":128,"Nanoseconds":600},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":11,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":12,"Operations":128,"Nanoseconds":900},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":13,"Operations":128,"Nanoseconds":400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":14,"Operations":128,"Nanoseconds":400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":15,"Operations":128,"Nanoseconds":400},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":16,"Operations":128,"Nanoseconds":500},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":17,"Operations":128,"Nanoseconds":300},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":18,"Operations":128,"Nanoseconds":2100},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":19,"Operations":128,"Nanoseconds":3700},{"IterationMode":"Overhead","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":20,"Operations":128,"Nanoseconds":1200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":1,"Operations":128,"Nanoseconds":858394100},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":2,"Operations":128,"Nanoseconds":756656200},{"IterationMode":"Workload","IterationStage":"Warmup","LaunchIndex":1,"IterationIndex":3,"Operations":128,"Nanoseconds":872570600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":1,"Operations":128,"Nanoseconds":1367994500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":2,"Operations":128,"Nanoseconds":1388230900},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":3,"Operations":128,"Nanoseconds":1338563400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":1418118400},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":5,"Operations":128,"Nanoseconds":1369650500},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":6,"Operations":128,"Nanoseconds":1373404000},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":7,"Operations":128,"Nanoseconds":1384529100},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":8,"Operations":128,"Nanoseconds":1340532600},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":9,"Operations":128,"Nanoseconds":1365766800},{"IterationMode":"Workload","IterationStage":"Actual","LaunchIndex":1,"IterationIndex":10,"Operations":128,"Nanoseconds":1256798000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":1,"Operations":128,"Nanoseconds":1367993900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":2,"Operations":128,"Nanoseconds":1388230300},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":3,"Operations":128,"Nanoseconds":1338562800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":4,"Operations":128,"Nanoseconds":1418117800},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":5,"Operations":128,"Nanoseconds":1369649900},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":6,"Operations":128,"Nanoseconds":1373403400},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":7,"Operations":128,"Nanoseconds":1384528500},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":8,"Operations":128,"Nanoseconds":1340532000},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":9,"Operations":128,"Nanoseconds":1365766200},{"IterationMode":"Workload","IterationStage":"Result","LaunchIndex":1,"IterationIndex":10,"Operations":128,"Nanoseconds":1256797400}],"Metrics":[{"Value":0,"Descriptor":{"Id":"Gen0Collects","DisplayName":"Gen0","Legend":"GC Generation 0 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":0}},{"Value":0,"Descriptor":{"Id":"Gen1Collects","DisplayName":"Gen1","Legend":"GC Generation 1 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":1}},{"Value":0,"Descriptor":{"Id":"Gen2Collects","DisplayName":"Gen2","Legend":"GC Generation 2 collects per 1000 operations","NumberFormat":"#0.0000","UnitType":0,"Unit":"Count","TheGreaterTheBetter":false,"PriorityInCategory":2}},{"Value":0,"Descriptor":{"Id":"Allocated Memory","DisplayName":"Allocated","Legend":"Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)","NumberFormat":"0.##","UnitType":2,"Unit":"B","TheGreaterTheBetter":false,"PriorityInCategory":3}}]}]}
diff --git a/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report-github.md b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report-github.md
new file mode 100644
index 00000000..1fef6a72
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report-github.md
@@ -0,0 +1,53 @@
+```
+
+BenchmarkDotNet v0.14.0, Windows 11 (10.0.26200.7019)
+Unknown processor
+.NET SDK 10.0.103
+ [Host] : .NET 10.0.3 (10.0.326.7603), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
+ Job-TVTRHL : .NET 10.0.3 (10.0.326.7603), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
+
+IterationCount=10 WarmupCount=3
+
+```
+| Method | Threads | Mode | Mean | Error | StdDev | Median | Prefill tok/s | Decode tok/s | Allocated |
+|--------------------------- |-------- |----------- |----------------:|--------------:|--------------:|----------------:|--------------:|-------------:|----------:|
+| **Dispatch(no-op)** | **2** | **EventBased** | **289.0 ns** | **56.73 ns** | **33.76 ns** | **298.4 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 2 | EventBased | 441.8 ns | 68.58 ns | 40.81 ns | 442.5 ns | N/A | N/A | - |
+| 'Single-thread same work' | 2 | EventBased | 257.4 ns | 4.05 ns | 2.12 ns | 258.0 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 2 | EventBased | 14,849.3 ns | 2,304.60 ns | 1,524.35 ns | 14,879.3 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **2** | **SpinWait** | **216.1 ns** | **52.38 ns** | **34.65 ns** | **203.1 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 2 | SpinWait | 365.0 ns | 99.82 ns | 66.02 ns | 354.5 ns | N/A | N/A | - |
+| 'Single-thread same work' | 2 | SpinWait | 264.4 ns | 11.12 ns | 7.35 ns | 261.9 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 2 | SpinWait | 11,552.7 ns | 2,391.65 ns | 1,581.93 ns | 11,360.9 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **4** | **EventBased** | **723.2 ns** | **188.44 ns** | **124.64 ns** | **757.3 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 4 | EventBased | 772.7 ns | 151.86 ns | 100.44 ns | 798.4 ns | N/A | N/A | - |
+| 'Single-thread same work' | 4 | EventBased | 261.5 ns | 7.82 ns | 5.17 ns | 260.9 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 4 | EventBased | 27,550.4 ns | 7,449.08 ns | 4,927.11 ns | 29,776.3 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **4** | **SpinWait** | **547.7 ns** | **222.80 ns** | **147.37 ns** | **569.7 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 4 | SpinWait | 497.3 ns | 101.35 ns | 67.03 ns | 488.1 ns | N/A | N/A | - |
+| 'Single-thread same work' | 4 | SpinWait | 259.9 ns | 5.45 ns | 3.25 ns | 260.4 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 4 | SpinWait | 17,680.9 ns | 5,708.21 ns | 3,775.63 ns | 16,896.5 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **8** | **EventBased** | **1,521.5 ns** | **120.69 ns** | **79.83 ns** | **1,514.8 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 8 | EventBased | 1,425.1 ns | 405.89 ns | 268.47 ns | 1,502.1 ns | N/A | N/A | - |
+| 'Single-thread same work' | 8 | EventBased | 259.5 ns | 12.38 ns | 8.19 ns | 262.1 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 8 | EventBased | 43,443.1 ns | 5,813.42 ns | 3,845.22 ns | 42,797.7 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **8** | **SpinWait** | **1,042.4 ns** | **139.69 ns** | **92.40 ns** | **1,050.8 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 8 | SpinWait | 1,167.6 ns | 223.73 ns | 147.98 ns | 1,172.8 ns | N/A | N/A | - |
+| 'Single-thread same work' | 8 | SpinWait | 264.6 ns | 7.66 ns | 5.07 ns | 264.3 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 8 | SpinWait | 32,705.2 ns | 7,094.87 ns | 4,692.82 ns | 30,661.7 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **16** | **EventBased** | **9,126.3 ns** | **6,714.01 ns** | **4,440.90 ns** | **9,515.5 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 16 | EventBased | 19,700.9 ns | 22,325.29 ns | 13,285.42 ns | 16,894.2 ns | N/A | N/A | - |
+| 'Single-thread same work' | 16 | EventBased | 253.6 ns | 24.48 ns | 16.19 ns | 260.5 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 16 | EventBased | 589,037.0 ns | 241,238.42 ns | 159,564.28 ns | 610,580.0 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **16** | **SpinWait** | **2,113.4 ns** | **369.97 ns** | **220.16 ns** | **2,085.0 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 16 | SpinWait | 1,609.3 ns | 165.01 ns | 109.14 ns | 1,663.0 ns | N/A | N/A | - |
+| 'Single-thread same work' | 16 | SpinWait | 267.9 ns | 9.72 ns | 6.43 ns | 266.9 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 16 | SpinWait | 52,266.0 ns | 6,036.53 ns | 3,592.24 ns | 53,018.5 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **32** | **EventBased** | **23,947.1 ns** | **22,323.45 ns** | **14,765.58 ns** | **16,776.2 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 32 | EventBased | 14,127.9 ns | 5,505.95 ns | 2,879.72 ns | 14,263.2 ns | N/A | N/A | - |
+| 'Single-thread same work' | 32 | EventBased | 263.9 ns | 4.64 ns | 2.42 ns | 264.1 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 32 | EventBased | 1,415,062.3 ns | 681,610.27 ns | 450,842.99 ns | 1,420,001.7 ns | N/A | N/A | - |
+| **Dispatch(no-op)** | **32** | **SpinWait** | **333,405.4 ns** | **79,674.13 ns** | **52,699.50 ns** | **352,126.5 ns** | **N/A** | **N/A** | **-** |
+| 'Dispatch(~1µs work)' | 32 | SpinWait | 338,844.3 ns | 89,757.23 ns | 59,368.85 ns | 368,067.4 ns | N/A | N/A | - |
+| 'Single-thread same work' | 32 | SpinWait | 259.9 ns | 9.51 ns | 6.29 ns | 262.0 ns | N/A | N/A | - |
+| '30× Dispatch(small work)' | 32 | SpinWait | 10,627,798.6 ns | 507,786.53 ns | 335,869.35 ns | 10,693,921.1 ns | N/A | N/A | - |
diff --git a/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report.csv b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report.csv
new file mode 100644
index 00000000..d51759f9
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report.csv
@@ -0,0 +1,41 @@
+Method,Job,AnalyzeLaunchVariance,EvaluateOverhead,MaxAbsoluteError,MaxRelativeError,MinInvokeCount,MinIterationTime,OutlierMode,Affinity,EnvironmentVariables,Jit,LargeAddressAware,Platform,PowerPlanMode,Runtime,AllowVeryLargeObjects,Concurrent,CpuGroups,Force,HeapAffinitizeMask,HeapCount,NoAffinitize,RetainVm,Server,Arguments,BuildConfiguration,Clock,EngineFactory,NuGetReferences,Toolchain,IsMutator,InvocationCount,IterationCount,IterationTime,LaunchCount,MaxIterationCount,MaxWarmupIterationCount,MemoryRandomization,MinIterationCount,MinWarmupIterationCount,RunStrategy,UnrollFactor,WarmupCount,Threads,Mode,Mean,Error,StdDev,Median,Prefill tok/s,Decode tok/s,Allocated
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,EventBased,289.0 ns,56.73 ns,33.76 ns,298.4 ns,N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,EventBased,441.8 ns,68.58 ns,40.81 ns,442.5 ns,N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,EventBased,257.4 ns,4.05 ns,2.12 ns,258.0 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,EventBased,"14,849.3 ns","2,304.60 ns","1,524.35 ns","14,879.3 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,SpinWait,216.1 ns,52.38 ns,34.65 ns,203.1 ns,N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,SpinWait,365.0 ns,99.82 ns,66.02 ns,354.5 ns,N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,SpinWait,264.4 ns,11.12 ns,7.35 ns,261.9 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,2,SpinWait,"11,552.7 ns","2,391.65 ns","1,581.93 ns","11,360.9 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,EventBased,723.2 ns,188.44 ns,124.64 ns,757.3 ns,N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,EventBased,772.7 ns,151.86 ns,100.44 ns,798.4 ns,N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,EventBased,261.5 ns,7.82 ns,5.17 ns,260.9 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,EventBased,"27,550.4 ns","7,449.08 ns","4,927.11 ns","29,776.3 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,SpinWait,547.7 ns,222.80 ns,147.37 ns,569.7 ns,N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,SpinWait,497.3 ns,101.35 ns,67.03 ns,488.1 ns,N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,SpinWait,259.9 ns,5.45 ns,3.25 ns,260.4 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,4,SpinWait,"17,680.9 ns","5,708.21 ns","3,775.63 ns","16,896.5 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,EventBased,"1,521.5 ns",120.69 ns,79.83 ns,"1,514.8 ns",N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,EventBased,"1,425.1 ns",405.89 ns,268.47 ns,"1,502.1 ns",N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,EventBased,259.5 ns,12.38 ns,8.19 ns,262.1 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,EventBased,"43,443.1 ns","5,813.42 ns","3,845.22 ns","42,797.7 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,SpinWait,"1,042.4 ns",139.69 ns,92.40 ns,"1,050.8 ns",N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,SpinWait,"1,167.6 ns",223.73 ns,147.98 ns,"1,172.8 ns",N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,SpinWait,264.6 ns,7.66 ns,5.07 ns,264.3 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,8,SpinWait,"32,705.2 ns","7,094.87 ns","4,692.82 ns","30,661.7 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,EventBased,"9,126.3 ns","6,714.01 ns","4,440.90 ns","9,515.5 ns",N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,EventBased,"19,700.9 ns","22,325.29 ns","13,285.42 ns","16,894.2 ns",N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,EventBased,253.6 ns,24.48 ns,16.19 ns,260.5 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,EventBased,"589,037.0 ns","241,238.42 ns","159,564.28 ns","610,580.0 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,SpinWait,"2,113.4 ns",369.97 ns,220.16 ns,"2,085.0 ns",N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,SpinWait,"1,609.3 ns",165.01 ns,109.14 ns,"1,663.0 ns",N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,SpinWait,267.9 ns,9.72 ns,6.43 ns,266.9 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,16,SpinWait,"52,266.0 ns","6,036.53 ns","3,592.24 ns","53,018.5 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,EventBased,"23,947.1 ns","22,323.45 ns","14,765.58 ns","16,776.2 ns",N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,EventBased,"14,127.9 ns","5,505.95 ns","2,879.72 ns","14,263.2 ns",N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,EventBased,263.9 ns,4.64 ns,2.42 ns,264.1 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,EventBased,"1,415,062.3 ns","681,610.27 ns","450,842.99 ns","1,420,001.7 ns",N/A,N/A,0 B
+Dispatch(no-op),Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,SpinWait,"333,405.4 ns","79,674.13 ns","52,699.50 ns","352,126.5 ns",N/A,N/A,0 B
+'Dispatch(~1µs work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,SpinWait,"338,844.3 ns","89,757.23 ns","59,368.85 ns","368,067.4 ns",N/A,N/A,0 B
+'Single-thread same work',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,SpinWait,259.9 ns,9.51 ns,6.29 ns,262.0 ns,N/A,N/A,0 B
+'30× Dispatch(small work)',Job-TVTRHL,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 10.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,10,Default,Default,Default,Default,Default,Default,Default,Default,16,3,32,SpinWait,"10,627,798.6 ns","507,786.53 ns","335,869.35 ns","10,693,921.1 ns",N/A,N/A,0 B
diff --git a/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report.html b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report.html
new file mode 100644
index 00000000..f0d9d969
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/bench/results/DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-report.html
@@ -0,0 +1,70 @@
+
+
+
+
+DotLLM.Benchmarks.ThreadPoolDispatchBenchmarks-20260424-170416
+
+
+
+
+
+BenchmarkDotNet v0.14.0, Windows 11 (10.0.26200.7019)
+Unknown processor
+.NET SDK 10.0.103
+ [Host] : .NET 10.0.3 (10.0.326.7603), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
+ Job-TVTRHL : .NET 10.0.3 (10.0.326.7603), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
+
+IterationCount=10 WarmupCount=3
+
+
+
+| Method | Threads | Mode | Mean | Error | StdDev | Median | Prefill tok/s | Decode tok/s | Allocated |
+
+| Dispatch(no-op) | 2 | EventBased | 289.0 ns | 56.73 ns | 33.76 ns | 298.4 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 2 | EventBased | 441.8 ns | 68.58 ns | 40.81 ns | 442.5 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 2 | EventBased | 257.4 ns | 4.05 ns | 2.12 ns | 258.0 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 2 | EventBased | 14,849.3 ns | 2,304.60 ns | 1,524.35 ns | 14,879.3 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 2 | SpinWait | 216.1 ns | 52.38 ns | 34.65 ns | 203.1 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 2 | SpinWait | 365.0 ns | 99.82 ns | 66.02 ns | 354.5 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 2 | SpinWait | 264.4 ns | 11.12 ns | 7.35 ns | 261.9 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 2 | SpinWait | 11,552.7 ns | 2,391.65 ns | 1,581.93 ns | 11,360.9 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 4 | EventBased | 723.2 ns | 188.44 ns | 124.64 ns | 757.3 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 4 | EventBased | 772.7 ns | 151.86 ns | 100.44 ns | 798.4 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 4 | EventBased | 261.5 ns | 7.82 ns | 5.17 ns | 260.9 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 4 | EventBased | 27,550.4 ns | 7,449.08 ns | 4,927.11 ns | 29,776.3 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 4 | SpinWait | 547.7 ns | 222.80 ns | 147.37 ns | 569.7 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 4 | SpinWait | 497.3 ns | 101.35 ns | 67.03 ns | 488.1 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 4 | SpinWait | 259.9 ns | 5.45 ns | 3.25 ns | 260.4 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 4 | SpinWait | 17,680.9 ns | 5,708.21 ns | 3,775.63 ns | 16,896.5 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 8 | EventBased | 1,521.5 ns | 120.69 ns | 79.83 ns | 1,514.8 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 8 | EventBased | 1,425.1 ns | 405.89 ns | 268.47 ns | 1,502.1 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 8 | EventBased | 259.5 ns | 12.38 ns | 8.19 ns | 262.1 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 8 | EventBased | 43,443.1 ns | 5,813.42 ns | 3,845.22 ns | 42,797.7 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 8 | SpinWait | 1,042.4 ns | 139.69 ns | 92.40 ns | 1,050.8 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 8 | SpinWait | 1,167.6 ns | 223.73 ns | 147.98 ns | 1,172.8 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 8 | SpinWait | 264.6 ns | 7.66 ns | 5.07 ns | 264.3 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 8 | SpinWait | 32,705.2 ns | 7,094.87 ns | 4,692.82 ns | 30,661.7 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 16 | EventBased | 9,126.3 ns | 6,714.01 ns | 4,440.90 ns | 9,515.5 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 16 | EventBased | 19,700.9 ns | 22,325.29 ns | 13,285.42 ns | 16,894.2 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 16 | EventBased | 253.6 ns | 24.48 ns | 16.19 ns | 260.5 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 16 | EventBased | 589,037.0 ns | 241,238.42 ns | 159,564.28 ns | 610,580.0 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 16 | SpinWait | 2,113.4 ns | 369.97 ns | 220.16 ns | 2,085.0 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 16 | SpinWait | 1,609.3 ns | 165.01 ns | 109.14 ns | 1,663.0 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 16 | SpinWait | 267.9 ns | 9.72 ns | 6.43 ns | 266.9 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 16 | SpinWait | 52,266.0 ns | 6,036.53 ns | 3,592.24 ns | 53,018.5 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 32 | EventBased | 23,947.1 ns | 22,323.45 ns | 14,765.58 ns | 16,776.2 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 32 | EventBased | 14,127.9 ns | 5,505.95 ns | 2,879.72 ns | 14,263.2 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 32 | EventBased | 263.9 ns | 4.64 ns | 2.42 ns | 264.1 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 32 | EventBased | 1,415,062.3 ns | 681,610.27 ns | 450,842.99 ns | 1,420,001.7 ns | N/A | N/A | - |
+
| Dispatch(no-op) | 32 | SpinWait | 333,405.4 ns | 79,674.13 ns | 52,699.50 ns | 352,126.5 ns | N/A | N/A | - |
+
| 'Dispatch(~1µs work)' | 32 | SpinWait | 338,844.3 ns | 89,757.23 ns | 59,368.85 ns | 368,067.4 ns | N/A | N/A | - |
+
| 'Single-thread same work' | 32 | SpinWait | 259.9 ns | 9.51 ns | 6.29 ns | 262.0 ns | N/A | N/A | - |
+
| '30× Dispatch(small work)' | 32 | SpinWait | 10,627,798.6 ns | 507,786.53 ns | 335,869.35 ns | 10,693,921.1 ns | N/A | N/A | - |
+
+
+
diff --git a/.perf-runs/cross-lib-20260424/dispatch-microbench.md b/.perf-runs/cross-lib-20260424/dispatch-microbench.md
new file mode 100644
index 00000000..9fad9391
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/dispatch-microbench.md
@@ -0,0 +1,116 @@
+---
+date: 2026-04-24
+benchmark: ThreadPoolDispatchBenchmarks
+model-context: SmolLM-135M decode matmul size (~576-element row per partition)
+host: Strix Halo (32 logical CPUs)
+bdn: v0.14.0, .NET 10.0.3, AVX-512F+CD+BW+DQ+VL+VBMI
+---
+
+## Headline
+
+At the thread count we actually use for SmolLM-135M decode (`ThreadingConfig.Auto` = 32), `ComputeThreadPool.Dispatch` costs **more than the single-threaded work it replaces** at every problem size relevant to decode. SpinWait mode collapses at 32 threads — one dispatch takes **333 µs** (1300× the 260 ns single-threaded baseline) and a 30-dispatch decode-layer burst takes **10.6 ms** versus 7.8 µs sequential.
+
+This is why the PerfView trace showed 76% of CPU in the worker spin loop: at 32 threads, SpinWait is not helping — the workers are saturating the cache line on `_dispatchGeneration` and spinning through their 10,000-iteration budget before the caller can even increment the generation counter for the next dispatch.
+
+## Per-dispatch cost (ns) — Dispatch(no-op) vs single-threaded baseline
+
+| Threads | EventBased | SpinWait | Single-thread work |
+|--------:|-----------:|---------:|-------------------:|
+| 2 | 289 | 216 | 260 |
+| 4 | 723 | 548 | 262 |
+| 8 | 1,522 | 1,042 | 260 |
+| 16 | 9,126 | 2,113 | 254 |
+| 32 | 23,947 | **333,405** | 264 |
+
+Any row where Dispatch > Single-thread means parallelism is a net loss at
+this work size. That's every row from 4 threads up.
+
+## 30-dispatch burst (one decode layer loop)
+
+Sequential single-threaded = 30 × 260 ns = **7,800 ns (7.8 µs)** ideal.
+
+| Threads | EventBased | SpinWait | Ratio vs sequential |
+|--------:|----------------:|------------------:|--------------------:|
+| 2 | 14,849 ns | 11,553 ns | 1.5× slower |
+| 4 | 27,550 ns | 17,681 ns | 2.3× slower |
+| 8 | 43,443 ns | 32,705 ns | 4.2× slower |
+| 16 | 589,037 ns | 52,266 ns | 6.7× slower |
+| 32 | 1,415,062 ns | **10,627,799 ns** | **1,362× slower** |
+
+SmolLM-135M decodes 30 layers per token. At 32 threads SpinWait, each
+token spends **~10 ms** just in worker-pool coordination. Across 150
+tokens that's 1.6 s of the 22 s total — but the profile showed 76% of
+CPU in the spin loop, i.e. ~17 s of summed thread-time across 32
+threads. The extra cost shows up as workers spinning *between* bursts:
+once a burst ends, workers spin their 10K-iteration budget before
+falling through to event wait; if the caller issues the next burst in
+under 1 ms, they never reach event wait and keep spinning.
+
+## Why SpinWait degrades so badly at 32T
+
+1. **Cache-line contention.** All 31 workers spin on `_dispatchGeneration`
+ (`Volatile.Read` into a single volatile int). When the caller
+ increments it, the cache line has to invalidate on 31 cores; the first
+ reader wins, others re-fetch. At 32 threads this becomes the bottleneck.
+2. **`_completion.Signal()` is contended.** `CountdownEvent.Signal()` is
+ an interlocked decrement on a shared counter. 31 threads finishing
+ roughly together all take that hit.
+3. **OS oversubscription.** 32 logical threads on a 32-core host with
+ any other load (explorer.exe, antivirus, the benchmark harness
+ itself) means the scheduler de-schedules workers mid-spin, adding
+ context-switch cost back on top of the spin cost.
+4. **False sharing around `_activeWorkerCount` / `_dispatchThreadCount`.**
+ These are read by every worker each iteration; they share a cache
+ line with `_dispatchGeneration`.
+
+EventBased scales better at 32T (24 µs vs SpinWait's 333 µs) because the
+kernel arbitrates the wake — only one worker runs the signal path at a
+time and the rest are parked, not spinning.
+
+## Implications for dotLLM CPU perf
+
+1. **`_decodeThreadCount` default is wrong for this host.** When no
+ `NumaTopology` is provided (the common path — `ThreadingConfig.Auto`
+ does not build one), `_decodeThreadCount` falls back to `threadCount`
+ (32), so SpinWait runs on all 31 workers. It should instead default
+ to `min(8, threadCount)` or similar — decode is memory-bandwidth
+ bound and two channels per Zen 5 CCD don't reward more than ~8
+ concurrent loads.
+2. **Single-thread fast path below a size threshold.** Any decode
+ matmul where per-thread work < ~500 ns should skip dispatch. For a
+ 576×576 Q8_0 matmul split across 32 threads, per-worker work is
+ roughly 18 rows × 576 elements × ~2 cycles/elem = ~40 µs — above the
+ threshold, but at 1 token × 1536-column Gate-Up-Down split across 32
+ the per-worker chunk drops well below. A size-gated dispatch would
+ pay off here.
+3. **Reduce cross-thread sharing on the hot path.** Cache-line-pad
+ `_dispatchGeneration`, `_activeWorkerCount`, and `_dispatchThreadCount`.
+ Use a tree-reduction completion (two-level `CountdownEvent`) rather
+ than 31 threads signalling a single atomic counter.
+4. **Prefill still benefits from all 32 threads** — EventBased scales
+ linearly through 16T and only starts hurting at 32T. The fix here is
+ less about thread count, more about (1)–(3) above.
+
+## Concrete next-step experiments
+
+A. Cap `_decodeThreadCount` to 8 when no topology is provided, rerun
+ the full InferenceBenchmarks + cross-lib harness. Cheap — one-line
+ change, re-test.
+
+B. Add a `Dispatch` fast-path that skips the pool when `workItems <
+ SingleThreadThreshold` (tunable, e.g. 1024 F32 ops). Requires MatMul
+ call sites to know their work size, which they already do (M×K).
+
+C. Cache-pad hot fields in `ComputeThreadPool`. Low risk, easy to
+ verify with this microbench.
+
+Do A first — it's one line and the microbench already shows 4-16T
+EventBased is sane at this dispatch cost, so clamping to 8 should
+recover most of the 76% spin CPU. If A doesn't close the gap to
+llama.cpp's 34 tok/s, try B and C.
+
+## Artefacts
+
+- `bench/results/ThreadPoolDispatchBenchmarks-report-github.md` — full BDN table
+- `bench/results/ThreadPoolDispatchBenchmarks-report.csv`
+- `bench/results/ThreadPoolDispatchBenchmarks-report-full-compressed.json`
diff --git a/.perf-runs/cross-lib-20260424/dotllm-kernel.log.txt b/.perf-runs/cross-lib-20260424/dotllm-kernel.log.txt
new file mode 100644
index 00000000..617fc03f
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/dotllm-kernel.log.txt
@@ -0,0 +1,138 @@
+[EXECUTING: PerfView /RestartingToElevelate:collect /DataFile:.\.perf-runs\cross-lib-20260424\dotllm-kernel.etl /BufferSizeMB:512 /StackCompression /MaxCollectSec:25 /KernelEvents:ThreadTime /NoGui /NoNGenRundown Collect]
+Circular buffer size = 0, setting to 500.
+[Kernel Log: C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.kernel.etl]
+Kernel keywords enabled: ThreadTime
+[User mode Log: C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl]
+Enabling Provider:PerfViewLogger Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:a8a71ac1-040f-54a2-07ca-00a89b5ab761
+Turning on more CLR GC, JScript and ASP.NET Events.
+Enabling Provider:DotNet Level:Verbose Keywords:0xffffffffffffffff Stacks:1 Values: Guid:319dc449-ada5-50f7-428e-957db6791668
+Enabling Provider:Microsoft-Windows-Kernel-EventTracing Level:Verbose Keywords:0x70 Stacks:1 Values: Guid:b675ec37-bdb6-4648-bc92-f3fdc74d3ca2
+Enabling Provider:Microsoft-Windows-Kernel-File Level:Verbose Keywords:0x80 Stacks:1 Values: Guid:edd08927-9cc4-4e65-b970-c2560fb5c289
+Enabling Provider:Microsoft-Windows-Kernel-Process Level:Informational Keywords:0x10 Stacks:1 Values: Guid:22fb2cd6-0e7b-422b-a0c7-2fad1fd0e716
+Enabling Provider:ASP.NET Level:Verbose Keywords:0xfffffffffffffffd Stacks:0 Values: Guid:aff081fe-0247-4275-9c4e-021f3dc1da35
+File C:\WINDOWS\System32\inetsrv\iiscore.dll does not exist, ASP.NET is not enabled on the machine
+Enabling Provider:*Microsoft-Windows-ASPNET Level:Informational Keywords:0xffffffffffffffff Stacks:1 Values: Guid:ee799f41-cfa5-550b-bf2c-344747c1c668
+Enabling Provider:Microsoft-Windows-IIS Level:Critical Keywords:0x0 Stacks:0 Values: Guid:de4649c9-15e8-4fea-9d85-1cdda520c334
+Enabling Provider:Microsoft-PerfTrack-IEFRAME Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:b2a40f1f-a05a-4dfd-886a-4c4f18c4334c
+Enabling Provider:Microsoft-PerfTrack-MSHTML Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:ffdb9886-80f3-4540-aa8b-b85192217ddf
+Enabling Provider:Microsoft-Windows-WinINet Level:Verbose Keywords:0x2 Stacks:0 Values: Guid:43d1a55c-76d6-4f7e-995c-64c711e5cafe
+Enabling Provider:Microsoft-Windows-Application Server-Applications Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:c651f5f6-1c0d-492e-8ae1-b4efd7c9d503
+Enabling Provider:Microsoft-IE Level:Informational Keywords:0x1300 Stacks:0 Values: Guid:9e3b3947-ca5d-4614-91a2-7b624e0e7244
+Enabling Provider:Microsoft-Windows-DNS-Client Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:1c95126e-7eea-49a9-a3fe-a378b03ddb4d
+Enabling Provider:Microsoft-Windows-DirectComposition Level:Verbose Keywords:0x4 Stacks:0 Values: Guid:c44219d0-f344-11df-a5e2-b307dfd72085
+Enabling Provider:Microsoft-Windows-Immersive-Shell Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:315a8872-923e-4ea2-9889-33cd4754bf64
+Enabling Provider:Microsoft-Windows-XAML Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:531a35ab-63ce-4bcf-aa98-f88c7a89e455
+Enabling Provider:Microsoft-JScript Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:57277741-3638-4a4b-bdba-0ac6e45da56c
+Enabling Provider:ClrPrivateNative Level:Informational Keywords:0x4002000b Stacks:0 Values: Guid:1095638c-8748-4c7a-b39e-baea27b9c589
+Enabling Provider:CLRPrivate Level:Informational Keywords:0x4002000b Stacks:0 Values: Guid:763fd754-7086-4dfe-95eb-c01a46faf4ca
+Enabling Provider:.NETFramework Level:Verbose Keywords:0x16 Stacks:1 Values: Guid:8e9f5090-2d75-4d03-8a81-e5afbf85daf1
+Enabling Provider:Microsoft.Tasks.Nuget Level:Informational Keywords:0x80 Stacks:0 Values: Guid:b734069e-103c-5376-2080-4f009207881a
+Enabling Provider:Microsoft-AdoNet-SystemData Level:Informational Keywords:0x1 Stacks:1 Values: Guid:6a4dfe53-eb50-5332-8473-7b7e10a94fd1
+Enabling Provider:ETWCLrProfiler Diagnostics Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:6652970f-1756-5d8d-0805-e9aad152aa79
+Enabling Provider:Microsoft-Diagnostics-DiagnosticSource Level:Informational Keywords:0xfffffffffffff7ff Stacks:1 Values:FilterAndPayloadSpecs=HttpHandlerDiagnosticListener/System.Net.Http.Request@Activity2Start:Request.RequestUri
+HttpHandlerDiagnosticListener/System.Net.Http.Response@Activity2Stop:Response.StatusCode Guid:adb401e1-5296-51f8-c125-5fda75826144
+Enabling Provider:Microsoft-AspNetCore-Hosting Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:9e620d2a-55d4-5ade-deb7-c26046d245a8
+Enabling Provider:Microsoft-ApplicationInsights-Core Level:Verbose Keywords:0xffffffffffffffff Stacks:1 Values: Guid:74af9f20-af6a-5582-9382-f21f674fb271
+Enabling Provider:Microsoft-Windows-Kernel-Power Level:Informational Keywords:0xffb Stacks:0 Values: Guid:331c3b3a-2005-44c2-ac5e-77220c37d6b4
+Enabling Provider:Microsoft-Windows-Kernel-Processor-Power Level:Informational Keywords:0xe5d Stacks:0 Values: Guid:0f67e49f-fe51-4e9f-b490-6f2948cc6027
+Enabling Provider:Microsoft-Windows-PowerCpl Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:b1f90b27-4551-49d6-b2bd-dfc6453762a6
+Enabling Provider:Microsoft-Windows-PowerCfg Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:9f0c4ea8-ec01-4200-a00d-b9701cbea5d8
+Enabling Provider:Microsoft-Antimalware-Engine Level:Verbose Keywords:0xffffffffffffffff Stacks:1 Values: Guid:0a002690-3839-4e3a-b3b6-96d8df868d99
+Enabling Provider:Microsoft-Antimalware-AMFilter Level:Verbose Keywords:0xffffffffffffffff Stacks:1 Values: Guid:cfeb0608-330e-4410-b00d-56d8da9986e6
+Enabling Provider:Microsoft-Antimalware-Service Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:751ef305-6c6e-4fed-b847-02ef79d26aef
+Enabling Provider:Microsoft-Antimalware-RTP Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:8e92deef-5e17-413b-b927-59b2f06a3cfc
+Enabling Provider:Microsoft-Antimalware-Protection Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:e4b70372-261f-4c54-8fa6-a5a7914d73da
+Enabling Provider:Microsoft-Windows-HttpService Level:Verbose Keywords:0xffffffffffffffff Stacks:1 Values: Guid:dd5ef90a-6398-47a4-ad34-4dcecdef795f
+Enabling Provider:Microsoft-Windows-TCPIP Level:Informational Keywords:0xffffffffffffffff Stacks:1 Values: Guid:2f07e2ee-15db-40f1-90ef-9d7ba282188a
+Enabling Provider:Microsoft-Windows-NDIS-PacketCapture Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:2ed6006e-4729-4609-b423-3ee7bcd678ef
+Enabling Provider:Microsoft-Windows-WebIO Level:Informational Keywords:0xffffffffffffffff Stacks:0 Values: Guid:50b3e73c-9370-461d-bb9f-26f32d68887d
+Enabling Provider:Microsoft-Windows-WinINet Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:43d1a55c-76d6-4f7e-995c-64c711e5cafe
+Enabling Provider:Microsoft-Windows-WinHttp Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:7d44233d-3055-4b9c-ba64-0d47ca40a232
+Enabling Provider:Microsoft-Windows-RPC Level:Informational Keywords:0x0 Stacks:0 Values: Guid:6ad52b32-d609-4be9-ae07-ce8dae937e39
+Turning on VS CodeMarkers and MeasurementBlock Providers.
+Enabling Provider:MeasurementBlock Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:143a31db-0372-40b6-b8f1-b4b16adb5f54
+Enabling Provider:CodeMarkers Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:641d7f6c-481c-42e8-ab7e-d18dc5e5cb9e
+Enabling CLR Events: GC, Binder, Security, AppDomainResourceManagement, Contention, Exception, Threading, JITSymbols, Type, GCHeapSurvivalAndMovement, GCHeapAndTypeNames, Stack, ThreadTransfer, Codesymbols, Compilation
+Enabling Provider:ClrNative Level:Verbose Keywords:0x14c14fcc9d Stacks:1 Values: Guid:47c3ba0c-77f1-4eb0-8d4d-aef447f16a85
+Enabling Provider:CLR Level:Verbose Keywords:0x14c14fcc9d Stacks:0 Values: Guid:e13c0d23-ccbc-4e12-931b-d9cc2eee27e4
+Enabling Providers specified by the user.
+[Starting collection at 24/04/2026 16:45:32]
+[Collecting 10 sec: Size=588.9 MB.]
+[Collecting 20 sec: Size=723.1 MB. Recycling started at 20 sec.]
+[Exceeded MaxCollectSec 25]
+Stopping tracing for sessions 'NT Kernel Logger' and 'PerfViewSession'.
+No Heap events were active for this trace.
+[Sending rundown command to CLR providers...]
+Enabling Provider:PerfViewLogger Level:Verbose Keywords:0xffffffffffffffff Stacks:0 Values: Guid:a8a71ac1-040f-54a2-07ca-00a89b5ab761
+Enabling Provider:CLRRundown Level:Verbose Keywords:0x108 Stacks:0 Values: Guid:a669021c-c450-4609-a035-5af59af4df18
+Enabling Provider:CLRRundown Level:Verbose Keywords:0x1080060111 Stacks:0 Values: Guid:a669021c-c450-4609-a035-5af59af4df18
+Waiting up to 120 sec for rundown events. Use /RundownTimeout to change.
+If you know your process has exited, use /noRundown qualifer to skip this step.
+Rundown File Length: 32.8MB delta: 32.8MB
+Rundown File Length: 32.8MB delta: 0.0MB
+Rundown file has stopped growing, assuming rundown complete.
+CLR Rundown took 2.714 sec.
+Done stopping sessions.
+Ensuring .NET Allocation profiler not installed.
+Detected 64 bit system, removing 32 bit keys.
+[Merging data files to dotllm-kernel.etl. Can take 10s of seconds... (can skip if data analyzed on same machine with PerfView)]
+WARNING NO _NT_SYMBOL_PATH set ...
+Symbol reader _NT_SYMBOL_PATH= {
+ C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424;
+ }
+This can be set using the File -> Set Symbol Path dialog on the Stack Viewer.
+Created SymbolReader with SymbolPath C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424
+Setting SymbolReaderOptions forces clearing Pdb lookup cache
+Starting Merging of C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl
+Starting Generating NGEN pdbs for C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl
+Found NGEN pdb C:\Users\james\AppData\Local\Temp\SymbolCache\System.ni.pdb\d4bdee93a2422be7ed56b39c2554e1ae1\System.ni.pdb
+NGEN PDB creation for C:\Windows\assembly\NativeImages_v4.0.30319_64\System\d4bdee93a2422be7ed56b39c2554e1ae\System.ni.dll took 0.01 Sec
+Found NGEN pdb C:\Users\james\AppData\Local\Temp\SymbolCache\mscorlib.ni.pdb\76cc8cd31b99e4657b7fa51813603db81\mscorlib.ni.pdb
+NGEN PDB creation for C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\76cc8cd31b99e4657b7fa51813603db8\mscorlib.ni.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\PowerShell\7\System.Management.Automation.dll, Giving up
+NGEN PDB creation for C:\Program Files\PowerShell\7\System.Management.Automation.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\PowerShell\7\System.Private.CoreLib.dll, Giving up
+NGEN PDB creation for C:\Program Files\PowerShell\7\System.Private.CoreLib.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Private.CoreLib.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Private.CoreLib.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\sdk\10.0.103\Roslyn\bincore\VBCSCompiler.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\sdk\10.0.103\Roslyn\bincore\VBCSCompiler.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Server.Kestrel.Core.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Server.Kestrel.Core.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Text.Json.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Text.Json.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Net.Sockets.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Net.Sockets.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Http.Extensions.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Http.Extensions.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Diagnostics.DiagnosticSource.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Diagnostics.DiagnosticSource.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.IO.Pipelines.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.IO.Pipelines.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.Extensions.DependencyInjection.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.Extensions.DependencyInjection.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Hosting.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Hosting.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.HostFiltering.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.HostFiltering.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Diagnostics.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Diagnostics.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Routing.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\10.0.3\Microsoft.AspNetCore.Routing.dll took 0.00 Sec
+Could not find CLR directory for NGEN image C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Threading.dll, Giving up
+NGEN PDB creation for C:\Program Files\dotnet\shared\Microsoft.NETCore.App\10.0.3\System.Threading.dll took 0.00 Sec
+Generating NGEN Pdbs took 12.9 sec
+Merging took 17.6 sec
+Moving C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl.new to C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl
+Deleting temp file
+Merge and NGEN PDB Generation took 17.594 sec.
+Merge output file C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl
+[Zipping ETL file C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl]
+[Writing 2 PDBS to Zip file]
+ZIP generation took 15.176 sec
+ZIP output file C:\Development\dotLLM-mamba3\.perf-runs\cross-lib-20260424\dotllm-kernel.etl.zip
+Time: 24/04/2026 16:46:33
+Stop Completed at 24/04/2026 16:46:33
+[DONE 16:46:33 SUCCESS: PerfView /RestartingToElevelate:collect /DataFile:.\.perf-runs\cross-lib-20260424\dotllm-kernel.etl /BufferSizeMB:512 /StackCompression /MaxCollectSec:25 /KernelEvents:ThreadTime /NoGui /NoNGenRundown Collect]
diff --git a/.perf-runs/cross-lib-20260424/dotllm-throughput.json b/.perf-runs/cross-lib-20260424/dotllm-throughput.json
new file mode 100644
index 00000000..80ded074
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/dotllm-throughput.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T14:09:38.865709+00:00",
+ "run_finished_at": "2026-04-24T14:10:03.968871+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424\\dotllm-throughput.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T14:09:39.854929+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 8.66,
+ "total_s": 22.175,
+ "cpu_avg_pct": 76.5,
+ "cpu_peak_pct": 76.8,
+ "ram_peak_gb": 1.27,
+ "gpu_util_avg": 18.7,
+ "gpu_util_peak": 21.8,
+ "gpu_mem_peak_gb": 3.16,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n## 1.1.1. Introduction\n\nDependency injection is a technique that allows you to write a single unit of code that can "
+ }
+ ]
+}
diff --git a/.perf-runs/cross-lib-20260424/kernel-profile.txt b/.perf-runs/cross-lib-20260424/kernel-profile.txt
new file mode 100644
index 00000000..ec60dc2c
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/kernel-profile.txt
@@ -0,0 +1,270 @@
+Opening .perf-runs/cross-lib-20260424/etl-extract/dotllm-kernel.etl...
+ Session: 16:45:31.298 16:46:00.604
+ Processes: 412, Events: 5970336
+ CPU samples for dotnet: 100059 (no-stack: 1665)
+
+=== top 200 by self ===
+ 66.43% 65362 coreclr!0x7FFF17B42B64
+ 3.97% 3908 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.WorkerLoop(class System.Object)
+ 1.75% 1718 kernel32!0x7FFFB0D42B00
+ 1.37% 1350 kernel32!0x7FFFB0D42B07
+ 1.32% 1302 coreclr!0x7FFF17B42B6F
+ 1.24% 1223 coreclr!0x7FFF17B42B6E
+ 0.86% 843 coreclr!0x7FFF17B42B47
+ 0.84% 827 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.VecDotQ8_0Avx512_4Rows(unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,int32,float32*)
+ 0.83% 818 ntoskrnl!0xFFFFF8039D64C4FA
+ 0.61% 598 ntoskrnl!0xFFFFF8039D676CE5
+ 0.50% 496 coreclr!0x7FFF17B42B68
+ 0.37% 368 ntoskrnl!0xFFFFF8039DAA7655
+ 0.34% 333 coreclr!0x7FFF17B42B35
+ 0.33% 323 ntoskrnl!0xFFFFF8039D671D9F
+ 0.32% 310 ntoskrnl!0xFFFFF8039D67897B
+ 0.31% 306 ntoskrnl!0xFFFFF8039D6B61E2
+ 0.29% 290 ntoskrnl!0xFFFFF8039DAA3279
+ 0.29% 281 ntdll!0x7FFFB1F42344
+ 0.29% 281 ntoskrnl!0xFFFFF8039DAA769E
+ 0.28% 275 ntoskrnl!0xFFFFF8039D671D9C
+ 0.27% 266 ntoskrnl!0xFFFFF8039D677B35
+ 0.27% 265 coreclr!0x7FFF17B42B4E
+ 0.25% 248 ntoskrnl!0xFFFFF8039D74514E
+ 0.23% 229 ntoskrnl!0xFFFFF8039D677BA4
+ 0.23% 225 ntoskrnl!0xFFFFF8039D72876F
+ 0.22% 216 coreclr!0x7FFF17B42B25
+ 0.21% 211 kernel32!0x7FFFB0D42B1A
+ 0.21% 207 ntoskrnl!0xFFFFF8039D6B7DE4
+ 0.21% 204 ntoskrnl!0xFFFFF8039D676CFD
+ 0.19% 191 ntoskrnl!0xFFFFF8039D74471F
+ 0.19% 186 ntoskrnl!0xFFFFF8039D676CCA
+ 0.18% 182 ntoskrnl!0xFFFFF8039D677EBF
+ 0.17% 169 ntoskrnl!0xFFFFF8039D6B7D75
+ 0.16% 161 ntoskrnl!0xFFFFF8039D82293D
+ 0.16% 160 ntoskrnl!0xFFFFF8039D7453FE
+ 0.16% 157 ntoskrnl!0xFFFFF8039D676CD6
+ 0.15% 149 coreclr!0x7FFF17B42B20
+ 0.15% 144 ntoskrnl!0xFFFFF8039D6B7D55
+ 0.14% 142 ntoskrnl!0xFFFFF8039D6795E8
+ 0.14% 140 ntoskrnl!0xFFFFF8039D677B93
+ 0.14% 137 ntoskrnl!0xFFFFF8039D72891C
+ 0.14% 135 ntoskrnl!0xFFFFF8039DAA6D0A
+ 0.13% 132 ntoskrnl!0xFFFFF8039D6B7F24
+ 0.13% 132 ntoskrnl!0xFFFFF8039DAA7766
+ 0.13% 129 ntoskrnl!0xFFFFF8039D676E9B
+ 0.13% 126 ntoskrnl!0xFFFFF8039D671DAA
+ 0.12% 122 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0R4Worker(int,int32,int32)
+ 0.12% 122 ntoskrnl!0xFFFFF8039DAB1EB4
+ 0.12% 118 ntoskrnl!0xFFFFF8039D76112E
+ 0.12% 118 ntoskrnl!0xFFFFF8039D6796BD
+ 0.11% 112 ntoskrnl!0xFFFFF8039D676CEE
+ 0.11% 111 ntoskrnl!0xFFFFF8039DAA6E8F
+ 0.11% 104 ntoskrnl!0xFFFFF8039D676CD0
+ 0.10% 103 ntoskrnl!0xFFFFF8039D677CDF
+ 0.10% 99 ntoskrnl!0xFFFFF8039D742460
+ 0.10% 98 ntoskrnl!0xFFFFF8039D6B2497
+ 0.10% 97 ntoskrnl!0xFFFFF8039D8227E9
+ 0.10% 96 ntoskrnl!0xFFFFF8039D676C7A
+ 0.09% 91 ntoskrnl!0xFFFFF8039D677D52
+ 0.09% 89 System.Private.CoreLib!System.Threading.SpinWait.SpinOnce(int32)
+ 0.09% 88 ntoskrnl!0xFFFFF8039D677B45
+ 0.09% 87 ntoskrnl!0xFFFFF8039D74AFDA
+ 0.09% 86 ntoskrnl!0xFFFFF8039D676CE3
+ 0.09% 85 ntoskrnl!0xFFFFF8039D677B5C
+ 0.09% 85 ntoskrnl!0xFFFFF8039D6742DB
+ 0.09% 84 ntoskrnl!0xFFFFF8039D675F32
+ 0.08% 82 ntoskrnl!0xFFFFF8039D677F15
+ 0.08% 82 ntoskrnl!0xFFFFF8039D76104F
+ 0.08% 79 ntoskrnl!0xFFFFF8039D677B54
+ 0.08% 79 ntoskrnl!0xFFFFF8039D676BC5
+ 0.08% 77 ntoskrnl!0xFFFFF8039D6B7D0D
+ 0.07% 72 ntoskrnl!0xFFFFF8039DAA6DC3
+ 0.07% 71 ntoskrnl!0xFFFFF8039D64FE84
+ 0.07% 68 ntoskrnl!0xFFFFF8039D81A08D
+ 0.07% 68 ntoskrnl!0xFFFFF8039D677D34
+ 0.07% 67 ntoskrnl!0xFFFFF8039D728691
+ 0.07% 67 ntoskrnl!0xFFFFF8039D676EA7
+ 0.07% 66 ntoskrnl!0xFFFFF8039D64A355
+ 0.07% 64 ntoskrnl!0xFFFFF8039D677DEA
+ 0.06% 63 ntoskrnl!0xFFFFF8039D742BDF
+ 0.06% 62 ntoskrnl!0xFFFFF8039D674CB5
+ 0.06% 62 ntoskrnl!0xFFFFF8039D676CDE
+ 0.06% 61 ntoskrnl!0xFFFFF8039D66AA70
+ 0.06% 60 ntoskrnl!0xFFFFF8039D677D02
+ 0.06% 58 ntoskrnl!0xFFFFF8039D674CB2
+ 0.06% 58 ntoskrnl!0xFFFFF8039D675F85
+ 0.06% 58 coreclr!0x7FFF17B42B62
+ 0.06% 57 ntoskrnl!0xFFFFF8039D677D09
+ 0.06% 57 System.Private.CoreLib!System.Threading.ManualResetEventSlim.get_IsSet()
+ 0.06% 56 ntoskrnl!0xFFFFF8039D677ED9
+ 0.06% 55 ntoskrnl!0xFFFFF8039D675F5F
+ 0.05% 54 ntoskrnl!0xFFFFF8039DAA6E41
+ 0.05% 54 ntoskrnl!0xFFFFF8039D677D54
+ 0.05% 54 ntoskrnl!0xFFFFF8039DAA764A
+ 0.05% 54 ntoskrnl!0xFFFFF8039D677BF6
+ 0.05% 52 ntoskrnl!0xFFFFF8039D677C00
+ 0.05% 52 ntoskrnl!0xFFFFF8039D7D2788
+ 0.05% 51 ntoskrnl!0xFFFFF8039D677CD7
+ 0.05% 50 ntoskrnl!0xFFFFF8039D760BD7
+ 0.05% 50 ntoskrnl!0xFFFFF8039D760C4D
+ 0.05% 49 ntoskrnl!0xFFFFF8039D677D14
+ 0.05% 48 ntoskrnl!0xFFFFF8039D728685
+ 0.05% 48 ntoskrnl!0xFFFFF8039D6B2476
+ 0.05% 48 ntoskrnl!0xFFFFF8039DAA6CC3
+ 0.05% 47 ntoskrnl!0xFFFFF8039D676C93
+ 0.05% 45 ntoskrnl!0xFFFFF8039DAA6EC6
+ 0.04% 44 ntoskrnl!0xFFFFF8039D677D0C
+ 0.04% 44 ntoskrnl!0xFFFFF8039DAA6C67
+ 0.04% 44 ntoskrnl!0xFFFFF8039D82290F
+ 0.04% 44 ntoskrnl!0xFFFFF8039D6788C8
+ 0.04% 44 ntoskrnl!0xFFFFF8039D66AD0D
+ 0.04% 44 ntoskrnl!0xFFFFF8039D760B48
+ 0.04% 44 ntoskrnl!0xFFFFF8039D676CD3
+ 0.04% 44 ntoskrnl!0xFFFFF8039D760F5C
+ 0.04% 43 coreclr!0x7FFF17B42B30
+ 0.04% 43 ntdll!0x7FFFB1EAEF34
+ 0.04% 43 ntoskrnl!0xFFFFF8039D677BC4
+ 0.04% 41 ntoskrnl!0xFFFFF8039DAB2060
+ 0.04% 40 ntoskrnl!0xFFFFF8039D677E0A
+ 0.04% 39 coreclr!0x7FFF17B42B29
+ 0.04% 39 ntoskrnl!0xFFFFF8039D6747A3
+ 0.04% 38 ntoskrnl!0xFFFFF8039D676035
+ 0.04% 38 ntoskrnl!0xFFFFF8039D64A397
+ 0.04% 38 ntoskrnl!0xFFFFF8039DAA7769
+ 0.04% 38 ntoskrnl!0xFFFFF8039D676D0A
+ 0.04% 38 ntoskrnl!0xFFFFF8039D675298
+ 0.04% 37 ntoskrnl!0xFFFFF8039D6B7CD8
+ 0.04% 37 ntoskrnl!0xFFFFF8039D677D28
+ 0.04% 36 ntoskrnl!0xFFFFF8039D6B7C5D
+ 0.04% 36 ntoskrnl!0xFFFFF8039D760C78
+ 0.04% 36 ntoskrnl!0xFFFFF8039D677CA1
+ 0.04% 36 ntoskrnl!0xFFFFF8039D677DE2
+ 0.04% 35 ntoskrnl!0xFFFFF8039D677BAB
+ 0.04% 35 ntoskrnl!0xFFFFF8039D675EC7
+ 0.04% 35 ntoskrnl!0xFFFFF8039D675EC0
+ 0.03% 33 coreclr!0x7FFF17B64DC3
+ 0.03% 33 ntoskrnl!0xFFFFF8039DAA763A
+ 0.03% 33 ntoskrnl!0xFFFFF8039D6B25CF
+ 0.03% 32 ntoskrnl!0xFFFFF8039D676BE7
+ 0.03% 31 ntoskrnl!0xFFFFF8039D675F0F
+ 0.03% 30 ntoskrnl!0xFFFFF8039D676CDA
+ 0.03% 30 ntoskrnl!0xFFFFF8039D677B3F
+ 0.03% 29 ntoskrnl!0xFFFFF8039D677FB1
+ 0.03% 29 ntoskrnl!0xFFFFF8039D7450EC
+ 0.03% 29 ntoskrnl!0xFFFFF8039D677B60
+ 0.03% 28 ntoskrnl!0xFFFFF8039D676CC7
+ 0.03% 28 ntoskrnl!0xFFFFF8039D677C36
+ 0.03% 28 coreclr!0x7FFF17AD1D62
+ 0.03% 27 ntoskrnl!0xFFFFF8039D6747FE
+ 0.03% 27 ntoskrnl!0xFFFFF8039D676F57
+ 0.03% 27 ntoskrnl!0xFFFFF8039D676CF3
+ 0.03% 27 ntoskrnl!0xFFFFF8039D676E42
+ 0.03% 26 ntoskrnl!0xFFFFF8039D803AA1
+ 0.03% 26 kernelbase!0x7FFFAF2135D0
+ 0.03% 26 ntoskrnl!0xFFFFF8039D676CBA
+ 0.03% 26 ntoskrnl!0xFFFFF8039D676E1A
+ 0.03% 26 ntoskrnl!0xFFFFF8039D742D18
+ 0.03% 26 ntoskrnl!0xFFFFF8039D678854
+ 0.03% 25 ntoskrnl!0xFFFFF8039D6795B4
+ 0.03% 25 ntoskrnl!0xFFFFF8039D676C78
+ 0.03% 25 ntoskrnl!0xFFFFF8039D82286F
+ 0.03% 25 ntoskrnl!0xFFFFF8039D677D9F
+ 0.03% 25 ntoskrnl!0xFFFFF8039DAA5C10
+ 0.03% 25 ntoskrnl!0xFFFFF8039D677CCC
+ 0.03% 25 ntoskrnl!0xFFFFF8039D677AED
+ 0.03% 25 ntoskrnl!0xFFFFF8039D677F70
+ 0.03% 25 ntoskrnl!0xFFFFF8039DAA6E3F
+ 0.02% 24 ntoskrnl!0xFFFFF8039D676BC0
+ 0.02% 24 coreclr!0x7FFF17B4964F
+ 0.02% 24 ntoskrnl!0xFFFFF8039D676C4B
+ 0.02% 24 ntoskrnl!0xFFFFF8039D819BAD
+ 0.02% 24 ntoskrnl!0xFFFFF8039D677B7A
+ 0.02% 23 kernelbase!0x7FFFAF2135B0
+ 0.02% 23 ntoskrnl!0xFFFFF8039D676CE8
+ 0.02% 23 ntoskrnl!0xFFFFF8039D677D3D
+ 0.02% 22 ntoskrnl!0xFFFFF8039D677DD8
+ 0.02% 22 ntoskrnl!0xFFFFF8039D676F86
+ 0.02% 22 coreclr!0x7FFF17B49649
+ 0.02% 22 ntoskrnl!0xFFFFF8039D677B71
+ 0.02% 22 ntoskrnl!0xFFFFF8039D677B83
+ 0.02% 21 ntoskrnl!0xFFFFF8039D742B71
+ 0.02% 21 ntoskrnl!0xFFFFF8039D675309
+ 0.02% 21 ntoskrnl!0xFFFFF8039D675F97
+ 0.02% 21 ntoskrnl!0xFFFFF8039D675EE5
+ 0.02% 20 ntoskrnl!0xFFFFF8039D742679
+ 0.02% 20 ntoskrnl!0xFFFFF8039D760B0A
+ 0.02% 20 ntoskrnl!0xFFFFF8039D677B37
+ 0.02% 20 ntoskrnl!0xFFFFF8039D82290C
+ 0.02% 20 ntoskrnl!0xFFFFF8039D677D62
+ 0.02% 20 ntoskrnl!0xFFFFF8039D677DFE
+ 0.02% 19 ntoskrnl!0xFFFFF8039DAA7684
+ 0.02% 19 ntoskrnl!0xFFFFF8039D677BE1
+ 0.02% 19 ntoskrnl!0xFFFFF8039D677C13
+ 0.02% 19 coreclr!0x7FFF17B42B3B
+ 0.02% 19 ntoskrnl!0xFFFFF8039D675ECE
+ 0.02% 19 ntoskrnl!0xFFFFF8039DC4CD62
+ 0.02% 19 ntoskrnl!0xFFFFF8039D676EFF
+ 0.02% 19 ntoskrnl!0xFFFFF8039D8F78E2
+ 0.02% 18 ntoskrnl!0xFFFFF8039D677CFE
+ 0.02% 18 ntoskrnl!0xFFFFF8039D675FAB
+
+=== top 200 by self (DotLLM) (any of: DotLLM) ===
+ 3.97% 3908 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.WorkerLoop(class System.Object)
+ 0.84% 827 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.VecDotQ8_0Avx512_4Rows(unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,int32,float32*)
+ 0.12% 122 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0R4Worker(int,int32,int32)
+ 0.01% 10 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRows(unsigned int8*,unsigned int8*,float32*,int32,int32)
+ 0.01% 7 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.AttentionWorker(int,int32,int32)
+ 0.00% 4 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.GemmInterleaved(int,value class DotLLM.Core.Configuration.QuantizationType,float32*,float32*,int32,int32,int32,unsigned int8*,value class RepackedWeight&)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsWorker(int,int32,int32)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.Dispatch(int,fnptr void(int,int32,int32))
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.FusedDecode2Worker(int,int32,int32)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0Interleaved(unsigned int8*,unsigned int8*,float32*,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.QuantizeF32ToQ8_0Avx512(float32*,unsigned int8*,int32)
+ 0.00% 1 DotLLM.Engine!DotLLM.Engine.TextGenerator.Generate(class System.String,class DotLLM.Core.Configuration.InferenceOptions,class System.Action`1)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,float32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.00% 1 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.FusedGateUpDecode(value class DotLLM.Models.Architectures.TransformerLayerWeights&,float32*,unsigned int8*,float32*,float32*)
+ 0.00% 1 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.QuantizeInput(float32*,unsigned int8*,int32,int32,value class DotLLM.Core.Configuration.QuantizationType)
+
+=== top 200 by inclusive (DotLLM) (any of: DotLLM) ===
+ 6.73% 97384 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.WorkerLoop(class System.Object)
+ 0.06% 939 DotLLM.Engine!DotLLM.Engine.TextGenerator.Generate(class System.String,class DotLLM.Core.Configuration.InferenceOptions,class System.Action`1)
+ 0.06% 939 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+<>c__DisplayClass3_0+<b__0>d.MoveNext()
+ 0.06% 939 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+<>c__DisplayClass3_0.b__0()
+ 0.06% 939 DotLLM.Server!DotLLM.Server.ServerState+d__73.MoveNext()
+ 0.06% 939 DotLLM.Server!DotLLM.Server.ServerState.ExecuteAsync(class System.Func`1,value class System.Threading.CancellationToken)
+ 0.06% 939 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+d__3.MoveNext()
+ 0.06% 939 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint.HandleNonStreamingAsync(class DotLLM.Server.Models.ChatCompletionRequest,class DotLLM.Engine.TextGenerator,class DotLLM.Server.ServerState,class Microsoft.AspNetCore.Http.HttpContext,class System.String,class DotLLM.Core.Configuration.InferenceOptions,class System.String,class System.String,class DotLLM.Tokenizers.ToolDefinition[],value class System.Threading.CancellationToken)
+ 0.06% 939 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+d__2.MoveNext()
+ 0.06% 939 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint.HandleAsync(class DotLLM.Server.Models.ChatCompletionRequest,class DotLLM.Server.ServerState,class Microsoft.AspNetCore.Http.HttpContext)
+ 0.06% 938 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.Forward(value class System.ReadOnlySpan`1,value class System.ReadOnlySpan`1,int32,class DotLLM.Core.Attention.IKvCache)
+ 0.06% 845 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.VecDotQ8_0Avx512_4Rows(unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,int32,float32*)
+ 0.03% 459 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRows(unsigned int8*,unsigned int8*,float32*,int32,int32)
+ 0.03% 398 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsWorker(int,int32,int32)
+ 0.03% 383 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.GemmInterleaved(int,value class DotLLM.Core.Configuration.QuantizationType,float32*,float32*,int32,int32,int32,unsigned int8*,value class RepackedWeight&)
+ 0.02% 323 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.FusedDecode2Worker(int,int32,int32)
+ 0.01% 197 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.FusedGateUpDecode(value class DotLLM.Models.Architectures.TransformerLayerWeights&,float32*,unsigned int8*,float32*,float32*)
+ 0.01% 194 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0Interleaved(unsigned int8*,unsigned int8*,float32*,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.01% 174 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.Dispatch(int,fnptr void(int,int32,int32))
+ 0.01% 174 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.FusedQkvDecode(value class DotLLM.Models.Architectures.TransformerLayerWeights&,float32*,unsigned int8*,float32*,float32*,float32*)
+ 0.01% 168 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,float32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.01% 168 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.01% 138 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.FusedDecode3Worker(int,int32,int32)
+ 0.01% 122 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0R4Worker(int,int32,int32)
+ 0.00% 26 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.AttentionWorker(int,int32,int32)
+ 0.00% 10 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.GemvQ8_0(unsigned int8*,float32*,float32*,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.00% 5 DotLLM.Engine!DotLLM.Engine.KvCache.PagedKvCache.GetValuesRef(int32)
+ 0.00% 2 DotLLM.Engine!DotLLM.Engine.KvCache.PagedKvCache.GetKeysRef(int32)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.FusedOps.RmsNormQuantizeQ8_0(float32*,value class System.ReadOnlySpan`1,float32,unsigned int8*,int32)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.FusedOps.RmsNormQuantize(float32*,value class System.ReadOnlySpan`1,float32,unsigned int8*,int32,value class DotLLM.Core.Configuration.QuantizationType)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.FusedOps.SwiGLU(value class System.ReadOnlySpan`1,value class System.ReadOnlySpan`1,value class System.Span`1)
+ 0.00% 2 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.QuantizeInput(float32*,unsigned int8*,int32,int32,value class DotLLM.Core.Configuration.QuantizationType)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.QuantizeF32ToQ8_0Avx512(float32*,unsigned int8*,int32)
+
+=== top 200 by self (VecDot/Quantize/Attention/Matmul) (any of: VecDot, Quantize, Attention, MatMul) ===
+ 0.84% 827 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.VecDotQ8_0Avx512_4Rows(unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,int32,float32*)
+ 0.12% 122 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0R4Worker(int,int32,int32)
+ 0.01% 10 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRows(unsigned int8*,unsigned int8*,float32*,int32,int32)
+ 0.01% 7 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.AttentionWorker(int,int32,int32)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsWorker(int,int32,int32)
+ 0.00% 2 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.FusedDecode2Worker(int,int32,int32)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0Interleaved(unsigned int8*,unsigned int8*,float32*,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.QuantizeF32ToQ8_0Avx512(float32*,unsigned int8*,int32)
+ 0.00% 1 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,float32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.00% 1 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.QuantizeInput(float32*,unsigned int8*,int32,int32,value class DotLLM.Core.Configuration.QuantizationType)
diff --git a/.perf-runs/cross-lib-20260424/llamacpp-throughput.json b/.perf-runs/cross-lib-20260424/llamacpp-throughput.json
new file mode 100644
index 00000000..dedb942b
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/llamacpp-throughput.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T14:02:19.573290+00:00",
+ "run_finished_at": "2026-04-24T14:02:31.739610+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424\\llamacpp-throughput.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T14:02:21.979783+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 34.73,
+ "total_s": 5.528,
+ "cpu_avg_pct": 0,
+ "cpu_peak_pct": 0,
+ "ram_peak_gb": 0.25,
+ "gpu_util_avg": 14.6,
+ "gpu_util_peak": 14.6,
+ "gpu_mem_peak_gb": 3.29,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n# Python\n\nimport requests\nimport json\nimport time\n\n# 1.\n# 2.\n# 3.\n# 4.\n# 5.\n# 6.\n# 7.\n# 8.\n# 9.\n# 10.\n# 11.\n# 12.\n# "
+ }
+ ]
+}
diff --git a/.perf-runs/cross-lib-20260424/profile-summary.txt b/.perf-runs/cross-lib-20260424/profile-summary.txt
new file mode 100644
index 00000000..e0e36678
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/profile-summary.txt
@@ -0,0 +1,84 @@
+profiles: 42 frames: 141 total time units: 1247317.9
+
+=== top 50 by self ===
+ 100.00% 1247290.8 UNMANAGED_CODE_TIME
+ 0.00% 26.2 CPU_TIME
+ 0.00% 0.6 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.WorkerLoop(class System.Object)
+ 0.00% 0.1 System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.PortableThreadPool+IOCompletionPoller.Poll()
+ 0.00% 0.0 Microsoft.AspNetCore.Server.Kestrel.Core.il!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.PortableThreadPool+GateThread.GateThreadStart()
+ 0.00% 0.0 DotLLM.Engine!DotLLM.Engine.TextGenerator.Generate(class System.String,class DotLLM.Core.Configuration.InferenceOptions,class System.Action`1)
+ 0.00% 0.0 Anonymously Hosted DynamicMethods Assembly!dynamicClass.lambda_method2(pMT: 00007FFEA82924C8,class System.Object,pMT: 00007FFEA81CF1E0,class System.Object)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.SpinWait.SpinOnceCore(int32)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int32)
+ 0.00% 0.0 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.Forward(value class System.ReadOnlySpan`1,value class System.ReadOnlySpan`1,int32,class DotLLM.Core.Attention.IKvCache)
+ 0.00% 0.0 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.GemmInterleaved(int,value class DotLLM.Core.Configuration.QuantizationType,float32*,float32*,int32,int32,int32,unsigned int8*,value class RepackedWeight&)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32,bool,class System.Object,value class WaitHandleWaitSourceMap)
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.GemmQ8_0(unsigned int8*,float32*,float32*,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool,unsigned int8*)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
+ 0.00% 0.0 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.FusedGateUpDecode(value class DotLLM.Models.Architectures.TransformerLayerWeights&,float32*,unsigned int8*,float32*,float32*)
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.Dispatch(int,fnptr void(int,int32,int32))
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRows(unsigned int8*,unsigned int8*,float32*,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,float32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.ManualResetEventSlim.Set(bool)
+ 0.00% 0.0 Process64 Program (25240) Args: samples/DotLLM.Sample.Server/bin/Release/net10.0/DotLLM.Sample.Server.dll C:/Users/james/.dotllm/test-cache/QuantFactory/SmolLM-135M-GGUF/SmolLM-135M.Q8_0.gguf --port 8000
+ 0.00% 0.0 (Non-Activities)
+ 0.00% 0.0 Threads
+ 0.00% 0.0 Thread (36556)
+ 0.00% 0.0 DotLLM.Sample.Server!Program.$(class System.String[])
+ 0.00% 0.0 Microsoft.Extensions.Hosting.Abstractions.il!Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(class Microsoft.Extensions.Hosting.IHost)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(class System.Threading.Tasks.Task,value class System.Threading.Tasks.ConfigureAwaitOptions)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Tasks.Task.InternalWait(int32,value class System.Threading.CancellationToken)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Tasks.Task.InternalWaitCore(int32,value class System.Threading.CancellationToken)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Tasks.Task.SpinThenBlockingWait(int32,value class System.Threading.CancellationToken)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Monitor.Wait(class System.Object,int32)
+ 0.00% 0.0 Thread (35824)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Thread.StartCallback()
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Thread.g__PollGCWorker|70_0()
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Thread.Sleep(int32)
+ 0.00% 0.0 System.Threading.il!System.Threading.CountdownEvent.Signal()
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Monitor.Enter_Slowpath(class System.Object)
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsWorker(int,int32,int32)
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.VecDotQ8_0Avx512_4Rows(unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,int32,float32*)
+ 0.00% 0.0 System.Private.CoreLib.il!System.Threading.Monitor.Exit_Slowpath(value class LeaveHelperAction,class System.Object)
+ 0.00% 0.0 Thread (25228)
+ 0.00% 0.0 Thread (25612)
+ 0.00% 0.0 Thread (34252)
+ 0.00% 0.0 Thread (13132)
+ 0.00% 0.0 Thread (24888)
+ 0.00% 0.0 Thread (26352)
+ 0.00% 0.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0R4Worker(int,int32,int32)
+ 0.00% 0.0 Thread (36240)
+ 0.00% 0.0 Thread (36768)
+
+=== top 40 by inclusive (dotLLM only) (filter=DotLLM) ===
+ 100.00% 1247317.9 Process64 Program (25240) Args: samples/DotLLM.Sample.Server/bin/Release/net10.0/DotLLM.Sample.Server.dll C:/Users/james/.dotllm/test-cache/QuantFactory/SmolLM-135M-GGUF/SmolLM-135M.Q8_0.gguf --port 8000
+ 74.66% 931208.1 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.WorkerLoop(class System.Object)
+ 2.41% 30038.9 DotLLM.Sample.Server!Program.$(class System.String[])
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint.HandleAsync(class DotLLM.Server.Models.ChatCompletionRequest,class DotLLM.Server.ServerState,class Microsoft.AspNetCore.Http.HttpContext)
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+d__2.MoveNext()
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint.HandleNonStreamingAsync(class DotLLM.Server.Models.ChatCompletionRequest,class DotLLM.Engine.TextGenerator,class DotLLM.Server.ServerState,class Microsoft.AspNetCore.Http.HttpContext,class System.String,class DotLLM.Core.Configuration.InferenceOptions,class System.String,class System.String,class DotLLM.Tokenizers.ToolDefinition[],value class System.Threading.CancellationToken)
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+d__3.MoveNext()
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.ServerState.ExecuteAsync(class System.Func`1,value class System.Threading.CancellationToken)
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.ServerState+d__73.MoveNext()
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+<>c__DisplayClass3_0.b__0()
+ 1.51% 18839.6 DotLLM.Server!DotLLM.Server.Endpoints.ChatCompletionEndpoint+<>c__DisplayClass3_0+<b__0>d.MoveNext()
+ 1.51% 18839.6 DotLLM.Engine!DotLLM.Engine.TextGenerator.Generate(class System.String,class DotLLM.Core.Configuration.InferenceOptions,class System.Action`1)
+ 1.51% 18836.1 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.Forward(value class System.ReadOnlySpan`1,value class System.ReadOnlySpan`1,int32,class DotLLM.Core.Attention.IKvCache)
+ 0.63% 7801.8 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.GemmInterleaved(int,value class DotLLM.Core.Configuration.QuantizationType,float32*,float32*,int32,int32,int32,unsigned int8*,value class RepackedWeight&)
+ 0.34% 4257.7 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.GemmQ8_0(unsigned int8*,float32*,float32*,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool,unsigned int8*)
+ 0.33% 4065.7 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRows(unsigned int8*,unsigned int8*,float32*,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.29% 3660.9 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.FusedGateUpDecode(value class DotLLM.Models.Architectures.TransformerLayerWeights&,float32*,unsigned int8*,float32*,float32*)
+ 0.29% 3657.9 DotLLM.Models!DotLLM.Models.Architectures.TransformerModel.FusedQkvDecode(value class DotLLM.Models.Architectures.TransformerLayerWeights&,float32*,unsigned int8*,float32*,float32*,float32*)
+ 0.29% 3657.9 DotLLM.Cpu!DotLLM.Cpu.Threading.ComputeThreadPool.Dispatch(int,fnptr void(int,int32,int32))
+ 0.26% 3297.4 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.26% 3297.4 DotLLM.Cpu!DotLLM.Cpu.Kernels.Attention.Execute(float32*,float32*,float32*,float32*,int32,int32,int32,int32,int32,int32,float32,class DotLLM.Cpu.Threading.ComputeThreadPool,value class System.Nullable`1)
+ 0.03% 392.9 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.VecDotQ8_0Avx512_4Rows(unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,unsigned int8*,int32,float32*)
+ 0.03% 392.5 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsWorker(int,int32,int32)
+ 0.02% 237.2 DotLLM.Engine!DotLLM.Engine.KvCache.PagedKvCache.GetKeysRef(int32)
+ 0.02% 192.0 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.GemvQ8_0(unsigned int8*,float32*,float32*,int32,int32,class DotLLM.Cpu.Threading.ComputeThreadPool)
+ 0.01% 103.7 DotLLM.Engine!DotLLM.Engine.KvCache.PagedKvCache.GetValuesRef(int32)
+ 0.00% 4.2 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRowsQ8_0R4Worker(int,int32,int32)
+ 0.00% 2.3 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.FusedDecode2Worker(int,int32,int32)
+ 0.00% 2.3 DotLLM.Cpu!DotLLM.Cpu.Kernels.MatMul.ComputeRows(unsigned int8*,unsigned int8*,float32*,int32,int32)
diff --git a/.perf-runs/cross-lib-20260424/throughput-resource-SmolLM-135M.json b/.perf-runs/cross-lib-20260424/throughput-resource-SmolLM-135M.json
new file mode 100644
index 00000000..9089a3fe
--- /dev/null
+++ b/.perf-runs/cross-lib-20260424/throughput-resource-SmolLM-135M.json
@@ -0,0 +1,52 @@
+{
+ "benchmark": "throughput_openai",
+ "run_started_at": "2026-04-24T14:09:38.865709+00:00",
+ "run_finished_at": "2026-04-24T14:10:03.970876+00:00",
+ "output_path": "C:\\Development\\dotLLM-mamba3\\.perf-runs\\cross-lib-20260424\\throughput-resource-SmolLM-135M.json",
+ "base_url": "http://127.0.0.1:8000",
+ "host_details": {
+ "captured_at": "2026-04-24T14:09:39.854929+00:00",
+ "hostname": "Strix",
+ "host_slug": "strix",
+ "os": {
+ "system": "Windows",
+ "release": "11",
+ "version": "10.0.26200",
+ "machine": "AMD64"
+ },
+ "manufacturer": "Bosgame",
+ "model": "BeyondMax Series",
+ "processor": "AMD64 Family 26 Model 112 Stepping 0, AuthenticAMD",
+ "logical_cpu_count": 32,
+ "total_memory_gb": 63.65,
+ "python_version": "3.12.10",
+ "ollama_version": "Warning: could not connect to a running Ollama instance",
+ "gpus": [
+ {
+ "Name": "AMD Radeon(TM) 8060S Graphics",
+ "AdapterRAM": 4293918720,
+ "DriverVersion": "32.0.12064.27",
+ "VideoProcessor": "AMD Radeon Graphics Processor (0x1586)",
+ "Status": "OK"
+ }
+ ]
+ },
+ "models": [
+ "SmolLM-135M"
+ ],
+ "results": [
+ {
+ "model": "SmolLM-135M",
+ "completion_tokens": 192,
+ "toks_per_s": 8.66,
+ "total_s": 22.175,
+ "cpu_avg_pct": 76.5,
+ "cpu_peak_pct": 76.8,
+ "ram_peak_gb": 1.27,
+ "gpu_util_avg": 18.7,
+ "gpu_util_peak": 21.8,
+ "gpu_mem_peak_gb": 3.16,
+ "response_preview": "Write a concise explanation of dependency injection with one short Python example.\n\n## 1.1.1. Introduction\n\nDependency injection is a technique that allows you to write a single unit of code that can "
+ }
+ ]
+}
diff --git a/.perf-runs/prefill-20260424/README.md b/.perf-runs/prefill-20260424/README.md
new file mode 100644
index 00000000..861a244f
--- /dev/null
+++ b/.perf-runs/prefill-20260424/README.md
@@ -0,0 +1,184 @@
+---
+date: 2026-04-24
+model: SmolLM-135M Q8_0 (~145 MB)
+host: Strix Halo (Ryzen AI Max+ 395 Zen5 32T), AMD Radeon 8060S iGPU
+test: pp512 (512-token prefill-only, llama-bench style)
+dotllm-branch: feature/prefill-tiled-gemm (from feature/mamba-3 @ 94b9a24)
+---
+
+## Phase 1 re-measurement summary
+
+The stale "7x prefill gap" predates the decode fix (`173fd29`) and the
+cross-lib harness. Re-running with the same 512-token prefill workload
+shows **dotLLM 334 tok/s vs llama.cpp 2198 tok/s = 6.6x gap**. Decode fix
+did not help prefill (as expected — prefill uses `EventBased` dispatch mode,
+never on the SpinWait collapse curve). Gate 1 **triggered** — proceed to
+Phase 2 profiling.
+
+## llama.cpp pp512 on CPU (Vulkan DLL hidden, true CPU-only)
+
+`C:/Development/llama.cpp/llama-bench.exe -m SmolLM-135M.Q8_0.gguf -p 512 -n 0 -t `
+
+| Threads | pp512 tok/s | Notes |
+|--------:|------------:|-------|
+| 8 | 1557 | |
+| 16 | 2198 | Best |
+| 32 | 2150 | Saturated / slight regression |
+
+llama-bench `-ngl 0 -dev none` still loads `ggml-vulkan.dll` and labels the
+row Vulkan even though weights are on CPU. To measure true CPU-only we
+temporarily renamed `ggml-vulkan.dll`; the numbers above are the result.
+
+## dotLLM prefill via `PrefillBenchmarks` (added this session)
+
+`benchmarks/DotLLM.Benchmarks --filter *PrefillBenchmarks*`
+
+The benchmark calls `model.Forward(tokenIds[512], positions[0..512], kvCache)`
+with a fresh `SimpleKvCache`, isolating prefill from tokenizer, sampling,
+and HTTP overhead. Uses `ThreadingConfig.Auto` = 32 threads on this box.
+
+| Mean wall | Tokens | Prefill tok/s |
+|----------:|-------:|--------------:|
+| 1.534 s | 512 | **334** |
+
+BDN notes: warmupCount=2, iterationCount=5, 1 mild outlier. StdDev 163 ms
+(10%). Mean converted to tok/s: 512 / 1.534 = 334.
+
+## Gap
+
+**dotLLM 334 vs llama.cpp 2198 = 6.6×.** Meaningful — proceed to Phase 2.
+
+## Phase 3 fix applied: 2D work partition for `GemmTiledQ8Worker`
+
+Change in `src/DotLLM.Cpu/Kernels/MatMul.cs`:
+
+* When `totalTiles >= threadCount`: pure row-tile partition (old path preserved
+ — threads are saturated on M alone, as in `lm_head`-shaped matmuls).
+* When `totalTiles < threadCount` (QKV/O/Gate/Up/Down on SmolLM-sized models):
+ split M into `totalTiles` uniform row-groups (aligned to 4 for 4-row VecDot)
+ and distribute the remaining threads across N tokens within each row-group.
+ Weight tile per thread stays L2-resident; all threads now have work.
+* `GemmQ8_0(pool)` no longer short-circuits `totalTiles < 2` to a
+ single-threaded path when N is big — a single-tile matmul with 512 tokens
+ now fans out across 32 threads via the 2D partition.
+
+## Before vs after — full `PrefillBenchmarks` pp512 (SmolLM-135M Q8_0, 32T)
+
+| Variant | Mean wall | Prefill tok/s | vs llama.cpp |
+|-------------------|----------:|--------------:|-------------:|
+| Before fix | 1534 ms | **334** | 15% |
+| After 2D partition| 570 ms | **898** | **41%** |
+| llama.cpp (16T) | 233 ms | 2198 | 100% |
+
+**2.7× speedup end-to-end.** Closed ~60% of the gap to llama.cpp.
+
+### Per-projection GFLOPS, before vs after (warmed-up steady state)
+
+| Kernel (m×k×n) | Before | After | Ratio |
+|------------------------|-------:|--------:|------:|
+| Q 576×576×512 | 32 | 178 | 5.6× |
+| K 192×576×512 | 24 | 113 | 4.7× |
+| V 192×576×512 | 24 | 111 | 4.6× |
+| O 576×576×512 | 48 | 188 | 3.9× |
+| Gate 1536×576×512 | 117 | 235 | 2.0× |
+| Up 1536×576×512 | 120 | 274 | 2.3× |
+| Down 576×1536×512 | 76 | 246 | 3.2× |
+| lm_head 49152×576×512 | 211 | 298 | 1.4× |
+
+Wider-M kernels (Gate, Up, lm_head) improve less because they already had
+6+ tiles on 32 threads under the old scheme. The biggest wins are on
+projections that had 1-3 tiles — exactly what the diagnosis predicted.
+
+## Phase 2 — kernel-level profile (`.perf-runs/prefill-20260424/profile/`)
+
+The `PrefillProfile` tool calls `MatMul.GemmQ8_0` at the shapes SmolLM-135M uses for
+N=512 prefill (Q, K, V, O, Gate, Up, Down, lm_head) with a warmed-up
+`ComputeThreadPool` at 32 threads, and reports ms/call and GFLOPS.
+
+```
+ name m k n ms/call GFLOPS x30 layers
+ ─────────────────────────────────────────────────────────────────────
+ Q 576 576 512 10.64 31.9 319.3 ms
+ K 192 576 512 4.70 24.1 140.9 ms
+ V 192 576 512 4.79 23.7 143.6 ms
+ O 576 576 512 7.16 47.5 214.7 ms
+ Gate 1536 576 512 7.72 117.3 231.6 ms
+ Up 1536 576 512 7.55 119.9 226.6 ms
+ Down 576 1536 512 11.91 76.1 357.3 ms
+
+ Sum of MatMul time per layer: 54.46 ms (× 30 = 1634 ms)
+ lm_head 49152 576 512 137.38 211.0 137.4 ms
+
+ Predicted pp512 from MatMul alone: 1634 + 137 = 1771 ms → ~290 tok/s
+ Measured pp512 end-to-end: 1534 ms → 334 tok/s
+```
+
+Predicted ≈ measured within noise. **`MatMul.GemmQ8_0` is the leaf — Gate 2 passes.**
+
+### Why the hypothesis in `.continue-here.md` §2 was wrong
+
+The section predicted tiled GEMM with 8×8 or 16×16 cache blocking would close
+the gap. Inspection of `MatMul.cs:ComputeGemmTiled` and `GemmTiledQ8Worker`
+plus the GFLOPS table above tells a different story:
+
+- `ComputeTileM` already picks tileM = min(256, L2_budget / rowBytes). For
+ SmolLM k=576, rowBytes = 612, L2 = 256KB → `tileM = 256`, aligned to 4.
+- For the narrow projections (M in {192, 576}), `totalTiles = ceil(M/tileM)`
+ is **1-3**. With `threadCount = 32` and
+ `tilesPerThread = ceil(totalTiles / threadCount) = 1`, only 1-3 threads
+ get any work — 29 of 32 sit idle on the barrier.
+- GFLOPS matches this exactly: 192-row projections hit 24 GFLOPS,
+ 576-row hit 32-48, 1536-row hit 117 (still under-utilised — 6 tiles on 32
+ threads), 49152-row `lm_head` hits 211 (saturating, closest to peak).
+
+The weight tile at 150KB fits in Zen 5's 48KB L1D only partially but sits
+fully in the 1MB per-core L2 — so each worker is fine on cache, it just has
+nothing to do. Cache blocking buys nothing when the problem is empty threads.
+
+### Real diagnosis
+
+**Work partition is coarse-grained on M only, never on N.** `GemmTiledQ8Worker`
+splits the M rows into tiles and assigns whole tiles to threads. For the
+QKVO projections on SmolLM (M∈{192, 576}), this leaves the majority of
+workers idle for the entire 30 × prefill calls. The MLP projections fare
+better (1536 rows, 6 tiles) but still under-utilise.
+
+The Nov 2026 `ThreadPoolDispatchBenchmarks` run in `.perf-runs/cross-lib-20260424/`
+confirmed EventBased dispatch at 32 threads costs ~24 µs per Dispatch — but
+our prefill kernels take 5-12 ms apiece, so dispatch overhead is only ~0.5%
+per call. The waste is idle workers, not dispatch cost.
+
+### Fix direction (Phase 3)
+
+Two candidate changes, smallest first:
+
+1. **Split work across both M and N** (row-and-token partition). Each thread
+ gets a rectangular subset of the output C[t, m] = sum_k B[t,k] × A[m,k]^T.
+ For Q/K/V where M is small, threads get token slices rather than row
+ slices, restoring parallelism. No kernel rewrite needed.
+
+2. **If (1) is not enough**, interleaved R4 weight layout currently only
+ runs on the N=1 decode path. Enabling it for prefill is a bigger change.
+
+Expected win from (1): ~3-4× on Q, ~6-8× on K/V, ~2× on O, ~2× on Gate/Up,
+negligible on lm_head. Weighted by their x30 contribution, total prefill
+should drop from ~1634 ms to ~500-700 ms → ~700-1000 tok/s. Still short of
+llama.cpp's 2200, but ~2-3× faster than today.
+
+## Repro
+
+```bash
+# llama.cpp (rename away Vulkan for CPU-only number)
+mv C:/Development/llama.cpp/ggml-vulkan.dll C:/Development/llama.cpp/ggml-vulkan.dll.bak
+C:/Development/llama.cpp/llama-bench.exe \
+ -m C:/Users/james/.dotllm/test-cache/QuantFactory/SmolLM-135M-GGUF/SmolLM-135M.Q8_0.gguf \
+ -p 512 -n 0 -t 16
+mv C:/Development/llama.cpp/ggml-vulkan.dll.bak C:/Development/llama.cpp/ggml-vulkan.dll
+
+# dotLLM end-to-end
+cd benchmarks/DotLLM.Benchmarks
+dotnet run -c Release -- --filter '*PrefillBenchmarks*'
+
+# dotLLM per-projection breakdown
+dotnet run --project .perf-runs/prefill-20260424/profile/PrefillProfile.csproj -c Release 512 32
+```
diff --git a/.perf-runs/prefill-20260424/profile/PrefillProfile.csproj b/.perf-runs/prefill-20260424/profile/PrefillProfile.csproj
new file mode 100644
index 00000000..9428c704
--- /dev/null
+++ b/.perf-runs/prefill-20260424/profile/PrefillProfile.csproj
@@ -0,0 +1,14 @@
+
+
+ Exe
+ net10.0
+ true
+ enable
+ preview
+ false
+
+
+
+
+
+
diff --git a/.perf-runs/prefill-20260424/profile/Program.cs b/.perf-runs/prefill-20260424/profile/Program.cs
new file mode 100644
index 00000000..1ee51ce4
--- /dev/null
+++ b/.perf-runs/prefill-20260424/profile/Program.cs
@@ -0,0 +1,153 @@
+using System.Diagnostics;
+using System.Runtime.InteropServices;
+using DotLLM.Cpu.Kernels;
+using DotLLM.Cpu.Threading;
+using DotLLM.Core.Configuration;
+
+const int Q8_0BlockBytes = 34;
+const int Q8_0GroupSize = 32;
+
+// SmolLM-135M shapes: hidden=576, ffn=1536, heads=9, kv=3, headDim=64, vocab=49152, layers=30
+const int Hidden = 576;
+const int QDim = 576; // 9 * 64
+const int KvDim = 192; // 3 * 64
+const int FfnDim = 1536;
+const int Vocab = 49152;
+const int Layers = 30;
+
+int N = args.Length > 0 && int.TryParse(args[0], out var n) ? n : 512;
+int Threads = args.Length > 1 && int.TryParse(args[1], out var t) ? t : Environment.ProcessorCount;
+
+var pool = new ComputeThreadPool(Threads, topology: null, ThreadingConfig.Auto);
+pool.SetDispatchMode(DispatchMode.EventBased);
+
+// Warm up the pool — spin up workers and let the OS schedule them before any measurement.
+unsafe
+{
+ nint wwarm = (nint)NativeMemory.AlignedAlloc(1024 * 1024, 64);
+ var inw = new float[Hidden * N];
+ var outw = new float[Hidden * N];
+ fixed (float* bp = inw)
+ fixed (float* cp = outw)
+ for (int i = 0; i < 5; i++)
+ MatMul.GemmQ8_0((byte*)wwarm, bp, cp, Hidden, Hidden, N, pool);
+ NativeMemory.AlignedFree((void*)wwarm);
+}
+
+Console.WriteLine($"Prefill kernel breakdown: SmolLM-135M Q8_0, N={N}, Threads={Threads}");
+Console.WriteLine(new string('-', 80));
+Console.WriteLine($" {"name",-12} {"m",6} {"k",6} {"n",4} {"ms/call",8} {"GFLOPS",7} {"x30",10}");
+Console.WriteLine(new string('-', 80));
+
+double totalMsPerLayer = 0;
+void Bench(string name, int m, int k, int nTok, int reps, bool inPerLayer)
+{
+ unsafe
+ {
+ nint w = AllocQ8(m, k);
+ var input = AllocF32(nTok * k);
+ var output = new float[(long)nTok * m];
+
+ fixed (float* bp = input)
+ fixed (float* cp = output)
+ {
+ // Warm up (3 calls) to let JIT tier-up complete and cache settle.
+ for (int i = 0; i < 3; i++)
+ MatMul.GemmQ8_0((byte*)w, bp, cp, m, k, nTok, pool);
+ var sw = Stopwatch.StartNew();
+ for (int r = 0; r < reps; r++)
+ MatMul.GemmQ8_0((byte*)w, bp, cp, m, k, nTok, pool);
+ sw.Stop();
+ double msPerCall = sw.Elapsed.TotalMilliseconds / reps;
+ double gflops = (2.0 * m * k * nTok) / (msPerCall / 1000.0) / 1e9;
+ double xLayers = inPerLayer ? msPerCall * Layers : msPerCall;
+ Console.WriteLine($" {name,-12} {m,6} {k,6} {nTok,4} {msPerCall,8:F2} {gflops,7:F1} {xLayers,10:F1} ms");
+ if (inPerLayer) totalMsPerLayer += msPerCall;
+ }
+ NativeMemory.AlignedFree((void*)w);
+ }
+}
+
+// Warmup the parallel path on a realistic shape — the *first* kernel measured
+// after pool setup consistently runs ~5× slower than subsequent calls on this
+// host, so we run a throwaway first before timing anything.
+Bench("warmup", FfnDim, Hidden, N, 10, false);
+Bench("Q", QDim, Hidden, N, 30, true);
+Bench("K", KvDim, Hidden, N, 30, true);
+Bench("V", KvDim, Hidden, N, 30, true);
+Bench("O", Hidden, QDim, N, 30, true);
+Bench("Gate", FfnDim, Hidden, N, 30, true);
+Bench("Up", FfnDim, Hidden, N, 30, true);
+Bench("Down", Hidden, FfnDim, N, 30, true);
+
+Console.WriteLine();
+double totalGemmTime = totalMsPerLayer * Layers;
+Console.WriteLine($"Sum of MatMul time per layer: {totalMsPerLayer,7:F2} ms (x {Layers} = {totalGemmTime,7:F1} ms for all layers)");
+
+Console.WriteLine();
+Bench("lm_head", Vocab, Hidden, N, 2, false);
+
+// Single-threaded comparisons for the same shapes — pool=null takes the single-thread path.
+Console.WriteLine();
+Console.WriteLine("Single-threaded (pool=null) same shapes — compare GFLOPS to parallel above:");
+void BenchST(string name, int m, int k, int nTok, int reps)
+{
+ unsafe
+ {
+ nint w = AllocQ8(m, k);
+ var input = AllocF32(nTok * k);
+ var output = new float[(long)nTok * m];
+ fixed (float* bp = input)
+ fixed (float* cp = output)
+ {
+ for (int i = 0; i < 3; i++)
+ MatMul.GemmQ8_0((byte*)w, bp, cp, m, k, nTok, null);
+ var sw = Stopwatch.StartNew();
+ for (int r = 0; r < reps; r++)
+ MatMul.GemmQ8_0((byte*)w, bp, cp, m, k, nTok, null);
+ sw.Stop();
+ double msPerCall = sw.Elapsed.TotalMilliseconds / reps;
+ double gflops = (2.0 * m * k * nTok) / (msPerCall / 1000.0) / 1e9;
+ Console.WriteLine($" {name,-12} {m,6} {k,6} {nTok,4} {msPerCall,8:F2} {gflops,7:F1}");
+ }
+ NativeMemory.AlignedFree((void*)w);
+ }
+}
+BenchST("Q-1T", QDim, Hidden, N, 3);
+BenchST("K-1T", KvDim, Hidden, N, 3);
+BenchST("Gate-1T", FfnDim, Hidden, N, 3);
+BenchST("Down-1T", Hidden, FfnDim, N, 3);
+
+Console.WriteLine();
+Console.WriteLine($"Expected pp{N} wall from MatMul alone (ignoring attention, RMSNorm, RoPE, SiLU):");
+Console.WriteLine($" {totalGemmTime:F0} ms → {(N / (totalGemmTime / 1000.0)):F1} tok/s");
+
+pool.Dispose();
+
+static unsafe nint AllocQ8(int m, int k)
+{
+ int blockCount = k / Q8_0GroupSize;
+ int rowBytes = blockCount * Q8_0BlockBytes;
+ nint ptr = (nint)NativeMemory.AlignedAlloc((nuint)((long)m * rowBytes), 64);
+ var rng = new Random(42);
+ byte* p = (byte*)ptr;
+ for (long r = 0; r < m; r++)
+ {
+ for (int b = 0; b < blockCount; b++)
+ {
+ *(Half*)p = (Half)(rng.NextSingle() * 0.1f);
+ for (int i = 0; i < Q8_0GroupSize; i++)
+ ((sbyte*)(p + 2))[i] = (sbyte)rng.Next(-127, 128);
+ p += Q8_0BlockBytes;
+ }
+ }
+ return ptr;
+}
+
+static float[] AllocF32(int elems)
+{
+ var a = new float[elems];
+ var rng = new Random(7);
+ for (int i = 0; i < elems; i++) a[i] = rng.NextSingle() * 2 - 1;
+ return a;
+}
diff --git a/.perf-runs/tools/EtlStackDump/EtlStackDump/EtlStackDump.csproj b/.perf-runs/tools/EtlStackDump/EtlStackDump/EtlStackDump.csproj
new file mode 100644
index 00000000..bc52313a
--- /dev/null
+++ b/.perf-runs/tools/EtlStackDump/EtlStackDump/EtlStackDump.csproj
@@ -0,0 +1,17 @@
+
+
+
+ Exe
+ net10.0
+ enable
+ enable
+
+ false
+
+
+
+
+
+
+
diff --git a/.perf-runs/tools/EtlStackDump/EtlStackDump/Program.cs b/.perf-runs/tools/EtlStackDump/EtlStackDump/Program.cs
new file mode 100644
index 00000000..e100118c
--- /dev/null
+++ b/.perf-runs/tools/EtlStackDump/EtlStackDump/Program.cs
@@ -0,0 +1,150 @@
+// Reads a PerfView .etl.zip (kernel CPU sample trace) and emits an aggregated
+// self-time + inclusive-time breakdown per method, restricted to a target
+// process name (default: dotnet).
+//
+// Usage:
+// EtlStackDump [process-name] [top-N]
+
+using Microsoft.Diagnostics.Tracing.Etlx;
+using Microsoft.Diagnostics.Tracing.Parsers.Kernel;
+
+if (args.Length < 1)
+{
+ Console.Error.WriteLine("Usage: EtlStackDump [process-name] [top-N]");
+ return 1;
+}
+
+string etlPath = args[0];
+string procName = args.Length > 1 ? args[1] : "dotnet";
+int topN = args.Length > 2 ? int.Parse(args[2]) : 40;
+
+Console.Error.WriteLine($"Opening {etlPath}...");
+
+using var traceLog = TraceLog.OpenOrConvert(etlPath);
+Console.Error.WriteLine($" Session: {traceLog.SessionStartTime:HH:mm:ss.fff} → {traceLog.SessionEndTime:HH:mm:ss.fff}");
+Console.Error.WriteLine($" Processes: {traceLog.Processes.Count}, Events: {traceLog.EventCount}");
+
+var selfTime = new Dictionary(StringComparer.Ordinal);
+var inclusiveTime = new Dictionary(StringComparer.Ordinal);
+long totalSamples = 0;
+int missingStacks = 0;
+
+foreach (var ev in traceLog.Events)
+{
+ if (ev is not SampledProfileTraceData sample) continue;
+ var proc = sample.Process();
+ if (proc == null || !string.Equals(proc.Name, procName, StringComparison.OrdinalIgnoreCase))
+ continue;
+
+ totalSamples++;
+
+ var stack = sample.CallStack();
+ if (stack == null)
+ {
+ missingStacks++;
+ continue;
+ }
+
+ string? leaf = null;
+ var seen = new HashSet(StringComparer.Ordinal);
+ var current = stack;
+ while (current != null)
+ {
+ string name = FrameName(current);
+ if (leaf == null) leaf = name;
+ if (seen.Add(name))
+ inclusiveTime[name] = inclusiveTime.GetValueOrDefault(name) + 1;
+ current = current.Caller;
+ }
+ if (leaf != null)
+ selfTime[leaf] = selfTime.GetValueOrDefault(leaf) + 1;
+}
+
+Console.Error.WriteLine($" CPU samples for {procName}: {totalSamples} (no-stack: {missingStacks})");
+
+PrintTop("self", selfTime, topN);
+PrintTop("self (DotLLM)", selfTime, topN, "DotLLM");
+PrintTop("inclusive (DotLLM)", inclusiveTime, topN, "DotLLM");
+PrintTop("self (VecDot/Quantize/Attention/Matmul)", selfTime, topN,
+ "VecDot", "Quantize", "Attention", "MatMul");
+
+// Second pass: for each unresolved coreclr! leaf with > 100 samples,
+// aggregate its nearest managed caller so we can tell what it is
+// (spin wait, JIT, wake, etc.).
+Console.WriteLine();
+Console.WriteLine("=== nearest managed caller of unresolved coreclr leaves (> 100 self samples) ===");
+var hotUnresolved = selfTime
+ .Where(kv => kv.Key.StartsWith("coreclr!0x", StringComparison.OrdinalIgnoreCase) && kv.Value > 100)
+ .Select(kv => kv.Key)
+ .ToHashSet();
+
+var callerAgg = new Dictionary>();
+foreach (var ev in traceLog.Events)
+{
+ if (ev is not SampledProfileTraceData sample) continue;
+ var proc = sample.Process();
+ if (proc == null || !string.Equals(proc.Name, procName, StringComparison.OrdinalIgnoreCase))
+ continue;
+ var s = sample.CallStack();
+ if (s == null) continue;
+ string leaf = FrameName(s);
+ if (!hotUnresolved.Contains(leaf)) continue;
+ // Walk up until a managed frame
+ var cur = s.Caller;
+ while (cur != null)
+ {
+ string n = FrameName(cur);
+ if (!n.StartsWith("coreclr!", StringComparison.OrdinalIgnoreCase)
+ && !n.StartsWith("ntoskrnl!", StringComparison.OrdinalIgnoreCase)
+ && !n.StartsWith("ntdll!", StringComparison.OrdinalIgnoreCase)
+ && !n.StartsWith("kernel", StringComparison.OrdinalIgnoreCase)
+ && !n.StartsWith("?!", StringComparison.Ordinal))
+ {
+ var d = callerAgg.GetValueOrDefault(leaf) ?? new Dictionary();
+ d[n] = d.GetValueOrDefault(n) + 1;
+ callerAgg[leaf] = d;
+ break;
+ }
+ cur = cur.Caller;
+ }
+}
+foreach (var (addr, d) in callerAgg.OrderByDescending(kv => kv.Value.Values.Sum()))
+{
+ long total = d.Values.Sum();
+ Console.WriteLine($"\n {addr} (attributed: {total})");
+ foreach (var (name, w) in d.OrderByDescending(kv => kv.Value).Take(6))
+ Console.WriteLine($" {100.0 * w / total,6:F1}% {w,8} {name}");
+}
+
+return 0;
+
+static string FrameName(TraceCallStack stack)
+{
+ var ca = stack.CodeAddress;
+ var method = ca.Method;
+ if (method != null)
+ {
+ var module = method.MethodModuleFile?.Name ?? "";
+ return $"{module}!{method.FullMethodName}";
+ }
+ var mod = ca.ModuleFile?.Name ?? "?";
+ return $"{mod}!0x{ca.Address:X}";
+}
+
+static void PrintTop(string label, Dictionary d, int n, params string[] filters)
+{
+ Console.WriteLine();
+ string filterLabel = filters.Length > 0 ? $" (any of: {string.Join(", ", filters)})" : "";
+ Console.WriteLine($"=== top {n} by {label}{filterLabel} ===");
+ long total = d.Values.Sum();
+ var items = d
+ .Where(kv => filters.Length == 0 ||
+ filters.Any(f => kv.Key.Contains(f, StringComparison.OrdinalIgnoreCase)))
+ .OrderByDescending(kv => kv.Value)
+ .Take(n);
+ foreach (var (name, w) in items)
+ {
+ double pct = 100.0 * w / Math.Max(total, 1);
+ Console.WriteLine($" {pct,6:F2}% {w,10} {name}");
+ }
+}
diff --git a/.perf-runs/vulkan-coopmat-attention-20260424/README.md b/.perf-runs/vulkan-coopmat-attention-20260424/README.md
new file mode 100644
index 00000000..e2d08ba3
--- /dev/null
+++ b/.perf-runs/vulkan-coopmat-attention-20260424/README.md
@@ -0,0 +1,74 @@
+# Vulkan attention_f32_coopmat — AMD Radeon 8060S iGPU (gfx1151, RDNA3.5)
+
+## Status
+
+Landed — shader compiles, probe works, runtime dispatch picks coopmat when
+available, numerical parity validated to abs 5e-4 / rel 5e-3 against the
+scalar CPU reference on five representative shapes (SmolLM decode, SmolLM
+prefill, Llama hd=128 decode, multi-tile online-softmax, single-head decode).
+
+**Perf: coopmat path is SLOWER than shared-mem / subgroup on this iGPU.**
+
+The target was ≥2× at the prefill shape; reality is a ~1.3× regression.
+Root cause appears to be RDNA3.5 WMMA + wave=64 shared-memory pressure plus
+the Q/K f32→f16 conversion overhead, not outweighed by the 16×16×16 tile
+throughput on the shape sizes tested. See perf table below.
+
+## Measured
+
+Host: Ryzen AI Max+ 395 iGPU (AMD Radeon 8060S, gfx1151). 50 iters per
+shape, warmed up 3×. Timer: `Stopwatch` around `AttentionF32Kernel.Launch`
+(includes `vkQueueSubmit` + `vkQueueWaitIdle`). Env-var toggling selects
+the variant.
+
+| Shape | shared-mem | subgroup | coopmat | coopmat vs shared |
+|------------------------------------------------------|-----------:|---------:|--------:|------------------:|
+| SmolLM decode (sq=1, sk=128, nh=9, nkv=3, hd=64) | 0.166 ms | 0.164 ms | 0.251 ms | 1.51× slower |
+| SmolLM prefill (sq=64, sk=64, nh=9, nkv=3, hd=64) | 0.222 ms | 0.216 ms | 0.282 ms | 1.27× slower |
+| Llama decode (sq=1, sk=128, nh=32, nkv=8, hd=128) | 0.275 ms | 0.264 ms | 0.429 ms | 1.56× slower |
+
+## Why is coopmat slower?
+
+The 16×16×16 tile shape RDNA3 exposes via `VK_KHR_cooperative_matrix` runs
+wave32 WMMA instructions even though the driver reports subgroupSize=64.
+That halves the usable lanes per coopmat multiply. Plus the shader does:
+
+- Explicit f32 → f16 Q/K/V conversion into shared memory (load, cast,
+ store, barrier). The shared-mem shader reads f32 straight through.
+- Zero-pad of unused `[headDim..MAX_HEAD_DIM]` tail (to match the coopmat
+ stride). Wasted work when `headDim` is 64 or 128.
+- Extra barriers for the per-row softmax update + correction feedback.
+
+For prefill (sq=64) the coopmat path processes 4 query tiles × 9 heads = 36
+workgroups, each doing 16 Q rows × N KV tiles. The shared-mem path does
+64 × 9 = 576 workgroups, each doing 1 Q row. RDNA3.5's 8 WGPs prefer more
+smaller workgroups for this shape; coopmat's fewer-but-wider dispatch
+under-fills the GPU.
+
+## Decisions
+
+- **Dispatch priority kept as `coopmat > subgroup > shared`** per the task.
+ NVIDIA tensor cores + AMD discrete (gfx110x) will make this the right
+ choice; leaving the dispatch default gives those paths the speedup and
+ lets AMD iGPU users escape via `DOTLLM_VULKAN_FORCE_NO_COOPMAT=1`.
+- Tolerance relaxed to **abs 5e-4 / rel 5e-3** on the coopmat-path arm of
+ the numeric-parity tests. The f16 input quantization adds ~3× abs noise
+ over the shared-mem baseline's 1e-4 / 1e-3; shared and subgroup stay on
+ the strict envelope.
+- `VkPhysicalDeviceCooperativeMatrixFeaturesKHR` sType is `1000506000`
+ (NOT `1000506001` — `1000506001` is the property struct). Had this
+ swapped in the first commit and the feature probe reported
+ `cooperativeMatrix = 0` on a device that clearly exposed the tile.
+
+## Next steps (out of scope for this commit)
+
+- Re-bench on an NVIDIA GPU (tensor cores at 16×16×16 tf32/f16) and on
+ AMD discrete (gfx110x actual wave32 WMMA). Expect coopmat to win cleanly
+ there.
+- Larger prefill shapes — as seq_q grows past ~256, the shared-mem path's
+ per-query-row workgroup overhead starts dominating and coopmat's
+ 16-row-per-WG dispatch becomes more favorable.
+- Use `VK_EXT_subgroup_size_control` to force wave32 on RDNA3 — that's
+ what llama.cpp does to match the WMMA instruction width and halves the
+ LDS footprint. Changing WG_SIZE from 64 to 32 plus explicit wave32 would
+ likely close the gap on AMD iGPU.
diff --git a/.perf-runs/vulkan-next-version-plan.md b/.perf-runs/vulkan-next-version-plan.md
new file mode 100644
index 00000000..c6197aed
--- /dev/null
+++ b/.perf-runs/vulkan-next-version-plan.md
@@ -0,0 +1,92 @@
+---
+date: 2026-04-24
+target-tag: v0.2.0-alpha.2 (or v0.2.0-beta if we beat llama.cpp)
+baseline: v0.2.0-alpha at commit 5d04691
+baseline-perf: 4.94 ms/forward = 202 tok/s decode on SmolLM-135M.Q8_0 (AMD Radeon 8060S, RDNA3.5)
+target: ≥349 tok/s decode (match llama.cpp Vulkan d0a6dfeb2 on same hardware)
+gap: 1.73×
+---
+
+## Current state of the 4.94 ms forward
+
+Dispatch count per decode step on SmolLM-135M (30 layers):
+
+| Category | Per layer | × layers | + outer | Total |
+|---|---:|---:|---:|---:|
+| matmul_q8_0 (GEMV) | 7 | 210 | 1 (lm_head) | 211 |
+| rmsnorm | 2 | 60 | 1 (output) | 61 |
+| rope | 1 | 30 | 0 | 30 |
+| kv-cache-update | 1 | 30 | 0 | 30 |
+| attention | 1 | 30 | 0 | 30 |
+| add (residual) | 2 | 60 | 0 | 60 |
+| copy-buffer | 4 | 120 | 2 | 122 |
+| swiglu | 1 | 30 | 0 | 30 |
+| **Total** | **19** | **570** | **4** | **~574** |
+
+~9 µs per dispatch on average. Only 40% of that is matmul compute; the rest is attention work (1.5-1.8 ms total), non-matmul overhead, and copies.
+
+## Reference: what llama.cpp's Vulkan backend does differently
+
+From `C:/Development/llama.cpp/llama.cpp-repo/ggml/src/ggml-vulkan/vulkan-shaders/`:
+
+- **Coopmat matmul:** `mul_mm_cm2.comp` (KHR_coopmat2 / NV) and coopmat-1 path via `mul_mat_vecq.comp`. 16×16×16 int8/fp16 tiles via `VK_KHR_cooperative_matrix`. AMD RDNA3.5 iGPU supports this — `llama-bench --verbose` reports `matrix cores: KHR_coopmat`.
+- **Coopmat flash attention:** `flash_attn_cm1.comp` / `flash_attn_cm2.comp` + `flash_attn_split_k_reduce.comp`. Tile-by-tile QK^T, online softmax, tile-by-tile AV — all in coopmat tiles.
+- **Split-K reduction:** both GEMM and attention have split-K variants with a `*_split_k_reduce.comp` second pass. More parallelism for small-output-dim decode shapes.
+
+Their decode on this hardware: 349.4 tok/s. These are the two unlocks we're missing.
+
+## Plan (ROI-ordered, parallelism-aware)
+
+### Round 1 (parallel, independent files)
+
+**A1. Coopmat Q8_0 GEMM** — issue [#2](https://github.com/jamesburton/dotLLM/issues/2).
+- New shader `matmul_q8_0_gemm_coopmat.comp` + new `MatMulQ8_0GemmCoopmatKernel.cs` + new test file.
+- Runtime probe `VK_KHR_cooperative_matrix` (pattern already in `VulkanDevice.cs` subgroup probe).
+- Dispatch coopmat when available, scalar fallback otherwise.
+- **Expected: 1.0 ms saving** (≈202 → 255 tok/s).
+- **Risk:** medium. Coopmat tile shape + Q8_0 dequant-to-tile is new territory but llama.cpp reference exists.
+- **Does NOT modify:** existing `matmul_q8_0_gemm.comp` (scalar), existing `matmul_q8_0.comp` (GEMV).
+
+**A2. Coopmat Flash Attention** — new issue (to be filed).
+- New shader `flash_attention_f32_coopmat.comp` + new `FlashAttentionCoopmatKernel.cs` + tests.
+- Also file-independent: does NOT modify existing `attention_f32.comp` or `attention_f32_sg.comp`.
+- **Expected: 1.2 ms saving** (≈255 → 340 tok/s).
+- **Risk:** high. Online softmax + coopmat + numerical stability.
+
+**B3. Upload embedding table once** — issue [#12](https://github.com/jamesburton/dotLLM/issues/12).
+- Touches `VulkanWeights.cs` + small bit of `VulkanTransformerModel.Forward`.
+- **Expected: 0.05 ms saving** (noise-level on decode; more meaningful on prefill).
+- **Risk:** low. Pattern identical to the perf-wave's device-local weight migration.
+- Runs parallel with A1/A2 — the files touched don't overlap enough to block.
+
+### Round 2 (after Round 1 lands)
+
+**B1. Eliminate copy dispatches via buffer swap** — new issue (to be filed).
+- `VulkanTransformerModel.Forward` currently does 4 `RecordCopyBuffer` per layer (HiddenState↔Residual↔AddScratch). Replace with pointer/handle swap — the buffers are identical shape so a ping-pong works.
+- **Expected: 0.3 ms saving** (~120 copy dispatches eliminated).
+- **Risk:** low. Pure refactor of forward loop.
+
+**B2. Fuse rmsnorm + matmul** — new issue (to be filed).
+- New shader `rmsnorm_matmul_q8_0.comp` doing rmsnorm → in-register matmul without round-tripping through `NormOutput` buffer. Attention and FFN in-rmsnorms both feed a Q8_0 matmul immediately after.
+- **Expected: 0.2 ms saving** (4 fused dispatches per layer × 30).
+- **Risk:** medium. New shader, reorderable numerics.
+
+### Round 3 (validation + tag)
+
+- Re-run `VulkanForwardPerfHarness` (32 decode steps, 4 warmup).
+- Re-run llama.cpp `llama-bench` for fresh reference.
+- Update `.continue-here.md` with final numbers.
+- Tag `v0.2.0-alpha.2` (or `v0.2.0-beta` if we match/beat llama.cpp).
+
+## Expected final
+
+| | Decode (tok/s) | ms/forward |
+|---|---:|---:|
+| Baseline (v0.2.0-alpha) | 202 | 4.94 |
+| + A1 (coopmat GEMM) | 255 | 3.9 |
+| + A2 (coopmat attention) | 340 | 2.9 |
+| + B1 (buffer swap) | 385 | 2.6 |
+| + B2 (fused norm+matmul) | 430 | 2.3 |
+| + B3 (embedding) | 435 | 2.3 |
+
+Target: ≥349 (llama.cpp parity). Stretch: >435 (decisively beat llama.cpp).
diff --git a/.perf-runs/vulkan-perf-wave-20260424/README.md b/.perf-runs/vulkan-perf-wave-20260424/README.md
new file mode 100644
index 00000000..b8f347ca
--- /dev/null
+++ b/.perf-runs/vulkan-perf-wave-20260424/README.md
@@ -0,0 +1,272 @@
+# Vulkan Perf Wave — 2026-04-24
+
+## Host
+
+- GPU: **AMD Radeon(TM) 8060S Graphics** (Strix Halo iGPU, RDNA 3.5, gfx1151)
+- CPU: Ryzen AI Max+ 395 (Zen 5, 32T)
+- Memory: UMA DDR5
+- Vulkan driver: AMD proprietary (`vulkan-1.dll`)
+- Build: `-c Release`, `net10.0`, JIT
+- Branch: `feature/vulkan-perf-wave` (branched off `feature/mamba-3` at `c256d19`)
+
+## Goal
+
+Take the CPU↔Vulkan end-to-end forward from "seconds per forward" (host-visible
+everything, one `vkQueueWaitIdle` per kernel, no descriptor reuse) to
+something useful — without touching kernel quality or shaders.
+
+Ground truth: `VulkanForwardPerfHarness.MeasureDecodeLatency` on
+SmolLM-135M.Q8_0, 16 decode steps after 3 warmup. Wall-clock per
+`VulkanTransformerModel.Forward` call.
+
+Reproduction:
+
+```bash
+cd .claude/worktrees/agent-a74786ce # or wherever feature/vulkan-perf-wave is checked out
+DOTLLM_VULKAN_PERF=1 DOTLLM_VULKAN_PERF_DECODE_STEPS=16 DOTLLM_VULKAN_PERF_WARMUP=3 \
+ dotnet test tests/DotLLM.Tests.Integration \
+ --filter "FullyQualifiedName~VulkanForwardPerfHarness" \
+ -c Release --logger "console;verbosity=detailed"
+```
+
+Correctness oracle at every step:
+
+```bash
+dotnet test tests/DotLLM.Tests.Integration --filter "FullyQualifiedName~VulkanTransformerModelTests" -c Release
+dotnet test tests/DotLLM.Tests.Unit --filter "FullyQualifiedName~Vulkan" -c Release
+```
+
+## Summary
+
+| Stage | Commit | decode avg (ms) | tok/s | Δ vs prev |
+| ---------------------------------- | --------- | --------------: | ----: | --------: |
+| Baseline (c256d19 as checked out) | c256d19 | 49.36 | 20.26 | — |
+| Step 1: device-local weights + KV | e8266d0 | 42.97 | 23.27 | -13% |
+| Step 2: fence-pipelined forward | d43ff71 | 15.70 | 63.71 | -63% |
+| Step 3: descriptor-set cache | 7566ee8 | 13.21 | 75.69 | -16% |
+
+**End-to-end: 49.36 → 13.21 ms/forward (3.7× speedup), 20.26 → 75.69 tok/s (3.7×).**
+
+All three steps kept the CPU↔Vulkan parity test at 8/9 strict argmax
+matches (the single step-1 swap is Q8_0-vs-F32 noise, unchanged from the
+baseline). Every Vulkan unit test (53/53) stays green across all three
+commits. Full unit suite stays at 1396/0/38.
+
+## Step-by-step
+
+### Step 1 — Device-local weights + KV cache (commit `e8266d0`)
+
+**Before.** Every `VulkanDevice.Allocate` returned a buffer backed by
+`VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | HOST_COHERENT_BIT`. Weights lived
+in the same host-coherent linear memory a compute shader would read via
+the same HOST memory bandwidth as a `memcpy` would. KV cache was
+similarly host-coherent and updated via host-mapped `System.Buffer.MemoryCopy`.
+
+**After.** Two allocator paths:
+
+- `VulkanDevice.Allocate(long)` — still host-visible host-coherent, for
+ activation / scratch buffers the forward pass reads from the host
+ (embedding upload, position upload, bias-add fallback, logits
+ download).
+- `VulkanDevice.AllocateDeviceLocal(long)` — device-local only (prefers
+ a `DEVICE_LOCAL_BIT` memory type that is *not* `HOST_VISIBLE` so the
+ driver picks its tiled / swizzled native layout). Falls back to
+ `DEVICE_LOCAL + HOST_VISIBLE` on older Intel / mobile GPUs that only
+ expose the combined pool.
+
+Weight upload (`VulkanWeights.Upload`) pre-allocates one reusable
+host-visible staging buffer sized to the widest matrix (lm_head
+49152×576 on SmolLM-135M). Each weight matrix:
+
+1. Map staging, dequantize / copy FP32 rows into it, unmap.
+2. `vkCmdCopyBuffer(staging → device-local)` on a one-shot command
+ buffer, fenced.
+
+KV cache (`VulkanKvCache`) now allocates device-local from the start.
+`UpdateDevice` records `vkCmdCopyBuffer` from the current forward's
+host-visible K/V scratch into the cache (one contiguous region when
+positions are consecutive, per-row otherwise) and fences.
+
+Measured decode avg: **49.36 → 42.97 ms** (13% drop).
+
+The gain is modest on this iGPU because the bytes still sit in the same
+DDR5 — there's no PCIe bus to cross. A discrete dGPU would see a much
+bigger jump on this step because the weights physically move off the
+system-memory bus into VRAM. The ~13% we do see on UMA is the driver
+switching from host-coherent linear memory to its tiled compute layout.
+
+### Step 2 — Fence-pipelined forward (commit `d43ff71`)
+
+**Before.** Each of the 10 kernels' `Launch(...)` allocated its own
+command buffer, submitted one dispatch, called `vkQueueWaitIdle`, and
+freed the command buffer. SmolLM-135M with 30 layers dispatches ~15
+kernels per layer for a total of ~450 per forward, plus many host-side
+`vkMapMemory` round-trips for the per-layer residual copies. Every one
+of those 450+ round-trips went host ↔ driver ↔ GPU ↔ driver ↔ host.
+
+**After.** Refactored every kernel into two entry points:
+
+- `Record(cmdBuf, ...)` — appends the dispatch to a caller-owned command
+ buffer. No submit, no fence.
+- `Launch(...)` — wraps `Record` in a one-shot submit context for
+ standalone unit tests and API compatibility.
+
+`VulkanTransformerModel` now owns one persistent `SubmitContext`
+(command buffer + fence). `Forward` calls `Begin`, records the entire
+transformer (with `COMPUTE→COMPUTE` memory barriers between kernels,
+`TRANSFER→COMPUTE` after each KV update, `HOST→COMPUTE` after the
+embedding/position uploads, `COMPUTE→HOST` before the logit download),
+then `SubmitAndWait` once. The per-kernel residual copies that used to
+be `vkMapMemory + Buffer.MemoryCopy` become `vkCmdCopyBuffer` in the
+same command buffer.
+
+Optional bias adds remain host-mapped (SmolLM-135M has no biases, so
+the whole forward lands in one submit on the test model). When a model
+*does* have biases, the record loop breaks into multiple submits around
+the host step — each submit still batches many dispatches behind one
+fence.
+
+`KernelSupport` centralizes descriptor-pool creation (now
+`maxSets=1024` per kernel) and the barrier shapes.
+
+Measured decode avg: **42.97 → 15.70 ms** (63% drop, 2.7× speedup).
+
+This is the biggest single lever. The scaffold wasn't slow because the
+kernels were slow — it was slow because every dispatch round-tripped
+through the host.
+
+### Step 3 — Descriptor-set cache per kernel (commit `7566ee8`)
+
+**Before.** Each `Record` call ran
+`vkAllocateDescriptorSets` + `vkUpdateDescriptorSets` to build a fresh
+descriptor set, then used it for exactly one dispatch. Across a SmolLM
+forward the matmul kernel alone does this 211 times per forward,
+RMSNorm 61 times, and so on.
+
+**After.** `DescriptorSetCache` (one per kernel) keys on the tuple of
+buffer handles bound to the set. First call with a new tuple allocates
+and writes; every subsequent call with the same tuple re-uses the
+cached handle and skips both the allocate and the update.
+
+Cache lifetime spans forward passes: weights and activation scratch
+have stable `VkBuffer` handles once the first forward has grown
+`VulkanForwardState` to the longest seqLen the caller will hit. So the
+cache warms through the prefill and stays hot for every decode step.
+`VulkanForwardState.EnsureCapacity` now returns a bool so the model can
+invalidate every kernel's cache on the rare scratch-regrow path; no
+reset on the steady-state decode loop.
+
+Capacity is 256 slots per kernel (SmolLM-135M hits ~211 distinct matmul
+tuples per forward). On overflow the pool resets and the cache drops
+every entry — a slow fallback that never fires in normal inference.
+
+Measured decode avg: **15.70 → 13.21 ms** (16% drop). Steady-state min
+/ max 11.90 ms / 15.00 ms.
+
+The remaining headroom from here is in kernel quality, not dispatch
+machinery — see "Not in scope" below.
+
+## Per-step latency distribution (after Step 3)
+
+Steady-state decode on SmolLM-135M.Q8_0:
+
+```
+warmup [0..2]: 17.6..21.4 ms (cache warming)
+decode [0]: 15.00 ms (one distinct tuple vs warmup — branch on kvCache.CurrentLength crossing 8)
+decode [1..15]: 11.90..14.44 ms (cache fully warm)
+```
+
+## Limits
+
+On this UMA box (Ryzen AI Max+ 395 iGPU), steady-state decode is
+~**75 tok/s** on the F32 Vulkan path. llama.cpp Vulkan's reference is
+~392 tok/s on the same hardware — but that path uses Q8_0 end-to-end
+weights, not the FP32-dequantized path we have today. Closing that gap
+is the Q8_0-end-to-end + cooperative-matrix work, tracked on the
+existing follow-up issues and explicitly out of scope for this wave.
+
+## Not in scope (follow-ups)
+
+1. **Q8_0 end-to-end** — a parallel agent owns this. Once Q8_0 blocks
+ stay on device and matmuls route through `matmul_q8_0` (GEMV) /
+ `matmul_q8_0_gemm` (batched), the weight memory and the kernel op
+ count both drop ~4×.
+2. **Cooperative-matrix Q8 GEMM** — RDNA3 WMMA 16×16×16 int8 via
+ `VK_KHR_cooperative_matrix`. Expected ~10× on the batched kernel.
+3. **`bias_add_f32` kernel** — pulls Phi / Qwen bias adds out of the
+ host-mapped bracket; removes the per-layer submit split when biases
+ are present.
+4. **Async scratch pool** — if a future model grows scratch mid-decode
+ (long-context), the current invalidate-and-reset cache flush is a
+ submit-wide stall. A small async-allocated scratch pool would keep
+ the cache warm across resizes.
+5. **Staging ring for prefill** — currently one reusable staging buffer
+ sized to the largest matrix. A small ring would let weight uploads
+ pipeline during model load; not load-path-critical today.
+
+## Files changed
+
+- `src/DotLLM.Vulkan/VulkanDevice.cs` — `AllocateDeviceLocal`,
+ `UploadToDeviceLocal`, `CopyBufferRangeSynchronous`, `SubmitContext`.
+- `src/DotLLM.Vulkan/VulkanWeights.cs` — staging-buffered device-local
+ weight upload.
+- `src/DotLLM.Vulkan/VulkanKvCache.cs` — device-local buffers,
+ `RecordUpdate(cmdBuf, ...)` for the pipelined forward plus the
+ legacy synchronous `UpdateDevice` fallback.
+- `src/DotLLM.Vulkan/VulkanForwardState.cs` — `EnsureCapacity` now
+ returns a `bool` to signal scratch regrow.
+- `src/DotLLM.Vulkan/VulkanTransformerModel.cs` — refactored
+ `Forward` to record the whole transformer into one persistent
+ command buffer + fence.
+- `src/DotLLM.Vulkan/Kernels/*.cs` — every kernel gained a
+ `Record(cmdBuf, ...)` entry point; `Launch` kept as a legacy
+ wrapper; each kernel owns a `DescriptorSetCache`.
+- `src/DotLLM.Vulkan/Kernels/KernelSupport.cs` (new) — shared
+ descriptor-pool / set / barrier helpers.
+- `src/DotLLM.Vulkan/Kernels/DescriptorSetCache.cs` (new) — per-kernel
+ buffer-handle-keyed descriptor set cache.
+- `src/DotLLM.Vulkan/Interop/VulkanApi.cs` +
+ `src/DotLLM.Vulkan/Interop/VulkanStructs.cs` — `vkCmdCopyBuffer`,
+ `vkCmdPipelineBarrier`, fence APIs, memory-barrier / buffer-copy /
+ fence structs, pipeline-stage / access flags.
+- `tests/DotLLM.Tests.Integration/Vulkan/VulkanForwardPerfHarness.cs`
+ (new) — `DOTLLM_VULKAN_PERF=1`-gated timing harness.
+
+## Raw measurements
+
+Each run: 3 warmup decodes (timed but reported separately) + 16 steady
+decodes. Wall time per `VulkanTransformerModel.Forward` via
+`System.Diagnostics.Stopwatch`. Host `dotnet test` default Release GC
+/ JIT, `SustainedLowLatency` not forced.
+
+### Baseline (c256d19)
+
+```
+prefill_len=5 prefill_ms=80.77
+warmup_avg_ms=51.74
+decode_avg_ms=49.36 min=39.87 max=107.99 tok/s=20.26
+```
+
+### After Step 1 (e8266d0)
+
+```
+prefill_len=5 prefill_ms=114.79 # includes staging copies
+warmup_avg_ms=46.61
+decode_avg_ms=42.97 min=38.27 max=49.18 tok/s=23.27
+```
+
+### After Step 2 (d43ff71)
+
+```
+prefill_len=5 prefill_ms=54.70
+warmup_avg_ms=22.79
+decode_avg_ms=15.70 min=14.55 max=18.07 tok/s=63.71
+```
+
+### After Step 3 (7566ee8)
+
+```
+prefill_len=5 prefill_ms=45.62
+warmup_avg_ms=18.31
+decode_avg_ms=13.21 min=11.90 max=15.00 tok/s=75.69
+```
diff --git a/.perf-runs/vulkan-q8-e2e-20260424/README.md b/.perf-runs/vulkan-q8-e2e-20260424/README.md
new file mode 100644
index 00000000..e5b49793
--- /dev/null
+++ b/.perf-runs/vulkan-q8-e2e-20260424/README.md
@@ -0,0 +1,242 @@
+# Vulkan Q8_0 end-to-end — 2026-04-24
+
+## Host
+
+- GPU: **AMD Radeon(TM) 8060S Graphics** (Strix Halo iGPU, RDNA 3.5, gfx1151)
+- CPU: Ryzen AI Max+ 395 (Zen 5, 32T)
+- Memory: UMA DDR5
+- Vulkan driver: AMD proprietary (`vulkan-1.dll`)
+- Build: `-c Debug` (default for `dotnet test`), `net10.0`, JIT
+- Branch: `feature/vulkan-q8-end-to-end` (branched off `feature/mamba-3` at `6f509de`)
+
+## Goal
+
+Close the last big lever called out at the end of the Vulkan perf wave
+(`vulkan-perf-wave-20260424`): stop dequantising Q8_0 weights to FP32 at
+load. Keep the 34-byte blocks on device, route matmuls through the
+already-validated `matmul_q8_0` (GEMV) and `matmul_q8_0_gemm` (batched
+GEMM) kernels. No shader edits — this is a host-side routing change.
+
+Ground truth: `VulkanForwardPerfHarness.MeasureDecodeLatency` on
+SmolLM-135M.Q8_0. Wall-clock per `VulkanTransformerModel.Forward` call.
+
+Reproduction:
+
+```bash
+DOTLLM_VULKAN_PERF=1 DOTLLM_VULKAN_PERF_DECODE_STEPS=32 DOTLLM_VULKAN_PERF_WARMUP=4 \
+ dotnet test tests/DotLLM.Tests.Integration \
+ --filter "FullyQualifiedName~VulkanForwardPerfHarness" \
+ --logger "console;verbosity=detailed"
+```
+
+Correctness oracle at every commit:
+
+```bash
+dotnet test tests/DotLLM.Tests.Integration --filter "FullyQualifiedName~VulkanTransformerModel"
+dotnet test tests/DotLLM.Tests.Unit --filter "FullyQualifiedName~Vulkan"
+```
+
+## Summary
+
+| Stage | Commit | decode avg (ms) | tok/s | Δ vs prev |
+| ----------------------------------------- | -------- | --------------: | -----: | --------: |
+| Baseline (6f509de — Vulkan perf wave tip) | 6f509de | 11.49 | 87.02 | — |
+| Step 1: `VulkanWeights` Q8_0 option | 73f9424 | 11.49 | 87.02 | 0% (no-op) |
+| Step 2: route matmuls through Q8_0 | 944cd67 | 6.23 | 160.40 | **-46%** |
+
+**End-to-end: 11.49 → 6.23 ms/forward (1.85× speedup), 87 → 160 tok/s (1.84×).**
+
+Correctness: `VulkanForward_MatchesCpuReference_OnEightDecodeSteps`
+holds at **8/9 strict argmax matches** across prefill + 8 decode steps,
+identical to the baseline count. The single swap on step 1 is a top-1 /
+top-2 neighbour tie that survives from the F32-dequant path — both
+backends now read the exact same Q8_0 block bytes, so the only
+arithmetic divergence is Vulkan's direct Q8_0 dot-with-FP32-activation
+vs CPU's `GemmQ8_0` (which quantises the activation to Q8_1). Top-5
+jaccard holds throughout.
+
+Unit tests: 53 passed / 1 skipped Vulkan filter — unchanged from baseline.
+
+## Step-by-step
+
+### Step 1 — `VulkanWeights` keeps Q8_0 on device (commit `73f9424`)
+
+**Before.** `VulkanWeights.UploadMatrix` unconditionally dequantised
+every Q8_0 weight row to FP32 through `Dequantize.ToFloat32` before
+staging. On SmolLM-135M's Q8_0 GGUF the matmul-weight total on device
+was ~520 MiB (FP32) vs ~144 MiB of source bytes.
+
+**After.** `Upload` gains an optional `dequantToFp32` parameter
+(default `false`). When `false` and the source quant type is Q8_0, the
+34-byte blocks are uploaded to device-local memory verbatim via
+`vkCmdCopyBuffer` through the same reusable staging buffer.
+`LayerBuffers` records the device-side quant type alongside each buffer
+handle so the forward dispatch can pick the right kernel. F16 and
+K-quant paths still dequant to F32 at upload (those Vulkan kernels
+don't exist yet); `dequantToFp32: true` preserves the old all-F32
+behaviour as a fallback switch.
+
+The `VulkanTransformerModel` call site passes `dequantToFp32: true` in
+this step so the forward behaviour is unchanged — Step 1 just makes the
+weights-side change without activating it. This keeps the tree green
+between commits.
+
+Decode avg: **11.49 → 11.49 ms** (no-op, flag forced to true).
+
+### Step 2 — Route matmuls through Q8_0 kernels (commit `944cd67`)
+
+**Before.** `VulkanTransformerModel.Forward` dispatched every linear
+projection through `MatMulF32Kernel.Record`, which reads FP32 weights.
+
+**After.** New `RecordMatmul(cmdBuf, weights, weightQt, ...)` helper
+picks the kernel by device-side quant type and seqLen:
+
+- Q8_0 + seqLen == 1 (decode) → `MatMulQ8_0Kernel.Record` (GEMV)
+- Q8_0 + seqLen > 1 (prefill) → `MatMulQ8_0GemmKernel.Record` (batched)
+- F32 → `MatMulF32Kernel.Record` (unchanged)
+
+All 7 per-layer projections (Q, K, V, O, Gate, Up, Down) **and** the LM
+head go through the dispatcher. The LM head on SmolLM-135M is Q8_0
+(49152×576), so the biggest single matmul in the forward now reads 34
+bytes per 32 elements instead of 128. Activations and scratch stay
+FP32 end-to-end — only the weight read-side changes.
+
+Weights upload now runs with `dequantToFp32: false` (default) so Q8_0
+matrices actually stay on device as blocks.
+
+Decode avg: **11.49 → 6.23 ms** (46% drop, 1.85× speedup).
+
+This is consistent with the "weight bandwidth bound" hypothesis. The
+weight read is ~4× fewer bytes (34 vs 128 per 32 elements = 3.76×);
+RDNA 3.5's int8 dot is fast enough on this iGPU that the Q8_0 GEMV
+kernel is still bandwidth-bound, so the observed 1.85× matches the
+fraction of the forward that is matmul-weight-read (roughly half —
+activations, KV cache, barriers, and the attention kernel eat the rest).
+
+## Per-step latency distribution (after Step 2)
+
+Steady-state decode on SmolLM-135M.Q8_0 (64 decode steps after 8 warmup):
+
+```
+warmup [0..7]: 5.74..6.55 ms (cache + shader compile warming)
+decode [0..63]: 4.92..9.27 ms
+ min 4.92 median ~6.0 max 9.27 avg 6.23 → 160.4 tok/s
+```
+
+Two-run jitter: a second 32-step run came in at decode_avg=5.85 ms
+(171 tok/s). Iterations are UMA-memory-bound and the DDR5 channel
+shares with background CPU traffic, so ~±6% sample-to-sample noise is
+expected.
+
+## Correctness detail
+
+From the Q8_0 end-to-end integration test at `944cd67`:
+
+```
+step 0 (prefill): cpu=7042 vk=7042 [match] ← Q8_0 GEMM (seqLen=5)
+step 1: cpu=30 vk=28 [swap-in-topK]
+step 2..8: cpu==vk everywhere [match] ← Q8_0 GEMV (seqLen=1)
+summary: 8/9 strict argmax matches
+```
+
+The step-1 swap is the same one the F32-dequant path produced at
+`6f509de` — it's a top-2 neighbour tie from the prefill-step KV cache,
+not an artefact of the Q8_0 routing. Both sides agree on the prefill
+argmax (token 7042 = " Paris") and on all 7 post-swap decode argmaxes.
+
+## VRAM footprint (SmolLM-135M)
+
+Matmul weight bytes on device (only the tensors routed through a matmul
+kernel, not bias / norm / embedding / KV):
+
+- Before: 30 layers × (Q + K + V + O + Gate + Up + Down) + LM head, all FP32
+ → ~139 MiB
+- After: same tensors at Q8_0 (34 bytes / 32 elems ≈ 27% of FP32)
+ → ~37 MiB
+
+The per-layer FP32 form was: Q 576×576, K 192×576, V 192×576, O 576×576,
+Gate/Up 1536×576, Down 576×1536 ≈ 3.94 MiB/layer × 30 = 118 MiB +
+LM head 49152×576 = 108 MiB × (Q8_0: ~27%) ≈ 29 MiB. Exact numbers
+depend on `AllocatedBytes` which the model reports, but the ratio is
+decisive on larger models — a 7B Q8_0 model goes from ~28 GiB on device
+(dequantised) to ~7 GiB (native).
+
+## Limits
+
+On this UMA box, steady-state decode is now ~**160 tok/s** on the
+Vulkan Q8_0 path. llama.cpp Vulkan's reference is ~392 tok/s on the
+same hardware. The remaining gap is kernel-quality, not dispatch
+machinery: a cooperative-matrix Q8_0 GEMM (RDNA3 WMMA 16×16×16 int8
+via `VK_KHR_cooperative_matrix`) is the single biggest next lever.
+
+## Not in scope (follow-ups)
+
+1. **Cooperative-matrix Q8_0 GEMM** — separate filed issue. Expected
+ ~10× on the batched kernel over the current 209 GFLOPS.
+2. **Q4_0 / Q5_0 / K-quant end-to-end** — those Vulkan kernels don't
+ exist yet; those quant types still dequant to F32 at upload as
+ they did pre-wave.
+3. **`bias_add_f32` kernel** — Phi / Qwen biases still host-mapped; a
+ dedicated kernel removes the per-layer submit split. Out of scope
+ here (scope limit: no new kernels).
+4. **GEMV k=32 M>1 stride bug** noted in `.continue-here.md` — not
+ hit by this change (prefill dispatches Q8_0 GEMM, decode uses M=1).
+ Worth a separate regression test + patch on the GEMV path.
+
+## Files changed
+
+- `src/DotLLM.Vulkan/VulkanWeights.cs` — `Upload(..., bool dequantToFp32=false)`;
+ per-LayerBuffers device quant type; `UploadMatrix` now branches on
+ `KeepQ8OnDevice` to upload either raw Q8_0 blocks or FP32.
+- `src/DotLLM.Vulkan/VulkanTransformerModel.cs` — `MatMulQ8_0Kernel` +
+ `MatMulQ8_0GemmKernel` fields, `RecordMatmul` dispatcher; every
+ `_matmul.Record(...)` call in `Forward` replaced with
+ `RecordMatmul(...)`; updated `InvalidateKernelCaches` and `Dispose`.
+
+## Raw measurements
+
+Each run: 4–8 warmup decodes (timed but reported separately) + 32–64
+steady decodes. Wall time per `VulkanTransformerModel.Forward` via
+`System.Diagnostics.Stopwatch`.
+
+### Baseline (6f509de)
+
+```
+load_ms=757.0
+prefill_len=5 prefill_ms=33.48
+warmup_avg_ms=11.80
+decode_avg_ms=11.49 min=10.69 max=13.34 tok/s=87.02
+```
+
+### After Step 1 (73f9424) — no-op (flag forced true at model site)
+
+Identical to baseline (same F32 dequant path, same forward).
+
+### After Step 2 (944cd67) — Q8_0 end-to-end
+
+Run 1 (32 steps, 4 warmup):
+```
+load_ms=428.0
+prefill_len=5 prefill_ms=47.81
+warmup_avg_ms=5.77
+decode_avg_ms=6.61 min=5.63 max=8.57 tok/s=151.34
+```
+
+Run 2 (32 steps, 4 warmup):
+```
+decode_avg_ms=5.85 min=5.08 max=6.99 tok/s=170.80
+```
+
+Run 3 (64 steps, 8 warmup):
+```
+load_ms=~430 (consistent)
+prefill_len=5 prefill_ms=~48
+warmup_avg_ms=6.07
+decode_avg_ms=6.23 min=4.92 max=9.27 tok/s=160.40
+```
+
+Load time improvement (757 → 428 ms, -43%) comes from skipping the
+per-row `Dequantize.ToFloat32` loop across 7 × 30 + 1 = 211 matrices
+in favour of a direct `Span.CopyTo` for the Q8_0 path. Prefill
+rose modestly (33 → 48 ms) — driver shader compile cost on the first
+`matmul_q8_0_gemm` dispatch, amortised across all future prefills.
diff --git a/.perf-runs/vulkan-subgroup-20260424/README.md b/.perf-runs/vulkan-subgroup-20260424/README.md
new file mode 100644
index 00000000..21eed0ac
--- /dev/null
+++ b/.perf-runs/vulkan-subgroup-20260424/README.md
@@ -0,0 +1,54 @@
+# Vulkan Subgroup-Arithmetic vs Shared-Memory Reduce — 2026-04-24
+
+## Host
+
+- GPU: **AMD Radeon(TM) 8060S Graphics** (Strix Halo iGPU, RDNA 3.5)
+- Vulkan driver: AMD proprietary (`vulkan-1.dll`)
+- Subgroup size: **64**
+- Subgroup arithmetic: **supported** (stage=compute, ops includes Arithmetic)
+- Build: `-c Release`, `net10.0`, JIT
+
+## Probe output
+
+```
+Device : AMD Radeon(TM) 8060S Graphics
+VendorId : 0x1002
+SubgroupSize : 64
+HasSubgroupArith : True
+```
+
+## Benchmark harness
+
+- `tests/DotLLM.Tests.Unit/Vulkan/VulkanSubgroupMicroBench.cs` gated by `DOTLLM_VULKAN_SUBGROUP_BENCH=1`.
+- Each sample: **10 warmup + 200 timed** dispatches. Each dispatch uses a dedicated kernel instance because the wave-2 descriptor pool has `maxSets=1` (documented deferred work).
+- Synchronous submission (`vkQueueSubmit` + `vkQueueWaitIdle`) per dispatch. End-to-end inference will batch behind fewer fences, so per-dispatch GPU submission overhead is amortized better in practice than these numbers suggest.
+
+## Results
+
+| Kernel | Shape | Shared-mem ms/iter | Subgroup ms/iter | Speedup |
+| -------------- | ----------------------------------------------- | -----------------: | ---------------: | ------: |
+| rmsnorm_f32 | rowCount=4, n=1536 | 0.046 | 0.045 | 1.02x |
+| attention_f32 | seqQ=1, seqKv=128, nh=9, nkv=3, hd=64 | 0.068 | 0.135 | 0.50x |
+| attention_f32 | seqQ=1, seqKv=512, nh=4, nkv=2, hd=128 (2-tile) | 0.168 | 0.177 | 0.95x |
+
+## Interpretation
+
+- **rmsnorm**: effectively a wash (1.02x). RDNA 3.5 handles the 8 `barrier()` ops in the shared-memory tree reduce cheaply; the subgroup path saves log₂(256)=8 barriers but the extra `subgroupAdd` lane shuffling and the broadcast-slot barrier eat most of the savings at N=1536.
+- **attention (small)**: the subgroup variant is *slower*, 0.50x. On a single 256-wide workgroup the shared-mem tree reduce at RDNA's 64-lane subgroup width is only 2 cross-subgroup iterations; `subgroupAdd` adds register pressure and an extra broadcast-slot barrier per reduction call. At small seq_kv the QKᵀ dot is cheap and reduction overhead dominates the delta.
+- **attention (multi-tile)**: approaches parity (0.95x) once the 256-tile inner loop dominates per-thread work, relegating reduction cost to the noise floor. At seqKv≥1024 the delta is below measurement jitter.
+
+**Conclusion.** The subgroup-arithmetic variants are *numerically* equivalent (verified by `VulkanSubgroupPathParityTests` within abs 1e-4 / rel 1e-3) but are not a win on RDNA 3.5 for these shapes. The wins the wave-2 handoff predicted (~4× barrier reduction) are real but barriers are not the bottleneck on modern AMD hardware at WG=256. Keeping both paths in the tree pays off on:
+- **NVIDIA Ampere/Ada** (subgroup size 32) where barrier count is 8 vs 5 — more headroom for wins.
+- **Mobile GPUs** (Adreno, Mali) where barrier latency is significantly higher than desktop.
+- **Large hidden sizes** (n≥4096 in rmsnorm) where the shared-memory accumulator path spills registers.
+
+Correctness is the deliverable for this session; leaving the selection logic in place + env-var override (`DOTLLM_VULKAN_FORCE_SHARED_REDUCE=1`) lets subsequent sessions tune per-vendor without code changes.
+
+## How to reproduce
+
+```bash
+cd /c/Development/dotLLM/.claude/worktrees/agent-a5844750
+DOTLLM_VULKAN_SUBGROUP_BENCH=1 dotnet test tests/DotLLM.Tests.Unit \
+ --filter "FullyQualifiedName~VulkanSubgroupMicroBench" \
+ -c Release --logger "console;verbosity=detailed"
+```
diff --git a/CONTINUE_ON_CUDA.md b/CONTINUE_ON_CUDA.md
new file mode 100644
index 00000000..0c36cf27
--- /dev/null
+++ b/CONTINUE_ON_CUDA.md
@@ -0,0 +1,238 @@
+# CONTINUE_ON_CUDA.md — Pickup instructions for CUDA work
+
+**Target audience:** an engineer with an NVIDIA GPU available, continuing
+dotLLM's CUDA backend from where `feature/mamba-3` left off. The Ryzen dev
+machine this branch was primarily developed on has only an AMD Radeon 8060S
+iGPU — the CUDA path has the scaffolding in place but cannot be executed
+there. All CUDA tests are present and skip cleanly without a device.
+
+Branch: `feature/mamba-3`. Checkout commit: `576ea91` or later.
+
+---
+
+## What exists today
+
+`src/DotLLM.Cuda/` is a full project with:
+
+- **P/Invoke layer** against NVIDIA's CUDA Driver API (`libcuda.so` / `nvcuda.dll`) and cuBLAS — see `Interop/CudaDriverApi.cs`, `Interop/CublasApi.cs`. No custom C shared library; we call NVIDIA's system libs directly. License-clean (no ManagedCuda dep).
+- **Model stack**: `CudaTransformerModel`, `CudaWeights`, `CudaForwardState`, `CudaKvCache`, `CudaQuantizedKvCache`.
+- **Kernel launcher**: `CudaModule`, `CudaKernels`, `CudaGemm`.
+- **PTX kernel catalog** (`native/kernels/*.cu`, compiled via `nvcc -ptx`): RMSNorm, RoPE (Norm + NeoX), Attention (naive), SwiGLU, Add, BiasAdd, Softmax, Embedding (F32/F16/Q8_0), Dequant (Q8_0/Q4_0/Q5_0/Q4_K/Q5_K/Q6_K), Quantized GEMV (Q8_0/Q4_K/Q6_K), Conversion (F16↔F32).
+- **cuBLAS GEMM** for prefill path (FP16 with Tensor Core auto-dispatch when dims are ×8).
+- **36 skipped unit tests** in `tests/DotLLM.Tests.Unit/Cuda/` — all runtime-probe CUDA availability and skip gracefully when no GPU is present.
+
+See `docs/CUDA.md` (~700 lines, authoritative design doc) and `docs/GPU.md`
+(data-flow overview).
+
+## What almost-certainly works vs what needs validation
+
+| Area | Status | Likely effort on an NVIDIA box |
+|---|---|---|
+| Project builds on machines without CUDA | ✅ verified (all dev so far) | — |
+| Project builds with CUDA Toolkit installed | likely (no blockers) | minutes |
+| `nvcc -ptx` on each `native/kernels/*.cu` | likely | minutes (`native/build.sh`) |
+| `CudaModule.LoadFromFile` picks up the PTX | likely | — |
+| 36 skipped tests become runnable | ✅ expected | — |
+| Individual kernel correctness vs CPU reference (`CudaKernelComparisonTests`) | not exercised | hours (expected to pass first try for most; any failure is informative) |
+| Full forward-pass on GGUF (SmolLM-135M, TinyLlama) | not exercised | day-ish — may hit rough edges |
+| cuBLAS path with Tensor Cores | not exercised | hours |
+| Quantized GEMV decode path | not exercised | hours |
+| MLA on CUDA | **not started** | weeks — Phase A/B/C CPU path is the oracle |
+| MoE on CUDA | **not started** | weeks |
+| Mamba-3 SSM on CUDA | **not started** | weeks |
+
+## Step-by-step pickup
+
+### 1. Clone + build on the CUDA machine
+```bash
+git clone https://github.com/jamesburton/dotLLM.git
+cd dotLLM
+git checkout feature/mamba-3
+dotnet build --configuration Debug
+```
+Expected: 0 warnings, 0 errors. The build succeeds whether or not CUDA is
+present — all CUDA interop is runtime-resolved.
+
+### 2. Verify CUDA is visible
+```bash
+nvidia-smi # confirm GPU + driver ≥ 525
+nvcc --version # confirm CUDA Toolkit 12.x for PTX build
+```
+
+### 3. Build the PTX kernels
+```bash
+cd native
+./build.sh # or build.ps1 on Windows
+# Expected output: one PTX file per .cu in native/kernels/
+ls native/ptx/
+```
+If any `.cu` fails to compile, it's almost certainly a nvcc version mismatch
+or a missing header. Fix one at a time; the kernels are independent.
+
+### 4. Run CUDA kernel comparison tests
+
+These are the 36 currently-skipped tests. They build reference outputs on
+CPU (via `DotLLM.Cpu.Kernels`) and compare against CUDA results.
+
+```bash
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj \
+ --filter "FullyQualifiedName~Cuda" \
+ --logger "console;verbosity=normal"
+```
+
+Expected: most pass, some may need tolerance tuning (`5e-4` is the common
+threshold; Q4_K dequant-then-matmul might want looser on F16 paths).
+
+### 5. Wire `ModelLoader.LoadFromGguf(... deviceId = 0)` end-to-end
+```csharp
+using var gguf = GgufFile.Open("path/to/SmolLM-135M.Q8_0.gguf");
+var (model, _, config) = ModelLoader.LoadFromGguf(gguf, threading: ...,
+ deviceId: 0); // CUDA
+using ITensor logits = model.Forward(tokenIds, positions, deviceId: 0);
+```
+
+Check: does `ModelLoader.LoadFromGguf` actually dispatch to `CudaTransformerModel`
+when `deviceId >= 0`? As of commit `576ea91` the dispatch exists in the
+code path but has not been exercised. If it throws `NotSupportedException`
+or similar, the dispatch arm is the first thing to wire up.
+
+### 6. Real-weight sanity
+
+Use the same fixtures the CPU tests use:
+
+```bash
+# On your CUDA machine, place (or env-var) the checkpoints:
+# C:/temp/dotllm-tinyllama/ (or $DOTLLM_TINYLLAMA_CHECKPOINT_PATH)
+# C:/temp/dotllm-phi35-mini/ (or $DOTLLM_PHI35_CHECKPOINT_PATH)
+# C:/temp/dotllm-granite31-moe/ (or $DOTLLM_GRANITE3_CHECKPOINT_PATH)
+
+dotnet test tests/DotLLM.Tests.Integration/DotLLM.Tests.Integration.csproj \
+ --filter "FullyQualifiedName~LoadsAndForwardsEndToEnd"
+```
+
+Currently these dispatch to CPU (`deviceId: -1`). For CUDA, introduce a
+`_Cuda` sibling test per arch once the dispatch works, keeping the CPU
+versions as oracles.
+
+### 7. PyTorch reference gate
+
+The CPU path already matches HF 5/5 argmax on 4 architectures (Qwen / Phi
+/ Llama / DeepSeek-V2-Lite). The reference JSONs are at
+`tests/DotLLM.Tests.Integration/Models/Loaders/references/`. Mirror the
+`*_LogitsMatchPyTorchReference` pattern for CUDA — same JSON, run CUDA
+forward, diff. Target: `DriftTolerances.Tight` or looser only if observed
+BF16-on-GPU-vs-F32-on-CPU drift warrants it.
+
+### 8. Benchmark
+
+`benchmarks/DotLLM.Benchmarks/InferenceBenchmarks.cs` runs `prefill + decode`
+on SmolLM-135M / Llama-3.2-1B / Llama-3.2-3B via the default CPU path. Add a
+CUDA variant and compare. Baseline (CPU, this branch): see
+`PLANS.md` → P1.3 pre-W2 section (~234 prefill / 17 decode tok/s on CPU
+single-thread for SmolLM-135M Q8_0).
+
+### 9. Hard architecture gaps to close on CUDA
+
+These are structural — not bugs, just not-yet-done:
+
+- **MLA on CUDA.** The CPU Phase A/B/C `MlaAttention` kernels are the
+ numerical oracle (split-call matches single-call ≤ 1e-3 / 1e-4; F32+eager
+ vs HF 5/5 argmax on DeepSeek-V2-Lite). Plan a port mirroring vLLM's
+ MLA backend (prefill-expand / decode-absorbed split). The Phase B
+ latent-cache layout (`MlaLatentKvState`) maps naturally to GPU HBM
+ locality.
+- **MoE on CUDA.** DeepSeek multi-shared-expert + Qwen-MoE + Granite-fused
+ all need kernel ports. The Mixtral-style routing is the simplest first
+ target.
+- **Mamba-3 SSM on CUDA.** Selective scan + conv1d + MIMO — multi-month
+ effort; low priority unless users ask.
+
+### 10. Known secondary HF divergences (apply to both CPU and CUDA)
+
+Tracked in our `P2.6` follow-ups, not blocking but worth fixing as long-context coverage grows:
+
+1. **YaRN RoPE frequency rescaling** — parsed in `MlaConfig`, not applied.
+ Matters for prompts > `original_max_position_embeddings` (4096 on V2-Lite).
+ Active session at time of writing has an agent implementing this; check
+ latest commits on `feature/mamba-3` for a `P2.6 fix: YaRN RoPE
+ frequency rescaling` commit.
+2. **`routed_scaling_factor`** — parsed, ignored. V2-Lite=1.0 no-op; V2-full
+ and V3 use ≠1.0 with `norm_topk_prob=false`.
+3. **`topk_method=group_limited_greedy`** — V3-only. V2 uses `greedy`.
+
+### 11. Multi-GPU (NCCL)
+
+`docs/MULTI_GPU.md` has the design. Not started.
+
+## Contact points
+
+- `docs/CUDA.md` — everything about the PTX architecture.
+- `docs/GPU.md` — data-flow diagrams.
+- `CLAUDE.md` — project coding conventions.
+- `PLANS.md` — tracks outstanding work and past decisions.
+- `docs/KV_CACHE.md` — Phase A/B/C MLA cache rationale (applies to GPU too).
+- `docs/SUPPORTED_MODELS.md` — per-arch matrix with `Verified on` evidence.
+
+## Reciprocal — what runs on this Ryzen box
+
+### Backends verified on AMD Radeon 8060S iGPU (Strix Halo, gfx1151)
+
+| Backend | Status on this iGPU | Notes |
+|---|---|---|
+| **CPU** (dotLLM.Cpu) | ✅ works | Default path; AVX-512F+CD+BW+DQ+VL+VBMI active at runtime. |
+| **Vulkan** (dotLLM.Vulkan) | ✅ Add kernel verified end-to-end | Full LLM kernels not yet ported. Device reports `fp16: 1`, `bf16: 0`, `KHR_coopmat` matrix cores. |
+| **HIP/ROCm** (dotLLM.Hip) | ⚠️ builds + initialises, `hipModuleLoadData` fails with `hipErrorInvalidKernelFile` | Known issue with gfx1151 iGPU on ROCm 7.1 Windows. Expected to work on discrete AMD GPUs (gfx1030/gfx1100). |
+| **CUDA** (dotLLM.Cuda) | ❌ no NVIDIA GPU | That's why you exist. |
+
+### Cross-library benchmarks (SmolLM-135M Q8_0, llama.cpp build d0a6dfeb2)
+
+Reference numbers captured on the Ryzen + Radeon 8060S box for
+cross-validation when the CUDA port is ready. Prompt sizes in tokens:
+`pp` = prompt processing (prefill), `tg` = token generation (decode).
+
+| Backend | pp128 (tok/s) | pp512 (tok/s) | tg128 (tok/s) |
+|----------------------|--------------:|--------------:|--------------:|
+| **dotLLM CPU** (PLANS.md P1.3, SmolLM Q8_0, 20-tok decode) | ~234 | — | **~17** |
+| llama.cpp `-ngl 0` | 1465 ± 155 | 2897 ± 640 | 109 ± 10 |
+| llama.cpp Vulkan `-ngl 99` (iGPU) | 15096 ± 689 | 25205 ± 3040 | **392 ± 45** |
+
+Key gaps for the CPU path:
+- dotLLM CPU decode is **~6.4× slower** than llama.cpp CPU decode.
+- dotLLM CPU decode is **~23× slower** than llama.cpp Vulkan iGPU decode.
+
+Candidate causes on our CPU path (tracked, not yet root-caused):
+1. Single-thread by default; `ThreadingConfig.Parallel` exists but the
+ `InferenceBenchmarks` run single-thread for stability.
+2. No fused quantized-matmul (we dequantize → GEMM; llama.cpp MMQ fuses).
+3. No SIMD in MLA (fixed in `c3f1c21` for MLA but not yet for the full
+ GQA attention — though `Attention.cs` already uses `TensorPrimitives`).
+4. Sampler overhead — llama.cpp's simple sampler is sub-microsecond;
+ ours has more abstraction layers.
+
+A CUDA comparison on an NVIDIA box should target **≥ llama.cpp's Vulkan
+iGPU numbers** as a floor. Real CUDA on discrete Ampere/Ada/Hopper
+should substantially exceed it.
+
+### What's happening on the Ryzen box while you work
+
+- **Vulkan backend** (`src/DotLLM.Vulkan/`) — working end-to-end on AMD
+ Radeon 8060S iGPU (`AddKernel_ProducesElementwiseSum` passes). Scope
+ is proof-of-pipeline; LLM-kernel port is the next task here.
+- **HIP backend** (`src/DotLLM.Hip/`) — builds and initialises on Strix Halo
+ iGPU (gfx1151) but `hipModuleLoadData` returns `hipErrorInvalidKernelFile`
+ on ROCm 7.1 + iGPU; test skips cleanly. Expected to work on discrete AMD
+ GPUs (gfx1030 / gfx1100) — validation left for a machine with one.
+- **CPU kernel work** — MLA vectorisation, long-context tests, benchmark
+ comparisons vs llama.cpp / Ollama (which already has a harness at
+ `C:/Development/OllamaBenchmarks/` per the parent-dir scan).
+
+## When you sync back
+
+The two tracks diverge cleanly. The Ryzen box touches
+`src/DotLLM.Vulkan/`, `src/DotLLM.Hip/`, `src/DotLLM.Cpu/Kernels/*`,
+`src/DotLLM.Models/Architectures/Mla*.cs`, `tests/DotLLM.Tests.Unit/**`,
+`tests/scripts/compare_logits_py_reference.py`. The CUDA box touches
+`src/DotLLM.Cuda/`, `native/kernels/*.cu`, `native/build.*`,
+`tests/DotLLM.Tests.Unit/Cuda/*`, `docs/CUDA.md`, `docs/GPU.md`. Minor
+rebase expected on `dotLLM.slnx` project list; other than that the trees
+should merge clean.
diff --git a/DESIGN.md b/DESIGN.md
new file mode 100644
index 00000000..121cb041
--- /dev/null
+++ b/DESIGN.md
@@ -0,0 +1,269 @@
+# DESIGN — Nemotron-3 (`nemotron_h`) support in dotLLM
+
+Branch: `feature/nemotron-and-mamba-2` (worktree: `C:/Development/dotLLM-mamba3`).
+Target model: `NVIDIA-Nemotron-3-Nano-4B-Q4_K_M.gguf` (2.7 GB).
+Goal: load and generate tokens with the dotLLM CPU backend.
+
+> **Note on scope.** This work was started under a branch originally named
+> `feature/mamba-3` with the intent of implementing the **Mamba-3** algorithm
+> (Lahoti et al., [arXiv 2603.15569](https://hf.co/papers/2603.15569),
+> Mar 2026 — complex-valued state updates, MIMO formulation). The plan below
+> was then (wrongly) written against the **Mamba-2**-based `nemotron_h`
+> hybrid architecture that NVIDIA ships in every Nemotron-3 family member
+> (Nano 4B / Nano 30B-A3B MoE / Super / Ultra). The "3" in Nemotron-3 is
+> the model family version, not the Mamba generation. Having already built
+> out most of this path, we finish the Mamba-2 / `nemotron_h` implementation
+> on the renamed branch and land it, *then* open a fresh `feature/mamba-3`
+> for the actual Mamba-3 algorithm as a separate workstream.
+
+## 1. Architecture summary
+
+Nemotron-3 (NVIDIA) is a **hybrid SSM + Transformer** language model. The
+`nemotron_h` GGUF architecture string identifies it. Each of the model's
+`block_count` layers is one of three exclusive types:
+
+```
+embed -> N x [ attn_norm -> { SSM | Attention | FFN } -> residual ] -> output_norm -> lm_head
+```
+
+This is unlike a standard Llama/Qwen block where attention **and** an FFN
+both run with two residuals; here each layer applies a *single* sub-layer
+with one residual add (mirroring llama.cpp `src/models/nemotron-h.cpp`).
+
+The three sub-layer flavours, in order from most novel to least:
+
+| Type | Sub-layer body (after RMSNorm) |
+|-----------|-----------------------------------------------------------------------------------------------------------------------------------|
+| SSM | linear `ssm_in` -> split (z, x, B, C, dt) -> conv1d+SiLU on (x|B|C) -> selective-scan -> +D*x -> SwiGLU(z, y) -> group-RMSNorm -> linear `ssm_out` |
+| Attention | linear Q/K/V (no bias) -> RoPE on first 78 dims of head -> GQA attention -> linear `attn_output` |
+| FFN | linear `ffn_up` -> ReLU^2 -> linear `ffn_down` |
+
+There is **no SwiGLU MLP** (no `ffn_gate.weight`). The FFN is a parallel
+non-gated MLP with squared-ReLU activation.
+
+## 2. Target GGUF inventory (NVIDIA-Nemotron-3-Nano-4B-Q4_K_M)
+
+Metadata extracted via `dotnet run --project src/DotLLM.Cli -c Debug -- debug gguf-metadata `:
+
+```
+general.architecture = "nemotron_h"
+nemotron_h.block_count = 42
+nemotron_h.embedding_length = 3136
+nemotron_h.attention.head_count = 40
+nemotron_h.attention.key_length = 128
+nemotron_h.attention.value_length = 128
+nemotron_h.attention.head_count_kv = INT32[42] <- per-layer, 0 means non-attn layer
+nemotron_h.feed_forward_length = INT32[42] <- per-layer, 0 means non-FFN layer
+nemotron_h.attention.layer_norm_rms_epsilon = 1e-5
+nemotron_h.context_length = 1048576 (1 M)
+nemotron_h.rope.dimension_count = 78 (partial RoPE, only first 78 of head_dim=128)
+nemotron_h.ssm.conv_kernel = 4 (d_conv)
+nemotron_h.ssm.group_count = 8 (n_group)
+nemotron_h.ssm.inner_size = 7680 (d_inner)
+nemotron_h.ssm.state_size = 128 (d_state)
+nemotron_h.ssm.time_step_rank = 96 (n_head for Mamba2 — heads of size d_inner/n_head = 80)
+nemotron_h.vocab_size = 131072
+tokenizer.ggml.model = "gpt2" (BPE)
+```
+
+Per-layer tensor inventory (sample from `debug gguf-tensors`):
+
+* SSM layers (21x: blk 0,2,4,6,7,9,11,14,16,19,21,23,26,28,30,31,34,35,36,38,40):
+ - `attn_norm.weight [3136] F32`
+ - `ssm_in.weight [3136, 17504] Q5_0` (17504 = 2*7680 + 2*8*128 + 96)
+ - `ssm_conv1d.weight [4, 9728] F32` (9728 = 7680 + 2*8*128)
+ - `ssm_conv1d.bias [9728] F32`
+ - `ssm_a [1, 96] F32` (Mamba2 scalar A per head)
+ - `ssm_d [1, 96] F32` (per-head skip gain)
+ - `ssm_dt.bias [96] F32`
+ - `ssm_norm.weight [960, 8] F32` (group RMSNorm, d_inner/n_group=960, n_group=8)
+ - `ssm_out.weight [7680, 3136] Q4_K`
+* Attention layers (4x: blk 12,17,24,32):
+ - `attn_norm.weight [3136] F32`
+ - `attn_q.weight [3136, 5120] Q5_0` (40 heads x 128)
+ - `attn_k.weight [3136, 1024] Q5_0` (8 KV heads x 128)
+ - `attn_v.weight [3136, 1024] Q5_0/Q8_0`
+ - `attn_output.weight [5120, 3136] Q4_K`
+* FFN layers (17x: blk 1,3,5,8,10,13,15,18,20,22,25,27,29,33,37,39,41):
+ - `attn_norm.weight [3136] F32` (named attn_norm but is the FFN pre-norm)
+ - `ffn_up.weight [3136, 12544] Q5_0`
+ - `ffn_down.weight [12544, 3136] Q4_K/Q6_K`
+ - **no `ffn_gate.weight`**
+* Global:
+ - `token_embd.weight [3136, 131072] Q5_0`
+ - `output_norm.weight [3136] F32`
+ - `output.weight [3136, 131072] Q8_0`
+
+Layer-type discriminator (matches llama.cpp `is_recurrent(il)` / `n_ff(il) == 0`):
+
+```
+if tensors contain blk.{i}.ssm_in.weight -> SSM layer
+elif tensors contain blk.{i}.attn_q.weight -> Attention layer
+elif tensors contain blk.{i}.ffn_up.weight -> FFN layer
+else error
+```
+
+## 3. dotLLM gap analysis
+
+What dotLLM has that we can reuse as-is:
+* `GgufFile.Open` / `GgufMetadata` / `GgufTensorDescriptor`
+* `RmsNorm`, `Attention`, `RoPE`, `SiLU`, `Add`, `Multiply` kernels (CPU)
+* `MatMul` GEMM/GEMV for Q4_K, Q5_0, Q5_K, Q6_K, Q8_0, F16, F32
+* `WeightRepacking` R4 interleave
+* `BpeTokenizer` (gpt2 pre-tokenizer regex is needed; verify already supported)
+* `TextGenerator`, KV cache (works for the attention layers we keep)
+* `IModel` / `ITensor` interfaces
+
+What is missing or needs change:
+1. **Architecture enum:** `Architecture.NemotronH` (extend `DotLLM.Core.Configuration.Architecture`).
+2. **Config extraction:** `GgufModelConfigExtractor.ParseArchitecture` recognises `nemotron_h`. We also need to parse the **array** metadata (`head_count_kv`, `feed_forward_length`) and the SSM hyperparameters into a new `MambaSsmConfig` record bag. `GgufMetadata` does not yet expose `int[]` arrays of length-N per-layer counts — `GetInt32Array` covers it because the underlying `GgufValueType.Array` already stores `int[]`. Good. Need to add an `int[]?` field to `ModelConfig` (or a side record `HybridLayerLayout`) for per-layer head_count_kv and ffn dim, plus the SSM dims.
+3. **Layer-type tag:** A `LayerKind` enum (`Ssm | Attention | Ffn`) computed at load time from tensor presence (preferred over array-zero check, more robust).
+4. **New CPU kernels:**
+ a. `Conv1dCausal` — depthwise causal 1D conv with kernel size 4, applied per channel along the time dim (input shape `[d_conv-1 + n_seq_tokens, channels]`). Bias add and SiLU follow.
+ b. `Mamba2SelectiveScan` — the recurrent state-space scan (see math below). Pure scalar reference first, vectorise later.
+ c. `ReluSquared` activation (one-line: `y = ReLU(x); y *= y`). Trivial.
+ d. **Group RMSNorm** — RMSNorm applied independently to each of n_group sub-vectors of length `d_inner/n_group = 960`. Reuse existing `RmsNorm.Execute` in a loop; weight tensor `ssm_norm.weight` has shape `[960, 8]` and is broadcast per group.
+5. **Recurrent state buffer (SSM cache):** Two per-layer tensors per sequence:
+ * `conv_state` shape `[d_conv-1, d_inner + 2*n_group*d_state]` = `[3, 9728]`
+ * `ssm_state` shape `[d_state, head_dim, n_head]` = `[128, 80, 96]` = ~3.9M floats = ~15 MB per SSM layer per sequence (large!). For 21 SSM layers that is ~315 MB; fine for a 4B model with 2.7 GB weights.
+ The current `IKvCache` in dotLLM is attention-only. We add a parallel `ISsmStateCache` (or a unified `IRecurrentStateCache`) that lives next to the KV cache. For the first cut: a non-paged `SsmStateCache` allocated in unmanaged memory, one per generation request, sized to the model's SSM-layer count.
+6. **Hybrid block dispatch:** A new `NemotronHTransformerModel : IModel` with its own `Forward()` that walks layers consulting the per-layer `LayerKind` and dispatches to one of three sub-layer routines. Cannot reuse `TransformerModel` directly because:
+ * its forward assumes attn+FFN per layer with two residuals,
+ * it pre-allocates Q/K/V/Gate/Up scratch globally,
+ * it has no place for SSM tensors or recurrent state.
+ A clean separate type is the right call. We should still factor shared helpers (`EmbeddingLookup`, `GemmInterleaved`, `QuantizeInput`) so we do not duplicate the dispatch table.
+7. **TransformerArchitecture factory:** add `Architecture.NemotronH` -> `NemotronHTransformerModel.LoadFromGguf` switch.
+8. **BPE tokenizer pre-tokenizer:** metadata says `tokenizer.ggml.pre = "pixtral"`. Need to check whether `GgufBpeTokenizerFactory` knows this regex; fall back to gpt2 default if not. (Out-of-scope concern — log a TODO.)
+9. **RoPE partial dimensions:** `rope.dimension_count = 78` means only the first 78 of each 128-dim head get RoPE; the rest are unrotated. The existing `RoPE.Execute` already takes a `ropeDim` parameter (`_ropeDim`). Confirm it implements the partial case correctly — it appears to.
+
+## 4. Mamba2 forward maths (CPU reference)
+
+Naming below matches the GGUF tensors. All shapes assume one sequence
+(`n_seqs = 1`), `T = n_seq_tokens` in the current step.
+
+Input `x_in` shape `[T, d_model]` where `d_model = 3136`.
+
+```
+# 1. Big input projection: hidden -> z, x, B, C, dt
+zxBCdt = x_in @ ssm_in # [T, d_in_proj] where d_in_proj = 17504
+ # = 2*d_inner + 2*n_group*d_state + n_head
+ # d_inner = 7680, n_group = 8, d_state = 128, n_head = 96
+# Layout along d_in_proj:
+# [0 .. d_inner) -> z (gate)
+# [d_inner .. 2*d_inner) -> x_part of xBC
+# [2*d_inner .. 2*d_inner + n_group*d_state) -> B
+# [... .. 2*d_inner + 2*n_group*d_state) -> C
+# [... .. + n_head) -> dt
+xBC = concat(x_part, B, C) # [T, d_inner + 2*n_group*d_state] = [T, 9728]
+
+# 2. Causal depthwise 1D conv over last d_conv-1 cached steps + new T steps
+conv_input = concat(conv_state, xBC^T) # along time, [d_conv-1+T, 9728]
+for ch in 0..9728:
+ for t in 0..T:
+ y[t, ch] = sum_{k in 0..4} conv_input[t+k, ch] * conv1d_w[k, ch]
+xBC = SiLU(y + conv1d_bias) # [T, 9728]
+conv_state = last (d_conv-1) rows of conv_input # [3, 9728]
+
+# 3. Re-split xBC into x, B, C (in-place views)
+x = xBC[:, 0..d_inner] # [T, 7680]
+B = xBC[:, d_inner..d_inner+n_group*d_state] # [T, 8*128]
+C = xBC[:, d_inner+n_group*d_state..end] # [T, 8*128]
+
+# 4. dt with bias
+dt = dt + ssm_dt_b # [T, n_head]
+
+# 5. Selective scan (THE recurrent loop). Groups: each head h is in group g = h / (n_head/n_group).
+# State shape: [n_head, head_dim, d_state] = [96, 80, 128].
+A = ssm_a # [n_head] (scalar per head)
+D = ssm_d # [n_head] (scalar per head)
+for t in 0..T:
+ for h in 0..n_head:
+ dt_sp = softplus(dt[t, h])
+ dA = exp(dt_sp * A[h])
+ g = h / (n_head / n_group)
+ for i in 0..head_dim: # head_dim = 80
+ x_dt = x[t, h*head_dim + i] * dt_sp
+ sumf = 0
+ for k in 0..d_state: # d_state = 128
+ state[h, i, k] = state[h, i, k] * dA + B[t, g*d_state + k] * x_dt
+ sumf += state[h, i, k] * C[t, g*d_state + k]
+ y[t, h*head_dim + i] = sumf
+
+# 6. Skip connection then SwiGLU gating with z
+y = y + x * D_per_head_broadcast # broadcast D across head_dim
+y = SiLU(z) * y # SwiGLU (note: gating param is z, not y)
+
+# 7. Group RMSNorm: split y [T, d_inner] -> [T, n_group, d_inner/n_group], per-group RMSNorm with ssm_norm
+y = group_rms_norm(y, ssm_norm) # [T, 7680]
+
+# 8. Output projection back to model width
+out = y @ ssm_out # [T, 3136]
+```
+
+Per Mamba2 paper: A is parameterised in log-space and `ggml_ssm_scan`
+applies `exp(dt_sp * A[h])` directly (no negation). The GGUF converter
+already stores `A` such that `A[h] < 0`, so `exp(dt_sp * A[h])` decays.
+We mirror llama.cpp byte-for-byte to avoid sign confusion.
+
+The expensive inner loop is `n_head * head_dim * d_state = 96 * 80 * 128 = 983,040`
+fused multiply-adds per token per layer, plus a second pass for the dot with C
+(or both fused, as llama.cpp does). For 21 SSM layers and a 1-token decode:
+~21 M FMAs per token. Q4_K weight matmuls dominate compute regardless.
+
+## 5. Staged implementation plan
+
+Each stage is a separate commit on `feature/mamba-3`. After every stage:
+`dotnet build src/dotLLM.sln -c Release` MUST pass before commit.
+
+1. **DESIGN.md** (this file). Commit first.
+2. **Architecture enum + ParseArchitecture + ModelConfig fields**
+ * Add `Architecture.NemotronH`
+ * Extend `ModelConfig` with optional `HybridLayoutConfig?` (per-layer `LayerKind[]`, per-layer `int[] HeadCountKv`, per-layer `int[] FeedForwardLength`) and `MambaSsmConfig?` (d_conv, d_inner, d_state, n_group, n_head/dt_rank).
+ * Update `GgufModelConfigExtractor.Extract` to populate them when `nemotron_h`.
+ * Defensive: existing transformer paths must still be unchanged for llama/qwen/...
+ * Verify against `dotnet run -- debug gguf-config ` (need to extend the CLI's debug command to print the new fields, optional).
+3. **NemotronHTransformerModel skeleton + loader**
+ * Walk all 42 blocks, classify by tensor presence, build per-layer weight bundles (`SsmLayerWeights`, `AttnLayerWeights`, `FfnLayerWeights`).
+ * Allocate scratch and recurrent state.
+ * `Forward()` returns logits when **all** layers are SSM-only or trivial — for now throws `NotImplementedException("Mamba2 selective scan not yet implemented")` with a clear pointer to step 5.
+ * Wire `TransformerArchitecture.SupportedArchitectures` and `CreateModel` to dispatch to it.
+ * Update `ModelLoader.LoadFromGguf` to dispatch to the new model class for `Architecture.NemotronH`.
+ * Result: `dotllm model-info ` no longer throws on architecture, but `dotllm run` throws a clean NotImplementedException.
+4. **Squared ReLU activation kernel** (small, easy)
+ * `DotLLM.Cpu/Kernels/ReluSquared.cs`. SIMD via `TensorPrimitives.Max` then `TensorPrimitives.Multiply`. Smoke test against scalar reference.
+5. **FFN sub-layer forward (no SSM, no attention yet)**
+ * Wire the 17 FFN layers in `NemotronHTransformerModel.Forward`.
+ * Path: RmsNorm -> ffn_up GEMM -> ReluSquared -> ffn_down GEMM -> residual.
+ * Still throws on first SSM layer encountered; useful for catching layout bugs in isolation.
+6. **Attention sub-layer forward**
+ * Reuse existing `Attention.Execute`, `RoPE.Execute`, `MatMul` GEMMs.
+ * Biases: none for nemotron-3.
+ * Per-layer KV cache slot — but only for the 4 attention layers; allocate a sparse KV cache that has entries only for those layer indices.
+7. **Mamba2 SSM forward**
+ a. `Conv1dCausal` kernel (depthwise, kernel size 4, channels = 9728). Scalar fallback first; SIMD later.
+ b. `Mamba2SelectiveScan` kernel — direct port of `ggml_compute_forward_ssm_scan_f32` (Mamba-2 branch where `src3->ne[0] == 1`).
+ c. Group RMSNorm helper (reuse RmsNorm in a loop initially).
+ d. SsmStateCache (unmanaged, aligned).
+ e. Wire SSM dispatch in `NemotronHTransformerModel.Forward`.
+ f. Numerical validation against llama.cpp logits (capture a few prompt tokens in llama.cpp with `--logit-bias` debug).
+8. **Smoke test** `tests/DotLLM.Tests.Integration/Engine/NemotronHTextGeneratorTests.cs`:
+ * Loads model, generates 5 greedy tokens, asserts non-empty and non-NaN logits.
+ * Skipped (or marked `Trait("RequiresModel")`) when GGUF is absent.
+
+## 6. Risks and notes
+
+* **SSM math correctness is fragile.** Tiny errors in head ordering, group repeat-interleave, or A sign convention will produce gibberish that *looks* like normal generation. Mitigate by capturing intermediate tensors (z, x_post_conv, y_pre_norm, y_post_norm) for token 0 of "Hello" in llama.cpp and comparing to ours element-wise within 1e-3.
+* **Performance (later).** First cut is scalar; fine. AVX2 vectorisation of the inner `d_state=128` loop is straightforward (512-byte rows = 4 YMM registers). The conv1d and selective-scan are memory-bound with small per-channel work, so a thread-per-head-block parallelisation across `n_head=96` should scale well on 16-32 cores.
+* **Memory:** 21 SSM layers x (3 * 9728 * 4B conv-state + 96*80*128 * 4B ssm-state) ~= 21 * (117 KB + 3.9 MB) ~= 84 MB recurrent state per sequence. Reasonable.
+* **Tokenizer pre-tokenizer:** `tokenizer.ggml.pre = "pixtral"`. Must confirm the BPE tokenizer factory accepts that pre-key (otherwise tokens off by one will completely corrupt generation). We should log it loudly during loading and fall back to gpt2 split if unknown — the actual byte-level merges still apply.
+* **Partial RoPE (rope_dim=78 of head_dim=128):** verify dotLLM's RoPE applies to first `ropeDim` dims of each head and leaves the rest untouched. From reading `TransformerModel.Forward` and `RoPE.Execute(... ropeDim ...)`, this looks correct. Add an assertion at load time that the four attention layers have `head_dim == 128` and `ropeDim == 78`.
+* **Tied embeddings:** `output.weight` is present (Q8_0), separate from `token_embd.weight` (Q5_0). No tying.
+* **MoE variant** `nemotron_h_moe` is **not** in scope; raise a clear error if encountered.
+* **Build:** all Mamba2 work is CPU-only. No CUDA, no native lib changes. The CUDA backend already ignores unsupported architectures via `TransformerArchitecture` filter.
+
+## 7. Done criteria
+
+* `dotllm run NVIDIA-Nemotron-3-Nano-4B-GGUF` (or equivalent path) produces text without throwing.
+* Greedy "The capital of France is" produces a sensible continuation containing "Paris" within the first 10 tokens (or, failing that, produces deterministic non-NaN logits whose argmax is in the vocab range).
+* All existing tests still green.
+* If we ship before step 7 (SSM forward) is done, the loader works end-to-end and the forward throws a clean `NotImplementedException("Mamba2 selective scan not yet implemented (stage 7 of feature/mamba-3)")` from a single named call site so a future agent can pick up exactly where we left off.
diff --git a/DESIGN_MAMBA_3.md b/DESIGN_MAMBA_3.md
new file mode 100644
index 00000000..62dd0525
--- /dev/null
+++ b/DESIGN_MAMBA_3.md
@@ -0,0 +1,289 @@
+# DESIGN — Mamba-3 support in dotLLM
+
+Branch: `feature/mamba-3` (this is the *real* one, for the Mamba-3 algorithm
+from Lahoti et al., [arXiv 2603.15569](https://arxiv.org/abs/2603.15569),
+published 16 Mar 2026, accepted to ICLR 2026).
+
+> **Scope difference from `feature/nemotron-and-mamba-2`.** That branch
+> implemented the Mamba-2 hybrid architecture used by NVIDIA's Nemotron-3
+> family. Mamba-3 is a distinct, newer algorithm that **cannot be validated
+> the same way**: there are no GGUF checkpoints on HuggingFace, no
+> llama.cpp support, and the only known reference implementation is
+> [VikramKarLex/mamba3-minimal](https://github.com/VikramKarLex/mamba3-minimal)
+> (minimal PyTorch, ~800 lines, endorsed by Albert Gu) plus the
+> unreleased authors' integration into `state-spaces/mamba`. The realistic
+> first deliverable here is **kernel-level parity with the minimal PyTorch
+> reference**, not end-to-end inference.
+
+## 1. What's new in Mamba-3
+
+Three core methodological improvements over Mamba-2, expressed concretely:
+
+| # | Change | Effect |
+|---|---|---|
+| 1 | Trapezoidal discretization (Prop. 1 of the paper) | Adds a second input term to the recurrence; removes the need for conv1d |
+| 2 | Complex-valued state update (applied as data-dependent RoPE on B, C) | Richer state tracking; no explicit complex arithmetic needed |
+| 3 | MIMO (multi-input multi-output) formulation | Half the state size for equivalent perplexity; 2× faster decode |
+
+Plus several architectural refinements (QK-normalization on B/C, learnable
+BC bias, removal of input-side gating).
+
+### 1.1 Trapezoidal recurrence (Eq. 9)
+
+```
+h_t = α_t · h_{t-1} + β_t · B̄_{t-1} · x_{t-1} + γ_t · B̄_t · x_t
+```
+
+where, per-token-per-head:
+
+```
+α_t = exp(dt · A) # decay, same as Mamba-2
+β_t = (1 - λ) · dt · α_t # previous-step coefficient (NEW)
+γ_t = λ · dt # current-step coefficient
+λ = sigmoid(λ_raw) # trapezoidal interpolation param, per-head
+```
+
+Vs Mamba-2's `h_t = α · h_{t-1} + B̄_t · x_t` — Mamba-3 weights the
+*previous* input as well as the current, giving a second-order
+integration rule. This also **eliminates the depthwise conv1d** that
+Mamba-2 uses — the β-term provides the cross-token memory that the conv
+was emulating.
+
+### 1.2 Complex state via RoPE on B, C
+
+The complex-valued update rule is implemented as a data-dependent
+rotation applied to B and C, rather than explicit complex arithmetic.
+Per-head rotation angles are cumulative:
+
+```
+raw_angles[t, h, :] = dt[t, h] · θ[h, :] # θ is learned, shape (n_heads, d_state/2)
+cum_angles = -cumsum(raw_angles, dim=time)
+B, C = rope_2d(B, C, cum_angles) # 2D rotation per adjacent d_state pair
+```
+
+The 2D rotation on pairs of state dimensions absorbs what would
+otherwise be a complex-valued state update — each pair `(x[2k], x[2k+1])`
+rotates as `(cos·x[2k] - sin·x[2k+1], sin·x[2k] + cos·x[2k+1])`.
+
+### 1.3 MIMO variant
+
+When enabled, B and C become rank-R matrices (shape `(d_state, R)`) and
+the SSM input is rank-expanded before the scan:
+
+```
+# SISO (Mamba-2 style): BX = outer(B, x) → (P, d_state)
+# MIMO (rank-R factored): x_exp = x ⊙ mimo_x_proj[R,P]
+ BX = B @ x_exp.T → (P, d_state)
+```
+
+Same scan kernel; different B/C/x shapes at the input side.
+
+### 1.4 Two-SSD decomposition
+
+Implementation trick from the minimal reference: the trapezoidal update
+is run as **two separate SSD calls** (one for the γ-term using current
+`B_t, x_t`, one for the β-term using previous `B_{t-1}, x_{t-1}`), then
+summed. This lets Mamba-3 reuse an unchanged Mamba-2 SSD kernel instead
+of needing a custom two-input recurrence kernel. **We can do the same.**
+
+## 2. Ecosystem status (as of 2026-04-18)
+
+| Item | Status |
+|---|---|
+| Paper | ICLR 2026, arXiv 2603.15569, Mar 2026 |
+| Official code | `state-spaces/mamba` repo (PyTorch) — production impl with Triton kernels |
+| Minimal reference | `VikramKarLex/mamba3-minimal` — ~800 lines pure PyTorch, Albert Gu-endorsed |
+| Alternative minimal | `yang3121099/mamba3` — another port (need to verify authenticity) |
+| HuggingFace checkpoints | **None.** `state-spaces/*` org hosts Mamba-1/Mamba-2 but no Mamba-3 yet |
+| GGUF converter | **None.** No `tensor_mapping.py` entries for Mamba-3 tensors in llama.cpp |
+| llama.cpp arch support | **None** in current `master`. No `LLM_ARCH_MAMBA3` or equivalent |
+
+This means **we cannot do the llama.cpp cross-reference validation that
+the Nemotron-H work relied on**. Our options for correctness validation:
+
+1. **Port tests against the minimal PyTorch reference.** Fix a random
+ seed, run Mamba-3 forward in PyTorch on synthetic inputs, save the
+ intermediate tensors to a `.npz` / `.bin` file, then have our C#
+ unit tests load and compare against those fixtures. Feasible today.
+2. **Wait for an official checkpoint + GGUF converter.** No known ETA.
+3. **Port the state-spaces/mamba Triton kernels semantically and trust
+ the reference.** Risky — complex math, easy to drift.
+
+We use **option 1** for initial kernel work.
+
+## 3. What dotLLM needs
+
+### 3.1 Reuseable from `feature/nemotron-and-mamba-2`
+
+All already merged on the base branch:
+* `Mamba2SelectiveScan.Execute` — the inner-loop scan. **Mamba-3 calls
+ this twice per layer** (γ and β decomposition), so it ships as-is.
+* `SsmStateCache` — generalizes. Mamba-3 additionally needs a
+ `prev_Bx` buffer per head, which is just another same-shape slot.
+* `RmsNorm` — used for QK-normalization on B, C.
+* `NemotronHForwardState` scratch pool pattern — reuse conceptually, not
+ literally. A fresh `Mamba3ForwardState` is cleaner.
+* `NemotronHDiagnostics` — the env-var-gated trace pattern applies as-is.
+
+### 3.2 New kernels
+
+| Kernel | Purpose | Complexity |
+|---|---|---|
+| `Mamba3Discretize` | Compute α, β, γ per token-head from dt, A, λ | Simple element-wise |
+| `Mamba3DataRoPE` | 2D rotation on B and C using cumulative angles derived from dt and θ | Like standard RoPE but data-dependent (not position-based); per-pair rotation |
+| `Mamba3QkNorm` | RMSNorm on B and C vectors | Trivial wrapper |
+| `Mamba3MimoProject` *(MIMO variant)* | Rank-R expansion of x into SSM input; rank-R contraction of y | Matmul-shaped |
+| `Mamba3SelectiveScan` | Orchestrator: two-SSD decomposition calling existing `Mamba2SelectiveScan` twice + sum | Thin wrapper |
+
+### 3.3 No new architecture wiring *yet*
+
+Until a checkpoint + GGUF format materialize, there is nothing to load
+end-to-end. The work here is:
+1. Kernels (stage A, below)
+2. PyTorch reference-fixture test harness (stage B)
+3. Mamba-3 block class that composes kernels the way the minimal
+ reference does — not wired into `ModelLoader` since there are no
+ models to load (stage C)
+
+When checkpoints + GGUF support land, a follow-up branch adds the
+`Architecture.Mamba3` enum, `Mamba3TransformerModel`, and the
+`ModelLoader` dispatch — mirroring what Stage 3 of the Nemotron-H plan did.
+
+## 4. Staged plan
+
+Commit-granular; each stage builds green and adds tests.
+
+### Stage A — Kernels, each validated against a scalar Python reference
+
+1. **`Mamba3Discretize`** — compute α, β, γ from dt, A, λ per (t, h).
+ Unit tests: scalar reference; a λ=1 sanity check (recovers
+ Mamba-2's first-order discretization when β→0, γ→dt).
+
+2. **`Mamba3DataRoPE`** — data-dependent 2D rotation on a
+ `[T, n_group, d_state]` tensor using angles derived from
+ `cumsum(-dt[:, h] * θ[h, :])`. Unit tests: identity when θ=0;
+ recoverable rotation when θ is piecewise constant.
+
+3. **`Mamba3QkNorm`** — trivial, existing `RmsNorm.Execute` in a loop;
+ wrap for clarity only.
+
+4. **`Mamba3SelectiveScan`** — orchestrator that:
+ - computes α, β, γ (kernel 1)
+ - data-RoPE's B, C (kernel 2)
+ - calls `Mamba2SelectiveScan` twice (once for γ-path with `B_t · x_t`,
+ once for β-path with `B_{t-1} · x_{t-1}`)
+ - sums the two outputs
+ Unit tests: tiny hand-computed example (n_head=1, d_state=2,
+ head_dim=2, T=3); matches a pure-scalar Python reference within 1e-4.
+
+5. *(MIMO variant)* — `Mamba3MimoProject` + `Mamba3SelectiveScan`
+ `useMimo=true` path. Unit tests: rank-1 MIMO degenerates to SISO.
+
+### Stage B — PyTorch reference fixtures
+
+**Schema decisions (derived from reading `.mamba3-reference/mamba3-minimal/mamba3.py`):**
+
+The Mamba-3 block input projection produces seven splits, not five:
+`z, x, B, C, dt_raw, lam_raw, theta` — where `theta` (shape `[T, dState/2]`)
+is **per-token data-dependent**, not a learned per-head table. `dt` is then
+`softplus(dt_raw + dt_bias)`, `lam` is `sigmoid(lam_raw)`. `A` comes from
+`-exp(A_log)` (per-head, always negative). BC-bias is learned per-head,
+per-channel (init 1.0), added *after* QK-Norm, *before* RoPE. For SISO the
+scan input is `B` shape `[T, nHead, dState]` (post-bias-broadcast) and our
+`Mamba3SelectiveScan` accepts `nGroup=nHead` in that layout.
+
+**Fixture capture points for the Stage B golden file:**
+1. Inputs: `u` (post-pre-norm hidden), all weights (`in_proj`, `A_log`,
+ `dt_bias`, `B_bias`, `C_bias`, `D`, `out_proj`, `B_norm.weight`,
+ `C_norm.weight`).
+2. Post-split intermediates: `z, x, B_raw, C_raw, dt_raw, lam_raw, theta`.
+3. Post-activation: `dt` (softplus), `lam` (sigmoid).
+4. Computed discretization: `dA, alpha, beta, gamma`.
+5. Post-QK-Norm: `B_qkn, C_qkn`.
+6. Post-BC-bias add (per-head broadcast): `B_biased, C_biased`.
+7. Cum angles: `cum_angles` shape `[T, nHead, dState/2]`.
+8. Post-RoPE: `B_roped, C_roped` shape `[T, nHead, dState]`.
+9. Post-scan: `y_scan` (pre-D), `ssm_state`, `last_Bx`.
+10. Post-D + gate: `y_gated` (after `y + x*D` and `y * silu(z)`).
+
+**Format:** single `fixture.json` with tensor name → flat F32 array +
+shape tuple. Simple for C# to parse (no NPZ/zip machinery). Config:
+tiny model (d_model=8, nheads=2, headdim=4, dState=4, seqLen=4, SISO
+first then MIMO R=2).
+
+
+
+Script (Python) that runs `VikramKarLex/mamba3-minimal` on a fixed seed
+and dumps:
+- A single-layer forward's input (hidden state, weights) as `.bin` files
+- Intermediate tensors after each step (discretize output, post-RoPE
+ B/C, scan output, final y) as `.bin` files
+
+Corresponding C# fixture loader + integration test that reads the
+.bin files, feeds the input into our kernels, and asserts byte-equal
+outputs to 4 decimal places. This is the direct analogue of what we did
+for Mamba-2 via llama-eval-callback, just with a PyTorch reference
+instead of a C reference.
+
+### Stage C — `Mamba3Block` orchestrator (no model loading)
+
+A `Mamba3Block` class that composes the kernels in the order the
+minimal reference does:
+
+```
+ RmsNorm → input-projection (GEMM) → split (z, x, B, C, dt, λ, θ)
+ → QK-Norm(B, C) → + BC-bias → DataRoPE(B, C)
+ → SelectiveScan → y·SiLU(z) → RmsNorm → output-projection
+```
+
+Followed by a separate `SwiGLU-MLP` block that's just the existing
+FFN wiring. No `Mamba3TransformerModel`, no GGUF loading — this is a
+composable primitive, validated via a third PyTorch fixture test that
+runs a full layer.
+
+### Stage D (deferred) — when a checkpoint exists
+
+- `Architecture.Mamba3` enum value
+- GGUF tensor-name mapping (needs to be reverse-engineered from whichever
+ converter the first checkpoint uses)
+- `Mamba3TransformerModel : IModel` analogous to `NemotronHTransformerModel`
+- `ModelLoader.LoadFromGguf` dispatch
+
+## 5. Open questions to confirm before starting
+
+1. **Use `VikramKarLex/mamba3-minimal` or `yang3121099/mamba3` as the
+ reference?** The former is explicitly Albert Gu-endorsed; defaulting
+ to it unless we find a reason otherwise.
+2. **Do we care about the MIMO variant for the initial deliverable,
+ or SISO-only?** Paper says MIMO gives the extra 1.2pp accuracy;
+ implementation complexity is low (rank-R expansion is one extra
+ matmul). Default: implement both in stage A, test both in stage B.
+3. **How strict should the fixture-comparison tolerance be?** PyTorch
+ vs our scalar C# on the same F32 weights should match to ~1e-5.
+ Proposed: 1e-4 element-wise absolute tolerance + 1e-3 relative.
+
+## 6. Risk assessment
+
+| Risk | Mitigation |
+|---|---|
+| Minimal PyTorch reference has subtle bugs or diverges from the paper | Cross-check key equations against the published paper; spot-check outputs against `state-spaces/mamba` full impl when it becomes easy to install |
+| RoPE-on-BC convention varies between references | Lock to the VikramKarLex convention; document the angle-sign and pair-ordering choices |
+| Complex state is actually complex arithmetic in some impls | Paper and minimal reference both express it as real-valued 2D rotation — we follow that; re-verify once the official code is reproduced |
+| Porting without a checkpoint means latent bugs won't be caught end-to-end | Stage B's PyTorch-fixture tests catch numerical drift at the layer level; stage D will catch any architecture-level bugs once a checkpoint lands |
+| Scope creep: MIMO + SISO + optimization kernels | Stage A first (SISO correctness), MIMO as optional stage A5, SIMD later under a separate branch |
+
+## 7. Done criteria
+
+**For this branch to be considered complete:**
+
+* All four Stage-A kernels land with passing unit tests.
+* Stage-B PyTorch fixture comparison test is green: our C# Mamba-3 scan
+ matches the minimal reference's layer output element-wise within
+ `1e-4` abs / `1e-3` rel on a 64-token synthetic input.
+* Stage C `Mamba3Block` composes the kernels correctly and passes a
+ full-layer reference comparison.
+* `DESIGN_MAMBA_3.md` (this file) is updated with any convention choices
+ that turn out to differ from the paper abstract.
+
+**Stage D is a separate branch** once checkpoints exist. Do not block
+this branch on upstream checkpoint availability.
diff --git a/PLANS.md b/PLANS.md
new file mode 100644
index 00000000..9c59a5a1
--- /dev/null
+++ b/PLANS.md
@@ -0,0 +1,348 @@
+# PLANS.md — Outstanding Work (feature/mamba-3)
+
+Coverage gaps, documentation deficits, and standards alignment for the
+`feature/mamba-3` branch at HEAD `b72f6a0`, ordered by ROI. Each item lists
+scope, acceptance criteria, files touched, and parallelization notes.
+
+**Session-to-date shipped capability** (for context): Mamba-3 end-to-end (real
+ib-ssm 370M, streaming decode, MIMO kernel/block); safetensors loader dispatch
+(Llama/Mistral/Phi/Qwen/Mixtral/Qwen-MoE/DeepSeek-V2/V3/Granite-MoE); HF
+tokenizer.json adapter (SPM+Metaspace+ByteFallback); multi-shard; MLA
+attention kernel + TransformerModel integration; MoE GroupedGEMM refactor
+(bit-identical); DeepSeek multi-shared-expert; Phi-3 fused-tensor loader;
+Granite-MoE fused-per-expert loader. Real-weight validation for: Phi-3.5-mini
+(7.6 GB), Granite-3.0-MoE (6.3 GB), Qwen2.5-0.5B (999 MB), TinyLlama-1.1B
+(2.1 GB), ib-ssm/mamba3-370M (1.55 GB).
+
+---
+
+## P0 — Blocks existing claims (highest ROI)
+
+### P0.1 HF ByteLevel BPE tokenizer adapter
+Our `tokenizer.json` adapter handles Metaspace pretokenizer + ByteFallback
+decoder (Llama-2/Mamba-3). It does **not** handle ByteLevel pretokenizer +
+ByteLevel decoder (GPT-2/Qwen/Llama-3/Granite/Phi-3). Without this we cannot
+encode real text for any of the dense transformer architectures we "support".
+
+- **Scope**: Extend `HfTokenizerJsonParser` + `HfBpeTokenizerFactory` to detect
+ and route ByteLevel pretokenizer (regex-split + byte-to-unicode mapping) and
+ ByteLevel decoder. Add the GPT-2 255-byte alphabet mapping. Handle
+ `Sequence` of pretokenizers (Qwen2 uses `[Split, ByteLevel]`).
+- **Acceptance**: round-trip "Hello world" on Qwen2.5-0.5B, Phi-3.5,
+ Granite-3, TinyLlama tokenizers produces identical token IDs to HF Python's
+ `AutoTokenizer.encode`. Unit tests for Split+ByteLevel composition.
+- **Files**: `src/DotLLM.Tokenizers/Hf/{HfTokenizerJsonParser,HfBpeTokenizerFactory}.cs`,
+ new `ByteLevelPreTokenizer.cs`, new `ByteLevelDecoder.cs`, tests under
+ `tests/DotLLM.Tests.Unit/Tokenizers/Hf/`.
+- **Deps**: none. **Parallel**: yes.
+
+### P0.2 DeepSeek-V2-Lite real-weight end-to-end
+Download in flight (~31 GB, `b3rt1lmmu`). Once present, the existing
+`RealHfSafetensorsEndToEndTests` pattern gives real-weight validation for MLA
++ multi-shared-expert MoE.
+
+- **Scope**: add `DeepSeekV2Lite_LoadsAndForwardsEndToEnd` gated by
+ `DOTLLM_DEEPSEEK_V2_LITE_PATH` + auto-detect `C:/temp/dotllm-deepseek-v2-lite/`.
+ Forward `[0, 1, 2]`, assert `[3, 102400]` finite, stddev nonzero, MLA config
+ populated (`kv_lora_rank=512`, `qk_nope_head_dim=128`, etc.), MoE populated
+ (`n_routed_experts=64`, `n_shared_experts=2`).
+- **Acceptance**: test passes. Documents drift max_abs vs PyTorch reference if
+ feasible, else just finite check.
+- **Files**: `tests/DotLLM.Tests.Integration/Models/Loaders/RealHfSafetensorsEndToEndTests.cs`.
+- **Deps**: download completion.
+- **Parallel**: independent of everything else.
+
+### P0.3 Mamba-3 MIMO weight loader + model wiring
+`Mamba3TransformerModel` currently throws `NotSupportedException` when the
+config has `IsMimo=true`. The MIMO kernel and Block paths are complete and
+test-covered (step 60f); the remaining gap is the weight loader needing
+`[H, R, N]`-shaped B_bias/C_bias + `mimo_x`/`mimo_z`/`mimo_o` tensors.
+
+- **Scope**: extend `Mamba3WeightLoader` to load MIMO tensors; remove the
+ `NotSupportedException`; route MIMO models through `ForwardMimo` with
+ streaming state.
+- **Acceptance**: synthetic MIMO model loads + forwards + generates. No public
+ real-MIMO checkpoint exists — synthetic fixture suffices. Existing 83
+ Mamba-3 unit tests stay green.
+- **Files**: `src/DotLLM.Models/Architectures/Mamba3WeightLoader.cs`,
+ `Mamba3TransformerModel.cs`, new unit test + synthesis script.
+- **Deps**: none. **Parallel**: yes.
+
+### P0.4 Unified "Supported Models" matrix
+A single, authoritative page listing every architecture dotLLM can load, with
+required tokenizer type, RoPE variant, quant support, and the exact HF
+`config.json` field set. Solves discoverability and prevents claim drift.
+
+- **Scope**: new `docs/SUPPORTED_MODELS.md` + link from README. Table:
+ arch, enum, config fields, tokenizer, RoPE, KV-cache, MoE, notes.
+ Populate from current code; reference real-weight proofs.
+- **Acceptance**: every `Architecture` enum variant is listed with accurate
+ field mapping and at least one verified checkpoint (or "verified with
+ tiny-random" / "synthetic only" tag).
+- **Files**: new `docs/SUPPORTED_MODELS.md`, `README.md` link.
+- **Deps**: none. **Parallel**: yes (docs-only).
+
+---
+
+## P1 — Standards alignment
+
+### P1.1 Shared attention tensor loader abstraction
+`TransformerWeightsSafetensorsLoader` now has `LoadLlamaAttentionLayer`,
+`LoadPhi3AttentionLayer` (fused QKV), `LoadDeepSeekMlaLayer`,
+`LoadGraniteMoeLayer` — each with ~30-60 lines of tensor-name resolution.
+Repeat pattern ripe for DRY.
+
+- **Scope**: extract a shared `AttentionTensorLoader` service that takes an
+ attention variant enum (Llama/GQA, Phi3/FusedQKV, MLA) + per-variant tensor
+ names and returns a populated `TransformerLayerWeights` attention slot.
+ Bit-identical output.
+- **Acceptance**: all 1285 unit + full integration tests pass unchanged. Each
+ per-arch layer loader is <20 lines.
+- **Files**: `src/DotLLM.Models/Architectures/TransformerWeightsSafetensors.cs`,
+ possibly new `AttentionTensorLoader.cs`.
+- **Deps**: **conflicts with P2.2 (YaRN)** on MLA loader lines. Sequence P1.1
+ before P2.2. **Parallel**: yes with everything else.
+
+### P1.2 Remove `goto FfnBranch` in TransformerModel.Forward
+The MLA branch introduced a `goto` label to jump past GQA Q/K/V/RoPE/O code
+to the shared FFN dispatch. Unusual for C#; refactor into explicit branch /
+helper method.
+
+- **Scope**: extract `RunAttentionLayer(config, lw, ...)` helper returning
+ post-attention hidden; both GQA and MLA paths call it; then FFN runs. No
+ numerical change.
+- **Acceptance**: bit-identical logits on all real-weight tests
+ (Phi-3.5, Qwen2.5, TinyLlama, Granite-MoE, DeepSeek-V2-tiny, Mamba-3 ib-ssm).
+- **Files**: `src/DotLLM.Models/Architectures/TransformerModel.cs`.
+- **Deps**: **conflicts with P1.1**. **Parallel**: no — must sequence against
+ P1.1 on same file.
+
+### P1.3 Benchmark regression sweep
+No benchmarks run since the session began. The GroupedGEMM refactor, pooled
+scratch, and MLA integration could silently regress throughput. Run the
+existing BDN suite and commit deltas.
+
+- **Scope**: `dotnet run -c Release --project benchmarks/DotLLM.Benchmarks`
+ against SmolLM-135M, Llama-3.2-1B (or equivalent); compare vs `main`.
+- **Acceptance**: prefill + decode tok/s within ±5% of main. Any regression
+ documented with root cause.
+- **Files**: `benchmarks/*.cs` (only if new benchmarks needed), report in
+ PLANS.md or separate doc.
+- **Deps**: none. **Parallel**: yes (separate artifact).
+
+**Pre-W2 baseline captured at commit `5ff5312`** (post P1.1+P0.1+P0.3+P0.4
+merges, before P1.2/P2.2/P2.3 land):
+
+| Benchmark | Config | Pre-W2 baseline |
+|---|---|---|
+| `InferenceBenchmarks` E2E | SmolLM-135M Q8_0, CPU, single-thread, 20-token decode | **prefill 233.8 tok/s**, **decode 16.8 tok/s**, E2E 1.191 s ± 0.025 s (5 iter, 2 warmup) |
+| `SpecialTokenEncodeBenchmarks` | 1024 chars, 5/20/100 specials | ~40 μs, 57.63 KB allocated |
+| `SpecialTokenEncodeBenchmarks` | 8192 chars, 5/20/100 specials | ~320 μs, 467.09 KB allocated |
+
+**Note on tokenizer allocation**: the ~467 KB-per-call at 8192 chars is consistent
+with P0.1's new one-string-per-regex-match pattern in `Gpt2TiktokenEncoding.ByteMap`
+(see follow-up note in P1.4). Pre-P0.1 numbers weren't captured; first regression
+check vs these numbers after W2 lands.
+
+### P1.4 Analyzer / style pass
+Spot-check for IDE warnings, CA-analyzer hits, stale XML doc references,
+unused using statements accumulated over the session's many commits.
+
+- **Scope**: `dotnet build -warnAsError=false`; triage warnings; fix trivial.
+ Run any `.editorconfig`-enforced formatter.
+- **Acceptance**: warning count on `feature/mamba-3` ≤ warning count on
+ `main`. No new suppressions.
+- **Files**: various, all new code primarily.
+- **Deps**: best done last to sweep up everything. **Parallel**: yes (touches
+ many files but non-semantic).
+
+---
+
+## P2 — Hardening & correctness expansion
+
+### P2.1 Transformer generation loop integration test
+We've only exercised generation end-to-end on Mamba-3. Each real transformer
+(Qwen2.5, TinyLlama, Phi-3.5, Granite-MoE) should have an equivalent
+`GeneratesText_FromTokenizedPrompt`-style test.
+
+- **Scope**: gated integration test per architecture; encodes a short prompt
+ via the HF tokenizer (requires **P0.1**), iteratively forwards growing
+ context, argmax, decode; asserts finite, valid token IDs, no infinite loops.
+- **Acceptance**: 5-token generation from "The capital of France is" or
+ similar completes for each arch, producing at least one non-trivial token.
+- **Files**: `tests/DotLLM.Tests.Integration/Models/Loaders/`
+ (new `*GenerationTests.cs` per arch or one shared).
+- **Deps**: **P0.1 (ByteLevel tokenizer)** for Qwen/Phi/Granite/Llama-3;
+ TinyLlama can go without (SPM).
+- **Parallel**: after P0.1.
+
+### P2.2 YaRN RoPE mscale
+`MlaConfig` stores YaRN scaling fields (`RopeScalingFactor`,
+`RopeScalingMscale`, `RopeScalingMscaleAllDim`,
+`RopeScalingOriginalMaxPositionEmbeddings`) but `MlaAttention.Execute`
+ignores them. DeepSeek-V3 uses YaRN for 128K+ context; without it, long
+prompts produce wrong logits.
+
+- **Scope**: apply mscale correction in the attention scale (`scale *=
+ mscale^2`) and in RoPE frequency scaling per YaRN spec. Preserve default
+ (non-YaRN) behavior when fields are null.
+- **Acceptance**: DeepSeek-V2/V3 long-context test (>4K tokens) produces
+ finite logits; drift vs known-good reference within tolerance.
+- **Files**: `src/DotLLM.Cpu/Kernels/MlaAttention.cs`,
+ `src/DotLLM.Models/Architectures/TransformerModel.cs` (if RoPE table
+ construction moves).
+- **Deps**: sequences after P1.1 on MLA loader lines.
+- **Parallel**: conflicts with P1.1 and P2.3 — serialize.
+
+### P2.3 MLA latent KV-cache + absorption
+MLA's production win is KV-memory reduction (store `kv_lora_rank=512` latent
+instead of `n_heads * (qk_nope + v) = 3072+` per head). Absorption fuses
+`W_q_nope @ W_k_nope^T` at load time, eliminating runtime GEMM. Currently
+PoC reruns the full attention forward per call.
+
+- **Scope**: `LatentMlaKvCache` storing compressed latent; `MlaAttention`
+ variant consuming it; absorption precomputed at load time.
+- **Acceptance**: DeepSeek-V2-Lite generates text; memory footprint drops
+ measurably; logits match pre-optimization within 1e-3.
+- **Files**: `src/DotLLM.Engine/KvCache/LatentMlaKvCache.cs` (new),
+ `src/DotLLM.Cpu/Kernels/MlaAttention.cs`,
+ `src/DotLLM.Models/Architectures/TransformerModel.cs`.
+- **Deps**: after P2.2 (shared file). **Parallel**: no.
+
+### P2.4 Sliding-window attention test
+Qwen2.5-0.5B config has `sliding_window=32768`; Mistral uses sliding window
+smaller. Currently untested — we feed 3 tokens so window never triggers.
+Kernel correctness for >window contexts is unverified.
+
+- **Scope**: test that feeds >`sliding_window` tokens (or a smaller
+ synthetic config with window=8, seqlen=16) and asserts attention pattern
+ respects the window. Compare against a reference Python implementation or
+ a masked-brute-force scalar.
+- **Acceptance**: sliding window masks correctly zero out pre-window
+ attention weights; output matches masked reference.
+- **Files**: `src/DotLLM.Cpu/Kernels/Attention.cs` (check existing),
+ `tests/DotLLM.Tests.Unit/Cpu/Kernels/AttentionSlidingWindowTests.cs` (new).
+- **Deps**: none. **Parallel**: yes.
+
+### P2.5 OLMoE-1B-7B real-weight Mixtral-convention validation
+Mixtral-convention MoE (`block_sparse_moe.experts.{j}.w{1,2,3}` with per-
+expert separate tensors) only tested with 522 KB tiny-random. OLMoE-1B-7B
+(14 GB) is current (2024), open, Mixtral-convention.
+
+- **Scope**: download `allenai/OLMoE-1B-7B-0924`; add
+ `OLMoE1B7B_LoadsAndForwardsEndToEnd` gated test.
+- **Acceptance**: load + forward + finite logits.
+- **Files**: download script, test addition to
+ `RealHfSafetensorsEndToEndTests.cs`.
+- **Deps**: 14 GB download + disk availability.
+- **Parallel**: yes (independent download + test).
+
+### P2.6 bf16/F16 correctness test
+Every real checkpoint so far is BF16 on disk, upcasted to F32 at load. The
+upcast is assumed correct but never numerically compared against a reference
+running on native BF16 (e.g., PyTorch). Drift in edge cases (denormals,
+NaN propagation through softmax) could pass finite checks but still be
+wrong.
+
+- **Scope**: pick one small BF16 model, load via dotLLM → F32 and via
+ PyTorch → BF16; compare forward outputs at N tokens.
+- **Acceptance**: max_abs <1e-2 (very loose; BF16 has ~7-bit mantissa so
+ drift vs F32 is expected ~1e-3 per layer, accumulating).
+- **Files**: new Python reference script + C# test.
+- **Deps**: none. **Parallel**: yes.
+
+---
+
+## P3 — Documentation refinement
+
+All of these are independent text-only edits; can run fully in parallel. One
+agent, one file each, trivial coordination.
+
+### P3.1 `docs/ATTENTION.md` — MLA section
+Current doc covers MHA/MQA/GQA. Add MLA section: down-projection via
+`q_a_proj`/`kv_a_proj_with_mqa`, RMSNorm, up-projection, decoupled RoPE,
+per-head SDPA with MQA-shared K_rope, o_proj. Reference `MlaAttention.cs`.
+
+### P3.2 `docs/TOKENIZERS.md` — HF adapter section
+Add section on `HfTokenizerJsonParser` + `HfBpeTokenizerFactory`. Document
+which pretokenizer/decoder combinations are supported (Metaspace +
+ByteFallback today; ByteLevel per P0.1 pending). Include how `model_type` →
+tokenizer style mapping works, and the `ModelLoader.LoadTokenizerFromHfDirectory`
+helper.
+
+### P3.3 Per-architecture config reference
+New `docs/ARCHITECTURES.md` (or section in SUPPORTED_MODELS.md) — for each
+arch, list the `config.json` fields consumed by `HfConfigExtractor` and any
+tensor naming quirks. Link to relevant kernel/loader sources.
+
+### P3.4 Stale `docs/*.md` sweep
+Spot-check every `docs/*.md` for references to out-of-date file paths,
+missing features, or claims that no longer hold (e.g., "supports Llama,
+Mistral, Phi, Qwen, DeepSeek" in the README may need updates for
+Mamba-3/MoE/Granite).
+
+### P3.5 Qwen2.5 + TinyLlama News entry
+Single README News bullet covering both (real-weight validations of Qwen-dense
+and Llama-family). Reference commit `b72f6a0`.
+
+### P3.6 PR #136 description refresh
+Per-commit comments have accumulated. PR description is the top-level view;
+ensure it reflects the full scope shipped (a bullet list of arches supported,
+tests passed, known gaps from this PLANS.md).
+
+---
+
+## P4 — Deferred / blocked
+
+### P4.1 Chat templates from `tokenizer_config.json`
+Not loaded today. Nice-to-have for generation UX. Defer.
+
+### P4.2 Quantized safetensors (int8 / GPTQ / AWQ)
+Significantly expands supported-checkpoint universe but is a full new feature
+(new kernels, new loader paths). Separate PR / issue.
+
+### P4.3 MIMO Mamba-3 real-weight verification
+Blocked — no public MIMO Mamba-3 checkpoint exists. P0.3 covers the code
+path; real-weight waits indefinitely.
+
+---
+
+## Parallelization analysis
+
+**Fully independent (any agent, any file — safe for parallel worktrees):**
+- P0.1 (ByteLevel tokenizer) — `src/DotLLM.Tokenizers/Hf/`
+- P0.2 (DeepSeek-V2-Lite test) — test file only
+- P0.3 (Mamba-3 MIMO loader) — `Mamba3WeightLoader.cs`, `Mamba3TransformerModel.cs`
+- P0.4 (Supported Models matrix) — `docs/SUPPORTED_MODELS.md` (new)
+- P1.3 (benchmarks) — `benchmarks/*`, separate artifact
+- P2.4 (sliding-window test) — new test file + `Attention.cs` (usually isolated)
+- P2.5 (OLMoE) — download + test file
+- P2.6 (bf16 correctness) — new test + Python script
+- All of P3 (docs) — one file each, text only
+
+**Serialized chains (same-file conflicts):**
+- P1.1 → P1.2 → P2.2 → P2.3 → P1.4 on `TransformerModel.cs` / `MlaAttention.cs`
+- P0.1 → P2.1 (generation tests depend on ByteLevel tokenizer)
+- P0.3 wiring → updating `.continue-here.md` / docs entries
+
+**Download-bound (no compute but network):**
+- P0.2 (DeepSeek-V2-Lite — already downloading)
+- P2.5 (OLMoE — needs ~14 GB download)
+
+**First-wave dispatch plan (maximum parallelism, no conflicts):**
+
+| Wave | Items | Expected time |
+|---|---|---|
+| W1 (parallel, 4 agents + 1 orchestrator task) | P0.1, P0.3, P0.4, P1.1, P1.3 | ~30-60 min each |
+| W2 (serial after W1 on MLA file) | P1.2, P2.2, P2.3 | ~30 min each |
+| W3 (parallel after W1) | P2.1 (after P0.1), P2.4, P2.5 (after OLMoE download), P2.6 | ~30 min each |
+| W4 (parallel, all W1-W3 prereqs satisfied) | P3.1, P3.2, P3.3, P3.4, P3.5, P3.6 | 10-20 min each |
+| W5 (final sweep) | P1.4 (analyzer pass) | 15 min |
+
+**Automated / passive items:**
+- DeepSeek-V2-Lite download (~hours, no action)
+- P0.2 test runs when download completes (automatic via the gated test)
+
+**Critical-path estimate** (serial MLA chain): W1 + W2 ≈ 2-3 hours if MLA
+chain is taken on one file sequentially. Everything else parallelizes around
+that.
diff --git a/README.md b/README.md
index 7092f8a8..3d97ea76 100644
--- a/README.md
+++ b/README.md
@@ -17,9 +17,9 @@
## About
-dotLLM is a ground-up LLM inference engine for .NET — not a wrapper around llama.cpp or Python libraries. All orchestration, model loading, tokenization, sampling, and CPU compute are implemented in pure C#, with CUDA GPU acceleration via PTX kernels loaded through the CUDA Driver API (no native shared library). It targets transformer-based models (Llama, Mistral, Phi, Qwen, DeepSeek) with SIMD-optimized CPU and CUDA GPU backends.
+dotLLM is a ground-up LLM inference engine for .NET — not a wrapper around llama.cpp or Python libraries. All orchestration, model loading, tokenization, sampling, and CPU compute are implemented in pure C#, with CUDA GPU acceleration via PTX kernels loaded through the CUDA Driver API (no native shared library). It targets transformer and SSM/MoE model families including Llama, Mistral, Phi, Qwen, DeepSeek-V2/V3, Mixtral, Granite-MoE, Nemotron-H, and Mamba-3 with SIMD-optimized CPU plus CUDA and Vulkan GPU backends.
-> **Status**: Phase 6 complete — speculative decoding, paged KV-cache, Native AOT (experimental), and startup warm-up on top of the OpenAI-compatible API server, built-in chat UI, constrained decoding (JSON/schema/regex/grammar), tool calling, and prompt caching. CUDA GPU backend with CPU/GPU hybrid offloading and KV-cache quantization. SIMD-optimized CPU inference with Q4_K_M, chat templates, streaming, multi-threading, NUMA pinning. Supports Llama, Mistral, Phi, Qwen. Phase 7 (diagnostics & interpretability) in progress — logprobs landed. See [Roadmap](#roadmap).
+> **Status**: Active post-Mamba-3 development — speculative decoding, paged KV-cache allocation, Native AOT (experimental), startup warm-up, constrained decoding (JSON/schema/regex/grammar), tool calling, prompt caching, logprobs, LoRA adapter loading, CUDA CPU/GPU hybrid offload, and native Vulkan real-weight parity for multiple architectures. See [docs/SUPPORTED_MODELS.md](docs/SUPPORTED_MODELS.md) for the current support matrix.
## Key Features
@@ -27,12 +27,12 @@ dotLLM is a ground-up LLM inference engine for .NET — not a wrapper around lla
- **Zero-GC inference** — unmanaged memory (`NativeMemory.AlignedAlloc`, 64-byte aligned) for all tensor data; no managed heap allocations on the hot path
- **SIMD vectorization** — `TensorPrimitives` + hand-tuned `System.Runtime.Intrinsics` for quantized matmul, RMSNorm, RoPE, softmax
- **Memory-mapped model loading** — GGUF files loaded via `MemoryMappedFile`; OS demand-paging means multi-GB models load in milliseconds
-- **Quantized inference** — FP16, Q8_0, Q4_K_M and other GGUF quantization formats; fused scale×int dot-product kernels operating directly on quantized blocks
+- **Quantized inference** — FP16/BF16, Q8_0, K-quants, and other GGUF quantization formats; fused scale×int dot-product kernels operating directly on quantized blocks where implemented
### Architecture Support
-- **Transformer models** — Llama, Mistral, Phi, Qwen, DeepSeek via parameterized `TransformerBlock` and `ModelConfig`
-- **Attention mechanisms** — MHA, MQA, GQA via parameterized `ModelConfig`, with `IAttentionStrategy` for kernel selection
-- **Position encoding** — RoPE, ALiBi, absolute, none — pluggable via `IPositionEncoding`
+- **Transformer/SSM models** — dense transformers, MLA/MoE DeepSeek variants, MoE families, hybrid Nemotron-H, and pure-SSM Mamba-3. See [docs/SUPPORTED_MODELS.md](docs/SUPPORTED_MODELS.md) for the full matrix.
+- **Attention mechanisms** — MHA, MQA, GQA, sliding-window masks, and DeepSeek MLA via parameterized `ModelConfig`, with `IAttentionStrategy` for kernel selection
+- **Position encoding** — RoPE variants and Mamba-3 data-dependent RoPE plumbing, with a pluggable `IPositionEncoding` surface for additional encodings
- **Composable sampling** — `ISamplerStep` chain: repetition penalty → temperature → top-k → top-p → min-p → categorical sample
### Serving
@@ -43,9 +43,9 @@ dotLLM is a ground-up LLM inference engine for .NET — not a wrapper around lla
- **(Planned) Continuous batching** — iteration-level scheduling with preemption and priority queuing — Phase 9, see [Roadmap](#roadmap)
### Extensibility
-- **Pluggable backends** — `IBackend` interface with separate packages per backend (CPU, CUDA, ROCm)
+- **Pluggable backends** — `IBackend` interface with separate packages per backend (CPU, CUDA, Vulkan, ROCm/HIP foundation)
- **Diagnostic hooks** — zero-cost `IInferenceHook` points for activation capture, logit lens, SAE integration
-- **(Planned) LoRA adapters** — runtime loading, no weight merging, concurrent multi-adapter serving — Phase 7, see [Roadmap](#roadmap)
+- **LoRA adapters** — runtime loading, no weight merging, CPU/Vulkan application paths, and concurrent multi-adapter serving
- **(Planned) OpenTelemetry observability** — `System.Diagnostics.Metrics` + `Activity` for throughput, latency, and per-request tracing — Phase 7 / Phase 9, see [Roadmap](#roadmap)
## Architecture Overview
@@ -581,13 +581,13 @@ dotLLM ships as a set of NuGet packages so you can reference only what you need
| [`DotLLM.Core`](https://www.nuget.org/packages/DotLLM.Core) | Core abstractions — tensor types, backend interfaces, model config, sampling, attention strategies, diagnostics hooks |
| [`DotLLM.Cpu`](https://www.nuget.org/packages/DotLLM.Cpu) | CPU backend — SIMD-optimized quantized matmul, RMSNorm, RoPE, softmax, attention |
| [`DotLLM.Cuda`](https://www.nuget.org/packages/DotLLM.Cuda) | CUDA GPU backend — PTX kernels via CUDA Driver API, cuBLAS prefill, CPU/GPU hybrid offload |
-| [`DotLLM.Models`](https://www.nuget.org/packages/DotLLM.Models) | Memory-mapped GGUF/SafeTensors loaders, parameterized `TransformerBlock` (Llama/Mistral/Phi/Qwen/DeepSeek) |
+| [`DotLLM.Models`](https://www.nuget.org/packages/DotLLM.Models) | Memory-mapped GGUF/SafeTensors loaders, parameterized model loaders for dense transformer, MLA/MoE, and Mamba-3 families |
| [`DotLLM.Tokenizers`](https://www.nuget.org/packages/DotLLM.Tokenizers) | BPE, SentencePiece, HuggingFace tokenizer.json, Jinja2-subset chat templates |
| [`DotLLM.Engine`](https://www.nuget.org/packages/DotLLM.Engine) | Inference engine — KV-cache, scheduler, samplers, constrained decoding, speculative decoding |
| [`DotLLM.Server`](https://www.nuget.org/packages/DotLLM.Server) | OpenAI-compatible HTTP server, tool calling, built-in chat UI |
| [`DotLLM.HuggingFace`](https://www.nuget.org/packages/DotLLM.HuggingFace) | HuggingFace Hub search and GGUF download/caching |
| [`DotLLM.Diagnostics`](https://www.nuget.org/packages/DotLLM.Diagnostics) | Interpretability hooks — activation capture, logit lens, logprobs |
-| [`DotLLM.Telemetry`](https://www.nuget.org/packages/DotLLM.Telemetry) | Placeholder package; `System.Diagnostics.Metrics` counters and `Activity`-based tracing are planned (Phase 7 / Phase 9) |
+| [`DotLLM.Telemetry`](https://www.nuget.org/packages/DotLLM.Telemetry) | Telemetry package for metrics/tracing integration as observability work lands |
| [`DotLLM.Cli`](https://www.nuget.org/packages/DotLLM.Cli) | `dotnet tool` — the `dotllm` command (run / chat / serve / model management) |
Install the engine plus CPU backend for a minimal setup:
@@ -672,6 +672,27 @@ Both modes transparently reuse the embedded chat UI assets if `serveUi: true`. T
## News
+- **2026-04** — **Vulkan MoE Strategy C grouped F16 coopmat path** — routed MoE expert rows can now stay grouped by expert on-GPU for native F16 expert banks when `VK_KHR_cooperative_matrix` is available. The path computes expert counts/offsets on device, groups routed rows, runs grouped F16 coopmat W1/W3/W2 projections, ungroups rows back to the existing weighted-scatter contract, and falls back to the prior indexed F32/Q8_0 path whenever the model has active MoE LoRA, non-F16 expert banks, incompatible dimensions, or no cooperative-matrix support. Covered by standalone group/offset/grouped-matmul kernel tests and an end-to-end synthetic Vulkan-vs-CPU MoE forward parity test.
+- **2026-04** — **Vulkan: K-quants (Q4_K_M / Q5_K_M / Q6_K_M) + native F16 / BF16 matmul kernels**, plus comprehensive real-weight Vulkan parity coverage. Vulkan's matmul library now ships GEMV (decode) + GEMM (prefill) shaders for every common GGUF quantisation: F32, F16, BF16, Q8_0, Q4_K_M, Q5_K_M, Q6_K_M — plus an opt-in F16 cooperative-matrix GEMM path on devices that enumerate F16xF16→F32 tiles (gfx1151 verified). K-quant + Q8_0 weights stay in source bytes on device and dequantise inside the shader inner loop, eliminating the 2× memory penalty of the previous F32-only upload path. F16 source weights also stay native; BF16 source preserves bytes via `uintBitsToFloat(bits<<16)` (no F16 extension required). The kernel half lands at `afb2272`/`15099b9`/`29a1459`/`c9c08c5`; upload-path + dispatcher routing at `b1ee6bc`/`83e0732`/`39b7646`/`6787492` covers `VulkanWeights` (standard transformer), `VulkanNemotronHWeights`, `VulkanMamba3Weights` and the MoE overlay slots. Real-weight CPU↔Vulkan parity at the integration tier now covers **TinyLlama-1.1B, Phi-3.5-mini, Qwen2.5-0.5B, Granite-3-MoE, Mamba-3 ib-ssm 370M, DeepSeek-V2-Lite-Q4_K_M (MLA + multi-shared MoE, 19m 27s end-to-end)** — `RealHfSafetensorsEndToEndVulkanTests` + `RealGgufVulkanParityTests`. Vulkan suite 264 → 360 (+96 unit tests). DeepSeek-V2-Lite via the BF16 SafeTensors path self-skips at 0.7 s with a clear actionable message (the F32-expanded weights would need ~58 GB; use the Q4_K_M GGUF instead — Phase 1 K-quants make it native).
+- **2026-04** — **Long-context YaRN ramped-freq verification end-to-end** — `DeepSeekV2Lite_LongContext_LogitsMatchPyTorchReference` was previously a 50 ms self-skip while its PyTorch reference JSON was missing. New `--prompt-file` + `--last-n-rows` flags on `tests/scripts/compare_logits_py_reference.py` plus a `logits_offset` field on the JSON schema make >4K-token references practical (a full 4920 × 102400 reference would be ~5 GB JSON; subset-to-last-8-rows is ~7.4 MB). The test now actually runs (36m 55s on this iGPU box's CPU forward) and validates positions 4912–4920 — past DeepSeek-V2-Lite's `original_max_position_embeddings=4096` where YaRN's ramped `inv_freq` actively diverges from plain RoPE — within `DriftTolerances.Tight` against the PyTorch BF16 oracle. Validates the YaRN fix landed at `1e0a930`. The C# reference comparator was extended to slice `ours[offset..offset+ref_seq_len, :]` for the comparison; short-prompt references with offset=0 are unaffected.
+- **2026-04** — **DeepSeek-V2/V3 default to Phase C latent KV-cache** at the loader level (HF + GGUF) — the Phase A expanded cache scales as `O(numLayers × maxSeqLen × numHeads × (qkNope + v) × 4 bytes)` which for V2-Lite at `max_position_embeddings=163840` is ~68 GB and OOMs on most hosts. Phase C's hybrid (latent persistence + Phase A-equivalent prefill expand + absorbed decode) keeps the same numerics but stores `kv_lora_rank=512` per token instead of expanded heads — ~9 GB for the same model. `HfConfigExtractor` and `GgufModelConfigExtractor` both now set `UseHybridMlaCache=true` by default for `Architecture.DeepSeekV2`/`Architecture.DeepSeekV3`; production code paths get the win automatically, unit tests that build `MlaConfig` directly (bypassing the loader) still default to Phase A.
+- **2026-04** — **Vulkan: fused rmsnorm + matmul_q8_0 (sub-tile design)**, 282 → 295 tok/s SmolLM-135M.Q8_0 decode on AMD Radeon 8060S iGPU (RDNA3.5; +4.6%, beats v0.2.0-alpha.2 baseline of 291). Eliminates 60 dispatches + 60 pipeline barriers per decode at the attn-norm+Q and ffn-norm+Gate fusion sites. Each WG outputs ROWS_PER_WG=8 matmul rows so the rmsnorm reduction is amortised 8× across the rows it feeds (72 vs 576 WGs for SmolLM Q proj); the matmul phase reads normalised values from on-chip shared memory, and a leader-WG slice of the dispatch writes the global normOutput so K/V/Up matmuls keep working unchanged. A first per-row variant was ~7% slower (576× redundant rmsnorm overwhelmed the dispatch+barrier savings); sub-tile-4 broke even; sub-tile-8 with K_MAX=1024 (4 KB sharedNorm) tipped the balance. Default-on; opt out via `DOTLLM_VULKAN_DISABLE_FUSED_RMSNORM_MATMUL=1`. Falls back to the standalone (rmsnorm + matmul) pair for prefill (seqLen>1), non-Q8_0 weights, and hidden sizes >1024. Strict-argmax CPU↔Vulkan parity holds (8/9, unchanged). 8 unit parity tests use a CPU reference and exercise K=32 with M>1, sidestepping the standalone GEMV's known `rowUints*4` over-stride bug (issue #1) — the fused shader uses the correct `blocksPerRow*34` byte stride ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Phi-3.5-mini-instruct + Granite-3.0-MoE end-to-end on real weights.** Two loader extensions enable loading recent production checkpoints as-is from HuggingFace. **Phi-3** (`Phi3ForCausalLM`) stores attention as a fused `self_attn.qkv_proj [Q+K+V, hidden]` and MLP as a fused `mlp.gate_up_proj [2*intermediate, hidden]`; the new `SplitFusedProjection` helper probes for these fused names, allocates one 64-byte-aligned F32 buffer per split at the correct row offsets, and BF16/F16/F32-decodes in place. Fusion-absent models unchanged. **Granite-MoE** (`GraniteMoeForCausalLM` / `model_type=granitemoe`) packs all experts per-layer into `block_sparse_moe.input_linear [E, 2*I, H]` + `block_sparse_moe.output_linear [E, H, I]` + a `router.layer` gate. New `Architecture.GraniteMoe` variant + `LoadGraniteMoeLayer` slices each expert's w1/w2/w3 from the fused tensors into per-expert F32 pointers; the existing `MoeSwiGluMlp` kernel handles top-8 routing natively. **Real-weight verification**: `microsoft/Phi-3.5-mini-instruct` (7.6 GB, 32-layer MHA) produces `[3, 32064]` logits 96192/96192 finite (stddev 7.00); `ibm-granite/granite-3.0-3b-a800m-instruct` (6.3 GB, 40 experts top-8, 24Q/8KV GQA) produces `[3, 49155]` logits 147465/147465 finite (stddev 16.13). Both gated by env vars (`DOTLLM_PHI35_CHECKPOINT_PATH`, `DOTLLM_GRANITE3_CHECKPOINT_PATH`) with auto-detect of `C:/temp/dotllm-{phi35-mini,granite3-moe}/` ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **DeepSeek-V2/V3 end-to-end MLA integration** — `TransformerModel.Forward` gains a per-layer MLA branch (RMSNorm → `MlaAttention.Execute` → residual → shared MoE/dense FFN dispatch), `LoadDeepSeekMlaLayer` safetensors weight loader resolves all MLA tensors (`q_a_proj`/`q_a_layernorm`/`q_b_proj` or monolithic `q_proj` when `q_lora_rank=0`, `kv_a_proj_with_mqa`/`kv_a_layernorm`/`kv_b_proj`, `o_proj`), and `ModelLoader.LoadFromSafetensors` now dispatches `Architecture.DeepSeekV2`/`DeepSeekV3` (removed `NotSupportedException`). FFN routes to dense MLP (first-K prefix) or plural-shared-experts MoE (step 58c). GQA path for Llama/Mistral/Phi/Qwen unchanged. Verified against real `yujiepan/deepseek-v2-tiny-random`: 4-token prefill produces 409600/409600 finite logits (stddev 0.056). 3 new unit tests (LoRA-Q prefill, monolithic-Q prefill, LoRA-Q single-token); 1285 unit tests pass. Follow-ups: latent KV-cache, YaRN mscale, absorption optimisation, real DeepSeek-V2-Lite (16 GB) validation ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **DeepSeek multi-shared-expert MoE (`n_shared_experts > 1`)** — extends the MoE shared-expert branch to run N shared experts in parallel (DeepSeek-V2/V3 convention; Qwen1.5-MoE at N=1 unchanged). `MoeConfig.NumSharedExperts` (default 1). `MoeLayerWeights.SharedGateProj/UpProj/DownProj` migrated from single `nint` to `nint[]`. `MoeSwiGluMlp.ExecuteWithSharedExpert` accepts per-shared-proj `ReadOnlySpan`; for k=0 the GEMM writes directly into the down-buffer (bit-identical with prior single-shared path), subsequent experts compute into pooled scratch then `TensorPrimitives.Add`. Plural HF naming `mlp.shared_experts.{k}.{gate,up,down}_proj` supported; singular Qwen1.5 `mlp.shared_expert.*` still loads as length-1 array. This path is now used by the DeepSeek-V2/V3 MLA forward integration as well as Qwen-MoE shared-expert variants ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mamba-3 MIMO streaming decode (kernel + block)** — extends the SISO streaming SSD from step 60e to MIMO (rank > 1). `Mamba3State.k_state` rank-extended to `[R, H, N]` (SISO keeps `[H, N]`); `ssm_state`, `cum_angle`, `v_state` remain rank-free per canonical `mamba3.py` `allocate_inference_cache()`. New `Mamba3CanonicalSsd.ExecuteMimoStreaming` mirrors SISO streaming with rank-aware K; boundary term derived from the tilelang MIMO forward: `ssm[h,p,n] += v_state[h,p] · (Σ_r k_state[r,h,n]) · DT[0,h] · (1-trap[0,h])` — the rank-sum analog of the SISO adjustment. Empty-span and single-chunk schedules are bit-identical to the existing one-shot `ExecuteMimo`. Synthetic MIMO drift (R=3) across all split schedules: y max_abs ≤ 4.7e-15, ssm_state max_abs ≤ 3.4e-13 — pure F32-reorder noise. 18 new unit tests (8 kernel + 9 block + 1 state shape). Model-level wiring (`Mamba3TransformerModel` MIMO path) awaits the weight-loader extension for `[H, R, N]` biases + `mimo_*` projections ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **MoE GroupedGEMM refactor (bit-identical)** — `MoeSwiGluMlp` moves from per-token per-expert scalar loop to gather/batched-GEMM/scatter: bucket tokens by their top-k-assigned experts (`ArrayPool`-rented index lists, zero sustained allocations), run one batched SwiGLU per expert across all its assigned tokens (gather → `w1`/`w3` GEMM → silu*up → `w2` GEMM), scatter back preserving the original per-(token, slot) accumulation order for bit-identity. Public signatures unchanged; both `Execute` (Mixtral) and `ExecuteWithSharedExpert` (Qwen-MoE) route through the same grouped core. Verified bit-identical on all 8 existing MoE unit tests + 16 integration tests including the real `yujiepan/qwen3-moe-tiny-random` end-to-end forward. Follow-up: fused GroupedGEMM with SIMD-batched SwiGLU for ~2-3× further ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **MLA attention kernel (PoC) + DeepSeek-V2/V3 detection** — new `MlaAttention.Execute` scalar kernel implements DeepSeek's Multi-head Latent Attention end-to-end: Q LoRA factorization (`q_a_proj` → `q_a_layernorm` → `q_b_proj`, or monolithic Q when `q_lora_rank=0`), KV compression via `kv_a_proj_with_mqa` → `kv_a_layernorm` → `kv_b_proj` (split into `K_nope` + per-head `V`), decoupled RoPE on `qk_rope_head_dim` sub-dim with an MQA-shared `K_rope`, per-head causal SDPA, and `o_proj`. New `Architecture.{DeepSeekV2, DeepSeekV3}` enum variants; `HfConfigExtractor` detects `DeepseekV2ForCausalLM` / `DeepseekV3ForCausalLM` and populates `MlaConfig` + DeepSeek-flavoured `MoeConfig` (`n_routed_experts`, `n_shared_experts`, `first_k_dense_replace`). Verified against `yujiepan/deepseek-v2-tiny-random` (config detection) and a reference implementation within 5e-4 absolute tolerance. Later work wired this into `TransformerModel.Forward`, safetensors/GGUF loading, latent KV-cache defaults, and real DeepSeek-V2-Lite verification ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mamba-3 streaming decode (chunk-boundary state)** — Mamba-3 prefill+decode now reproduces one-shot prefill at chunk boundaries, closing the canonical `shifted_γ[t] = DT[t+1]·(1-trap[t+1])` 1-token lookahead gap. `Mamba3State` gains two new persistent buffers per layer — `k_state [n_head, d_state]` (previous chunk's last-token post-RoPE, pre-scale K) and `v_state [n_head, head_dim]` (previous chunk's last-token V) — matching canonical `mamba3_ssm.modules.mamba3` commit `7438488` (`mamba3.py:142` / `mamba3_siso_fwd.py:341-352`). `Mamba3Block.Forward` gains `kState` / `vState` span parameters: at chunk start it folds the deferred `ssm += v_state · k_state · DT[0] · (1 - trap[0])` term into the SSM state BEFORE the scan — exactly the contribution a one-shot forward would have produced via `scale[T_prev-1] = γ + shifted_γ`; at chunk end it persists this chunk's last-token post-RoPE K (from `bHRN[T-1]`) and V (from `xBuf[T-1]`). `Mamba3TransformerModel.Forward(..., Mamba3State)` threads all four buffers. **User-visible win:** on the real `ib-ssm/mamba3-370M-10BT` (48 layers, 32 heads, `d_state=128`) the last-token logit drift max_abs drops from ≈6.5 to ≈3.3 and — critically — the top-1 argmax now matches between one-shot and 2-prefill+1-decode (previously differed, so next-token sampling was inconsistent across modes). On tiny synthetic fixtures prefill+decode reproduces one-shot to F32-reorder noise (1e-8 on 2+2 splits, ~2e-6 on three-way splits). Opens the door to true O(N) streaming-decode generation (vs today's O(N²) growing-context prefill). Tightened assertions: `PrefillThenDecode_ApproximatesOneShot_WithExpectedDrift` renamed to `PrefillThenDecode_BitEqualsOneShot` (abs_tol 5e-5→1e-5, three schedules); `DecodeMatchesPrefillOnRealCheckpoint` ceiling 100→10 with a new `Assert.Equal(argmaxPrefill, argmaxSplit)`. New `Mamba3BlockStreamingTests` (7 split schedules × {y, ssm_state, k_state, v_state}) covers the kernel-level property. MIMO streaming deferred (canonical's tilelang MIMO kernel has no input-state signature; the MIMO Block path is unreachable today). 65 Mamba-3 unit tests + 20 integration tests all green ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mamba-3 pooled forward scratch** — eliminates per-call managed allocations on the Mamba-3 hot path. New `Mamba3ForwardScratch : IDisposable` (mirrors the existing `NemotronHForwardState` pattern) owns 13 named `NativeMemory.AlignedAlloc`-backed 64-byte-aligned scratch buffers (`Proj`, `X`, `Z`, `Dt`, `Adt`, `Trap`, `Gamma`, `Scale`, `AnglesRaw`, `B`, `C`, `QkPreDot`, `YScan`) grown power-of-two on demand via `EnsureCapacity(seqLen)` and reused across every layer of every `Mamba3Block.Forward` / `ForwardMimo` call. `Mamba3TransformerModel` allocates one scratch at construction and disposes it — on the ib-ssm 370M checkpoint this collapses ≈480 per-step managed float[] allocations (48 layers × ≈10 buffers) to zero after the first Forward. Numerics preserved bit-exact: every pre-existing Mamba-3 test (57 unit + 20 integration, including the canonical-Python SISO/MIMO block comparators and all 9 real-weight ib-ssm integration tests with end-to-end 5-token generation) still passes. `ComputeMemoryBytes` now reports the scratch footprint. Scratch handles both SISO and MIMO B/C widths in one instance (backed by `max(1, mimoRank)·nHead·dState`) ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Qwen-MoE support (Qwen1.5/2/3-MoE + shared experts)** — extends the Mixtral MoE plumbing to the HF Qwen-MoE naming convention (`mlp.gate` + `mlp.experts.{j}.{gate_proj,up_proj,down_proj}` instead of Mixtral's `block_sparse_moe.gate` + `experts.{j}.w1/w2/w3`), with optional shared-expert branch (Qwen1.5-MoE-A2.7B: `mlp.shared_expert.*` + optional `mlp.shared_expert_gate.weight` sigmoid scalar) and the `norm_topk_prob=false` raw-softmax gating used by Qwen1.5. New `Architecture.QwenMoe` enum variant dispatches `Qwen{2,3}MoeForCausalLM` / `model_type=qwen{2,3}_moe`. `MoeConfig` gains `NormTopKProb`, `SharedExpertIntermediateSize`, `HasSharedExpertGate`, `DecoderSparseStep`, and `MlpOnlyLayers` — the last two let Qwen3-MoE interleave dense MLP and MoE layers in the same model (`decoder_sparse_step=2` → layer 0 dense, layer 1 MoE). New `MoeSwiGluMlp.ExecuteWithSharedExpert` overload runs a parallel dense SwiGLU on every token and (optionally) multiplies it by `sigmoid(hidden . shared_expert_gate)` before adding to the routed top-k sum. Existing Mixtral `Execute` call-sites are untouched — the kernel change is additive. Verified end-to-end against the real `yujiepan/qwen3-moe-tiny-random` HF checkpoint (~20 MB, 2 layers × 8 experts × top-2, `decoder_sparse_step=2`, no shared expert): detection → load → 3-token forward → finite logits. Synthetic-fixture coverage for the shared-expert + sigmoid-gate + raw-softmax path (Qwen1.5-MoE convention). DeepSeek-V2/V3 reuse the plural shared-expert loader path through the MLA integration ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mamba-3 end-to-end text generation verified on ib-ssm 370M** — full pipeline confirmation: `ModelLoader.LoadFromSafetensors` + `ModelLoader.LoadTokenizerFromHfDirectory` + iterative `Mamba3TransformerModel.Forward` → argmax → decode. Test generates 5 tokens from `"The capital of France is"` → `[263, 1407, 4100, 760, 310]` → `"The capital of France is a very important part of"` (coherent English from a 10BT-trained base model). Uses growing-context one-shot prefill per step (O(N²) total, every forward canonical) to sidestep the `shifted_γ` chunk-edge lookahead drift that affects streaming decode; ~1.4 s/token on CPU, 7 s end-to-end for 5 tokens. Streaming-SSD decode (O(N) per token, needs `k_state` + `v_state` threading plus a streaming kernel) remains a follow-up. Gated integration test `IbSsmMamba3GenerationTests` runs on `DOTLLM_IBSSM_CHECKPOINT_PATH` / conventional paths ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Multi-shard safetensors loader (`model.safetensors.index.json`)** — HuggingFace sharded-checkpoint support for models above the default 5 GiB shard cap (e.g. Llama-3-8B-Instruct, Mistral-7B-Instruct). New `SafetensorsIndex` parses the `weight_map` + `metadata.total_size` sidecar; `MultiShardSafetensorsFile` opens every shard, resolves tensor-name → owning shard, and exposes the same lookup surface as the single-file reader via new `ISafetensorsTensorSource` interface. Consumers (`TransformerWeightsSafetensorsLoader`, `Mamba3WeightLoader`, `TransformerModel.LoadFromSafetensors`, `Mamba3TransformerModel.LoadFromSafetensors`) accept the interface; existing single-file callers unchanged. `ModelLoader.LoadFromSafetensors` auto-detects: directory path → probe for `model.safetensors.index.json` first (multi-shard) else single `.safetensors`; index.json path → multi-shard; single-file path → single-shard, but prefers sibling index.json when present. Duplicate tensor-name handling favours the index's authoritative mapping and throws on conflicts in non-index shards. Verified end-to-end via resharded `hf-internal-testing/tiny-random-LlamaForCausalLM` (2 shards, matching index.json, full forward pass: `[3, 32000]` finite logits, stddev=0.080) ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **HF `tokenizer.json` ingest for Mamba-3 / Llama-family** — JSON-driven BPE adapter bridging HuggingFace `tokenizer.json` into the existing `BpeTokenizer` / `SentencePieceEncoding` infrastructure. `HfTokenizerJsonParser` decodes vocab, merges (both `"a b"` and modern `["a","b"]` forms), added tokens, Metaspace pretokenizer, ByteFallback decoder. `HfBpeTokenizerFactory` maps merge rank → synthetic score (`score = -rank`) so the score-driven merge queue reproduces HF's earliest-rank-wins order. Special added tokens route through the control-token pre-split path; BOS/EOS auto-detected. `ModelLoader.LoadTokenizerFromHfDirectory` surfaces it next to `LoadFromSafetensors`. Verified end-to-end on real `ib-ssm/mamba3-370M-10BT`: `"Hello world"` → `[15043, 3186]` (matches Llama-2 canonical IDs), non-ASCII round-trips via byte fallback, `"The quick brown fox"` → forward through 48-layer 370M SSM → argmax decodes ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mamba-3 persistent state buffers** — `Mamba3State : IDisposable` owns per-layer `ssm_state` + `cum_angle` (`NativeMemory.AlignedAlloc`, 64-byte) and threads them across `Mamba3TransformerModel.Forward(..., Mamba3State)` calls for stateful multi-call sequence advance. Building block for streaming decode (full streaming kernel — `k_state` + `v_state` + chunk-boundary `shifted_γ` continuation — is the next stage). State threading is deterministic bit-for-bit on re-run; prefill+decode tracks one-shot up to the canonical chunk-edge lookahead drift, which is documented and pinned by a regression-ceiling assertion. Verified on the real `ib-ssm/mamba3-370M-10BT` checkpoint (gated by `DOTLLM_IBSSM_CHECKPOINT_PATH`) ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mixtral-family MoE support** — dense-routing top-k Mixture-of-Experts for Mixtral-convention models (Mixtral, Qwen*-MoE without shared experts, Phi-3.5-MoE). New `MoeConfig` on `ModelConfig` (`NumExperts`, `NumExpertsPerTok`, `MoeIntermediateSize`), `Architecture.Mixtral` enum variant, `HfConfigExtractor` detects `num_local_experts` / `num_experts` + `num_experts_per_tok` and surfaces Phi-3.5's `moe_intermediate_size` override. `MoeSwiGluMlp` kernel: full softmax over experts → top-k partial max-scan (stable tiebreak: lower index wins, matching `torch.topk`) → renormalise by sum (Mixtral convention, NOT a second softmax) → per-expert SwiGLU MLP via existing `FusedOps.SwiGLU` → weighted sum. `TransformerModel.Forward` branches on `TransformerLayerWeights.Moe`; safetensors loader resolves `block_sparse_moe.gate` + `experts.{j}.w1/w2/w3`, F16/BF16 → F32 upcast at load time. Verified against real `yujiepan/mixtral-tiny-random` (config + detection) and a synthetic 2-layer, 4-expert, top-2 fixture (full forward pass). Out of scope: shared experts (DeepSeek-V3), Qwen-MoE `mlp.experts` naming adapter, fused GroupedGEMM, expert parallelism, real Mixtral-8x7B validation ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — **Mamba-3 architecture (prefill)** — pure-SSM Mamba-3 end-to-end loading and prefill forward pass. `Mamba3ConfigExtractor` parses HF `config.json` (`model_type=mamba3`), `Mamba3WeightLoader` ingests F32 safetensors, and `Mamba3TransformerModel : IModel` composes `embed → N × (RMSNorm + Mamba3Block.Forward + residual) → final RMSNorm → LM head`. Canonical `state-spaces/mamba` semantics: 8-slice in_proj (`z, x, B, C, dd_dt, dd_A, trap, angles`), per-token per-head `A = -softplus(dd_A)` clamp, data-dependent RoPE with `tanh(angle)·π·DT` cumsum, attention-style SSD scan, SISO and MIMO entry points. `ModelLoader.LoadFromSafetensors` now dispatches `Architecture.Mamba3`. Prefill only — persistent decode state, tokenizer ingest, and generation loop are follow-ups. Verified end-to-end against canonical Python fixtures, a synthetic tiny checkpoint, and the real 1.55 GB `ib-ssm/mamba3-370M-10BT` checkpoint (48 layers × 32 heads × d_state=128 × vocab=32000, gated integration test via `DOTLLM_IBSSM_CHECKPOINT_PATH`) ([#136](https://github.com/kkokosa/dotLLM/pull/136))
+- **2026-04** — Safetensors loader for dense transformers — `ModelLoader.LoadFromSafetensors` + `TransformerModel.LoadFromSafetensors` ingest HuggingFace `model.safetensors` + `config.json` for Llama/Mistral/Phi/Qwen. `HfConfigExtractor` mirrors the GGUF extractor pattern over HF JSON fields (`hidden_size`, `num_hidden_layers`, `num_key_value_heads`, `rope_theta`, `tie_word_embeddings`, …). bf16 tensors are upcast into 64-byte-aligned scratch at load time; F32 tensors are zero-copy mmap views. `ModelLoader.Load(path)` auto-detects `.gguf` vs `.safetensors`. Verified end-to-end on `hf-internal-testing/tiny-random-LlamaForCausalLM`
- **2026-04** — **First public release (v0.1.0-preview.1)** — dotLLM goes public. [NuGet packages](#nuget-packages) for all 10 libraries + `DotLLM.Cli` as a global `dotnet tool`. Self-contained single-file downloads for Windows / Linux / macOS (Apple Silicon) and experimental Native AOT builds for Linux / Windows attached to every [GitHub Release](https://github.com/kkokosa/dotLLM/releases). Companion website at [dotllm.dev](https://dotllm.dev/) ([#119](https://github.com/kkokosa/dotLLM/issues/119))
- **2026-04** — **Wave 7**: CPU performance cleanup pass — `TopKSampler` replaces full `Array.Sort` with a hand-rolled size-K min-heap (`O(N log K)`, stack-resident scratch); `JsonSchemaConstraint` adds first-char bucketing to skip the ~160 MB of struct clones per mask build when the tracker rejects most leading characters, plus LRU eviction instead of the previous full-flush cache overflow; `Dequantize.Q5_0` gains an AVX2 path matching Q8_0's throughput (reuses `MatMulQ5_0.ExtractQ5HighBits` / `vpshufb` bit-extraction); `BpeTokenizer` pre-splits special tokens via the existing `Trie.TryMatchLongest` instead of the O(n × m) linear scan; `ComputeThreadPool` now pins the caller (inference) thread to the first candidate P-core on first `Dispatch`, eliminating the hybrid-CPU stall where pinned P-core workers idled at the barrier waiting for an E-core caller. New BenchmarkDotNet suites for TopK sampling, schema mask build, and special-token encode ([#109](https://github.com/kkokosa/dotLLM/issues/109))
- **2026-04** — **Phase 7 begins**: Logprobs — OpenAI-compatible `logprobs: true` + `top_logprobs: N` (0-20) on `/v1/chat/completions` and `/v1/completions`. Per-token log-softmax captured before sampling, returned in both streaming SSE chunks and non-streaming responses. Chat UI gains opt-in logprobs visualization: color-coded token confidence (green/lime/yellow/orange/red), hover tooltips with top-K alternatives and probabilities, diagnostic cues for low confidence, ambiguity, and sampling effect. `DotLLM.Sample.Logprobs` console sample with ANSI-colored output ([#101](https://github.com/kkokosa/dotLLM/issues/101))
@@ -716,13 +737,13 @@ Both modes transparently reuse the embedded chat UI assets if `serveUi: true`. T
| Phase | Description | Status |
|-------|-------------|--------|
| **1 — End-to-End Generation** | GGUF loading, dequantization, CPU ops, tokenizer, attention, forward pass, KV-cache, sampling | Done (9/9) |
-| **2 — Practical Local Inference** | Engine metrics, benchmarks, Q4_K_M, chat templates, streaming, multi-threading, more architectures | Done (10/10) |
+| **2 — Practical Local Inference** | Engine metrics, benchmarks, Q4_K_M, chat templates, streaming, multi-threading, more architectures, safetensors loader | Done (12/12) |
| **3 — CPU Performance** | Decode dispatch, Q8_1 input, weight repacking, outer-product GEMM, tiled attention, fast exp, fusion, NUMA | In Progress (7/8) |
| **4 — GPU Acceleration** | CUDA backend, CPU/GPU hybrid, KV-cache quantization | Done (3/3) |
| **5 — Constrained Decoding & API** | JSON mode, JSON Schema, regex/CFG, tool calling, OpenAI API server, chat UI, prompt caching | Done (7/7) |
| **6 — Improved Serving** | Warm-up, NativeAOT, paged KV-cache, speculative decoding | Done (4/4) |
| **7 — Diagnostics & Interpretability** | Logprobs, hook system, logit lens, SAE integration, LoRA adapters | In Progress (1/5) |
-| **8 — Model Expansion** | MLA attention, ALiBi, SmolLM3, Gemma 4, Mixture of Experts | Planned (0/5) |
+| **8 — Model Expansion** | MLA attention, ALiBi, SmolLM3, Gemma 4, Mixture of Experts, Mamba-3 | In Progress (2/6) |
| **9 — Production Serving** | Continuous batching, prefix sharing, advanced scheduling, rate limiting, metrics & tracing | Planned (0/5) |
See [docs/ROADMAP.md](docs/ROADMAP.md) for detailed steps, dependencies, and milestones.
diff --git a/benchmarks/DotLLM.Benchmarks/CudaInferenceBenchmarks.cs b/benchmarks/DotLLM.Benchmarks/CudaInferenceBenchmarks.cs
new file mode 100644
index 00000000..683d837d
--- /dev/null
+++ b/benchmarks/DotLLM.Benchmarks/CudaInferenceBenchmarks.cs
@@ -0,0 +1,246 @@
+using System.Runtime.CompilerServices;
+using BenchmarkDotNet.Attributes;
+using DotLLM.Benchmarks.Columns;
+using DotLLM.Core.Attention;
+using DotLLM.Core.Configuration;
+using DotLLM.Core.Models;
+using DotLLM.Engine;
+using DotLLM.HuggingFace;
+using DotLLM.Models.Gguf;
+using DotLLM.Tokenizers.Bpe;
+
+namespace DotLLM.Benchmarks;
+
+///
+/// CUDA variant of the end-to-end inference benchmark. Mirrors
+/// but always loads the model on GPU 0 via
+/// CudaTransformerModel.LoadFromGguf. Skips with an informative error
+/// if no CUDA driver/device is present, so CI-on-CPU does not break.
+///
+/// Runs the same prompt + max-tokens defaults as the CPU variant; the metrics
+/// key is prefixed Cuda_ to avoid collision with CPU runs.
+///
+[SimpleJob(warmupCount: 2, iterationCount: 5)]
+public class CudaInferenceBenchmarks
+{
+ private static readonly Dictionary s_models = new()
+ {
+ // Q4_K_M is the end-to-end-verified quantization on CUDA (see
+ // CudaLogitComparisonTest.CompareLogits_PrefillAndDecode_Q4KM).
+ [BenchmarkModel.SmolLM_135M] = ("QuantFactory/SmolLM-135M-GGUF", "SmolLM-135M.Q4_K_M.gguf", 84),
+ [BenchmarkModel.Llama32_1B] = ("bartowski/Llama-3.2-1B-Instruct-GGUF", "Llama-3.2-1B-Instruct-Q4_K_M.gguf", 800),
+ [BenchmarkModel.Llama32_3B] = ("bartowski/Llama-3.2-3B-Instruct-GGUF", "Llama-3.2-3B-Instruct-Q4_K_M.gguf", 2100),
+ };
+
+ private const string DefaultPrompt = "The capital of France is";
+ private const int DefaultMaxTokens = 20;
+
+ /// Model to benchmark. Use --filter *SmolLM* etc. to select one.
+ [Params(BenchmarkModel.SmolLM_135M)]
+ public BenchmarkModel Model { get; set; }
+
+ private GgufFile _gguf = null!;
+ private IModel _model = null!;
+ private BpeTokenizer _tokenizer = null!;
+ private TextGenerator _generator = null!;
+ private string _modelPath = null!;
+ private string _prompt = DefaultPrompt;
+ private int _maxTokens = DefaultMaxTokens;
+ private string _metricsKey = null!;
+
+ private readonly List _timings = new();
+
+ [GlobalSetup]
+ public void Setup()
+ {
+ // Gate-check: CUDA must be available. Throw a clear message rather than
+ // letting a later P/Invoke fail with DllNotFoundException.
+ if (!ProbeCudaAvailable())
+ {
+ throw new InvalidOperationException(
+ "CUDA not available on this system (no driver or no GPU). " +
+ "Skip CudaInferenceBenchmarks by filtering, e.g. --filter '*InferenceBenchmarks*'.");
+ }
+
+ var envModelPath = Environment.GetEnvironmentVariable("DOTLLM_BENCH_MODEL_PATH");
+ if (!string.IsNullOrEmpty(envModelPath) && File.Exists(envModelPath))
+ {
+ _modelPath = envModelPath;
+ _metricsKey = "Cuda_" + Path.GetFileNameWithoutExtension(envModelPath);
+ Console.WriteLine($"*** Model override active: {_metricsKey} ***");
+ Console.WriteLine($" Path: {envModelPath}");
+ }
+ else
+ {
+ var (repoId, filename, approxMB) = s_models[Model];
+ // Prefer local model cache (CLI convention: ~/.dotllm/models//),
+ // fall back to benchmark test-cache and finally HF download.
+ _modelPath = ResolveModelPath(repoId, filename, approxMB);
+ _metricsKey = "Cuda_" + Model.ToString();
+ }
+
+ var envPrompt = Environment.GetEnvironmentVariable("DOTLLM_BENCH_PROMPT");
+ if (!string.IsNullOrEmpty(envPrompt))
+ _prompt = envPrompt;
+
+ var envMaxTokens = Environment.GetEnvironmentVariable("DOTLLM_BENCH_MAX_TOKENS");
+ if (!string.IsNullOrEmpty(envMaxTokens) && int.TryParse(envMaxTokens, out var parsedTokens))
+ _maxTokens = parsedTokens;
+
+ var promptPreview = _prompt.Length > 60 ? _prompt[..60] + "..." : _prompt;
+ Console.WriteLine($"Prompt: \"{promptPreview}\", MaxTokens: {_maxTokens}");
+
+ _gguf = GgufFile.Open(_modelPath);
+ var config = GgufModelConfigExtractor.Extract(_gguf.Metadata);
+ _tokenizer = GgufBpeTokenizerFactory.Load(_gguf.Metadata);
+
+ int gpuId = 0;
+ var envGpuId = Environment.GetEnvironmentVariable("DOTLLM_BENCH_GPU_ID");
+ if (!string.IsNullOrEmpty(envGpuId))
+ int.TryParse(envGpuId, out gpuId);
+
+ var kvFactory = LoadGpuModel(_gguf, config, gpuId);
+ _generator = new TextGenerator(_model, _tokenizer, kvFactory);
+ }
+
+ ///
+ /// Separated into its own method with
+ /// so the JIT only resolves DotLLM.Cuda types (and their native cublas/nvcuda
+ /// dependencies) when actually called — not when Setup() is compiled.
+ ///
+ [MethodImpl(MethodImplOptions.NoInlining)]
+ private Func LoadGpuModel(GgufFile gguf, ModelConfig config, int gpuId)
+ {
+ try
+ {
+ var cudaModel = Cuda.CudaTransformerModel.LoadFromGguf(gguf, config, gpuId);
+ _model = cudaModel;
+
+ var device = Cuda.CudaDevice.GetDevice(gpuId);
+ Console.WriteLine($"Device: GPU ({device})");
+ if (!string.IsNullOrEmpty(cudaModel.VramWarning))
+ Console.WriteLine($"VRAM: {cudaModel.VramWarning}");
+
+ return (_, size) => cudaModel.CreateKvCache(size);
+ }
+ catch (DllNotFoundException ex)
+ {
+ throw new InvalidOperationException(
+ $"CUDA libraries not found. Install CUDA Toolkit/driver. ({ex.Message})", ex);
+ }
+ }
+
+ [MethodImpl(MethodImplOptions.NoInlining)]
+ private static bool ProbeCudaAvailable()
+ {
+ try { return Cuda.CudaDevice.IsAvailable(); }
+ catch { return false; }
+ }
+
+ [Benchmark(Description = "CUDA E2E inference (prefill + decode)")]
+ public InferenceResponse Inference()
+ {
+ var options = new InferenceOptions
+ {
+ Temperature = 0f, // greedy
+ MaxTokens = _maxTokens
+ };
+
+ var response = _generator.Generate(_prompt, options);
+ _timings.Add(response.Timings);
+ return response;
+ }
+
+ [GlobalCleanup]
+ public void Cleanup()
+ {
+ if (_timings.Count > 0)
+ {
+ var prefillTokPerSecAll = _timings.Select(t => t.PrefillTokensPerSec).ToArray();
+ var decodeTokPerSecAll = _timings.Select(t => t.DecodeTokensPerSec).ToArray();
+ var prefillMsAll = _timings.Select(t => t.PrefillTimeMs).ToArray();
+ var decodeMsAll = _timings.Select(t => t.DecodeTimeMs).ToArray();
+
+ var prefillSorted = prefillTokPerSecAll.OrderBy(v => v).ToList();
+ var decodeSorted = decodeTokPerSecAll.OrderBy(v => v).ToList();
+ var prefillMsSorted = prefillMsAll.OrderBy(v => v).ToList();
+ var decodeMsSorted = decodeMsAll.OrderBy(v => v).ToList();
+
+ var metrics = new InferenceMetricsFile(
+ MedianPrefillTokPerSec: Median(prefillSorted),
+ MedianDecodeTokPerSec: Median(decodeSorted),
+ MedianPrefillMs: Median(prefillMsSorted),
+ MedianDecodeMs: Median(decodeMsSorted),
+ PrefillTokenCount: _timings[0].PrefillTokenCount,
+ DecodeTokenCount: _timings[0].DecodeTokenCount,
+ Iterations: _timings.Count,
+ BestPrefillTokPerSec: prefillTokPerSecAll.Max(),
+ BestDecodeTokPerSec: decodeTokPerSecAll.Max(),
+ BestPrefillMs: prefillMsAll.Min(),
+ BestDecodeMs: decodeMsAll.Min(),
+ DecodeCv: Cv(decodeTokPerSecAll),
+ PrefillCv: Cv(prefillTokPerSecAll),
+ AllDecodeTokPerSec: decodeTokPerSecAll,
+ AllPrefillTokPerSec: prefillTokPerSecAll,
+ AllDecodeMs: decodeMsAll,
+ AllPrefillMs: prefillMsAll);
+
+ InferenceMetricsFile.Write(_metricsKey, metrics);
+
+ Console.WriteLine($"[{_metricsKey}] prefill={Median(prefillSorted):F1} tok/s ({metrics.PrefillTokenCount} tok), " +
+ $"decode={Median(decodeSorted):F1} tok/s ({metrics.DecodeTokenCount} tok), n={_timings.Count}");
+ }
+
+ _model?.Dispose();
+ _gguf?.Dispose();
+ }
+
+ private static double Median(List sorted)
+ {
+ int n = sorted.Count;
+ if (n == 0) return 0;
+ if (n % 2 == 1) return sorted[n / 2];
+ return (sorted[n / 2 - 1] + sorted[n / 2]) / 2.0;
+ }
+
+ private static double StdDev(double[] values)
+ {
+ if (values.Length < 2) return 0;
+ double mean = values.Average();
+ double sumSq = values.Sum(v => (v - mean) * (v - mean));
+ return Math.Sqrt(sumSq / (values.Length - 1));
+ }
+
+ private static double Cv(double[] values)
+ {
+ if (values.Length < 2) return 0;
+ double mean = values.Average();
+ if (mean == 0) return 0;
+ return StdDev(values) / mean;
+ }
+
+ ///
+ /// Looks in the CLI model cache (~/.dotllm/models/<repo>/<file>)
+ /// before falling back to the benchmark test-cache + HF download.
+ ///
+ private static string ResolveModelPath(string repoId, string filename, int approxMB)
+ {
+ string home = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
+
+ string cliCached = Path.Combine(
+ home, ".dotllm", "models",
+ repoId.Replace('/', Path.DirectorySeparatorChar), filename);
+ if (File.Exists(cliCached))
+ return cliCached;
+
+ string benchCacheDir = Path.Combine(home, ".dotllm", "test-cache");
+ string benchCached = Path.Combine(benchCacheDir,
+ repoId.Replace('/', Path.DirectorySeparatorChar), filename);
+ if (File.Exists(benchCached))
+ return benchCached;
+
+ Console.WriteLine($"Downloading {repoId}/{filename} (~{approxMB} MB)...");
+ using var downloader = new HuggingFaceDownloader();
+ return downloader.DownloadFileAsync(repoId, filename, benchCacheDir).GetAwaiter().GetResult();
+ }
+}
diff --git a/benchmarks/DotLLM.Benchmarks/Lora/LoraDeltaOverheadBenchmark.cs b/benchmarks/DotLLM.Benchmarks/Lora/LoraDeltaOverheadBenchmark.cs
new file mode 100644
index 00000000..9dd720f9
--- /dev/null
+++ b/benchmarks/DotLLM.Benchmarks/Lora/LoraDeltaOverheadBenchmark.cs
@@ -0,0 +1,123 @@
+using System.Runtime.InteropServices;
+using BenchmarkDotNet.Attributes;
+using DotLLM.Core.Lora;
+using DotLLM.Cpu.Kernels;
+
+namespace DotLLM.Benchmarks.Lora;
+
+///
+/// Phase 4d.3 — Measures LoRA delta overhead vs the bare base projection.
+/// Baseline = a single F32 GEMM at TinyLlama-1.1B q_proj shapes
+/// (hidden=2048, q_out=2048, seq=128 typical prefill chunk).
+/// LoRA path = baseline + scale × (x · B) · A at r=16.
+/// Target: <5% overhead on the bare projection.
+///
+///
+/// We benchmark at the kernel level (no model load) because the spec target
+/// is the additional cost of the delta itself, and a kernel bench is fully
+/// reproducible without checkpoint download. The macro-bench against a real
+/// TinyLlama checkpoint is tracked as a follow-up — once a public checkpoint
+/// path is wired into the bench harness, replace this file with a
+/// model-level forward-pass bench.
+///
+[MemoryDiagnoser]
+[ShortRunJob]
+public unsafe class LoraDeltaOverheadBenchmark
+{
+ /// Sequence length (prefill chunk size).
+ [Params(1, 128)]
+ public int SeqLen { get; set; }
+
+ /// LoRA rank.
+ [Params(16)]
+ public int Rank { get; set; }
+
+ // TinyLlama q_proj shape.
+ private const int HiddenSize = 2048;
+ private const int OutputDim = 2048;
+
+ private nint _xPtr;
+ private nint _yBasePtr;
+ private nint _yLoraPtr;
+ private nint _wPtr; // base weight [OutputDim, HiddenSize]
+ private nint _bPtr; // LoRA B [Rank, HiddenSize]
+ private nint _aPtr; // LoRA A [OutputDim, Rank]
+
+ [GlobalSetup]
+ public void Setup()
+ {
+ var rng = new Random(123);
+
+ _xPtr = AllocAligned(SeqLen * HiddenSize);
+ _yBasePtr = AllocAligned(SeqLen * OutputDim);
+ _yLoraPtr = AllocAligned(SeqLen * OutputDim);
+ _wPtr = AllocAligned(OutputDim * HiddenSize);
+ _bPtr = AllocAligned(Rank * HiddenSize);
+ _aPtr = AllocAligned(OutputDim * Rank);
+
+ FillRandom((float*)_xPtr, SeqLen * HiddenSize, rng, 0.05f);
+ FillRandom((float*)_wPtr, OutputDim * HiddenSize, rng, 0.05f);
+ FillRandom((float*)_bPtr, Rank * HiddenSize, rng, 0.05f);
+ FillRandom((float*)_aPtr, OutputDim * Rank, rng, 0.05f);
+ }
+
+ [GlobalCleanup]
+ public void Cleanup()
+ {
+ FreeAligned(_xPtr);
+ FreeAligned(_yBasePtr);
+ FreeAligned(_yLoraPtr);
+ FreeAligned(_wPtr);
+ FreeAligned(_bPtr);
+ FreeAligned(_aPtr);
+ }
+
+ /// Baseline: only the base GEMM projection.
+ [Benchmark(Baseline = true)]
+ public void BaseProjectionOnly()
+ {
+ // C[N, M] = B[N, K] × A[M, K]^T, so y = x · w^T.
+ MatMul.GemmF32((float*)_wPtr, (float*)_xPtr, (float*)_yBasePtr,
+ OutputDim, HiddenSize, SeqLen);
+ }
+
+ /// Base GEMM + F32 LoRA delta (Phase 4a path).
+ [Benchmark]
+ public void BasePlusLoraF32()
+ {
+ MatMul.GemmF32((float*)_wPtr, (float*)_xPtr, (float*)_yLoraPtr,
+ OutputDim, HiddenSize, SeqLen);
+ LoraDelta.Apply(
+ (float*)_xPtr, (float*)_bPtr, (float*)_aPtr, (float*)_yLoraPtr,
+ SeqLen, HiddenSize, OutputDim, Rank, scale: 0.5f);
+ }
+
+ /// Base GEMM + F16 LoRA delta (Phase 4d.1 path).
+ [Benchmark]
+ public void BasePlusLoraF16()
+ {
+ MatMul.GemmF32((float*)_wPtr, (float*)_xPtr, (float*)_yLoraPtr,
+ OutputDim, HiddenSize, SeqLen);
+ // Reinterpret existing F32 buffers as F16 for the dispatch test —
+ // we measure dispatch + dequant overhead, not the math (the test
+ // suite already verifies numerical parity).
+ LoraDelta.Apply(
+ (float*)_xPtr, (void*)_bPtr, (void*)_aPtr, (float*)_yLoraPtr,
+ SeqLen, HiddenSize, OutputDim, Rank, scale: 0.5f,
+ LoraWeightDType.F16, LoraWeightDType.F16);
+ }
+
+ private static nint AllocAligned(long elementCount)
+ => (nint)NativeMemory.AlignedAlloc((nuint)(elementCount * sizeof(float)), 64);
+
+ private static void FreeAligned(nint p)
+ {
+ if (p != 0) NativeMemory.AlignedFree((void*)p);
+ }
+
+ private static void FillRandom(float* p, long n, Random rng, float scale)
+ {
+ for (long i = 0; i < n; i++)
+ p[i] = ((float)rng.NextDouble() * 2f - 1f) * scale;
+ }
+}
diff --git a/benchmarks/DotLLM.Benchmarks/PrefillBenchmarks.cs b/benchmarks/DotLLM.Benchmarks/PrefillBenchmarks.cs
new file mode 100644
index 00000000..addb4c1e
--- /dev/null
+++ b/benchmarks/DotLLM.Benchmarks/PrefillBenchmarks.cs
@@ -0,0 +1,99 @@
+using BenchmarkDotNet.Attributes;
+using DotLLM.Core.Attention;
+using DotLLM.Core.Configuration;
+using DotLLM.Core.Models;
+using DotLLM.Engine.KvCache;
+using DotLLM.HuggingFace;
+using DotLLM.Models.Architectures;
+using DotLLM.Models.Gguf;
+
+namespace DotLLM.Benchmarks;
+
+///
+/// Pure-prefill benchmark (llama.cpp pp512 equivalent). Measures a single
+/// call on an N-token synthetic sequence
+/// with a fresh KV-cache, isolating the prefill kernel path from tokenizer,
+/// sampling, and HTTP overhead.
+///
+///
+/// Env var DOTLLM_BENCH_MODEL_PATH overrides the default SmolLM-135M model.
+/// Env var DOTLLM_BENCH_PREFILL_TOKENS overrides the prompt length (default 512).
+///
+[SimpleJob(warmupCount: 2, iterationCount: 5)]
+public unsafe class PrefillBenchmarks
+{
+ private const string DefaultModelRepo = "QuantFactory/SmolLM-135M-GGUF";
+ private const string DefaultModelFile = "SmolLM-135M.Q8_0.gguf";
+ private const int DefaultModelSizeMB = 145;
+
+ private GgufFile _gguf = null!;
+ private IModel _model = null!;
+ private IKvCache _kvCache = null!;
+ private int[] _tokenIds = null!;
+ private int[] _positions = null!;
+ private int _prefillTokens;
+
+ [GlobalSetup]
+ public void Setup()
+ {
+ var envTokens = Environment.GetEnvironmentVariable("DOTLLM_BENCH_PREFILL_TOKENS");
+ _prefillTokens = !string.IsNullOrEmpty(envTokens) && int.TryParse(envTokens, out var parsed)
+ ? parsed
+ : 512;
+
+ var envModelPath = Environment.GetEnvironmentVariable("DOTLLM_BENCH_MODEL_PATH");
+ string modelPath = !string.IsNullOrEmpty(envModelPath) && File.Exists(envModelPath)
+ ? envModelPath
+ : DownloadModel(DefaultModelRepo, DefaultModelFile, DefaultModelSizeMB);
+
+ _gguf = GgufFile.Open(modelPath);
+ var config = GgufModelConfigExtractor.Extract(_gguf.Metadata);
+ _model = TransformerModel.LoadFromGguf(_gguf, config, ThreadingConfig.Auto);
+
+ _tokenIds = new int[_prefillTokens];
+ _positions = new int[_prefillTokens];
+ var rng = new Random(42);
+ int vocab = config.VocabSize;
+ for (int i = 0; i < _prefillTokens; i++)
+ {
+ _tokenIds[i] = rng.Next(1, Math.Min(vocab, 32000));
+ _positions[i] = i;
+ }
+
+ _kvCache = new SimpleKvCache(config.NumLayers, config.NumKvHeads, config.HeadDim, _prefillTokens);
+
+ Console.WriteLine($"Model: {Path.GetFileName(modelPath)} ({config.Architecture})");
+ Console.WriteLine($"Prefill tokens: {_prefillTokens}");
+ Console.WriteLine($"Device: CPU ({ThreadingConfig.Auto.EffectiveThreadCount} threads)");
+ }
+
+ [Benchmark(Description = "Prefill forward pass (fresh KV-cache)")]
+ public void Prefill()
+ {
+ // Rollback to 0 so every iteration measures a true prefill from position 0.
+ _kvCache.Rollback(0);
+ using var logits = _model.Forward(_tokenIds, _positions, deviceId: -1, _kvCache);
+ }
+
+ [GlobalCleanup]
+ public void Cleanup()
+ {
+ _kvCache?.Dispose();
+ _model?.Dispose();
+ _gguf?.Dispose();
+ }
+
+ private static string DownloadModel(string repoId, string filename, int approxMB)
+ {
+ string cacheDir = Path.Combine(
+ Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
+ ".dotllm", "test-cache");
+ string cachedPath = Path.Combine(cacheDir, repoId.Replace('/', Path.DirectorySeparatorChar), filename);
+ if (File.Exists(cachedPath))
+ return cachedPath;
+
+ Console.WriteLine($"Downloading {repoId}/{filename} (~{approxMB} MB)...");
+ using var downloader = new HuggingFaceDownloader();
+ return downloader.DownloadFileAsync(repoId, filename, cacheDir).GetAwaiter().GetResult();
+ }
+}
diff --git a/benchmarks/DotLLM.Benchmarks/Profile/CudaDecodeProfile.cs b/benchmarks/DotLLM.Benchmarks/Profile/CudaDecodeProfile.cs
new file mode 100644
index 00000000..ce3041ba
--- /dev/null
+++ b/benchmarks/DotLLM.Benchmarks/Profile/CudaDecodeProfile.cs
@@ -0,0 +1,355 @@
+using System.Diagnostics;
+using DotLLM.Core.Attention;
+using DotLLM.Core.Configuration;
+using DotLLM.Cuda;
+using DotLLM.HuggingFace;
+using DotLLM.Models.Gguf;
+using DotLLM.Tokenizers.Bpe;
+
+namespace DotLLM.Benchmarks.Profile;
+
+///
+/// Standalone profiler for the CUDA decode forward pass. Measures
+///
+/// wall_ms — Stopwatch around CudaTransformerModel.Forward (includes
+/// host dispatch + stream sync + final D2H memcpy of FP32 logits).
+/// gpu_ms — cuEventElapsedTime between the first and last kernel of
+/// the launch sequence (pure GPU wallclock; excludes the cost of
+/// cuStreamSynchronize returning to host).
+/// overhead — wall_ms − gpu_ms; bounds the host-side dispatch + sync round-trip.
+///
+/// The ratio gpu_ms / wall_ms is the single decision input for whether
+/// CUDA Graphs (collapses host dispatch into one packet submission) is the right
+/// next step. Below ~70 % means launches dominate; above ~85 % means kernels
+/// themselves are the bottleneck and fusion / better algorithms matter more.
+///
+internal static class CudaDecodeProfile
+{
+ private const string DefaultRepoId = "QuantFactory/SmolLM-135M-GGUF";
+ private const string DefaultFilename = "SmolLM-135M.Q4_K_M.gguf";
+ private const string DefaultPrompt = "The capital of France is Paris. The capital of Germany is Berlin. The capital of Italy is Rome. The capital of Spain is Madrid. The capital of Portugal is";
+ private const int DefaultPrefillTokens = 96;
+ private const int DefaultDecodeTokens = 200;
+ private const int DefaultWarmupTokens = 16;
+
+ public static int Run(string[] args)
+ {
+ if (!CudaDevice.IsAvailable())
+ {
+ Console.Error.WriteLine("CUDA device not available — install driver/toolkit and retry.");
+ return 2;
+ }
+
+ // --graph : (legacy, now redundant) force CUDA-Graphs decode path
+ // --no-graph : force eager decode (overrides the new default-on)
+ // --compare : run BOTH eager and graph back-to-back, side-by-side report
+ // --no-profiling : measure eager wall WITHOUT cuEventRecord overhead so the
+ // comparison vs graph is true-apples-to-apples
+ // (default) : graph capture (mirrors CudaTransformerModel default-on; flipped
+ // in commit "CUDA: graph capture default-ON for all k").
+ bool noGraph = args.Contains("--no-graph") ||
+ Environment.GetEnvironmentVariable("DOTLLM_DISABLE_GRAPH_CAPTURE") == "1";
+ bool useGraph = !noGraph;
+ bool compare = args.Contains("--compare");
+ bool noProfiling = args.Contains("--no-profiling");
+ // --kv-quant : run with the mixed-precision quantized KV cache
+ // (Q8_0 stored region + 16-row FP16 window). Validates the
+ // quantized-cache CUDA Graphs decode path lands the same
+ // ~2× speedup as the FP16 cache.
+ bool kvQuant = args.Contains("--kv-quant");
+ KvCacheConfig kvCfg = kvQuant
+ ? new KvCacheConfig(KvCacheDType.Q8_0, KvCacheDType.Q8_0, MixedPrecisionWindowSize: 16)
+ : KvCacheConfig.Default;
+
+ string modelPath = ResolveModelPath();
+ Console.WriteLine($"Model: {modelPath}");
+ Console.WriteLine($"Device: {CudaDevice.GetDevice(0)}");
+
+ using var gguf = GgufFile.Open(modelPath);
+ var config = GgufModelConfigExtractor.Extract(gguf.Metadata);
+ var tokenizer = GgufBpeTokenizerFactory.Load(gguf.Metadata);
+ int[] promptTokens = tokenizer.Encode(DefaultPrompt).ToArray();
+ Console.WriteLine($"Prompt tokens: {promptTokens.Length} (using up to {DefaultPrefillTokens})");
+
+ if (compare)
+ {
+ string cacheLabel = kvQuant ? "KV-quant Q8_0 + W16 window" : "FP16 KV";
+ Console.WriteLine($"Cache config: {cacheLabel}");
+ Console.WriteLine();
+ Console.WriteLine("════════ EAGER (with per-category profiling) ════════");
+ var eagerResult = RunOne(gguf, config, promptTokens, useGraphCapture: false, disableProfiling: false, kvCfg: kvCfg);
+ Console.WriteLine();
+ Console.WriteLine("════════ EAGER (no profiling — true wall) ════════");
+ var eagerCleanResult = RunOne(gguf, config, promptTokens, useGraphCapture: false, disableProfiling: true, kvCfg: kvCfg);
+ Console.WriteLine();
+ Console.WriteLine("════════ CUDA GRAPH (capture+replay) ════════");
+ var graphResult = RunOne(gguf, config, promptTokens, useGraphCapture: true, disableProfiling: true, kvCfg: kvCfg);
+ Console.WriteLine();
+ Console.WriteLine("════════ SUMMARY ════════");
+ Console.WriteLine($" cache config = {cacheLabel}");
+ Console.WriteLine($" eager+profile median tok/s = {1000.0 / eagerResult.MedianWallMs,7:F1} (wall {eagerResult.MedianWallMs:F2} ms)");
+ Console.WriteLine($" eager (clean) median tok/s = {1000.0 / eagerCleanResult.MedianWallMs,7:F1} (wall {eagerCleanResult.MedianWallMs:F2} ms)");
+ Console.WriteLine($" graph median tok/s = {1000.0 / graphResult.MedianWallMs,7:F1} (wall {graphResult.MedianWallMs:F2} ms)");
+ Console.WriteLine($" speedup vs clean eager = {eagerCleanResult.MedianWallMs / graphResult.MedianWallMs,7:F2}×");
+ return 0;
+ }
+
+ var single = RunOne(gguf, config, promptTokens, useGraphCapture: useGraph, disableProfiling: noProfiling, kvCfg: kvCfg);
+ return 0;
+ }
+
+ private readonly struct DecodeStats
+ {
+ public required double MedianWallMs { get; init; }
+ public required double MedianGpuMs { get; init; }
+ }
+
+ private static DecodeStats RunOne(GgufFile gguf, DotLLM.Core.Models.ModelConfig config,
+ int[] promptTokens, bool useGraphCapture,
+ bool disableProfiling = false,
+ KvCacheConfig kvCfg = default)
+ {
+ using var model = CudaTransformerModel.LoadFromGguf(gguf, config, deviceId: 0);
+ model.UseGraphCapture = useGraphCapture;
+
+ int prefillLen = Math.Min(promptTokens.Length, DefaultPrefillTokens);
+ int[] prefill = promptTokens[..prefillLen];
+
+ int kvCapacity = prefillLen + DefaultWarmupTokens + DefaultDecodeTokens + 8;
+ using var kv = kvCfg.IsQuantized
+ ? (IKvCache)model.CreateKvCache(kvCapacity, kvCfg)
+ : (IKvCache)model.CreateKvCache(kvCapacity);
+
+ int[] prefillPositions = new int[prefillLen];
+ for (int i = 0; i < prefillLen; i++) prefillPositions[i] = i;
+ using (var _ = model.Forward(prefill, prefillPositions, deviceId: 0, kv))
+ { }
+
+ int nextPos = prefillLen;
+ int currentToken = promptTokens[prefillLen - 1];
+
+ int[] tokBuf = new int[1];
+ int[] posBuf = new int[1];
+
+ for (int i = 0; i < DefaultWarmupTokens; i++)
+ {
+ tokBuf[0] = currentToken;
+ posBuf[0] = nextPos;
+ using var t = model.Forward(tokBuf, posBuf, deviceId: 0, kv);
+ currentToken = ArgmaxFirstRow(t);
+ nextPos++;
+ }
+
+ // Per-category profiling is disabled on the graph path (event-record between
+ // launches breaks stream capture). For the graph path we still get wall-clock
+ // and a single GPU bracket via the cuEvent on entry/exit of the graph launch.
+ int categoryCount = CudaTransformerModel.ProfileCategoryCount;
+ var wallTimes = new double[DefaultDecodeTokens];
+ var gpuTimes = new double[DefaultDecodeTokens];
+ var categoryTimes = new double[categoryCount, DefaultDecodeTokens];
+ var sw = new Stopwatch();
+
+ if (!useGraphCapture && !disableProfiling)
+ {
+ // Eager: full per-category profiling.
+ model.ProfilingEnabled = true;
+ for (int i = 0; i < DefaultDecodeTokens; i++)
+ {
+ tokBuf[0] = currentToken;
+ posBuf[0] = nextPos;
+ sw.Restart();
+ using var t = model.Forward(tokBuf, posBuf, deviceId: 0, kv);
+ sw.Stop();
+ wallTimes[i] = sw.Elapsed.TotalMilliseconds;
+ gpuTimes[i] = model.LastGpuLaunchMs;
+ for (int c = 0; c < categoryCount; c++)
+ categoryTimes[c, i] = model.LastCategoryMs[c];
+ currentToken = ArgmaxFirstRow(t);
+ nextPos++;
+ }
+ }
+ else
+ {
+ // Graph, or eager with profiling disabled: wall only.
+ model.ProfilingEnabled = false;
+ for (int i = 0; i < DefaultDecodeTokens; i++)
+ {
+ tokBuf[0] = currentToken;
+ posBuf[0] = nextPos;
+ sw.Restart();
+ using var t = model.Forward(tokBuf, posBuf, deviceId: 0, kv);
+ sw.Stop();
+ wallTimes[i] = sw.Elapsed.TotalMilliseconds;
+ gpuTimes[i] = double.NaN;
+ currentToken = ArgmaxFirstRow(t);
+ nextPos++;
+ }
+ }
+
+ // Reporting: show per-category breakdown only when we captured it.
+ // hideCategoryBreakdown is semantically what Report's last arg means in practice.
+ bool hadProfiling = !useGraphCapture && !disableProfiling;
+ Report(wallTimes, gpuTimes, categoryTimes, prefillLen, kvCapacity,
+ config.NumLayers, config.HiddenSize,
+ useGraphCapture: useGraphCapture, hideCategory: !hadProfiling);
+
+ var sortedWall = (double[])wallTimes.Clone();
+ Array.Sort(sortedWall);
+ var sortedGpu = (double[])gpuTimes.Clone();
+ Array.Sort(sortedGpu);
+ return new DecodeStats { MedianWallMs = Median(sortedWall), MedianGpuMs = Median(sortedGpu) };
+ }
+
+ private static unsafe int ArgmaxFirstRow(DotLLM.Core.Tensors.ITensor logits)
+ {
+ // logits shape [1, vocab]; FP32; already on host after Forward()'s D2H
+ int n = checked((int)logits.Shape.ElementCount);
+ var span = new ReadOnlySpan((void*)logits.DataPointer, n);
+ int best = 0;
+ float v = span[0];
+ for (int i = 1; i < span.Length; i++)
+ {
+ if (span[i] > v) { v = span[i]; best = i; }
+ }
+ return best;
+ }
+
+ private static readonly string[] CategoryNames =
+ {
+ "Embed", "QkvProj", "Rope+Bias+QkNorm", "KvUpdate", "Attention",
+ "OProj", "Norm (rmsnorm/fused-add)", "MlpUp (gate+up)", "Swiglu",
+ "MlpDown", "LmHead", "Convert+ResidAdd"
+ };
+
+ private static void Report(double[] wall, double[] gpu, double[,] categoryTimes,
+ int prefillLen, int kvCapacity, int layers, int hidden,
+ bool useGraphCapture = false, bool hideCategory = false)
+ {
+ Array.Sort(wall);
+ var gpuSorted = (double[])gpu.Clone();
+ Array.Sort(gpuSorted);
+
+ double wallMedian = Median(wall);
+ double gpuMedian = Median(gpuSorted);
+ double wallP10 = Percentile(wall, 10);
+ double wallP90 = Percentile(wall, 90);
+ double gpuP10 = Percentile(gpuSorted, 10);
+ double gpuP90 = Percentile(gpuSorted, 90);
+ double wallMin = wall[0];
+ double gpuMin = gpuSorted[0];
+ double overhead = wallMedian - gpuMedian;
+ double gpuFraction = gpuMedian / wallMedian;
+ double tokPerSec = 1000.0 / wallMedian;
+
+ Console.WriteLine();
+ Console.WriteLine("──────── CUDA decode profile ────────");
+ string pathLabel = useGraphCapture ? "GRAPH" : (hideCategory ? "EAGER (no profiling)" : "EAGER");
+ Console.WriteLine($"Layers={layers} Hidden={hidden} Prefill={prefillLen} KvCapacity={kvCapacity} Path={pathLabel}");
+ Console.WriteLine($"Iterations: {wall.Length} timed (after {DefaultWarmupTokens} warmup)");
+ Console.WriteLine();
+ Console.WriteLine($" {"min",8} {"p10",8} {"p50",8} {"p90",8}");
+ Console.WriteLine($" wall ms {wallMin,8:F3} {wallP10,8:F3} {wallMedian,8:F3} {wallP90,8:F3}");
+ if (!hideCategory)
+ {
+ Console.WriteLine($" gpu ms {gpuMin,8:F3} {gpuP10,8:F3} {gpuMedian,8:F3} {gpuP90,8:F3}");
+ }
+ Console.WriteLine();
+ if (!hideCategory)
+ {
+ Console.WriteLine($" median wall − gpu = {overhead,7:F3} ms (host dispatch + sync + D2H)");
+ Console.WriteLine($" median gpu / wall = {gpuFraction,7:P1}");
+ }
+ Console.WriteLine($" median tok/s = {tokPerSec,7:F1}");
+ Console.WriteLine();
+
+ if (hideCategory)
+ {
+ if (useGraphCapture)
+ Console.WriteLine("Per-category breakdown disabled in graph mode (event-record between launches breaks stream capture).");
+ else
+ Console.WriteLine("Per-category breakdown disabled (run without --no-profiling for the full breakdown).");
+ Console.WriteLine("─────────────────────────────────────");
+ return;
+ }
+
+ // Per-category breakdown — sort by median time descending.
+ int categoryCount = categoryTimes.GetLength(0);
+ int n = categoryTimes.GetLength(1);
+ var medians = new double[categoryCount];
+ for (int c = 0; c < categoryCount; c++)
+ {
+ var col = new double[n];
+ for (int i = 0; i < n; i++) col[i] = categoryTimes[c, i];
+ Array.Sort(col);
+ medians[c] = Median(col);
+ }
+ var order = Enumerable.Range(0, categoryCount).OrderByDescending(c => medians[c]).ToArray();
+
+ Console.WriteLine("Per-category GPU time (median ms / token):");
+ Console.WriteLine($" {"category",-26} {"ms",8} {"%gpu",8}");
+ double sumCat = 0;
+ for (int k = 0; k < order.Length; k++)
+ {
+ int c = order[k];
+ double pct = medians[c] / gpuMedian * 100;
+ sumCat += medians[c];
+ Console.WriteLine($" {CategoryNames[c],-26} {medians[c],8:F3} {pct,7:F1}%");
+ }
+ double accountedPct = sumCat / gpuMedian * 100;
+ Console.WriteLine($" {"-- accounted --",-26} {sumCat,8:F3} {accountedPct,7:F1}%");
+ Console.WriteLine($" (unaccounted = event-record overhead between marks)");
+ Console.WriteLine();
+
+ if (gpuFraction < 0.70)
+ {
+ Console.WriteLine("VERDICT: launch-bound. CUDA Graphs is the highest-ROI fix");
+ Console.WriteLine(" (collapses ~400 launches/token into one packet submission).");
+ }
+ else if (gpuFraction > 0.85)
+ {
+ Console.WriteLine("VERDICT: kernel-bound. Top-3 categories above are the targets.");
+ }
+ else
+ {
+ Console.WriteLine("VERDICT: mixed. Both CUDA Graphs and kernel fusion would help.");
+ }
+ Console.WriteLine("─────────────────────────────────────");
+ }
+
+ private static double Median(double[] sorted)
+ {
+ int n = sorted.Length;
+ if (n == 0) return 0;
+ return n % 2 == 1 ? sorted[n / 2] : (sorted[n / 2 - 1] + sorted[n / 2]) / 2.0;
+ }
+
+ private static double Percentile(double[] sorted, double p)
+ {
+ if (sorted.Length == 0) return 0;
+ double rank = (p / 100.0) * (sorted.Length - 1);
+ int lo = (int)Math.Floor(rank);
+ int hi = (int)Math.Ceiling(rank);
+ if (lo == hi) return sorted[lo];
+ return sorted[lo] + (rank - lo) * (sorted[hi] - sorted[lo]);
+ }
+
+ private static string ResolveModelPath()
+ {
+ string envPath = Environment.GetEnvironmentVariable("DOTLLM_BENCH_MODEL_PATH") ?? "";
+ if (!string.IsNullOrEmpty(envPath) && File.Exists(envPath)) return envPath;
+
+ string home = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
+ string cli = Path.Combine(home, ".dotllm", "models",
+ DefaultRepoId.Replace('/', Path.DirectorySeparatorChar), DefaultFilename);
+ if (File.Exists(cli)) return cli;
+
+ string benchCacheDir = Path.Combine(home, ".dotllm", "test-cache");
+ string benchCached = Path.Combine(benchCacheDir,
+ DefaultRepoId.Replace('/', Path.DirectorySeparatorChar), DefaultFilename);
+ if (File.Exists(benchCached)) return benchCached;
+
+ Console.WriteLine($"Downloading {DefaultRepoId}/{DefaultFilename}...");
+ using var dl = new HuggingFaceDownloader();
+ return dl.DownloadFileAsync(DefaultRepoId, DefaultFilename, benchCacheDir).GetAwaiter().GetResult();
+ }
+}
diff --git a/benchmarks/DotLLM.Benchmarks/Program.cs b/benchmarks/DotLLM.Benchmarks/Program.cs
index a22425ee..411aebe0 100644
--- a/benchmarks/DotLLM.Benchmarks/Program.cs
+++ b/benchmarks/DotLLM.Benchmarks/Program.cs
@@ -1,9 +1,17 @@
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;
using DotLLM.Benchmarks.Columns;
+using DotLLM.Benchmarks.Profile;
+
+// Subcommands not understood by BenchmarkSwitcher are routed first.
+if (args.Length > 0 && args[0] == "profile-cuda-decode")
+{
+ return CudaDecodeProfile.Run(args[1..]);
+}
var config = ManualConfig.Create(DefaultConfig.Instance)
.AddColumn(new PrefillTokPerSecColumn())
.AddColumn(new DecodeTokPerSecColumn());
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);
+return 0;
diff --git a/benchmarks/DotLLM.Benchmarks/ThreadPoolDispatchBenchmarks.cs b/benchmarks/DotLLM.Benchmarks/ThreadPoolDispatchBenchmarks.cs
new file mode 100644
index 00000000..7f80a1e1
--- /dev/null
+++ b/benchmarks/DotLLM.Benchmarks/ThreadPoolDispatchBenchmarks.cs
@@ -0,0 +1,164 @@
+using System.Runtime.InteropServices;
+using BenchmarkDotNet.Attributes;
+using DotLLM.Cpu.Threading;
+
+namespace DotLLM.Benchmarks;
+
+///
+/// Microbenchmarks for coordination cost.
+/// Motivated by a PerfView kernel CPU profile showing ~76% of dotLLM CPU burned in
+/// the worker spin loop during decode. These benchmarks isolate the three cost
+/// components so we can tell whether the loss is per-dispatch barrier cost,
+/// inter-dispatch idle spin, or worker wake latency.
+///
+/// Benchmark design:
+///
+/// -
+///
+/// — Dispatch with a no-op kernel.
+/// Measures pure round-trip: wake N workers, execute empty fn, barrier.
+/// Decode gap between dispatches in real workloads is dominated by this.
+///
+///
+/// -
+///
+/// — Dispatch with ~1 µs of actual
+/// SIMD-shape work per thread (sum a fixed-size buffer). Mirrors one row
+/// of a SmolLM-135M-size decode matmul. The ratio of this vs
+/// tells us whether the pool is
+/// helping at this problem size or only hurting.
+///
+///
+/// -
+///
+/// — the same work on the caller
+/// thread only, no dispatch. Fixed-cost baseline.
+///
+///
+/// -
+///
+/// — 30 back-to-back small-work
+/// dispatches. Mirrors one SmolLM decode step (30 layers × ~1 matmul
+/// dispatch per layer). Lets us see whether batching amortises the
+/// spin-fallback cost or whether each dispatch pays it again.
+///
+///
+///
+///
+/// All timings reported by BenchmarkDotNet include barrier overhead —
+/// the point of the microbench is to make that overhead visible, not to hide
+/// it. Comparing 1-thread runs to N-thread runs at the same work size shows
+/// the dispatch floor.
+///
+[MemoryDiagnoser]
+[SimpleJob(warmupCount: 3, iterationCount: 10)]
+public unsafe class ThreadPoolDispatchBenchmarks : IDisposable
+{
+ ///
+ /// Per-thread work buffer length. 576 × 4 bytes matches a SmolLM-135M hidden
+ /// row (F32 view for a scalar reduction — we're measuring coordination cost,
+ /// not kernel quality, so a plain sum keeps the worker work simple and
+ /// predictable.)
+ ///
+ private const int WorkBufferLen = 576;
+
+ ///
+ /// Back-to-back dispatch count in .
+ /// 30 matches SmolLM-135M layer count; other Llama-family decode-layer loops
+ /// sit in a similar 20–40 range.
+ ///
+ private const int BurstCount = 30;
+
+ private ComputeThreadPool? _pool;
+ private float* _work;
+
+ [Params(2, 4, 8, 16, 32)]
+ public int Threads { get; set; }
+
+ [Params(DispatchMode.EventBased, DispatchMode.SpinWait)]
+ public DispatchMode Mode { get; set; }
+
+ [GlobalSetup]
+ public void Setup()
+ {
+ _pool = new ComputeThreadPool(Threads);
+ _pool.SetDispatchMode(Mode);
+
+ _work = (float*)NativeMemory.AlignedAlloc((nuint)(WorkBufferLen * sizeof(float)), 64);
+ var rng = new Random(42);
+ for (int i = 0; i < WorkBufferLen; i++)
+ _work[i] = rng.NextSingle();
+
+ // Warm the pool so the first measured iteration isn't paying for
+ // first-touch pin + event initialisation.
+ for (int i = 0; i < 8; i++)
+ _pool.Dispatch((nint)_work, &NoOpWorker);
+ }
+
+ [GlobalCleanup]
+ public void Cleanup()
+ {
+ NativeMemory.AlignedFree(_work);
+ _pool?.Dispose();
+ }
+
+ public void Dispose() => Cleanup();
+
+ /// Dispatch a no-op to every thread — pure barrier cost.
+ [Benchmark(Description = "Dispatch(no-op)")]
+ public void Dispatch_NoWork()
+ {
+ _pool!.Dispatch((nint)_work, &NoOpWorker);
+ }
+
+ /// Dispatch ~576-element sum per thread — barrier plus realistic row work.
+ [Benchmark(Description = "Dispatch(~1µs work)")]
+ public void Dispatch_SmallWork()
+ {
+ _pool!.Dispatch((nint)_work, &SmallWorkWorker);
+ }
+
+ /// Same small work, executed on the caller thread only (no dispatch).
+ [Benchmark(Description = "Single-thread same work")]
+ public float SingleThreaded_SmallWork()
+ {
+ return SumBuffer(_work, WorkBufferLen);
+ }
+
+ /// 30 back-to-back dispatches — one SmolLM-135M decode step's layer loop.
+ [Benchmark(Description = "30× Dispatch(small work)")]
+ public void DispatchBurst_DecodePattern()
+ {
+ var pool = _pool!;
+ for (int i = 0; i < BurstCount; i++)
+ pool.Dispatch((nint)_work, &SmallWorkWorker);
+ }
+
+ // ---- kernel functions (static, pointer-dispatched) ----
+
+ private static void NoOpWorker(nint context, int threadIdx, int threadCount)
+ {
+ // Intentionally empty — measures pure dispatch barrier.
+ _ = context; _ = threadIdx; _ = threadCount;
+ }
+
+ private static void SmallWorkWorker(nint context, int threadIdx, int threadCount)
+ {
+ float* buf = (float*)context;
+ int chunk = WorkBufferLen / threadCount;
+ int start = threadIdx * chunk;
+ int end = threadIdx == threadCount - 1 ? WorkBufferLen : start + chunk;
+ float sum = SumBuffer(buf + start, end - start);
+ // Prevent the JIT from eliding the sum. Writing back into the buffer
+ // is cheap and keeps the partition realistic.
+ if (sum == float.NegativeInfinity) buf[start] = sum;
+ }
+
+ private static float SumBuffer(float* buf, int len)
+ {
+ float s = 0f;
+ for (int i = 0; i < len; i++)
+ s += buf[i];
+ return s;
+ }
+}
diff --git a/docs/ATTENTION.md b/docs/ATTENTION.md
index bd1f22d3..fb3dbb3e 100644
--- a/docs/ATTENTION.md
+++ b/docs/ATTENTION.md
@@ -41,6 +41,15 @@ DeepSeek-V2/V3. Compresses KV into low-rank latent space.
Requires its own attention implementation with `LatentKvCache`.
+**CPU**: complete. Three phases coexist behind `MlaConfig` flags:
+- `MlaAttention.Execute` — Phase A naive expanded (per-head K_nope/V cache); the numerical oracle.
+- `MlaAttention.ExecuteLatent` — Phase B absorbed-form attention over latent `[c_kv, k_pe]` cache (`MlaLatentKvState`).
+- `MlaAttention.ExecuteLatentHybrid` — Phase C vLLM-style: prefill expands + MHA, decode runs absorbed.
+- All three handle low-rank Q (`q_a_proj` + `q_a_layernorm` + `q_b_proj`), low-rank KV with `kv_a_layernorm`, RoPE on the rope-only sub-dim, causal mask, and YaRN's `mscale²` softmax-scale multiplier (`MlaConfig.ComputeYarnSoftmaxScaleMultiplier`).
+- Verified end-to-end on tiny-random DeepSeek-V2/V3 fixtures and (gated by checkpoint availability) DeepSeek-V2-Lite real weights.
+
+**CUDA**: Phase A primitives landed (`CudaMlaAttention.Forward`, `attention_mla_f32` kernel, `mla_helpers.cu`, `CudaMlaWeights`, `CudaMlaKvCache`). F32 throughout for now, validated against the CPU oracle within FP16 noise. **Not yet wired into `CudaTransformerModel.Forward`** — that wiring blocks on the CUDA MoE FFN port (DeepSeek-V2/V3 layers are MLA + MoE FFN, and the FFN GPU path doesn't exist). Phase B/C and FP16/quantized weight paths are deferred follow-ups.
+
## IAttentionStrategy — Kernel Selection
```
diff --git a/docs/CUDA.md b/docs/CUDA.md
index 04c8bc2b..b330e864 100644
--- a/docs/CUDA.md
+++ b/docs/CUDA.md
@@ -640,11 +640,25 @@ All kernels compiled to PTX, loaded via `cuModuleLoadData`, launched via `cuLaun
**Quantized GEMV (decode path — operate directly on quantized weights):**
-| Kernel | File | Function Name |
-|---|---|---|
-| Q8_0 GEMV | `quantized_gemv.cu` | `quantized_gemv_q8_0` |
-| Q4_K GEMV | `quantized_gemv.cu` | `quantized_gemv_q4_k` |
-| Q6_K GEMV | `quantized_gemv.cu` | `quantized_gemv_q6_k` |
+| Kernel | File | Function Name | Notes |
+|---|---|---|---|
+| Q8_0 GEMV | `quantized_gemv.cu` | `quantized_gemv_q8_0` | FP fmuladd, legacy |
+| Q4_K GEMV | `quantized_gemv.cu` | `quantized_gemv_q4_k` | FP fmuladd, legacy |
+| Q5_K GEMV | `quantized_gemv.cu` | `quantized_gemv_q5_k` | FP fmuladd, legacy |
+| Q6_K GEMV | `quantized_gemv.cu` | `quantized_gemv_q6_k` | FP fmuladd, legacy |
+| Q4_K MMQ | `quantized_gemv_mmq.cu` | `quantized_gemv_q4_k_mmq` | dp4a, 4 rows/block; default for k<1024 |
+| Q5_K MMQ | `quantized_gemv_mmq.cu` | `quantized_gemv_q5_k_mmq` | dp4a, 4 rows/block |
+| Q6_K MMQ | `quantized_gemv_mmq.cu` | `quantized_gemv_q6_k_mmq` | dp4a, 4 rows/block; lifts LmHead in Q4_K_M GGUFs |
+| Q{4,5,6}_K MMVQ-large | `quantized_gemv_mmq.cu` | `quantized_gemv_q*_k_mmvq_large` | dp4a, 1 row/block × 128 threads; default for k≥1024 (paired with `_preq` variants) |
+| Q{4,5,6}_K MMQ/MMVQ `_preq` | `quantized_gemv_mmq.cu` | `*_preq` suffix | Read pre-quantized x from scratch; skips Stage 1 input quant |
+| Pre-Q8_1 input quant | `quantize_x.cu` | `quantize_x_to_q8_1` | Quantizes activation x[k] once per fused-GEMV-input — feeds the `_preq` variants. Auto-engages for k≥1024 |
+
+**Embedding lookup (per-row dequant, no full-table FP16):**
+
+| Kernel | File | Function Name | Notes |
+|---|---|---|---|
+| F32 / F16 / Q8_0 | `embedding.cu` | `embedding_lookup_*` | Existing fast paths |
+| Q4_K / Q5_K / Q6_K | `embedding.cu` | `embedding_lookup_q{4,5,6}_k_f16` | Per-row dequant, llama.cpp `get_rows_q*_K` pattern. Saves ~1.16 GiB on Qwen3-8B vocab×hidden |
**Conversion:**
@@ -691,8 +705,13 @@ This is well-proven — llama.cpp, vLLM, and every CUDA inference engine uses th
## Future Work
+- **CUDA MoE FFN port**: top-k routing + per-expert grouped-GEMM on GPU. CPU has `MoeSwiGluMlp`; CUDA equivalent doesn't exist yet. **Concrete blocker for end-to-end DeepSeek-V2/V3 on CUDA** (MLA attention Phase 1 primitives landed; they need a matching MoE FFN to complete the layer forward).
+- **MLA Phase B + C** (CUDA decode efficiency): latent KV cache + W_UK absorption (`MlaAttention.ExecuteLatent`/`ExecuteLatentHybrid` CPU equivalents). Phase A naive expanded forward is already in tree (`CudaMlaAttention.Forward`).
+- **MLA FP16/quantized weight paths**: current Phase A is F32 throughout. FP16 follow-up; quantized extends `Project` patterns from the GQA path.
- **Flash Attention**: replace naive attention kernel with tiled flash attention (shared memory, online softmax). Full Tensor Core access via `wmma` intrinsics in PTX.
-- **Fused quantized GEMM for prefill**: Marlin-style dequant-in-register or llama.cpp MMQ-style fused matmul to eliminate per-projection dequant→scratch overhead during prefill. Decode path already uses custom quantized GEMV kernels (Q8_0, Q4_K, Q6_K).
+- **Fused quantized GEMM for prefill**: Marlin-style dequant-in-register. Decode is now MMQ + MMVQ-large + pre-Q8_1 (Qwen3-8B Q4_K_M decode hits 33 tok/s eager on RTX 3060 — inside llama.cpp's reported range); prefill still uses dequant→cuBLAS HGEMM.
+- **Continuous batching scheduler** (engine-layer prerequisite for tensor-core mma kernel value — see `docs/perf/MMA_BATCHED_MMQ.md` for the design analysis).
+- **Tensor-core (mma) batched MMQ**: only valuable once batched decode is the call shape. See `docs/perf/MMA_BATCHED_MMQ.md` for thresholds.
- **Multi-stream pipelining** (Step 32): overlap H2D transfer with compute across layers.
- **NCCL integration** (Step 51): multi-GPU tensor parallelism. NCCL is another system library — same P/Invoke pattern, no shared library needed.
- **Fatbin distribution**: ship pre-compiled SASS for common architectures to eliminate JIT overhead.
diff --git a/docs/GPU.md b/docs/GPU.md
index 2d82f607..3f2d07ec 100644
--- a/docs/GPU.md
+++ b/docs/GPU.md
@@ -192,34 +192,44 @@ See [KV_CACHE.md](KV_CACHE.md) for general KV-cache design (KV-cache quantizatio
Supported GGUF quantization formats for GPU inference:
-| Format | Embedding | Dequant→FP16 | Quantized GEMV | Notes |
-|--------|-----------|--------------|----------------|-------|
-| F32 | Yes | Yes | — | Converted to FP16 on upload |
-| F16 | Yes | — | — | Direct upload, used by cuBLAS directly |
-| Q8_0 | Yes | Yes | Yes | Best quality quantized format |
-| Q4_0 | No | Yes | No | Dequant-only (no custom GEMV kernel) |
-| Q5_0 | No | Yes | No | Dequant-only (no custom GEMV kernel) |
-| Q4_K | No | Yes | Yes | Good quality-to-size ratio |
-| Q5_K | No | Yes | No | Dequant-only (no custom GEMV kernel) |
-| Q6_K | No | Yes | Yes | High quality, larger than Q4_K |
-
-**Decode** uses custom quantized GEMV kernels (Q8_0, Q4_K, Q6_K) that operate directly on quantized weights — no dequantization needed. For formats without a custom GEMV kernel, the weight is dequantized on-the-fly into a scratch buffer and cuBLAS GEMV is used.
+| Format | Embedding | Dequant→FP16 | Quantized GEMV | MMQ (dp4a) | MMVQ-large + pre-Q8_1 | Notes |
+|--------|-----------|--------------|----------------|-----------|---------------------|-------|
+| F32 | Yes | Yes | — | — | — | Converted to FP16 on upload |
+| F16 | Yes | — | — | — | — | Direct upload, used by cuBLAS directly |
+| Q8_0 | Yes | Yes | Yes | — | — | Best quality quantized format |
+| Q4_0 | No | Yes | No | — | — | Dequant-only (no custom GEMV kernel) |
+| Q5_0 | No | Yes | No | — | — | Dequant-only (no custom GEMV kernel) |
+| Q4_K | **Yes (per-row)** | Yes | Yes | Yes | Yes (k≥1024) | Good quality-to-size ratio |
+| Q5_K | **Yes (per-row)** | Yes | Yes | Yes | Yes (k≥1024) | |
+| Q6_K | **Yes (per-row)** | Yes | Yes | Yes | Yes (k≥1024) | High quality, larger than Q4_K |
+
+**Decode dispatch** (per call, picked by `LaunchQuantizedGemvMmq`):
+- **k ≥ 1024 + Q4_K/Q5_K/Q6_K**: MMVQ-large + pre-Q8_1 (1 row/block × 128 threads, dp4a, input pre-quantized once per fused-GEMV-input). Hits ~33 tok/s on Qwen3-8B Q4_K_M (inside llama.cpp's reported 25-42 range).
+- **k < 1024 + Q4_K/Q5_K/Q6_K**: MMQ-4-rows (4 rows/block × 256 threads, dp4a, on-the-fly Stage 1 quant). Right for SmolLM-class.
+- **Q8_0**: legacy FP-fmuladd GEMV (no MMQ variant).
+- **Other formats (Q4_0, Q5_0)**: dequant→FP16 scratch + cuBLAS GEMV.
+
+**Embedding** uses per-row K-quant lookup kernels for Q4_K/Q5_K/Q6_K (llama.cpp `get_rows_q*_K` pattern) — keeps the embedding table quantized in VRAM and dequantizes only the looked-up rows at forward time. Saves ~1.16 GiB on Qwen3-8B vocab×hidden vs the bulk-dequant path.
**Prefill** always dequantizes into a scratch buffer before calling cuBLAS HGEMM. The scratch holds one projection at a time and is reused across all projections.
See [QUANTIZATION.md](QUANTIZATION.md) for block layouts.
-## Weight Strategy: On-the-Fly Dequantization
+## Weight Strategy: Quantized + on-demand dequant
-### Problem
+### Problem (historical)
-Storing quantized weights provides compression (e.g., Q4_K is ~4.5 bits/param), but cuBLAS GEMM for prefill requires FP16 input. A naive approach stores both the quantized copy (for decode GEMV) and a permanent FP16 copy (for cuBLAS). This **doubles** VRAM usage, negating the benefit of quantization:
+Storing quantized weights provides compression (e.g., Q4_K is ~4.5 bits/param), but cuBLAS GEMM for prefill requires FP16 input. A naive approach stores both the quantized copy (for decode GEMV) and a permanent FP16 copy (for cuBLAS). This would **double** VRAM usage, negating the benefit of quantization:
| Model | Quantized only | Quantized + FP16 copy | Overhead |
|-------|---------------|----------------------|----------|
| Llama-3.2-1B Q8_0 | 1.1 GB | 3.3 GB | +200% |
| Llama-3.1-8B Q4_K_M | 4.9 GB | 21 GB | +330% |
+### Current state — only-quantized + scratch + dedup
+
+dotLLM today stores **only the quantized copy** for formats with a custom GEMV (Q8_0/Q4_K/Q5_K/Q6_K) and uses a single per-projection FP16 scratch for prefill HGEMM. Plus: the fused QkvPacked / GateUpPacked buffers are now the *source of truth* — per-tensor `qQuant`/`kQuant`/etc. pointers slice into them rather than allocating duplicates (saves ~2.5 GiB on Qwen3-8B). Net VRAM at load on Qwen3-8B Q4_K_M: ~5.3 GiB (was OOMing on a 12 GB RTX 3060 before the dedup).
+
### Industry Approaches
Three strategies exist in the ecosystem:
diff --git a/docs/HIP.md b/docs/HIP.md
new file mode 100644
index 00000000..953794db
--- /dev/null
+++ b/docs/HIP.md
@@ -0,0 +1,181 @@
+# HIP/ROCm Backend Architecture — dotLLM
+
+## Overview
+
+The HIP backend provides the AMD GPU execution path for dotLLM. It is an architectural mirror of [docs/CUDA.md](CUDA.md): same layering, same conventions, same approach — the only differences are library names (`libamdhip64` / `hipblas`) and the code-object format (`.co`) instead of PTX.
+
+HIP is AMD's source-level CUDA clone. The kernels in `native/kernels/*.cu` translate to HIP with a mechanical rewrite (`hipify-perl` or manual search-and-replace of `cuda_` → `hip_`), and the runtime API reads like the CUDA Driver API with `cu` replaced by `hip`. This repository adopts the same design choice as the CUDA backend: **P/Invoke directly against AMD's runtime libraries; no custom C/C++ shared library.**
+
+## Status
+
+This is a **proof-of-pipeline scaffold**. The RmsNorm kernel is ported and a round-trip test confirms the P/Invoke → module-load → kernel-launch → memcpy pipeline. The full LLM forward pass is deferred — see [Kernel Catalog](#kernel-catalog-to-port) for the port list.
+
+## Chosen Architecture: Code-Object Loading via HIP Module API
+
+Kernels are written in HIP (`.hip` / `.cu`) and compiled to **code objects** (AMDGPU ELF, `.co` / `.hsaco`) using `hipcc --genco`. The HIP runtime loads them at application startup via `hipModuleLoadData` and exposes kernel entry points via `hipModuleGetFunction`. Launches go through `hipModuleLaunchKernel`.
+
+```
+┌──────────────────┐ hipcc --genco ┌──────────────┐
+│ rmsnorm.hip │ ──────────────────► │ rmsnorm.co │ (fat ELF; ships with app)
+│ rope.hip │ (bundled ISA │ rope.co │
+│ attention.hip │ for each │ attention.co │
+│ dequant.hip │ --offload-arch) │ dequant.co │
+└──────────────────┘ └──────┬───────┘
+ │ loaded at runtime
+┌───────────────────────────────────────────────▼──────────────────┐
+│ C# application │
+│ │
+│ [LibraryImport("amdhip64")] ← AMD HIP runtime (on system) │
+│ hipModuleLoadData(coBytes) ← loads code object │
+│ hipModuleGetFunction(module) ← gets kernel handle │
+│ hipModuleLaunchKernel(func, ...) ← launches on GPU │
+│ │
+│ [LibraryImport("hipblas")] ← hipBLAS (on system) │
+│ hipblasHgemm(...) ← FP16 GEMM, matrix cores │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+Unlike NVIDIA's PTX (text IR, JIT'd per GPU), a `.co` file is a fat ELF bundle containing pre-compiled AMDGPU ISA for each `--offload-arch` target. Cross-architecture distribution therefore requires compiling for every target at build time. For the scaffold we target `gfx1030` (RDNA2, RX 6000) and `gfx1100` (RDNA3, RX 7000); override with the `HIP_ARCHS` env var in `native/hip/build.sh` / `build.ps1`.
+
+## Library Layering
+
+```
+DotLLM.Hip (managed, this project)
+├── Interop/
+│ ├── HipDriverApi.cs — [LibraryImport("amdhip64")] ~25 fns
+│ ├── HipBlasApi.cs — [LibraryImport("hipblas")] ~6 fns
+│ ├── HipLibraryResolver.cs — DllImportResolver: versioned .so / .dll
+│ ├── HipException.cs
+│ └── HipErrorHelper.cs — ThrowOnError / ThrowOnHipBlasError
+├── HipContext.cs — RAII context (hipCtxCreate/hipCtxDestroy)
+├── HipStream.cs — RAII stream (hipStreamCreate/hipStreamDestroy)
+├── HipModule.cs — Code-object loader (hipModuleLoadData + cache)
+├── HipDevice.cs — Enumeration, device props, host/device memcpy
+└── Kernels/
+ └── RmsNormKernel.cs — Wraps rmsnorm.co, exposes typed Launch()
+```
+
+Exactly mirrors `DotLLM.Cuda`. Every type here has a direct counterpart there.
+
+## P/Invoke Approach
+
+No custom C library. `[LibraryImport]` source-generated P/Invoke declarations call AMD's runtime directly:
+
+- **`amdhip64`** — the HIP runtime. `libamdhip64.so[.6]` on Linux, `amdhip64.dll` on Windows. Installed with ROCm (Linux) or AMD's ROCm for Windows release.
+- **`hipblas`** — hipBLAS, AMD's cuBLAS counterpart. `libhipblas.so` / `hipblas.dll`.
+
+`HipLibraryResolver.Register()` is called before the first P/Invoke and routes requests through a `NativeLibrary.SetDllImportResolver` that tries versioned Linux names (`libamdhip64.so.6`, `.5`, etc.) — the same pattern as the CUDA resolver.
+
+`HipDevice.IsAvailable()` probes for the runtime library with `NativeLibrary.TryLoad` **before** touching any generated P/Invoke stub. This lets the assembly load cleanly on systems without ROCm installed and lets callers gracefully degrade.
+
+## HIP vs CUDA: Key API Mappings
+
+Mechanical translation covers ~95% of call sites. A reference for anyone porting a kernel or runtime call:
+
+| CUDA Driver API | HIP Runtime API | Notes |
+|--------------------------|--------------------------|-------|
+| `cuInit` | `hipInit` | |
+| `cuDeviceGet(Count)` | `hipDeviceGet` / `hipGetDeviceCount` | |
+| `cuCtxCreate_v2` | `hipCtxCreate` | HIP drops the `_v2` suffixes |
+| `cuMemAlloc_v2` | `hipMalloc` | |
+| `cuMemcpyHtoD_v2` | `hipMemcpyHtoD` | |
+| `cuModuleLoadData` | `hipModuleLoadData` | takes .co instead of PTX |
+| `cuModuleGetFunction` | `hipModuleGetFunction` | |
+| `cuLaunchKernel` | `hipModuleLaunchKernel` | name differs, signature identical |
+| `cuStreamCreate(flags)` | `hipStreamCreate()` | HIP's variant has no flags arg |
+| `cuGetErrorString` | `hipGetErrorString` | returns `const char*` directly, no out-param |
+| `cublasHgemm` | `hipblasHgemm` | |
+| `cublasGemmEx` | `hipblasGemmEx` | constant values differ — see below |
+
+For **kernel source**, the hipify mapping is just as mechanical:
+
+| CUDA | HIP |
+|----------------------------|---------------------------------|
+| `` | `` |
+| `` | `` |
+| `__global__` / `__device__` | unchanged |
+| `__syncthreads()` | unchanged |
+| `__shfl_down_sync(mask,v,n)`| `__shfl_down(v,n)` — HIP drops the mask (implicit full wave) |
+| `cudaStream_t` | `hipStream_t` |
+| `warpSize` | unchanged (32 on RDNA wave32 / CDNA, 64 on older wave64) |
+
+**Note on constants**: `hipblas` uses enum values that do NOT match cuBLAS. `HIPBLAS_OP_N = 111` (not 0), `HIPBLAS_R_16F = 150` (not 2), etc. These are wired correctly in `HipBlasApi.cs`.
+
+## Kernel Catalog (to port)
+
+All kernels from [docs/CUDA.md §Kernel Catalog](CUDA.md#kernel-catalog) have a hipify-perl-ready counterpart. Ported kernels live in `native/hip/kernels/*.hip`, compile to `native/hip/co/*.co`, and are wrapped by typed classes under `src/DotLLM.Hip/Kernels/`.
+
+**Ported in this scaffold:**
+
+| Kernel | File | Entry Points | Wrapper |
+|--------|------|--------------|---------|
+| RMS Norm | `rmsnorm.hip` | `rmsnorm_f32`, `rmsnorm_f16` | `RmsNormKernel` |
+
+**To port (mechanical hipify pass):**
+
+| Kernel | Source (CUDA) | Entry Points |
+|--------|---------------|--------------|
+| Fused Add + RmsNorm | `fused_add_rmsnorm.cu` | `fused_add_rmsnorm_f16` |
+| Per-Head RmsNorm | `per_head_rmsnorm.cu` / `per_head_rmsnorm_f32.cu` | `per_head_rmsnorm_f16`, `per_head_rmsnorm_f32` |
+| RmsNorm (FP32-in→FP16-out) | `rmsnorm_f32in.cu` | `rmsnorm_f32_in_f16_out` |
+| RoPE | `rope.cu` / `rope_f32.cu` | `rope_f16`, `rope_f32` |
+| Attention (naive) | `attention.cu` / `attention_f32.cu` | `attention_f16`, `attention_f32` |
+| SwiGLU | `swiglu.cu` / `swiglu_f32.cu` | `swiglu_f16`, `swiglu_f32` |
+| Add | `add.cu` / `add_f32.cu` | `add_f16`, `add_f32`, `add_f32_f16` |
+| Bias Add | `bias_add.cu` / `bias_add_f32.cu` | `bias_add_f16`, `bias_add_f32` |
+| Softmax | `softmax.cu` | `softmax_f16` |
+| Embedding | `embedding.cu` / `embedding_f32out.cu` | `embedding_lookup_{f32,f16,q8_0}`, plus f32out variants |
+| Convert | `convert.cu` | `convert_f16_to_f32`, `convert_f32_to_f16` |
+| Dequant | `dequant.cu` | `dequant_{q8_0,q4_0,q5_0,q4_k,q5_k,q6_k}_f16` |
+| Quantized GEMV | `quantized_gemv.cu` / `quantized_gemv_f32in.cu` | `quantized_gemv_{q8_0,q4_k,q5_0,q5_k,q6_k}` + f32-in variants |
+| KV quant | `quant_kv.cu` | `quant_kv_*` family |
+
+**hipBLAS for GEMM** — prefill path uses `hipblasHgemm` / `hipblasGemmEx` directly, mirroring cuBLAS. The P/Invoke declarations already exist in `HipBlasApi.cs`; a `HipGemm.cs` wrapper (mirror of `CudaGemm.cs`) lands with the first GEMM-consuming caller.
+
+## Build Workflow
+
+### Developer with ROCm installed
+
+```bash
+# one-off: compile all kernels
+bash native/hip/build.sh
+# or on Windows:
+pwsh native/hip/build.ps1
+
+# then a normal solution build copies .co files into the output directory
+dotnet build
+```
+
+`DotLLM.Hip.csproj` also runs an incremental `hipcc --genco` MSBuild target before build when `hipcc` is in `PATH`, matching the CUDA project's `CompileCudaPtx` target.
+
+### Downstream user without ROCm
+
+Pre-compiled `.co` files ship alongside the .NET binaries under `runtimes/*/native/` (to be wired up when distribution tooling lands). The assembly builds and loads cleanly without ROCm; `HipDevice.IsAvailable()` returns `false` and the backend is skipped.
+
+## Runtime Requirements
+
+- **AMD GPU**: supported architectures depend on which `--offload-arch` targets are compiled in. Default: `gfx1030` (RDNA2 — RX 6000 series) and `gfx1100` (RDNA3 — RX 7000 series). Add `gfx942` for MI300, `gfx90a` for MI250, `gfx900` for Vega, etc.
+- **ROCm**: 6.x recommended (5.x likely works but untested). Linux: install `rocm-hip-runtime`, `hipblas`, and the `amdgpu` kernel driver. Windows: install AMD's ROCm for Windows release (HIP SDK).
+- **Driver**: `amdgpu` kernel driver (Linux) / AMD Adrenalin with ROCm support (Windows).
+
+## hipify-perl Workflow
+
+When porting a new kernel:
+
+1. Copy the `.cu` file to `native/hip/kernels/` renamed to `.hip`.
+2. Run `hipify-perl -inplace` (if available) — translates headers, atomics, shuffle intrinsics, typedefs. The tool is part of the ROCm HIPIFY package.
+3. Manual fix-ups: any `__shfl_*_sync` calls simplify to `__shfl_*` (drop the mask arg). Shared-memory sizes that assume 32-wave reduction may need to grow to 64 to cover wave64 GPUs (see `rmsnorm.hip` for an example).
+4. Compile with `hipcc --genco --offload-arch= -o out.co in.hip`.
+5. Verify numerical parity against the CPU reference in a unit test (pattern: `HipRmsNormKernelTests`).
+
+Reference: [ROCm HIPIFY docs](https://rocm.docs.amd.com/projects/HIPIFY/en/latest/).
+
+## Future Work
+
+- Port remaining kernels per [Kernel Catalog (to port)](#kernel-catalog-to-port).
+- `HipGemm.cs` — row-major wrapper around `hipblasHgemm` using the same transpose trick as cuBLAS.
+- `HipKvCache.cs`, `HipWeights.cs`, `HipForwardState.cs` — mirror the CUDA counterparts.
+- `HipTransformerModel.cs` full implementation (currently a NotImplemented stub).
+- CI job on an AMD runner to exercise the GPU tests.
+- Distribution: package pre-built `.co` files under `runtimes/{rid}/native/` for NuGet consumption.
+- RCCL (ROCm's NCCL counterpart) for multi-AMD-GPU tensor parallelism — same P/Invoke pattern, same library layering.
diff --git a/docs/KV_CACHE.md b/docs/KV_CACHE.md
index 98f9cd47..0ac24763 100644
--- a/docs/KV_CACHE.md
+++ b/docs/KV_CACHE.md
@@ -119,6 +119,77 @@ On each new token write, the oldest window entry is quantized and appended to th
Orthogonal to weight quantization — Q4_K_M model can use Q8_0 KV-cache.
+## MLA KV-Cache (DeepSeek-V2 / V3)
+
+MLA decouples Q head-dim from V head-dim (V2-Lite: qk=192, v=128) and adds
+a shared MQA-style rope-K that broadcasts across heads — neither fits the
+per-head-uniform `IKvCache` shape that GQA/MHA caches assume. A dedicated
+`MlaExpandedKvState` lives next to `TransformerModel` for this reason.
+
+**Loader default**: HF and GGUF config extractors both set `MlaConfig.UseHybridMlaCache = true` for `Architecture.DeepSeekV2` / `Architecture.DeepSeekV3`, so production code paths get **Phase C** (hybrid latent + absorbed decode) without needing per-call configuration. Phase A remains active and is the numerical oracle; tests that build `MlaConfig` directly (bypassing the loader) still default to Phase A. The default flip lives at commits `4b54a72` (HF) and `4724397` (GGUF) — pre-flip, V2-Lite at `max_position_embeddings=163840` allocated ~68 GB and OOM'd on most hosts.
+
+### Phase A — expanded reference cache
+
+`src/DotLLM.Models/Architectures/MlaExpandedKvState.cs`. Per layer:
+
+- `K_nope[layer]` : `[maxSeqLen, numHeads * qkNopeHeadDim]` — per-head non-rope K
+- `V[layer]` : `[maxSeqLen, numHeads * vHeadDim]` — per-head V
+- `KPe[layer]` : `[maxSeqLen, qkRopeHeadDim]` — shared rope-K (post-rotation)
+
+All 64-byte aligned native memory, lazily constructed on the first MLA
+forward and reset when the caller signals a fresh sequence by passing
+`positions[0] == 0`. Not re-entrant; single-stream only (beam search or
+batching needs per-sequence instances). Caller-supplied `IKvCache` is
+ignored for MLA layers.
+
+This layout is the PoC scalar kernel's scratch layout made persistent —
+storage is 1:1 with what the kernel already computes, so zero shape
+translation. Memory: ~16.6 KB per token per layer at F32 for V2-Lite;
+on a 27-layer 8K context that's ~3.6 GB. The purpose of Phase A is
+**correctness oracle**: generation works end-to-end and a split call
+(prefill + step-by-step decode) produces logits that match a single-call
+forward over the combined range within 1e-4.
+
+### Phase B — pure latent + W_UK absorbed (landed)
+
+The production memory win (per the DeepSeek-V2 paper, §2.1.2): store
+the *compressed* latent `c_kv[kv_lora_rank]` per token (512 floats for
+V2-Lite) alongside the shared `k_pe[qk_rope_head_dim]` (64 floats) —
+a single `[kv_lora_rank + qk_rope_head_dim] = 576` value per token per
+layer, 7.2× smaller than Phase A at F32. Attention math absorbs `W_UK`
+into Q on-the-fly: `Q_latent[h] = Q_nope[h] @ W_UK_T[h]` (size
+kv_lora_rank) and `score[h, t, s] = Q_latent[h] · c_kv[s] + Q_pe[h]
+· k_pe[s]`. Output uses absorbed `W_UV`: `out[h] = W_UV[h] @ (softmax
+· c_kv)`. vLLM's MLA backend is the reference implementation.
+
+Lives at `src/DotLLM.Models/Architectures/MlaLatentKvState.cs` +
+`src/DotLLM.Cpu/Kernels/MlaAttention.ExecuteLatent`. Selected via
+`MlaConfig.UseLatentCache = true` (mutually exclusive with
+`UseHybridMlaCache`).
+
+### Phase C — hybrid: latent persistence + Phase A-equivalent prefill expand + absorbed decode (landed, default)
+
+The production-shipping path mirrors vLLM's MLA backend: prefill
+(`seqLen > 1`) expands cached latents through `W_UK` / `W_UV` into
+local scratch and runs the standard 192-dim per-head MHA loop
+(compute-bound at long seqKv); decode (`seqLen == 1`) delegates to
+`ExecuteLatent` — the absorbed 576-dim MQA-style read of the compact
+latent cache (bandwidth-bound at decode). Both paths persist the
+SAME latent form (c_kv + k_pe per token) to `MlaLatentKvState` —
+Phase A's expanded per-head K_nope/V is local prefill scratch and is
+discarded. A decode step therefore consumes exactly the latents a
+pure-Phase-B prefill would have written, so the absorbed kernel can
+run over them without re-expansion.
+
+Lives at `src/DotLLM.Cpu/Kernels/MlaAttention.ExecuteLatentHybrid`.
+Selected via `MlaConfig.UseHybridMlaCache = true` (mutually
+exclusive with `UseLatentCache`). **Default for DeepSeek-V2/V3 from
+the loaders.**
+
+Phase A is the numerical oracle for Phase B / Phase C: oracle tests
+prove split-call match against the expanded-cache reference at 1e-3
+drift on real-weight prompts.
+
## Simple Prompt Caching (Step 54)
Live KV-cache reuse for multi-turn conversations. No paged attention required — works with `SimpleKvCache`.
diff --git a/docs/MODEL_CONFIG.md b/docs/MODEL_CONFIG.md
index d62c371f..dce4c24b 100644
--- a/docs/MODEL_CONFIG.md
+++ b/docs/MODEL_CONFIG.md
@@ -6,7 +6,7 @@ Comprehensive record describing any transformer variant. Populated from GGUF met
```
ModelConfig:
- Architecture Llama | Mistral | Phi | Qwen | DeepSeek
+ Architecture Llama | Mistral | Phi | Qwen | DeepSeekV2 | DeepSeekV3 | NemotronH | Mamba3 | Mixtral | QwenMoe | GraniteMoe
VocabSize int
HiddenSize int
IntermediateSize int (FFN intermediate dim)
@@ -110,4 +110,4 @@ var config = new ModelConfig
2. If yes: add a new `Architecture` enum value, map GGUF metadata keys to ModelConfig, done.
3. If the attention mechanism is different (like MLA): implement a dedicated attention path in the forward pass.
4. If the FFN structure is different: parameterize or add a new FFN variant.
-5. Verify numerical output against HuggingFace transformers reference for the new architecture.
\ No newline at end of file
+5. Verify numerical output against HuggingFace transformers reference for the new architecture.
diff --git a/docs/QUANTIZATION.md b/docs/QUANTIZATION.md
index 72b59fcd..9efb59e2 100644
--- a/docs/QUANTIZATION.md
+++ b/docs/QUANTIZATION.md
@@ -97,3 +97,21 @@ GGUF files can have different types per tensor. Dispatch to correct kernel based
- Vec_dot dominant for decode (GEMV). Dequant+BLAS may win for prefill (GEMM).
- GPU: custom CUDA kernels dequantize in shared memory, use tensor cores. Ref: llama.cpp `ggml-cuda/mmq.cu`.
- Block alignment awkward for SIMD — handle tail elements carefully.
+
+## Vulkan Backend Coverage
+
+The Vulkan backend ships native matmul kernels (GEMV decode + GEMM prefill, with an opt-in F16 cooperative-matrix tile when the device enumerates F16xF16→F32) for the following source dtypes / quant formats. Source bytes stay on device — dequantisation happens in the shader inner loop, so memory cost is the GGUF source size (not 2-4× expanded F32):
+
+| Format | GEMV | GEMM | Coopmat | Reference |
+|---|---|---|---|---|
+| F32 | ✓ | ✓ | — | baseline |
+| F16 | ✓ | ✓ | ✓ (F16xF16→F32, M=N=K=16 tile) | `c9c08c5` |
+| BF16 | ✓ | ✓ | — (BF16 tiles not enumerated on RDNA3.5; use shift-left-16 reinterpret) | `c9c08c5` |
+| Q8_0 | ✓ | ✓ | ✓ (`MatMulQ8_0GemmCoopmatKernel`) | pre-existing |
+| Q4_K_M | ✓ | ✓ | — (Phase 1 follow-up) | `afb2272` + `b1ee6bc` |
+| Q5_K_M | ✓ | ✓ | — | `15099b9` + `83e0732` |
+| Q6_K_M | ✓ | ✓ | — | `29a1459` + `39b7646` |
+
+Q4_0 / Q4_1 / Q5_0 / Q5_1 / Q2_K / Q3_K / IQ-family Vulkan kernels are not yet shipped — the upload path falls back to F32 dequant for those formats, so weight memory is doubled / quadrupled. K-quant Q4/5/6 priority was chosen because they cover the majority of production GGUF deployments (`*-Q4_K_M.gguf` is the de-facto default for most checkpoints). Q2_K + Q3_K are present in the CUDA backend (lighter-weight reference kernels) and are tracked as a Vulkan follow-up.
+
+See [docs/VULKAN.md](VULKAN.md) for runtime selection details and [docs/CUDA.md](CUDA.md) for the CUDA backend's coverage (Q2_K through Q8_0 plus pre-Q8_1 + MMVQ-large + MMQ + grouped-MoE-GEMV variants).
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index b0b9a995..41e2c55a 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -43,6 +43,10 @@ Each step is designed to be a discrete unit of work suitable for a single implem
| 16 | **Chat template engine** :white_check_mark: | Jinja2-subset interpreter. Parse `chat_template` from GGUF metadata or `tokenizer_config.json`. Compile to `IChatTemplate`. | 4 |
| 17 | **Streaming generation** :white_check_mark: | `IAsyncEnumerable` token-by-token output. Yield each decoded token as it's generated. | 8 |
| 20 | **Additional architectures** :white_check_mark: | Mistral (add sliding window attention mask), Phi, Qwen. Should be mostly `ModelConfig` parameterization, minimal new code. | 6 |
+| 20b | **Safetensors loader (dense transformers)** :white_check_mark: | HuggingFace `model.safetensors` + `config.json` ingest for Llama/Mistral/Phi/Qwen. `HfConfigExtractor` mirrors `GgufModelConfigExtractor`; `TransformerModel.LoadFromSafetensors` reads HF tensor names (`model.layers.{i}.self_attn.*`, `model.mlp.*`, `lm_head`), handles `tie_word_embeddings`, and upcasts bf16 → f32 into 64-byte-aligned scratch. `ModelLoader.Load(path)` auto-detects `.gguf` vs `.safetensors`. Verified end-to-end on `hf-internal-testing/tiny-random-LlamaForCausalLM`. | 20 |
+| 20c | **Multi-shard safetensors (`model.safetensors.index.json`)** :white_check_mark: | HuggingFace-convention sharded-checkpoint ingest for models above the 5 GiB shard cap (Llama-3-8B-Instruct, Mistral-7B-Instruct, …). `SafetensorsIndex` parses the `weight_map` + `metadata.total_size` sidecar; `MultiShardSafetensorsFile` opens every shard file, resolves tensor-name → owning shard, and exposes the same lookup surface as `SafetensorsFile` via new `ISafetensorsTensorSource` interface. Consumers (`TransformerWeightsSafetensorsLoader`, `Mamba3WeightLoader`, `TransformerModel.LoadFromSafetensors`, `Mamba3TransformerModel.LoadFromSafetensors`) accept the interface; existing single-file callers unchanged. `ModelLoader.LoadFromSafetensors` auto-detects: directory path → probe for `model.safetensors.index.json` first (multi-shard) else single `.safetensors`; index.json path → multi-shard; single-file path → single-shard, but prefers sibling index.json when present. Verified end-to-end via resharded `hf-internal-testing/tiny-random-LlamaForCausalLM` (2 shards, index.json, forward pass produces finite `[3, 32000]` logits). | 20b |
+| 20d | **Phi-3 fused-tensor safetensors loader** :white_check_mark: | Phi-3/Phi-3.5 ships `self_attn.qkv_proj` (fused `[Q + K + V]` rows) and `mlp.gate_up_proj` (fused `[gate; up]`) — dotLLM's separate-tensor path previously couldn't load these. `TransformerWeightsSafetensorsLoader.SplitFusedProjection(file, name, partRows[], ...)` probes for fused names, allocates one 64-byte-aligned F32 buffer per split (`[Q, K, V]` at row offsets `[0, qOut, qOut+kvOut]`; `[gate, up]` at `[0, intermediate]`), and decodes BF16/F16/F32 in place. Fusion-absent models (Llama/Mistral/Qwen/etc.) unaffected. **Verified end-to-end on real `microsoft/Phi-3.5-mini-instruct` (7.6 GB, 2 shards, 32 layers, MHA)**: 3-token prefill produces `[3, 32064]` logits, 96192/96192 finite, stddev 7.00. | 20b |
+| 20e | **Granite-MoE fused-per-expert safetensors loader** :white_check_mark: | IBM Granite's MoE (`GraniteMoeForCausalLM` / `model_type=granitemoe`) stores all experts in one fused tensor per projection: `block_sparse_moe.input_linear [E, 2*I, H]` (w1+w3 for all experts, w1 at rows `[0:I]`, w3 at `[I:2I]`), `block_sparse_moe.output_linear [E, H, I]` (w2 for all experts), `block_sparse_moe.router.layer` (gate). New `Architecture.GraniteMoe` variant + `HfConfigExtractor` detection. `LoadGraniteMoeLayer` slices the fused tensor per expert e at element offsets `e*(2*I*H)` / `e*(H*I)` into 64-byte-aligned per-expert F32 allocations, populating `MoeLayerWeights.RoutedW{1,2,3} nint[]` arrays. `MoeSwiGluMlp` handles top-8 routing unchanged (no shared expert in Granite). Attention is standard GQA. **Verified end-to-end on real `ibm-granite/granite-3.0-3b-a800m-instruct` (6.3 GB, 2 shards, 40 experts, top-8, 24Q/8KV heads GQA, vocab=49155)**: 3-token prefill produces `[3, 49155]` logits, 147465/147465 finite, stddev 16.13. | 58, 58c |
| 22 | **Multi-threaded CPU inference** :white_check_mark: | Parallelize GEMV/GEMM, attention, and FFN across cores. Custom zero-alloc `ComputeThreadPool` with `delegate*` dispatch for compute-bound loops in `MatMul`, `Attention`, per-layer token processing. Thread count configurable via `--threads` CLI option and `ThreadingConfig`. Target: ~4-8× speedup on multi-core CPUs. | 6 |
**Milestone**: Chat interactively with Q4_K_M models, stream responses, support multiple model architectures.
@@ -135,11 +139,22 @@ Step 22 (done) ──────► Step 30 (NUMA + Spin-wait)
| Step | Feature | Description | Depends On |
|------|---------|-------------|------------|
-| 48 | **MLA attention** | DeepSeek-V2/V3 Multi-head Latent Attention. Down-project KV to latent, up-project during attention. `LatentKvCache`. | Phase 1 |
+| 48 | **MLA attention (kernel PoC)** :white_check_mark: | DeepSeek-V2/V3 Multi-head Latent Attention — standalone scalar kernel + HF config detection. `MlaAttention.Execute`: hidden → Q LoRA factorisation (`q_a_proj` → `q_a_layernorm` → `q_b_proj`, or monolithic Q when `q_lora_rank=0`), KV compress via `kv_a_proj_with_mqa` → `kv_a_layernorm` → `kv_b_proj` (split `K_nope`/`V`), decoupled RoPE on `qk_rope_head_dim` sub-dim with MQA-shared `K_rope`, per-head causal SDPA with scale `1/sqrt(qk_nope + qk_rope)`, `o_proj`. `Architecture.{DeepSeekV2, DeepSeekV3}` enum variants + `HfConfigExtractor` populates `MlaConfig` (`KvLoraRank`, `QLoraRank`, `QkNopeHeadDim`, `QkRopeHeadDim`, `VHeadDim`, `RopeTheta`, YaRN scaling fields) and DeepSeek `MoeConfig` (`n_routed_experts`, `n_shared_experts`, `first_k_dense_replace`). Verified against `yujiepan/deepseek-v2-tiny-random` (config detection) and a reference implementation within 5e-4. Out of scope (follow-ups): TransformerModel forward integration, tensor-name loader, latent KV-cache, YaRN mscale, absorption, multi-shared-expert. | Phase 1 |
+| 48a | **MLA TransformerModel integration (DeepSeek-V2/V3 end-to-end)** :white_check_mark: | Wires the MLA kernel (step 48) into `TransformerModel`'s forward path and adds the DeepSeek safetensors weight loader. New `MlaLayerWeights` class (referenced via `TransformerLayerWeights.Mla`) holds `Q{A,B}Proj` / `QProj` (monolithic when `q_lora_rank=0`), `KvAProjWithMqa` / `KvBProj`, `Q/KvALayernormWeight`, and per-layer dims. `LoadDeepSeekMlaLayer` resolves all attention tensors and routes FFN to dense MLP (first-K prefix) or `LoadQwenMoeLayer` (plural `shared_experts` already supported). `TransformerModel.Forward` gets an MLA branch: `RMSNorm → MlaAttention.Execute → residual → goto FfnBranch` (jumps past GQA Q/K/V/RoPE/Attention/O to the shared MoE/dense FFN dispatch), preserving Llama/Mistral/Phi/Qwen path unchanged. `ModelLoader.LoadFromSafetensors` now dispatches `Architecture.DeepSeekV2 / DeepSeekV3` (removed `NotSupportedException`). RoPE tables sized to `QkRopeHeadDim` for MLA. **Verified against real `yujiepan/deepseek-v2-tiny-random`** (DeepSeekV2, MLA, hidden=8, layers=2, heads=2, qk_head=4, vocab=102400, kv_lora_rank=2, q_lora_rank=2): 4-token prefill produces `[4, 102400]` logits, 409600/409600 finite, stddev 0.056, argmax varies per position. 3 new unit tests (LoRA-Q prefill, monolithic-Q prefill, LoRA-Q single-token). KV-cache is PoC — rerun full MLA forward per call; latent-KV-cache and absorption optimisation remain follow-ups. | 48 |
| 49 | **ALiBi position encoding** | Additive linear bias to attention scores. `AlibiPositionEncoding` implementing `IPositionEncoding`. | Phase 1 |
| 56 | **SmolLM3 architecture** | HuggingFace SmolLM3-3B. NoPE layer support in attention (skip RoPE application on marked layers). YARN context extension for 128k. GQA with 4 groups. Tool calling via `xml_tools` (Hermes-compatible) or `python_tools` (`PythonicToolCallParser`). | Phase 1 |
| 57 | **Gemma 4 architecture** | Google Gemma 4 model family. GeGLU activation, RMS pre-norm with per-layer scaling, interleaved local/global attention, logit soft-capping. `GemmaModel` implementing `IModel` via `TransformerBlock` parameterization. | Phase 1 |
-| 58 | **Mixture of Experts** | MoE FFN with top-K expert routing. `IExpertRouter` interface, `MoeFFN` block replacing standard FFN. Sparse activation — only K of N experts compute per token. Shared expert support (DeepSeek-style). Memory: all expert weights loaded, only active experts computed. Covers: DeepSeek-V2 MoE, Granite hybrid MoE, Qwen-MoE. | Phase 1 |
+| 58 | **Mixture of Experts** :white_check_mark: | MoE FFN with top-K expert routing. Dense-routing Mixtral-family support: `MoeConfig` on `ModelConfig`, `Architecture.Mixtral`, `MoeSwiGluMlp` kernel (softmax over experts → top-k → renormalise → per-expert SwiGLU → weighted combine, scalar tiebreaker matching `torch.topk`). HF safetensors loader resolves `block_sparse_moe.gate` + `experts.{j}.w1/w2/w3` with F16/BF16 → F32 upcast; `ModelLoader.LoadFromSafetensors` dispatches Mixtral through the existing `TransformerModel` forward path (attention unchanged, FFN branches on `TransformerLayerWeights.Moe`). Verified against real HF `yujiepan/mixtral-tiny-random` config detection + synthetic-fixture forward pass. Out of scope (future): shared experts (DeepSeek-V3, Qwen1.5-MoE), Qwen-MoE `mlp.experts.{j}.{gate_proj,up_proj,down_proj}` naming, fused GroupedGEMM, expert parallelism. | Phase 1 |
+| 58c | **DeepSeek multi-shared-expert (`n_shared_experts > 1`)** :white_check_mark: | Extends MoE's shared-expert branch to run N shared experts in parallel (DeepSeek-V2/V3 convention; Qwen1.5-MoE at N=1 unchanged). `MoeConfig.NumSharedExperts` (default 1). `MoeLayerWeights.SharedGateProj/UpProj/DownProj` migrated from single `nint` to `nint[]` (length = `NumSharedExperts`). `MoeSwiGluMlp.ExecuteWithSharedExpert` now accepts `ReadOnlySpan` per shared-proj kind — for k=0 the GEMM writes directly into the down-buffer (preserving bit-identity with prior single-shared path); subsequent experts compute into pooled scratch then `TensorPrimitives.Add` into the down buffer. Plural HF naming `model.layers.{i}.mlp.shared_experts.{k}.{gate,up,down}_proj` loader support; singular `mlp.shared_expert.*` (Qwen1.5-MoE) still loads as length-1 array. Sigmoid `shared_expert_gate` only applies when `NumSharedExperts=1 && HasSharedExpertGate` (Qwen1.5-MoE). DeepSeek arches (`DeepSeekV2`/`DeepSeekV3`) still throw `NotSupportedException` in `ModelLoader` pending MLA TransformerModel integration; multi-shared MoE loader path is ready for when MLA lands. 2 new `MoeSwiGluMlp` tests (multi-shared sum + length-1-array bit-identity), 1 new synthetic-fixture safetensors-loader test (`DeepSeekStyleMoE_PluralSharedExperts_LoadsAndProducesFiniteLogits`). 1282 unit tests pass. | 58, 58a |
+| 58b | **MoE GroupedGEMM refactor** :white_check_mark: | Replace per-token per-expert scalar loop in `MoeSwiGluMlp` with a gather/batched-GEMM/scatter implementation: bucket tokens by their top-k-assigned experts (ArrayPool-rented index lists, zero sustained allocations), run one batched SwiGLU per expert across all its assigned tokens (gather `[B, hidden]` → `w1/w3` GEMM → silu*up → `w2` GEMM), scatter back preserving original per-(token, slot) accumulation order. Bit-identical to the prior scalar path — verified across all 8 `MoeSwiGluMlp` unit tests (Mixtral 2-expert top-1, 4-expert top-2, Qwen-MoE with/without shared expert + sigmoid gate + `norm_topk_prob` on/off) and 16 MoE integration tests including real `yujiepan/qwen3-moe-tiny-random` end-to-end forward. Public signatures unchanged; both `Execute` and `ExecuteWithSharedExpert` route through the same grouped core; shared-expert runs once per token on the un-gathered batch. Follow-up: fused GroupedGEMM with SIMD-batched SwiGLU for ~2-3× further. | 58 |
+| 58a | **Qwen-MoE naming + shared experts (Qwen1.5/2/3-MoE)** :white_check_mark: | Extends step 58 to the HF Qwen-MoE convention. New `Architecture.QwenMoe` enum variant maps from `model_type=qwen2_moe` / `qwen3_moe` and `architectures[0]=Qwen{2,3}MoeForCausalLM`. `HfConfigExtractor` surfaces `norm_topk_prob`, `shared_expert_intermediate_size`, `decoder_sparse_step`, and `mlp_only_layers` into `MoeConfig` (new fields: `NormTopKProb`, `SharedExpertIntermediateSize`, `HasSharedExpertGate`, `DecoderSparseStep`, `MlpOnlyLayers`, plus an `IsMoeLayer(layerIdx)` helper). `TransformerWeightsSafetensorsLoader.LoadQwenMoeLayer` resolves Qwen-style names (`mlp.gate`, `mlp.experts.{j}.{gate_proj,up_proj,down_proj}`), optional `mlp.shared_expert.{gate,up,down}_proj` (Qwen1.5-MoE-A2.7B dense SwiGLU), and optional `mlp.shared_expert_gate.weight` (sigmoid scalar). Layer-level dispatch: in Qwen3-MoE (`decoder_sparse_step=2`), dense-MLP layers use the existing Llama path; MoE layers use the new kernel. `MoeSwiGluMlp.ExecuteWithSharedExpert` extends the routed kernel with a parallel shared-expert branch (dense SwiGLU at a configurable intermediate width, optionally multiplied by a per-token sigmoid scalar) and a `normTopKProb` flag (Mixtral+Qwen3=true; Qwen1.5-MoE=false). Existing Mixtral call-sites unchanged — the single-arg `Execute` overload still dispatches the same Mixtral kernel. Verified: 3 new `MoeSwiGluMlp` unit tests (routed+shared no-gate, routed+shared+sigmoid-no-renorm, shared-disabled byte-identity with Mixtral path), 4 `HfConfigExtractor` Qwen-MoE detection tests (Qwen3-MoE tiny-random config, Qwen1.5-MoE-A2.7B with shared expert + `norm_topk_prob=false`, `mlp_only_layers` override, NeoX RoPE), 2 synthetic-fixture forward-pass tests (Qwen-MoE plain 2-layer + Qwen-MoE with shared expert), and the real `yujiepan/qwen3-moe-tiny-random` checkpoint (~20 MB, 2 layers × 8 experts × top-2, `decoder_sparse_step=2` so layer 0 dense / layer 1 MoE) — detection + load + 3-token forward, finite logits with nonzero variance. Out of scope (future): DeepSeek-V2/V3 multi-shared-expert (`n_shared_experts > 1`), MLA attention for DeepSeek, real Qwen1.5-MoE-A2.7B validation (~14 GB). | 58 |
+| 60 | **Mamba-3 architecture (prefill)** :white_check_mark: | Pure-SSM Mamba-3 end-to-end loading and prefill forward pass. `Mamba3ConfigExtractor` parses HF `config.json` (`model_type=mamba3`). `Mamba3WeightLoader` ingests F32 safetensors; `Mamba3TransformerModel : IModel` wires embed → N × (RMSNorm + `Mamba3Block.Forward` + residual) → final RMSNorm → LM head. Canonical `state-spaces/mamba` semantics: 8-slice in_proj (z, x, B, C, dd_dt, dd_A, trap, angles), per-token per-head `A = -softplus(dd_A)` clamp, data-dependent RoPE with tanh-PI accumulation, attention-style SSD scan, SISO and MIMO entry points. Prefill only — persistent decode state, tokenizer ingest, and generation loop are follow-ups. `ModelLoader.LoadFromSafetensors` dispatches `Architecture.Mamba3`. Validated end-to-end on the real `ib-ssm/mamba3-370M-10BT` checkpoint (1.55 GB, 48 layers, 32 heads, d_state=128, vocab=32000) via a gated integration test (`DOTLLM_IBSSM_CHECKPOINT_PATH` env var or `%USERPROFILE%/dotllm-ibssm-370m/`): 5-token prefill produces finite logits with nonzero per-position variance; canonical Python reference comparison deferred (requires Triton+CUDA at 370M dims, infeasible on Windows CPU). | Phase 1 |
+| 60a | **Mamba-3 persistent state (`Mamba3State`)** :white_check_mark: | Building block for streaming decode. `Mamba3State : IDisposable` owns per-layer `ssm_state [n_head, head_dim, d_state]` + `cum_angle [n_head, num_rope_angles]` buffers via `NativeMemory.AlignedAlloc` (64-byte). New `Mamba3TransformerModel.Forward(tokens, positions, deviceId, Mamba3State)` overload reads state at entry, writes back at exit; existing parameterless overloads route through a shared core with an ephemeral state. State threading determinism is bit-exact (max_abs=0 across re-runs of the same chunk schedule). Prefill+decode does NOT exactly equal one-shot at chunk boundaries by design: canonical's `shifted_γ[t] = DT[t+1]·(1-trap[t+1])` is a 1-token lookahead that drops to 0 at chunk edges. Closing that gap requires threading two additional canonical buffers (`k_state`, `v_state`) plus a streaming-decode SSD kernel — both future work. Verified on real `ib-ssm/mamba3-370M-10BT` (`DecodeMatchesPrefillOnRealCheckpoint`): finite logits, state actually advances, regression ceiling pinned. | 60 |
+| 60c | **Mamba-3 end-to-end text generation (growing-context prefill)** :white_check_mark: | End-to-end text generation verified on real `ib-ssm/mamba3-370M-10BT`: encode prompt via HF `tokenizer.json` → iterative argmax decode (one-shot prefill per step over the full growing context — O(N²) total but every forward is canonical, sidestepping the `shifted_γ` chunk-edge drift documented under step 60a) → decode with tokenizer round-trip. Gated integration test `IbSsmMamba3GenerationTests.Mamba3_GeneratesText_FromTokenizedPrompt` generates 5 tokens from `"The capital of France is"`, asserts all logits finite, all argmax IDs in `[0, vocab)`, and the prompt is a prefix of the decoded output. No pipeline code changed — this step is the explicit confirmation that load → tokenize → forward → argmax → decode composes correctly. Streaming-SSD decode (O(N) per token) is the follow-up: requires threading `k_state` + `v_state` plus a streaming-decode kernel that consumes them (step 60a notes). | 60, 60a, 60b |
+| 60f | **Mamba-3 MIMO streaming decode** :white_check_mark: | Extends step 60e's SISO streaming SSD to MIMO (rank > 1) at the kernel + block level. `Mamba3State.k_state` is rank-extended to `[R, H, N]` when `IsMimo=true` (SISO keeps `[H, N]`); `ssm_state`, `cum_angle`, `v_state` remain rank-free per canonical `mamba3.py:434-452`'s `allocate_inference_cache()` (rank is summed into state inside the scan kernel; V is not rank-expanded). New `Mamba3CanonicalSsd.ExecuteMimoStreaming(...)` mirrors `ExecuteSisoStreaming` with rank-aware K; boundary math derived from the tilelang MIMO fwd state update (rank-sum of K): `ssm[h,p,n] += v_state[h,p] · (Σ_r k_state[r,h,n]) · DT[0,h] · (1-trap[0,h])`. `Mamba3Block.ForwardMimo` gains a streaming overload (empty-span preserves one-shot). Empty-span and single-chunk schedules are **bit-identical** to the existing one-shot `ExecuteMimo`. Synthetic MIMO drift (R=3, T=6, H=4, P=4, N=8) across all split schedules `[6]`, `[3,3]`, `[2,2,2]`, `[1,1,1,1,1,1]`, `[4,2]`, `[2,4]`, `[5,1]`, `[1,2,3]`: y max_abs ≤ 4.7e-15, ssm_state max_abs ≤ 3.4e-13 — pure F32-reorder noise, orders of magnitude tighter than the 1e-5/1e-4 gate. Adds 8 kernel-level + 9 block-level + 1 state-shape unit tests (18 total new). `Mamba3TransformerModel` continues to throw `NotSupportedException` for MIMO pending the weight-loader follow-up (`Mamba3WeightLoader` needs `[H, R, N]` B_bias/C_bias + `mimo_x`/`mimo_z`/`mimo_o` tensors) — kernel/block paths are complete and test-covered; model-level is a one-liner once the loader lands. | 60e |
+| 60e | **Mamba-3 streaming decode (SSD chunk-boundary state)** :white_check_mark: | Closes the canonical `shifted_γ[t] = DT[t+1]·(1-trap[t+1])` 1-token lookahead gap that step 60a documented. `Mamba3State` gains two new persistent buffers per layer — `k_state [n_head, d_state]` (previous chunk's last-token post-RoPE, pre-scale K) and `v_state [n_head, head_dim]` (previous chunk's last-token V) — matching canonical `mamba3.py:142` and `mamba3_siso_fwd.py:341-352` / `mamba3_siso_step_kernel` semantics. `Mamba3Block.Forward` gains new `kState` / `vState` span parameters (empty-span overload preserves one-shot prefill). At chunk start, a new private `ApplyChunkBoundaryAdjustment` folds the deferred boundary term `ssm += v_state · k_state · DT[0] · (1 - trap[0])` into the SSM hidden state BEFORE the SSD scan — exactly the term a one-shot forward would have produced via `scale[T_prev-1] = γ + shifted_γ`. At chunk end, the block persists this chunk's last-token post-RoPE K (from `bHRN[T-1]`) and V (from `xBuf[T-1]`) back into `kState` / `vState`. The SSD kernel itself is untouched — pre-adjustment and post-save live in the block where scratch and token widths are known. `Mamba3TransformerModel.Forward(..., Mamba3State)` threads all four buffers. **Numerical results.** Tiny synthetic fixture (2-layer, 4-head): prefill+decode vs one-shot max_abs drops from ~2.5e-6 (pre-P3) to ~2e-6 on 2+1+1 and 1e-8 on 2+2 — within F32-reorder noise. Real `ib-ssm/mamba3-370M-10BT` (48 layers, 32 heads, `d_state=128`): last-token logit max_abs drops from ≈6.5 to ≈3.3, and — critically — top-1 argmax now MATCHES between one-shot and 2-prefill+1-decode (previously differed, so sampled tokens would diverge). New `Mamba3BlockStreamingTests` (7 split schedules × {y, ssm_state, k_state, v_state}) verify the kernel-level property directly. Tightened assertions: `PrefillThenDecode_ApproximatesOneShot_WithExpectedDrift` renamed to `PrefillThenDecode_BitEqualsOneShot` (abs_tol 5e-5→1e-5, extended to 1+1+1+1); `DecodeMatchesPrefillOnRealCheckpoint` ceiling tightened from 100 to 10 with a new `Assert.Equal(argmaxPrefill, argmaxSplit)`. MIMO streaming is deferred — canonical `mamba3_mimo_combined` does not itself thread input states (only `step()` does), the tilelang MIMO kernel has no input-state signature, and the MIMO Block path is unreachable from `Mamba3TransformerModel` today (throws `NotSupportedException`). SISO covers every validated checkpoint (ib-ssm 370M). | 60, 60a |
+| 60d | **Mamba-3 pooled forward scratch (`Mamba3ForwardScratch`)** :white_check_mark: | Eliminate per-Forward-call managed allocations on the Mamba-3 hot path. New `Mamba3ForwardScratch : IDisposable` (mirrors `NemotronHForwardState`) owns 13 named 64-byte-aligned `NativeMemory.AlignedAlloc` buffers (`Proj`, `X`, `Z`, `Dt`, `Adt`, `Trap`, `Gamma`, `Scale`, `AnglesRaw`, `B`, `C`, `QkPreDot`, `YScan`) with power-of-two `EnsureCapacity(seqLen)` growth and lazy first-call init — subsequent Forwards at the same or smaller T reuse the allocation. `Mamba3Block.Forward` / `ForwardMimo` now take the scratch as a required first parameter; `Mamba3TransformerModel` owns one instance and threads it through every layer of every Forward call (48 layers × ~10 buffers/call → 0 allocations/step on the 370M checkpoint, vs ≈480 managed-float[] allocations before). Scratch sizing includes both SISO and MIMO B/C widths (backed by `max(1, mimoRank) · nHead · dState`), and `ComputeMemoryBytes` now reports the scratch footprint. Numerics preserved bit-exact — every pre-existing Mamba-3 test (57 unit + 20 integration) still passes, including the canonical Python-reference SISO/MIMO block comparators and all 9 real-weight `ib-ssm/mamba3-370M-10BT` integration tests. 9 new unit tests exercise scratch lifecycle: zero-capacity lazy init, power-of-two growth, no-op when EnsureCapacity(≤cap), accessor bounds check, idempotent Dispose, accessor-after-Dispose throws. | 60 |
+| 60b | **HF `tokenizer.json` ingest (Llama-family / Mamba-3)** :white_check_mark: | JSON-driven BPE adapter bridging HuggingFace `tokenizer.json` into the existing `BpeTokenizer` / `SentencePieceEncoding` infrastructure. `HfTokenizerJsonParser` decodes vocab, merges (both string `"a b"` and modern `[a,b]` array forms), added tokens, Metaspace pre-tokenizer, ByteFallback decoder stages. `HfBpeTokenizerFactory` maps merge rank → synthetic score (`score = -rank`) so the score-driven priority queue reproduces HF's earliest-rank-wins merge order. Special added tokens flagged `special:true` route through the control-token pre-split path; BOS/EOS auto-detected from `` / `` added-token entries with Llama-2 fallbacks (1/2). `ModelLoader.LoadTokenizerFromHfDirectory` surfaces it alongside `LoadFromSafetensors` without breaking the tuple contract. End-to-end verified on real `ib-ssm/mamba3-370M-10BT`: `"Hello world"` → `[15043, 3186]` (matches Llama-2 canonical IDs), non-ASCII `café` round-trips via byte fallback, `"The quick brown fox"` → forward through 48-layer 370M SSM → argmax decodes cleanly. Byte-level (GPT-2) pretokenizer intentionally deferred — separate adapter into `CreateTiktoken`. | 60 |
**Milestone**: DeepSeek-V2/V3 inference, SmolLM3 with NoPE, Gemma 4, and MoE models running correctly.
diff --git a/docs/SUPPORTED_MODELS.md b/docs/SUPPORTED_MODELS.md
new file mode 100644
index 00000000..223356a3
--- /dev/null
+++ b/docs/SUPPORTED_MODELS.md
@@ -0,0 +1,170 @@
+# Supported Models
+
+Authoritative matrix of every architecture variant currently declared by the
+`DotLLM.Core.Configuration.Architecture` enum, cross-referenced with the
+loader dispatch in `ModelLoader`, the field consumption in
+`HfConfigExtractor` / `GgufModelConfigExtractor`, and the verification
+evidence in the test tree.
+
+**How to read.** Each row is one `Architecture` enum variant. The `Verified
+on` column reports the strongest evidence that exists today, tagged per the
+legend at the bottom of this page. Rows tagged `planned` have an enum value
+but no working load-and-forward path. For deeper detail on a given row see
+its subsection below the matrix and the linked loader source.
+
+**Contributing a new architecture.** Add the enum variant in
+`src/DotLLM.Core/Configuration/Architecture.cs`, wire `ResolveArchitecture`
+in `src/DotLLM.Models/SafeTensors/HfConfigExtractor.cs` (or
+`GgufModelConfigExtractor` for GGUF-first), add a loader method under
+`src/DotLLM.Models/Architectures/`, extend the dispatch in
+`ModelLoader.LoadFromSafetensors` / `ModelLoader.LoadFromGguf`, then add a
+tiny-random integration test under
+`tests/DotLLM.Tests.Integration/Models/Loaders/` and a row here. See
+`docs/MODEL_CONFIG.md` for the config plumbing and `docs/ARCHITECTURE.md`
+for system context.
+
+## Matrix
+
+| Architecture | Enum | Tokenizer | RoPE | KV-cache | MoE | Required config fields | Verified on | Notes |
+|---|---|---|---|---|---|---|---|---|
+| Meta Llama | `Architecture.Llama` | HF tokenizer.json (BPE + ByteLevel) | RoPE Norm (interleaved pairs) | GQA | no | `hidden_size`, `num_hidden_layers`, `num_attention_heads`, `num_key_value_heads`, `intermediate_size`, `vocab_size`, `max_position_embeddings`, `rope_theta`, `rms_norm_eps`, `tie_word_embeddings` | `verified: real weights` (CPU + Vulkan) — TinyLlama-1.1B-Chat-v1.0 (2.1 GB, CPU `TinyLlama_11B_LoadsAndForwardsEndToEnd`; Vulkan `TinyLlama_VulkanForward_MatchesCpuReference_OnEightDecodeSteps`); Llama-3.2-1B-Instruct.Q8_0 GGUF (1.3 GB, Vulkan `Llama32_1B_Q8_0_VulkanForward_MatchesCpuReference`); Bielik-1.5B-v3.0 Q4_K_M GGUF (Vulkan `Bielik15B_Q4_K_M_VulkanForward_MatchesCpuReference`) | Dense SwiGLU, standard `q/k/v/o_proj`, `mlp.{gate,up,down}_proj`. GGUF `general.architecture = llama`. |
+| Mistral AI | `Architecture.Mistral` | HF tokenizer.json (BPE + ByteLevel or SPM) | RoPE Norm | GQA, optional `sliding_window` | no | Same as Llama plus optional `sliding_window` | `verified: tiny-random` (config only) — `HfConfigExtractorTests.Mistral_UsesNormRoPE`; `verified: real weights (gated)` — Mistral-7B-Instruct-v0.2.Q4_K_M GGUF via Vulkan `Mistral7B_Q4_K_M_VulkanForward_MatchesCpuReference` when `DOTLLM_MISTRAL_7B_Q4_K_M_GGUF` is set or the conventional cache path exists | Routes through the same `LoadLayer` path as Llama; GGUF `general.architecture = mistral` (or `mistral3`). |
+| Microsoft Phi | `Architecture.Phi` | HF tokenizer.json (BPE + ByteLevel) | RoPE NeoX (non-interleaved) | GQA | no | Llama set plus `architectures[0]` starting with `phi` or `model_type` in `{phi, phi2, phi3}`. `tie_word_embeddings` defaults to true. | `verified: real weights` (CPU + Vulkan) — Phi-3.5-mini-instruct (7.6 GB, CPU `Phi35Mini_LoadsAndForwardsEndToEnd`; Vulkan `Phi35Mini_VulkanForward_MatchesCpuReference_OnEightDecodeSteps`) | Fused tensors: `self_attn.qkv_proj.weight [Q+K+V, H]` and `mlp.gate_up_proj.weight [2*I, H]` are split at load time into independent F32 slabs (see `SplitFusedProjection`). |
+| Alibaba Qwen | `Architecture.Qwen` | HF tokenizer.json (BPE + ByteLevel) | RoPE NeoX | GQA, optional `sliding_window` | no | Llama set plus optional `head_dim` (Qwen3). Qwen2/3 commonly ship `q/k/v` biases; Qwen3 ships per-head `q_norm`/`k_norm` RMSNorms. | `verified: real weights` (CPU + Vulkan) — Qwen2.5-0.5B (999 MB, CPU `Qwen25_0_5B_LoadsAndForwardsEndToEnd`; Vulkan `Qwen25_0_5B_VulkanForward_MatchesCpuReference_OnEightDecodeSteps`) | `ResolveOptionalBias` picks up Qwen2 biases; `ResolveOptionalNorm` picks up Qwen3 QK-norms. `tied_embeddings=true` typical for small SKUs. |
+| DeepSeek (legacy) | `Architecture.DeepSeek` (**obsolete**) | GGUF tokenizer only (no HF safetensors dispatch) | RoPE | GQA (no MLA) | no | GGUF `general.architecture = deepseek` only | `legacy placeholder` — no working forward path. `TransformerArchitecture.CreateModel` throws for the pre-V2 arch and points users at DeepSeek-V2/V3. | Kept for GGUF metadata parsing back-compat. New checkpoints land on `DeepSeekV2`/`DeepSeekV3`; this variant is intentionally not a capability claim. |
+| DeepSeek-V2 | `Architecture.DeepSeekV2` | HF tokenizer.json (BPE + ByteLevel) | RoPE Norm + YaRN softmax mscale² + ramped frequency rescaling (long context >4K) | **MLA Phase A / B / C — Phase C (hybrid latent + absorbed decode) is now the loader default for both HF and GGUF**, with Phase A retained as the numerical oracle. Vulkan keeps Phase A (no-cache forward path) | yes: routed + multi-shared expert; `first_k_dense_replace` dense prefix | Llama set plus MLA block (`kv_lora_rank`, `qk_nope_head_dim`, `qk_rope_head_dim`, `v_head_dim`, optional `q_lora_rank`), MoE block (`n_routed_experts`, `num_experts_per_tok`, `moe_intermediate_size`, `n_shared_experts`, `first_k_dense_replace`), optional `rope_scaling` | `verified: real weights` (CPU + Vulkan) — DeepSeek-V2-Lite (30 GB BF16 SafeTensors CPU `DeepSeekV2Lite_LoadsAndForwardsEndToEnd` + `_LogitsMatchPyTorchReference` short-prompt + `_LongContext_LogitsMatchPyTorchReference` 4920-token ramped-YaRN — both within `DriftTolerances.Tight`); DeepSeek-V2-Lite-Q4_K_M GGUF (10.4 GB, Vulkan `DeepSeekV2Lite_Q4_K_M_VulkanForward_MatchesCpuReference` 19m 27s end-to-end). The HF SafeTensors path on Vulkan self-skips on hosts where the F32-expansion would exceed the configured budget (default 24 GB; override via `DOTLLM_VULKAN_F32_EXPANDED_BUDGET_GB`) | Attention routes through `LoadDeepSeekMlaLayer` → `MlaLayerWeights`; scalar `MlaAttention` kernel with optional cache pointers. `MlaExpandedKvState` caches expanded K_nope / V / K_pe across calls (Phase A); `MlaLatentKvState` stores compact `c_kv + k_pe` per token (Phase B/C). Phase C wires expand-to-MHA on prefill + absorbed MQA-style read on decode through the SAME latent cache (commit `5b2add7` merged `feature/mamba-3-phase-c`). Loader default to Phase C at `4b54a72` (HF) + `4724397` (GGUF). |
+| DeepSeek-V3 | `Architecture.DeepSeekV3` | HF tokenizer.json (BPE + ByteLevel) | RoPE Norm with optional YaRN | MLA | yes: sigmoid-router top-k, multi-shared expert | Same as DeepSeek-V2. Router discriminator is `architectures[0] = DeepseekV3ForCausalLM` or `model_type = deepseek_v3`. | `verified: tiny-random` — `yujiepan/deepseek-v3-tiny-random`, `katuni4ka/tiny-random-deepseek-v3` (same test) | Same attention path as V2; V3's node-level aux-loss-free routing + sigmoid scoring are V3-specific MoE refinements. No real-weight forward run in CI. |
+| NVIDIA Nemotron-H | `Architecture.NemotronH` | GGUF BPE (via `GgufBpeTokenizerFactory`) | RoPE on attention layers, none on SSM layers | GQA per attention layer; per-layer SSM state cache | no | GGUF `general.architecture = nemotron_h` with per-layer `head_count_kv` + `feed_forward_length` arrays (hybrid layout), plus SSM config keys (`ssm.*`) | `planned`/`verified: real weights (gated)` — `NemotronHTextGeneratorTests` run end-to-end against a local Nemotron-3-Nano-4B Q4_K_M GGUF when `DOTLLM_NEMOTRON_H_GGUF` is set; CI does not pull the checkpoint. Config-level detection is covered by `GgufModelConfigExtractorTests`. **Vulkan**: full forward path landed (`VulkanNemotronHTransformerModel` at `4abe4c2`) including SSM scan, attention, FFN, hybrid layer dispatch, plus Q8_0 / Q4_K / Q5_K / Q6_K / F16 / BF16 projection upload via the Phase 1 + Phase 8 Vulkan kernels. | **GGUF-only** — safetensors dispatch in `ModelLoader.LoadFromSafetensors` does NOT enumerate `NemotronH`. Activation function is `ReluSquared`, not SiLU. Dedicated `NemotronHTransformerModel` (CPU) and `VulkanNemotronHTransformerModel` (Vulkan) with a hybrid Mamba-2 / attention forward. |
+| Mamba-3 (pure SSM) | `Architecture.Mamba3` | HF tokenizer.json (SPM + Metaspace + ByteFallback for `ib-ssm`) | Data-dependent RoPE on B/C inside the SSM block | SSM state cache (no KV-cache) | no | `model_type = mamba3`, `hidden_size`, `num_hidden_layers`, `num_heads`, `head_dim`, `state_size`, `vocab_size`, `expand`, `n_groups`, `chunk_size`, `mimo_rank`, `is_mimo`, `is_outproj_norm`, `use_l2warp`, `tie_word_embeddings`, `rescale_prenorm_residual`, `rope_fraction` | `verified: real weights` (CPU + Vulkan) — `ib-ssm/mamba3-370M-10BT` (1.55 GB, CPU `IbSsmMamba3RealWeightsLoadTests`, Vulkan `Mamba3_VulkanForward_MatchesCpuReference_OnPromptPrefill`); `verified: tiny-random (synthetic)` — `TinyMamba3SafetensorsLoadTests` synthesises a deterministic miniature on disk; MIMO forward exercised by CPU `Mamba3TransformerModelMimoTests` and Vulkan `VulkanMamba3TransformerModelForwardTests` (SISO + MIMO + streaming-chunk + Q4_K + Q5_K + Q6_K + Q8_0 + F16 + BF16) synthetic fixtures | Non-MIMO real-weight + MIMO synthetic (commit `0499465`). No public MIMO checkpoint — real-weight MIMO verification deferred P4.3. No upstream GGUF mapping. **Vulkan**: full forward path landed (`VulkanMamba3TransformerModel` at `e40ada4` SISO + `7142f31` MIMO + `dfc9759` streaming-chunk + `effd8fc` Q8_0). |
+| Mistral Mixtral | `Architecture.Mixtral` | HF tokenizer.json | RoPE Norm | GQA | yes: `block_sparse_moe.gate` + per-expert `experts.{j}.w{1,2,3}` | Llama set plus `num_local_experts`, `num_experts_per_tok`. `architectures[0] = MixtralForCausalLM` or `model_type = mixtral`. | `verified: tiny-random` — `yujiepan/mixtral-tiny-random` (520 KB, `TinyMixtralSafetensorsLoadTests`); `verified: real weights (gated)` — `Mixtral8x7B_LoadsAndForwardsEndToEnd_WhenCheckpointPresent` when `DOTLLM_MIXTRAL_8X7B_CHECKPOINT_PATH` or `C:/temp/dotllm-mixtral-8x7b` is present. **Vulkan**: kernel-level + synthetic-fixture parity for the Mixtral MoE convention via `VulkanTransformerModelMoe*ForwardTests` (Q8_0 router/shared variants too). | Mixtral-convention MoE loader (`LoadMixtralMoeLayer`); no shared experts by design. |
+| Qwen-MoE (1.5 / 2 / 3) | `Architecture.QwenMoe` | HF tokenizer.json (BPE + ByteLevel) | RoPE NeoX | GQA, optional `sliding_window` | yes: `mlp.gate` + per-expert `experts.{j}.{gate,up,down}_proj`; optional shared-expert branch (Qwen1.5-MoE) with sigmoid gate; layer-level sparsity (Qwen3-MoE) via `decoder_sparse_step` + `mlp_only_layers` | Llama set plus `num_experts` or `num_local_experts`, `num_experts_per_tok`, `moe_intermediate_size`, optional `shared_expert_intermediate_size`, optional `norm_topk_prob`, optional `decoder_sparse_step`, optional `mlp_only_layers` | `verified: tiny-random` — `yujiepan/qwen3-moe-tiny-random` (20 MB, `TinyQwenMoeSafetensorsLoadTests`) and synthetic unit fixtures in `TransformerSafetensorsLoadTests` covering shared-expert + sigmoid-gate paths; `verified: real weights (gated)` — `Qwen15MoeA27B_LoadsAndForwardsEndToEnd_WhenCheckpointPresent` when `DOTLLM_QWEN15_MOE_A27B_CHECKPOINT_PATH` or `C:/temp/dotllm-qwen15-moe-a27b` is present | `LoadQwenMoeLayer` resolves both singular `shared_expert.*` (Qwen1.5-MoE-A2.7B) and plural `shared_experts.{k}.*` (DeepSeek, reused). |
+| IBM Granite-3.x MoE | `Architecture.GraniteMoe` | HF tokenizer.json (BPE + ByteLevel) | RoPE Norm | GQA | yes: fused per-layer `block_sparse_moe.{router.layer, input_linear, output_linear}` | Llama set plus `num_local_experts`, `num_experts_per_tok`, `moe_intermediate_size`. `architectures[0] = GraniteMoeForCausalLM` or `model_type = granitemoe`. | `verified: real weights` (CPU + Vulkan) — `ibm-granite/granite-3.0-3b-a800m-instruct` (6.3 GB, CPU `Granite3Moe_LoadsAndForwardsEndToEnd`; Vulkan `Granite3Moe_VulkanForward_MatchesCpuReference_OnEightDecodeSteps` 3m 56s) | Fused per-expert layout: `input_linear [E, 2*I, H]` packs w1 (rows `[0..I)`) + w3 (rows `[I..2*I)`), `output_linear [E, H, I]` packs w2. Each expert is upcast into its own F32 slab via `AllocPartAsF32`. No shared expert; typical top-k is unusually high (8 of 40). |
+
+**Row count: 12 / 12 `Architecture` enum variants covered.**
+
+## Per-architecture notes
+
+### Llama (`Architecture.Llama`)
+Dense SwiGLU transformer; `q/k/v/o_proj` split, optional biases off by
+default, no QK-norm. GGUF path: `TransformerArchitecture` →
+`TransformerModel.LoadFromGguf`. Safetensors path:
+`TransformerWeightsSafetensorsLoader.LoadLayer` in
+[`TransformerWeightsSafetensors.cs`](../src/DotLLM.Models/Architectures/TransformerWeightsSafetensors.cs).
+See [docs/MODEL_CONFIG.md](MODEL_CONFIG.md) and [docs/ATTENTION.md](ATTENTION.md).
+
+### Mistral (`Architecture.Mistral`)
+Same tensor shape as Llama; the discriminator is `architectures[0]` /
+`model_type`. Sliding-window attention is declared via `sliding_window`; the
+kernel correctness for contexts longer than the window is covered by
+PLANS.md P2.4. No end-to-end test in CI — config-only coverage.
+
+### Phi (`Architecture.Phi`)
+Phi-3 convention fuses `qkv_proj` and `gate_up_proj`; the safetensors loader
+splits them into Q/K/V and gate/up via `SplitFusedProjection` so the forward
+path is uniform with Llama/Mistral. Uses NeoX-style (non-interleaved) RoPE
+pairs. See [docs/POSITION_ENCODING.md](POSITION_ENCODING.md).
+
+### Qwen (`Architecture.Qwen`)
+Dense Qwen2 / Qwen3 models. Qwen2 commonly ships `q/k/v_proj.bias`; Qwen3
+additionally ships per-head `q_norm` / `k_norm` RMSNorm tensors. Both are
+resolved optionally so Qwen2 weights load without Qwen3 tensors and vice
+versa. Small SKUs tie embeddings (`tied_embeddings=true`).
+
+### DeepSeek legacy (`Architecture.DeepSeek`)
+Obsolete placeholder for older `general.architecture = deepseek` GGUF labels.
+`TransformerArchitecture.CreateModel` explicitly throws for this pre-V2
+variant and directs users to `DeepSeekV2` / `DeepSeekV3`, which are the
+supported MLA-based DeepSeek paths. This row exists as a capability claim
+audit: the enum member remains visible for public API and metadata
+compatibility, but it is intentionally not a forward-path claim.
+
+### DeepSeek-V2 (`Architecture.DeepSeekV2`) and DeepSeek-V3 (`Architecture.DeepSeekV3`)
+MLA attention (see [docs/ATTENTION.md](ATTENTION.md) and
+`MlaConfig` in [`src/DotLLM.Core/Models/`](../src/DotLLM.Core/Models/)):
+low-rank factorised Q (`q_a_proj` / `q_b_proj` — or monolithic `q_proj` on
+the Lite variant when `q_lora_rank = 0`) and KV (`kv_a_proj_with_mqa`,
+`kv_a_layernorm`, `kv_b_proj`) with decoupled RoPE on the `qk_rope_head_dim`
+slice only. MoE side: multi-shared-expert (DeepSeek uses plural
+`mlp.shared_experts.{k}.*` with no sigmoid gate), plus a dense-MLP prefix
+for the first `first_k_dense_replace` layers that is folded into
+`MlpOnlyLayers`. YaRN softmax mscale² correction is applied when
+`rope_scaling.factor > 1` and `mscale_all_dim != 0`; ramped YaRN frequency
+rescaling is covered by the long-context DeepSeek-V2-Lite reference test.
+**Phase C hybrid latent KV-cache** is the loader default for HF and GGUF:
+prefill keeps Phase-A-equivalent numerics while decode stores compact
+`[kv_lora_rank + qk_rope_head_dim]` latent state and uses the absorbed
+read path. `MlaExpandedKvState` remains the Phase A oracle for tests that
+construct `MlaConfig` directly.
+
+### NemotronH (`Architecture.NemotronH`)
+Hybrid Mamba-2 SSM + attention per-layer. Loaded **only** from GGUF —
+`ModelLoader.LoadFromGguf` dispatches to `NemotronHTransformerModel`, and
+`ModelLoader.LoadFromSafetensors` does not enumerate this arch. Config
+parsing uses per-layer `head_count_kv` + `feed_forward_length` arrays
+(hybrid layout) rather than the scalar keys; zero entries mark layers of
+the "other" kind. Activation is ReLU-squared, not SiLU.
+
+### Mamba3 (`Architecture.Mamba3`)
+Pure SSM — no attention, no convolution. Safetensors-first (no upstream
+GGUF mapping as of 2026-04). Loaded via `Mamba3ConfigExtractor` (HF config
+parsing), `Mamba3WeightLoader`, and `Mamba3TransformerModel`. The
+**non-MIMO** path is real-weight verified against `ib-ssm/mamba3-370M-10BT`
+(1.55 GB) and synthetic-fixture verified via `TinyMamba3SafetensorsLoadTests`.
+The **MIMO** path was blocked until P0.3 landed (commit `0499465`): the
+weight loader now resolves `mimo_x` / `mimo_z` / `mimo_o` tensors and
+`[H, R, N]`-shaped B_bias / C_bias, `Mamba3TransformerModel.Forward`
+dispatches to `ForwardMimo`, and `Mamba3TransformerModelMimoTests`
+exercises end-to-end MIMO forward on a synthetic fixture. Real-weight
+MIMO verification is indefinitely deferred (P4.3) because no public MIMO
+checkpoint exists.
+
+### Mixtral (`Architecture.Mixtral`)
+Dense transformer with top-k MoE FFN in every layer. Discriminator is
+`architectures[0] = MixtralForCausalLM` / `model_type = mixtral` — checked
+before the generic "mistral" substring match so that `mistralai/Mixtral-*`
+repo names don't shadow it. Tensor-name convention:
+`block_sparse_moe.gate` + `experts.{j}.(w1|w2|w3)`. No shared expert.
+Routed by `LoadMixtralMoeLayer`.
+
+### Qwen-MoE (`Architecture.QwenMoe`)
+Covers Qwen1.5-MoE-A2.7B, Qwen2-MoE, and Qwen3-MoE. Tensor naming follows
+HF Llama convention (`mlp.experts.{j}.{gate,up,down}_proj`), not Mixtral's
+`w1/w2/w3`. Qwen1.5 ships a singular `mlp.shared_expert.*` branch
+optionally gated by a sigmoid over `shared_expert_gate.weight`; Qwen3
+drops the shared branch entirely but adds layer-level sparsity via
+`decoder_sparse_step` + `mlp_only_layers` — dense Qwen-MoE layers fall
+through to the standard Llama SwiGLU loader. `LoadQwenMoeLayer` is also
+reused by the DeepSeek FFN path (with plural `mlp.shared_experts.{k}.*`).
+
+### GraniteMoe (`Architecture.GraniteMoe`)
+Fused-per-expert layout: all experts of one layer live in three rank-3
+tensors (`router.layer`, `input_linear`, `output_linear`). The loader
+(`LoadGraniteMoeLayer`) slices expert slabs out of `input_linear`
+(`[E, 2*I, H]` — w1 top half, w3 bottom half) and `output_linear`
+(`[E, H, I]` — w2), allocating per-expert F32 buffers via `AllocPartAsF32`.
+Unusually high top-k (8 of 40 on the 3B-A800M SKU). No shared expert.
+
+## Legend
+
+| Tag | Meaning |
+|---|---|
+| `verified: real weights` | A real HuggingFace checkpoint is downloaded (off-CI, gated on env var or conventional path) and `Forward` returns finite logits with non-zero stddev. Exact test name + checkpoint cited. |
+| `verified: tiny-random` | A tiny-random HF checkpoint (KB–MB, weights are random but shapes match the architecture) is fetched by the test runner into `~/.dotllm/test-cache/`, config is parsed and asserted, and — where the tiny-random ships usable weights — a forward pass runs. |
+| `verified: tiny-random (synthetic)` | The test builds a deterministic miniature checkpoint on disk at run time because no public tiny-random exists (Mamba-3 is the sole case). |
+| `planned` | Enum variant exists but no working load-and-forward path. Tagged when the loader throws `NotSupportedException` or no dispatch arm covers the variant. |
+| `legacy placeholder` | Public enum value is retained for metadata/API compatibility but is obsolete and intentionally not a supported forward path. |
+
+Evidence citations point to test types under
+[`tests/DotLLM.Tests.Integration/Models/Loaders/`](../tests/DotLLM.Tests.Integration/Models/Loaders/)
+and
+[`tests/DotLLM.Tests.Integration/Engine/`](../tests/DotLLM.Tests.Integration/Engine/).
+
+## See also
+
+- [docs/MODEL_CONFIG.md](MODEL_CONFIG.md) — `ModelConfig` schema and the parameterised architecture pattern
+- [docs/ARCHITECTURE.md](ARCHITECTURE.md) — system data flow
+- [docs/GGUF_FORMAT.md](GGUF_FORMAT.md) — GGUF key conventions consumed by `GgufModelConfigExtractor`
+- [docs/ATTENTION.md](ATTENTION.md) — GQA / MLA kernel selection
+- [docs/POSITION_ENCODING.md](POSITION_ENCODING.md) — RoPE Norm vs NeoX pair conventions
+- [docs/ROADMAP.md](ROADMAP.md) — step-by-step plan
+- [PLANS.md](../PLANS.md) — outstanding gaps on `feature/mamba-3`
diff --git a/docs/TOKENIZERS.md b/docs/TOKENIZERS.md
index 759397c1..7a153655 100644
--- a/docs/TOKENIZERS.md
+++ b/docs/TOKENIZERS.md
@@ -25,6 +25,105 @@ JSON format containing: model type, vocabulary, merges, pre-tokenizer config, no
Used when loading models from SafeTensors (which don't embed tokenizer in the weight file).
+#### Adapter: `HfTokenizerJsonParser` + `HfBpeTokenizerFactory`
+
+The `DotLLM.Tokenizers.Hf` namespace parses `tokenizer.json` and routes the
+pipeline to the matching BPE encoder. The parser strips down to the fields
+actually used for encode/decode (model vocab/merges, pre-tokenizer,
+normalizer, decoder, added tokens); post-processor / template logic is
+handled elsewhere.
+
+Two pipelines are supported today:
+
+| Pipeline | Pre-tokenizer (JSON) | Decoder (JSON) | Norm | BPE encoder | Checkpoints |
+|---|---|---|---|---|---|
+| **SentencePiece / Metaspace** | `Metaspace` or `null` | `Sequence[Replace, ByteFallback, Fuse, Strip]` | — | `BpeTokenizer.CreateSentencePiece` | Llama 1/2, Mistral, TinyLlama, Phi-3.5-mini, ib-ssm Mamba-3 |
+| **GPT-2 / ByteLevel** | `ByteLevel` **or** `Sequence[Split, ByteLevel]` | `ByteLevel` | NFC (Qwen) | `BpeTokenizer.CreateTiktoken{,WithRegex}` | Qwen2 / Qwen2.5, Granite-3 dense, GPT-2 proper |
+
+##### ByteLevel contract
+
+`ByteLevelPreTokenizer` owns the GPT-2 `bytes_to_unicode` mapping that
+defines the alphabet the BPE merges operate over:
+
+- Printable ASCII 33–126 and Latin-1 supplement 161–255 (minus 173) map to
+ their own code points.
+- All remaining bytes (0–32, 127–160, 173) — the control and whitespace
+ bytes — are pushed into `U+0100 + n` where `n` counts unassigned bytes in
+ ascending order. The effect is that BPE never has to merge across
+ boundaries caused by control bytes, because those bytes now live in a
+ private-range of valid Latin-extended characters.
+- UTF-8 continuation bytes are naturally handled: `é` = `0xC3 0xA9` →
+ `"é"` in the byte-level alphabet, which the merge table resolves to a
+ single token if the vocabulary contains that byte pair.
+
+`ByteLevelDecoder` is the inverse: reverse each token's chars to bytes,
+concatenate across the full sequence, UTF-8 decode.
+
+##### Regex ordering (correctness-critical)
+
+HF `ByteLevel` applies its pre-tokenization regex to the **raw text**
+before byte-mapping. The dotLLM encoder mirrors that order — applying the
+regex to byte-mapped text would misclassify multi-byte chars (`é` splits
+to `Ã` + `©` across segment boundaries, losing the merge).
+
+Regex source priority inside the factory:
+
+1. If `Sequence[Split(regex=X), ByteLevel(use_regex=false)]` (Qwen2 shape),
+ use `X` compiled from the JSON. The Qwen2 pattern is
+ `(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+`
+ — close to but distinct from the built-in GGUF `llama3` pattern.
+2. Else if standalone `ByteLevel(use_regex=true)` (Granite-3, GPT-2), use
+ the cached default GPT-2 pattern in `ByteLevelPreTokenizer.DefaultGpt2Regex`.
+3. Else (`use_regex=false` with no upstream Split), feed the whole byte-
+ mapped string to BPE as one segment.
+
+##### Normalizer
+
+When the JSON declares a `normalizer` of kind NFC/NFD/NFKC/NFKD, the
+factory wraps the inner tokenizer in a `NormalizingTokenizer` decorator
+that `text.Normalize(form)`s input before encode. Decode is untouched.
+Qwen2/Qwen2.5 declare NFC — necessary to keep composed vs decomposed
+forms of accented chars on the same code path the merge table was
+trained against.
+
+##### Architecture → pipeline map
+
+| Arch | Tokenizer | Pipeline |
+|---|---|---|
+| Llama 1 / 2 | SPM BPE | Metaspace |
+| Mistral | SPM BPE | Metaspace |
+| TinyLlama | SPM BPE | Metaspace |
+| Phi-3.5-mini | SPM BPE | Metaspace (not ByteLevel as some other Phi variants) |
+| Mamba-3 (ib-ssm) | SPM BPE | Metaspace |
+| Qwen2 / Qwen2.5 | GPT-2 BPE | `Sequence[Split, ByteLevel]` + NFC |
+| Llama 3 | GPT-2 BPE | ByteLevel (standalone, use_regex=true) |
+| Granite-3 dense | GPT-2 BPE | ByteLevel (standalone, use_regex=true) |
+| Granite-3 MoE | GPT-2 BPE (vocab.json, no tokenizer.json) | — out of adapter scope; uses slow-tokenizer path |
+| GPT-2 / GPT-4 | GPT-2 BPE | ByteLevel |
+
+##### Known gaps
+
+- **Multi-step Sequence pre-tokenizers.** DeepSeek-V2/V2-Lite uses
+ `Sequence[Split, Split, Split, Split, Split, Digits, ByteLevel]`. The
+ adapter only recognizes the `[Split, ByteLevel]` shape; other
+ compositions surface as `HfPreTokenizerKind.Sequence` and the factory
+ throws. Tracked as a P0.1 follow-up.
+- **Slow-tokenizer checkpoints.** Granite-3 MoE (and older Llama-family
+ models) ship `vocab.json` + `merges.txt` rather than `tokenizer.json`.
+ A separate parser path is required — not in scope for the
+ `tokenizer.json` adapter.
+- **Chat templates and post-processors** (BOS/EOS insertion, token type
+ IDs) are handled by callers, not the adapter.
+
+##### Entry points
+
+- `HfTokenizerJsonParser.Parse(jsonContent)` → `HfTokenizerSpec`.
+- `HfBpeTokenizerFactory.Create(spec, bosId=-1, eosId=-1)` → `ITokenizer`.
+- `HfBpeTokenizerFactory.TryLoadFromDirectory(dir)` → `ITokenizer?`
+ (one-liner for HF checkpoint dirs).
+- `ModelLoader.LoadTokenizerFromHfDirectory(path)` → same, from the model
+ loader's public API.
+
## ITokenizer Interface
```
diff --git a/docs/VULKAN.md b/docs/VULKAN.md
new file mode 100644
index 00000000..bccfd8e7
--- /dev/null
+++ b/docs/VULKAN.md
@@ -0,0 +1,225 @@
+# Vulkan Backend Architecture — dotLLM
+
+## Why a Vulkan Backend
+
+The `DotLLM.Cuda` backend is NVIDIA-only: it P/Invokes the CUDA Driver API
+(`libcuda.so` / `nvcuda.dll`) and cuBLAS, and loads PTX text files. On
+non-NVIDIA GPUs — AMD Radeon, Intel Arc, Apple Silicon (via MoltenVK),
+mobile Adreno/Mali — that path returns nothing.
+
+`DotLLM.Vulkan` exists to cover that gap. It uses the same P/Invoke
+philosophy as the CUDA backend — no custom C shared library — but targets
+the Vulkan loader (`vulkan-1.dll` / `libvulkan.so.1`) and loads SPIR-V
+compute shaders instead of PTX. The architectural parallel is exact:
+
+| | CUDA backend | Vulkan backend |
+|---|---|---|
+| Native loader | `libcuda.so` / `nvcuda.dll` | `libvulkan.so.1` / `vulkan-1.dll` |
+| Shader IR | PTX (text) | SPIR-V (u32 binary) |
+| Kernel source | CUDA C++ (`.cu`) | GLSL compute (`.comp`) |
+| Compiler | `nvcc -ptx` | `glslc --target-env=vulkan1.2` |
+| Module type | `CUmodule` | `VkShaderModule` |
+| Launch | `cuLaunchKernel` | `vkCmdDispatch` |
+| Vendor reach | NVIDIA only | AMD, NVIDIA, Intel, Apple (MoltenVK), Qualcomm, ARM |
+
+The existing gap-analysis in `docs/CUDA.md` §1 concludes that Vulkan
+compute is the only cross-vendor path with full custom-kernel expressivity
+and a credible route to Tensor-Core-class throughput via
+`VK_NV_cooperative_matrix2` (October 2024). That extension adds
+dequantization callbacks specifically for quantized LLM inference and is
+implemented by NVIDIA and AMD; Intel support is tracked in the Mesa ANV
+driver. Recent benchmarks in llama.cpp's `ggml-vulkan` backend reach
+~70–95% of native CUDA throughput on RTX 4090.
+
+## Scope of This PR
+
+**Proof of pipeline only.** This PR establishes the plumbing:
+
+- `DotLLM.Vulkan` project with raw `[LibraryImport("vulkan-1")]` P/Invoke
+ — no Silk.NET, no external bindings package. ~15 Vulkan entry points
+ covering instance, physical device, logical device, memory, buffer,
+ shader module, compute pipeline, descriptor sets, command buffer,
+ queue submit.
+- Shader build pipeline: `native/vulkan/shaders/*.comp` → `glslc` →
+ `native/vulkan/spv/*.spv`, driven by `build.sh` / `build.ps1` and also
+ wired into MSBuild (`CompileVulkanShaders` target) so shaders rebuild
+ incrementally when sources change and `glslc` is on PATH.
+- One working kernel: `add.comp` implements `c[i] = a[i] + b[i]` over
+ FP32 buffers.
+- `VulkanDevice` — instance creation, physical-device selection
+ (discrete > integrated; prefer AMD/NVIDIA over Intel integrated),
+ compute queue + command pool, host-visible buffer allocation,
+ upload/download.
+- `VulkanModule` — loads one `.spv`, creates compute pipelines by
+ entry-point name.
+- `AddKernel` — wraps `add.comp`; records a one-shot command buffer,
+ binds descriptor set, pushes `n` via push constant, dispatches,
+ waits.
+- Smoke test `VulkanAddKernelTests` — verifies 1024-element addition
+ round-trips correctly; skips when no Vulkan loader/device is present.
+
+**Not in scope.** Real LLM kernels, multi-GPU, staging ring, fence-based
+pipelining, `VK_NV_cooperative_matrix2`, descriptor-set pooling across
+launches, memory-type heuristics beyond HOST_VISIBLE|HOST_COHERENT.
+
+## SPIR-V Compilation Pipeline
+
+End users need only the compiled `.spv` blobs, which ship as MSBuild
+`Content` alongside the managed DLL. Shader *authors* need the Vulkan
+SDK (https://vulkan.lunarg.com/) for `glslc`. The MSBuild target is
+idempotent: if `glslc` is not on PATH it logs a warning and uses the
+committed `.spv` files.
+
+```
+native/vulkan/shaders/add.comp (author edits)
+ │
+ │ glslc --target-env=vulkan1.2 -o add.spv add.comp
+ ▼
+native/vulkan/spv/add.spv (checked in — ships to users)
+ │
+ │ in DotLLM.Vulkan.csproj
+ ▼
+bin/Debug/net10.0/spv/add.spv (loaded at runtime)
+ │
+ │ File.ReadAllBytes → vkCreateShaderModule
+ ▼
+VkShaderModule handle
+ │
+ │ vkCreateComputePipelines
+ ▼
+VkPipeline → vkCmdBindPipeline → vkCmdDispatch
+```
+
+The driver compiles SPIR-V to vendor ISA at `vkCreateComputePipelines`
+time and caches the result on disk (AMDGPU-PRO cache, Mesa shader cache,
+NVIDIA internal blob). First-launch cost is amortized across process
+restarts, same as PTX JIT under CUDA.
+
+## P/Invoke Strategy
+
+Identical to `DotLLM.Cuda`:
+
+- `[LibraryImport("vulkan-1")]` with source-generated marshalling.
+- `VulkanLibraryResolver` rewrites "vulkan-1" to the correct OS binary
+ (`vulkan-1.dll`, `libvulkan.so.1`, `libvulkan.dylib`).
+- Handles (`VkInstance`, `VkDevice`, `VkBuffer`, `VkDeviceMemory`, etc.)
+ cross the boundary as opaque `nint` — tensor bytes never traverse
+ P/Invoke.
+- `VkResult` returned as `int`; negative values are errors, zero is
+ `VK_SUCCESS`, positive values are non-error status codes
+ (`VK_INCOMPLETE`).
+- Structs declared `[StructLayout(LayoutKind.Sequential)]`. We only
+ declare the fields actually used; extension tails are left as
+ `fixed byte` padding (see `VkPhysicalDeviceProperties.limits`).
+
+No Silk.NET dependency. Adding Silk.NET.Vulkan would give us ergonomic
+bindings but ~15 MB of transitive DLLs and another layer to audit; the
+~15 Vulkan entry points we actually need fit in a single file.
+
+## Target Kernel Catalog
+
+Future sessions port the `DotLLM.Cuda` catalog (see
+`docs/CUDA.md` §Kernel Catalog) one kernel at a time. Order is chosen so
+each lands a testable end-to-end slice:
+
+| Phase | Kernel | CUDA file | Notes |
+|---|---|---|---|
+| 1 | `add` | `add.cu` | ✅ Done |
+| 2 | `rmsnorm_f32` | `rmsnorm_f32.cu` | ✅ Done; warp reduction → subgroup shuffle (`GL_KHR_shader_subgroup`); fused rmsnorm+matmul_q8_0 sub-tile variant landed |
+| 2 | `rope_f32` | `rope_f32.cu` | ✅ Done; sin/cos lookup; no reduction |
+| 2 | `swiglu_f32` | `swiglu_f32.cu` | ✅ Done; pointwise |
+| 3 | `embedding_*` | `embedding.cu` | ✅ Done; gather, F32 + Q8_0 in tree |
+| 3 | `bias_add_f32` | `bias_add_f32.cu` | ✅ Done |
+| 3 | `softmax` | `softmax.cu` | ✅ Done; two-pass (max, exp-sum), subgroup reduction |
+| 4 | `attention_f32` | `attention_f32.cu` | ✅ Done; standard + MLA (`AttentionMlaF32Kernel`) variants |
+| 5 | `dequant_q8_0` | `dequant.cu` | ✅ Done (CPU-side); Vulkan reads source bytes inline |
+| 5 | `dequant_q4_k` / `q5_k` / `q6_k` | `dequant.cu` | ✅ Done (CPU-side); Vulkan kernels read source bytes inline (Phase 1 K-quant work) |
+| 6 | `matmul_q8_0_*` | `quantized_gemv.cu` | ✅ Done; GEMV (`MatMulQ8_0Kernel`) + GEMM (`MatMulQ8_0GemmKernel`) + coopmat (`MatMulQ8_0GemmCoopmatKernel`) |
+| 6 | `matmul_q4_k_*` / `q5_k_*` / `q6_k_*` | `quantized_gemv.cu` | ✅ Done (Phase 1, commits `afb2272`/`b1ee6bc` Q4_K, `15099b9`/`83e0732` Q5_K, `29a1459`/`39b7646` Q6_K); GEMV + GEMM each |
+| 7 | F16 / BF16 pipeline | `*_f16.cu` | ✅ Done (Phase 8 brought forward, commits `c9c08c5`/`6787492`); F16 GEMV + GEMM + coopmat (F16xF16→F32, gfx1151), BF16 GEMV + GEMM (`uintBitsToFloat(bits<<16)`); F32 activations + native F16/BF16 weights on device |
+| 8 | Cooperative matrix | (new) | ✅ Partial (F16 + Q8_0 GEMM coopmat tiles via `GL_KHR_cooperative_matrix`; gfx1151 verified). MoE Strategy C now has a gated F16 grouped path: expert-count/offset, group-by-expert, grouped F16 coopmat matmul, ungroup scatter, and end-to-end synthetic forward parity. K-quant coopmat variants and non-F16 grouped MoE remain follow-ups |
+
+Milestone 7 (FP16) is a significant enabler: most Vulkan drivers expose
+`shaderFloat16` only through those two extensions. `VK_KHR_16bit_storage`
+lets us read/write FP16 in storage buffers; `shader_float16_int8` lets
+shaders operate on `float16_t` natively.
+
+Milestone 8 (cooperative matrix) is the cross-vendor equivalent of
+NVIDIA's `mma.sync` / Tensor Cores. `VK_NV_cooperative_matrix2` (Oct
+2024) is the ticket to ~90% of cuBLAS FP16 GEMM throughput on a 4090,
+and AMD's equivalent is on its RDNA3+ driver roadmap.
+
+## GLSL Conventions
+
+All kernels live under `native/vulkan/shaders/*.comp`. Conventions
+mirror the CUDA kernels where possible:
+
+- One `.comp` file per kernel; file name matches the CUDA file name
+ (e.g. `rmsnorm_f32.comp` ↔ `rmsnorm_f32.cu`).
+- `#version 450` for the baseline; bump to `#version 460` when
+ cooperative matrix is needed.
+- `layout(local_size_x = 256, local_size_y = 1, local_size_z = 1) in;`
+ matches the CUDA block size of 256.
+- Storage buffers use `std430` layout and `readonly`/`writeonly`
+ qualifiers as appropriate.
+- Scalar uniforms (sizes, strides, configuration) are passed through
+ push constants (max 128 bytes per pipeline). Larger config goes in
+ a uniform buffer.
+- Entry point is always `void main()`. The C# `AddKernel` passes
+ `"main"` as the entry-point name to match.
+
+## Physical-Device Selection
+
+`VulkanDevice.Create` enumerates all physical devices and scores them:
+
+- Device type: discrete (+1000), integrated (+500), virtual (+100), other (0).
+- Vendor: NVIDIA/AMD (+20), Intel (+10), other (+5).
+
+Highest score wins. Tie-breaking is first-enumerated. This prioritizes
+a discrete GPU over an integrated one on hybrid laptops, and prefers
+AMD/NVIDIA discrete over the (rare) Intel Arc discrete when both are
+present. The heuristic is deliberately dumb — real placement policy
+(explicit `--device` CLI flag, per-request device hints) is
+infrastructure for later.
+
+## Deferred Work
+
+Tracked for future sessions:
+
+1. **Staging buffers for large uploads.** The scaffold allocates
+ host-visible buffers directly, which is OK for 1024 floats but will
+ burn bandwidth on multi-GB model weights. Real uploads need a
+ device-local destination plus a host-visible staging ring and
+ `vkCmdCopyBuffer`.
+2. **Descriptor-set pool reuse.** `AddKernel` currently allocates a
+ descriptor set per launch. Port the per-kernel cache pattern from
+ `CudaKernels`.
+3. **Fence-based pipelining.** Every launch currently does
+ `vkQueueWaitIdle` — synchronous, no overlap with host work.
+ Replace with `VkFence` + per-in-flight command-buffer arena.
+4. **`IBackend` integration.** `DotLLM.Vulkan` does not yet implement
+ `DotLLM.Core.Backends.IBackend`. Hooking it up needs the
+ `VulkanTransformerModel` equivalent of `CudaTransformerModel`, which
+ in turn needs the attention/rmsnorm/rope kernels.
+5. **Model loading.** GGUF weights → Vulkan buffers requires the mmap
+ + staging path above.
+6. **Validation layers.** `VK_LAYER_KHRONOS_validation` at
+ instance-create time gives llama.cpp-style debug output. Opt-in via
+ env var (`DOTLLM_VULKAN_VALIDATION=1`) once the SDK-install-check
+ story is sorted.
+
+## Building
+
+```bash
+# .NET build — compiles C# and (if glslc is on PATH) shaders.
+dotnet build src/DotLLM.Vulkan/DotLLM.Vulkan.csproj
+
+# Shader-only rebuild.
+./native/vulkan/build.sh # Linux / macOS / WSL / Git Bash
+pwsh ./native/vulkan/build.ps1 # Windows PowerShell
+
+# Run the scaffold test — passes if a Vulkan device is present, else skips.
+dotnet test tests/DotLLM.Tests.Unit --filter FullyQualifiedName~Vulkan
+```
+
+Opt out on CI without a usable driver: `DOTLLM_SKIP_VULKAN=1`.
diff --git a/docs/perf/DEEPSEEK_QUANTIZED_GPU_PATH.md b/docs/perf/DEEPSEEK_QUANTIZED_GPU_PATH.md
new file mode 100644
index 00000000..c6e060e0
--- /dev/null
+++ b/docs/perf/DEEPSEEK_QUANTIZED_GPU_PATH.md
@@ -0,0 +1,281 @@
+# Quantized DeepSeek-V2 GPU Weight Paths (Tasks #9 + #10)
+
+Design note. Maps the work needed to take real
+`DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf` from "metadata loads" to
+"prefill + decode runs end-to-end on the RTX 3060".
+
+## Status (2026-04-27)
+
+- **#9-i, #9-ii, #9-iii (quantized MLA)** — COMPLETE.
+- **#10-i, #10-ii, #10-iii (quantized MoE)** — COMPLETE.
+- **#11 (real-checkpoint smoke)** — COMPLETE (1/2/4/8-layer V2-Lite Q4_K_M).
+- **Phase A (direct quantized GEMV in CudaMoeFfn)** — COMPLETE in commit
+ `4a09dbe`. Replaces the dequant→F16→F32→LinearF32 chain with
+ `LaunchConvertF32ToF16` + `LaunchQuantizedGemv` + `LaunchConvertF16ToF32`
+ for K-aligned projections (gate_proj + up_proj at hidden=2048).
+ down_proj at K=intermediate=1408 stays on the dequant fallback (K not
+ 256-aligned). This is a perf-only improvement; doesn't move the GPU
+ memory cap.
+- **8-layer ceiling on RTX 3060 12 GB** — empirical, see Round 10.
+ 16+ layers GPU-OOM. The path to fitting more layers is **NOT this
+ document's design**; it's separate (expert offloading or grouped-GEMM
+ cross-expert batching).
+
+## Constraint summary
+
+- DeepSeek-V2-Lite has 16B params on disk at Q4_K_M (~10.4 GB).
+- F32 dequant of the full MoE expert block: **~57 GB host RAM** (64 experts ×
+ 3 projections × 1408 × 2048 × 4 bytes × 26 MoE layers). Untenable.
+- F16 dequant on GPU once at load: **~28 GB GPU RAM**. Doesn't fit in 12 GB.
+- **Only viable path**: keep raw Q4_K bytes on GPU, dequant per call.
+
+## What's already in place (from prior sessions)
+
+- `quantized_gemv_*` kernels for Q4_K / Q5_K / Q6_K / Q8_0 (`native/kernels/quantized_gemv*.cu`)
+- `LaunchDequantize` GPU kernel (Q4_K → F16 device scratch)
+- The GQA `Project` helper in `CudaTransformerModel.cs` already routes
+ decode through `LaunchQuantizedGemv` and prefill through
+ `LaunchDequantize` + cuBLAS HGEMM
+- Pre-Q8_1 + MMVQ-large infra for fastest decode-batch=1 path
+- GGUF DeepSeek-V2/V3 loader (config + MLA tensors + MoE 3D-stacked
+ experts), CPU-side, F32-dequant only
+
+## The five sub-tasks (in implementation order)
+
+### #9-i. CPU GGUF MLA loader: skip F32 dequant, store raw quant bytes
+
+`TransformerWeights.LoadMlaLayer` currently dequants every MLA tensor to
+an F32 host buffer via `DequantToF32`. The fix:
+
+```csharp
+// Today
+nint qAProj = DequantToF32(dataBase, tensors[$"{prefix}.attn_q_a.weight"], ..., owned);
+
+// Target
+(nint qAProj, QuantizationType qAQt) = LoadLinearRaw(dataBase, tensors[$"{prefix}.attn_q_a.weight"]);
+```
+
+`LoadLinearRaw` returns the mmap pointer + quant type without copying
+or dequanting. The `MlaLayerWeights` struct gains parallel `nint`+`QuantizationType`
+fields per projection (or a discriminated wrapper struct). The CPU
+`MlaAttention.Execute` path stays F32-only by keeping the existing
+`DequantToF32` route gated on a flag — but the GPU path uses the raw
+pointers directly.
+
+**Cost: ~2h.** Touches `TransformerWeights.cs`, `MlaLayerWeights`,
+the CPU `MlaAttention.Execute` if it needs an F32-dequant fallback.
+
+### #9-ii. CUDA MLA loader: upload raw Q4_K + add per-projection dequant
+
+`CudaMlaWeightsLoader.LoadLayerF16` today reads F32 host pointers via
+`UploadF32AsF16`. The new path:
+
+```csharp
+public static CudaMlaLayerWeights LoadLayerQuant(
+ in TransformerLayerWeights cpuLayer, int hiddenSize, List allocs)
+{
+ // Upload raw quantized bytes — same RowByteSize math as GQA path
+ nint qAProjQuant = UploadQuant(cpuLayer.Mla.QAProjPtr, cpuLayer.Mla.QAProjQt, qLora, hidden, allocs);
+ // ... etc for q_b, kv_a, kv_b, o_proj
+ return new CudaMlaLayerWeights(..., MlaPrecision.Quantized);
+}
+```
+
+Add `MlaPrecision.Quantized` enum variant. Per-projection scratch sized
+for the largest projection (q_b or kv_b expansion = ~2048 × 1408 × 2 = 5.6 MB).
+The scratch is reused across all 27 layers' projections — single allocation.
+
+**Cost: ~2h.** Touches `CudaMlaWeights.cs` + `CudaMlaWeightsLoader.cs`.
+
+### #9-iii. CudaMlaAttention.ForwardF16: branch on precision
+
+```csharp
+if (layer.Precision == MlaPrecision.Quantized)
+{
+ // Decode (seqLen=1): direct quantized GEMV
+ if (seqLen == 1 && _kernels.HasQuantizedGemv(layer.QAProjQt))
+ {
+ _kernels.LaunchQuantizedGemv(
+ layer.QAProj, layer.QAProjQt, scratch.NormHidden, scratch.QLatent,
+ qLora, hiddenSize, stream);
+ }
+ else
+ {
+ // Prefill: dequant to scratch, then HGEMM
+ _kernels.LaunchDequantize(layer.QAProj, layer.QAProjQt, dequantScratch, qLora, hiddenSize, stream);
+ CudaGemm.LinearF16(cublasHandle, scratch.NormHidden, dequantScratch, scratch.QLatent,
+ seqLen, hiddenSize, qLora, stream);
+ }
+}
+else { /* existing HGEMM path */ }
+```
+
+Same pattern for q_b, kv_a_proj_with_mqa, kv_b, o_proj. The scratch
+buffer is owned by the caller-passed `CudaMlaScratchF16`.
+
+**Cost: ~3h.** Touches `CudaMlaAttention.cs` (mostly the F16 sibling — it
+has the existing 5 cuBLAS LinearF16 calls; each gets a precision branch).
+
+### #10-i. CPU GGUF MoE loader: skip F32 dequant, store raw 3D-stacked
+
+`TransformerWeights.LoadDeepSeekMoeLayer` today dequants each expert's
+slice via `SliceExpertsToF32`. New path: store the raw GGUF tensor
+pointer + per-expert byte stride + quant type. The `MoeLayerWeights`
+struct gains a parallel "quant" view.
+
+**Cost: ~2h.** Touches `TransformerWeights.cs`, `MoeLayerWeights`.
+
+### #10-ii. CUDA MoE loader + Forward: per-expert quantized GEMV
+
+```csharp
+// Per-expert in CudaMoeWeightsLoader.LoadLayer:
+var w1Quant = new nint[numExperts];
+var w1Qt = quantType;
+for (int e = 0; e < numExperts; e++)
+{
+ w1Quant[e] = UploadQuantSlice(rawBase, e * perExpertBytes, perExpertElements, qt, allocs);
+}
+```
+
+In `CudaMoeFfn.Forward`, the per-expert SwiGLU loop branches:
+
+```csharp
+if (weights.Precision == MoePrecision.Quantized)
+{
+ foreach (active expert e):
+ _kernels.LaunchQuantizedGemv(weights.W1Quant[e], weights.W1QuantType,
+ gatheredInput, gateBatch_e_slice, intermediate, hidden, stream);
+ // similar for W3 (up), then SwiGLU, then W2 (down).
+}
+else { /* existing F32 cuBLAS path */ }
+```
+
+For DeepSeek-V2-Lite Q4_K_M:
+- 64 routed experts × 3 projections × Q4_K decode = 192 kernel launches/layer/token
+- @ 22 µs/launch on WDDM = 4.2 ms/layer just for FFN dispatch overhead — too slow.
+- **Mitigation**: graph capture for the active-expert subset, OR grouped-GEMM (single
+ kernel that walks the bucketed assignments). Stretch goal — sequential first.
+
+**Cost: ~4h.** Touches `CudaMoeWeights.cs` + `CudaMoeFfn.cs`.
+
+### Phase B: cross-expert grouped-GEMM kernel — NOT YET STARTED
+
+The literal "grouped GEMM" — single CUDA kernel that walks K_active
+experts in one launch instead of issuing one cuBLAS call per active
+expert per projection. Combined with raw-quant-on-the-fly weight reads
+(avoiding the per-expert resident GPU buffers), this is the path to:
+
+1. **Reducing dispatch overhead** further: today (post-Phase A) we
+ issue 3 launches per active expert per projection (F32→F16, GEMV,
+ F16→F32). Grouped kernel handles all K active experts in 1 launch
+ per projection. For K=6, V2-Lite, 26 MoE layers: ~40% of remaining
+ dispatch overhead.
+2. **(Stretch) Reducing GPU resident memory**: if the kernel streams
+ raw quant blocks from a single contiguous per-tensor buffer (keeping
+ one ffn_*_exps tensor on GPU rather than 64 per-expert allocations),
+ bookkeeping shrinks ~10-15%. Doesn't unblock 16+ layers on its own.
+
+Sketch:
+
+```c
+extern "C" __global__ void moe_grouped_gemv_q4_k_f16(
+ const half* __restrict__ x, // [K] F16 input row, shared
+ const uint8_t* const* __restrict__ weights, // K_active per-expert weight ptrs
+ half* const* __restrict__ outputs, // K_active per-expert output ptrs
+ int M, int K, int K_active)
+{
+ // Each block: one (expert, output_row) pair.
+ int blocks_per_expert = M / 1; // one block per output row
+ int expert_idx = blockIdx.x / blocks_per_expert;
+ int row_idx = blockIdx.x % blocks_per_expert;
+
+ if (expert_idx >= K_active) return;
+
+ // Standard Q4_K dot product over K elements...
+ // Output one F16 to outputs[expert_idx][row_idx].
+}
+```
+
+**Implementation cost:** ~1-2 days focused work — new CUDA kernel +
+PTX module + per-quant-type variants (Q4_K, Q5_K, Q6_K, Q8_0) + tests.
+Output goes through `LaunchMoeGroupedGemv` helper that mirrors
+`LaunchQuantizedGemv` but accepts arrays of pointers.
+
+The Phase A speedup was already meaningful — Phase B's incremental win
+shrinks now that the dequant scratch overhead is eliminated. Phase B's
+real value is opening the door to memory-streaming variants that fit
+larger models, not the dispatch reduction itself.
+
+### #11. Smoke test on cached real DeepSeek-V2-Lite GGUF
+
+```csharp
+[SkippableFact]
+public void RealGguf_DeepSeekV2Lite_PrefillDecode_OnCuda()
+{
+ string path = "~/.dotllm/models/bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF/...Q4_K_M.gguf";
+ Skip.If(!File.Exists(path));
+
+ using var gguf = GgufFile.Open(path);
+ var config = GgufModelConfigExtractor.Extract(gguf.Metadata);
+ using var gpu = CudaTransformerModel.LoadFromGguf(gguf, config);
+ int[] tokenIds = gpu.Tokenizer.Encode("def fibonacci(n):");
+
+ using var prefill = gpu.Forward(tokenIds, positions: [0,1,...], deviceId: 0, kvCache: null);
+ AssertFiniteLogits(prefill);
+
+ int curTok = ArgMax(prefill);
+ for (int i = 0; i < 3; i++)
+ {
+ using var step = gpu.Forward([curTok], [tokenIds.Length + i], 0, null);
+ AssertFiniteLogits(step);
+ curTok = ArgMax(step);
+ }
+}
+```
+
+Doesn't compare against a reference (no CPU oracle at this scale on this
+box) — just asserts the path runs without crashing or producing NaN.
+Real correctness validation comes from a follow-up CPU↔CUDA parity run
+on the Ryzen+iGPU box (where the safetensors HF model already loads).
+
+**Cost: ~1h.**
+
+## Total estimated cost
+
+**14 hours of focused work.** Likely 2 sessions of 6-8h each. The first
+session lands #9-i through #9-iii (MLA only) — that already gets the
+attention block on real-quant weights, even though MoE layers fall back
+to F32 dequant. Second session does #10 + #11.
+
+## Risks
+
+1. **Quantized GEMV kernels assume `K % 256 == 0`** for Q4_K/Q5_K/Q6_K
+ block alignment. V2-Lite hidden=2048 = 8×256 ✓; intermediate=10944 = 42.75×256
+ ✗ for the dense FFN at layer 0 (existing GQA path may have a fallback —
+ need to verify). MoE intermediate=1408 = 5.5×256 ✗ — definitely needs
+ the dequant-then-HGEMM fallback for some projections.
+
+2. **Per-expert kernel launch count** at 192/layer/token: if WDDM overhead
+ stays at 22 µs, that's 4.2 ms/layer × 26 MoE layers = 109 ms FFN overhead
+ per token. Decode would be < 10 tok/s. Only acceptable if we accept it
+ for the v1; otherwise grouped-GEMM is required.
+
+3. **MoE 3D-stacked tensor on-device upload**: each layer's `ffn_gate_exps`
+ is `[hidden, intermediate, num_experts]` raw bytes. We need to keep
+ the 3D layout AND extract per-expert slice pointers. Probably easier to
+ upload the whole tensor as one contiguous buffer and compute per-expert
+ slice offsets at GEMV-launch time.
+
+## What to do first next session
+
+1. `git pull origin feature/mamba-3-cuda` — should match `9ebba26`.
+2. Confirm the 10.4 GB GGUF is still cached at
+ `~/.dotllm/models/bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF/...Q4_K_M.gguf`.
+3. Run `RealGguf_ConfigExtractor_ParsesDeepSeekV2Lite` to confirm the
+ metadata extractor still works.
+4. Start on #9-i. Acceptance gate: existing CPU MLA tests still pass
+ AND the F32-dequant fallback path produces bit-identical results.
+5. Once #9 lands, attempt the smoke test #11. Even with MoE F32-host-OOM,
+ you can `DebugMaxLayers = 2` on `CudaTransformerModel` to test only
+ layers 0 (dense FFN, MLA attention) — that exercises the new
+ quantized MLA path without hitting MoE.
diff --git a/docs/perf/MLPUP_GEMV_GAP.md b/docs/perf/MLPUP_GEMV_GAP.md
new file mode 100644
index 00000000..8c6e4cef
--- /dev/null
+++ b/docs/perf/MLPUP_GEMV_GAP.md
@@ -0,0 +1,476 @@
+# MlpUp GEMV bandwidth gap vs llama.cpp — research note
+
+**Branch:** `cuda-mlpup-research` (forked from `feature/mamba-3-cuda` @ `a247909`)
+**Author:** dotnet-perf-expert agent, 2026-04-25
+**Status:** Research only. No kernel changes in this commit.
+**Issue:** dotLLM Qwen3-8B Q4_K_M decode at 10.5 tok/s on RTX 3060;
+llama.cpp CUDA reports ~30–42 tok/s on the same model + GPU. The gap is
+concentrated in `MlpUp` (fused gate+up Q4_K GEMV at n=24576, k=4096), which
+consumes 41% of GPU time and runs at ~50 GB/s effective HBM bandwidth — only
+14% of the RTX 3060's 360 GB/s peak.
+
+## TL;DR
+
+The dotLLM MMQ Q4_K kernel (`native/kernels/quantized_gemv_mmq.cu`) was tuned
+for **small models** (SmolLM-135M at k=576), where each row covers only 2
+super-blocks and the natural "1 thread per super-block" parallelism leaves 254/256
+threads idle. The 4-rows-per-block tiling fixed that.
+
+**For large models (k≥4096), that tiling is now actively hurting.** Each block
+processes 4 rows × 16 super-blocks = 64 work units across 256 threads (a
+4-units-per-thread stride loop), then performs a 256-wide cross-warp shared-memory
+reduction per row × 4 rows. Most of the 256 threads do tiny work then sit on
+`__syncthreads()` waiting for the reduction — and the reduction itself adds
+shared-memory traffic and a barrier that aren't on the critical path of a
+well-saturated GEMV.
+
+llama.cpp's `mul_mat_vec_q` kernel (the GEMV/batch=1 path) takes the exact
+opposite approach for ncols_dst=1 on Ampere:
+- **1 row per CUDA block** (not 4)
+- **4 warps × 32 = 128 threads per block** (not 256)
+- **No cross-warp shared-memory tree** — each warp handles its own dot product
+ slice, and the final reduction is `warp_reduce_sum` after a small shmem fan-in
+- All 128 threads contribute to the same row's dot product, walking
+ super-blocks via `blocks_per_iter = vdr * nwarps * warp_size / qi = 8`
+ super-blocks per iteration
+
+**Recommended single change:** restructure the MMQ Q4_K kernel for the
+large-k regime to mirror llama.cpp's mmvq: 1 row per block, 4 warps (128
+threads), warp-coherent super-block stride, single warp-shuffle reduction at
+the end. Keep the 4-rows-per-block variant as a separate kernel selected when
+`k ≤ 1024` (≤ 4 super-blocks/row), where the small-row amortization still
+helps. Estimated win: **2–4× on MlpUp** (50 → 100–200 GB/s), which is
+**~5 tok/s → ~12–20 tok/s** on Qwen3-8B at the system level.
+
+---
+
+## 1. Setup verification — the 50 GB/s number
+
+The bandwidth claim is derivable from numbers already in `.continue-here.md`
+without re-running the profiler. Showing the math here:
+
+**Per-call work (one MlpUp GEMV per layer):**
+- n = 24576 (= 2 × intermediate_size 12288, fused gate+up)
+- k = 4096
+- Q4_K_M weight bytes per row: `(k / 256) × 144 = 16 × 144 = 2304 bytes`
+- Total weight bytes read: `n × 2304 = 24576 × 2304 = 56.6 MiB`
+- Activation read: `k × 2 = 8 KiB` FP16 (negligible vs weight)
+- Output write: `n × 2 = 48 KiB` FP16 (negligible)
+
+**Per-call timing** (from profiler median, eager path, Qwen3-8B Q4_K_M):
+- MlpUp category total: 40.111 ms / token / 36 layers = **1.1142 ms / call**
+
+**Effective HBM bandwidth:**
+- 56.6 MiB / 1.1142 ms = **50.8 GB/s**
+
+**RTX 3060 12GB peak HBM bandwidth:** 360 GB/s (192-bit GDDR6 @ 15 Gbps).
+
+**Achieved fraction:** 50.8 / 360 = **14.1%**.
+
+This matches the number in `.continue-here.md` §3c. Confirmed without a fresh
+benchmark run.
+
+> **Operational note:** the box has background VRAM hogs (RustDesk, Edge, Ollama)
+> that consume ~6 GiB. Qwen3-8B fits with cleared background processes but
+> Q4_K_M loads at the very edge of the available 12 GiB after weight dedup
+> (commit `15becbf` brought resident weights from 7.8 → 5.3 GiB). HBM
+> contention with these background apps will manifest as further bandwidth
+> drop, but the 50 GB/s number is from a profiling run with cleared
+> background — the kernel is leaving headroom on the GPU side.
+
+---
+
+## 2. llama.cpp structure summary — what they actually do for batch=1 Q4_K
+
+Source files analyzed (master branch, accessed 2026-04-25):
+- `ggml/src/ggml-cuda/mmvq.cu` — the GEMV (batch=1, "matrix×vector") path
+- `ggml/src/ggml-cuda/mmq.cu` and `mmq.cuh` — the **batched** GEMM path
+ (only used at `ne11 ≥ MMQ_DP4A_MAX_BATCH_SIZE`, not for token-by-token decode)
+- `ggml/src/ggml-cuda/vecdotq.cuh` — per-quant-type vec_dot kernels (the
+ `vec_dot_q4_K_q8_1_impl_vmmq` function below)
+- `ggml/src/ggml-common.h` — `QK_K=256`, `QR4_K=2`, `QI4_K=32`, `QK8_1=32`
+
+### 2a. MMQ vs MMVQ — which one does decode use?
+
+**Both kernels use dp4a on Ampere for Q4_K.** They differ in batch dimension:
+- **`mul_mat_vec_q` (mmvq.cu)** — batch=1 (and small batches up to 8). This
+ is what decode-step GEMV actually goes through. It does **not** use tensor
+ cores — only `dp4a` via `ggml_cuda_dp4a`.
+- **`mul_mat_q` (mmq.cu)** — batch ≥ ~8. On Ampere with Q4_K it uses
+ **tensor cores (mma.sync m16n8k16 INT8)** via the `MMQ_MMA_TILE_X_K_Q8_1`
+ path. This is the prefill/batched path.
+
+Implication for the dotLLM gap: **tensor cores are NOT in the decode-path
+critical kernel for llama.cpp either.** Hypothesis 2 ("mma.m16n16k16 INT8")
+in the original task brief is therefore lower-priority than initially
+estimated — llama.cpp itself doesn't pull the tensor-core lever for
+batch=1 GEMV on Ampere. The 30–42 tok/s they hit on this hardware comes
+from a well-tuned dp4a kernel, not from tensor cores.
+
+### 2b. The mmvq launch parameters (Q4_K, ncols_dst=1, Ampere)
+
+From `mmvq.cu` `calc_nwarps` and `calc_rows_per_block` (both `constexpr` —
+fully resolved at compile time per `` template):
+
+```cuda
+// MMVQ_PARAMETERS_GENERIC (= NVIDIA non-MMA, NVIDIA Ampere, NVIDIA Hopper, ...)
+case ncols_dst == 1: nwarps = 4
+case ncols_dst == 1: rows_per_cuda_block = small_k ? nwarps : 1
+```
+
+`small_k` is a runtime path that dotLLM's k=4096 does not trigger. So at
+**k=4096, ncols_dst=1**:
+- `rows_per_cuda_block = 1`
+- `nwarps = 4`
+- block size = `(warp_size, nwarps, 1) = (32, 4, 1)` = **128 threads/block**
+- grid = `(nrows_x / 1, 1, 1) = (24576, 1, 1)` blocks for our MlpUp shape
+
+For Q4_K specifically, `vdr = VDR_Q4_K_Q8_1_MMVQ = 2`, `qi = QI4_K = 32`,
+so `blocks_per_iter = vdr × nwarps × warp_size / qi = 2 × 4 × 32 / 32 = 8`
+super-blocks per iteration. With `blocks_per_row_x = 4096/256 = 16`
+super-blocks/row, each block does **2 iterations** total to cover one row.
+
+### 2c. The mul_mat_vec_q outer loop (cited; not copied)
+
+The structural pattern from `mmvq.cu` `mul_mat_vec_q`:
+
+```
+tid = warp_size * threadIdx.y + threadIdx.x // 0..127
+row0 = blockIdx.x * 1 // one row per block
+for (kbx = tid/(qi/vdr); kbx < blocks_per_row_x; kbx += blocks_per_iter):
+ kqs = vdr * (tid % (qi/vdr))
+ // single dot-product per row, vec_dot_q_cuda walks one super-block chunk
+ tmp[0][0] += vec_dot_q4_K_q8_1(vx, &y[kby], row0*stride + kbx, kqs)
+
+// final reduction: warps 1..3 dump tmp into shared memory,
+// warp 0 sums across the (nwarps-1) staged values + warp_reduce_sum
+```
+
+The key points to compare with dotLLM:
+1. **No on-the-fly input quantization in this kernel.** The `vy` argument is
+ already Q8_1-quantized (32 INT8 + half scale + half partial sum, packed
+ into `block_q8_1`). llama.cpp runs a separate `quantize_row_q8_1_cuda`
+ kernel earlier in the forward pass that converts FP16 activations to Q8_1
+ *once per matmul group*.
+2. **One row per block.** A given dp4a result lives in registers from
+ start to finish on a single warp's lanes — no per-row shmem accumulator.
+3. **Block size is 128, not 256.** Fewer threads, so each thread does
+ more work (better arithmetic intensity per thread, less reduction
+ overhead).
+
+### 2d. The Q4_K vec_dot inner kernel (from `vecdotq.cuh`)
+
+```cuda
+static __device__ __forceinline__ float vec_dot_q4_K_q8_1_impl_vmmq(
+ const int * __restrict__ v, const int * __restrict__ u,
+ const uint8_t * __restrict__ sc, const uint8_t * __restrict__ m,
+ const half2 & dm4, const float * __restrict__ d8) {
+
+ float sumf_d = 0.0f;
+ float sumf_m = 0.0f;
+
+ #pragma unroll
+ for (int i = 0; i < QR4_K; ++i) { // QR4_K = 2 → 2-iter unroll
+ const int v0i = (v[0] >> (4*i)) & 0x0F0F0F0F;
+ const int v1i = (v[1] >> (4*i)) & 0x0F0F0F0F;
+
+ // Two dp4a chained per i (8 INT4×INT8 mac per dp4a → 32 macs / inner unroll)
+ const int dot1 = ggml_cuda_dp4a(v1i, u[2*i+1],
+ ggml_cuda_dp4a(v0i, u[2*i+0], 0));
+ const int dot2 = ggml_cuda_dp4a(0x01010101, u[2*i+1],
+ ggml_cuda_dp4a(0x01010101, u[2*i+0], 0));
+
+ sumf_d += d8[i] * (dot1 * sc[i]);
+ sumf_m += d8[i] * (dot2 * m[i]);
+ }
+ const float2 dm4f = __half22float2(dm4);
+ return dm4f.x*sumf_d - dm4f.y*sumf_m;
+}
+```
+
+The `dot2` term computes `Σ xq[j]` cheaply via `dp4a(0x01010101, xq, 0)` —
+4 INT8 sums per dp4a — instead of pre-computing it during input quantization.
+This is a different design choice from dotLLM (which precomputes `s_sx[c]` in
+Stage 1) but the cost is similar (2 extra dp4a per super-block ≈ negligible).
+
+> **Licensing:** llama.cpp source quoted under MIT for *citation* (research
+> reference). Not redistributed. dotLLM's GPLv3 kernel will be a re-derivation
+> from the GGUF format spec, not a copy.
+
+---
+
+## 3. Hypothesis ranking
+
+### H1 — Block-shape mismatch (recommended target)
+
+dotLLM's MMQ Q4_K block has 256 threads × 4 rows. At Qwen3-8B's k=4096:
+- 16 super-blocks × 4 rows = 64 work units distributed across 256 threads
+ via `for (unit = tid; unit < total_units; unit += blockDim.x)`.
+- Each thread does ~1 work unit, then writes to `s_acc[r*256 + tid]` and
+ hits `__syncthreads()`.
+- Stage 3 reduction reads back 4 × 256 = 1024 floats from shmem and reduces
+ per row across 4 warps (the other 4 warps idle).
+
+llama.cpp's mmvq Q4_K block has 128 threads × 1 row:
+- 16 super-blocks × 1 row = 16 work units, but they're processed in the
+ loop `kbx += 8` — so 4 lanes × 2 iterations × 1 row = each warp owns
+ a stride-8 slice of super-blocks.
+- All 128 threads contribute to the same accumulator, reduced by
+ `warp_reduce_sum + 3 warp-shfls fan-in across nwarps-1=3`.
+- No cross-row shmem, no per-thread `s_acc` slot.
+
+**Estimated impact:** +2–4× on MlpUp throughput at k=4096. Reasoning:
+- The 4-rows-per-block tiling was the *fix* for SmolLM's 254/256-idle
+ problem. At k=4096 the natural problem is that **rows are huge**
+ (16 super-blocks each), not small. We'd be paying the cross-row reduction
+ cost without the tile amortization paying for itself in shared work.
+- Smaller block (128 vs 256) doubles the number of resident blocks per SM
+ (`__launch_bounds__(256, 2)` → 2 blocks of 256, `(128, 4)` could give 4–8
+ blocks of 128), which helps hide HBM latency.
+- Removing the per-row shmem accumulator + `__syncthreads()` removes a
+ serializing barrier that buys nothing at this k.
+
+**Implementation effort:** 1–2 days of focused work.
+- New kernel `quantized_gemv_q4_k_mmvq_large` in
+ `native/kernels/quantized_gemv_mmq.cu` modeled on llama.cpp's mmvq pattern.
+- Keep input-quantization-on-the-fly (don't yet require a separate Q8_1
+ pre-pass — that's H4 below). Stage 1 already produces a usable INT8 stream
+ in shmem; the new kernel can read from `s_xq` exactly as today.
+- Routing in `CudaKernels.LaunchQuantizedGemvMmq`: pick mmvq-large when
+ `k ≥ 1024` (≥4 super-blocks/row); pick the existing 4-rows-per-block
+ kernel otherwise.
+- Block dim = (128, 1, 1), grid dim = (n, 1, 1).
+
+**Risk:**
+- Correctness: the same `vec_dot_q4_K` math, just re-tiled. Existing
+ `CudaMmqKernelTests` will gate it. Add a (n=24576, k=4096) test case to
+ cover Qwen3-8B's MlpUp shape.
+- Compatibility: kernel selection by k means SmolLM (k=576) keeps the old
+ kernel, so no SmolLM regression. New kernel only kicks in for Qwen3-class
+ models.
+- The existing `DOTLLM_DISABLE_MMQ_Q4K=1` env var still bypasses everything
+ to the legacy FP-fmuladd path — A/B comparison still possible.
+
+**Recommendation:** **DO THIS FIRST.** Single largest expected lift, well
+within the kernel author's existing skill set, full A/B test infrastructure
+already in place.
+
+### H2 — Tensor cores via mma.m16n8k16 INT8
+
+**Estimated impact:** uncertain, possibly 0–1.5× over a well-tuned dp4a.
+
+**Reasoning against (this is the surprise):**
+- llama.cpp does not use tensor cores in their batch=1 GEMV path on Ampere.
+ They have full mma machinery (`mmq.cuh`) but it's gated to batch ≥ ~8.
+- For a single token, GEMV is bandwidth-bound (we're at 14% of peak HBM,
+ not 14% of peak FLOPS). Tensor cores are FLOP accelerators. They wouldn't
+ help if the bottleneck is memory.
+- mma.sync requires gathering input fragments into a 16×16 tile shape that
+ doesn't naturally exist in a single-token GEMV — you'd be pre-replicating
+ the activation across the M dimension just to feed the tensor unit, which
+ is wasted work.
+
+**Estimated impact: likely zero or modestly negative for batch=1.** Skip
+unless and until decode batching is on the roadmap (then it becomes
+batch ≥ 8 prefill, and the existing cuBLAS HGEMM path is already the right
+answer there).
+
+**Recommendation:** **SKIP for batch=1 decode.** Revisit when adding
+multi-request decode batching (multi-week future work).
+
+### H3 — Memory layout / weight repack
+
+**Estimated impact:** small (~1.1–1.3×) at best, possibly zero.
+
+**Reasoning:** Q4_K rows are already row-major and contiguous in HBM. Each
+super-block is 144 bytes; with `__ldg` reads of 4-byte qpacked words, a
+warp reading 32 contiguous super-blocks reads 32 × 144 = 4.5 KiB per
+super-block-stride iteration — that should be fine for L2/L1 prefetching.
+What might help:
+- Repack so all 8 super-blocks of a row's `qs[]` bytes are contiguous as a
+ single 1024-byte stream rather than interleaved with the 16-byte
+ scale/min headers. This would let `vec_dot` read `qpacked` with denser
+ coalescing.
+- Pre-reorder Q4_K to a tensor-core-friendly tile layout — but H2 says
+ tensor cores aren't worth chasing for batch=1.
+
+**Implementation effort:** medium (1 week — touches `CudaWeights.LoadFromGguf`
+to add a repack-on-load step, plus matching dequant).
+
+**Risk:** moderate — repack has to match an updated kernel one-to-one, easy
+to introduce subtle alignment bugs.
+
+**Recommendation:** **SKIP for now.** Try after H1 lands and re-measure
+bandwidth — if H1 takes us to ~150–200 GB/s, the layout opt would have
+diminishing returns relative to the time investment.
+
+### H4 — Pre-quantize activations to Q8_1 once, share across QKV+GateUp+Down
+
+**Estimated impact:** +5–10% on MlpUp specifically (low), more meaningful
+for the *non-fused* projections (would compound across QkvProj and the
+attention-block path).
+
+**Reasoning:** dotLLM's MMQ kernel re-runs the input-quantization Stage 1
+on every call. Activation `x` for Qwen3-8B MlpUp is 4096 elements — the
+quantization pass itself costs maybe 5–10 µs. dotLLM's QkvProj/GateUp are
+already fused (one call quantizes once, reused across all output rows in
+the same kernel), so within MlpUp this hypothesis only saves the time it
+would take to quantize 4096 floats. That's not the dominant cost.
+
+**Bigger lever:** if we *also* split the kernel into two stages —
+`quantize_x_to_q8_1` (one launch) + `mmvq_q4_k` (one launch reusing the
+quantized x) — then the second-stage kernel has access to a precomputed
+INT8 stream and doesn't need shared memory for it. That frees up the
+9 KiB currently held by `s_xq + s_dx + s_sx` and might bump occupancy
+beyond 2 blocks/SM. But adding launches re-introduces WDDM overhead
+(~22 µs per launch on this box), which would eat the gain on Qwen3-8B's
+36 layers × 5 GEMV groups = 180 launches.
+
+**Recommendation:** **SKIP unless graph capture is on for this model.**
+Once on the graph path, launch overhead is amortized — but Qwen3-8B
+currently runs eager (graph hurts at this scale per `.continue-here.md`).
+Defer to after H1 lands and graph perf is re-evaluated.
+
+### H5 — Multiple GEMVs in flight
+
+**Estimated impact:** ≤ 0.8 ms/token (≤ 1%).
+
+**Reasoning:** 36 layers × 1 GEMV = 36 launches at 22 µs each = 792 µs
+total launch overhead for MlpUp specifically. Out of 40 ms MlpUp time
+that's 2%. Not worth chasing.
+
+**Recommendation:** **SKIP.**
+
+### H6 — KV-cache bandwidth contention
+
+Not relevant to MlpUp specifically. The MlpUp kernel reads weights and
+activations only — KV cache isn't touched. (KV bandwidth contention would
+show up in the Attention category, which is at 1.3% — not a problem.)
+
+### Hypothesis summary table
+
+| # | Hypothesis | Est. impact on MlpUp | Effort | Risk | Verdict |
+|---|---|---|---|---|---|
+| 1 | Block-shape: 128 thr × 1 row vs 256 × 4 | **+100–300%** | 1–2 days | Low | **DO** |
+| 2 | Tensor cores (mma INT8) | ~0% (bandwidth-bound) | 2–3 weeks | High | SKIP |
+| 3 | Weight repack | +10–30% maybe | 1 week | Medium | SKIP for now |
+| 4 | Pre-quantized activations | +5–10% | 2 days | Medium | SKIP for now |
+| 5 | Launch coalescing | +2% | 1 day | Low | SKIP |
+| 6 | KV bandwidth | N/A | — | — | N/A |
+
+---
+
+## 4. Recommended single next change
+
+**Implement a 1-row-per-block, 4-warp (128-thread) MMVQ-style Q4_K kernel
+for k ≥ 1024, modeled on llama.cpp's `mul_mat_vec_q`.**
+
+### Files to touch
+
+1. **`native/kernels/quantized_gemv_mmq.cu`** — add new kernel
+ `quantized_gemv_q4_k_mmvq_large` with these properties:
+ - `blockDim = (128, 1, 1)`; `gridDim = (n, 1, 1)` — one row per block.
+ - Stage 1 (input quant) restructured so the warp owning the row's
+ dot product also handles input quant for *its* super-block stride.
+ Or: keep a small one-time Stage 1 across all 128 threads (k=4096 →
+ 128 chunks of 32 = 4096 elements covered in one warp-stride pass per
+ warp), then __syncthreads(), then dot-product loop.
+ - Dot-product loop: `for (sb = warp_id*4 + (tid & 31); ...)` so each
+ warp owns 4 of the 16 super-blocks; lanes within a warp split each
+ super-block 8-ways (the 8 dp4a calls go 1-per-lane-pair). Final per-warp
+ `warp_reduce_sum`, then 4 warp partial sums combined via shmem +
+ one more `warp_reduce_sum` in warp 0.
+ - Write `y[row]` from threadIdx.x==0, threadIdx.y==0.
+ - **No per-row shmem accumulator** — the partial lives in registers
+ until the final reduction.
+
+2. **`src/DotLLM.Cuda/CudaKernels.cs`** lines ~880-980 — register
+ `_quantizedGemvQ4_KMmvqLargeFunc`, expose `HasMmvqLargeQ4K`, add
+ `LaunchQuantizedGemvMmqLarge` with the new (128, 1, 1) launch shape.
+ Update `LaunchQuantizedGemvMmq` to pick the `_large` variant when
+ `k >= 1024`, fall back to existing 4-rows-per-block kernel otherwise.
+ Both still gated by `HasMmqQ4K` and `DOTLLM_DISABLE_MMQ_Q4K`.
+
+3. **`src/DotLLM.Cuda/Tests/CudaMmqKernelTests.cs`** — add test cases at
+ (n=4096, k=4096), (n=11008, k=4096), (n=24576, k=4096) to cover
+ Qwen3-8B-class shapes. Validate against the legacy
+ `quantized_gemv_q4_k` kernel within the existing K-quant tolerance.
+
+4. **`benchmarks/DotLLM.Benchmarks/...`** — add a per-kernel microbench
+ `Q4KGemvVariants` that hits MMQ-old, MMQ-mmvq-large, legacy at the
+ three shapes above, prints GB/s.
+
+### Validation strategy
+
+1. Build PTX, run unit tests: `dotnet test tests/DotLLM.Tests.Unit/ --filter "Category=GPU"`
+ — must stay 47/47 green.
+2. Microbench: `Q4KGemvVariants` must show mmvq-large ≥ 2× MMQ-old at
+ (24576, 4096) on RTX 3060.
+3. End-to-end: `profile-cuda-decode --compare` on Qwen3-8B Q4_K_M.
+ Target: MlpUp category drops from 40 ms/token to ≤ 20 ms/token.
+ Total decode wall ≤ 60 ms/token (≥ 17 tok/s, vs 10.5 today).
+4. Run with `DOTLLM_DISABLE_MMQ_Q4K=1` to confirm the legacy path is
+ unchanged (control).
+5. SmolLM-135M sanity: confirm 217 graph / 117 eager tok/s unchanged
+ (small-k path keeps using the old MMQ kernel via the k-threshold).
+
+---
+
+## 5. (Optional) Tiny prototype — not implemented in this commit
+
+The recommended change is **not** a one-line tile-factor bump. The original
+task brief's hypothesis 1 ("bump `MMQ_ROWS_PER_BLOCK` from 4 to 8 or 16")
+turns out to be the **wrong direction** — llama.cpp evidence is that for
+batch=1 GEMV at large k, you want **1** row per block, not 8 or 16.
+Bumping MMQ_ROWS_PER_BLOCK higher would make the cross-row shmem reduction
+more expensive without adding work-per-thread (you'd have *more* idle
+threads per block, not fewer).
+
+For completeness, one could try `MMQ_ROWS_PER_BLOCK = 2` as a 5-minute
+micro-experiment. But the architectural problem is the cross-row shmem
+reduction itself, not its width — going from 4 rows to 2 rows just halves
+a small constant. The real win requires removing the cross-row shmem
+accumulator entirely (the H1 restructure).
+
+So this commit is **research-only**: the note above + the branch.
+The implementation is a follow-up issue.
+
+---
+
+## 6. References
+
+### llama.cpp source (MIT — cited, not redistributed)
+
+- [`ggml/src/ggml-cuda/mmvq.cu`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/mmvq.cu)
+ — the GEMV (batch=1) path. `mul_mat_vec_q` template + `calc_nwarps` /
+ `calc_rows_per_block`.
+- [`ggml/src/ggml-cuda/vecdotq.cuh`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/vecdotq.cuh)
+ — `vec_dot_q4_K_q8_1_impl_vmmq` and constants.
+- [`ggml/src/ggml-cuda/mmq.cuh`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/mmq.cuh)
+ — the *batched* MMQ path with mma tensor cores. **Not** used for batch=1.
+- [`ggml/src/ggml-common.h`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-common.h)
+ — `QK_K=256`, `QR4_K=2`, `QI4_K=32`, `QK8_1=32`.
+- [PR #5394](https://github.com/ggerganov/llama.cpp/pull/5394) — the
+ original NVIDIA MMVQ tuning that introduced 4-warp `nwarps` for
+ ncols_dst=1: "use of more warps seems to be beneficial in scenarios where
+ the compute per memory bandwidth is relatively high." 1.6× win on
+ RTX 3090 with Q6_K at the time.
+
+### dotLLM source
+
+- `native/kernels/quantized_gemv_mmq.cu` — current MMQ Q4_K kernel.
+- `src/DotLLM.Cuda/CudaKernels.cs` lines 880-980 — dispatcher and tile
+ constant `MmqRowsPerBlock = 4`.
+- `src/DotLLM.Cuda/CudaTransformerModel.cs` line 382 — fused QKV/GateUp
+ call site.
+- `.continue-here.md` §3c — the gap framing this note addresses.
+
+### General references
+
+- llama.cpp benchmark scoreboard ([discussion #15013](https://github.com/ggml-org/llama.cpp/discussions/15013))
+ — RTX 3060 with Llama-2 7B Q4_0: ~75 tok/s (this is dense FP16 attention,
+ smaller weight than Q4_K_M Qwen3-8B, but anchors achievable order of
+ magnitude on this GPU).
+- 8B-class on RTX 3060 reported in third-party benchmarks at
+ ~22–42 tok/s for Q4_K_M depending on backend (CUDA vs Vulkan) and
+ flash-attention setting. dotLLM at 10.5 is at the bottom of that range.
diff --git a/docs/perf/MMA_BATCHED_MMQ.md b/docs/perf/MMA_BATCHED_MMQ.md
new file mode 100644
index 00000000..e5b35051
--- /dev/null
+++ b/docs/perf/MMA_BATCHED_MMQ.md
@@ -0,0 +1,474 @@
+# Tensor-core (mma) batched MMQ — design note
+
+**Branch:** `cuda-mma-batched-research` (forked from `feature/mamba-3-cuda` @ `69ef507`)
+**Author:** dotnet-perf-expert agent, 2026-04-25
+**Status:** Research only — no kernel code in this commit.
+**Predecessor:** `docs/perf/MLPUP_GEMV_GAP.md` (closed the batch=1 GEMV gap).
+
+## TL;DR
+
+llama.cpp's tensor-core MMQ kernel (mma.m16n8k16 INT8 on sm_75+) activates at
+**batch ≥ 64** on Ampere (`MMQ_DP4A_MAX_BATCH_SIZE = 64`, with `MMVQ_MAX_BATCH_SIZE = 8`
+gating the GEMV path below that). dotLLM today runs at **batch = 1** for every
+decode step — the multi-request scheduler that would produce non-trivial batch
+sizes is `IScheduler` with no implementation (Phase 9 / step 35 on the
+roadmap). Without that scheduler, an mma kernel **has nowhere to apply** in the
+production hot path.
+
+The recommended single next change is **option B — SKIP the mma kernel for now;
+land continuous batching first**. Option A (write the kernel speculatively
+behind a gate, validate on synthetic data) is a defensible Plan B if the kernel
+author wants to keep CUDA momentum, but the work cannot translate into
+end-user perf until the engine produces real batches.
+
+The honest upper bound on mma's contribution today: **+0%** (no batched
+workload exists). With continuous batching at a typical server batch of
+B = 16 the upper bound is roughly **+30–60% throughput** on the GEMV-heavy
+categories (MlpUp / MlpDown / QkvProj / LmHead) versus dp4a, but *only when
+B ≥ 8-ish per decode step*. Speculative decoding alone (B = K = 4–8) lands at
+the bottom of that band.
+
+---
+
+## 1. Threshold analysis
+
+### 1a. Per-warp arithmetic peak — dp4a vs mma INT8
+
+Both use INT8 inputs and FP32 accumulators. Per-cycle, per-warp throughput on
+sm_86:
+
+| Path | Op | Per thread / cycle | Per warp / cycle | Notes |
+|---|---|---|---|---|
+| dp4a | `__dp4a(int, int, int)` | 4 INT8 MACs | **128 INT8 MACs** | Issued on the INT pipe, sm_61+ |
+| mma | `mma.m16n8k16.s8.s8.s32` | 4×8×16 / 32 = **64 MACs / lane** | **2048 INT8 MACs** | Tensor core, sm_75+ |
+
+So per-warp peak is **16× higher** for mma than dp4a. The catch is the M
+dimension: mma.m16n8k16 produces a 16×8 output tile per warp from a 16×16
+fragment of A and an 8×16 fragment of B. **The M=16 lanes are physically
+allocated whether the work is there or not.**
+
+### 1b. Effective utilisation as a function of batch B
+
+In a transformer GEMV-as-GEMM, M is the active batch (number of concurrent
+sequences in this decode step) and N is the output dimension (output rows of
+the projection). For batch=1 decode:
+
+| B | mma M lanes used | Effective utilisation | dp4a path |
+|---|---|---|---|
+| 1 | 1 / 16 | **6.25%** | 100% (no M-fragment waste) |
+| 4 | 4 / 16 | 25% | 100% |
+| 8 | 8 / 16 | 50% | 100% |
+| 16 | 16 / 16 | **100%** | 100% |
+| 32 | 32 / 16 = 2 mma issues × 100% | 100% | 100% |
+
+So **B=16 is the natural breakeven on raw arithmetic alone**: at B=16, mma
+delivers 16× the per-warp throughput of dp4a *and* uses every M lane.
+
+But arithmetic peak is not the whole story.
+
+### 1c. Bandwidth-bound regime — when does compute matter?
+
+The MMQ_GEMV_GAP note already established that at batch=1, MlpUp on Qwen3-8B
+runs at ~50–200 GB/s depending on the kernel — fundamentally **bandwidth-bound
+at this scale on a 360 GB/s RTX 3060**. Switching from dp4a to mma at B=1 buys
+nothing because the bottleneck is memory, not compute.
+
+Crossover heuristic — at what B does compute start to dominate?
+
+- **Weight bytes per layer** (Qwen3-8B Q4_K_M MlpUp): ~57 MiB
+- **Compute per layer per output column at B=1**: 2·k = 8192 INT8 MACs per row,
+ × 24576 rows = ~200M MACs.
+- **Arithmetic intensity** at B=1: ~200M MACs / 57 MiB ≈ 3.5 op/byte. Way
+ below the RTX 3060 ridge point (~30 op/byte for INT8).
+- At **B = 16**, weight reads are amortised across 16 outputs: arithmetic
+ intensity scales linearly to ~56 op/byte → **compute-bound**, mma should
+ shine.
+- **Crossover ≈ B = 8–10** — below that, weight bandwidth dominates and the
+ kernel choice barely matters; above, the per-warp throughput multiplier
+ starts paying off.
+
+This matches llama.cpp's empirically chosen `MMVQ_MAX_BATCH_SIZE = 8`: below
+B=8 they use the MMVQ (GEMV) path; above, they switch to the batched MMQ. And
+within MMQ, `MMQ_DP4A_MAX_BATCH_SIZE = 64` keeps dp4a until B is very clearly
+in the compute-bound regime.
+
+### 1d. The actual llama.cpp dispatch (cited verbatim)
+
+From `ggml/src/ggml-cuda/mmvq.cuh`:
+
+```c
+#define MMVQ_MAX_BATCH_SIZE 8 // Max. batch size for which to use MMVQ kernels.
+```
+
+From `ggml/src/ggml-cuda/mmq.cuh`:
+
+```c
+#define MMQ_DP4A_MAX_BATCH_SIZE 64 // Max. batch size to use for dp4a MMQ kernels when FP16 tensor cores are available.
+```
+
+From `ggml/src/ggml-cuda/ggml-cuda.cu`, the per-op dispatch:
+
+```c
+bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear
+ && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32
+ && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE;
+```
+
+And from `ggml/src/ggml-cuda/mmq.cu` (`ggml_cuda_should_use_mmq`):
+
+```c
+if (GGML_CUDA_CC_IS_NVIDIA(cc)) {
+ return !fp16_mma_hardware_available(cc) || ne11 < MMQ_DP4A_MAX_BATCH_SIZE;
+}
+```
+
+(That return value is "use the dp4a MMQ path"; when `false`, MMA is selected.)
+
+### 1e. Putting the thresholds together — Ampere routing
+
+| Active batch B | llama.cpp path | dotLLM today |
+|---|---|---|
+| 1 | MMVQ (GEMV, dp4a) | MMVQ-large + pre-Q8_1 (matches structurally) |
+| 2..8 | MMVQ | Same kernel, no special path |
+| 9..63 | MMQ dp4a (batched, no tensor cores) | **No path** |
+| 64+ | MMQ mma (tensor cores) | **No path** |
+
+The interesting takeaway: **dp4a-based batched MMQ at B = 9..63 is a
+distinct optimisation tier from mma**. If we ever build out a batched path,
+the dp4a-batched-MMQ may be more impactful than mma — most realistic server
+batches sit in the 8..32 range, not ≥64.
+
+### 1f. The mma instruction itself
+
+Used by llama.cpp on sm_75+ (Turing) and reused on sm_80/86/89/90:
+
+```
+mma.sync.aligned.m16n8k16.row.col.s32.s8.s8.s32
+```
+
+This consumes:
+- A: 16×16 INT8 row-major (held across the warp's 32 threads, 4 INT8s per
+ thread per A fragment)
+- B: 8×16 INT8 column-major (2 INT8s per thread)
+- C: 16×8 INT32 (4 ints per thread)
+
+Each warp produces a 16×8 INT32 output tile per `mma.sync` call. Per dp4a:
+warp produces 32 INT32 outputs (1 per thread). So **per-warp throughput
+ratio = 128 / 32 = 4× more outputs per warp from mma**, on top of the
+arithmetic-density advantage from working on a 16×K tile rather than a
+single dot product.
+
+---
+
+## 2. dotLLM applicability
+
+### 2a. Today: B = 1, single-stream decode
+
+`src/DotLLM.Engine/TextGenerator.cs` is the one and only generation path. It
+calls `_model.Forward(...)` with a single token at every decode step. There
+is no batched-forward API on `IModel`. There is no scheduler. `IScheduler`
+exists in `src/DotLLM.Engine/IScheduler.cs` as an interface contract with a
+`SchedulerMetrics` struct, but **no implementation exists in the repo**
+(grep for `: IScheduler` returns zero matches; `ContinuousBatching` matches
+only docs and roadmap, no code).
+
+**MMA value to dotLLM today: zero.** No batched workload is produced
+anywhere in the engine.
+
+### 2b. Speculative decoding (already on the branch)
+
+`SpeculativeDecoder.DraftAndVerify` already runs target verification with a
+batch of K candidate tokens. K defaults to 5 (`_speculativeCandidates = 5`
+in `TextGenerator`); the implementation accepts up to ~8 in practice.
+
+This **is** a real batched-forward call, but it has three constraints:
+
+1. **B = K = 4–8** sits in MMVQ territory (B ≤ 8). llama.cpp would not
+ activate even the dp4a-batched MMQ at this size, let alone mma.
+2. Greedy-only today (`IsEffectivelyGreedy(options)` gate in
+ `TextGenerator`). Wave 8 would lift this, but the batch size doesn't
+ change.
+3. Speculative + graph capture is not yet on (Open §4 in `.continue-here.md`).
+
+For mma specifically, speculative decoding is a **non-trigger**. It does
+not get into the M ≥ 16 regime where mma starts winning meaningfully.
+
+### 2c. Continuous batching (Phase 9, step 35 — not implemented)
+
+This is the realistic mma trigger. From `docs/SCHEDULING.md`:
+
+> Each scheduler iteration:
+> 1. Check completions: Sequences hitting EOS/max tokens/stop conditions → evict, free KV blocks.
+> 2. Admit new requests: Fill freed capacity from the priority queue.
+> 3. Prefill: For newly admitted sequences, process full prompt tokens (batch prefill).
+> 4. Decode: For all active sequences, generate one token each (batched decode).
+
+In step 4, the scheduler aggregates **B active sequences** into a single
+forward pass. Production server batches typically run B = 8–32 depending on
+KV-cache headroom and request mix. At B = 16, the MlpUp GEMV becomes a
+24576 × 4096 × 16 GEMM — **mma's natural sweet spot**.
+
+But this requires:
+
+- An `IScheduler` implementation (continuous batching loop, sequence state
+ machine, KV-block admission, completion eviction, sampler-per-sequence
+ fan-out).
+- A batched-forward signature on `IModel` (`Forward(span,
+ span, span)` or equivalent fan-out).
+- Per-sequence sampling pipelines run after a single forward (already exist
+ per-request, just need to be invoked B times per step from disjoint logit
+ rows).
+- Paged KV-cache (step 36) so concurrent sequences don't fragment memory.
+
+That is a **multi-week engine project**, and mma is one of many kernels
+needed under it (also: the KV-cache concat path, the attention kernel needs
+batched seq_kv, etc.).
+
+### 2d. Dynamic batching across multi-request server
+
+Same as 2c, just a slightly different admission policy (window-based vs
+iteration-based). Same kernel requirements. Production batch sizes B = 4–32.
+
+### 2e. Per-case feasibility / cost / kernel speedup summary
+
+| Trigger | Realistic B | Feasible today? | Kernel-only speedup vs current dp4a | E2E throughput uplift |
+|---|---|---|---|---|
+| Single-stream decode (today) | 1 | n/a | 0% (mma loses at B=1) | 0% |
+| Speculative + graph (post Open §4 fix) | 4–8 | ~1 week eng | Marginal: mma maybe break-even at B=8, often slower due to M=8/16 waste | <5% |
+| Continuous batching (step 35) | 8–32 | Multi-week eng | dp4a-batched-MMQ: ~2× over per-request dp4a (amortised weight reads). mma on top: another 1.3–1.8× | **Realistic +30–60%** in server scenarios; 0% in single-user CLI |
+| Dynamic batching (server) | 8–32 | Same as continuous batching | Same | Same |
+
+**Critical observation**: even when continuous batching lands, the **first**
+big win is the dp4a-batched MMQ kernel (re-using llama.cpp's MMQ-dp4a
+structure, which keeps weight reads coalesced across multiple sequences). mma
+is the *second* lever, only kicking in meaningfully at the high end of
+production batch sizes.
+
+---
+
+## 3. Recommended path
+
+### 3a. Step ordering
+
+1. **Continuous batching scheduler** (`IScheduler` impl, batched-forward on
+ `IModel`, paged KV-cache integration). Effort: **3–6 weeks**. Unlocks any
+ batched kernel work. Risk: medium-high — touches engine architecture,
+ sampler pipeline fan-out, KV memory model, prefix cache integration.
+2. **Batched-MMQ-dp4a kernels** (single weight read amortised across B
+ active sequences; `mul_mat_q` analog from llama.cpp, dp4a only). Effort:
+ **1–2 weeks** per quant type. Risk: low — same dp4a primitive we already
+ use, just multi-row M instead of M=1. Expected speedup at B=16:
+ **~2×** versus running B independent GEMVs.
+3. **mma-MMQ kernels** (tensor-core variant of #2 for B ≥ 32-ish). Effort:
+ **2–3 weeks** per quant type, plus the load-tile path for K-quants which
+ is fiddly. Risk: medium — fragment layout debugging, ldmatrix vs manual
+ loads, INT8 saturation behaviour. Expected speedup at B=64:
+ **~1.5–1.8×** over batched dp4a; at B=16 break-even or slightly worse.
+4. **Dispatcher gate** that picks the right kernel per launch based on B
+ (mirrors llama.cpp's MMVQ_MAX_BATCH_SIZE / MMQ_DP4A_MAX_BATCH_SIZE
+ logic). Effort: **1 day** — trivial once the kernels exist.
+
+### 3b. End-to-end perf model
+
+Take Qwen3-8B Q4_K_M as the anchor. Single-user decode is at 30.9 tok/s and
+**already inside llama.cpp's range**.
+
+- B = 1 (current): 30.9 tok/s per sequence, **30.9 tok/s total**.
+- B = 4 server, dp4a-batched-MMQ: ~22 tok/s per seq (weight reads share
+ modestly), **~88 tok/s aggregate**.
+- B = 16 server, dp4a-batched-MMQ: ~14 tok/s per seq, **~225 tok/s aggregate**.
+- B = 16 server, mma-MMQ: ~18 tok/s per seq (faster compute), **~290 tok/s
+ aggregate**. **Δ vs dp4a-batched ~30%.**
+- B = 32 server, mma-MMQ: ~12 tok/s per seq (KV / attention bottleneck
+ takes over), **~380 tok/s aggregate**. Δ vs dp4a-batched ~40%.
+
+The **mma-specific lift** is +30–40% on top of a properly batched dp4a
+kernel. The total step from "no batching" to "batching + mma" is much
+larger (~10× aggregate throughput on 16-way concurrent server load), but
+**most of that comes from the scheduler, not the mma kernel.**
+
+### 3c. Honest sizing of the mma upper bound
+
+To be blunt: **mma is a refinement on top of the engine work, not a
+standalone win**. The work is meaningful when:
+
+1. dotLLM is being deployed as a server, not embedded as a CLI.
+2. The server consistently runs B ≥ 16 active sequences.
+3. The user's bottleneck is aggregate throughput, not single-request latency.
+
+For a single-user developer workflow (which is dotLLM's most common
+demonstrated use case today, per the bench scripts and `samples/`),
+**mma does not apply**.
+
+---
+
+## 4. Recommended single next change
+
+**B — SKIP the mma kernel work entirely until continuous batching lands.**
+
+### Per-option rationale
+
+#### A) Write the mma kernel as a microbenchmark, gate it off, validate on synthetic batched inputs
+
+- **Effort**: 1 week (write kernel, validate vs dp4a reference, microbench).
+- **End-user perf today**: 0% — no caller exists.
+- **Future leverage**: kernel exists when the scheduler arrives; a contributor
+ to the scheduler doesn't have to also write the kernel.
+- **Risk of bitrot**: high — without integration, the kernel may not even
+ match the eventual batched-forward calling convention. mma kernel APIs
+ depend on how the batched activation buffer is laid out (interleaved vs
+ stacked), and that's a scheduler-level decision.
+- **Recommendation: SKIP.** The leverage isn't worth the risk of building
+ the wrong kernel. Speculative kernel work on a non-existent engine layer
+ is a classic premature-optimisation trap.
+
+#### B) Wait — implement nothing until continuous batching lands
+
+- **Effort**: 0 (today).
+- **Future cost**: pay the kernel cost when it's actually needed.
+- **Recommendation: DO.** This is the right answer. The engine work is the
+ bottleneck and should be scoped, designed, and started without a kernel
+ dependency.
+
+#### C) Implement speculative-decoding extension first (small batch B = K = 4–8)
+
+- **Speculative decode is already greedy-on this branch**. The remaining
+ work is the Wave 8 / issue #121 pipeline-aware acceptance — that's an
+ engine change, not a kernel change.
+- B = K = 5 (default) does **not** activate mma in any reasonable
+ threshold model (M = 5 / 16 = 31% lane utilisation, well under the
+ dp4a crossover).
+- Even if speculative-decode ends up using a batched verification kernel,
+ that kernel should be a small-batch dp4a-MMQ variant, not mma.
+- **Recommendation: SKIP for the mma question.** Speculative is worth
+ finishing for its own (greedy-lift) reasons, but it does not unlock mma.
+
+#### D) Other — implement dp4a-batched MMQ first (without mma), gate by B ≥ 4
+
+- The dp4a-batched-MMQ has a meaningful payoff at B = 4..32, well below
+ mma's threshold.
+- Same kernel structure as the existing single-row dp4a; the change is
+ expanding the per-block tile to (M_block, N_block) outputs and amortising
+ weight reads across M.
+- Effort: ~1 week per quant type (Q4_K, Q5_K, Q6_K).
+- **Still requires** the scheduler to produce B ≥ 4 — same blocker as A
+ and the mma path.
+- **Recommendation: SKIP for now, but rank it ABOVE mma when batching
+ arrives.** It's the correct kernel to write **first** when there's a
+ caller for it.
+
+### Final ranking
+
+1. **DO**: Land continuous batching in the engine (option B = wait + redirect
+ effort to the actual blocker).
+2. **DO**: When batching is producing real B ≥ 4 batches, add the
+ dp4a-batched MMQ kernel (option D).
+3. **DO**: When server deployments demonstrably run B ≥ 16 consistently, add
+ the mma MMQ kernel (the topic of this note).
+4. **SKIP**: Writing the mma kernel speculatively today (option A).
+
+---
+
+## 5. Optional tiny prototype
+
+**Not implemented in this commit.** Per option B above, a prototype today
+would be premature work without a caller. The kernel author's time is more
+valuably spent on:
+
+- The CUDA MLA Phase 1 work currently in flight on the parallel agent (real
+ end-user perf for DeepSeek-V2/V3 on CUDA), or
+- Engine-layer scheduling design (the actual blocker for any batched kernel
+ work).
+
+If a future session decides to override this recommendation and write the
+prototype anyway (option A), the entry point should be:
+
+- File: `native/kernels/quantized_gemm_mma_q4_k.cu` (new — sibling to
+ `quantized_gemv_mmq.cu`, NOT a replacement).
+- Signature: `void quantized_gemm_q4_k_mma_b8(const uint8_t* weight,
+ const int8_t* x_q8_1, const half* x_d, half* y, int n, int k, int batch)`.
+- Tile: M = 16 (one mma fragment), N_per_block = 16 (so 16 output rows per
+ block), K iterated in chunks of 16 INT8s (one mma.m16n8k16 step).
+- One warp per mma; 4 warps per block (128 threads) for occupancy parity
+ with the existing MMVQ-large kernel.
+- Validation: synthetic random batched inputs, compare against
+ `quantized_gemv_q4_k_mmq_preq` called B times in a loop. INT8 noise
+ budget: same ≤1% peak-relative drift as the existing MMQ tests.
+- Microbench harness: extend `Q4KGemvVariants` (proposed in MLPUP_GEMV_GAP)
+ to a `Q4KGemmBatched` variant that sweeps B = 1, 4, 8, 16, 32, 64.
+
+The prototype is documented as a follow-up; **not** done here.
+
+---
+
+## 6. Constraints and engineering risk for option B
+
+If the recommendation (option B = wait) is accepted, the engine work
+sequencing is:
+
+1. **`IScheduler` skeleton** — port the loop in `docs/SCHEDULING.md` to a
+ real implementation behind the existing interface. Single-sequence
+ smoke test first. Effort: 1 week.
+2. **Batched `Forward`** on `IModel` — new method
+ `Forward(ReadOnlySpan tokens, ReadOnlySpan positions,
+ ReadOnlySpan sequenceIds, IReadOnlyList kvCaches)`,
+ where `tokens.Length` is the total batch size and `sequenceIds[i]`
+ selects the per-token KV cache. CUDA backend will need batched RoPE,
+ batched KV-write, and a per-sequence attention dispatch. Effort:
+ 2–3 weeks.
+3. **Paged KV-cache integration** — use the existing `PagedKvCache`
+ (already in repo at `src/DotLLM.Engine/KvCache/PagedKvCache.cs`)
+ as the multi-sequence backing store. Block-table indirection lets
+ us pack multiple sequences contiguously. Effort: 1 week.
+4. **Sampler fan-out** — run `SamplerPipeline.Sample` once per active
+ sequence on disjoint logit rows. Existing per-request pipeline
+ already supports this; just needs to be driven from the scheduler.
+ Effort: <1 week.
+5. **Then** kernel work (option D, then this note's mma kernel).
+
+Total: **~6–8 weeks** of engine work before mma becomes addressable.
+That work is high-value on its own — every server-class deployment
+needs continuous batching regardless of mma — so it is **not** wasted
+even if mma never lands.
+
+---
+
+## 7. References
+
+### llama.cpp source (MIT — cited, not redistributed)
+
+- [`ggml/src/ggml-cuda/mmvq.cuh`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/mmvq.cuh)
+ — `MMVQ_MAX_BATCH_SIZE = 8` define.
+- [`ggml/src/ggml-cuda/mmq.cuh`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/mmq.cuh)
+ — `MMQ_DP4A_MAX_BATCH_SIZE = 64` define; `MMQ_NWARPS = 8`,
+ `MMQ_TILE_NE_K = 32`, MMQ tile constants for the mma kernel.
+- [`ggml/src/ggml-cuda/mmq.cu`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/mmq.cu)
+ — `ggml_cuda_should_use_mmq` dispatch logic.
+- [`ggml/src/ggml-cuda/ggml-cuda.cu`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/ggml-cuda.cu)
+ — `ggml_cuda_mul_mat` MMVQ-vs-MMQ entry-point dispatch.
+- [`ggml/src/ggml-cuda/mma.cuh`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cuda/mma.cuh)
+ — `mma.sync.aligned.m16n8k16.row.col.s32.s8.s8.s32` PTX, Turing/Ampere
+ fragment layout helpers.
+
+### NVIDIA documentation
+
+- [PTX ISA — Warp-level matrix instructions (mma)](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions)
+- [PTX ISA — `mma.sync` for INT8](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-fragments-for-mma-m16n8k16-mma-m16n8k32)
+
+### dotLLM source
+
+- `docs/perf/MLPUP_GEMV_GAP.md` — predecessor note (closed the batch=1 gap).
+- `src/DotLLM.Engine/IScheduler.cs` — interface contract; **no impl exists**.
+- `src/DotLLM.Engine/TextGenerator.cs` — the only generation path today
+ (single-request, B=1).
+- `src/DotLLM.Engine/SpeculativeDecoder.cs` — produces small batches
+ (K=4–8) via target-model verification; greedy-only today.
+- `src/DotLLM.Engine/KvCache/PagedKvCache.cs` — paged KV available, ready
+ for multi-sequence integration.
+- `native/kernels/quantized_gemv_mmq.cu` — current MMQ kernels; mma kernel
+ would sit alongside as a sibling, not replace.
+- `src/DotLLM.Cuda/CudaKernels.cs` `LaunchQuantizedGemvMmq` /
+ `HasMmqQ4K` — the dispatcher that would gate by B once both kernels
+ exist.
+- `docs/SCHEDULING.md`, `docs/ROADMAP.md` step 35 — engine layer that
+ must land before mma is addressable.
diff --git a/docs/superpowers/plans/2026-04-27-q2k-phase1-plan.md b/docs/superpowers/plans/2026-04-27-q2k-phase1-plan.md
new file mode 100644
index 00000000..db77b472
--- /dev/null
+++ b/docs/superpowers/plans/2026-04-27-q2k-phase1-plan.md
@@ -0,0 +1,1300 @@
+# Q2_K Quantization Phase 1 Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add full L3 support (CPU dequant + GPU dequant + per-call GEMV + MMQ + MMQ_preq + MMVQ-large + MMVQ-large_preq + grouped-MoE-GEMV) for the Q2_K GGUF quantization type so V2-Lite-Q2_K models load and run end-to-end on dotLLM CUDA.
+
+**Architecture:** Q2_K is a port-style addition — structurally identical to existing Q3_K/Q4_K/Q5_K/Q6_K K-quants (256-element superblocks, linear quantization, dp4a-friendly inner loops). Each new kernel is built by copying the corresponding Q4_K kernel and substituting the per-superblock decode body (16 sub-blocks of 16 elements with 4-bit scale + 4-bit min coef per sub-block, vs Q4_K's 8 sub-blocks of 32 elements with 6-bit packed scales). No new abstractions; all extension via switch cases on `QuantizationType.Q2_K`.
+
+**Tech Stack:** C# / .NET 10 (CPU side), CUDA C++ → PTX (GPU kernels), xUnit + SkippableFact (tests), `cuLaunchKernel` driver API, GGUF binary format (Q2_K = type ID 10).
+
+**Spec reference:** `docs/superpowers/specs/2026-04-27-q2k-iquant-coverage-design.md` — Section 1, 2, 3, 4, 5, 6 cover the architecture; this plan implements Phase 1 only.
+
+**Effort estimate:** ~1 week of focused work, ~27 new tests, 1 PR.
+
+---
+
+## File Structure
+
+| Action | Path | Responsibility |
+|---|---|---|
+| Modify | `src/DotLLM.Core/Configuration/QuantizationType.cs` | Add `Q2_K = 10` enum entry |
+| Modify | `src/DotLLM.Cpu/Kernels/Dequantize.cs` | Add `Q2_K_BlockBytes = 84` constant; switch cases in `RowByteSize` + `ToFloat32` |
+| Modify | `src/DotLLM.Cpu/Kernels/DequantizeKQuants.cs` | Add `DequantizeQ2_K` scalar impl |
+| Modify | `tests/DotLLM.Tests.Unit/Cpu/Kernels/DequantizeKQuantTests.cs` | Add Q2_K hand-calc + RowByteSize + non-aligned-throws tests |
+| Modify | `native/kernels/dequant.cu` | Add `dequant_q2_k_f16` kernel + `Q2_K_BLOCK_BYTES` macro |
+| Modify | `native/kernels/quantized_gemv.cu` | Add `quantized_gemv_q2_k` kernel |
+| Modify | `native/kernels/quantized_gemv_mmq.cu` | Add `quantized_gemv_q2_k_mmq` + `_preq` + `_mmvq_large` + `_mmvq_large_preq` kernels |
+| Modify | `native/kernels/moe_grouped_gemv.cu` | Add `moe_grouped_gemv_q2_k_f16` kernel |
+| Modify | `src/DotLLM.Cuda/CudaKernels.cs` | Module function fields, `Has*Q2K` properties, `MinKAlignmentFor` case, dispatch switches in `LaunchDequantToF16`, `LaunchQuantizedGemv`, `LaunchQuantizedGemvMmq`, `LaunchMoeGroupedGemv` |
+| Modify | `tests/DotLLM.Tests.Unit/Cuda/CudaQuantizedGemvAlignmentTests.cs` | Add Q2_K theory entries (M=2048, K=2048; M=1408, K=2048) |
+| Modify | `tests/DotLLM.Tests.Unit/Cuda/CudaMmqKernelTests.cs` | Add `MmqQ2K_MatchesLegacyWithinTolerance` + `MmvqLargeQ2K_MatchesLegacy` + pre-Q8_1 theories |
+| Modify | `tests/DotLLM.Tests.Unit/Cuda/CudaMoeGroupedGemvTests.cs` | Add `GroupedQ2K_MatchesPerCallWithinFp16Tolerance` theory |
+| Modify | `tests/DotLLM.Tests.Unit/Models/Architectures/DeepSeekV2GgufLoadTests.cs` | Add `RealGguf_Q2K_FullModel_27LayerSmoke` |
+| Modify | `.continue-here.md` | Round 16 entry documenting Q2_K landing |
+
+No new files created in this phase; all additions extend existing K-quant infrastructure. Phase 2 (IQ-family) is the phase that introduces new files.
+
+---
+
+## Q2_K Block Layout Reference
+
+Pinning the encoding so all kernels agree (verified against `ggml-quants.h` `block_q2_K`):
+
+```
+Offset Size Field Description
+─────────────────────────────────────────────────────────────────────
+0 16 scales[16] Per-sub-block scale (low nibble) + min coef (high nibble), 4 bits each
+16 64 qs[64] 2-bit quant values (4 elements per byte, packed low-to-high)
+80 2 d (half) Super-block scale
+82 2 dmin (half) Super-block min
+─────────────────────────────────────────────────────────────────────
+Total: 84 bytes per 256 elements (16 sub-blocks of 16 elements)
+```
+
+**Decode**: for element index `t` (0..255):
+- `sub = t / 16` (0..15) → which sub-block
+- `j = t % 16` (0..15) → position within sub-block
+- `byte_idx = (sub * 16 + j) / 4 = t / 4`
+- `bit_off = ((sub * 16 + j) % 4) * 2 = (t % 4) * 2`
+- `q2 = (qs[byte_idx] >> bit_off) & 0x3` (0..3, unsigned)
+- `scale = scales[sub] & 0xF` (0..15, unsigned)
+- `dmin_coef = (scales[sub] >> 4) & 0xF` (0..15, unsigned)
+- `value = d * scale * q2 - dmin * dmin_coef`
+
+The ½-byte scale + ½-byte min-coef packing per sub-block is what differs from Q3_K (12-byte 6-bit packed scales for 16 sub-blocks). The 2-bits-per-element packing is the same density as Q3_K's qs but without the hmask high-bit.
+
+---
+
+### Task 1: Q2_K enum + block-bytes constant + CPU dequant
+
+**Files:**
+- Modify: `src/DotLLM.Core/Configuration/QuantizationType.cs`
+- Modify: `src/DotLLM.Cpu/Kernels/Dequantize.cs`
+- Modify: `src/DotLLM.Cpu/Kernels/DequantizeKQuants.cs`
+- Test: `tests/DotLLM.Tests.Unit/Cpu/Kernels/DequantizeKQuantTests.cs`
+
+- [ ] **Step 1: Write the failing tests for Q2_K dequant**
+
+Add to `tests/DotLLM.Tests.Unit/Cpu/Kernels/DequantizeKQuantTests.cs` (place after the Q3_K block at line ~274; mirrors the Q3_K hand-calc pattern):
+
+```csharp
+ // ──────────────────── Q2_K dequant ────────────────────
+
+ private const int Q2_K_BlockBytes = 84;
+
+ [Fact]
+ public void Q2_K_SingleBlock_HandCalculated()
+ {
+ // Block layout: scales[16] + qs[64] + d[2] + dmin[2] = 84 bytes.
+ nuint totalBytes = Q2_K_BlockBytes;
+ nint ptr = (nint)NativeMemory.AlignedAlloc(totalBytes, 64);
+ try
+ {
+ NativeMemory.Clear((void*)ptr, totalBytes);
+ byte* block = (byte*)ptr;
+
+ // d = 1.0, dmin = 0.5
+ Unsafe.WriteUnaligned(block + 80, (Half)1.0f);
+ Unsafe.WriteUnaligned(block + 82, (Half)0.5f);
+
+ // scales[0]: low nibble = scale (we want scale = 3), high nibble = dmin coef (we want 2).
+ // Packed as: (dmin_coef << 4) | scale = (2 << 4) | 3 = 0x23
+ block[0] = 0x23;
+
+ // qs[0] (offset 16): set element 0's 2 low bits to 0b10 (= 2).
+ // qs encoding: 4 elements per byte, low-to-high.
+ // byte 0, bits 0-1 → element 0
+ // byte 0, bits 2-3 → element 1
+ // byte 0, bits 4-5 → element 2
+ // byte 0, bits 6-7 → element 3
+ block[16 + 0] = 0x02; // element 0 = 2, elements 1-3 = 0
+
+ // Element 0: q2 = 2, scale = 3, dmin_coef = 2
+ // value = d * scale * q2 - dmin * dmin_coef
+ // = 1.0 * 3 * 2 - 0.5 * 2
+ // = 6 - 1 = 5
+ float[] dest = new float[KQuantGroupSize];
+ Dequantize.ToFloat32(ptr, KQuantGroupSize, QuantizationType.Q2_K, dest);
+
+ Assert.Equal(5.0f, dest[0], 0.01f);
+
+ // Element 1: q2 = 0, scale = 3, dmin_coef = 2
+ // value = 1.0 * 3 * 0 - 0.5 * 2 = -1
+ Assert.Equal(-1.0f, dest[1], 0.01f);
+
+ // Sub-block 1 (elements 16..31): scale = 0, dmin_coef = 0 (all-zero scales[1..15])
+ // value = 1.0 * 0 * 0 - 0.5 * 0 = 0
+ Assert.Equal(0.0f, dest[16], 0.01f);
+ }
+ finally
+ {
+ NativeMemory.AlignedFree((void*)ptr);
+ }
+ }
+
+ [Fact]
+ public void Q2_K_RowByteSize_Matches()
+ {
+ // 256 elements = 1 super-block = 84 bytes.
+ Assert.Equal(84L, Dequantize.RowByteSize(256, QuantizationType.Q2_K));
+ // 1024 elements = 4 super-blocks = 336 bytes.
+ Assert.Equal(336L, Dequantize.RowByteSize(1024, QuantizationType.Q2_K));
+ }
+
+ [Fact]
+ public void Q2_K_NonAlignedCount_Throws()
+ {
+ float[] dest = new float[100];
+ Assert.Throws(() =>
+ Dequantize.ToFloat32(nint.Zero, 100, QuantizationType.Q2_K, dest));
+ }
+```
+
+- [ ] **Step 2: Run tests to verify they fail (compile error)**
+
+Run: `dotnet build src/DotLLM.Cpu/DotLLM.Cpu.csproj -c Release -nologo 2>&1 | tail -10`
+
+Expected: build error referencing `QuantizationType.Q2_K` (enum value doesn't exist yet).
+
+- [ ] **Step 3: Add Q2_K to the QuantizationType enum**
+
+Edit `src/DotLLM.Core/Configuration/QuantizationType.cs`. Find the line with `Q8_0 = 8,` and add `Q2_K = 10` after it (matching GGUF type ID and slotting between Q8_0 and Q3_K which is type 11):
+
+```csharp
+ /// 8-bit quantization, group size 32.
+ Q8_0 = 8,
+
+ /// 2-bit K-quant, super-block of 256.
+ Q2_K = 10,
+
+ /// 3-bit K-quant, super-block of 256.
+ Q3_K = 11,
+```
+
+- [ ] **Step 4: Add Q2_K_BlockBytes constant + RowByteSize + ToFloat32 dispatch**
+
+Edit `src/DotLLM.Cpu/Kernels/Dequantize.cs`. Find the K-quant block-bytes constants block; add `Q2_K_BlockBytes = 84` (place between Q3_K_BlockBytes=110 and Q4_K_BlockBytes=144 or wherever the alphabetical/numeric ordering puts it):
+
+```csharp
+ /// Q2_K super-block byte size: scales[16] + qs[64] + d[2] + dmin[2] = 84.
+ public const int Q2_K_BlockBytes = 84;
+```
+
+In the same file, find `RowByteSize` and add the Q2_K case before the Q3_K case:
+
+```csharp
+ QuantizationType.Q2_K => elementCount / KQuantGroupSize * Q2_K_BlockBytes,
+```
+
+In `ToFloat32`, find the K-quant case block and add Q2_K:
+
+```csharp
+ case QuantizationType.Q2_K:
+ if (elementCount % KQuantGroupSize != 0)
+ throw new ArgumentException(
+ $"Q2_K requires elementCount to be a multiple of {KQuantGroupSize}.",
+ nameof(elementCount));
+ DequantizeKQuants.DequantizeQ2_K(src, dest.Slice(0, (int)elementCount), elementCount);
+ return;
+```
+
+- [ ] **Step 5: Add DequantizeQ2_K scalar implementation**
+
+Edit `src/DotLLM.Cpu/Kernels/DequantizeKQuants.cs`. Add (after `DequantizeQ3_K` impl):
+
+```csharp
+ ///
+ /// Dequantizes Q2_K-quantized data to float32. Block layout:
+ /// scales[16] (4-bit scale + 4-bit dmin coef per sub-block, packed) +
+ /// qs[64] (2-bit elements, 4 per byte) + d (half) + dmin (half) = 84 bytes per 256 elements.
+ /// Per-element decode: value = d × scale × q2 − dmin × dmin_coef.
+ ///
+ public static unsafe void DequantizeQ2_K(nint src, Span dest, long elementCount)
+ {
+ if (elementCount % KQuantGroupSize != 0)
+ throw new ArgumentException(
+ $"Q2_K requires elementCount to be a multiple of {KQuantGroupSize}.", nameof(elementCount));
+
+ long superBlocks = elementCount / KQuantGroupSize;
+ byte* basePtr = (byte*)src;
+
+ for (long sb = 0; sb < superBlocks; sb++)
+ {
+ byte* block = basePtr + sb * Dequantize.Q2_K_BlockBytes;
+ byte* scales = block; // 16 bytes
+ byte* qs = block + 16; // 64 bytes
+ float d = (float)Unsafe.ReadUnaligned(block + 80);
+ float dmin = (float)Unsafe.ReadUnaligned(block + 82);
+
+ int outOffset = (int)(sb * KQuantGroupSize);
+ for (int t = 0; t < KQuantGroupSize; t++)
+ {
+ int sub = t >> 4; // t / 16
+ int byteIdx = t >> 2; // t / 4
+ int bitOff = (t & 0x3) << 1; // (t % 4) * 2
+ int q2 = (qs[byteIdx] >> bitOff) & 0x3;
+ int scale = scales[sub] & 0xF;
+ int dmCoef = (scales[sub] >> 4) & 0xF;
+ dest[outOffset + t] = d * scale * q2 - dmin * dmCoef;
+ }
+ }
+ }
+```
+
+- [ ] **Step 6: Run tests to verify they pass**
+
+Run: `dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~Q2_K" -nologo`
+
+Expected: 3 tests passed (`Q2_K_SingleBlock_HandCalculated`, `Q2_K_RowByteSize_Matches`, `Q2_K_NonAlignedCount_Throws`).
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/DotLLM.Core/Configuration/QuantizationType.cs \
+ src/DotLLM.Cpu/Kernels/Dequantize.cs \
+ src/DotLLM.Cpu/Kernels/DequantizeKQuants.cs \
+ tests/DotLLM.Tests.Unit/Cpu/Kernels/DequantizeKQuantTests.cs
+git commit -m "$(cat <<'EOF'
+CPU: Q2_K dequantization
+
+Adds Q2_K (GGUF type ID 10, K-quant family, 84-byte 256-element
+super-blocks) to the CPU dequantization path. Block layout:
+scales[16] + qs[64] + d (half) + dmin (half).
+
+Per-element decode: value = d × (scales[sub] & 0xF) × q_2bit
+ − dmin × ((scales[sub] >> 4) & 0xF)
+
+Matches the encoding spec in ggml-quants.h block_q2_K. Single-block
+hand-calc test pins the math; RowByteSize + non-aligned-throws tests
+mirror the existing K-quant pattern.
+
+Phase 1 of Spec 1 (Q2_K + IQ-family quantization coverage).
+EOF
+)"
+```
+
+---
+
+### Task 2: Q2_K GPU dequant kernel
+
+**Files:**
+- Modify: `native/kernels/dequant.cu`
+- Modify: `src/DotLLM.Cuda/CudaKernels.cs`
+- Modify: `tests/DotLLM.Tests.Unit/Cuda/CudaKernelComparisonTests.cs` OR add a new lean test class
+
+- [ ] **Step 1: Add Q2_K GPU dequant kernel to dequant.cu**
+
+Edit `native/kernels/dequant.cu`. Add (place after `dequant_q3_k_f16`, before `dequant_q4_k_f16`):
+
+```cuda
+// ── Q2_K: 84 bytes per 256 values ──────────────────────────────────
+// struct block_q2_K { uint8_t scales[16]; uint8_t qs[64]; half d; half dmin; };
+// - scales[i]: low nibble = sub-block i scale, high nibble = sub-block i dmin coef
+// - qs[i]: 2-bit elements packed 4 per byte (low-to-high)
+//
+// 256 threads/block, one element per thread, FP16 store.
+
+#define Q2_K_SUPER_BLOCK_SIZE 256
+#define Q2_K_BLOCK_BYTES 84
+
+extern "C" __global__ void __launch_bounds__(256) dequant_q2_k_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_superblocks)
+{
+ int t = threadIdx.x; // 0..255
+
+ for (int sb_idx = blockIdx.x; sb_idx < total_superblocks; sb_idx += gridDim.x)
+ {
+ const uint8_t* block = src + (size_t)sb_idx * Q2_K_BLOCK_BYTES;
+ const uint8_t* scales = block; // 16 bytes
+ const uint8_t* qs = block + 16; // 64 bytes
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ int sub = t >> 4; // t / 16
+ int byte_idx = t >> 2; // t / 4
+ int bit_off = (t & 0x3) << 1; // (t % 4) * 2
+ int q2 = (qs[byte_idx] >> bit_off) & 0x3;
+ int scale = scales[sub] & 0xF;
+ int dm_coef = (scales[sub] >> 4) & 0xF;
+
+ float result = d * (float)scale * (float)q2 - dmin * (float)dm_coef;
+ dst[(size_t)sb_idx * Q2_K_SUPER_BLOCK_SIZE + t] = __float2half(result);
+ }
+}
+```
+
+- [ ] **Step 2: Build PTX**
+
+Run: `bash scripts/gpu-lock.sh acquire q2k-build "build dequant.ptx" 600 && bash native/build_ptx.sh dequant 2>&1 | tail -10; bash scripts/gpu-lock.sh release q2k-build`
+
+(Or `native/build_ptx.bat dequant` on Windows-only shells.)
+
+Expected: clean compilation, `native/ptx/dequant.ptx` updated.
+
+- [ ] **Step 3: Add Q2_K dispatch case in CudaKernels.cs**
+
+Edit `src/DotLLM.Cuda/CudaKernels.cs`. Find the dequant module function fields (around line 340); add:
+
+```csharp
+ _dequantQ2_KFunc = _dequantModule.TryGetFunction("dequant_q2_k_f16");
+```
+
+Find `LaunchDequantToF16` (around line 1599); add the Q2_K case before Q3_K:
+
+```csharp
+ case QuantizationType.Q2_K:
+ {
+ int totalSuperblocks = totalElements / 256;
+ int tsbArg = totalSuperblocks;
+ void** args = stackalloc void*[] {&srcArg, &dstArg, &tsbArg};
+ uint gridDim = (uint)Math.Min(totalSuperblocks, MaxDequantGridSize);
+ CudaDriverApi.cuLaunchKernel(_dequantQ2_KFunc,
+ gridDim, 1, 1, BlockSize, 1, 1,
+ 0, stream, (nint)args, 0).ThrowOnError();
+ return;
+ }
+```
+
+Add the `_dequantQ2_KFunc` field declaration in the class field section.
+
+- [ ] **Step 4: Write GPU dequant parity test**
+
+Add to `tests/DotLLM.Tests.Unit/Cuda/CudaKernelComparisonTests.cs` (or extend the existing dequant comparison area). The test synthesizes random Q2_K block bytes, dequants on CPU + GPU, asserts max-abs-diff < 1e-3.
+
+```csharp
+ [SkippableFact]
+ public void DequantQ2K_GpuMatchesCpu()
+ {
+ Skip.IfNot(_available, "No CUDA GPU available or PTX missing");
+
+ const int superBlocks = 16; // 16 × 256 = 4096 elements
+ const int elementCount = superBlocks * 256;
+ const int blockBytes = 84;
+ long totalBytes = (long)superBlocks * blockBytes;
+
+ var rng = new Random(0xC0FFEE);
+ byte[] hostBytes = new byte[totalBytes];
+ rng.NextBytes(hostBytes);
+
+ // Make d / dmin reasonable halves at offset 80 / 82 of each super-block
+ unsafe {
+ fixed (byte* p = hostBytes) {
+ for (int sb = 0; sb < superBlocks; sb++) {
+ byte* block = p + sb * blockBytes;
+ *(Half*)(block + 80) = (Half)((rng.NextDouble() - 0.5) * 0.04);
+ *(Half*)(block + 82) = (Half)((rng.NextDouble() - 0.5) * 0.02);
+ }
+ }
+ }
+
+ // CPU reference
+ float[] cpuRef = new float[elementCount];
+ unsafe {
+ fixed (byte* p = hostBytes) {
+ Dequantize.ToFloat32((nint)p, elementCount, QuantizationType.Q2_K, cpuRef);
+ }
+ }
+
+ // GPU path
+ Half[] gpuOut = new Half[elementCount];
+ nint devSrc = 0, devDst = 0;
+ try {
+ CudaDriverApi.cuMemAlloc_v2(out devSrc, (nuint)totalBytes).ThrowOnError();
+ CudaDriverApi.cuMemAlloc_v2(out devDst, (nuint)((long)elementCount * sizeof(ushort))).ThrowOnError();
+ unsafe {
+ fixed (byte* p = hostBytes)
+ CudaDriverApi.cuMemcpyHtoD_v2(devSrc, (nint)p, (nuint)totalBytes).ThrowOnError();
+ }
+ _kernels!.LaunchDequantToF16(devSrc, QuantizationType.Q2_K, devDst, elementCount, _stream!.Handle);
+ _stream.Synchronize();
+ unsafe {
+ fixed (Half* p = gpuOut)
+ CudaDriverApi.cuMemcpyDtoH_v2((nint)p, devDst, (nuint)((long)elementCount * sizeof(ushort))).ThrowOnError();
+ }
+ }
+ finally {
+ if (devSrc != 0) CudaDriverApi.cuMemFree_v2(devSrc);
+ if (devDst != 0) CudaDriverApi.cuMemFree_v2(devDst);
+ }
+
+ float maxAbs = 0f;
+ for (int i = 0; i < elementCount; i++) {
+ float diff = MathF.Abs(cpuRef[i] - (float)gpuOut[i]);
+ if (diff > maxAbs) maxAbs = diff;
+ }
+ _output.WriteLine($"Q2_K dequant max-abs-diff (GPU vs CPU): {maxAbs:F6}");
+ Assert.True(maxAbs < 1e-3f, $"Q2_K GPU dequant diverges from CPU (max-abs-diff={maxAbs}).");
+ }
+```
+
+- [ ] **Step 5: Run tests to verify they pass**
+
+Run:
+```bash
+bash scripts/gpu-lock.sh acquire q2k-test "Q2_K dequant parity" 600
+$env:CUDA_PATH="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" # or rely on Round-15 fallback
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~DequantQ2K_GpuMatchesCpu" -nologo
+bash scripts/gpu-lock.sh release q2k-test
+```
+
+Expected: 1 test passed.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add native/kernels/dequant.cu native/ptx/dequant.ptx \
+ src/DotLLM.Cuda/CudaKernels.cs \
+ tests/DotLLM.Tests.Unit/Cuda/CudaKernelComparisonTests.cs
+git commit -m "$(cat <<'EOF'
+CUDA: Q2_K GPU dequant kernel
+
+Adds dequant_q2_k_f16 mirroring the existing dequant_q4_k_f16 pattern:
+one CUDA block per super-block, 256 threads (one element per thread),
+FP16 store. Substituted decode body for Q2_K's 16 sub-blocks of
+16 elements with 4-bit scale + 4-bit min coef per sub-block.
+
+GPU vs CPU parity test passes within FP16 rounding (1e-3 max-abs-diff).
+EOF
+)"
+```
+
+---
+
+### Task 3: Q2_K per-call quantized GEMV
+
+**Files:**
+- Modify: `native/kernels/quantized_gemv.cu`
+- Modify: `src/DotLLM.Cuda/CudaKernels.cs`
+- Modify: `tests/DotLLM.Tests.Unit/Cuda/CudaQuantizedGemvAlignmentTests.cs`
+
+- [ ] **Step 1: Add quantized_gemv_q2_k kernel**
+
+Edit `native/kernels/quantized_gemv.cu`. Add (after `quantized_gemv_q4_k`):
+
+```cuda
+// ── Q2_K: 84 bytes per 256 values ──────────────────────────────────
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q2_k(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ // 16 sub-blocks of 16 elements
+ for (int sub = 0; sub < 16; sub++) {
+ int sc = scales[sub] & 0xF;
+ int dm = (scales[sub] >> 4) & 0xF;
+
+ float sub_acc = 0.0f;
+ float xsum_sub = 0.0f;
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++) {
+ int t = sub * 16 + j;
+ int byte_idx = t >> 2;
+ int bit_off = (t & 0x3) << 1;
+ int q2 = (qs[byte_idx] >> bit_off) & 0x3;
+ float xv = __half2float(x[sb * 256 + t]);
+ sub_acc += (float)q2 * xv;
+ xsum_sub += xv;
+ }
+ acc += d * (float)sc * sub_acc - dmin * (float)dm * xsum_sub;
+ }
+ }
+
+ // Warp reduction
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0) {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+ if (threadIdx.x == 0) y[row] = __float2half(acc);
+}
+```
+
+- [ ] **Step 2: Build PTX**
+
+Run: `bash scripts/gpu-lock.sh acquire q2k-build "build quantized_gemv.ptx" 600 && bash native/build_ptx.sh quantized_gemv 2>&1 | tail -10; bash scripts/gpu-lock.sh release q2k-build`
+
+- [ ] **Step 3: Wire dispatch in CudaKernels.cs**
+
+Find the per-call GEMV module fields:
+
+```csharp
+ _quantizedGemvQ2_KFunc = _quantizedGemvModule.TryGetFunction("quantized_gemv_q2_k");
+```
+
+Add the field declaration. Then update `LaunchQuantizedGemv` switch:
+
+```csharp
+ QuantizationType.Q2_K => _quantizedGemvQ2_KFunc,
+```
+
+Update `HasQuantizedGemv`:
+
+```csharp
+ public static bool HasQuantizedGemv(QuantizationType qt) =>
+ qt is QuantizationType.Q8_0 or QuantizationType.Q4_K or QuantizationType.Q5_0
+ or QuantizationType.Q5_K or QuantizationType.Q6_K
+ or QuantizationType.Q2_K; // NEW
+```
+
+Update `MinKAlignmentFor` — add Q2_K to the K-quant 256-alignment branch:
+
+```csharp
+ QuantizationType.Q3_K or QuantizationType.Q4_K
+ or QuantizationType.Q5_K or QuantizationType.Q6_K
+ or QuantizationType.Q2_K => 256, // NEW
+```
+
+- [ ] **Step 4: Write Q2_K GEMV parity test**
+
+Add to `tests/DotLLM.Tests.Unit/Cuda/CudaQuantizedGemvAlignmentTests.cs`. Add a new theory:
+
+```csharp
+ [SkippableTheory]
+ [InlineData(QuantizationType.Q2_K, 2048, 2048, 84)]
+ [InlineData(QuantizationType.Q2_K, 2048, 1408, 84)]
+ [InlineData(QuantizationType.Q2_K, 256, 256, 84)]
+ public void Q2K_GemvMatchesScalarReference(
+ QuantizationType qt, int M, int K, int blockBytes)
+ {
+ Skip.IfNot(IsCudaDriverPresent(), "No CUDA GPU available");
+ // Q2_K requires K%256==0. Tests with K=1408 are *expected* to throw via
+ // the K-alignment gate (1408 is not multiple of 256). Skip those for Q2_K.
+ if (qt == QuantizationType.Q2_K && (K % 256) != 0)
+ Skip.If(true, "Q2_K requires K%256==0; this shape exercises a different path.");
+
+ RunGemvVsScalar(qt, M, K, blockBytes);
+ }
+```
+
+Extend `RunGemvVsScalar` (already exists in this file) to handle Q2_K — add a branch in the synthesis loop and the `ComputeScalarReference` switch:
+
+```csharp
+ else if (qt == QuantizationType.Q2_K)
+ {
+ // Q2_K block: scales[16] + qs[64] + d + dmin = 84 bytes
+ Half d = (Half)((rng.NextDouble() - 0.5) * 0.04);
+ Half dmin = (Half)((rng.NextDouble() - 0.5) * 0.02);
+ fixed (byte* pBlk = blk) {
+ *(Half*)(pBlk + 80) = d;
+ *(Half*)(pBlk + 82) = dmin;
+ }
+ for (int j = 0; j < 16; j++) blk[j] = (byte)rng.Next(0, 256);
+ for (int j = 16; j < 80; j++) blk[j] = (byte)rng.Next(0, 256);
+ }
+```
+
+And in `ComputeScalarReference`:
+
+```csharp
+ else if (qt == QuantizationType.Q2_K)
+ {
+ Half d, dmin;
+ fixed (byte* pW = weight) {
+ d = *(Half*)(pW + blkOff + 80);
+ dmin = *(Half*)(pW + blkOff + 82);
+ }
+ float dF = (float)d, dminF = (float)dmin;
+ // Walk the 256 elements of this block (we're inside the per-block loop).
+ // ... we need to refactor — for Q2_K the inner loop walks 256 elements per superblock,
+ // not 32 (Q8_0/Q5_0). The existing test infrastructure was for block_size=32 quants.
+ throw new NotImplementedException("Q2_K scalar reference goes here — see plan");
+ }
+```
+
+NOTE: the existing `CudaQuantizedGemvAlignmentTests` was built for block_size=32 quants (Q8_0, Q5_0). Q2_K is block_size=256. **Either extend `RunGemvVsScalar` to handle both block sizes, OR add a new `RunGemvVsScalarKQuant` helper.** The cleaner choice is the latter:
+
+```csharp
+ [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining)]
+ private unsafe void RunGemvVsScalarKQuant(QuantizationType qt, int M, int K, int blockBytes)
+ {
+ // Test infrastructure for K-quants (block_size=256). Mirrors RunGemvVsScalar
+ // structurally but iterates over superblocks (K/256) per row.
+ using var ctx = CudaContext.Create(0);
+ using var stream = CudaStream.Create();
+ string? ptxDir = FindPtxDir();
+ Skip.If(ptxDir == null, "PTX files not found");
+ using var kernels = new CudaKernels(ptxDir!);
+ Skip.IfNot(CudaKernels.HasQuantizedGemv(qt), $"No GEMV kernel for {qt}");
+ Assert.Equal(0, K % 256);
+
+ int sbPerRow = K / 256;
+ long rowBytes = (long)sbPerRow * blockBytes;
+ long weightBytes = (long)M * rowBytes;
+ var rng = new Random(0xBEEF ^ (int)qt ^ M ^ K);
+
+ byte[] hostW = new byte[weightBytes];
+ rng.NextBytes(hostW);
+ // Per-super-block: write d/dmin halves at known offset (Q2_K: +80/+82)
+ unsafe {
+ fixed (byte* p = hostW) {
+ for (int row = 0; row < M; row++) {
+ for (int sb = 0; sb < sbPerRow; sb++) {
+ byte* blk = p + row * rowBytes + sb * blockBytes;
+ if (qt == QuantizationType.Q2_K) {
+ *(Half*)(blk + 80) = (Half)((rng.NextDouble() - 0.5) * 0.04);
+ *(Half*)(blk + 82) = (Half)((rng.NextDouble() - 0.5) * 0.02);
+ }
+ }
+ }
+ }
+ }
+
+ Half[] hostX = new Half[K];
+ for (int i = 0; i < K; i++) {
+ double u1 = 1.0 - rng.NextDouble();
+ double u2 = 1.0 - rng.NextDouble();
+ double g = Math.Sqrt(-2.0 * Math.Log(u1)) * Math.Cos(2.0 * Math.PI * u2);
+ hostX[i] = (Half)(g * 0.4);
+ }
+
+ // Scalar reference: dequant whole row → F32 dot.
+ float[] yRef = new float[M];
+ float[] xF32 = new float[K];
+ for (int i = 0; i < K; i++) xF32[i] = (float)hostX[i];
+ unsafe {
+ fixed (byte* p = hostW) {
+ float[] rowDequant = new float[K];
+ for (int row = 0; row < M; row++) {
+ Dequantize.ToFloat32((nint)(p + row * rowBytes), K, qt, rowDequant);
+ float acc = 0;
+ for (int i = 0; i < K; i++) acc += rowDequant[i] * xF32[i];
+ yRef[row] = acc;
+ }
+ }
+ }
+
+ // GPU GEMV
+ long xBytes = (long)K * sizeof(ushort);
+ long yBytes = (long)M * sizeof(ushort);
+ nint devW = 0, devX = 0, devY = 0;
+ Half[] yGpu = new Half[M];
+ try {
+ CudaDriverApi.cuMemAlloc_v2(out devW, (nuint)weightBytes).ThrowOnError();
+ CudaDriverApi.cuMemAlloc_v2(out devX, (nuint)xBytes).ThrowOnError();
+ CudaDriverApi.cuMemAlloc_v2(out devY, (nuint)yBytes).ThrowOnError();
+ unsafe {
+ fixed (byte* pW = hostW) CudaDriverApi.cuMemcpyHtoD_v2(devW, (nint)pW, (nuint)weightBytes).ThrowOnError();
+ fixed (Half* pX = hostX) CudaDriverApi.cuMemcpyHtoD_v2(devX, (nint)pX, (nuint)xBytes).ThrowOnError();
+ }
+ kernels.LaunchQuantizedGemv(devW, qt, devX, devY, M, K, stream.Handle);
+ stream.Synchronize();
+ unsafe { fixed (Half* p = yGpu) CudaDriverApi.cuMemcpyDtoH_v2((nint)p, devY, (nuint)yBytes).ThrowOnError(); }
+ }
+ finally {
+ if (devW != 0) CudaDriverApi.cuMemFree_v2(devW);
+ if (devX != 0) CudaDriverApi.cuMemFree_v2(devX);
+ if (devY != 0) CudaDriverApi.cuMemFree_v2(devY);
+ }
+
+ float maxAbs = 0f, refMax = 0f;
+ for (int i = 0; i < M; i++) {
+ float diff = MathF.Abs((float)yGpu[i] - yRef[i]);
+ if (diff > maxAbs) maxAbs = diff;
+ if (MathF.Abs(yRef[i]) > refMax) refMax = MathF.Abs(yRef[i]);
+ }
+ _out.WriteLine($"{qt} M={M} K={K}: ref|max|={refMax:F3} max-abs-diff={maxAbs:F5}");
+ Assert.True(maxAbs < 0.05f,
+ $"Q2_K GEMV diverges from scalar reference (max-abs-diff={maxAbs}, refMax={refMax}).");
+ }
+```
+
+Then change the Q2_K theory to call `RunGemvVsScalarKQuant`:
+
+```csharp
+ [SkippableTheory]
+ [InlineData(QuantizationType.Q2_K, 2048, 2048, 84)]
+ [InlineData(QuantizationType.Q2_K, 2048, 1024, 84)]
+ [InlineData(QuantizationType.Q2_K, 256, 256, 84)]
+ public void GemvKQuantMatchesScalarReference(
+ QuantizationType qt, int M, int K, int blockBytes)
+ {
+ Skip.IfNot(IsCudaDriverPresent(), "No CUDA GPU available");
+ RunGemvVsScalarKQuant(qt, M, K, blockBytes);
+ }
+```
+
+- [ ] **Step 5: Run tests to verify they pass**
+
+Run:
+```bash
+bash scripts/gpu-lock.sh acquire q2k-gemv "Q2_K GEMV parity" 600
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~Q2K|FullyQualifiedName~GemvKQuant" -nologo
+bash scripts/gpu-lock.sh release q2k-gemv
+```
+
+Expected: 3 new tests passed.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add native/kernels/quantized_gemv.cu native/ptx/quantized_gemv.ptx \
+ src/DotLLM.Cuda/CudaKernels.cs \
+ tests/DotLLM.Tests.Unit/Cuda/CudaQuantizedGemvAlignmentTests.cs
+git commit -m "$(cat <<'EOF'
+CUDA: Q2_K per-call quantized GEMV
+
+Adds quantized_gemv_q2_k mirroring quantized_gemv_q4_k pattern (one
+CUDA block per output row, FP32 accumulation, FP16 store, warp+block
+reduction). Inner loop walks 16 sub-blocks of 16 elements per
+super-block with 4-bit scale + 4-bit dmin coef each.
+
+HasQuantizedGemv + MinKAlignmentFor extended to recognize Q2_K
+(K%256 alignment requirement, same as other K-quants).
+
+GEMV vs scalar reference parity within 5e-2 max-abs-diff at
+V2-Lite-class shapes (M=2048, K=2048).
+EOF
+)"
+```
+
+---
+
+### Task 4: Q2_K MMQ kernel (port from Q4_K MMQ)
+
+**Files:**
+- Modify: `native/kernels/quantized_gemv_mmq.cu`
+- Modify: `src/DotLLM.Cuda/CudaKernels.cs`
+- Modify: `tests/DotLLM.Tests.Unit/Cuda/CudaMmqKernelTests.cs`
+
+- [ ] **Step 1: Read the existing Q4_K MMQ kernel as the porting source**
+
+Run: `sed -n '47,219p' native/kernels/quantized_gemv_mmq.cu`
+
+Read it carefully — note especially:
+- Pre-Q8_1 input scratch contract (fixed format from `quantize_x_to_q8_1`)
+- Per-warp 4-rows-per-block tile layout
+- dp4a inner loop structure (`__dp4a(W_int8x4, X_int8x4, acc)`)
+- Per-super-block scale extraction (Q4_K's 6-bit packed → for Q2_K we need 4-bit scale + 4-bit dmin coef)
+- Final FP16 store with d × scale - dmin × dmin_coef × xsum applied
+
+- [ ] **Step 2: Add quantized_gemv_q2_k_mmq kernel**
+
+Edit `native/kernels/quantized_gemv_mmq.cu`. Add (after the Q4_K MMQ kernel — find `quantized_gemv_q4_k_mmq` and place Q2_K's MMQ before it OR add after the Q6_K MMQ before the MMVQ-large kernels, your choice based on file flow). Substitute the Q4_K decode body for Q2_K's:
+
+```cuda
+// ── Q2_K MMQ: 84 bytes per 256 values, 4-bit scale + 4-bit dmin coef per sub-block ──
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q2_k_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ // Layout mirrors Q4_K MMQ:
+ // - blockDim.x = 256 threads, block_y = 1
+ // - 32 warps × 8 lanes per warp tile = 4 output rows × 32 elements per warp
+ // - W_tile[4][32] in shared mem stores int8 weight bytes
+ // - X_tile[32] in shared mem stores int8 input bytes (pre-Q8_1)
+ // - dp4a inner loop accumulates int32 per (row, K-position)
+ //
+ // Per-super-block decode:
+ // scales[sub] (1 byte): low nibble = sc, high nibble = dm_coef
+ // qs[byte_idx]: 4 elements packed 2-bit
+ // d, dmin: half values at offset +80, +82
+
+ // Read the Q4_K MMQ kernel and copy structure here, with these substitutions:
+ // Q4_K_BLOCK_BYTES (144) → Q2_K_BLOCK_BYTES (84)
+ // 8 sub-blocks of 32 → 16 sub-blocks of 16
+ // 6-bit packed scales (12 bytes from offset 4) → 16 bytes of (sc|dm) at offset 0
+ // nibble decode (4-bit) → 2-bit decode (4 elements per byte)
+ // q4 sign convention: nibble × scale → q2 × scale - dmin × dm_coef × xsum
+ //
+ // The dp4a fma stays identical — we only change how W_tile is filled
+ // (decode body) and how the final FP32 → FP16 cast is performed (Q2_K
+ // applies BOTH d-scale-positive AND dmin-coef-negative terms).
+
+ /* PASTE Q4_K MMQ kernel body here, then apply substitutions per
+ comment block above. ~170 lines total. Pin the Q4_K reference at
+ quantized_gemv_mmq.cu:47-219 in the commit message. */
+}
+```
+
+**This task does NOT include the full ~170-line MMQ kernel inline because the porting work is mechanical given the existing Q4_K reference.** The implementing engineer reads the Q4_K MMQ source, applies the documented substitutions, and produces a new ~150-180-line kernel. The output is a Q2_K MMQ that:
+- Consumes the same Q8_1-style FP16 input (calls `quantize_x_to_q8_1` per call to convert input).
+- Lays out the same 4-rows × 32-elements warp tiles in shared memory.
+- Runs the same dp4a accumulation chain.
+- Differs only in the per-superblock decode (16 sub-blocks of 16 elements + 4-bit scale + 4-bit dmin coef) and the final FP32 → FP16 cast formula (`d*sc*sub_acc - dmin*dm*xsum_sub`).
+
+If during implementation a structural difference emerges that prevents direct porting (e.g., the 16-sub-block × 16-element layout creates a warp-tile mismatch), the engineer falls back to the same approach Q3_K uses — which currently has dequant-only support — and this task expands to also adding a Q3_K MMQ kernel as a sibling. Pin that decision in the PR description.
+
+- [ ] **Step 3: Build PTX**
+
+Run: `bash scripts/gpu-lock.sh acquire q2k-mmq "build q2_k mmq" 600 && bash native/build_ptx.sh quantized_gemv_mmq 2>&1 | tail -10; bash scripts/gpu-lock.sh release q2k-mmq`
+
+- [ ] **Step 4: Wire dispatch**
+
+In `CudaKernels.cs`:
+- Add field: `private readonly nint _quantizedGemvQ2_KMmqFunc;`
+- Load at ctor: `_quantizedGemvQ2_KMmqFunc = _quantizedGemvMmqModule.TryGetFunction("quantized_gemv_q2_k_mmq");`
+- Add property: `public bool HasMmqQ2K => _quantizedGemvQ2_KMmqFunc != 0 && !DisableMmqQ2K;` plus `static bool DisableMmqQ2K`.
+- Update `HasMmq(qt)`: `Q2_K => HasMmqQ2K,`
+- Update `LaunchQuantizedGemvMmq` switch: `QuantizationType.Q2_K => _quantizedGemvQ2_KMmqFunc,`
+
+- [ ] **Step 5: Add MMQ parity test**
+
+Add to `tests/DotLLM.Tests.Unit/Cuda/CudaMmqKernelTests.cs`. Mirror the Q4_K theory pattern:
+
+```csharp
+ [SkippableTheory]
+ [InlineData(4, 256)]
+ [InlineData(8, 512)]
+ [InlineData(64, 1024)]
+ [InlineData(2048, 2048)]
+ public void MmqQ2K_MatchesLegacyWithinTolerance(int n, int k)
+ {
+ Skip.IfNot(IsCudaDriverPresent(), "No CUDA GPU available");
+ RunMmqEquivalence(QuantizationType.Q2_K, n, k, blockBytes: 84,
+ (rng, span) => SynthesizeQ2KBlock(rng, span));
+ }
+
+ private static unsafe void SynthesizeQ2KBlock(Random rng, Span block)
+ {
+ Assert.Equal(84, block.Length);
+ Half d = (Half)((rng.NextDouble() - 0.5) * 0.04);
+ Half dmin = (Half)((rng.NextDouble() - 0.5) * 0.02);
+ fixed (byte* pBlk = block) {
+ *(Half*)(pBlk + 80) = d;
+ *(Half*)(pBlk + 82) = dmin;
+ }
+ for (int i = 0; i < 80; i++) block[i] = (byte)rng.Next(0, 256);
+ }
+```
+
+- [ ] **Step 6: Run tests**
+
+Run:
+```bash
+bash scripts/gpu-lock.sh acquire q2k-mmq-test "Q2_K MMQ parity" 600
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~MmqQ2K" -nologo
+bash scripts/gpu-lock.sh release q2k-mmq-test
+```
+
+Expected: 4 tests passed within 3% peak-relative tolerance (matches existing Q4_K MMQ test bar).
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add native/kernels/quantized_gemv_mmq.cu native/ptx/quantized_gemv_mmq.ptx \
+ src/DotLLM.Cuda/CudaKernels.cs \
+ tests/DotLLM.Tests.Unit/Cuda/CudaMmqKernelTests.cs
+git commit -m "$(cat <<'EOF'
+CUDA: Q2_K MMQ kernel (dp4a fast path)
+
+Ports quantized_gemv_q4_k_mmq pattern to Q2_K with substituted
+per-super-block decode body: 16 sub-blocks of 16 elements with
+4-bit scale + 4-bit dmin coef per sub-block, vs Q4_K's 8 sub-blocks
+of 32 elements with 6-bit packed scales.
+
+Inner dp4a loop unchanged — int8 W_tile × int8 X_tile (pre-Q8_1
+quantized input). Only the W_tile fill (decode prefix) and final
+FP16 store (d*sc*acc - dmin*dm*xsum) differ from Q4_K.
+
+MMQ vs legacy GEMV parity within 3% peak-relative on synthetic
+Llama70B-class shapes (n=2048, k=2048).
+
+Q4_K reference: native/kernels/quantized_gemv_mmq.cu:47-219.
+EOF
+)"
+```
+
+---
+
+### Task 5: Q2_K MMQ pre-Q8_1 variant
+
+**Files:**
+- Modify: `native/kernels/quantized_gemv_mmq.cu`
+- Modify: `src/DotLLM.Cuda/CudaKernels.cs`
+- Modify: `tests/DotLLM.Tests.Unit/Cuda/CudaMmqKernelTests.cs`
+
+- [ ] **Step 1: Add quantized_gemv_q2_k_mmq_preq kernel**
+
+The `_preq` variant skips the per-call input quantization step; the input scratch is pre-quantized once per MoE/MLA projection (existing `quantize_x_to_q8_1` kernel writes the scratch; subsequent kernel calls read it). Body is identical to `quantized_gemv_q2_k_mmq` minus the per-warp input convert lane.
+
+Reference the Q4_K _preq pattern at `native/kernels/quantized_gemv_mmq.cu:1110-1228`. Substitute the Q2_K decode body. Same structural rule as Task 4.
+
+- [ ] **Step 2: Build PTX, wire dispatch, add test, commit**
+
+Repeat the Task 4 step pattern:
+- Build PTX
+- Add field `_quantizedGemvQ2_KMmqPreqFunc`, load via `TryGetFunction`
+- Update `LaunchQuantizedGemvMmqPreq` switch
+- Add test theory `MmqQ2K_PreQ8_1_MatchesOnTheFly` mirroring existing pre-Q8_1 tests
+- Run tests
+- Commit with message `CUDA: Q2_K MMQ pre-Q8_1 variant`
+
+---
+
+### Task 6: Q2_K MMVQ-large + MMVQ-large pre-Q8_1
+
+**Files:**
+- Modify: `native/kernels/quantized_gemv_mmq.cu`
+- Modify: `src/DotLLM.Cuda/CudaKernels.cs`
+- Modify: `tests/DotLLM.Tests.Unit/Cuda/CudaMmqKernelTests.cs`
+
+- [ ] **Step 1: Add quantized_gemv_q2_k_mmvq_large kernel**
+
+Reference Q4_K MMVQ-large pattern at `native/kernels/quantized_gemv_mmq.cu:630-784`. The MMVQ-large layout is 1 CUDA block per output row, multiple warps span the K-reduction, optimized for M ≥ 1024. Substitute Q2_K decode body.
+
+- [ ] **Step 2: Add MMVQ-large pre-Q8_1 variant + dispatch + test**
+
+Mirrors Task 5 pattern.
+
+Run tests:
+```bash
+bash scripts/gpu-lock.sh acquire q2k-mmvq "Q2_K MMVQ-large parity" 600
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~MmvqLargeQ2K" -nologo
+bash scripts/gpu-lock.sh release q2k-mmvq
+```
+
+Expected: 4 new tests passed. Commit with `CUDA: Q2_K MMVQ-large + pre-Q8_1 variants`.
+
+---
+
+### Task 7: Q2_K grouped-MoE-GEMV
+
+**Files:**
+- Modify: `native/kernels/moe_grouped_gemv.cu`
+- Modify: `src/DotLLM.Cuda/CudaKernels.cs`
+- Modify: `tests/DotLLM.Tests.Unit/Cuda/CudaMoeGroupedGemvTests.cs`
+
+- [ ] **Step 1: Add moe_grouped_gemv_q2_k_f16 kernel**
+
+Edit `native/kernels/moe_grouped_gemv.cu`. The grouped variant is structurally identical to the per-call `quantized_gemv_q2_k`, except the per-expert weight pointer is loaded from a device array indexed by `blockIdx.y`:
+
+```cuda
+extern "C" __global__ void moe_grouped_gemv_q2_k_f16(
+ const half* __restrict__ x,
+ const uintptr_t* __restrict__ weights_ptrs,
+ uintptr_t* __restrict__ outputs_ptrs,
+ const int M, const int K, const int K_active)
+{
+ int m = blockIdx.x, e = blockIdx.y;
+ if (m >= M || e >= K_active) return;
+
+ const uint8_t* W = (const uint8_t*)weights_ptrs[e];
+ half* y = (half*)outputs_ptrs[e];
+
+ // Body identical to quantized_gemv_q2_k inner loop; just substitute
+ // weight_ptr-from-array and write result to per-expert output.
+
+ // ... 16 sub-blocks decode, dp4a-free FP32 accumulation,
+ // warp + block reduction, single FP16 store at y[m].
+}
+```
+
+(Full kernel ~80-100 lines. Reference moe_grouped_gemv_q4_k_f16 in the same file as the porting source.)
+
+- [ ] **Step 2: Build PTX, wire dispatch, add test**
+
+```csharp
+// In CudaKernels.cs:
+_moeGroupedGemvQ2_KFunc = _moeGroupedGemvModule.TryGetFunction("moe_grouped_gemv_q2_k_f16");
+
+public bool HasMoeGroupedGemv(QuantizationType qt) => qt switch {
+ /* existing entries */
+ Q2_K => _moeGroupedGemvQ2_KFunc != 0,
+ /* ... */
+};
+
+// Add to LaunchMoeGroupedGemv switch:
+QuantizationType.Q2_K => _moeGroupedGemvQ2_KFunc,
+```
+
+Add test theory in `CudaMoeGroupedGemvTests.cs`:
+
+```csharp
+ [SkippableTheory]
+ [InlineData(4, 256, 256)]
+ [InlineData(4, 1408, 2048)]
+ [InlineData(2, 256, 512)]
+ public void GroupedQ2K_MatchesPerCallWithinFp16Tolerance(int kActive, int M, int K)
+ {
+ Skip.IfNot(IsCudaDriverPresent(), "No CUDA GPU available");
+ RunGroupedEquivalence(QuantizationType.Q2_K, kActive, M, K, blockBytes: 84,
+ (rng, span) => SynthesizeQ2KBlock(rng, span));
+ }
+```
+
+(Reuses the existing `RunGroupedEquivalence` helper. Add `SynthesizeQ2KBlock` if not already present from Task 4.)
+
+- [ ] **Step 3: Run tests**
+
+```bash
+bash scripts/gpu-lock.sh acquire q2k-grouped "Q2_K grouped MoE GEMV" 600
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~GroupedQ2K" -nologo
+bash scripts/gpu-lock.sh release q2k-grouped
+```
+
+Expected: 3 new tests passed within 1e-3 max-abs-diff (matches existing grouped tests).
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add native/kernels/moe_grouped_gemv.cu native/ptx/moe_grouped_gemv.ptx \
+ src/DotLLM.Cuda/CudaKernels.cs \
+ tests/DotLLM.Tests.Unit/Cuda/CudaMoeGroupedGemvTests.cs
+git commit -m "CUDA: Q2_K grouped-MoE-GEMV variant"
+```
+
+---
+
+### Task 8: Real-GGUF V2-Lite Q2_K end-to-end smoke
+
+**Files:**
+- Modify: `tests/DotLLM.Tests.Unit/Models/Architectures/DeepSeekV2GgufLoadTests.cs`
+
+- [ ] **Step 1: Cache the V2-Lite Q2_K GGUF**
+
+Manual step (~10 min one-time):
+```
+Download bartowski/DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf (~5 GB)
+Place at: ~/.dotllm/models/bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF/DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf
+```
+
+If absent, the test added in Step 2 skips gracefully via `Skip.If`.
+
+- [ ] **Step 2: Add the smoke test**
+
+Edit `tests/DotLLM.Tests.Unit/Models/Architectures/DeepSeekV2GgufLoadTests.cs`. Add (mirroring `RealGguf_Q3KM_FullModel_27LayerSmoke` at line ~301):
+
+```csharp
+ ///
+ /// Full 27-layer V2-Lite at Q2_K. Smaller than Q3_K_M (~5 GB on disk vs 8 GB),
+ /// so the full model fits more comfortably in 12 GB GPU. Asserts: 27-layer load +
+ /// prefill on 4 tokens + 3 decode steps, every step's logits finite. Skips if
+ /// the GGUF isn't cached.
+ ///
+ [SkippableFact]
+ [Trait("Category", "GPU")]
+ public void RealGguf_Q2K_FullModel_27LayerSmoke()
+ {
+ Skip.IfNot(CudaDevice.IsAvailable(), "No CUDA GPU available");
+ string path = Path.Combine(
+ Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
+ ".dotllm", "models", "bartowski", "DeepSeek-Coder-V2-Lite-Instruct-GGUF",
+ "DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf");
+ Skip.If(!File.Exists(path), $"Q2_K GGUF not cached at {path}");
+
+ using var gguf = GgufFile.Open(path);
+ var fullConfig = GgufModelConfigExtractor.Extract(gguf.Metadata);
+ Assert.Equal(27, fullConfig.NumLayers);
+ Assert.NotNull(fullConfig.MlaConfig);
+ Assert.NotNull(fullConfig.Moe);
+
+ // Trim KV cache horizon to 16 to fit smoke (matches Q3_K_M smoke pattern).
+ var config = fullConfig with { MaxSequenceLength = 16 };
+
+ using var model = CudaTransformerModel.LoadFromGguf(gguf, config);
+
+ // Prefill 4 tokens, decode 3 more.
+ int[] tokenIds = [100000, 261, 1559, 11];
+ int[] positions = [0, 1, 2, 3];
+
+ using ITensor logits = model.Forward(tokenIds, positions, deviceId: 0, kvCache: null);
+ AssertAllFinite(logits, "Q2_K prefill");
+
+ // ... continue with 3 decode steps, asserting finite logits each time
+ // (mirror the Q3_K_M smoke pattern verbatim).
+ }
+```
+
+- [ ] **Step 3: Run smoke**
+
+```bash
+bash scripts/gpu-lock.sh acquire q2k-smoke "V2-Lite Q2_K 27-layer smoke" 1800
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~RealGguf_Q2K" -nologo
+bash scripts/gpu-lock.sh release q2k-smoke
+```
+
+Expected: 1 smoke passed (or skipped if GGUF not cached) within ~30-60 s. The Q3_K_M baseline is 31 s; Q2_K is smaller so we expect similar or faster.
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add tests/DotLLM.Tests.Unit/Models/Architectures/DeepSeekV2GgufLoadTests.cs
+git commit -m "$(cat <<'EOF'
+CUDA: V2-Lite Q2_K full-27-layer real-GGUF smoke
+
+Asserts the full chain — Q2_K dequant + GEMV + MMQ + grouped-MoE-GEMV
+— produces finite logits end-to-end on the cached
+bartowski/DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf (5 GB) at all
+27 layers (1 dense + 26 MoE).
+
+Decisive evidence Phase 1 of Spec 1 ships V2-Lite-Q2_K on a single
+RTX 3060.
+EOF
+)"
+```
+
+---
+
+### Task 9: Final regression sweep + .continue-here.md update
+
+**Files:**
+- Modify: `.continue-here.md`
+
+- [ ] **Step 1: Run the full critical-path regression sweep**
+
+```bash
+bash scripts/gpu-lock.sh acquire q2k-regression "full Q2_K regression" 1200
+dotnet test tests/DotLLM.Tests.Unit/DotLLM.Tests.Unit.csproj -c Release --filter "FullyQualifiedName~CudaMoeGroupedGemvTests|FullyQualifiedName~CudaTransformerMlaForwardTests|FullyQualifiedName~CudaQuantizedGemvAlignmentTests|FullyQualifiedName~RealGguf_QuantizedMlaMoe|FullyQualifiedName~RealGguf_Q3KM|FullyQualifiedName~RealGguf_Q2K|FullyQualifiedName~Q2K|FullyQualifiedName~MmqQ2K|FullyQualifiedName~MmvqLargeQ2K|FullyQualifiedName~GroupedQ2K" -nologo --no-build
+bash scripts/gpu-lock.sh release q2k-regression
+```
+
+Expected:
+- All previous tests (24 critical CUDA + Q3_K_M smoke) still pass.
+- 27+ new Q2_K tests pass.
+- 0 failures.
+
+- [ ] **Step 2: Update .continue-here.md with Round 16 entry**
+
+Insert at the top of the file (replacing the date in the front-matter and prepending a Round 16 section):
+
+```markdown
+last_commit:
+last_updated: 2026-MM-DD (Round 16 — Q2_K full L3, Phase 1 of Spec 1)
+
+## Round 16: Q2_K full L3 (Phase 1 of Spec 1)
+
+The Q2_K K-quant family quantization type is now fully supported on CUDA:
+CPU dequant + GPU dequant + per-call GEMV + MMQ (legacy + pre-Q8_1) +
+MMVQ-large (legacy + pre-Q8_1) + grouped-MoE-GEMV. V2-Lite-Q2_K loads
+end-to-end and produces finite logits across all 27 layers on the
+12 GB RTX 3060.
+
+### Commits this round (`..`)
+
+```
+ CPU: Q2_K dequantization
+ CUDA: Q2_K GPU dequant kernel
+ CUDA: Q2_K per-call quantized GEMV
+ CUDA: Q2_K MMQ kernel (dp4a fast path)
+ CUDA: Q2_K MMQ pre-Q8_1 variant
+ CUDA: Q2_K MMVQ-large + pre-Q8_1 variants
+ CUDA: Q2_K grouped-MoE-GEMV variant
+ CUDA: V2-Lite Q2_K full-27-layer real-GGUF smoke
+```
+
+### What landed
+
+(Filled in by engineer during Task 9 Step 2 from the regression-sweep output:
+new Q2_K test count, V2-Lite Q2_K smoke timing vs the 31 s Q3_K_M baseline,
+critical-tests-still-green count, total CPU pass count, total failure count.)
+
+### Next: Plan 1.2 (IQ4_NL + IQ4_XS)
+
+Phase 2 of Spec 1. Brings the IQ-family infrastructure online —
+codebook constants in IQuantGrids.cs, new dequant_iquants.cu PTX,
+shared MMQ pattern with codebook expansion before dp4a fires.
+```
+
+- [ ] **Step 3: Final commit**
+
+```bash
+git add .continue-here.md
+git commit -m "$(cat <<'EOF'
+.continue-here.md: Round 16 — Q2_K full L3 lands (Phase 1 of Spec 1)
+
+Documents the Q2_K phase: 8 commits across CPU dequant, GPU dequant,
+per-call GEMV, MMQ + pre-Q8_1, MMVQ-large + pre-Q8_1,
+grouped-MoE-GEMV, real-GGUF V2-Lite Q2_K 27-layer smoke.
+
+Phase 2 (IQ4_NL + IQ4_XS) is next.
+EOF
+)"
+```
+
+- [ ] **Step 4: Optional — open the Phase 1 PR**
+
+```bash
+git push origin feature/mamba-3-cuda
+gh pr create --base feature/mamba-3 --title "CUDA: Q2_K full L3 (Phase 1 of quantization-coverage spec)" --body "$(cat <<'EOF'
+## Summary
+- Phase 1 of Spec 1 (`docs/superpowers/specs/2026-04-27-q2k-iquant-coverage-design.md`).
+- Adds Q2_K K-quant family support: CPU + GPU dequant, per-call GEMV, MMQ + MMVQ-large (legacy + pre-Q8_1), grouped-MoE-GEMV.
+- V2-Lite-Q2_K loads + decodes end-to-end on RTX 3060 (27 layers, 5 GB GGUF, ~30-60 s smoke).
+
+## Validation
+- 27+ new Q2_K-specific tests, 0 failures.
+- 24 critical CUDA tests (3 MLA + 12 grouped GEMV + 4 V2-Lite Q4_K_M smokes + 5 K=1408 alignment) still green.
+- 1467+ CPU tests still green.
+- Q3_K_M 27-layer smoke unchanged at ~31 s.
+- Build: 0 warnings, 0 errors.
+
+## Next steps
+Phase 2 (IQ4_NL + IQ4_XS) — separate plan, separate PR.
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+EOF
+)"
+```
+
+---
+
+## Self-Review
+
+**Spec coverage:** Each section of `2026-04-27-q2k-iquant-coverage-design.md` Phase 1 row maps to a task:
+- Section 1 (architecture) ↔ implicit in every task
+- Section 2 (CPU dequant) ↔ Task 1
+- Section 3 (GPU dequant) ↔ Task 2
+- Section 4 (per-call GEMV) ↔ Task 3
+- Section 5 (MMQ + MMVQ-large + pre-Q8_1) ↔ Tasks 4, 5, 6
+- Section 6 (grouped-MoE-GEMV) ↔ Task 7
+- Section 7 (testing strategy: kernel parity + real-GGUF smoke + regression) ↔ inline in each task + Task 8 + Task 9
+
+**Placeholder scan:** Tasks 4 / 5 / 6 reference Q4_K source-line numbers (`quantized_gemv_mmq.cu:47-219` etc.) and instruct the implementer to mechanically port the kernel body. This is intentional: the kernel bodies are 150-200 lines each and the porting work is largely mechanical given the existing reference. Embedding 600+ lines of CUDA inline would obscure the core changes (decode body substitution + final-cast formula). The plan documents the decode delta precisely; the implementing engineer reads the reference, applies the delta, and produces the new kernel. This is a deliberate granularity tradeoff.
+
+If the implementing agent prefers more granular per-line guidance for the MMQ kernels, expand Tasks 4-6 in-session by reading the Q4_K kernel and writing a per-section diff inline.
+
+**Type consistency:**
+- `_quantizedGemvQ2_KFunc` (Task 3), `_quantizedGemvQ2_KMmqFunc` (Task 4), `_quantizedGemvQ2_KMmqPreqFunc` (Task 5), `_quantizedGemvQ2_KMmvqLargeFunc` (Task 6), `_moeGroupedGemvQ2_KFunc` (Task 7) — naming matches existing `_quantizedGemvQ4_KFunc` etc. patterns.
+- `HasMmqQ2K` (Task 4), `HasMmvqLargeQ2K` (Task 6), `DisableMmqQ2K`, `DisableMmvqLargeQ2K` — match existing `HasMmqQ4K` / `DisableMmvqLargeQ4K` patterns.
+- `QuantizationType.Q2_K` consistent across all task references.
+- `MinKAlignmentFor(Q2_K) = 256` consistent (it's a K-quant).
+
+**Scope:** This plan is 8 tasks covering a single quantization type (Q2_K) end-to-end. ~1 week of work; one shippable PR. Right granularity.
+
+---
+
+## Execution Handoff
+
+Plan complete and saved to `docs/superpowers/plans/2026-04-27-q2k-phase1-plan.md`. Two execution options:
+
+**1. Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration. Each subagent gets the task content + relevant spec context; the parent agent reviews diffs + test output before approving the next task.
+
+**2. Inline Execution** — Execute tasks in this session using executing-plans, batch execution with checkpoints for review.
+
+Which approach?
diff --git a/docs/superpowers/specs/2026-04-27-q2k-iquant-coverage-design.md b/docs/superpowers/specs/2026-04-27-q2k-iquant-coverage-design.md
new file mode 100644
index 00000000..72f778d4
--- /dev/null
+++ b/docs/superpowers/specs/2026-04-27-q2k-iquant-coverage-design.md
@@ -0,0 +1,385 @@
+# Spec 1 — Q2_K + IQ-family quantization coverage (full L3)
+
+**Status**: Approved (2026-04-27). Ready for plan generation.
+**Owner**: feature/mamba-3-cuda branch.
+**Phasing**: 1 design (this doc) → 5 implementation plans → 5 PRs.
+
+## Goal
+
+Add full L3 (CPU dequant + GPU dequant + per-call GEMV + MMQ + MMVQ-large + grouped-MoE-GEMV) support for nine GGUF quantization types currently unsupported by dotLLM:
+
+- **Q2_K** (GGUF type 10, K-quant family, 84-byte 256-element superblocks)
+- **IQ4_NL** (20, 18-byte 32-element blocks, 16-entry signed lookup)
+- **IQ4_XS** (23, 136-byte 256-element superblocks, same 16-entry lookup)
+- **IQ3_S** (21, 110-byte superblocks, 512-entry int32 codebook)
+- **IQ3_XXS** (18, 98-byte superblocks, 256-entry int32 codebook)
+- **IQ2_S** (22, 82-byte superblocks, 1024-entry int64 codebook + sign bytes)
+- **IQ2_XS** (17, 74-byte superblocks, 512-entry int64 codebook + sign mask)
+- **IQ2_XXS** (16, 66-byte superblocks, 256-entry int64 codebook + sign mask)
+- **IQ1_S** (19, 50-byte superblocks, 2048-entry int16 grid + qh sub-block deltas)
+
+After Spec 1 lands, every GGUF quant type registered in upstream `ggml-quants.h` is loadable end-to-end on dotLLM CUDA — no more `Unsupported GGUF quantization type` errors at load, no more dequant-then-cuBLAS fallback for these types.
+
+## Non-goals
+
+- Q8_K (intermediate type used internally for MMQ, not stored in production GGUFs).
+- IQ1_M (rarer 1.5-bit variant; can be added later if demand surfaces).
+- Imatrix (importance-matrix calibration) tooling — that's quant-time, not load-time.
+- Pre-V2 DeepSeek architecture (separate spec).
+- GGUF loaders for Mixtral / QwenMoe / GraniteMoe / Mamba3 (separate spec).
+- Vulkan / HIP backend extensions (deferred to other hardware).
+
+## Approach choice
+
+Hybrid: port-style for Q-family, design-style for IQ-family.
+
+**Q2_K** is structurally close to the existing `Q3_K`, `Q4_K`, `Q5_K`, `Q6_K` K-quants — same 256-element superblocks, same dispatch surface, linear `value = scale × signed_int + min` decode math. We port `quantized_gemv_q4_k_mmq` as the starting point and substitute the per-superblock decode body. Predictable.
+
+**IQ-family** uses signed-grid lookup encoding fundamentally unlike linear quantization. The dp4a inner loop pattern from K-quants does not transfer directly: each weight value is a codebook index, not a signed int8. Rather than translate llama.cpp's IQ kernels line-by-line (which would import their idioms wholesale), we design new PTX kernels using the same shape as our existing K-quant MMQ (4-rows-per-warp, 256-thread blocks, FP32 accum, FP16 store) but with a codebook-expansion prefix that fills the int8 weight tile in shared memory before dp4a fires. Codebook constants (the wire-format-defined int8/int16 lookup tables) are copied verbatim from `ggml-quants.c` — they're a published spec.
+
+## Architecture overview
+
+### What gets added
+
+| Layer | New surface | Files touched |
+|---|---|---|
+| Core enum | 9 new entries in `QuantizationType` matching GGUF type IDs | `src/DotLLM.Core/Configuration/QuantizationType.cs` |
+| Block-size constants | `Q2_K_BlockBytes = 84`, `IQ2_XXS_BlockBytes = 66`, `IQ2_XS_BlockBytes = 74`, `IQ2_S_BlockBytes = 82`, `IQ3_XXS_BlockBytes = 98`, `IQ3_S_BlockBytes = 110`, `IQ4_NL_BlockBytes = 18`, `IQ4_XS_BlockBytes = 136`, `IQ1_S_BlockBytes = 50` | `src/DotLLM.Cpu/Kernels/Dequantize.cs` |
+| CPU dequant impl | `DequantizeQ2_K` in existing K-quant file; 8 IQ scalar impls in new file; codebooks in new file | `DequantizeKQuants.cs` (extend), `DequantizeIQuants.cs` (new), `IQuantGrids.cs` (new) |
+| GGUF reader | `Enum.IsDefined` accepts the new IDs once added to enum. Zero changes. | none |
+| GPU dequant | `dequant_q2_k_f16` in existing file; 8 IQ kernels + codebook `__device__ __constant__` arrays in new file | `dequant.cu` (extend), `dequant_iquants.cu` (new) |
+| GPU GEMV (per-call) | `quantized_gemv_q2_k`; 8 IQ GEMV kernels | `quantized_gemv.cu` (extend), `quantized_gemv_iquants.cu` (new) |
+| GPU MMQ + MMVQ-large + pre-Q8_1 variants | Full quartet per quant (mmq, mmq_preq, mmvq_large, mmvq_large_preq), 36 kernel functions across 9 quants | `quantized_gemv_mmq.cu` (extend for Q2_K), `quantized_gemv_iquants_mmq.cu` (new for IQ) |
+| GPU grouped-MoE-GEMV | One per quant: 9 new kernels | `moe_grouped_gemv.cu` (extend for Q2_K), `moe_grouped_gemv_iquants.cu` (new for IQ) |
+| `HasQuantizedGemv` / `HasMmq*` / `HasMoeGroupedGemv` / `MinKAlignmentFor` | All extended; IQ4_NL takes block-32 path (alignment 32), all others 256 | `CudaKernels.cs` |
+
+### File-organization rule
+
+Q-family extensions go into existing files (they're idiomatically siblings). IQ-family gets a new sibling file in each pair (`dequant_iquants.cu`, `quantized_gemv_iquants.cu`, `quantized_gemv_iquants_mmq.cu`, `moe_grouped_gemv_iquants.cu`) — keeps codebook-aware inner-loop code separate from linear-quant code, and PTX modules stay reasonably sized.
+
+### Codebook strategy
+
+Single source of truth in `src/DotLLM.Cpu/Kernels/IQuantGrids.cs` as `static readonly` arrays. Identical byte representation in PTX modules via `__device__ __constant__` arrays in `dequant_iquants.cu` (compiled into the GPU module's `.const` segment at PTX load). Total codebook footprint: ~30 KB on each side, well under the 64 KB CUDA constant-memory cache limit.
+
+A unit test (`IQuantGridsByteEquivalenceTest`) reads each PTX `__device__ __constant__` array via `cuModuleGetGlobal` after PTX load, copies bytes back to host, asserts byte-for-byte equality with the C# `static readonly` array. Diverging copies fail loud — catches the day someone updates one but not the other.
+
+### Dispatch flow
+
+```
+GGUF load → QuantizationType (extended enum)
+ → Dequantize.RowByteSize / ToFloat32 (extended switches)
+ → CPU oracle path (always works, scalar reference)
+ → GPU upload of raw quant bytes (zero-copy mmap → cuMemAlloc)
+ → forward pass:
+ if MoE → CudaMoeFfn.Forward → ProjectF32OrQuant
+ (gate uses MinKAlignmentFor, quant type from MoeLayerWeights)
+ if MLA → CudaMlaAttention.ForwardF16 → ProjectF16OrQuant (same)
+ else (GQA) → CudaGemm.LinearF32 with quant fast paths via existing dispatch
+```
+
+No new dispatch layers introduced — every new quant slots into the existing decision tree on the back of `HasQuantizedGemv(qt)`, `HasMmq(qt)`, `HasMoeGroupedGemv(qt)`, `MinKAlignmentFor(qt)`. New code goes in switch cases + new kernel functions, not in new abstractions.
+
+## Component detail
+
+### CPU dequant
+
+Per-quant scalar implementations that decode raw bytes → F32. These are the canonical math reference; every GPU kernel parity test compares against them.
+
+**Block layouts** (verified against `ggml-quants.h`):
+
+| Quant | Block size | Block bytes | Layout |
+|---|---|---|---|
+| Q2_K | 256 | 84 | scales[16] (4-bit scale + 4-bit min coef per sub-block, packed) + qs[64] (2-bit elements) + d (half) + dmin (half) |
+| IQ2_XXS | 256 | 66 | d + qs[32] (uint16; 256-entry codebook) |
+| IQ2_XS | 256 | 74 | d + qs[32] + scales[8] (4-bit packed) |
+| IQ2_S | 256 | 82 | d + qs[64] + qh[8] + scales[8] |
+| IQ3_XXS | 256 | 98 | d + qs[96] |
+| IQ3_S | 256 | 110 | d + qs[64] + qh[8] + signs[32] + scales[4] |
+| IQ4_NL | **32** | 18 | d + qs[16] (4-bit elements; 16-entry signed lookup) |
+| IQ4_XS | 256 | 136 | d + scales_h[2] + scales_l[4] + qs[128] |
+| IQ1_S | 256 | 50 | d + qs[32] + qh[8] (16-bit) |
+
+**Decode math** (Q-family vs IQ-family):
+
+- **Q2_K**: per-element `value = d × (scales[sub] & 0xF) × q_2bit − dmin × ((scales[sub] >> 4) & 0xF)`. 16 sub-blocks of 16 elements; 4-bit scale + 4-bit min per sub-block, both pre-quantized.
+- **IQ4_NL**: per-element `value = d × kvalues_iq4nl[qs_4bit]` where `kvalues_iq4nl[16]` is a signed-int8 lookup table baked at compile time.
+- **IQ4_XS**: like IQ4_NL but with 8 sub-blocks of 32 elements each, 6-bit scale per sub-block (6-bit = low 4 from `scales_l` + high 2 from `scales_h`).
+- **IQ2_XXS / IQ2_XS / IQ2_S**: codebook-indexed. Each `qs` uint16 indexes into `iq2*_grid[][8]`, producing 8 packed-int8 values; sign bits unpack from the same uint16's high bits or a separate `signs[]` byte. `value = d × scale_sub × signed_grid_value`.
+- **IQ3_XXS / IQ3_S**: same pattern as IQ2_S but with 256/512-entry int32 codebook (`iq3xxs_grid` produces 4 packed-int8 values per index).
+- **IQ1_S**: 2048-entry int16 codebook (`iq1s_grid`) with embedded sign info; sub-block delta from `qh`. Most exotic — most likely to surprise during implementation.
+
+**Codebook storage** (`IQuantGrids.cs`). The bracketed `[/* N values */]` comments below are spec-display abbreviations — the actual file contains the full literal arrays copied verbatim from `ggml/src/ggml-quants.c` (specifically the `iq2xxs_grid`, `iq2xs_grid`, `iq2s_grid`, `iq3xxs_grid`, `iq3s_grid`, `iq1s_grid`, `kvalues_iq4nl`, `ksigns_iq2xs` constants). A header comment in the file pins the upstream source URL and the `ggml-quants.c` revision SHA at copy-time.
+
+```csharp
+internal static class IQuantGrids
+{
+ // 256 × uint64. Each uint64 packs 8 signed int8 grid values.
+ internal static readonly ulong[] Iq2XxsGrid = [/* 256 values from ggml-quants.c */];
+
+ // 512 × uint64.
+ internal static readonly ulong[] Iq2XsGrid = [/* 512 */];
+
+ // 1024 × uint64.
+ internal static readonly ulong[] Iq2SGrid = [/* 1024 */];
+
+ // 256 × uint32.
+ internal static readonly uint[] Iq3XxsGrid = [/* 256 */];
+
+ // 512 × uint32.
+ internal static readonly uint[] Iq3SGrid = [/* 512 */];
+
+ // 2048 × uint16.
+ internal static readonly ushort[] Iq1SGrid = [/* 2048 */];
+
+ // 16 × sbyte. Signed lookup for IQ4_NL/XS — full literal:
+ internal static readonly sbyte[] KvaluesIq4Nl = [-127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113];
+
+ // 128 × uint64. Sign-mask lookup for IQ2_XS/S, IQ3_XXS/S.
+ internal static readonly ulong[] KsignsIq2Xs = [/* 128 */];
+}
+```
+
+### GPU dequant kernels
+
+Each new quant gets a `dequant__f16` PTX kernel matching the existing pattern: one CUDA block per superblock, 256 threads (or 32 for IQ4_NL), one element per thread, FP16 store at the end. IQ-family kernels read codebook constants from the `.const` segment populated at module load.
+
+Grid sizing: `min(totalSuperblocks, MaxDequantGridSize)` with grid-stride loops in the kernel body, matching the existing K-quant pattern.
+
+### GPU per-call GEMV
+
+Each new quant gets a `quantized_gemv_` kernel: one CUDA block per output row, 256 threads, threads iterate over superblocks per row, FP32 accumulation, single FP16 store via warp + block reduction.
+
+For Q2_K: structurally identical to `quantized_gemv_q4_k` with substituted per-superblock decode body.
+
+For IQ-family with codebook: each thread decodes its assigned codebook indices via `__constant__` reads, accumulates element-products from input × signed-codebook-value × per-sub-block-scale.
+
+For IQ4_NL (block-32): block iteration uses `k/32` instead of `k/256`; signed lookup via `kvalues_iq4nl[16]` is broadcast-friendly from constant memory.
+
+### GPU MMQ + MMVQ-large + Pre-Q8_1
+
+The fast-path kernels — dp4a-based int8×int8 fma chains consuming pre-Q8_1-quantized input scratch.
+
+**Q2_K MMQ**: port of `quantized_gemv_q4_k_mmq` with substituted per-superblock decode (16 sub-blocks of 16 elements vs Q4_K's 8 sub-blocks of 32; 4-bit scale + 4-bit min coef vs Q4_K's 6-bit packed scales). Inner dp4a stays identical because the int8 weight tile doesn't care how it was decoded.
+
+**IQ-family MMQ**: same overall block layout (4 rows × 32 elements per warp, 256 threads, dp4a inner loop) but the per-superblock decode prefix uses codebook lookup to **realize int8 values in the W_tile shared-memory array** before dp4a fires:
+
+```
+Per CUDA block, 4 rows × 32 elements (warp), 32 warps per block (256 threads):
+ 1. Each warp's 32 threads cooperatively decode 1 row × 32 elements:
+ - Read codebook indices from raw quant bytes
+ - Each thread does N constant-memory grid lookups (via __ldg or .const)
+ producing N × {4 or 8} signed int8 values
+ - Apply sign mask from ksigns_iq2xs (if applicable)
+ - Multiply by per-sub-block scale (FP32 staged before final accum)
+ - Pack 4 × int8 into one int32 in shared memory tile (W_tile[row][col])
+ 2. Pre-Q8_1 input scratch is already shaped as 4 × int32 lanes per element-tile;
+ read directly from scratch (no per-call re-quantize).
+ 3. dp4a fma loop: for each warp lane,
+ acc_lane[j] += __dp4a(W_tile[lane_row][k], X_tile[lane_col][k], 0)
+ across the K reduction.
+ 4. Apply per-block FP16 d × per-sub-block-scale at the end (FP32 → FP16 cast).
+ 5. Single FP16 store per output element.
+```
+
+The novelty over Q-family MMQ: step 1 has a codebook-expansion prefix, but the resulting int8 W_tile is structurally identical → step 3+ is byte-identical to existing K-quant MMQ inner loop. Step 1 is per-quant-specific; step 2-5 reuses the existing K-quant patterns.
+
+**MMVQ-large variants** (1 CUDA block per output row, multiple warps span K reduction) follow the same per-quant codebook-decode-then-dp4a pattern, triggered when M ≥ 1024 (existing threshold).
+
+**Pre-Q8_1 variants** consume the existing `quantize_x_to_q8_1` scratch (already exists for K-quants); the kernel saves a per-call re-quantize of the input. No new pre-Q8_1 input quantizer needed — the existing one quantizes whatever K is supplied (already handles K%32 alignment via blocks-per-row).
+
+**IQ4_XS** is a special case — its codebook is only 16 entries (`kvalues_iq4nl`), small enough to live entirely in registers. Single shared 16-entry signed-int8 lookup; no codebook scatter cost. Closer to a Q-family MMQ in cost than the IQ2/IQ3 codebook-heavy variants.
+
+### GPU grouped-MoE-GEMV
+
+Phase B fast path: collapse `K_active` per-expert GEMVs into a single launch where the FP16 input row is shared. Each new quant gets `moe_grouped_gemv__f16` mirroring the existing four (Q4_K, Q5_K, Q6_K, Q8_0).
+
+Grid: `(M, K_active, 1)`. Block: 256 threads. Each block computes one `(expert, output_row)` pair using the **same inner-loop body as the per-call `quantized_gemv_` kernel** — only difference is per-expert weight pointer is loaded from the `weights_ptrs[e]` device array.
+
+The grouped GEMV launcher's K-alignment check uses `MinKAlignmentFor(qt)`: 32 for IQ4_NL, 256 for everything else. This mirrors the gate fix from Round 15 (down_proj K=1408 unlock for Q8_0/Q5_0).
+
+```csharp
+public bool HasMoeGroupedGemv(QuantizationType qt) => qt switch {
+ Q4_K => _moeGroupedGemvQ4_KFunc != 0,
+ Q5_K => _moeGroupedGemvQ5_KFunc != 0,
+ Q6_K => _moeGroupedGemvQ6_KFunc != 0,
+ Q8_0 => _moeGroupedGemvQ8_0Func != 0,
+ Q2_K => _moeGroupedGemvQ2_KFunc != 0,
+ IQ2_XXS => _moeGroupedGemvIq2XxsFunc != 0,
+ IQ2_XS => _moeGroupedGemvIq2XsFunc != 0,
+ IQ2_S => _moeGroupedGemvIq2SFunc != 0,
+ IQ3_XXS => _moeGroupedGemvIq3XxsFunc != 0,
+ IQ3_S => _moeGroupedGemvIq3SFunc != 0,
+ IQ4_NL => _moeGroupedGemvIq4NlFunc != 0,
+ IQ4_XS => _moeGroupedGemvIq4XsFunc != 0,
+ IQ1_S => _moeGroupedGemvIq1SFunc != 0,
+ _ => false,
+};
+```
+
+`CudaMoeFfn.Forward`'s existing `useGrouped` decision logic is unchanged — it already keys on `kernels.HasMoeGroupedGemv(weights.GateProjQuantType)`. New quants are picked up automatically.
+
+## Testing strategy
+
+### Test pyramid (~190 new tests across 5 phases)
+
+```
+ ╱╲
+ ╱ ╲ Real-GGUF end-to-end smokes (~9 tests)
+ ╱────╲
+ ╱ ╲ Cross-kernel parity (~80 tests)
+ ╱────────╲ Grouped vs per-call MMQ vs MMVQ-large
+ ╱ ╲ per quant × 3-4 shapes
+ ╱────────────╲
+ ╱ ╲ Kernel correctness (~110 tests)
+ ╱ ╲ CPU + GPU dequant parity, GEMV
+ ╱──────────────────╲ vs scalar oracle, per quant × 3 shapes
+```
+
+Per-quant test inventory (multiplied by 9 quants):
+
+| Test class | Per-quant count | Total |
+|---|---|---|
+| `DequantizeQ2KTests` + `DequantizeIQuantsTests` (CPU) | ~3 | 27 |
+| GPU `LaunchDequantToF16` parity (extend `CudaKernelComparisonTests`) | ~3 shapes | 27 |
+| `CudaQuantizedGemvAlignmentTests` (extend) | ~3 shapes | 27 |
+| `CudaMmqKernelTests` MMQ parity (extend) | ~3 shapes | 27 |
+| `CudaMmqKernelTests` MMVQ-large parity (extend) | ~2 shapes | 18 |
+| `CudaMmqKernelTests` Pre-Q8_1 paths (extend) | ~2 shapes | 18 |
+| `CudaMoeGroupedGemvTests` (extend) | ~3-4 shapes | ~30 |
+| `IQuantGridsByteEquivalenceTest` (NEW) | 1 per IQ codebook | ~7 |
+| Real-GGUF end-to-end smoke per quant | 1 model each | 9 |
+| **Total new tests** | | **~190** |
+
+### Real-GGUF fixtures
+
+| Phase | Fixture (cached at `~/.dotllm/models/bartowski/`) | Disk |
+|---|---|---|
+| 1 | `DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf` (~5 GB) | 5 GB |
+| 2 | `Meta-Llama-3.1-8B-Instruct-IQ4_XS.gguf` (~4.5 GB) | 4.5 GB |
+| 3 | `Phi-3.5-mini-instruct-IQ3_M.gguf` (~2 GB) | 2 GB |
+| 4 | `Meta-Llama-3.1-8B-Instruct-IQ2_XS.gguf` (~3 GB) | 3 GB |
+| 5 | smallest IQ1_S available (~1.5 GB) or synthesize-only | 0-1.5 GB |
+| **Total** | | **~14-16 GB** |
+
+All gated `Skip.If(!File.Exists)` — fresh-clone CI green without fixtures. Disk impact fits on the existing E:\ junction.
+
+### Per-phase test gates
+
+| Phase | Gate |
+|---|---|
+| 1 (Q2_K) | All 27 Phase-1 tests green + V2-Lite Q2_K 27-layer smoke produces finite logits within 60 s (compares favorably to Round-13's Q3_K_M 31 s baseline since Q2_K is smaller). Existing 24 critical CUDA tests still green. CPU sweep still 0 failures. |
+| 2 (IQ4_NL/XS) | All Phase-1 still green + 50 new Phase-2 tests + Llama-3.1 IQ4_XS smoke produces finite logits + finite logits parity vs llama.cpp on a 32-token reference (within 5e-2 of llama.cpp's reference logits via JSON sidecar fixture). |
+| 3 (IQ3) | Phase 1+2 green + 50 new Phase-3 tests + Phi-3.5 IQ3_M smoke. |
+| 4 (IQ2) | Phases 1-3 green + 75 new Phase-4 tests + Llama-3.1 IQ2_XS smoke. |
+| 5 (IQ1_S) | All previous green + 25 new Phase-5 tests + (smoke optional if no real model cached). |
+
+### Existing-test discipline (regression bar)
+
+Every phase preserves these as a hard regression gate:
+- 24 critical CUDA tests (3 MLA + 12 grouped GEMV + 4 real-GGUF Q4_K_M smokes + 5 K=1408 alignment) — unchanged.
+- 27-layer V2-Lite Q3_K_M smoke — unchanged baseline (~31 s).
+- 1467+ CPU tests — unchanged.
+- Build: 0 warnings, 0 errors.
+
+### Llama.cpp parity sidecar (cross-validation, recommended)
+
+Per phase, a one-shot reference run produces a JSON sidecar:
+
+```json
+{
+ "model": "Meta-Llama-3.1-8B-Instruct-IQ4_XS",
+ "prompt": "Hello, how are",
+ "logits_first10_position3": [3.421, -1.105, ..., 0.892]
+}
+```
+
+Generated once via `llama-cli --logits` against the same GGUF, committed to `tests/DotLLM.Tests.Unit/Cuda/llama_cpp_reference/`. Test loads the GGUF + sidecar, runs same prompt through CUDA, asserts argmax-stable + max-abs-diff < 5e-2 on first-10 logits. Catches IQ kernel bugs that pass synthetic parity but produce drift on real attention/MLP composition.
+
+### CI runtime budget
+
+Worst-case full sweep grows from ~10 minutes (today) to ~15-20 minutes after Spec 1 lands all 5 phases (mostly real-GGUF smokes). Critical-path subset stays under 5 minutes.
+
+## Phasing
+
+### Per-phase deliverables
+
+| Phase | Quant types | Files (new/extended) | Tests added | Effort | Real-GGUF |
+|---|---|---|---|---|---|
+| **1: Q2_K** | Q2_K | `Dequantize.cs`, `DequantizeKQuants.cs`, `dequant.cu`, `quantized_gemv.cu`, `quantized_gemv_mmq.cu`, `moe_grouped_gemv.cu`, `CudaKernels.cs` (extend) | ~27 | ~1 week | V2-Lite Q2_K (5 GB) |
+| **2: IQ4_NL + IQ4_XS** | IQ4_NL, IQ4_XS | + new `DequantizeIQuants.cs`, `IQuantGrids.cs`, `dequant_iquants.cu`, `quantized_gemv_iquants.cu`, `quantized_gemv_iquants_mmq.cu`, `moe_grouped_gemv_iquants.cu` | ~50 | ~2 weeks | Llama-3.1 IQ4_XS (4.5 GB) |
+| **3: IQ3_S + IQ3_XXS** | IQ3_S, IQ3_XXS | extend IQ files: + IQ3 codebooks + 8 kernels (dequant/GEMV/MMQ/MMVQ-large × 2 quants) + 2 grouped | ~50 | ~2-3 weeks | Phi-3.5 IQ3_M (2 GB) |
+| **4: IQ2 family** | IQ2_S, IQ2_XS, IQ2_XXS | extend IQ files: + IQ2 codebooks (256/512/1024-entry grids) + sign-mask handling + 12 kernels + 3 grouped | ~75 | ~2-3 weeks | Llama-3.1 IQ2_XS (3 GB) |
+| **5: IQ1_S** | IQ1_S | extend IQ files: + IQ1 codebook (2048-entry int16 grid) + 1.5-bit-with-qh packing + 4 kernels + 1 grouped | ~25 | ~1-2 weeks | Optional (~1.5 GB) |
+
+**5 phases, 5 PRs, ~7-12 weeks calendar time, ~190 new tests, ~14-16 GB of fixture downloads.**
+
+### Phase ordering rationale
+
+- Phase 1 first because Q2_K is a Q-family port — easy infra warmup before IQ.
+- Phase 2 next because IQ4_NL/XS are the most-used IQ quants in production today (Llama-3.1 / Qwen2.5 IQ4_XS). Highest production value per unit of work.
+- Phases 3-4-5 in increasing codebook complexity: IQ3 is 256-entry grids; IQ2 adds full sign-mask scaffolding; IQ1 has the most exotic 2048-entry grid + qh-packed sub-block deltas.
+
+### Inter-phase dependencies
+
+- Phase 1 → none (independent).
+- Phase 2 introduces `DequantizeIQuants.cs`, `IQuantGrids.cs`, `dequant_iquants.cu`, `quantized_gemv_iquants*.cu`, `moe_grouped_gemv_iquants.cu` — these become the trunk for Phases 3-5 to extend.
+- Phases 3-5 only extend Phase 2's files. None of them touches Phase 1 work.
+
+## Error handling
+
+Per `CLAUDE.md` — error surfaces stay boundary-only:
+
+| Surface | Existing behavior | Spec-1 extension |
+|---|---|---|
+| Unknown GGUF type ID at load | `Enum.IsDefined` throws `InvalidDataException` | New IDs added to enum → no longer throws |
+| `NotSupportedException` from CPU `Dequantize.ToFloat32` | Already thrown for unknown types | Removed for the 9 new types |
+| `NotSupportedException` from `LaunchDequantToF16` (GPU) | Existing for unsupported types | Removed for the 9 new types |
+| K misaligned for grouped/MMQ | `ArgumentException` "must be multiple of 256" | Now uses per-quant `MinKAlignmentFor` (32 for IQ4_NL, 256 elsewhere) |
+| Stale PTX (function not in module) | `Has*` properties report `false`; dispatcher falls through to next-best path | Same pattern; new `Has*` properties for each new quant |
+| **NEW: codebook drift** between C#/PTX | n/a | `IQuantGridsByteEquivalenceTest` reads each PTX `__device__ __constant__` array via `cuModuleGetGlobal`, compares bytes to C# `static readonly`, fails loud |
+
+No new fallbacks, no new validation — the existing "unknown → throw with explicit message" pattern absorbs the new types.
+
+## Risks + mitigations
+
+1. **IQ-family MMQ correctness pain** — IQ kernels in llama.cpp had multiple rounds of post-merge bugfixes (sign-bit unpacking, sub-block scale extraction, qh-byte-pair edge cases). Mitigations:
+ - Hand-calc parity test on one block per IQ quant (deterministic; catches bit-pack mistakes early).
+ - Bulk parity vs the C# scalar `ToFloat32` (single source of truth).
+ - Optional sidecar logits parity vs llama.cpp on a real model per phase (5e-2 max-abs-diff).
+
+2. **PTX module bloat** — adding new kernels grows total PTX size from ~50 KB to ~150 KB. New kernel function count: 9 dequant + 9 per-call GEMV + 9 MMQ + 9 MMQ_preq + ~7 MMVQ-large (skipped where M never reaches 1024 in practice, e.g. tiny IQ1_S models) + ~7 MMVQ-large_preq + 9 grouped-MoE-GEMV ≈ **~60 new kernel functions** across the 5 phases. Mitigation: PTX is per-file; existing `_quantizedGemvMmqModule`, `_dequantModule` patterns split kernels across modules; loading time scales linearly. Measured impact ~50-200 ms additional startup on cold load. Acceptable.
+
+3. **Real-GGUF availability** — bartowski catalogs evolve; HF revisions can change file SHAs. Mitigation: `Skip.If(!File.Exists)` gates everything; per-test comment pins HF revision SHA at time of last-green run.
+
+4. **Codebook PTX `.const` segment limit** — total IQ codebook footprint ~30 KB in PTX `.const`, well under 64 KB CUDA constant-memory cache limit. No risk.
+
+5. **Pre-Q8_1 input quant extension** — the existing `quantize_x_to_q8_1` kernel quantizes whatever K is supplied (already handles K%32 via blocks-per-row). IQ4_NL at K%32 alignment uses the same pre-Q8_1 input scratch as Q4_K at K%256 — no new pre-Q8_1 kernel needed.
+
+6. **MMQ vs llama.cpp performance** — our MMQ kernels for K-quants today are slightly behind llama.cpp's at the same M/K (3-7%). IQ MMQ kernels designed from scratch may sit similarly behind. Mitigation: target "within 10% of llama.cpp at same M/K" rather than "match exactly." Acceptable for L3.
+
+7. **Dispatch-table fanout** — `LaunchQuantizedGemv` switch grows from 5 to 14 cases; `LaunchDequantToF16` from 11 to 20 cases. Pure switch dispatch, no runtime cost. Mitigation: split per-family helpers if any switch exceeds ~25 cases (none will after Spec 1).
+
+## Resolved questions (answered during brainstorm)
+
+| Q | Resolution |
+|---|---|
+| Coverage tier per quant? | Full L3 for all 9 (CPU dequant + GPU dequant + GEMV + MMQ + MMVQ-large + grouped-MoE-GEMV + pre-Q8_1 variants) |
+| Approach: port-from-llama.cpp vs design-from-scratch? | Hybrid (Approach C): port for Q-family, design for IQ-family with codebooks copied verbatim |
+| Spec / plan structure? | One spec (this doc), five plans, five PRs |
+| OK to add ~14-16 GB to local model cache? | Yes |
+| Generate llama.cpp reference logits sidecars? | Yes (one-time per phase, gated `Skip.If(!File.Exists)`) |
+| PR per phase or one PR with 5 commits? | PR per phase |
+| Phase 5 (IQ1_S) optional? | No — full L3 means all 9. Fixture is optional given rarity. |
+
+## Out of scope (explicit non-goals)
+
+- Q8_K (intermediate type used internally for MMQ accumulation, not stored in GGUFs).
+- IQ1_M (rarer 1.5-bit variant with sub-block scaling — can be added later).
+- Imatrix calibration tooling (quant-time concern, not load-time).
+- Pre-V2 DeepSeek architecture support (separate spec).
+- GGUF loaders for Mixtral / QwenMoe / GraniteMoe / Mamba3 (separate spec).
+- Vulkan / HIP backend extensions (deferred to other hardware).
+
+## Next step
+
+Once this spec is approved, invoke `superpowers:writing-plans` to produce `Plan 1.1 (Q2_K)` as the first implementation plan. Plans 1.2 - 1.5 follow as their preceding phase lands.
diff --git a/dotLLM.slnx b/dotLLM.slnx
index 22a7bf05..15b197ab 100644
--- a/dotLLM.slnx
+++ b/dotLLM.slnx
@@ -9,6 +9,7 @@
+
@@ -22,6 +23,7 @@
+
@@ -37,11 +39,13 @@
+
+
diff --git a/native/build.ps1 b/native/build.ps1
index 53960aed..e5ab3c49 100644
--- a/native/build.ps1
+++ b/native/build.ps1
@@ -2,7 +2,8 @@
# Requires: nvcc (CUDA Toolkit) on PATH
# Output: native\ptx\*.ptx
#
-# PTX is forward-compatible: compute_61 PTX runs on all GPUs from Pascal onward.
+# PTX is forward-compatible: compute_75 PTX runs on all GPUs from Turing onward.
+# CUDA 13 dropped Pascal (SM 6.x) and Volta (SM 7.0); Turing (SM 7.5) is the floor.
$ErrorActionPreference = "Stop"
@@ -12,7 +13,7 @@ $kernelDir = Join-Path $scriptDir "kernels"
if (-not (Test-Path $outDir)) { New-Item -ItemType Directory -Path $outDir | Out-Null }
-$arch = "compute_61"
+$arch = "compute_75"
Write-Host "Compiling CUDA kernels -> PTX (target: $arch)..."
diff --git a/native/build.sh b/native/build.sh
index 68143e3b..2859d895 100644
--- a/native/build.sh
+++ b/native/build.sh
@@ -3,8 +3,9 @@
# Requires: nvcc (CUDA Toolkit)
# Output: native/ptx/*.ptx
#
-# PTX is forward-compatible: compute_61 PTX runs on all GPUs from Pascal onward.
+# PTX is forward-compatible: compute_75 PTX runs on all GPUs from Turing onward.
# The CUDA driver JIT-compiles PTX → SASS for the specific GPU at load time.
+# CUDA 13 dropped Pascal (SM 6.x) and Volta (SM 7.0); Turing (SM 7.5) is the floor.
set -e
@@ -14,9 +15,9 @@ KERNEL_DIR="$SCRIPT_DIR/kernels"
mkdir -p "$OUT_DIR"
-# Target the lowest compute capability we support (Pascal / GTX 10xx).
-# The driver will JIT to the actual GPU's native ISA at load time.
-ARCH="compute_61"
+# Target the CUDA-13 floor (Turing / RTX 20xx). Driver JITs to the actual GPU's
+# native ISA at load time (Ampere / Ada / Hopper / Blackwell all derive from this).
+ARCH="compute_75"
# Kernels where --use_fast_math is safe (element-wise ops, no precision-sensitive math):
FAST_MATH_KERNELS="add add_f32 swiglu swiglu_f32 convert bias_add bias_add_f32 embedding embedding_f32out dequant quant_kv"
diff --git a/native/build_ptx.bat b/native/build_ptx.bat
new file mode 100644
index 00000000..5f942b20
--- /dev/null
+++ b/native/build_ptx.bat
@@ -0,0 +1,82 @@
+@echo off
+REM Build all CUDA kernels to PTX.
+REM Requires: %CUDA_PATH% set to a CUDA toolkit that supports the host MSVC
+REM (CUDA 13.x supports VS 2022/2026 MSVC; CUDA 11.8 does not).
+REM Usage: build_ptx.bat [arch] (default: compute_75)
+REM
+REM compute_75 = Turing, the CUDA 13 floor. PTX is forward-compatible so this
+REM runs on any Turing (SM 7.5), Ampere (8.0/8.6), Ada (8.9), Hopper (9.0),
+REM or Blackwell (10.0/12.0) GPU. CUDA 13 dropped Pascal/Volta (SM 6.x/7.0).
+setlocal EnableDelayedExpansion
+
+set ARCH=%1
+if "%ARCH%"=="" set ARCH=compute_75
+
+if not defined CUDA_PATH (
+ echo CUDA_PATH is not set. Install a CUDA toolkit and ensure CUDA_PATH points at it.
+ exit /b 1
+)
+set "NVCC=%CUDA_PATH%\bin\nvcc.exe"
+if not exist "%NVCC%" (
+ echo nvcc.exe not found at %NVCC%
+ exit /b 1
+)
+
+REM Locate a CUDA-compatible MSVC toolchain. Preference order:
+REM 1. VS 2022 Build Tools at E:\VS2022\BuildTools (MSVC 14.3x-14.4x — CUDA 13.x compatible)
+REM 2. VS 2026 Community at E:\Program Files\Microsoft Visual Studio\18\Community
+REM (MSVC 14.50 — _MSC_VER==1950, rejected by CUDA 13.1's host_config.h and by nvcc's OS-target check)
+REM A user can override by pre-setting MSVC_DIR before invoking this script.
+set "BT_ROOT=E:\VS2022\BuildTools"
+set "VS2026_ROOT=E:\Program Files\Microsoft Visual Studio\18\Community"
+if not defined MSVC_DIR if exist "%BT_ROOT%\VC\Tools\MSVC\" (
+ for /d %%D in ("%BT_ROOT%\VC\Tools\MSVC\*") do set "MSVC_DIR=%%D"
+)
+if not defined MSVC_DIR if exist "%VS2026_ROOT%\VC\Tools\MSVC\" (
+ for /d %%D in ("%VS2026_ROOT%\VC\Tools\MSVC\*") do set "MSVC_DIR=%%D"
+)
+if not defined MSVC_DIR (
+ echo Could not locate any VC Tools install. Install VS 2022 Build Tools to E:\VS2022\BuildTools.
+ exit /b 1
+)
+set "MSVC_BIN=%MSVC_DIR%\bin\Hostx64\x64"
+if not exist "%MSVC_BIN%\cl.exe" (
+ echo cl.exe not found at %MSVC_BIN%
+ exit /b 1
+)
+set "PATH=%MSVC_BIN%;%PATH%"
+
+set SCRIPT_DIR=%~dp0
+set KERNEL_DIR=%SCRIPT_DIR%kernels
+set OUT_DIR=%SCRIPT_DIR%ptx
+if not exist "%OUT_DIR%" mkdir "%OUT_DIR%"
+
+REM Kernels safe under --use_fast_math (elementwise; no expf/rsqrtf/sin/cos/pow):
+set "FAST_MATH=add add_f32 swiglu swiglu_f32 convert bias_add bias_add_f32 embedding embedding_f32out dequant quant_kv"
+
+echo Using nvcc: %NVCC%
+echo Compiling CUDA kernels -^> PTX (target: %ARCH%)...
+
+set FAIL=0
+for %%F in ("%KERNEL_DIR%\*.cu") do (
+ set "BASE=%%~nF"
+ set "FAST_FLAG="
+ for %%M in (%FAST_MATH%) do (
+ if /I "%%~nF"=="%%M" set "FAST_FLAG=--use_fast_math"
+ )
+ "%NVCC%" -ptx -arch=%ARCH% !FAST_FLAG! -allow-unsupported-compiler -o "%OUT_DIR%\!BASE!.ptx" "%%F"
+ if errorlevel 1 (
+ echo FAILED: %%~nxF
+ set FAIL=1
+ ) else (
+ if defined FAST_FLAG (
+ echo %%~nxF -^> !BASE!.ptx ^(fast_math^)
+ ) else (
+ echo %%~nxF -^> !BASE!.ptx ^(precise^)
+ )
+ )
+)
+
+if "%FAIL%"=="1" exit /b 1
+echo Done. PTX files in %OUT_DIR%
+exit /b 0
diff --git a/native/hip/build.ps1 b/native/hip/build.ps1
new file mode 100644
index 00000000..eb602e1d
--- /dev/null
+++ b/native/hip/build.ps1
@@ -0,0 +1,41 @@
+# Compile all .hip kernels to code objects (.co) for dotLLM HIP backend.
+# Requires: hipcc (ROCm on Windows)
+# Output: native/hip/co/*.co
+#
+# hipcc --genco produces a fat code-object ELF containing bundled AMDGPU ISA
+# for each --offload-arch target. The HIP runtime picks the matching arch at
+# hipModuleLoadData time.
+
+$ErrorActionPreference = 'Stop'
+
+$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
+$OutDir = Join-Path $ScriptDir 'co'
+$KernelDir = Join-Path $ScriptDir 'kernels'
+
+New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
+
+# Default architectures: RDNA2 (gfx1030, RX 6000), RDNA3 (gfx1100, RX 7000).
+# Override with $env:HIP_ARCHS, e.g. "gfx942 gfx90a" for MI300/MI250.
+$Archs = if ($env:HIP_ARCHS) { $env:HIP_ARCHS -split '\s+' } else { @('gfx1030', 'gfx1100') }
+
+$OffloadFlags = @()
+foreach ($a in $Archs) { $OffloadFlags += "--offload-arch=$a" }
+
+$hipcc = Get-Command hipcc -ErrorAction SilentlyContinue
+if (-not $hipcc) {
+ Write-Error 'hipcc not found in PATH. Install ROCm to build HIP kernels.'
+ exit 1
+}
+
+Write-Host "Compiling HIP kernels to code objects (archs: $($Archs -join ' '))..."
+
+Get-ChildItem -Path $KernelDir -Filter '*.hip' | ForEach-Object {
+ $src = $_.FullName
+ $name = [System.IO.Path]::GetFileNameWithoutExtension($_.Name)
+ $out = Join-Path $OutDir "$name.co"
+ Write-Host " hipcc --genco $($OffloadFlags -join ' ') -o $out $src"
+ & hipcc --genco @OffloadFlags -o $out $src
+ if ($LASTEXITCODE -ne 0) { throw "hipcc failed for $src" }
+}
+
+Write-Host "Done. Code objects in $OutDir"
diff --git a/native/hip/build.sh b/native/hip/build.sh
new file mode 100644
index 00000000..a13ca2e0
--- /dev/null
+++ b/native/hip/build.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+# Compile all .hip kernels to code objects (.co) for dotLLM HIP backend.
+# Requires: hipcc (ROCm)
+# Output: native/hip/co/*.co
+#
+# hipcc --genco produces a fat code-object ELF containing bundled AMDGPU ISA for
+# each --offload-arch target. The HIP runtime picks the matching arch at
+# hipModuleLoadData time.
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+OUT_DIR="$SCRIPT_DIR/co"
+KERNEL_DIR="$SCRIPT_DIR/kernels"
+
+mkdir -p "$OUT_DIR"
+
+# Default architectures: RDNA2 (gfx1030, RX 6000), RDNA3 (gfx1100, RX 7000).
+# Override with HIP_ARCHS env var, e.g. HIP_ARCHS="gfx942 gfx90a" for MI300/MI250.
+ARCHS="${HIP_ARCHS:-gfx1030 gfx1100}"
+
+OFFLOAD_FLAGS=""
+for a in $ARCHS; do
+ OFFLOAD_FLAGS="$OFFLOAD_FLAGS --offload-arch=$a"
+done
+
+if ! command -v hipcc >/dev/null 2>&1; then
+ echo "hipcc not found in PATH. Install ROCm to build HIP kernels." >&2
+ exit 1
+fi
+
+echo "Compiling HIP kernels → code objects (archs: $ARCHS)..."
+
+for src in "$KERNEL_DIR"/*.hip; do
+ [ -e "$src" ] || continue
+ name="$(basename "$src" .hip)"
+ out="$OUT_DIR/$name.co"
+ echo " hipcc --genco $OFFLOAD_FLAGS -o $out $src"
+ hipcc --genco $OFFLOAD_FLAGS -o "$out" "$src"
+done
+
+echo "Done. Code objects in $OUT_DIR"
diff --git a/native/hip/co/.gitkeep b/native/hip/co/.gitkeep
new file mode 100644
index 00000000..e69de29b
diff --git a/native/hip/co/rmsnorm.co b/native/hip/co/rmsnorm.co
new file mode 100644
index 00000000..e51be081
Binary files /dev/null and b/native/hip/co/rmsnorm.co differ
diff --git a/native/hip/kernels/rmsnorm.hip b/native/hip/kernels/rmsnorm.hip
new file mode 100644
index 00000000..30adbc6b
--- /dev/null
+++ b/native/hip/kernels/rmsnorm.hip
@@ -0,0 +1,113 @@
+// RMS Normalization kernel for dotLLM — HIP port of native/kernels/rmsnorm.cu and rmsnorm_f32.cu.
+// out[i] = (input[i] / rms) * weight[i], rms = sqrt(mean(input^2) + eps)
+// One block per row, wavefront shuffle reduction.
+//
+// Mechanical hipify translation: cuda_fp16.h -> hip/hip_fp16.h, __shfl_down_sync works
+// on HIP with the same signature. __syncthreads() is unchanged. warpSize is the
+// wavefront size (32 on RDNA in wave32 mode / CDNA, 64 on older/wave64).
+
+#include
+#include
+
+// FP32 variant — matches CPU reference exactly for the scaffold test.
+extern "C" __global__ void __launch_bounds__(256) rmsnorm_f32(
+ const float* __restrict__ input,
+ const float* __restrict__ weight,
+ float* __restrict__ output,
+ const int n,
+ const float eps)
+{
+ const int row = blockIdx.x;
+ const float* x = input + (size_t)row * n;
+ float* y = output + (size_t)row * n;
+
+ // Step 1: sum of squares, FP32 accumulation
+ float sum_sq = 0.0f;
+ for (int i = threadIdx.x; i < n; i += blockDim.x)
+ {
+ float v = x[i];
+ sum_sq += v * v;
+ }
+
+ // Wavefront-level reduction. On AMD, warpSize is 32 (wave32) or 64 (wave64).
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ sum_sq += __shfl_down(sum_sq, offset);
+
+ // Cross-wavefront reduction via shared memory. Oversize to 64 slots so wave64
+ // GPUs with 256-thread blocks (= 4 waves) still fit; wave32 fits trivially.
+ __shared__ float warp_sums[64];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0)
+ warp_sums[warp_id] = sum_sq;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ sum_sq = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ sum_sq += __shfl_down(sum_sq, offset);
+ }
+
+ __shared__ float rms_inv;
+ if (threadIdx.x == 0)
+ rms_inv = rsqrtf(sum_sq / (float)n + eps);
+ __syncthreads();
+
+ // Step 2: normalize and scale
+ for (int i = threadIdx.x; i < n; i += blockDim.x)
+ y[i] = x[i] * rms_inv * weight[i];
+}
+
+// FP16 variant — kept for parity with the CUDA kernel catalog.
+extern "C" __global__ void __launch_bounds__(256) rmsnorm_f16(
+ const __half* __restrict__ input,
+ const __half* __restrict__ weight,
+ __half* __restrict__ output,
+ const int n,
+ const float eps)
+{
+ const int row = blockIdx.x;
+ const __half* x = input + (size_t)row * n;
+ __half* y = output + (size_t)row * n;
+
+ float sum_sq = 0.0f;
+ for (int i = threadIdx.x; i < n; i += blockDim.x)
+ {
+ float v = __half2float(x[i]);
+ sum_sq += v * v;
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ sum_sq += __shfl_down(sum_sq, offset);
+
+ __shared__ float warp_sums[64];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0)
+ warp_sums[warp_id] = sum_sq;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ sum_sq = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ sum_sq += __shfl_down(sum_sq, offset);
+ }
+
+ __shared__ float rms_inv;
+ if (threadIdx.x == 0)
+ rms_inv = rsqrtf(sum_sq / (float)n + eps);
+ __syncthreads();
+
+ for (int i = threadIdx.x; i < n; i += blockDim.x)
+ {
+ float v = __half2float(x[i]);
+ float w = __half2float(weight[i]);
+ y[i] = __float2half(v * rms_inv * w);
+ }
+}
diff --git a/native/kernels/attention.cu b/native/kernels/attention.cu
index 97d7bc44..46fa19c3 100644
--- a/native/kernels/attention.cu
+++ b/native/kernels/attention.cu
@@ -10,12 +10,34 @@
// 2. Parallel warp-shuffle reductions (not serial thread-0 scan)
// 3. Tiled online softmax — bounded shared memory O(TILE_KV + headDim)
// regardless of sequence length (no crash at long contexts)
+//
+// Two entry points share the body:
+// attention_f16 — scalar seq_kv / position_offset (eager / prefill).
+// attention_f16_dyn — reads seq_kv / position_offset from device pointers.
+// Used by CUDA-Graphs decode replay where launch params
+// are baked into the graph at instantiate time but the
+// KV length grows by 1 per replay. Zero extra sync —
+// host bumps a 4-byte cuMemcpyHtoD before each launch.
#include
#include
#define TILE_KV 256
+// Body shared by both entry points. Inlined into each.
+__device__ __forceinline__ void attention_f16_body(
+ const half* __restrict__ q,
+ const half* __restrict__ k,
+ const half* __restrict__ v,
+ half* __restrict__ output,
+ int seq_q,
+ int seq_kv,
+ int num_heads,
+ int num_kv_heads,
+ int head_dim,
+ int position_offset,
+ int sliding_window);
+
extern "C" __global__ void __launch_bounds__(256) attention_f16(
const half* __restrict__ q,
const half* __restrict__ k,
@@ -27,7 +49,46 @@ extern "C" __global__ void __launch_bounds__(256) attention_f16(
const int num_kv_heads,
const int head_dim,
const int position_offset,
- const int sliding_window) // 0 = no sliding window
+ const int sliding_window)
+{
+ attention_f16_body(q, k, v, output, seq_q, seq_kv, num_heads, num_kv_heads,
+ head_dim, position_offset, sliding_window);
+}
+
+// Graph-friendly entry point: seq_kv and position_offset are dereferenced from
+// 4-byte device buffers. Host increments these via cuMemcpyHtoD between
+// cuGraphLaunch calls (~1 µs vs 22 µs/launch on WDDM).
+extern "C" __global__ void __launch_bounds__(256) attention_f16_dyn(
+ const half* __restrict__ q,
+ const half* __restrict__ k,
+ const half* __restrict__ v,
+ half* __restrict__ output,
+ const int seq_q,
+ const int* __restrict__ seq_kv_ptr,
+ const int num_heads,
+ const int num_kv_heads,
+ const int head_dim,
+ const int* __restrict__ position_offset_ptr,
+ const int sliding_window)
+{
+ int seq_kv = seq_kv_ptr[0];
+ int position_offset = position_offset_ptr[0];
+ attention_f16_body(q, k, v, output, seq_q, seq_kv, num_heads, num_kv_heads,
+ head_dim, position_offset, sliding_window);
+}
+
+__device__ __forceinline__ void attention_f16_body(
+ const half* __restrict__ q,
+ const half* __restrict__ k,
+ const half* __restrict__ v,
+ half* __restrict__ output,
+ int seq_q,
+ int seq_kv,
+ int num_heads,
+ int num_kv_heads,
+ int head_dim,
+ int position_offset,
+ int sliding_window)
{
int block_id = blockIdx.x;
int total_blocks = seq_q * num_heads;
diff --git a/native/kernels/attention_mla.cu b/native/kernels/attention_mla.cu
new file mode 100644
index 00000000..588a0ef0
--- /dev/null
+++ b/native/kernels/attention_mla.cu
@@ -0,0 +1,371 @@
+// Multi-head Latent Attention (MLA) Phase A naive forward kernel — FP32 + FP16.
+//
+// Implements the equivalent of CPU MlaAttention.Execute's attention loop.
+// Each block computes attention for one (query_token, head) pair, scanning
+// the cached K_nope (per-head), shared K_pe (broadcast), and V (per-head)
+// over [0..seqKv) with a causal mask.
+//
+// Two entry points share the same online-softmax algorithm:
+// attention_mla_f32 — F32 inputs/outputs (Phase A reference / debug).
+// attention_mla_f16 — F16 inputs/outputs, FP32 softmax accumulator.
+// Matches the GQA `attention_f16` precision pattern
+// (FP32 reduction, FP16 in/out for cache + bandwidth).
+//
+// Distinctive vs the standard FP32 attention kernel:
+// - Per-token Q is split into nope (qkNope) + rope (qkRope) sub-dims.
+// Q lives at [seqLen, numHeads, qkNope + qkRope] row-major. The kernel
+// reads both halves and dot-products against:
+// K_nope[s, h, :qkNope] (per-head, [seqKv, numHeads, qkNope])
+// K_pe[s, :qkRope] (shared across heads, [seqKv, qkRope])
+// - Per-head V dim (vHead) may differ from qkHead (DeepSeek-V2 typical:
+// qk_head_dim = 192, v_head_dim = 128). Output per (token, head) has
+// vHead floats.
+// - Softmax scale is the explicit "softmax_scale" passed in (callers
+// fold YaRN mscale^2 into it).
+//
+// Matches MlaAttention.Execute byte-for-byte algorithmically; FP32 throughout
+// for bit-near-equivalence with the CPU oracle.
+
+#include
+#include
+
+#define TILE_KV 128
+
+extern "C" __global__ void __launch_bounds__(128) attention_mla_f32(
+ const float* __restrict__ q, // [seqLen, numHeads, qkNope + qkRope]
+ const float* __restrict__ k_nope, // [seqKv, numHeads, qkNope]
+ const float* __restrict__ k_pe, // [seqKv, qkRope] (shared)
+ const float* __restrict__ v, // [seqKv, numHeads, vHead]
+ float* __restrict__ output, // [seqLen, numHeads, vHead]
+ const int seq_q,
+ const int seq_kv,
+ const int num_heads,
+ const int qk_nope_head_dim,
+ const int qk_rope_head_dim,
+ const int v_head_dim,
+ const int position_offset,
+ const float softmax_scale)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_q * num_heads) return;
+
+ int tq = block_id / num_heads;
+ int hq = block_id % num_heads;
+ int pos_q = position_offset + tq;
+
+ int qk_head_dim = qk_nope_head_dim + qk_rope_head_dim;
+ int q_stride = num_heads * qk_head_dim;
+ int k_nope_stride = num_heads * qk_nope_head_dim;
+ int v_stride = num_heads * v_head_dim;
+
+ // Shared layout:
+ // q_nope_shared [qk_nope_head_dim]
+ // q_pe_shared [qk_rope_head_dim]
+ // score_tile [TILE_KV]
+ // out_accum [v_head_dim]
+ // warp_scratch [32]
+ extern __shared__ float smem[];
+ float* q_nope_shared = smem;
+ float* q_pe_shared = q_nope_shared + qk_nope_head_dim;
+ float* score_tile = q_pe_shared + qk_rope_head_dim;
+ float* out_accum = score_tile + TILE_KV;
+ float* warp_scratch = out_accum + v_head_dim;
+
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ // Load Q for this (tq, hq) into shared.
+ const float* q_vec_nope = q + (size_t)tq * q_stride + hq * qk_head_dim;
+ const float* q_vec_pe = q_vec_nope + qk_nope_head_dim;
+ for (int d = threadIdx.x; d < qk_nope_head_dim; d += blockDim.x)
+ q_nope_shared[d] = q_vec_nope[d];
+ for (int d = threadIdx.x; d < qk_rope_head_dim; d += blockDim.x)
+ q_pe_shared[d] = q_vec_pe[d];
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ out_accum[d] = 0.0f;
+ __syncthreads();
+
+ float running_max = -FLT_MAX;
+ float running_sum = 0.0f;
+
+ for (int t_start = 0; t_start < seq_kv; t_start += TILE_KV)
+ {
+ int t_end = t_start + TILE_KV;
+ if (t_end > seq_kv) t_end = seq_kv;
+ int tile_len = t_end - t_start;
+
+ // Compute scores for tokens [t_start, t_end)
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x)
+ {
+ int tkv = t_start + t;
+ if (tkv > pos_q)
+ {
+ score_tile[t] = -FLT_MAX;
+ continue;
+ }
+
+ const float* k_nope_vec = k_nope
+ + (size_t)tkv * k_nope_stride
+ + hq * qk_nope_head_dim;
+ const float* k_pe_vec = k_pe + (size_t)tkv * qk_rope_head_dim;
+
+ float dot = 0.0f;
+ for (int d = 0; d < qk_nope_head_dim; d++)
+ dot += q_nope_shared[d] * k_nope_vec[d];
+ for (int d = 0; d < qk_rope_head_dim; d++)
+ dot += q_pe_shared[d] * k_pe_vec[d];
+
+ score_tile[t] = dot * softmax_scale;
+ }
+ __syncthreads();
+
+ // ── Tile max reduction ──
+ float tile_max = -FLT_MAX;
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x)
+ tile_max = fmaxf(tile_max, score_tile[t]);
+
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_max = fmaxf(tile_max, __shfl_down_sync(0xFFFFFFFF, tile_max, off));
+ if (lane == 0) warp_scratch[warp_id] = tile_max;
+ __syncthreads();
+ if (warp_id == 0) {
+ int nw = (blockDim.x + warpSize - 1) / warpSize;
+ tile_max = (lane < nw) ? warp_scratch[lane] : -FLT_MAX;
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_max = fmaxf(tile_max, __shfl_down_sync(0xFFFFFFFF, tile_max, off));
+ }
+ if (threadIdx.x == 0) warp_scratch[0] = tile_max;
+ __syncthreads();
+ tile_max = warp_scratch[0];
+
+ // ── Online softmax rescale ──
+ float new_max = fmaxf(running_max, tile_max);
+ // Avoid expf(very-negative) on the first tile when running_max == -FLT_MAX.
+ float correction = (running_max > -FLT_MAX + 1.0f)
+ ? expf(running_max - new_max) : 0.0f;
+ running_sum *= correction;
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ out_accum[d] *= correction;
+ running_max = new_max;
+ __syncthreads();
+
+ // ── Attention weights for this tile ──
+ float tile_sum = 0.0f;
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x) {
+ float w = (score_tile[t] > -FLT_MAX + 1.0f)
+ ? expf(score_tile[t] - running_max) : 0.0f;
+ score_tile[t] = w;
+ tile_sum += w;
+ }
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_sum += __shfl_down_sync(0xFFFFFFFF, tile_sum, off);
+ if (lane == 0) warp_scratch[warp_id] = tile_sum;
+ __syncthreads();
+ if (warp_id == 0) {
+ int nw = (blockDim.x + warpSize - 1) / warpSize;
+ tile_sum = (lane < nw) ? warp_scratch[lane] : 0.0f;
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_sum += __shfl_down_sync(0xFFFFFFFF, tile_sum, off);
+ if (lane == 0) warp_scratch[0] = tile_sum;
+ }
+ __syncthreads();
+ running_sum += warp_scratch[0];
+
+ // ── Accumulate weighted V ──
+ // Each thread handles a subset of v_head_dim, scanning the tile.
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x) {
+ float v_acc = 0.0f;
+ for (int t = 0; t < tile_len; t++) {
+ if (score_tile[t] > 0.0f) {
+ const float* v_vec = v
+ + (size_t)(t_start + t) * v_stride
+ + hq * v_head_dim;
+ v_acc += score_tile[t] * v_vec[d];
+ }
+ }
+ out_accum[d] += v_acc;
+ }
+ __syncthreads();
+ }
+
+ // Normalize and write
+ float sum_inv = (running_sum > 1e-10f) ? (1.0f / running_sum) : 0.0f;
+ float* out_vec = output + (size_t)tq * v_stride + hq * v_head_dim;
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ out_vec[d] = out_accum[d] * sum_inv;
+}
+
+// ── #region MLA FP16 ─────────────────────────────────────────────────────
+//
+// FP16 sibling of attention_mla_f32. Inputs (Q, K_nope, K_pe, V) and output
+// are __half; all reductions and the softmax accumulator run in FP32 for
+// numerical stability, matching the precision pattern used by the GQA
+// `attention_f16` kernel in attention.cu.
+//
+// Shared memory layout is identical to the F32 kernel (all FP32 scratch):
+// q_nope_shared [qk_nope_head_dim] FP32
+// q_pe_shared [qk_rope_head_dim] FP32
+// score_tile [TILE_KV] FP32
+// out_accum [v_head_dim] FP32
+// warp_scratch [32] FP32
+//
+// Q is widened to FP32 once on shared-memory load. K and V FP16 reads are
+// converted on the fly inside the inner dot products / V accumulation.
+
+#include
+
+extern "C" __global__ void __launch_bounds__(128) attention_mla_f16(
+ const half* __restrict__ q, // [seqLen, numHeads, qkNope + qkRope]
+ const half* __restrict__ k_nope, // [seqKv, numHeads, qkNope]
+ const half* __restrict__ k_pe, // [seqKv, qkRope] (shared)
+ const half* __restrict__ v, // [seqKv, numHeads, vHead]
+ half* __restrict__ output, // [seqLen, numHeads, vHead]
+ const int seq_q,
+ const int seq_kv,
+ const int num_heads,
+ const int qk_nope_head_dim,
+ const int qk_rope_head_dim,
+ const int v_head_dim,
+ const int position_offset,
+ const float softmax_scale)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_q * num_heads) return;
+
+ int tq = block_id / num_heads;
+ int hq = block_id % num_heads;
+ int pos_q = position_offset + tq;
+
+ int qk_head_dim = qk_nope_head_dim + qk_rope_head_dim;
+ int q_stride = num_heads * qk_head_dim;
+ int k_nope_stride = num_heads * qk_nope_head_dim;
+ int v_stride = num_heads * v_head_dim;
+
+ extern __shared__ float smem[];
+ float* q_nope_shared = smem;
+ float* q_pe_shared = q_nope_shared + qk_nope_head_dim;
+ float* score_tile = q_pe_shared + qk_rope_head_dim;
+ float* out_accum = score_tile + TILE_KV;
+ float* warp_scratch = out_accum + v_head_dim;
+
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ // Load Q for this (tq, hq) into shared (FP16 → FP32 widen on load).
+ const half* q_vec_nope = q + (size_t)tq * q_stride + hq * qk_head_dim;
+ const half* q_vec_pe = q_vec_nope + qk_nope_head_dim;
+ for (int d = threadIdx.x; d < qk_nope_head_dim; d += blockDim.x)
+ q_nope_shared[d] = __half2float(q_vec_nope[d]);
+ for (int d = threadIdx.x; d < qk_rope_head_dim; d += blockDim.x)
+ q_pe_shared[d] = __half2float(q_vec_pe[d]);
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ out_accum[d] = 0.0f;
+ __syncthreads();
+
+ float running_max = -FLT_MAX;
+ float running_sum = 0.0f;
+
+ for (int t_start = 0; t_start < seq_kv; t_start += TILE_KV)
+ {
+ int t_end = t_start + TILE_KV;
+ if (t_end > seq_kv) t_end = seq_kv;
+ int tile_len = t_end - t_start;
+
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x)
+ {
+ int tkv = t_start + t;
+ if (tkv > pos_q)
+ {
+ score_tile[t] = -FLT_MAX;
+ continue;
+ }
+
+ const half* k_nope_vec = k_nope
+ + (size_t)tkv * k_nope_stride
+ + hq * qk_nope_head_dim;
+ const half* k_pe_vec = k_pe + (size_t)tkv * qk_rope_head_dim;
+
+ float dot = 0.0f;
+ for (int d = 0; d < qk_nope_head_dim; d++)
+ dot += q_nope_shared[d] * __half2float(k_nope_vec[d]);
+ for (int d = 0; d < qk_rope_head_dim; d++)
+ dot += q_pe_shared[d] * __half2float(k_pe_vec[d]);
+
+ score_tile[t] = dot * softmax_scale;
+ }
+ __syncthreads();
+
+ // ── Tile max reduction ──
+ float tile_max = -FLT_MAX;
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x)
+ tile_max = fmaxf(tile_max, score_tile[t]);
+
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_max = fmaxf(tile_max, __shfl_down_sync(0xFFFFFFFF, tile_max, off));
+ if (lane == 0) warp_scratch[warp_id] = tile_max;
+ __syncthreads();
+ if (warp_id == 0) {
+ int nw = (blockDim.x + warpSize - 1) / warpSize;
+ tile_max = (lane < nw) ? warp_scratch[lane] : -FLT_MAX;
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_max = fmaxf(tile_max, __shfl_down_sync(0xFFFFFFFF, tile_max, off));
+ }
+ if (threadIdx.x == 0) warp_scratch[0] = tile_max;
+ __syncthreads();
+ tile_max = warp_scratch[0];
+
+ // ── Online softmax rescale ──
+ float new_max = fmaxf(running_max, tile_max);
+ float correction = (running_max > -FLT_MAX + 1.0f)
+ ? expf(running_max - new_max) : 0.0f;
+ running_sum *= correction;
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ out_accum[d] *= correction;
+ running_max = new_max;
+ __syncthreads();
+
+ // ── Attention weights for this tile ──
+ float tile_sum = 0.0f;
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x) {
+ float w = (score_tile[t] > -FLT_MAX + 1.0f)
+ ? expf(score_tile[t] - running_max) : 0.0f;
+ score_tile[t] = w;
+ tile_sum += w;
+ }
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_sum += __shfl_down_sync(0xFFFFFFFF, tile_sum, off);
+ if (lane == 0) warp_scratch[warp_id] = tile_sum;
+ __syncthreads();
+ if (warp_id == 0) {
+ int nw = (blockDim.x + warpSize - 1) / warpSize;
+ tile_sum = (lane < nw) ? warp_scratch[lane] : 0.0f;
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_sum += __shfl_down_sync(0xFFFFFFFF, tile_sum, off);
+ if (lane == 0) warp_scratch[0] = tile_sum;
+ }
+ __syncthreads();
+ running_sum += warp_scratch[0];
+
+ // ── Accumulate weighted V (FP16 → FP32 on read) ──
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x) {
+ float v_acc = 0.0f;
+ for (int t = 0; t < tile_len; t++) {
+ if (score_tile[t] > 0.0f) {
+ const half* v_vec = v
+ + (size_t)(t_start + t) * v_stride
+ + hq * v_head_dim;
+ v_acc += score_tile[t] * __half2float(v_vec[d]);
+ }
+ }
+ out_accum[d] += v_acc;
+ }
+ __syncthreads();
+ }
+
+ // Normalize and write (FP32 → FP16 narrow on store).
+ float sum_inv = (running_sum > 1e-10f) ? (1.0f / running_sum) : 0.0f;
+ half* out_vec = output + (size_t)tq * v_stride + hq * v_head_dim;
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ out_vec[d] = __float2half(out_accum[d] * sum_inv);
+}
+// ── #endregion MLA FP16 ──────────────────────────────────────────────────
diff --git a/native/kernels/attention_mla_latent.cu b/native/kernels/attention_mla_latent.cu
new file mode 100644
index 00000000..33cd00da
--- /dev/null
+++ b/native/kernels/attention_mla_latent.cu
@@ -0,0 +1,287 @@
+// Multi-head Latent Attention (MLA) Phase B absorbed-attention kernel — FP32.
+//
+// Implements the equivalent of CPU MlaAttention.ExecuteLatent's attention loop:
+// score[h, t, s] = Q_absorbed[h, t] · c_kv[s] + Q_pe[h, t] · k_pe[s]
+// softmax over causal mask (s <= cachedLength + t == position_offset + t)
+// c_v_out[h, t] = Σ_s softmax · c_kv[s] (shape [kv_lora_rank])
+//
+// The key Phase B win: K and V are NOT materialised per head. Instead the
+// kernel reads the compact latent c_kv[seq_kv, kv_lora_rank] (one shared row
+// per token, no head dim) and the shared k_pe[seq_kv, qk_rope_head_dim] —
+// 8-16× smaller than Phase A's expanded per-head K_nope/V/K_pe cache.
+//
+// Q-side absorption (W_UK^T @ Q_nope) is precomputed by the caller into
+// Q_absorbed[seq_q, num_heads, kv_lora_rank]. The post-attention W_UV
+// expansion is also caller-side.
+//
+// Per-(query_token, head) one CUDA block. Block size 128. Online-softmax
+// loop tiled over seq_kv in TILE_KV-sized chunks, identical structure to
+// the Phase A kernel — only the per-tile dot product changes.
+//
+// F32 throughout for bit-near-equivalence with the CPU oracle.
+
+#include
+#include
+
+#define TILE_KV 128
+
+extern "C" __global__ void __launch_bounds__(128) attention_mla_latent_f32(
+ const float* __restrict__ q_absorbed, // [seq_q, num_heads, kv_lora_rank]
+ const float* __restrict__ q_pe, // [seq_q, num_heads, qk_rope_head_dim]
+ const float* __restrict__ c_kv, // [seq_kv, kv_lora_rank] (shared across heads)
+ const float* __restrict__ k_pe, // [seq_kv, qk_rope_head_dim] (shared across heads)
+ float* __restrict__ c_v_out, // [seq_q, num_heads, kv_lora_rank] (latent V output)
+ const int seq_q,
+ const int seq_kv,
+ const int num_heads,
+ const int kv_lora_rank,
+ const int qk_rope_head_dim,
+ const int position_offset,
+ const float softmax_scale)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_q * num_heads) return;
+
+ int tq = block_id / num_heads;
+ int hq = block_id % num_heads;
+ int pos_q = position_offset + tq;
+
+ int q_abs_stride = num_heads * kv_lora_rank;
+ int q_pe_stride = num_heads * qk_rope_head_dim;
+
+ // Shared layout:
+ // q_abs_shared [kv_lora_rank]
+ // q_pe_shared [qk_rope_head_dim]
+ // score_tile [TILE_KV]
+ // out_accum [kv_lora_rank] (latent-dim output)
+ // warp_scratch [32]
+ extern __shared__ float smem[];
+ float* q_abs_shared = smem;
+ float* q_pe_shared = q_abs_shared + kv_lora_rank;
+ float* score_tile = q_pe_shared + qk_rope_head_dim;
+ float* out_accum = score_tile + TILE_KV;
+ float* warp_scratch = out_accum + kv_lora_rank;
+
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ // Load Q_absorbed and Q_pe for this (tq, hq) into shared memory.
+ const float* q_abs_vec = q_absorbed + (size_t)tq * q_abs_stride + hq * kv_lora_rank;
+ const float* q_pe_vec = q_pe + (size_t)tq * q_pe_stride + hq * qk_rope_head_dim;
+ for (int d = threadIdx.x; d < kv_lora_rank; d += blockDim.x)
+ q_abs_shared[d] = q_abs_vec[d];
+ for (int d = threadIdx.x; d < qk_rope_head_dim; d += blockDim.x)
+ q_pe_shared[d] = q_pe_vec[d];
+ for (int d = threadIdx.x; d < kv_lora_rank; d += blockDim.x)
+ out_accum[d] = 0.0f;
+ __syncthreads();
+
+ float running_max = -FLT_MAX;
+ float running_sum = 0.0f;
+
+ for (int t_start = 0; t_start < seq_kv; t_start += TILE_KV)
+ {
+ int t_end = t_start + TILE_KV;
+ if (t_end > seq_kv) t_end = seq_kv;
+ int tile_len = t_end - t_start;
+
+ // ── Compute scores for tokens [t_start, t_end) ──
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x)
+ {
+ int tkv = t_start + t;
+ if (tkv > pos_q)
+ {
+ score_tile[t] = -FLT_MAX;
+ continue;
+ }
+
+ const float* c_kv_vec = c_kv + (size_t)tkv * kv_lora_rank;
+ const float* k_pe_vec = k_pe + (size_t)tkv * qk_rope_head_dim;
+
+ float dot = 0.0f;
+ for (int d = 0; d < kv_lora_rank; d++)
+ dot += q_abs_shared[d] * c_kv_vec[d];
+ for (int d = 0; d < qk_rope_head_dim; d++)
+ dot += q_pe_shared[d] * k_pe_vec[d];
+
+ score_tile[t] = dot * softmax_scale;
+ }
+ __syncthreads();
+
+ // ── Tile max reduction ──
+ float tile_max = -FLT_MAX;
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x)
+ tile_max = fmaxf(tile_max, score_tile[t]);
+
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_max = fmaxf(tile_max, __shfl_down_sync(0xFFFFFFFF, tile_max, off));
+ if (lane == 0) warp_scratch[warp_id] = tile_max;
+ __syncthreads();
+ if (warp_id == 0) {
+ int nw = (blockDim.x + warpSize - 1) / warpSize;
+ tile_max = (lane < nw) ? warp_scratch[lane] : -FLT_MAX;
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_max = fmaxf(tile_max, __shfl_down_sync(0xFFFFFFFF, tile_max, off));
+ }
+ if (threadIdx.x == 0) warp_scratch[0] = tile_max;
+ __syncthreads();
+ tile_max = warp_scratch[0];
+
+ // ── Online softmax rescale ──
+ float new_max = fmaxf(running_max, tile_max);
+ float correction = (running_max > -FLT_MAX + 1.0f)
+ ? expf(running_max - new_max) : 0.0f;
+ running_sum *= correction;
+ for (int d = threadIdx.x; d < kv_lora_rank; d += blockDim.x)
+ out_accum[d] *= correction;
+ running_max = new_max;
+ __syncthreads();
+
+ // ── Attention weights for this tile ──
+ float tile_sum = 0.0f;
+ for (int t = threadIdx.x; t < tile_len; t += blockDim.x) {
+ float w = (score_tile[t] > -FLT_MAX + 1.0f)
+ ? expf(score_tile[t] - running_max) : 0.0f;
+ score_tile[t] = w;
+ tile_sum += w;
+ }
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_sum += __shfl_down_sync(0xFFFFFFFF, tile_sum, off);
+ if (lane == 0) warp_scratch[warp_id] = tile_sum;
+ __syncthreads();
+ if (warp_id == 0) {
+ int nw = (blockDim.x + warpSize - 1) / warpSize;
+ tile_sum = (lane < nw) ? warp_scratch[lane] : 0.0f;
+ for (int off = warpSize / 2; off > 0; off >>= 1)
+ tile_sum += __shfl_down_sync(0xFFFFFFFF, tile_sum, off);
+ if (lane == 0) warp_scratch[0] = tile_sum;
+ }
+ __syncthreads();
+ running_sum += warp_scratch[0];
+
+ // ── Accumulate weighted c_kv into out_accum (latent-dim output) ──
+ // Each thread handles a subset of kv_lora_rank, scanning the tile.
+ // c_kv is shared across heads — same row for every head's block,
+ // good L2 reuse across the (h, t) blocks running concurrently.
+ for (int d = threadIdx.x; d < kv_lora_rank; d += blockDim.x) {
+ float v_acc = 0.0f;
+ for (int t = 0; t < tile_len; t++) {
+ if (score_tile[t] > 0.0f) {
+ const float* c_kv_vec = c_kv + (size_t)(t_start + t) * kv_lora_rank;
+ v_acc += score_tile[t] * c_kv_vec[d];
+ }
+ }
+ out_accum[d] += v_acc;
+ }
+ __syncthreads();
+ }
+
+ // ── Normalize and write the latent output ──
+ float sum_inv = (running_sum > 1e-10f) ? (1.0f / running_sum) : 0.0f;
+ float* out_vec = c_v_out + (size_t)tq * q_abs_stride + hq * kv_lora_rank;
+ for (int d = threadIdx.x; d < kv_lora_rank; d += blockDim.x)
+ out_vec[d] = out_accum[d] * sum_inv;
+}
+
+// ── Q absorption: Q_absorbed[h, t] = W_UK[h]^T @ Q_nope[h, t] ─────────────
+//
+// W_UK[h] is the per-head slice of kv_b_proj: rows [h * (qkNope + vHead) ..
+// h * (qkNope + vHead) + qkNope) of the full row-major kv_b_proj weight,
+// each row being kv_lora_rank wide.
+// W_UK[h][j][k] = kv_b_proj[(h * (qkNope + vHead) + j) * kv_lora_rank + k]
+//
+// Q_absorbed[h, t][k] = Σ_j W_UK[h][j][k] · Q_nope[h, t][j]
+//
+// Q layout: [seq_q, num_heads, qkNope + qkRope] — Q_nope[h, t] sits at
+// offset t * (num_heads * qkHead) + h * qkHead, length qkNope.
+// Output layout: [seq_q, num_heads, kv_lora_rank].
+//
+// Grid: blocks = seq_q * num_heads, each block handles one (t, h) and
+// computes kv_lora_rank outputs. Block size 128, threads cooperate on
+// emitting the kv_lora_rank-wide output via a strided loop.
+extern "C" __global__ void __launch_bounds__(128) mla_q_absorb_uk_f32(
+ const float* __restrict__ q, // [seq_q, num_heads, qkNope + qkRope]
+ const float* __restrict__ kv_b_proj, // [num_heads * (qkNope + vHead), kv_lora_rank]
+ float* __restrict__ q_absorbed, // [seq_q, num_heads, kv_lora_rank]
+ const int seq_q,
+ const int num_heads,
+ const int qk_nope_head_dim,
+ const int qk_rope_head_dim,
+ const int v_head_dim,
+ const int kv_lora_rank)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_q * num_heads) return;
+
+ int t = block_id / num_heads;
+ int h = block_id % num_heads;
+
+ int qk_head_dim = qk_nope_head_dim + qk_rope_head_dim;
+ int per_head_kvb_out = qk_nope_head_dim + v_head_dim;
+ int q_stride = num_heads * qk_head_dim;
+ int q_abs_stride = num_heads * kv_lora_rank;
+
+ // Q_nope[h, t] — first qk_nope_head_dim of the (t, h) head block.
+ const float* q_nope = q + (size_t)t * q_stride + h * qk_head_dim;
+ // W_UK[h] starts at row (h * per_head_kvb_out), spans qk_nope_head_dim rows.
+ const float* w_uk_base = kv_b_proj + (size_t)(h * per_head_kvb_out) * kv_lora_rank;
+ // Output Q_absorbed[h, t].
+ float* q_abs = q_absorbed + (size_t)t * q_abs_stride + h * kv_lora_rank;
+
+ for (int k = threadIdx.x; k < kv_lora_rank; k += blockDim.x)
+ {
+ float acc = 0.0f;
+ for (int j = 0; j < qk_nope_head_dim; j++)
+ acc += w_uk_base[(size_t)j * kv_lora_rank + k] * q_nope[j];
+ q_abs[k] = acc;
+ }
+}
+
+// ── V expansion: out[h, t] = W_UV[h] @ c_v_out[h, t] ─────────────────────
+//
+// W_UV[h] is the per-head slice of kv_b_proj: rows [h * (qkNope + vHead) +
+// qkNope .. h * (qkNope + vHead) + qkNope + vHead) of kv_b_proj.
+// W_UV[h][v][k] = kv_b_proj[(h * (qkNope + vHead) + qkNope + v) * kv_lora_rank + k]
+//
+// out[h, t][v] = Σ_k W_UV[h][v][k] · c_v_out[h, t][k]
+//
+// c_v_out layout: [seq_q, num_heads, kv_lora_rank].
+// Output layout: [seq_q, num_heads, v_head_dim] (= attention output before
+// o_proj — same shape Phase A's attn_out has).
+//
+// Grid: blocks = seq_q * num_heads, each block emits v_head_dim outputs.
+extern "C" __global__ void __launch_bounds__(128) mla_v_expand_uv_f32(
+ const float* __restrict__ c_v_out, // [seq_q, num_heads, kv_lora_rank]
+ const float* __restrict__ kv_b_proj, // [num_heads * (qkNope + vHead), kv_lora_rank]
+ float* __restrict__ attn_out, // [seq_q, num_heads, v_head_dim]
+ const int seq_q,
+ const int num_heads,
+ const int qk_nope_head_dim,
+ const int v_head_dim,
+ const int kv_lora_rank)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_q * num_heads) return;
+
+ int t = block_id / num_heads;
+ int h = block_id % num_heads;
+
+ int per_head_kvb_out = qk_nope_head_dim + v_head_dim;
+ int c_v_stride = num_heads * kv_lora_rank;
+ int out_stride = num_heads * v_head_dim;
+
+ const float* c_v_in = c_v_out + (size_t)t * c_v_stride + h * kv_lora_rank;
+ // W_UV[h] starts at row (h * per_head_kvb_out + qk_nope_head_dim).
+ const float* w_uv_base = kv_b_proj
+ + (size_t)(h * per_head_kvb_out + qk_nope_head_dim) * kv_lora_rank;
+ float* out_vec = attn_out + (size_t)t * out_stride + h * v_head_dim;
+
+ for (int v = threadIdx.x; v < v_head_dim; v += blockDim.x)
+ {
+ float acc = 0.0f;
+ const float* w_row = w_uv_base + (size_t)v * kv_lora_rank;
+ for (int k = 0; k < kv_lora_rank; k++)
+ acc += w_row[k] * c_v_in[k];
+ out_vec[v] = acc;
+ }
+}
diff --git a/native/kernels/dequant.cu b/native/kernels/dequant.cu
index a52a90fc..e73da472 100644
--- a/native/kernels/dequant.cu
+++ b/native/kernels/dequant.cu
@@ -107,6 +107,159 @@ extern "C" __global__ void __launch_bounds__(256) dequant_q5_0_f16(
}
}
+// ── Q4_1: 20 bytes per 32 values ────────────────────────────────────
+// struct block_q4_1 { half d; half m; uint8_t qs[16]; };
+// value = d * nibble + m (unsigned + min, vs Q4_0's signed-after-bias-of-8)
+#define Q4_1_BLOCK_SIZE 32
+#define Q4_1_BLOCK_BYTES 20
+
+extern "C" __global__ void __launch_bounds__(256) dequant_q4_1_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_blocks)
+{
+ int lane = threadIdx.x % Q4_1_BLOCK_SIZE;
+ int warp_in_block = threadIdx.x / Q4_1_BLOCK_SIZE;
+ int warps_per_grid = (gridDim.x * blockDim.x) / Q4_1_BLOCK_SIZE;
+ int start_block = blockIdx.x * (blockDim.x / Q4_1_BLOCK_SIZE) + warp_in_block;
+
+ for (int block_idx = start_block; block_idx < total_blocks; block_idx += warps_per_grid)
+ {
+ const uint8_t* block = src + (size_t)block_idx * Q4_1_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ float m = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* qs = block + 4;
+
+ int byte_idx = lane / 2;
+ uint8_t packed = qs[byte_idx];
+ int val = (lane & 1) ? (int)(packed >> 4) : (int)(packed & 0x0F);
+
+ dst[(size_t)block_idx * Q4_1_BLOCK_SIZE + lane] = __float2half(d * (float)val + m);
+ }
+}
+
+// ── Q5_1: 24 bytes per 32 values ────────────────────────────────────
+// struct block_q5_1 { half d; half m; uint32_t qh; uint8_t qs[16]; };
+// value = d * ((qh_bit << 4) | nibble) + m (5-bit unsigned + min)
+#define Q5_1_BLOCK_SIZE 32
+#define Q5_1_BLOCK_BYTES 24
+
+extern "C" __global__ void __launch_bounds__(256) dequant_q5_1_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_blocks)
+{
+ int lane = threadIdx.x % Q5_1_BLOCK_SIZE;
+ int warp_in_block = threadIdx.x / Q5_1_BLOCK_SIZE;
+ int warps_per_grid = (gridDim.x * blockDim.x) / Q5_1_BLOCK_SIZE;
+ int start_block = blockIdx.x * (blockDim.x / Q5_1_BLOCK_SIZE) + warp_in_block;
+
+ for (int block_idx = start_block; block_idx < total_blocks; block_idx += warps_per_grid)
+ {
+ const uint8_t* block = src + (size_t)block_idx * Q5_1_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ float m = __half2float(*reinterpret_cast(block + 2));
+ // Read qh (4 bytes, may be unaligned).
+ unsigned int qh = (unsigned int)block[4] | ((unsigned int)block[5] << 8) |
+ ((unsigned int)block[6] << 16) | ((unsigned int)block[7] << 24);
+ const uint8_t* qs = block + 8;
+
+ int j = lane < 16 ? lane : lane - 16;
+ uint8_t packed = qs[j];
+ int nibble = (lane < 16) ? (packed & 0x0F) : (packed >> 4);
+ int high_bit = (qh >> lane) & 1;
+ int val = nibble | (high_bit << 4);
+
+ dst[(size_t)block_idx * Q5_1_BLOCK_SIZE + lane] = __float2half(d * (float)val + m);
+ }
+}
+
+// ── Q2_K: 84 bytes per 256 values ──────────────────────────────────
+// struct block_q2_K { uint8_t scales[16]; uint8_t qs[64]; half d; half dmin; };
+// - scales[i]: low nibble = sub-block i scale, high nibble = sub-block i dmin coef
+// - qs[i]: 2-bit elements packed 4 per byte (low-to-high)
+//
+// 256 threads/block, one element per thread, FP16 store.
+
+#define Q2_K_SUPER_BLOCK_SIZE 256
+#define Q2_K_BLOCK_BYTES 84
+
+extern "C" __global__ void __launch_bounds__(256) dequant_q2_k_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_superblocks)
+{
+ int t = threadIdx.x; // 0..255
+
+ for (int sb_idx = blockIdx.x; sb_idx < total_superblocks; sb_idx += gridDim.x)
+ {
+ const uint8_t* block = src + (size_t)sb_idx * Q2_K_BLOCK_BYTES;
+ const uint8_t* scales = block; // 16 bytes
+ const uint8_t* qs = block + 16; // 64 bytes
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ int sub = t >> 4; // t / 16
+ int byte_idx = t >> 2; // t / 4
+ int bit_off = (t & 0x3) << 1; // (t % 4) * 2
+ int q2 = (qs[byte_idx] >> bit_off) & 0x3;
+ int scale = scales[sub] & 0xF;
+ int dm_coef = (scales[sub] >> 4) & 0xF;
+
+ float result = d * (float)scale * (float)q2 - dmin * (float)dm_coef;
+ dst[(size_t)sb_idx * Q2_K_SUPER_BLOCK_SIZE + t] = __float2half(result);
+ }
+}
+
+// ── Q3_K: 110 bytes per 256 values (super-block with 16 sub-blocks of 16) ──
+// struct block_q3_K { uint8_t hmask[32]; uint8_t qs[64]; uint8_t scales[12]; half d; };
+// hmask: 1 high bit per element (32 × 8 = 256 bits)
+// qs: 2 low bits per element (64 × 4 = 256)
+// scales: 16 × 6-bit signed-after-bias-of-32 packed into 12 bytes
+// d: FP16 super-block delta
+// Per-element value: d × (signed_scale[sub]) × (((hmask_bit << 2) | qs_bits) - 4)
+#define Q3_K_SUPER_BLOCK_SIZE 256
+#define Q3_K_BLOCK_BYTES 110
+
+extern "C" __global__ void __launch_bounds__(256) dequant_q3_k_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_superblocks)
+{
+ int t = threadIdx.x; // 0..255
+
+ for (int sb_idx = blockIdx.x; sb_idx < total_superblocks; sb_idx += gridDim.x)
+ {
+ const uint8_t* block = src + (size_t)sb_idx * Q3_K_BLOCK_BYTES;
+ const uint8_t* hmask = block; // 32 bytes
+ const uint8_t* qs = block + 32; // 64 bytes
+ const uint8_t* scales12 = block + 32 + 64; // 12 bytes
+ float d = __half2float(*reinterpret_cast(block + 32 + 64 + 12));
+
+ // Sub-block index (0..15) for this thread; 16 threads share a sub-block
+ // (= 16 elements per sub-block).
+ int sub = t / 16;
+
+ // Unpack 6-bit scale for this sub-block from the 12 packed bytes:
+ // scales12[sub<8 ? sub : sub-4] → low nibble (sub<8) or high nibble (sub>=8)
+ // scales12[8 + sub/4] → high 2 bits at offset (sub%4)*2
+ int lowSrcByte = sub < 8 ? sub : sub - 4;
+ int lowNibble = sub < 8
+ ? (scales12[lowSrcByte] & 0x0F)
+ : ((scales12[lowSrcByte] >> 4) & 0x0F);
+ int hiBits = (scales12[8 + (sub >> 2)] >> ((sub & 3) * 2)) & 0x03;
+ int signedScale = (lowNibble | (hiBits << 4)) - 32; // [-32, 31]
+
+ // Per-element 3-bit unpacking.
+ int qBits = (qs[t >> 2] >> ((t & 3) * 2)) & 0x03;
+ int hBit = (hmask[t >> 3] >> (t & 7)) & 0x01;
+ int signed3 = ((hBit << 2) | qBits) - 4; // [-4, 3]
+
+ dst[(size_t)sb_idx * Q3_K_SUPER_BLOCK_SIZE + t] =
+ __float2half(d * (float)signedScale * (float)signed3);
+ }
+}
+
// ── Q4_K: 144 bytes per 256 values (super-block with 8 sub-blocks) ──
// struct block_q4_K { half d; half dmin; uint8_t scales[12]; uint8_t qs[128]; };
#define Q4_K_SUPER_BLOCK_SIZE 256
@@ -202,23 +355,65 @@ extern "C" __global__ void __launch_bounds__(256) dequant_q5_k_f16(
float scale = d * (float)sc;
float min_val = dmin * (float)m;
- const uint8_t* sub_qs = qs + sub * 16;
- const uint8_t* sub_qh = qh + sub * 4;
-
- // pos 0..31: interleaved low/high nibbles
- // sub_out[2*j+0]=lo, sub_out[2*j+1]=hi for j=0..15
- int j = pos / 2;
- uint8_t packed = sub_qs[j];
- int nibble = (pos & 1) ? (packed >> 4) : (packed & 0x0F);
+ int pair_idx = sub >> 1;
+ int nibble_half = sub & 1;
+ uint8_t packed = qs[pair_idx * 32 + pos];
+ int nibble = nibble_half ? (packed >> 4) : (packed & 0x0F);
- // Extract 5th bit from qh
- int bit = (sub_qh[j / 4] >> ((j % 4) * 2 + (pos & 1))) & 1;
+ // qh is indexed by element position; bit `sub` supplies the fifth bit.
+ int bit = (qh[pos] >> sub) & 1;
int val = nibble | (bit << 4);
dst[(size_t)sb_idx * Q5_K_SUPER_BLOCK_SIZE + t] = __float2half(scale * (float)val - min_val);
}
}
+extern "C" __global__ void __launch_bounds__(256) dequant_q5_k_f32(
+ const uint8_t* __restrict__ src,
+ float* __restrict__ dst,
+ const int total_superblocks)
+{
+ int t = threadIdx.x; // 0..255
+
+ for (int sb_idx = blockIdx.x; sb_idx < total_superblocks; sb_idx += gridDim.x)
+ {
+ const uint8_t* block = src + (size_t)sb_idx * Q5_K_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qh = block + 16;
+ const uint8_t* qs = block + 48;
+
+ int sub = t / 32;
+ int pos = t % 32;
+
+ int sc, m;
+ if (sub < 4)
+ {
+ sc = scales_raw[sub] & 0x3F;
+ m = scales_raw[sub + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sub + 4] & 0x0F) | ((scales_raw[sub - 4] >> 6) << 4);
+ m = (scales_raw[sub + 4] >> 4) | ((scales_raw[sub] >> 6) << 4);
+ }
+
+ float scale = d * (float)sc;
+ float min_val = dmin * (float)m;
+
+ int pair_idx = sub >> 1;
+ int nibble_half = sub & 1;
+ uint8_t packed = qs[pair_idx * 32 + pos];
+ int nibble = nibble_half ? (packed >> 4) : (packed & 0x0F);
+
+ int bit = (qh[pos] >> sub) & 1;
+ int val = nibble | (bit << 4);
+
+ dst[(size_t)sb_idx * Q5_K_SUPER_BLOCK_SIZE + t] = scale * (float)val - min_val;
+ }
+}
+
// ── Q6_K: 210 bytes per 256 values ──────────────────────────────────
// struct block_q6_K { uint8_t ql[128]; uint8_t qh[64]; int8_t scales[16]; half d; };
#define Q6_K_SUPER_BLOCK_SIZE 256
diff --git a/native/kernels/dequant_iquants.cu b/native/kernels/dequant_iquants.cu
new file mode 100644
index 00000000..145428ff
--- /dev/null
+++ b/native/kernels/dequant_iquants.cu
@@ -0,0 +1,122 @@
+// IQ-family dequantization kernels for dotLLM.
+// Layouts mirror ggml-common.h block_iq4_nl and block_iq4_xs.
+
+#include
+#include
+
+__device__ __constant__ int8_t kvalues_iq4nl_device[16] = {
+ -127, -104, -83, -65, -49, -35, -22, -10,
+ 1, 13, 25, 38, 53, 69, 89, 113
+};
+
+#define IQ4_NL_BLOCK_SIZE 32
+#define IQ4_NL_BLOCK_BYTES 18
+
+extern "C" __global__ void __launch_bounds__(256) dequant_iq4_nl_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_blocks)
+{
+ int t = threadIdx.x;
+ if (t >= IQ4_NL_BLOCK_SIZE) {
+ return;
+ }
+
+ for (int block_idx = blockIdx.x; block_idx < total_blocks; block_idx += gridDim.x) {
+ const uint8_t* block = src + (size_t)block_idx * IQ4_NL_BLOCK_BYTES;
+ const uint8_t* qs = block + 2;
+ float d = __half2float(*reinterpret_cast(block));
+
+ int j = t & 15;
+ uint8_t packed = qs[j];
+ int q = t < 16 ? (packed & 0x0F) : (packed >> 4);
+ dst[(size_t)block_idx * IQ4_NL_BLOCK_SIZE + t] =
+ __float2half(d * (float)kvalues_iq4nl_device[q]);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256) dequant_iq4_nl_f32(
+ const uint8_t* __restrict__ src,
+ float* __restrict__ dst,
+ const int total_blocks)
+{
+ int t = threadIdx.x;
+ if (t >= IQ4_NL_BLOCK_SIZE) {
+ return;
+ }
+
+ for (int block_idx = blockIdx.x; block_idx < total_blocks; block_idx += gridDim.x) {
+ const uint8_t* block = src + (size_t)block_idx * IQ4_NL_BLOCK_BYTES;
+ const uint8_t* qs = block + 2;
+ float d = __half2float(*reinterpret_cast(block));
+
+ int j = t & 15;
+ uint8_t packed = qs[j];
+ int q = t < 16 ? (packed & 0x0F) : (packed >> 4);
+ dst[(size_t)block_idx * IQ4_NL_BLOCK_SIZE + t] =
+ d * (float)kvalues_iq4nl_device[q];
+ }
+}
+
+#define IQ4_XS_SUPER_BLOCK_SIZE 256
+#define IQ4_XS_BLOCK_BYTES 136
+
+extern "C" __global__ void __launch_bounds__(256) dequant_iq4_xs_f16(
+ const uint8_t* __restrict__ src,
+ half* __restrict__ dst,
+ const int total_superblocks)
+{
+ int t = threadIdx.x; // 0..255
+
+ for (int sb_idx = blockIdx.x; sb_idx < total_superblocks; sb_idx += gridDim.x) {
+ const uint8_t* block = src + (size_t)sb_idx * IQ4_XS_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+
+ int ib = t >> 5;
+ int elem = t & 31;
+ int q_index = elem & 15;
+
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+
+ uint8_t packed = qs[ib * 16 + q_index];
+ int q = elem < 16 ? (packed & 0x0F) : (packed >> 4);
+ dst[(size_t)sb_idx * IQ4_XS_SUPER_BLOCK_SIZE + t] =
+ __float2half(dl * (float)kvalues_iq4nl_device[q]);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256) dequant_iq4_xs_f32(
+ const uint8_t* __restrict__ src,
+ float* __restrict__ dst,
+ const int total_superblocks)
+{
+ int t = threadIdx.x; // 0..255
+
+ for (int sb_idx = blockIdx.x; sb_idx < total_superblocks; sb_idx += gridDim.x) {
+ const uint8_t* block = src + (size_t)sb_idx * IQ4_XS_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+
+ int ib = t >> 5;
+ int elem = t & 31;
+ int q_index = elem & 15;
+
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+
+ uint8_t packed = qs[ib * 16 + q_index];
+ int q = elem < 16 ? (packed & 0x0F) : (packed >> 4);
+ dst[(size_t)sb_idx * IQ4_XS_SUPER_BLOCK_SIZE + t] =
+ dl * (float)kvalues_iq4nl_device[q];
+ }
+}
diff --git a/native/kernels/embedding.cu b/native/kernels/embedding.cu
index 0eb77610..f4d323ef 100644
--- a/native/kernels/embedding.cu
+++ b/native/kernels/embedding.cu
@@ -1,6 +1,15 @@
// Embedding lookup kernel for dotLLM.
// output[t] = embedTable[tokenIds[t]]
-// Supports F32, F16, Q8_0 source embedding tables → FP16 output.
+// Supports F32, F16, Q8_0, Q4_K, Q5_K, Q6_K source embedding tables → FP16 output.
+//
+// K-quant variants (Q4_K/Q5_K/Q6_K) dequantize the requested rows on the fly,
+// avoiding the 1.16 GiB FP16 expansion of a 151k-vocab × 4096-hidden table at
+// load. Math is bit-identical to the bulk dequant_q{4,5,6}_k_f16 kernels — same
+// per-superblock arithmetic, just gated by token-id row selection.
+//
+// Required precondition: hidden_size must be a multiple of 256 (the K-quant
+// super-block size). Caller (CudaWeights) must verify this before routing
+// through these kernels and fall back to bulk-dequant otherwise.
#include
#include
@@ -9,6 +18,12 @@
#define Q8_0_BLOCK_SIZE 32
#define Q8_0_BLOCK_BYTES 34
+// K-quant super-block sizes (256 elements per super-block).
+#define K_QUANT_SUPER_BLOCK_SIZE 256
+#define Q4_K_BLOCK_BYTES 144
+#define Q5_K_BLOCK_BYTES 176
+#define Q6_K_BLOCK_BYTES 210
+
extern "C" __global__ void __launch_bounds__(256) embedding_lookup_f32(
const float* __restrict__ embed_table,
const int* __restrict__ token_ids,
@@ -72,3 +87,180 @@ extern "C" __global__ void __launch_bounds__(256) embedding_lookup_q8_0(
out_row[b * Q8_0_BLOCK_SIZE + j] = __float2half(d * (float)qs[j]);
}
}
+
+// ── Per-row Q4_K embedding lookup ─────────────────────────────────────────
+// Dequant math mirrors dequant_q4_k_f16 in dequant.cu (bit-identical).
+// One CUDA block per token; 256 threads, each thread emits one element of
+// each super-block in the row (grid-stride over super-blocks via b).
+extern "C" __global__ void __launch_bounds__(256) embedding_lookup_q4_k(
+ const uint8_t* __restrict__ embed_table,
+ const int* __restrict__ token_ids,
+ half* __restrict__ output,
+ const int seq_len,
+ const int hidden_size)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ int token_id = token_ids[t];
+ int superblocks_per_row = hidden_size / K_QUANT_SUPER_BLOCK_SIZE;
+ const uint8_t* row = embed_table +
+ (size_t)token_id * superblocks_per_row * Q4_K_BLOCK_BYTES;
+ half* out_row = output + (size_t)t * hidden_size;
+
+ int tid = threadIdx.x; // 0..255 — one thread per element of a super-block
+
+ // Pre-compute thread's intra-superblock indices once (loop-invariant)
+ int pair = tid / 64;
+ int pos_in_pair = tid % 64;
+ int is_odd = pos_in_pair / 32;
+ int j = pos_in_pair % 32;
+ int sb_even = pair * 2;
+ int sb_odd = pair * 2 + 1;
+ int sb_cur = is_odd ? sb_odd : sb_even;
+
+ for (int b = 0; b < superblocks_per_row; b++)
+ {
+ const uint8_t* block = row + b * Q4_K_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qs = block + 16;
+
+ int sc, m;
+ if (sb_cur < 4)
+ {
+ sc = scales_raw[sb_cur] & 0x3F;
+ m = scales_raw[sb_cur + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sb_cur + 4] & 0x0F) | ((scales_raw[sb_cur - 4] >> 6) << 4);
+ m = (scales_raw[sb_cur + 4] >> 4) | ((scales_raw[sb_cur] >> 6) << 4);
+ }
+
+ uint8_t byte_val = qs[pair * 32 + j];
+ int nibble = is_odd ? (byte_val >> 4) : (byte_val & 0x0F);
+
+ float result = d * (float)sc * (float)nibble - dmin * (float)m;
+ out_row[b * K_QUANT_SUPER_BLOCK_SIZE + tid] = __float2half(result);
+ }
+}
+
+// ── Per-row Q5_K embedding lookup ─────────────────────────────────────────
+// Dequant math mirrors dequant_q5_k_f16 in dequant.cu (bit-identical).
+extern "C" __global__ void __launch_bounds__(256) embedding_lookup_q5_k(
+ const uint8_t* __restrict__ embed_table,
+ const int* __restrict__ token_ids,
+ half* __restrict__ output,
+ const int seq_len,
+ const int hidden_size)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ int token_id = token_ids[t];
+ int superblocks_per_row = hidden_size / K_QUANT_SUPER_BLOCK_SIZE;
+ const uint8_t* row = embed_table +
+ (size_t)token_id * superblocks_per_row * Q5_K_BLOCK_BYTES;
+ half* out_row = output + (size_t)t * hidden_size;
+
+ int tid = threadIdx.x; // 0..255
+
+ int sub = tid / 32;
+ int pos = tid % 32;
+
+ for (int b = 0; b < superblocks_per_row; b++)
+ {
+ const uint8_t* block = row + b * Q5_K_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qh = block + 16;
+ const uint8_t* qs = block + 48;
+
+ int sc, m;
+ if (sub < 4)
+ {
+ sc = scales_raw[sub] & 0x3F;
+ m = scales_raw[sub + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sub + 4] & 0x0F) | ((scales_raw[sub - 4] >> 6) << 4);
+ m = (scales_raw[sub + 4] >> 4) | ((scales_raw[sub] >> 6) << 4);
+ }
+
+ float scale = d * (float)sc;
+ float min_val = dmin * (float)m;
+
+ int pair_idx = sub >> 1;
+ int nibble_half = sub & 1;
+ uint8_t packed = qs[pair_idx * 32 + pos];
+ int nibble = nibble_half ? (packed >> 4) : (packed & 0x0F);
+ int bit = (qh[pos] >> sub) & 1;
+ int val = nibble | (bit << 4);
+
+ out_row[b * K_QUANT_SUPER_BLOCK_SIZE + tid] = __float2half(scale * (float)val - min_val);
+ }
+}
+
+// ── Per-row Q6_K embedding lookup ─────────────────────────────────────────
+// Dequant math mirrors dequant_q6_k_f16 in dequant.cu (bit-identical).
+extern "C" __global__ void __launch_bounds__(256) embedding_lookup_q6_k(
+ const uint8_t* __restrict__ embed_table,
+ const int* __restrict__ token_ids,
+ half* __restrict__ output,
+ const int seq_len,
+ const int hidden_size)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ int token_id = token_ids[t];
+ int superblocks_per_row = hidden_size / K_QUANT_SUPER_BLOCK_SIZE;
+ const uint8_t* row = embed_table +
+ (size_t)token_id * superblocks_per_row * Q6_K_BLOCK_BYTES;
+ half* out_row = output + (size_t)t * hidden_size;
+
+ int tid = threadIdx.x; // 0..255
+
+ int half_idx = tid / 128;
+ int pos_in_half = tid % 128;
+ int group = pos_in_half / 32;
+ int l = pos_in_half % 32;
+ int isc = l / 16;
+
+ for (int b = 0; b < superblocks_per_row; b++)
+ {
+ const uint8_t* block = row + b * Q6_K_BLOCK_BYTES;
+ const uint8_t* ql = block;
+ const uint8_t* qh_base = block + 128;
+ const int8_t* scales = reinterpret_cast(block + 192);
+ float d = __half2float(*reinterpret_cast(block + 208));
+
+ const uint8_t* ql_half = ql + half_idx * 64;
+ const uint8_t* qh_half = qh_base + half_idx * 32;
+ const int8_t* sc_half = scales + half_idx * 8;
+
+ int q_val;
+ switch (group)
+ {
+ case 0:
+ q_val = ((ql_half[l] & 0x0F) | (((qh_half[l] >> 0) & 3) << 4)) - 32;
+ break;
+ case 1:
+ q_val = ((ql_half[l + 32] & 0x0F) | (((qh_half[l] >> 2) & 3) << 4)) - 32;
+ break;
+ case 2:
+ q_val = ((ql_half[l] >> 4) | (((qh_half[l] >> 4) & 3) << 4)) - 32;
+ break;
+ default: // case 3
+ q_val = ((ql_half[l + 32] >> 4) | (((qh_half[l] >> 6) & 3) << 4)) - 32;
+ break;
+ }
+
+ float sc = d * (float)sc_half[isc + group * 2];
+ out_row[b * K_QUANT_SUPER_BLOCK_SIZE + tid] = __float2half(sc * (float)q_val);
+ }
+}
diff --git a/native/kernels/fused_add_rmsnorm.cu b/native/kernels/fused_add_rmsnorm.cu
index aa89c533..b4f38885 100644
--- a/native/kernels/fused_add_rmsnorm.cu
+++ b/native/kernels/fused_add_rmsnorm.cu
@@ -1,8 +1,22 @@
// Fused residual-add + RMS normalization kernel for dotLLM.
-// Eliminates one FP16 truncation at the critical residual junction:
+// Numerical contract: avoids the FP16 truncation of `sum` between Add and RmsNorm
+// (the original benefit of fusing) AND now also avoids the FP16 round-trip in
+// the second pass — the FP32 sum is cached in shared memory.
+//
// 1. sum = FP32(residual) + FP32(x) — keep in FP32
-// 2. residual[i] = FP16(sum) — update residual buffer
-// 3. output[i] = FP16(sum * rsqrt(rms) * w) — normalize from FP32 sum
+// 2. residual[i] = FP16(sum) — update residual buffer (FP16 store)
+// 3. shmem[i] = sum — cache FP32 sum for pass 2
+// 4. output[i] = FP16(sum * rsqrt(rms) * w) — normalize from cached FP32 sum
+//
+// Optimizations vs the original two-pass shmem-tree-reduction kernel:
+// * half2 vectorized loads/stores (one transaction per 2 elements)
+// * __shfl_xor_sync warp reduction (symmetric — every lane holds the sum)
+// * Pre-folds 1/n into the rsqrt argument via fmaf
+// * Pass 2 reads FP32 sums from shared memory — eliminates the second global
+// read of residual AND the extra __half2float conversion
+//
+// Shared memory layout (dynamic): [n floats sum cache] [up to 32 floats warp scratch / rms_inv]
+// Caller MUST pass sharedBytes = (n + 32) * sizeof(float) (rounded up internally).
#include
@@ -10,63 +24,101 @@ extern "C" __global__ void __launch_bounds__(256) fused_add_rmsnorm_f16(
half* __restrict__ residual, // [n] in/out: updated with sum
const half* __restrict__ x, // [n] layer output to add
const half* __restrict__ weight, // [n] norm weights
- half* __restrict__ output, // [n] normalized output
+ half* __restrict__ output, // [n] normalized output (may alias `x`)
const int n,
const float eps)
{
- // Each block processes one row
+ extern __shared__ float smem[];
+ // smem[0 .. n-1] : FP32 sum cache
+ // smem[smem_scratch_off ..] : warp-sum scratch (max 32 floats); slot [0] also stores rms_inv
+ const int smem_scratch_off = (n + 1) & ~1; // align to 2 floats (8 bytes)
+ float* warp_scratch = smem + smem_scratch_off;
+
const int row = blockIdx.x;
half* res_row = residual + (size_t)row * n;
const half* x_row = x + (size_t)row * n;
half* out_row = output + (size_t)row * n;
+ const int tid = threadIdx.x;
+ const int n2 = n >> 1;
- // Pass 1: compute sum in FP32, write sum back to residual (FP16),
- // and accumulate sum-of-squares for RMS
+ // ── Pass 1: add, store FP16 sum to residual, cache FP32 sum in shmem, accumulate sum_sq ──
float sum_sq = 0.0f;
- for (int i = threadIdx.x; i < n; i += blockDim.x)
- {
- float r = __half2float(res_row[i]);
- float xi = __half2float(x_row[i]);
- float sum = r + xi;
- // Update residual with the sum (FP16 storage)
- res_row[i] = __float2half(sum);
+ const half2* __restrict__ res2_in = reinterpret_cast(res_row);
+ const half2* __restrict__ x2 = reinterpret_cast(x_row);
+ half2* __restrict__ res2_out = reinterpret_cast(res_row);
- // Accumulate for RMS from the FP32 sum (no extra truncation!)
- sum_sq += sum * sum;
+ for (int i = tid; i < n2; i += blockDim.x)
+ {
+ half2 r = res2_in[i];
+ half2 xi = x2[i];
+ float r0 = __low2float(r), r1 = __high2float(r);
+ float x0 = __low2float(xi), x1 = __high2float(xi);
+ float s0 = r0 + x0;
+ float s1 = r1 + x1;
+ res2_out[i] = __floats2half2_rn(s0, s1);
+ smem[2 * i] = s0;
+ smem[2 * i + 1] = s1;
+ sum_sq = fmaf(s0, s0, sum_sq);
+ sum_sq = fmaf(s1, s1, sum_sq);
}
- // Warp reduction for sum_sq
- for (int offset = warpSize / 2; offset > 0; offset >>= 1)
- sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, offset);
+ // Tail element if n is odd
+ if ((n & 1) && tid == 0)
+ {
+ int last = n - 1;
+ float r = __half2float(res_row[last]);
+ float xi = __half2float(x_row[last]);
+ float s = r + xi;
+ res_row[last] = __float2half(s);
+ smem[last] = s;
+ sum_sq = fmaf(s, s, sum_sq);
+ }
- __shared__ float warp_sums[32];
- int lane = threadIdx.x % warpSize;
- int warp_id = threadIdx.x / warpSize;
+ // ── Warp reduction (symmetric: all lanes hold the warp sum) ──
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ sum_sq += __shfl_xor_sync(0xFFFFFFFF, sum_sq, offset);
- if (lane == 0) warp_sums[warp_id] = sum_sq;
+ int lane = tid & 31;
+ int warp_id = tid >> 5;
+ if (lane == 0) warp_scratch[warp_id] = sum_sq;
__syncthreads();
+ // First warp aggregates the per-warp sums and publishes rms_inv
if (warp_id == 0)
{
- int num_warps = (blockDim.x + warpSize - 1) / warpSize;
- sum_sq = (lane < num_warps) ? warp_sums[lane] : 0.0f;
- for (int offset = warpSize / 2; offset > 0; offset >>= 1)
- sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, offset);
- }
+ int num_warps = (blockDim.x + 31) >> 5;
+ sum_sq = (lane < num_warps) ? warp_scratch[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ sum_sq += __shfl_xor_sync(0xFFFFFFFF, sum_sq, offset);
- __shared__ float rms_inv;
- if (threadIdx.x == 0)
- rms_inv = rsqrtf(sum_sq / (float)n + eps);
+ if (lane == 0)
+ warp_scratch[0] = rsqrtf(fmaf(sum_sq, 1.0f / (float)n, eps));
+ }
__syncthreads();
- // Pass 2: normalize — read from residual (which now has the sum in FP16)
- // Note: we read back from FP16, so there IS one truncation. But the RMS
- // was computed from the FP32 sum, which is the key improvement.
- for (int i = threadIdx.x; i < n; i += blockDim.x)
+ const float rms_inv = warp_scratch[0];
+
+ // ── Pass 2: read FP32 sum from shmem, scale by rms_inv * weight, write FP16 ──
+ const half2* __restrict__ w2 = reinterpret_cast(weight);
+ half2* __restrict__ y2 = reinterpret_cast(out_row);
+
+ for (int i = tid; i < n2; i += blockDim.x)
+ {
+ float s0 = smem[2 * i];
+ float s1 = smem[2 * i + 1];
+ half2 wh = w2[i];
+ float w0 = __low2float(wh), w1 = __high2float(wh);
+ y2[i] = __floats2half2_rn(s0 * rms_inv * w0, s1 * rms_inv * w1);
+ }
+
+ if ((n & 1) && tid == 0)
{
- float v = __half2float(res_row[i]);
- float w = __half2float(weight[i]);
- out_row[i] = __float2half(v * rms_inv * w);
+ int last = n - 1;
+ float s = smem[last];
+ float w = __half2float(weight[last]);
+ out_row[last] = __float2half(s * rms_inv * w);
}
}
diff --git a/native/kernels/fused_rope_kv_write.cu b/native/kernels/fused_rope_kv_write.cu
new file mode 100644
index 00000000..6d888f53
--- /dev/null
+++ b/native/kernels/fused_rope_kv_write.cu
@@ -0,0 +1,240 @@
+// Fused decode-step RoPE + KV-cache write.
+//
+// Replaces three separate launches per layer on the eager decode path:
+// 1. rope_f16 — rotate Q (in-place) and K (in-place) on the projection scratch
+// 2. cuMemcpyDtoDAsync(K_scratch → K_cache_layer + pos*stride)
+// 3. cuMemcpyDtoDAsync(V_scratch → V_cache_layer + pos*stride)
+//
+// Per-launch WDDM overhead is ~22 µs; on a 30-layer model that's 30*3*22 µs ≈ 2 ms
+// of pure submission cost per token. The actual compute is trivial — for
+// SmolLM-135M (num_heads=9, num_kv_heads=3, head_dim=64, rope_dim=64) it's
+// 288 RoPE pairs + 96 K-rope-and-write + 192 V-copy = 576 elements/layer.
+//
+// Decode-only contract: seq_len == 1 hard-coded — we read positions[0] and write
+// to one cache row per K-head and per V-head. Prefill keeps the original
+// rope+memcpy path (different per-token destinations make the fused write awkward).
+//
+// Region layout in the flat thread grid:
+// [0, r0) r0 = num_heads * (rope_dim/2) → Q rotation pair (in-place on q_src)
+// [r0, r1) r1 = r0 + num_kv_heads * (rope_dim/2) → K rotation pair (write to k_cache row)
+// [r1, r2) r2 = r1 + num_kv_heads * (head_dim - rope_dim) → K tail plain copy (only if rope_dim < head_dim)
+// [r2, r3) r3 = r2 + num_kv_heads * head_dim → V plain copy (write to v_cache row)
+//
+// Q is laid out [num_heads * head_dim] in `q_src`; K [num_kv_heads * head_dim] in
+// `k_src`; V [num_kv_heads * head_dim] in `v_src`. The fused-QKV decode path
+// passes slice offsets into the packed [Q|K|V] buffer — the kernel doesn't care
+// whether they alias or are separate allocations.
+//
+// `cache_pos` is a host-side int (decode token's absolute KV row index). The
+// graph-friendly variant `fused_rope_kv_write_f16_dyn` reads it from a device
+// pointer instead so the address `cache_K + cache_pos * kv_stride` is computed
+// device-side — preventing CUDA Graphs from baking in the row index.
+
+#include
+
+extern "C" __global__ void __launch_bounds__(256) fused_rope_kv_write_f16(
+ half* __restrict__ q_src, // [num_heads * head_dim] in/out (rotated in place)
+ const half* __restrict__ k_src, // [num_kv_heads * head_dim] in (read-only)
+ const half* __restrict__ v_src, // [num_kv_heads * head_dim] in (read-only)
+ half* __restrict__ k_cache, // [max_seq_len, kv_stride] base ptr for this layer
+ half* __restrict__ v_cache, // [max_seq_len, kv_stride] base ptr for this layer
+ const int* __restrict__ positions, // [1] device int (RoPE position)
+ int cache_pos, // host-side absolute KV row index
+ int num_heads,
+ int num_kv_heads,
+ int head_dim,
+ int rope_dim,
+ int kv_stride, // = num_kv_heads * head_dim
+ float theta,
+ int rope_type) // 0 = standard (interleaved pairs), 1 = neox (split halves)
+{
+ const int half_rope = rope_dim / 2;
+ const int tail = head_dim - rope_dim;
+
+ const int r0 = num_heads * half_rope;
+ const int r1 = r0 + num_kv_heads * half_rope;
+ const int r2 = r1 + num_kv_heads * tail;
+ const int r3 = r2 + num_kv_heads * head_dim;
+
+ const int tid = blockIdx.x * blockDim.x + threadIdx.x;
+ if (tid >= r3) return;
+
+ const int pos = positions[0];
+ const size_t cache_row_offset = (size_t)cache_pos * (size_t)kv_stride;
+
+ // ── Region 0: Q rotation pairs (in-place on q_src) ──
+ if (tid < r0)
+ {
+ const int pair = tid % half_rope;
+ const int head = tid / half_rope;
+
+ const float freq = 1.0f / powf(theta, (float)(2 * pair) / (float)rope_dim);
+ const float angle = (float)pos * freq;
+ const float c = cosf(angle);
+ const float s = sinf(angle);
+
+ const int base_idx = head * head_dim;
+ int i0, i1;
+ if (rope_type == 1) // neox
+ {
+ i0 = base_idx + pair;
+ i1 = base_idx + pair + half_rope;
+ }
+ else // standard
+ {
+ i0 = base_idx + 2 * pair;
+ i1 = base_idx + 2 * pair + 1;
+ }
+
+ const float v0 = __half2float(q_src[i0]);
+ const float v1 = __half2float(q_src[i1]);
+ q_src[i0] = __float2half(v0 * c - v1 * s);
+ q_src[i1] = __float2half(v0 * s + v1 * c);
+ return;
+ }
+
+ // ── Region 1: K rotation pairs (read from k_src, write rotated to k_cache row) ──
+ if (tid < r1)
+ {
+ const int local = tid - r0;
+ const int pair = local % half_rope;
+ const int head = local / half_rope;
+
+ const float freq = 1.0f / powf(theta, (float)(2 * pair) / (float)rope_dim);
+ const float angle = (float)pos * freq;
+ const float c = cosf(angle);
+ const float s = sinf(angle);
+
+ const int base_idx = head * head_dim;
+ int i0, i1;
+ if (rope_type == 1) // neox
+ {
+ i0 = base_idx + pair;
+ i1 = base_idx + pair + half_rope;
+ }
+ else // standard
+ {
+ i0 = base_idx + 2 * pair;
+ i1 = base_idx + 2 * pair + 1;
+ }
+
+ const float v0 = __half2float(k_src[i0]);
+ const float v1 = __half2float(k_src[i1]);
+
+ half* k_dst = k_cache + cache_row_offset;
+ k_dst[i0] = __float2half(v0 * c - v1 * s);
+ k_dst[i1] = __float2half(v0 * s + v1 * c);
+ return;
+ }
+
+ // ── Region 2: K tail copy (rope_dim < head_dim, no rotation) ──
+ if (tid < r2)
+ {
+ const int local = tid - r1;
+ const int t = local % tail; // tail offset within head
+ const int head = local / tail;
+ const int idx = head * head_dim + rope_dim + t;
+ half* k_dst = k_cache + cache_row_offset;
+ k_dst[idx] = k_src[idx];
+ return;
+ }
+
+ // ── Region 3: V plain copy (no rotation) ──
+ {
+ const int idx = tid - r2;
+ half* v_dst = v_cache + cache_row_offset;
+ v_dst[idx] = v_src[idx];
+ }
+}
+
+// Graph-friendly variant: cache_pos is read from a device pointer.
+// Used by the CUDA Graphs decode replay path so the row index can change between
+// replays without re-instantiating the graph.
+extern "C" __global__ void __launch_bounds__(256) fused_rope_kv_write_f16_dyn(
+ half* __restrict__ q_src,
+ const half* __restrict__ k_src,
+ const half* __restrict__ v_src,
+ half* __restrict__ k_cache,
+ half* __restrict__ v_cache,
+ const int* __restrict__ positions,
+ const int* __restrict__ cache_pos_ptr,
+ int num_heads,
+ int num_kv_heads,
+ int head_dim,
+ int rope_dim,
+ int kv_stride,
+ float theta,
+ int rope_type)
+{
+ const int half_rope = rope_dim / 2;
+ const int tail = head_dim - rope_dim;
+
+ const int r0 = num_heads * half_rope;
+ const int r1 = r0 + num_kv_heads * half_rope;
+ const int r2 = r1 + num_kv_heads * tail;
+ const int r3 = r2 + num_kv_heads * head_dim;
+
+ const int tid = blockIdx.x * blockDim.x + threadIdx.x;
+ if (tid >= r3) return;
+
+ const int pos = positions[0];
+ const int cache_pos = cache_pos_ptr[0];
+ const size_t cache_row_offset = (size_t)cache_pos * (size_t)kv_stride;
+
+ if (tid < r0)
+ {
+ const int pair = tid % half_rope;
+ const int head = tid / half_rope;
+ const float freq = 1.0f / powf(theta, (float)(2 * pair) / (float)rope_dim);
+ const float angle = (float)pos * freq;
+ const float c = cosf(angle);
+ const float s = sinf(angle);
+ const int base_idx = head * head_dim;
+ int i0, i1;
+ if (rope_type == 1) { i0 = base_idx + pair; i1 = base_idx + pair + half_rope; }
+ else { i0 = base_idx + 2 * pair; i1 = base_idx + 2 * pair + 1; }
+ const float v0 = __half2float(q_src[i0]);
+ const float v1 = __half2float(q_src[i1]);
+ q_src[i0] = __float2half(v0 * c - v1 * s);
+ q_src[i1] = __float2half(v0 * s + v1 * c);
+ return;
+ }
+
+ if (tid < r1)
+ {
+ const int local = tid - r0;
+ const int pair = local % half_rope;
+ const int head = local / half_rope;
+ const float freq = 1.0f / powf(theta, (float)(2 * pair) / (float)rope_dim);
+ const float angle = (float)pos * freq;
+ const float c = cosf(angle);
+ const float s = sinf(angle);
+ const int base_idx = head * head_dim;
+ int i0, i1;
+ if (rope_type == 1) { i0 = base_idx + pair; i1 = base_idx + pair + half_rope; }
+ else { i0 = base_idx + 2 * pair; i1 = base_idx + 2 * pair + 1; }
+ const float v0 = __half2float(k_src[i0]);
+ const float v1 = __half2float(k_src[i1]);
+ half* k_dst = k_cache + cache_row_offset;
+ k_dst[i0] = __float2half(v0 * c - v1 * s);
+ k_dst[i1] = __float2half(v0 * s + v1 * c);
+ return;
+ }
+
+ if (tid < r2)
+ {
+ const int local = tid - r1;
+ const int t = local % tail;
+ const int head = local / tail;
+ const int idx = head * head_dim + rope_dim + t;
+ half* k_dst = k_cache + cache_row_offset;
+ k_dst[idx] = k_src[idx];
+ return;
+ }
+
+ {
+ const int idx = tid - r2;
+ half* v_dst = v_cache + cache_row_offset;
+ v_dst[idx] = v_src[idx];
+ }
+}
diff --git a/native/kernels/kv_write.cu b/native/kernels/kv_write.cu
new file mode 100644
index 00000000..044ed3ec
--- /dev/null
+++ b/native/kernels/kv_write.cu
@@ -0,0 +1,189 @@
+// Decode-step KV cache write. Replaces a host-side cuMemcpyDtoDAsync where the
+// destination address is `dst_base + posPtr[0] * row_bytes` — fine in eager
+// mode but fatal under CUDA Graphs because the host-computed address gets baked
+// into the graph at instantiate time and the next decode step would clobber the
+// same row.
+//
+// Single decode token, FP16. Launch with grid=(num_blocks_for_row), block=256.
+// Writes one row of `kv_stride` FP16 elements from `src` to
+// `dst_base + posPtr[0] * row_stride_fp16`.
+//
+// We pass row_stride_fp16 (in element units, not bytes) to keep arithmetic in
+// the kernel cheap and unambiguous.
+
+#include
+
+extern "C" __global__ void kv_write_one_f16(
+ const half* __restrict__ src, // [kv_stride] FP16 (one row of new K or V)
+ half* __restrict__ dst_base, // [maxSeqLen, kv_stride] FP16 cache
+ const int kv_stride, // = num_kv_heads * head_dim
+ const int* __restrict__ pos_ptr) // device-resident write index
+{
+ int pos = pos_ptr[0];
+ int tid = blockIdx.x * blockDim.x + threadIdx.x;
+ if (tid >= kv_stride) return;
+
+ half* dst = dst_base + (size_t)pos * kv_stride;
+ dst[tid] = src[tid];
+}
+
+// ───────────────────────────────────────────────────────────────────
+// Quantized-KV graph-friendly helpers.
+//
+// kv_write_one_f16_ring: writes the new FP16 row into the per-layer
+// ring buffer at slot `pos % window_size`. Same role as
+// kv_write_one_f16, just modulo-indexed.
+//
+// kv_dequant_q{8,4}_0_dyn: dequantizes the [0, quant_len) prefix of
+// the per-layer Q-cache into the FP16 attention scratch, where
+// quant_len = max(0, pos+1 - window_size) is read device-side.
+// Grid-stride loop, predicated.
+//
+// kv_window_to_scratch_dyn: copies the live FP16 window into the
+// contiguous scratch starting at row `quant_len`. Each block handles
+// one ring slot; predicated on whether that slot is currently live.
+// ───────────────────────────────────────────────────────────────────
+
+#include
+
+#define KV_Q8_0_BLOCK_SIZE 32
+#define KV_Q8_0_BLOCK_BYTES 34
+#define KV_Q4_0_BLOCK_SIZE 32
+#define KV_Q4_0_BLOCK_BYTES 18
+
+extern "C" __global__ void kv_write_one_f16_ring(
+ const half* __restrict__ src, // [kv_stride] FP16
+ half* __restrict__ ring_base, // [window_size, kv_stride] FP16
+ const int kv_stride,
+ const int window_size,
+ const int* __restrict__ pos_ptr)
+{
+ int pos = pos_ptr[0];
+ int slot = pos % window_size;
+ int tid = blockIdx.x * blockDim.x + threadIdx.x;
+ if (tid >= kv_stride) return;
+ half* dst = ring_base + (size_t)slot * kv_stride;
+ dst[tid] = src[tid];
+}
+
+extern "C" __global__ void kv_dequant_q8_0_dyn(
+ const uint8_t* __restrict__ quant_base, // [maxSeqLen, kv_stride/32 * 34]
+ half* __restrict__ scratch_base, // [maxSeqLen, kv_stride]
+ const int kv_stride,
+ const int window_size,
+ const int* __restrict__ pos_ptr)
+{
+ int pos = pos_ptr[0];
+ int quant_len = pos + 1 - window_size;
+ if (quant_len <= 0) return;
+
+ int blocks_per_row = kv_stride / KV_Q8_0_BLOCK_SIZE;
+ int total_blocks = quant_len * blocks_per_row;
+
+ int lane = threadIdx.x % KV_Q8_0_BLOCK_SIZE;
+ int warp_in_block = threadIdx.x / KV_Q8_0_BLOCK_SIZE;
+ int warps_per_grid = (gridDim.x * blockDim.x) / KV_Q8_0_BLOCK_SIZE;
+ int start_block = blockIdx.x * (blockDim.x / KV_Q8_0_BLOCK_SIZE) + warp_in_block;
+
+ for (int block_idx = start_block; block_idx < total_blocks; block_idx += warps_per_grid)
+ {
+ const uint8_t* block = quant_base + (size_t)block_idx * KV_Q8_0_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ int8_t q = reinterpret_cast(block + 2)[lane];
+ scratch_base[(size_t)block_idx * KV_Q8_0_BLOCK_SIZE + lane] = __float2half(d * (float)q);
+ }
+}
+
+extern "C" __global__ void kv_dequant_q4_0_dyn(
+ const uint8_t* __restrict__ quant_base,
+ half* __restrict__ scratch_base,
+ const int kv_stride,
+ const int window_size,
+ const int* __restrict__ pos_ptr)
+{
+ int pos = pos_ptr[0];
+ int quant_len = pos + 1 - window_size;
+ if (quant_len <= 0) return;
+
+ int blocks_per_row = kv_stride / KV_Q4_0_BLOCK_SIZE;
+ int total_blocks = quant_len * blocks_per_row;
+
+ int lane = threadIdx.x % KV_Q4_0_BLOCK_SIZE;
+ int warp_in_block = threadIdx.x / KV_Q4_0_BLOCK_SIZE;
+ int warps_per_grid = (gridDim.x * blockDim.x) / KV_Q4_0_BLOCK_SIZE;
+ int start_block = blockIdx.x * (blockDim.x / KV_Q4_0_BLOCK_SIZE) + warp_in_block;
+
+ for (int block_idx = start_block; block_idx < total_blocks; block_idx += warps_per_grid)
+ {
+ const uint8_t* block = quant_base + (size_t)block_idx * KV_Q4_0_BLOCK_BYTES;
+ float d = __half2float(*reinterpret_cast(block));
+ const uint8_t* qs = block + 2;
+ int byte_idx = lane / 2;
+ uint8_t packed = qs[byte_idx];
+ int val = (lane & 1) ? ((int)(packed >> 4) - 8) : ((int)(packed & 0x0F) - 8);
+ scratch_base[(size_t)block_idx * KV_Q4_0_BLOCK_SIZE + lane] = __float2half(d * (float)val);
+ }
+}
+
+// Scatters the live FP16 window from the per-layer ring buffer into the
+// contiguous attention scratch starting at scratch row `quant_len`. One
+// CUDA block per ring slot — each block decides whether its slot is live
+// at the current decode position and, if so, what absolute scratch row to
+// land at.
+//
+// Grid: (window_size, 1, 1). Block: (kv_stride threads), capped — caller
+// uses ceil(kv_stride / 256) along grid.x_inner if kv_stride exceeds the
+// block dim. We keep it simple: each block has `min(kv_stride, 1024)`
+// threads and uses a per-thread stride loop over kv_stride.
+extern "C" __global__ void kv_window_to_scratch_dyn(
+ const half* __restrict__ ring_base, // [window_size, kv_stride]
+ half* __restrict__ scratch_base, // [maxSeqLen, kv_stride]
+ const int kv_stride,
+ const int window_size,
+ const int* __restrict__ pos_ptr)
+{
+ int pos = pos_ptr[0];
+ int current_len = pos + 1;
+ int quant_len = current_len - window_size;
+ if (quant_len < 0) quant_len = 0;
+ int window_len = current_len - quant_len; // == min(current_len, window_size)
+
+ int slot = blockIdx.x; // which ring slot this block owns
+ if (slot >= window_size) return;
+ if (slot >= window_len) return; // ring slot not yet populated
+
+ // Determine absolute position whose row sits in `slot` right now.
+ // Ring is filled by `pos % window_size`. The set of absolute positions
+ // ever stored at this slot is {slot, slot + W, slot + 2W, ...}. The
+ // most recent one ≤ pos is the current occupant.
+ //
+ // Special case pos < window_size: ring is partially filled — slot s
+ // holds position s itself.
+ int abs_pos;
+ if (pos < window_size)
+ {
+ abs_pos = slot;
+ }
+ else
+ {
+ // Two candidates: slot + (quant_len / window_size) * window_size
+ // and the same plus window_size. Pick whichever is
+ // ≤ pos AND ≥ quant_len.
+ int base = (quant_len / window_size) * window_size;
+ int cand_a = base + slot;
+ if (cand_a >= quant_len && cand_a <= pos)
+ {
+ abs_pos = cand_a;
+ }
+ else
+ {
+ abs_pos = cand_a + window_size; // wraps once
+ }
+ }
+
+ const half* src_row = ring_base + (size_t)slot * kv_stride;
+ half* dst_row = scratch_base + (size_t)abs_pos * kv_stride;
+
+ for (int i = threadIdx.x; i < kv_stride; i += blockDim.x)
+ dst_row[i] = src_row[i];
+}
diff --git a/native/kernels/mla_helpers.cu b/native/kernels/mla_helpers.cu
new file mode 100644
index 00000000..8db9e117
--- /dev/null
+++ b/native/kernels/mla_helpers.cu
@@ -0,0 +1,329 @@
+// Helper kernels for the MLA Phase A forward path. F32 + FP16 siblings.
+// F32 versions match the CPU oracle (DotLLM.Cpu.Kernels.MlaAttention.Execute)
+// byte-for-byte algorithmically; FP16 versions match the GQA precision
+// pattern (FP16 storage / activations, FP32 reduction).
+
+#include
+#include
+
+// ── Split kv_b_proj output into per-head K_nope and per-head V ──────────
+//
+// Input layout (per token): [numHeads, qkNopeHeadDim + vHeadDim] contiguous.
+// Per head h, the first qkNopeHeadDim floats go to kNope[t, h, :], the
+// remaining vHeadDim floats go to v[t, h, :].
+//
+// Grid: blocks = seqLen * numHeads, one block per (token, head) pair.
+// Threads = 128 (cover qkNope + vHead with a strided loop — typical
+// V2-Lite has qkNope=128, vHead=128 → 256 elems/block).
+extern "C" __global__ void __launch_bounds__(128) mla_split_kv_b_f32(
+ const float* __restrict__ kv_b_expanded, // [seqLen, numHeads * (qkNope + vHead)]
+ float* __restrict__ k_nope_dst, // [seqLen, numHeads * qkNope]
+ float* __restrict__ v_dst, // [seqLen, numHeads * vHead]
+ const int seq_len, const int num_heads,
+ const int qk_nope_head_dim, const int v_head_dim)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_len * num_heads) return;
+
+ int t = block_id / num_heads;
+ int h = block_id % num_heads;
+
+ int per_head = qk_nope_head_dim + v_head_dim;
+ int kv_b_stride = num_heads * per_head;
+ int k_stride = num_heads * qk_nope_head_dim;
+ int v_stride = num_heads * v_head_dim;
+
+ const float* src_block = kv_b_expanded + (size_t)t * kv_b_stride + h * per_head;
+ float* k_dst = k_nope_dst + (size_t)t * k_stride + h * qk_nope_head_dim;
+ float* v_out = v_dst + (size_t)t * v_stride + h * v_head_dim;
+
+ for (int d = threadIdx.x; d < qk_nope_head_dim; d += blockDim.x)
+ k_dst[d] = src_block[d];
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ v_out[d] = src_block[qk_nope_head_dim + d];
+}
+
+// ── Apply RoPE (Norm-pair convention) to the per-head rope portion of Q ──
+//
+// For each (t, h, i) with i in [0, halfRope):
+// q_pe = q[t, h, qkNope + 2i .. qkNope + 2i + 2)
+// v0 = q_pe[0] * cos[t,i] - q_pe[1] * sin[t,i]
+// v1 = q_pe[1] * cos[t,i] + q_pe[0] * sin[t,i]
+//
+// Cos/Sin tables: [maxSeq, halfRope] row-major. positionOffset shifts the
+// row index so token t reads row (positionOffset + t).
+//
+// Grid: blocks = seqLen * numHeads, threads = halfRope (rounded up to next
+// pow2 / capped at 128 with strided loop).
+extern "C" __global__ void __launch_bounds__(128) mla_rope_q_pe_f32(
+ float* __restrict__ q, // [seqLen, numHeads, qkNope + qkRope]
+ const float* __restrict__ cos_tab, // [maxSeq, qkRope/2]
+ const float* __restrict__ sin_tab, // [maxSeq, qkRope/2]
+ const int seq_len, const int num_heads,
+ const int qk_nope_head_dim, const int qk_rope_head_dim,
+ const int position_offset)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_len * num_heads) return;
+
+ int t = block_id / num_heads;
+ int h = block_id % num_heads;
+ int half = qk_rope_head_dim / 2;
+ int pos = position_offset + t;
+
+ int qk_head_dim = qk_nope_head_dim + qk_rope_head_dim;
+ int q_stride = num_heads * qk_head_dim;
+ float* q_pe = q + (size_t)t * q_stride + h * qk_head_dim + qk_nope_head_dim;
+ const float* cos_row = cos_tab + (size_t)pos * half;
+ const float* sin_row = sin_tab + (size_t)pos * half;
+
+ for (int i = threadIdx.x; i < half; i += blockDim.x)
+ {
+ float a = q_pe[2 * i];
+ float b = q_pe[2 * i + 1];
+ float c = cos_row[i];
+ float s = sin_row[i];
+ q_pe[2 * i] = a * c - b * s;
+ q_pe[2 * i + 1] = b * c + a * s;
+ }
+}
+
+// ── Apply RoPE to the MQA-shared K_pe (one rope vector per token, no head dim) ──
+//
+// Same Norm-pair convention as mla_rope_q_pe_f32 but operates on the
+// shared k_pe[seqLen, qkRope] buffer (one row per token, broadcast across
+// heads at attention time).
+//
+// Grid: blocks = seqLen, threads = halfRope (capped at 128 strided loop).
+extern "C" __global__ void __launch_bounds__(128) mla_rope_k_pe_f32(
+ float* __restrict__ k_pe, // [seqLen, qkRope]
+ const float* __restrict__ cos_tab, // [maxSeq, qkRope/2]
+ const float* __restrict__ sin_tab, // [maxSeq, qkRope/2]
+ const int seq_len, const int qk_rope_head_dim,
+ const int position_offset)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+ int half = qk_rope_head_dim / 2;
+ int pos = position_offset + t;
+
+ float* row = k_pe + (size_t)t * qk_rope_head_dim;
+ const float* cos_row = cos_tab + (size_t)pos * half;
+ const float* sin_row = sin_tab + (size_t)pos * half;
+
+ for (int i = threadIdx.x; i < half; i += blockDim.x)
+ {
+ float a = row[2 * i];
+ float b = row[2 * i + 1];
+ float c = cos_row[i];
+ float s = sin_row[i];
+ row[2 * i] = a * c - b * s;
+ row[2 * i + 1] = b * c + a * s;
+ }
+}
+
+// ── F32 RMSNorm with explicit (numRows, dim) layout ──────────────────────
+//
+// Used by the MLA pipeline for q_a_layernorm (numRows=seqLen, dim=qLora)
+// and kv_a_layernorm (numRows=seqLen, dim=kvLora). The repo already ships
+// rmsnorm_f32 with the same algorithmic shape — this kernel exists as a
+// drop-in alias that takes (numRows, dim) at the launch level rather than
+// caller-side gridding.
+//
+// One block per row, threads cooperate on reduction.
+extern "C" __global__ void __launch_bounds__(128) mla_rmsnorm_f32(
+ const float* __restrict__ input,
+ const float* __restrict__ weight,
+ float* __restrict__ output,
+ const int dim, const float epsilon)
+{
+ int row = blockIdx.x;
+ const float* in_row = input + (size_t)row * dim;
+ float* out_row = output + (size_t)row * dim;
+
+ // Sum of squares — block-strided.
+ float sum_sq = 0.0f;
+ for (int i = threadIdx.x; i < dim; i += blockDim.x)
+ {
+ float x = in_row[i];
+ sum_sq += x * x;
+ }
+
+ // Block reduction via shared memory + warp shfl.
+ __shared__ float warp_sums[4]; // up to 128 threads = 4 warps
+ int lane = threadIdx.x & 31;
+ int warp = threadIdx.x >> 5;
+
+ for (int off = 16; off > 0; off >>= 1)
+ sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, off);
+ if (lane == 0) warp_sums[warp] = sum_sq;
+ __syncthreads();
+
+ if (warp == 0)
+ {
+ sum_sq = (lane < (blockDim.x + 31) / 32) ? warp_sums[lane] : 0.0f;
+ for (int off = 16; off > 0; off >>= 1)
+ sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, off);
+ if (lane == 0) warp_sums[0] = sum_sq;
+ }
+ __syncthreads();
+
+ float rms = sqrtf(warp_sums[0] / (float)dim + epsilon);
+ float inv = 1.0f / rms;
+
+ for (int i = threadIdx.x; i < dim; i += blockDim.x)
+ out_row[i] = in_row[i] * inv * weight[i];
+}
+
+// ── #region MLA FP16 helpers ─────────────────────────────────────────────
+//
+// FP16 siblings of the F32 helpers. Same algorithm, FP16 in/out, FP32
+// accumulation for RMSNorm. Used by the FP16 MLA forward path.
+
+// Per-head split of kv_b expansion — FP16 in / FP16 out.
+extern "C" __global__ void __launch_bounds__(128) mla_split_kv_b_f16(
+ const half* __restrict__ kv_b_expanded, // [seqLen, numHeads * (qkNope + vHead)]
+ half* __restrict__ k_nope_dst, // [seqLen, numHeads * qkNope]
+ half* __restrict__ v_dst, // [seqLen, numHeads * vHead]
+ const int seq_len, const int num_heads,
+ const int qk_nope_head_dim, const int v_head_dim)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_len * num_heads) return;
+
+ int t = block_id / num_heads;
+ int h = block_id % num_heads;
+
+ int per_head = qk_nope_head_dim + v_head_dim;
+ int kv_b_stride = num_heads * per_head;
+ int k_stride = num_heads * qk_nope_head_dim;
+ int v_stride = num_heads * v_head_dim;
+
+ const half* src_block = kv_b_expanded + (size_t)t * kv_b_stride + h * per_head;
+ half* k_dst = k_nope_dst + (size_t)t * k_stride + h * qk_nope_head_dim;
+ half* v_out = v_dst + (size_t)t * v_stride + h * v_head_dim;
+
+ for (int d = threadIdx.x; d < qk_nope_head_dim; d += blockDim.x)
+ k_dst[d] = src_block[d];
+ for (int d = threadIdx.x; d < v_head_dim; d += blockDim.x)
+ v_out[d] = src_block[qk_nope_head_dim + d];
+}
+
+// In-place RoPE on the rope sub-dim of Q (per head) — FP16. Same Norm-pair
+// convention as the F32 sibling; cos/sin tables stay F32 for precision.
+extern "C" __global__ void __launch_bounds__(128) mla_rope_q_pe_f16(
+ half* __restrict__ q, // [seqLen, numHeads, qkNope + qkRope]
+ const float* __restrict__ cos_tab, // [maxSeq, qkRope/2]
+ const float* __restrict__ sin_tab, // [maxSeq, qkRope/2]
+ const int seq_len, const int num_heads,
+ const int qk_nope_head_dim, const int qk_rope_head_dim,
+ const int position_offset)
+{
+ int block_id = blockIdx.x;
+ if (block_id >= seq_len * num_heads) return;
+
+ int t = block_id / num_heads;
+ int h = block_id % num_heads;
+ int half_d = qk_rope_head_dim / 2;
+ int pos = position_offset + t;
+
+ int qk_head_dim = qk_nope_head_dim + qk_rope_head_dim;
+ int q_stride = num_heads * qk_head_dim;
+ half* q_pe = q + (size_t)t * q_stride + h * qk_head_dim + qk_nope_head_dim;
+ const float* cos_row = cos_tab + (size_t)pos * half_d;
+ const float* sin_row = sin_tab + (size_t)pos * half_d;
+
+ for (int i = threadIdx.x; i < half_d; i += blockDim.x)
+ {
+ float a = __half2float(q_pe[2 * i]);
+ float b = __half2float(q_pe[2 * i + 1]);
+ float c = cos_row[i];
+ float s = sin_row[i];
+ q_pe[2 * i] = __float2half(a * c - b * s);
+ q_pe[2 * i + 1] = __float2half(b * c + a * s);
+ }
+}
+
+// In-place RoPE on the MQA-shared K_pe — FP16. One row per token, no head dim.
+extern "C" __global__ void __launch_bounds__(128) mla_rope_k_pe_f16(
+ half* __restrict__ k_pe, // [seqLen, qkRope]
+ const float* __restrict__ cos_tab, // [maxSeq, qkRope/2]
+ const float* __restrict__ sin_tab, // [maxSeq, qkRope/2]
+ const int seq_len, const int qk_rope_head_dim,
+ const int position_offset)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+ int half_d = qk_rope_head_dim / 2;
+ int pos = position_offset + t;
+
+ half* row = k_pe + (size_t)t * qk_rope_head_dim;
+ const float* cos_row = cos_tab + (size_t)pos * half_d;
+ const float* sin_row = sin_tab + (size_t)pos * half_d;
+
+ for (int i = threadIdx.x; i < half_d; i += blockDim.x)
+ {
+ float a = __half2float(row[2 * i]);
+ float b = __half2float(row[2 * i + 1]);
+ float c = cos_row[i];
+ float s = sin_row[i];
+ row[2 * i] = __float2half(a * c - b * s);
+ row[2 * i + 1] = __float2half(b * c + a * s);
+ }
+}
+
+// FP16 RMSNorm with explicit (numRows, dim) layout — FP16 input, FP32 weight,
+// FP16 output, FP32 accumulation. Matches the GQA `rmsnorm_f16` precision
+// pattern but with explicit (numRows, dim) launch-level addressing.
+//
+// Note: weight is FP32 here because MLA's q_a_layernorm / kv_a_layernorm
+// weight tensors are typically already F32 on the safetensors side; uploading
+// them as F32 saves one F16 round-trip at kernel time and keeps the dot
+// against (input * inv) in FP32. (This matches the existing F32 path's weight
+// type, just with FP16 input/output.)
+extern "C" __global__ void __launch_bounds__(128) mla_rmsnorm_f16(
+ const half* __restrict__ input,
+ const float* __restrict__ weight,
+ half* __restrict__ output,
+ const int dim, const float epsilon)
+{
+ int row = blockIdx.x;
+ const half* in_row = input + (size_t)row * dim;
+ half* out_row = output + (size_t)row * dim;
+
+ // Sum of squares — FP32 accumulation.
+ float sum_sq = 0.0f;
+ for (int i = threadIdx.x; i < dim; i += blockDim.x)
+ {
+ float x = __half2float(in_row[i]);
+ sum_sq = fmaf(x, x, sum_sq);
+ }
+
+ __shared__ float warp_sums[4];
+ int lane = threadIdx.x & 31;
+ int warp = threadIdx.x >> 5;
+
+ for (int off = 16; off > 0; off >>= 1)
+ sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, off);
+ if (lane == 0) warp_sums[warp] = sum_sq;
+ __syncthreads();
+
+ if (warp == 0)
+ {
+ sum_sq = (lane < (blockDim.x + 31) / 32) ? warp_sums[lane] : 0.0f;
+ for (int off = 16; off > 0; off >>= 1)
+ sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, off);
+ if (lane == 0) warp_sums[0] = sum_sq;
+ }
+ __syncthreads();
+
+ float rms = sqrtf(warp_sums[0] / (float)dim + epsilon);
+ float inv = 1.0f / rms;
+
+ for (int i = threadIdx.x; i < dim; i += blockDim.x)
+ {
+ float x = __half2float(in_row[i]);
+ out_row[i] = __float2half(x * inv * weight[i]);
+ }
+}
+// ── #endregion MLA FP16 helpers ──────────────────────────────────────────
diff --git a/native/kernels/moe_ffn.cu b/native/kernels/moe_ffn.cu
new file mode 100644
index 00000000..d66b240a
--- /dev/null
+++ b/native/kernels/moe_ffn.cu
@@ -0,0 +1,352 @@
+// MoE (Mixture-of-Experts) helper kernels for the SwiGLU FFN forward path.
+// All F32 to mirror the CPU oracle (DotLLM.Cpu.Kernels.MoeSwiGluMlp.Execute /
+// ExecuteWithSharedExpert). Phase 1: a tiny set of orchestration kernels —
+// the bulk of the math (gate / up / down GEMVs, swiglu, RMSNorm) reuses
+// existing F32 launchers.
+//
+// Kernels in this file:
+// moe_softmax_topk_f32 — per-token softmax over [num_experts]
+// logits + lower-index-stable top-k pick
+// (matches MoeSwiGluMlp.SelectTopK).
+// moe_renorm_topk_f32 — per-token renormalise topk weights to sum
+// to 1.0 (Mixtral / Qwen3-MoE convention).
+// moe_zero_f32 — zero a flat F32 buffer (used to clear the
+// output before per-expert accumulation).
+// moe_axpy_f32 — y[i] += alpha * x[i] across a row, where
+// alpha is read from a host-supplied scalar.
+// moe_axpy_scaled_row_f32 — same but alpha is a fp32 read from a
+// device pointer at index `slot`. Used to
+// accumulate weight[slot] * down[hidden].
+// moe_sigmoid_logit_f32 — y = 1 / (1 + exp(-Σ x[k] * g[k])) for one
+// token (Qwen1.5-MoE shared_expert_gate).
+
+#include
+
+// ── Helpers ──────────────────────────────────────────────────────────────
+
+// Block-wide reduction (sum / max) using shared memory + warp shuffles.
+// Block size assumed to be 128 threads (4 warps).
+__device__ __forceinline__ float block_reduce_sum_128(float v, float* shared)
+{
+ int lane = threadIdx.x & 31;
+ int warp = threadIdx.x >> 5;
+
+ for (int off = 16; off > 0; off >>= 1)
+ v += __shfl_down_sync(0xFFFFFFFF, v, off);
+ if (lane == 0) shared[warp] = v;
+ __syncthreads();
+
+ if (warp == 0)
+ {
+ v = (lane < 4) ? shared[lane] : 0.0f;
+ for (int off = 2; off > 0; off >>= 1)
+ v += __shfl_down_sync(0xFFFFFFFF, v, off);
+ if (lane == 0) shared[0] = v;
+ }
+ __syncthreads();
+ return shared[0];
+}
+
+__device__ __forceinline__ float block_reduce_max_128(float v, float* shared)
+{
+ int lane = threadIdx.x & 31;
+ int warp = threadIdx.x >> 5;
+
+ for (int off = 16; off > 0; off >>= 1)
+ {
+ float other = __shfl_down_sync(0xFFFFFFFF, v, off);
+ if (other > v) v = other;
+ }
+ if (lane == 0) shared[warp] = v;
+ __syncthreads();
+
+ if (warp == 0)
+ {
+ v = (lane < 4) ? shared[lane] : -INFINITY;
+ for (int off = 2; off > 0; off >>= 1)
+ {
+ float other = __shfl_down_sync(0xFFFFFFFF, v, off);
+ if (other > v) v = other;
+ }
+ if (lane == 0) shared[0] = v;
+ }
+ __syncthreads();
+ return shared[0];
+}
+
+// ── moe_softmax_topk_f32 ─────────────────────────────────────────────────
+//
+// Per-token: softmax over `num_experts` logits, then select the top-k largest
+// entries (with ties broken in favour of the lower index — matches
+// torch.topk's CPU forward-order behaviour and the CPU oracle's
+// MoeSwiGluMlp.SelectTopK).
+//
+// Output (per token):
+// topk_idx[t * top_k + slot] : int32 — chosen expert index
+// topk_weight[t * top_k + slot] : float — softmax probability
+//
+// This kernel does NOT renormalise the top-k weights — the renormalise step
+// runs as a separate kernel so the caller can disable it for
+// `norm_topk_prob=false` configs (Qwen1.5-MoE).
+//
+// Layout: one block per token, 128 threads. We fan out over the experts in a
+// strided loop. The top-k selection is a serialised single-thread max-scan
+// (k iterations × num_experts comparisons). For the configurations we care
+// about (E ≤ 256, k ≤ 8) this is ~2k cycles — negligible compared to the
+// router GEMV / per-expert SwiGLU compute and easy to keep bit-near-identical
+// to the CPU oracle.
+extern "C" __global__ void __launch_bounds__(128) moe_softmax_topk_f32(
+ const float* __restrict__ logits, // [seq_len, num_experts]
+ int* __restrict__ topk_idx, // [seq_len, top_k]
+ float* __restrict__ topk_weight, // [seq_len, top_k]
+ const int seq_len, const int num_experts, const int top_k)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ extern __shared__ float s_softmax[];
+ // Layout:
+ // s_softmax[0 .. num_experts) : softmax probabilities
+ // s_softmax[num_experts .. n+4) : warp-reduce scratch (4 floats)
+ float* s_probs = s_softmax;
+ float* s_scratch = s_softmax + num_experts;
+
+ const float* row = logits + (size_t)t * num_experts;
+
+ // Pass 1: row max.
+ float local_max = -INFINITY;
+ for (int i = threadIdx.x; i < num_experts; i += blockDim.x)
+ {
+ float v = row[i];
+ if (v > local_max) local_max = v;
+ }
+ float row_max = block_reduce_max_128(local_max, s_scratch);
+
+ // Pass 2: exp(x - max), accumulate sum.
+ float local_sum = 0.0f;
+ for (int i = threadIdx.x; i < num_experts; i += blockDim.x)
+ {
+ float e = expf(row[i] - row_max);
+ s_probs[i] = e;
+ local_sum += e;
+ }
+ float row_sum = block_reduce_sum_128(local_sum, s_scratch);
+ float inv_sum = 1.0f / row_sum;
+
+ // Pass 3: divide.
+ for (int i = threadIdx.x; i < num_experts; i += blockDim.x)
+ s_probs[i] = s_probs[i] * inv_sum;
+ __syncthreads();
+
+ // Pass 4: top-k selection. Single-threaded max-scan with index-stable ties
+ // (matches MoeSwiGluMlp.SelectTopK exactly: strict `>` keeps the lower
+ // index on ties).
+ if (threadIdx.x == 0)
+ {
+ // Local stack-resident "claimed" array and outputs. top_k is small
+ // (typically 2..8); cap at 64 to be safe — DeepSeek-V2 / V3 use 6.
+ int sel_idx[64];
+ float sel_prob[64];
+ for (int slot = 0; slot < top_k; slot++)
+ {
+ int best_idx = -1;
+ float best_val = -INFINITY;
+ for (int i = 0; i < num_experts; i++)
+ {
+ bool claimed = false;
+ for (int p = 0; p < slot; p++) { if (sel_idx[p] == i) { claimed = true; break; } }
+ if (claimed) continue;
+ float v = s_probs[i];
+ if (v > best_val) { best_val = v; best_idx = i; }
+ }
+ sel_idx[slot] = best_idx;
+ sel_prob[slot] = best_val;
+ }
+ for (int slot = 0; slot < top_k; slot++)
+ {
+ topk_idx[(size_t)t * top_k + slot] = sel_idx[slot];
+ topk_weight[(size_t)t * top_k + slot] = sel_prob[slot];
+ }
+ }
+}
+
+// ── moe_renorm_topk_f32 ──────────────────────────────────────────────────
+//
+// In-place per-token renormalisation of the top-k weights so they sum to 1.0
+// (Mixtral / Qwen3-MoE convention; the CPU oracle does this iff
+// `normTopKProb` is true). Single-threaded per token — top_k is tiny.
+extern "C" __global__ void __launch_bounds__(32) moe_renorm_topk_f32(
+ float* __restrict__ topk_weight, // [seq_len, top_k]
+ const int seq_len, const int top_k)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+ if (threadIdx.x != 0) return;
+
+ float* row = topk_weight + (size_t)t * top_k;
+ float sum = 0.0f;
+ for (int i = 0; i < top_k; i++) sum += row[i];
+ float inv = sum > 0.0f ? 1.0f / sum : 0.0f;
+ for (int i = 0; i < top_k; i++) row[i] *= inv;
+}
+
+// ── moe_zero_f32 ─────────────────────────────────────────────────────────
+//
+// Zero a flat F32 buffer. The CPU oracle clears the per-token accumulator at
+// the start of FinalAccumulate; the GPU equivalent is to clear `output`
+// before the per-expert weighted-add loop begins.
+extern "C" __global__ void __launch_bounds__(256) moe_zero_f32(
+ float* __restrict__ buf, const int n)
+{
+ int idx = blockIdx.x * blockDim.x + threadIdx.x;
+ int stride = gridDim.x * blockDim.x;
+ for (int i = idx; i < n; i += stride)
+ buf[i] = 0.0f;
+}
+
+// ── moe_axpy_scaled_row_f32 ──────────────────────────────────────────────
+//
+// Per-token weighted accumulator:
+// out[t * hidden + i] += weight[t * top_k + slot] * down[t * hidden + i]
+//
+// Used after each expert's down projection lands its rows into
+// `down_per_token` (one row per output-side selected token). Reading the
+// scalar weight from device memory keeps the GPU pipeline pure — no extra
+// HtoD per-expert.
+//
+// Grid: one block per token. Block size 256 with strided loop over `hidden`.
+// `weight_offset_in_row` is the slot index inside each token's top_k stripe.
+extern "C" __global__ void __launch_bounds__(256) moe_axpy_scaled_row_f32(
+ float* __restrict__ output, // [seq_len, hidden]
+ const float* __restrict__ down, // [seq_len, hidden] (one row per token in this token-set)
+ const float* __restrict__ topk_weight, // [seq_len, top_k]
+ const int* __restrict__ token_indices, // [batch] absolute token ids ⇒ which output row each batch row updates
+ const int batch_size,
+ const int hidden,
+ const int top_k,
+ const int slot_index)
+{
+ int b = blockIdx.x;
+ if (b >= batch_size) return;
+
+ int t = token_indices[b];
+ float w = topk_weight[(size_t)t * top_k + slot_index];
+
+ if (w == 0.0f) return;
+
+ const float* down_row = down + (size_t)b * hidden;
+ float* out_row = output + (size_t)t * hidden;
+
+ for (int i = threadIdx.x; i < hidden; i += blockDim.x)
+ {
+ float v = down_row[i] * w;
+ // Atomic-free: each token row is written by exactly one block here
+ // (batch is a permutation of distinct token ids per expert, and
+ // different experts run in serialised launches on the same stream).
+ out_row[i] = out_row[i] + v;
+ }
+}
+
+// ── moe_axpy_unweighted_row_f32 ──────────────────────────────────────────
+//
+// Plain accumulator: `out[t * hidden + i] += down[t * hidden + i]`. Used by
+// the shared-expert path (output is summed unconditionally; DeepSeek has no
+// shared_expert_gate). Per-token over all `seq_len` tokens — no batch
+// indirection.
+extern "C" __global__ void __launch_bounds__(256) moe_axpy_unweighted_f32(
+ float* __restrict__ output, // [seq_len, hidden]
+ const float* __restrict__ down, // [seq_len, hidden]
+ const int seq_len, const int hidden)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ const float* down_row = down + (size_t)t * hidden;
+ float* out_row = output + (size_t)t * hidden;
+
+ for (int i = threadIdx.x; i < hidden; i += blockDim.x)
+ out_row[i] = out_row[i] + down_row[i];
+}
+
+// ── moe_axpy_scaled_per_token_f32 ────────────────────────────────────────
+//
+// Per-token sigmoid-gated accumulator for Qwen1.5-MoE shared_expert_gate:
+// out[t, i] += scale[t] * down[t, i]
+//
+// Used when the shared-expert branch carries an explicit per-token sigmoid
+// scale. DeepSeek-V2/V3 do NOT use this — they pass an empty gate and call
+// the unweighted variant instead.
+extern "C" __global__ void __launch_bounds__(256) moe_axpy_scaled_per_token_f32(
+ float* __restrict__ output, // [seq_len, hidden]
+ const float* __restrict__ down, // [seq_len, hidden]
+ const float* __restrict__ scale, // [seq_len]
+ const int seq_len, const int hidden)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ float s = scale[t];
+ if (s == 0.0f) return;
+
+ const float* down_row = down + (size_t)t * hidden;
+ float* out_row = output + (size_t)t * hidden;
+
+ for (int i = threadIdx.x; i < hidden; i += blockDim.x)
+ out_row[i] = out_row[i] + s * down_row[i];
+}
+
+// ── moe_sigmoid_logit_f32 ────────────────────────────────────────────────
+//
+// Per-token sigmoid-gated dot product:
+// scale[t] = 1 / (1 + exp(-Σ x[t, k] * g[k]))
+//
+// Used by Qwen1.5-MoE shared_expert_gate (a [hidden] weight vector). One
+// block per token, 128 threads. Trivial cost (single dot product per
+// token); kept as a dedicated kernel rather than going through cuBLAS
+// because the output is a scalar per token and we want to fold the
+// sigmoid in the same launch.
+extern "C" __global__ void __launch_bounds__(128) moe_sigmoid_logit_f32(
+ const float* __restrict__ hidden, // [seq_len, hidden_size]
+ const float* __restrict__ g, // [hidden_size]
+ float* __restrict__ scale_out, // [seq_len]
+ const int seq_len, const int hidden_size)
+{
+ int t = blockIdx.x;
+ if (t >= seq_len) return;
+
+ extern __shared__ float s_scratch[];
+
+ const float* x_row = hidden + (size_t)t * hidden_size;
+ float local = 0.0f;
+ for (int i = threadIdx.x; i < hidden_size; i += blockDim.x)
+ local += x_row[i] * g[i];
+
+ float total = block_reduce_sum_128(local, s_scratch);
+
+ if (threadIdx.x == 0)
+ scale_out[t] = 1.0f / (1.0f + expf(-total));
+}
+
+// ── moe_gather_token_rows_f32 ────────────────────────────────────────────
+//
+// Gathers `batch_size` rows from a [seq_len, hidden] source into a contiguous
+// [batch_size, hidden] destination, indexed by `token_indices[b]` for b ∈
+// [0, batch_size). Used by the per-expert grouped path: when expert e has
+// been routed to by tokens {t0, t1, ...}, we copy those tokens' hidden rows
+// into a contiguous batch buffer for the SwiGLU GEMMs.
+extern "C" __global__ void __launch_bounds__(256) moe_gather_token_rows_f32(
+ const float* __restrict__ src, // [seq_len, hidden]
+ float* __restrict__ dst, // [batch_size, hidden]
+ const int* __restrict__ token_indices, // [batch_size]
+ const int batch_size, const int hidden)
+{
+ int b = blockIdx.x;
+ if (b >= batch_size) return;
+ int t = token_indices[b];
+
+ const float* src_row = src + (size_t)t * hidden;
+ float* dst_row = dst + (size_t)b * hidden;
+
+ for (int i = threadIdx.x; i < hidden; i += blockDim.x)
+ dst_row[i] = src_row[i];
+}
diff --git a/native/kernels/moe_grouped_gemv.cu b/native/kernels/moe_grouped_gemv.cu
new file mode 100644
index 00000000..abd02a7d
--- /dev/null
+++ b/native/kernels/moe_grouped_gemv.cu
@@ -0,0 +1,559 @@
+// MoE grouped quantized GEMV: walks K_active experts in a single launch.
+//
+// Each expert e in [0, K_active) computes:
+// outputs[e][m] = W_e[m, k] @ x[k] for m in [0, M)
+//
+// where x is a single [K] FP16 input row shared across all experts (decode
+// batch=1 — all K_active routed experts feed off the same hidden vector for
+// that one token), and each W_e is a [M, K] quantized matrix with raw blocks
+// pointed at by weights[e].
+//
+// Grid: (M, K_active, 1). Each block computes one (expert, output_row) pair.
+// Block: 256 threads. Body matches `quantized_gemv_q4_k` — FP32 accumulation,
+// warp-then-block reduction, single half store at the end.
+//
+// Shared input x is read by all K_active experts; we rely on L1 / SM scope
+// to coalesce reads across blocks issued in the same SM (typical K=2048
+// ⇒ 4 KB ⇒ fits in L1). Per-block we don't __shared__-cache it because the
+// inner loop already streams via half-unrolled reads with good locality.
+
+#include
+#include
+
+__device__ __constant__ int8_t kvalues_iq4nl_grouped[16] = {
+ -127, -104, -83, -65, -49, -35, -22, -10,
+ 1, 13, 25, 38, 53, 69, 89, 113
+};
+
+// ── Q2_K: 84 bytes per 256 values ───────────────────────────────────
+// struct block_q2_K { uint8_t scales[16]; uint8_t qs[64]; half d; half dmin; };
+// scales: 16 × (4-bit scale | 4-bit dmin coef), one byte per sub-block of 16 elements
+// qs: 2 bits per element (256 elements × 2 bits = 64 bytes)
+// d: FP16 super-block delta
+// dmin: FP16 super-block min delta
+// Body mirrors quantized_gemv_q2_k — per-expert weight pointer indirection at
+// block.y plus per-expert output writes at block.y. FP32 accumulation, single
+// FP16 store at y[row].
+
+extern "C" __global__ void __launch_bounds__(256, 2) moe_grouped_gemv_q2_k_f16(
+ const half* __restrict__ x, // [K]
+ const uint8_t* const* __restrict__ weights, // [K_active] per-expert ptrs
+ half* const* __restrict__ outputs, // [K_active] per-expert ptrs
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int superblocks_per_row = K / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ // 16 sub-blocks of 16 elements
+ for (int sub = 0; sub < 16; sub++) {
+ int sc = scales[sub] & 0xF;
+ int dm = (scales[sub] >> 4) & 0xF;
+
+ float sub_acc = 0.0f;
+ float xsum_sub = 0.0f;
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++) {
+ int t = sub * 16 + j;
+ int byte_idx = t >> 2;
+ int bit_off = (t & 0x3) << 1;
+ int q2 = (qs[byte_idx] >> bit_off) & 0x3;
+ float xv = __half2float(x[sb * 256 + t]);
+ sub_acc += (float)q2 * xv;
+ xsum_sub += xv;
+ }
+ acc += d * (float)sc * sub_acc - dmin * (float)dm * xsum_sub;
+ }
+ }
+
+ // Warp reduction
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0) {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+ if (threadIdx.x == 0) y[row] = __float2half(acc);
+}
+
+// ── Q4_K: 144 bytes per 256 values ──────────────────────────────────
+
+extern "C" __global__ void __launch_bounds__(256, 2) moe_grouped_gemv_q4_k_f16(
+ const half* __restrict__ x, // [K]
+ const uint8_t* const* __restrict__ weights, // [K_active] per-expert ptrs
+ half* const* __restrict__ outputs, // [K_active] per-expert ptrs
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int superblocks_per_row = K / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 144;
+
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 144;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qs = block + 16;
+
+ // Q4_K: 4 pairs of sub-blocks, each pair shares 32 qs bytes.
+ // Lower nibbles → even sub-block, upper nibbles → odd sub-block.
+ for (int pair = 0; pair < 4; pair++)
+ {
+ int sb_even = pair * 2;
+ int sb_odd = pair * 2 + 1;
+
+ int sc0, m0, sc1, m1;
+ if (sb_even < 4)
+ {
+ sc0 = scales_raw[sb_even] & 0x3F;
+ m0 = scales_raw[sb_even + 4] & 0x3F;
+ sc1 = scales_raw[sb_odd] & 0x3F;
+ m1 = scales_raw[sb_odd + 4] & 0x3F;
+ }
+ else
+ {
+ sc0 = (scales_raw[sb_even + 4] & 0x0F) | ((scales_raw[sb_even - 4] >> 6) << 4);
+ m0 = (scales_raw[sb_even + 4] >> 4) | ((scales_raw[sb_even] >> 6) << 4);
+ sc1 = (scales_raw[sb_odd + 4] & 0x0F) | ((scales_raw[sb_odd - 4] >> 6) << 4);
+ m1 = (scales_raw[sb_odd + 4] >> 4) | ((scales_raw[sb_odd] >> 6) << 4);
+ }
+
+ float scale0 = d * (float)sc0;
+ float min0 = dmin * (float)m0;
+ float scale1 = d * (float)sc1;
+ float min1 = dmin * (float)m1;
+
+ const uint8_t* pair_qs = qs + pair * 32;
+ int base_x_idx = sb * 256 + pair * 64;
+
+ for (int j = 0; j < 32; j++)
+ {
+ uint8_t byte_val = pair_qs[j];
+ float x_even = __half2float(x[base_x_idx + j]);
+ float x_odd = __half2float(x[base_x_idx + j + 32]);
+
+ acc += (scale0 * (float)(byte_val & 0x0F) - min0) * x_even;
+ acc += (scale1 * (float)(byte_val >> 4) - min1) * x_odd;
+ }
+ }
+ }
+
+ // Warp reduction
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
+// ── Q5_K: 176 bytes per 256 values ──────────────────────────────────
+// struct block_q5_K { half d, dmin; uint8_t scales[12]; uint8_t qh[32]; uint8_t qs[128]; };
+
+extern "C" __global__ void __launch_bounds__(256, 2) moe_grouped_gemv_q5_k_f16(
+ const half* __restrict__ x,
+ const uint8_t* const* __restrict__ weights,
+ half* const* __restrict__ outputs,
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int superblocks_per_row = K / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 176;
+
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 176;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qh = block + 16; // 32 bytes
+ const uint8_t* qs = block + 48; // 128 bytes
+ int base_x = sb * 256;
+
+ for (int sub = 0; sub < 8; sub++)
+ {
+ int sc, m;
+ if (sub < 4)
+ {
+ sc = scales_raw[sub] & 0x3F;
+ m = scales_raw[sub + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sub + 4] & 0x0F) | ((scales_raw[sub - 4] >> 6) << 4);
+ m = (scales_raw[sub + 4] >> 4) | ((scales_raw[sub] >> 6) << 4);
+ }
+
+ float scale = d * (float)sc;
+ float min_val = dmin * (float)m;
+
+ const uint8_t* sub_qs = qs + sub * 16;
+ const uint8_t* sub_qh = qh + sub * 4;
+ int x_off = base_x + sub * 32;
+
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = sub_qs[j];
+ int bit_lo = (sub_qh[j / 4] >> ((j % 4) * 2)) & 1;
+ int bit_hi = (sub_qh[j / 4] >> ((j % 4) * 2 + 1)) & 1;
+ int lo = (packed & 0x0F) | (bit_lo << 4);
+ int hi = (packed >> 4) | (bit_hi << 4);
+
+ acc += (scale * (float)lo - min_val) * __half2float(x[x_off + 2 * j]);
+ acc += (scale * (float)hi - min_val) * __half2float(x[x_off + 2 * j + 1]);
+ }
+ }
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
+// ── Q6_K: 210 bytes per 256 values ──────────────────────────────────
+
+extern "C" __global__ void __launch_bounds__(256, 2) moe_grouped_gemv_q6_k_f16(
+ const half* __restrict__ x,
+ const uint8_t* const* __restrict__ weights,
+ half* const* __restrict__ outputs,
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int superblocks_per_row = K / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 210;
+
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 210;
+ const uint8_t* ql = block;
+ const uint8_t* qh = block + 128;
+ const int8_t* scales = reinterpret_cast(block + 192);
+ float d = __half2float(*reinterpret_cast(block + 208));
+
+ int base_x = sb * 256;
+
+ for (int half_idx = 0; half_idx < 2; half_idx++)
+ {
+ const uint8_t* ql_half = ql + half_idx * 64;
+ const uint8_t* qh_half = qh + half_idx * 32;
+ const int8_t* sc_half = scales + half_idx * 8;
+ int x_off = base_x + half_idx * 128;
+
+ for (int l = 0; l < 32; l++)
+ {
+ int isc = l / 16;
+
+ int q1 = ((ql_half[l] & 0x0F) | (((qh_half[l] >> 0) & 3) << 4)) - 32;
+ int q2 = ((ql_half[l + 32] & 0x0F) | (((qh_half[l] >> 2) & 3) << 4)) - 32;
+ int q3 = ((ql_half[l] >> 4) | (((qh_half[l] >> 4) & 3) << 4)) - 32;
+ int q4 = ((ql_half[l + 32] >> 4) | (((qh_half[l] >> 6) & 3) << 4)) - 32;
+
+ float s0 = d * (float)sc_half[isc];
+ float s1 = d * (float)sc_half[isc + 2];
+ float s2 = d * (float)sc_half[isc + 4];
+ float s3 = d * (float)sc_half[isc + 6];
+
+ acc += s0 * (float)q1 * __half2float(x[x_off + l]);
+ acc += s1 * (float)q2 * __half2float(x[x_off + l + 32]);
+ acc += s2 * (float)q3 * __half2float(x[x_off + l + 64]);
+ acc += s3 * (float)q4 * __half2float(x[x_off + l + 96]);
+ }
+ }
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
+// ── Q8_0: 34 bytes per 32 values ────────────────────────────────────
+// Note: K must be a multiple of 32 (the Q8_0 block size). Q5_K/Q6_K above
+// require K%256==0, but Q8_0 has finer granularity — same K%256==0 gate at
+// the dispatch layer keeps the API uniform.
+
+extern "C" __global__ void __launch_bounds__(256) moe_grouped_gemv_iq4_nl_f16(
+ const half* __restrict__ x,
+ const uint8_t* const* __restrict__ weights,
+ half* const* __restrict__ outputs,
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int blocks_per_row = K / 32;
+ const uint8_t* w_row = weight + (size_t)row * blocks_per_row * 18;
+ float acc = 0.0f;
+
+ for (int b = threadIdx.x; b < blocks_per_row; b += blockDim.x)
+ {
+ const uint8_t* block = w_row + b * 18;
+ float d = __half2float(*reinterpret_cast(block));
+ const uint8_t* qs = block + 2;
+
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = qs[j];
+ acc += d * (float)kvalues_iq4nl_grouped[packed & 0x0F] * __half2float(x[b * 32 + j]);
+ acc += d * (float)kvalues_iq4nl_grouped[packed >> 4] * __half2float(x[b * 32 + j + 16]);
+ }
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) moe_grouped_gemv_iq4_xs_f16(
+ const half* __restrict__ x,
+ const uint8_t* const* __restrict__ weights,
+ half* const* __restrict__ outputs,
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int superblocks_per_row = K / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 136;
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 136;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+ int base_x = sb * 256;
+
+ for (int ib = 0; ib < 8; ib++)
+ {
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+ const uint8_t* sub_qs = qs + ib * 16;
+ int x_off = base_x + ib * 32;
+
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = sub_qs[j];
+ acc += dl * (float)kvalues_iq4nl_grouped[packed & 0x0F] * __half2float(x[x_off + j]);
+ acc += dl * (float)kvalues_iq4nl_grouped[packed >> 4] * __half2float(x[x_off + j + 16]);
+ }
+ }
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
+extern "C" __global__ void __launch_bounds__(256) moe_grouped_gemv_q8_0_f16(
+ const half* __restrict__ x,
+ const uint8_t* const* __restrict__ weights,
+ half* const* __restrict__ outputs,
+ const int M,
+ const int K,
+ const int K_active)
+{
+ int expert_idx = blockIdx.y;
+ int row = blockIdx.x;
+ if (expert_idx >= K_active || row >= M) return;
+
+ const uint8_t* weight = weights[expert_idx];
+ half* y = outputs[expert_idx];
+
+ const int blocks_per_row = K / 32;
+ const uint8_t* w_row = weight + (size_t)row * blocks_per_row * 34;
+
+ float acc = 0.0f;
+
+ for (int b = threadIdx.x; b < blocks_per_row; b += blockDim.x)
+ {
+ const uint8_t* block = w_row + b * 34;
+ float d = __half2float(*reinterpret_cast(block));
+ const int8_t* qs = reinterpret_cast(block + 2);
+
+ float block_sum = 0.0f;
+ #pragma unroll 8
+ for (int j = 0; j < 32; j++)
+ block_sum += (float)qs[j] * __half2float(x[b * 32 + j]);
+
+ acc += d * block_sum;
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
diff --git a/native/kernels/per_head_rmsnorm.cu b/native/kernels/per_head_rmsnorm.cu
index 1f806a8d..95884247 100644
--- a/native/kernels/per_head_rmsnorm.cu
+++ b/native/kernels/per_head_rmsnorm.cu
@@ -1,5 +1,10 @@
// Per-head RMS normalization kernel (QK-norm, Qwen3-style) for dotLLM.
// Normalizes each head vector independently: qk[t, h, :headDim]
+//
+// Optimizations:
+// * half2 vectorized loads/stores
+// * __shfl_xor_sync warp reduction
+// * Pre-folds 1/head_dim via fmaf
#include
@@ -15,48 +20,70 @@ extern "C" __global__ void __launch_bounds__(256) per_head_rmsnorm_f16(
int block_id = blockIdx.x;
int t = block_id / num_heads;
int h = block_id % num_heads;
-
if (t >= seq_len) return;
int stride = num_heads * head_dim;
half* vec = qk + (size_t)t * stride + h * head_dim;
+ const int tid = threadIdx.x;
+ const int hd2 = head_dim >> 1;
+
+ const half2* __restrict__ v2_in = reinterpret_cast(vec);
+ half2* __restrict__ v2_out = reinterpret_cast(vec);
+ const half2* __restrict__ w2 = reinterpret_cast(weight);
- // Compute sum of squares
+ // ── Pass 1: sum of squares ──
float sum_sq = 0.0f;
- for (int i = threadIdx.x; i < head_dim; i += blockDim.x)
+ for (int i = tid; i < hd2; i += blockDim.x)
+ {
+ half2 v = v2_in[i];
+ float v0 = __low2float(v), v1 = __high2float(v);
+ sum_sq = fmaf(v0, v0, sum_sq);
+ sum_sq = fmaf(v1, v1, sum_sq);
+ }
+ if ((head_dim & 1) && tid == 0)
{
- float v = __half2float(vec[i]);
- sum_sq += v * v;
+ float v = __half2float(vec[head_dim - 1]);
+ sum_sq = fmaf(v, v, sum_sq);
}
- // Warp reduction
- for (int offset = warpSize / 2; offset > 0; offset >>= 1)
- sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, offset);
+ // ── Warp reduction (symmetric) ──
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ sum_sq += __shfl_xor_sync(0xFFFFFFFF, sum_sq, offset);
__shared__ float warp_sums[32];
- int lane = threadIdx.x % warpSize;
- int warp_id = threadIdx.x / warpSize;
+ int lane = tid & 31;
+ int warp_id = tid >> 5;
if (lane == 0) warp_sums[warp_id] = sum_sq;
__syncthreads();
if (warp_id == 0)
{
- int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ int num_warps = (blockDim.x + 31) >> 5;
sum_sq = (lane < num_warps) ? warp_sums[lane] : 0.0f;
- for (int offset = warpSize / 2; offset > 0; offset >>= 1)
- sum_sq += __shfl_down_sync(0xFFFFFFFF, sum_sq, offset);
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ sum_sq += __shfl_xor_sync(0xFFFFFFFF, sum_sq, offset);
+ if (lane == 0)
+ warp_sums[0] = rsqrtf(fmaf(sum_sq, 1.0f / (float)head_dim, eps));
}
-
- __shared__ float rms_inv;
- if (threadIdx.x == 0)
- rms_inv = rsqrtf(sum_sq / (float)head_dim + eps);
__syncthreads();
+ const float rms_inv = warp_sums[0];
- // Normalize and scale
- for (int i = threadIdx.x; i < head_dim; i += blockDim.x)
+ // ── Pass 2: vectorized normalize + scale ──
+ for (int i = tid; i < hd2; i += blockDim.x)
+ {
+ half2 v = v2_in[i];
+ half2 wh = w2[i];
+ float v0 = __low2float(v), v1 = __high2float(v);
+ float w0 = __low2float(wh), w1 = __high2float(wh);
+ v2_out[i] = __floats2half2_rn(v0 * rms_inv * w0, v1 * rms_inv * w1);
+ }
+ if ((head_dim & 1) && tid == 0)
{
- float v = __half2float(vec[i]);
- float w = __half2float(weight[i]);
- vec[i] = __float2half(v * rms_inv * w);
+ int last = head_dim - 1;
+ float v = __half2float(vec[last]);
+ float w = __half2float(weight[last]);
+ vec[last] = __float2half(v * rms_inv * w);
}
}
diff --git a/native/kernels/quant_kv.cu b/native/kernels/quant_kv.cu
index 9caddcdf..f2385e3c 100644
--- a/native/kernels/quant_kv.cu
+++ b/native/kernels/quant_kv.cu
@@ -1,6 +1,18 @@
// ─────────────────────────────────────────────────────────────────────
// KV-cache quantization kernels: FP16 → Q8_0 and FP16 → Q4_0
// Used for quantize-on-evict in CudaQuantizedKvCache.
+//
+// Two flavours per quant type:
+// quant_f16_to_q{8_0,4_0} — eager path. Host computes src/dst
+// and how many blocks to write.
+// quant_f16_to_q{8_0,4_0}_dyn — graph-capture path. Reads the decode
+// position from a device-resident int and
+// decides per launch whether to quantize
+// the just-evicted FP16 row, derives
+// src/dst addresses device-side. Predicated:
+// until the FP16 window fills, the kernel
+// no-ops. This keeps the graph topology
+// constant across decode steps.
// ─────────────────────────────────────────────────────────────────────
#include
@@ -107,3 +119,132 @@ extern "C" __global__ void __launch_bounds__(256) quant_f16_to_q4_0(
}
}
}
+
+// ── Dynamic (graph-capture) variants ────────────────────────────────
+//
+// On every decode step we launch one of these per layer per K/V. The
+// kernel reads the absolute decode position from `pos_ptr` and decides:
+// evict_pos = pos - window_size; // FP16 row that just fell out
+// if (evict_pos < 0) return; // window not full yet → no-op
+// Source row sits in the FP16 ring buffer at slot `evict_pos % window_size`
+// (kv_stride FP16 elements). Destination is the quantized row at row index
+// `evict_pos` in the per-layer Q-cache.
+//
+// Launch geometry: one CUDA block of `kv_stride / 32` threads (caller
+// rounds the block size up to a multiple of warp size). The grid is fixed
+// at (1, 1, 1) — all blocks-of-32 for a single row fit in one CUDA block
+// for typical kv_stride values (≤ 8192 ≡ 256 quant blocks ≡ 256 threads).
+//
+// `kv_stride` is the row width in FP16 elements ( = num_kv_heads * head_dim ).
+// `total_blocks_per_row` = `kv_stride / 32`.
+
+extern "C" __global__ void quant_f16_to_q8_0_dyn(
+ const half* __restrict__ window_base, // [window_size, kv_stride] FP16 ring
+ uint8_t* __restrict__ quant_base, // [maxSeqLen, kv_stride/32 * 34] Q8_0
+ const int kv_stride, // row width (FP16 elements)
+ const int window_size, // ring slots
+ const int* __restrict__ pos_ptr) // device-resident decode position
+{
+ int pos = pos_ptr[0];
+ int evict_pos = pos - window_size;
+ if (evict_pos < 0) return; // predicated no-op
+
+ int block_idx_in_row = blockIdx.x * blockDim.x + threadIdx.x;
+ int total_blocks_per_row = kv_stride / Q8_0_BLOCK_SIZE;
+ if (block_idx_in_row >= total_blocks_per_row) return;
+
+ int ring_slot = evict_pos % window_size;
+ const half* in = window_base
+ + (size_t)ring_slot * kv_stride
+ + (size_t)block_idx_in_row * Q8_0_BLOCK_SIZE;
+ uint8_t* out = quant_base
+ + (size_t)evict_pos * total_blocks_per_row * Q8_0_BLOCK_BYTES
+ + (size_t)block_idx_in_row * Q8_0_BLOCK_BYTES;
+
+ float max_abs = 0.0f;
+ float vals[Q8_0_BLOCK_SIZE];
+ #pragma unroll 8
+ for (int j = 0; j < Q8_0_BLOCK_SIZE; j++)
+ {
+ vals[j] = __half2float(in[j]);
+ float a = fabsf(vals[j]);
+ if (a > max_abs) max_abs = a;
+ }
+
+ float d = max_abs / 127.0f;
+ *reinterpret_cast(out) = __float2half(d);
+
+ int8_t* qs = reinterpret_cast(out + 2);
+ if (d == 0.0f)
+ {
+ #pragma unroll 8
+ for (int j = 0; j < Q8_0_BLOCK_SIZE; j++)
+ qs[j] = 0;
+ }
+ else
+ {
+ float inv_d = 1.0f / d;
+ #pragma unroll 8
+ for (int j = 0; j < Q8_0_BLOCK_SIZE; j++)
+ {
+ int v = __float2int_rn(vals[j] * inv_d);
+ qs[j] = (int8_t)max(-127, min(127, v));
+ }
+ }
+}
+
+extern "C" __global__ void quant_f16_to_q4_0_dyn(
+ const half* __restrict__ window_base,
+ uint8_t* __restrict__ quant_base,
+ const int kv_stride,
+ const int window_size,
+ const int* __restrict__ pos_ptr)
+{
+ int pos = pos_ptr[0];
+ int evict_pos = pos - window_size;
+ if (evict_pos < 0) return;
+
+ int block_idx_in_row = blockIdx.x * blockDim.x + threadIdx.x;
+ int total_blocks_per_row = kv_stride / Q4_0_BLOCK_SIZE;
+ if (block_idx_in_row >= total_blocks_per_row) return;
+
+ int ring_slot = evict_pos % window_size;
+ const half* in = window_base
+ + (size_t)ring_slot * kv_stride
+ + (size_t)block_idx_in_row * Q4_0_BLOCK_SIZE;
+ uint8_t* out = quant_base
+ + (size_t)evict_pos * total_blocks_per_row * Q4_0_BLOCK_BYTES
+ + (size_t)block_idx_in_row * Q4_0_BLOCK_BYTES;
+
+ float max_abs = 0.0f;
+ float vals[Q4_0_BLOCK_SIZE];
+ #pragma unroll 8
+ for (int j = 0; j < Q4_0_BLOCK_SIZE; j++)
+ {
+ vals[j] = __half2float(in[j]);
+ float a = fabsf(vals[j]);
+ if (a > max_abs) max_abs = a;
+ }
+
+ float d = max_abs / 7.0f;
+ *reinterpret_cast(out) = __float2half(d);
+
+ uint8_t* qs = out + 2;
+ if (d == 0.0f)
+ {
+ #pragma unroll 8
+ for (int j = 0; j < 16; j++)
+ qs[j] = 0x88;
+ }
+ else
+ {
+ float inv_d = 1.0f / d;
+ #pragma unroll 8
+ for (int j = 0; j < 16; j++)
+ {
+ int lo = max(0, min(15, __float2int_rn(vals[2 * j] * inv_d) + 8));
+ int hi = max(0, min(15, __float2int_rn(vals[2 * j + 1] * inv_d) + 8));
+ qs[j] = (uint8_t)((hi << 4) | lo);
+ }
+ }
+}
diff --git a/native/kernels/quantize_x.cu b/native/kernels/quantize_x.cu
new file mode 100644
index 00000000..641cac28
--- /dev/null
+++ b/native/kernels/quantize_x.cu
@@ -0,0 +1,79 @@
+// Pre-Q8_1 input quantization kernel.
+//
+// Quantizes an FP16 activation vector x[k] into INT8 with one FP16 scale per
+// 32-element chunk and per-half-chunk FP16 sums. Output layout matches the
+// shared-memory scratch produced by the legacy MMQ kernels' Stage 1, but lives
+// in device global memory so it can be shared across all GEMV calls in a fused
+// projection (and across QKV/GateUp/Down/O/LmHead in a forward pass).
+//
+// Layout (scratch is one contiguous device allocation per CudaForwardState):
+// int8_t xq [num_chunks * 32] // bytes [0, k)
+// half dx [num_chunks] // bytes [k, k + 2C) C = num_chunks
+// half sx2[num_chunks * 2] // bytes [k + 2C, k + 2C + 4C)
+//
+// where num_chunks = k / 32. sx2[c*2 + 0] = Σ xq[c*32 + 0..15] (lo-half),
+// sx2[c*2 + 1] = Σ xq[c*32 + 16..31] (hi-half). Q4_K / Q5_K consume the
+// full-chunk sum (sx2[c*2+0] + sx2[c*2+1]) lazily inside the GEMV kernel —
+// one extra add per chunk. Q6_K uses both halves separately (its 16-element
+// sub-blocks need per-half sums).
+//
+// Launch: blockDim = (32, 8, 1) = 256 threads. Each warp owns one chunk; one
+// warp-stride pass covers num_chunks chunks. gridDim = ceil(num_chunks / 8).
+//
+// Numerics: bit-near-identical to the legacy in-kernel Stage 1 — same rounding
+// (__float2int_rn), clamp to [-127, 127], inv_scale = 127 / max|x|, dx = max|x| / 127.
+
+#include
+#include
+
+#define QX_THREADS_X 32
+#define QX_WARPS_PER_BLOCK 8
+#define QX_THREADS (QX_THREADS_X * QX_WARPS_PER_BLOCK)
+
+extern "C" __global__ void __launch_bounds__(QX_THREADS) quantize_x_to_q8_1(
+ const half* __restrict__ x,
+ int8_t* __restrict__ xq,
+ half* __restrict__ dx,
+ half* __restrict__ sx2,
+ const int k)
+{
+ const int num_chunks = k >> 5; // k / 32
+ const int warp_id = threadIdx.y; // 0..QX_WARPS_PER_BLOCK-1
+ const int lane = threadIdx.x; // 0..31
+ const int chunk = blockIdx.x * QX_WARPS_PER_BLOCK + warp_id;
+ if (chunk >= num_chunks) return;
+
+ const int idx = chunk * 32 + lane;
+ float v = __half2float(x[idx]);
+ float a = fabsf(v);
+
+ // Full-warp max-abs reduction.
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ xq[idx] = (int8_t)qi;
+
+ // Half-warp sum reduction (offset stops at 8 — lanes 0..15 stay isolated
+ // from lanes 16..31). Each half-warp holds the sum of its 16 lanes' qi.
+ int s = qi;
+ #pragma unroll
+ for (int offset = 8; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ dx[chunk] = __float2half(a / 127.0f);
+ sx2[chunk * 2 + 0] = __float2half((float)s);
+ }
+ if (lane == 16)
+ {
+ sx2[chunk * 2 + 1] = __float2half((float)s);
+ }
+}
diff --git a/native/kernels/quantized_gemv.cu b/native/kernels/quantized_gemv.cu
index 25a1d8ce..f10f538d 100644
--- a/native/kernels/quantized_gemv.cu
+++ b/native/kernels/quantized_gemv.cu
@@ -7,6 +7,11 @@
#include
#include
+__device__ __constant__ int8_t kvalues_iq4nl_gemv[16] = {
+ -127, -104, -83, -65, -49, -35, -22, -10,
+ 1, 13, 25, 38, 53, 69, 89, 113
+};
+
// ── Q8_0: 34 bytes per 32 values ────────────────────────────────────
// scale (half) applied once per block: acc += float(scale) * sum(qs[j] * x[j])
@@ -62,6 +67,79 @@ extern "C" __global__ void __launch_bounds__(256) quantized_gemv_q8_0(
y[row] = __float2half(acc);
}
+// ── Q2_K: 84 bytes per 256 values ──────────────────────────────────
+// struct block_q2_K { uint8_t scales[16]; uint8_t qs[64]; half d; half dmin; };
+// scales: 16 × (4-bit scale | 4-bit dmin coef), one byte per sub-block of 16 elements
+// qs: 2 bits per element (256 elements × 2 bits = 64 bytes)
+// d: FP16 super-block delta
+// dmin: FP16 super-block min delta
+// Per-element value: d × sc[sub] × q2 − dmin × dm[sub]
+// Identity: Σ_i (d·sc[sub_i]·q2[i] − dmin·dm[sub_i]) · x[i]
+// = Σ_sub d·sc[sub] · Σ_j q2·x − dmin·dm[sub] · Σ_j x
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q2_k(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ // 16 sub-blocks of 16 elements
+ for (int sub = 0; sub < 16; sub++) {
+ int sc = scales[sub] & 0xF;
+ int dm = (scales[sub] >> 4) & 0xF;
+
+ float sub_acc = 0.0f;
+ float xsum_sub = 0.0f;
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++) {
+ int t = sub * 16 + j;
+ int byte_idx = t >> 2;
+ int bit_off = (t & 0x3) << 1;
+ int q2 = (qs[byte_idx] >> bit_off) & 0x3;
+ float xv = __half2float(x[sb * 256 + t]);
+ sub_acc += (float)q2 * xv;
+ xsum_sub += xv;
+ }
+ acc += d * (float)sc * sub_acc - dmin * (float)dm * xsum_sub;
+ }
+ }
+
+ // Warp reduction
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0) {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+ if (threadIdx.x == 0) y[row] = __float2half(acc);
+}
+
// ── Q4_K: 144 bytes per 256 values ──────────────────────────────────
extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q4_k(
@@ -236,6 +314,119 @@ extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q6_k(
// ── Q5_0: 22 bytes per 32 values ────────────────────────────────────
// struct block_q5_0 { half d; uint32_t qh; uint8_t qs[16]; };
+extern "C" __global__ void __launch_bounds__(256) quantized_gemv_iq4_nl(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int blocks_per_row = k / 32;
+ const uint8_t* w_row = weight + (size_t)row * blocks_per_row * 18;
+ float acc = 0.0f;
+
+ for (int b = threadIdx.x; b < blocks_per_row; b += blockDim.x)
+ {
+ const uint8_t* block = w_row + b * 18;
+ float d = __half2float(*reinterpret_cast(block));
+ const uint8_t* qs = block + 2;
+
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = qs[j];
+ acc += d * (float)kvalues_iq4nl_gemv[packed & 0x0F] * __half2float(x[b * 32 + j]);
+ acc += d * (float)kvalues_iq4nl_gemv[packed >> 4] * __half2float(x[b * 32 + j + 16]);
+ }
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_iq4_xs(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 136;
+ float acc = 0.0f;
+
+ for (int sb = threadIdx.x; sb < superblocks_per_row; sb += blockDim.x)
+ {
+ const uint8_t* block = w_row + sb * 136;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+ int base_x = sb * 256;
+
+ for (int ib = 0; ib < 8; ib++)
+ {
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+ const uint8_t* sub_qs = qs + ib * 16;
+ int x_off = base_x + ib * 32;
+
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = sub_qs[j];
+ acc += dl * (float)kvalues_iq4nl_gemv[packed & 0x0F] * __half2float(x[x_off + j]);
+ acc += dl * (float)kvalues_iq4nl_gemv[packed >> 4] * __half2float(x[x_off + j + 16]);
+ }
+ }
+ }
+
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+
+ __shared__ float warp_sums[32];
+ int lane = threadIdx.x % warpSize;
+ int warp_id = threadIdx.x / warpSize;
+ if (lane == 0) warp_sums[warp_id] = acc;
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ int num_warps = (blockDim.x + warpSize - 1) / warpSize;
+ acc = (lane < num_warps) ? warp_sums[lane] : 0.0f;
+ for (int offset = warpSize / 2; offset > 0; offset >>= 1)
+ acc += __shfl_down_sync(0xFFFFFFFF, acc, offset);
+ }
+
+ if (threadIdx.x == 0)
+ y[row] = __float2half(acc);
+}
+
extern "C" __global__ void __launch_bounds__(256) quantized_gemv_q5_0(
const uint8_t* __restrict__ weight,
const half* __restrict__ x,
@@ -344,22 +535,19 @@ extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q5_k(
float scale = d * (float)sc;
float min_val = dmin * (float)m;
- const uint8_t* sub_qs = qs + sub * 16;
- const uint8_t* sub_qh = qh + sub * 4;
int x_off = base_x + sub * 32;
+ const int pair_idx = sub >> 1;
+ const int nibble_half = sub & 1;
- // Element ordering matches dequant: interleaved lo/hi within sub-block
- // sub_out[2j] = lo(qs[j]), sub_out[2j+1] = hi(qs[j])
- for (int j = 0; j < 16; j++)
+ // Q5_K layout stores 4 pairs of 64 elements. Each sub-block has
+ // 32 elements at qs[pair_idx * 32 + pos], taking lo/hi nibbles
+ // from consecutive sub-blocks and bit `sub` from qh[pos].
+ for (int pos = 0; pos < 32; pos++)
{
- uint8_t packed = sub_qs[j];
- int bit_lo = (sub_qh[j / 4] >> ((j % 4) * 2)) & 1;
- int bit_hi = (sub_qh[j / 4] >> ((j % 4) * 2 + 1)) & 1;
- int lo = (packed & 0x0F) | (bit_lo << 4);
- int hi = (packed >> 4) | (bit_hi << 4);
-
- acc += (scale * (float)lo - min_val) * __half2float(x[x_off + 2 * j]);
- acc += (scale * (float)hi - min_val) * __half2float(x[x_off + 2 * j + 1]);
+ uint8_t packed = qs[pair_idx * 32 + pos];
+ int bit = (qh[pos] >> sub) & 1;
+ int q = (((packed >> (4 * nibble_half)) & 0x0F) | (bit << 4));
+ acc += (scale * (float)q - min_val) * __half2float(x[x_off + pos]);
}
}
}
diff --git a/native/kernels/quantized_gemv_mmq.cu b/native/kernels/quantized_gemv_mmq.cu
new file mode 100644
index 00000000..726d8d99
--- /dev/null
+++ b/native/kernels/quantized_gemv_mmq.cu
@@ -0,0 +1,2900 @@
+// MMQ-style fused dequant+matmul GEMV kernels.
+// Input activation x is quantized on-the-fly to INT8 with a per-32-element FP16 scale,
+// matching the Q4_K sub-block layout. Dot products use __dp4a (4× INT8×INT8 multiply-add
+// per cycle on sm_61+) instead of the FP fmuladd path used by quantized_gemv.cu.
+//
+// Q4_K math:
+// w[i] = d * sc_s * q[i] - dmin * m_s for sub-block s, q[i] ∈ [0,15]
+// xq[i] = round(x[i] / dx_c * 127) for input chunk c (32 elements per chunk)
+// x[i] ≈ dx_c * xq[i]
+// dot_s = Σ_i w[i]*x[j_i]
+// ≈ dx_c * (d*sc_s * Σ_i q[i]*xq[j_i] - dmin*m_s * Σ_i xq[j_i])
+//
+// The Σ q[i]*xq[j_i] sum is the dp4a accumulator (8 dp4a calls per 32-element sub-block).
+// Σ xq[j_i] is precomputed once per chunk during the input-quantization pass.
+//
+// Tile parallelism: one CUDA block processes MMQ_ROWS_PER_BLOCK output rows. All rows in
+// the block share the input-quantization pass, so for small models like SmolLM-135M
+// (k=576, 2 superblocks per row) the dp4a phase across MMQ_ROWS_PER_BLOCK*2 superblocks
+// fans out across BlockSize=256 threads instead of leaving most idle.
+
+#include
+#include
+
+__device__ __constant__ int8_t kvalues_iq4nl_mmq[16] = {
+ -127, -104, -83, -65, -49, -35, -22, -10,
+ 1, 13, 25, 38, 53, 69, 89, 113
+};
+
+// Per-chunk Stage 1 scratch lives in dynamic shared memory, sized at launch from `k`.
+// Layout (one extern __shared__ block, host passes `sharedMemBytes` to cuLaunchKernel):
+// int8_t s_xq [num_chunks * 32] // INT8-quantized x bytes
+// half s_dx [num_chunks] // per-chunk FP16 scale
+// half s_sx [num_chunks] (Q4_K, Q5_K) // Σ xq per full chunk
+// half s_sx2[num_chunks * 2] (Q6_K) // Σ xq per half-chunk (lo, hi)
+// The static __shared__ regions (s_acc, s_warp_partials) sit alongside; the dynamic
+// region starts after them. With 32-element chunks, num_chunks*32 is always multiple
+// of 32 so the half pointers downstream stay naturally aligned.
+//
+// Limits: dynamic shmem caps at the device's MAX_SHARED_MEMORY_PER_BLOCK_OPTIN
+// (≥ 100 KB on sm_86 / RTX 3060). The host side calls cuFuncSetAttribute once
+// per kernel to opt in, so any k that fits in that budget works at runtime.
+// - SmolLM-135M k=576 → 18 chunks → ~660 bytes
+// - Qwen3-8B k=12288 → 384 chunks → ~13.5 KB (Q4_K/Q5_K), ~14.2 KB (Q6_K)
+// - Llama-70B k=14336 → 448 chunks → ~15.7 KB
+// - Llama-405B k=53248 → 1664 chunks → ~58.4 KB (still under 100 KB optin)
+
+// Output rows per CUDA block. 4 rows × 2 superblocks/row (for SmolLM-135M) = 8 superblocks
+// distributed across 256 threads → 32 superblocks per warp before grouping. Larger models
+// with k≥1024 (≥4 superblocks/row) become work-saturated regardless.
+#define MMQ_ROWS_PER_BLOCK 4
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_iq4_nl_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int blocks_per_row = k / 32;
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+ __syncthreads();
+
+ const int total_units = rows_in_block * blocks_per_row;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / blocks_per_row;
+ int b = unit % blocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * blocks_per_row * 18;
+ const uint8_t* block = w_row + b * 18;
+ float d = __half2float(*reinterpret_cast(block));
+ const uint8_t* qs = block + 2;
+
+ float row_acc = 0.0f;
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = qs[j];
+ row_acc += d * (float)kvalues_iq4nl_mmq[packed & 0x0F] * __half2float(x[b * 32 + j]);
+ row_acc += d * (float)kvalues_iq4nl_mmq[packed >> 4] * __half2float(x[b * 32 + j + 16]);
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_iq4_xs_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+ __syncthreads();
+
+ const int total_units = rows_in_block * superblocks_per_row;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 136;
+ const uint8_t* block = w_row + sb * 136;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+ int base_x = sb * 256;
+
+ float row_acc = 0.0f;
+ for (int ib = 0; ib < 8; ib++)
+ {
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+ const uint8_t* sub_qs = qs + ib * 16;
+ int x_off = base_x + ib * 32;
+
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = sub_qs[j];
+ row_acc += dl * (float)kvalues_iq4nl_mmq[packed & 0x0F] * __half2float(x[x_off + j]);
+ row_acc += dl * (float)kvalues_iq4nl_mmq[packed >> 4] * __half2float(x[x_off + j + 16]);
+ }
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q4_k_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32; // 8 chunks per super-block
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx]. Sized by the host as
+ // num_chunks * (32 + 2 + 2) bytes. Static s_acc lives separately.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx = s_dx + num_chunks;
+
+ // Per-row partial sums staged in shared memory between Stage 2 and Stage 3.
+ // Each row's dp4a contributions are scattered across all threads (whichever
+ // thread happens to own that row+superblock pair); we sum them with a single
+ // cross-block reduction at the end.
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+ const int num_warps = blockDim.x >> 5;
+
+ // ── Stage 1: quantize x to INT8 per 32-element chunk ────────────────
+ for (int c = warp_id; c < num_chunks; c += num_warps)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ int s = qi;
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx[c] = __float2half((float)s);
+ }
+ }
+
+ // Zero the per-row accumulator scratch in parallel with the input-quant
+ // tail (whichever finishes last, the syncthreads below pairs them).
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation across (rows × superblocks) ──────────
+ // Map each thread to a (row_in_block, superblock) work unit. Total work
+ // = rows_in_block * superblocks_per_row. Distribute round-robin across
+ // blockDim threads via stride loop.
+ const int total_units = rows_in_block * superblocks_per_row;
+
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 144;
+ const uint8_t* block = w_row + sb * 144;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qs = block + 16;
+
+ float row_acc = 0.0f;
+
+ #pragma unroll
+ for (int pair = 0; pair < 4; pair++)
+ {
+ int sb_even = pair * 2;
+ int sb_odd = pair * 2 + 1;
+
+ int sc0, m0, sc1, m1;
+ if (sb_even < 4)
+ {
+ sc0 = scales_raw[sb_even] & 0x3F;
+ m0 = scales_raw[sb_even + 4] & 0x3F;
+ sc1 = scales_raw[sb_odd] & 0x3F;
+ m1 = scales_raw[sb_odd + 4] & 0x3F;
+ }
+ else
+ {
+ sc0 = (scales_raw[sb_even + 4] & 0x0F) | ((scales_raw[sb_even - 4] >> 6) << 4);
+ m0 = (scales_raw[sb_even + 4] >> 4) | ((scales_raw[sb_even] >> 6) << 4);
+ sc1 = (scales_raw[sb_odd + 4] & 0x0F) | ((scales_raw[sb_odd - 4] >> 6) << 4);
+ m1 = (scales_raw[sb_odd + 4] >> 4) | ((scales_raw[sb_odd] >> 6) << 4);
+ }
+
+ const uint8_t* pair_qs = qs + pair * 32;
+ int chunk_even = sb * 8 + sb_even;
+ int chunk_odd = sb * 8 + sb_odd;
+
+ const int8_t* xq_even = s_xq + chunk_even * 32;
+ const int8_t* xq_odd = s_xq + chunk_odd * 32;
+
+ int dot0 = 0;
+ int dot1 = 0;
+
+ #pragma unroll
+ for (int g = 0; g < 8; g++)
+ {
+ uint32_t qpacked = *reinterpret_cast(pair_qs + g * 4);
+ int lo = (int)(qpacked & 0x0F0F0F0F);
+ int hi = (int)((qpacked >> 4) & 0x0F0F0F0F);
+
+ int xq_e_packed = *reinterpret_cast(xq_even + g * 4);
+ int xq_o_packed = *reinterpret_cast(xq_odd + g * 4);
+
+ dot0 = __dp4a(lo, xq_e_packed, dot0);
+ dot1 = __dp4a(hi, xq_o_packed, dot1);
+ }
+
+ float dx_e = __half2float(s_dx[chunk_even]);
+ float dx_o = __half2float(s_dx[chunk_odd]);
+ float sx_e = __half2float(s_sx[chunk_even]);
+ float sx_o = __half2float(s_sx[chunk_odd]);
+
+ row_acc += dx_e * (d * (float)sc0 * (float)dot0 - dmin * (float)m0 * sx_e);
+ row_acc += dx_o * (d * (float)sc1 * (float)dot1 - dmin * (float)m1 * sx_o);
+ }
+
+ // Stage to shared memory at this thread's per-row slot. Threads working on
+ // the same row but different superblocks land at distinct tid offsets, so
+ // no atomics needed.
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ // ── Stage 3: per-row reduction. One warp per row; each lane sums 256/32 = 8
+ // of the row's slots, then a warp-shfl sums the lanes. With MMQ_ROWS_PER_BLOCK=4
+ // we use 4 warps (of 8 total); the remaining warps are idle in this stage.
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ // Each lane reads 8 of the 256 slots: lane 0 reads slots 0,32,64,...,224.
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// Q2_K MMQ — 84 bytes per 256 values
+//
+// Q2_K math (per sub-block s, 16 elements each — 16 sub-blocks per super-block):
+// w[i] = d * sc_s * q[i] - dmin * dm_s for q[i] ∈ [0, 3] (2-bit)
+// xq[i] = round(x[i] / dx_c * 127) for input chunk c (32 elements)
+// x[i] ≈ dx_c * xq[i]
+// dot_s ≈ dx_c * (d*sc_s * Σ q[i]*xq[j_i] - dmin*dm_s * Σ xq[j_i])
+//
+// Block layout:
+// scales[16] at offset 0 — one byte per sub-block: low nibble = sc, high nibble = dm
+// qs[64] at offset 16 — 2 bits/element, 4 elements/byte; sub_block s spans
+// qs[s*4 + 0..3] (16 elements as 4 bytes)
+// d at offset 80 — half super-block delta
+// dmin at offset 82 — half super-block min delta
+//
+// Tile geometry: each chunk (32 input elements) covers TWO Q2_K sub-blocks
+// (16 + 16 elements). Same chunk-vs-sub-block ratio as Q6_K, so we reuse Q6_K's
+// half-chunk-sum scheme: s_sx2[c*2 + 0] = Σ xq[c*32..c*32+15],
+// s_sx2[c*2 + 1] = Σ xq[c*32+16..c*32+31].
+//
+// Per super-block: 8 chunks (q_quad ∈ [0, 8)) × 2 sub-blocks/chunk = 16 sub-blocks.
+// dp4a inner loop: 4 calls per sub-block (16 elements / 4 elements per dp4a).
+// Weight bytes are pre-multiplied by sc into INT8 dp4a-friendly form? No — we
+// keep q ∈ [0,3] in the dp4a accumulator and apply sc as an FP multiplier post-dot
+// (matches Q5_K MMQ; q small enough that even 16-element dot fits int32 trivially).
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q2_k_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32; // 8 chunks per super-block
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx2]. s_sx2 is 2 halves per chunk (one per
+ // half-chunk = one per Q2_K sub-block), so dynamic budget is num_chunks*(32+2+4) bytes.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx2 = s_dx + num_chunks; // [a, b] per chunk
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+ const int num_warps = blockDim.x >> 5;
+
+ // ── Stage 1: input quantization with per-half-chunk sums (mirrors Q6_K MMQ) ──
+ // dx is per-32-element chunk. sx is split: sx_a = Σ xq[0..15], sx_b = Σ xq[16..31].
+ // Half-warp reductions: stop at offset=8 so lanes 0..15 and 16..31 stay isolated.
+ for (int c = warp_id; c < num_chunks; c += num_warps)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ int s = qi;
+ #pragma unroll
+ for (int offset = 8; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx2[c * 2 + 0] = __float2half((float)s);
+ }
+ if (lane == 16)
+ {
+ s_sx2[c * 2 + 1] = __float2half((float)s);
+ }
+ }
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation (one work unit per row × superblock) ────
+ const int total_units = rows_in_block * superblocks_per_row;
+
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ float row_acc = 0.0f;
+
+ // 8 chunks per super-block, each chunk covers 2 sub-blocks (isc=0,1).
+ #pragma unroll
+ for (int q_quad = 0; q_quad < 8; q_quad++)
+ {
+ int chunk_idx = sb * 8 + q_quad;
+ const uint8_t* chunk_qs = qs + q_quad * 8; // 8 bytes = 2 sub-blocks of 4 bytes
+
+ #pragma unroll
+ for (int isc = 0; isc < 2; isc++)
+ {
+ int sub = q_quad * 2 + isc; // sub-block index 0..15
+ int sc = scales[sub] & 0x0F; // 4-bit scale
+ int dm = (scales[sub] >> 4) & 0x0F; // 4-bit dmin coef
+
+ const uint8_t* sub_qs = chunk_qs + isc * 4; // 4 bytes for this sub-block
+ int l_base = isc * 16;
+ const int8_t* xq = s_xq + chunk_idx * 32 + l_base;
+
+ int dot = 0;
+ // 16 elements / 4 per dp4a = 4 dp4a calls. Each byte qs[g] holds 4
+ // consecutive 2-bit values at bit offsets 0,2,4,6.
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ uint32_t qbyte = sub_qs[g];
+ int q0 = (int)((qbyte >> 0) & 0x3);
+ int q1 = (int)((qbyte >> 2) & 0x3);
+ int q2v = (int)((qbyte >> 4) & 0x3);
+ int q3 = (int)((qbyte >> 6) & 0x3);
+ int wpack = (q0 & 0xFF)
+ | ((q1 & 0xFF) << 8)
+ | ((q2v & 0xFF) << 16)
+ | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast(xq + 4 * g);
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(s_dx[chunk_idx]);
+ float sx = __half2float(s_sx2[chunk_idx * 2 + isc]);
+
+ // Per-element identity:
+ // Σ (d*sc*q - dmin*dm) * x ≈ dx * (d*sc * Σ q*xq - dmin*dm * Σ xq)
+ row_acc += dx * (d * (float)sc * (float)dot - dmin * (float)dm * sx);
+ }
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ // ── Stage 3: per-row reduction (same shape as Q4_K/Q5_K/Q6_K MMQ) ────
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// Q5_K MMQ — 176 bytes per 256 values
+//
+// Q5_K math (per sub-block s, 32 elements each):
+// w[i] = d * sc_s * q[i] - dmin * m_s for q[i] ∈ [0, 31] (5-bit)
+// xq[i] = round(x[i] / dx_c * 127) for input chunk c (32 elements)
+// x[i] ≈ dx_c * xq[i]
+// dot_s ≈ dx_c * (d*sc_s * Σ q[i]*xq[j_i] - dmin*m_s * Σ xq[j_i])
+//
+// Element layout within a sub-block (matches dequant order):
+// out[2j] = (qs[j] & 0x0F) | (qh_bit_lo[j] << 4) for j ∈ [0, 16)
+// out[2j+1] = (qs[j] >> 4) | (qh_bit_hi[j] << 4)
+// That is, sub_out[0..31] = [lo_0, hi_0, lo_1, hi_1, ..., lo_15, hi_15].
+//
+// dp4a chunk packing: emit weight bytes in input memory order so they line up
+// with s_xq[chunk_idx*32 + 4g .. 4g+3] = x bytes 4g..4g+3 within the chunk.
+// For chunk g (0..7), we cover j = 2g and j = 2g+1; pack [lo_e, hi_e, lo_o, hi_o].
+//
+// Sub-block s ∈ [0, 8) maps 1:1 to chunk c = sb*8 + s, same as Q4_K MMQ — so
+// the input-quantization pass shared with Q4_K MMQ produces the right s_dx/s_sx.
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q5_k_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32;
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx]. See Q4_K MMQ above for layout.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx = s_dx + num_chunks;
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+ const int num_warps = blockDim.x >> 5;
+
+ // ── Stage 1: input quantization (identical to Q4_K MMQ) ────────────────
+ for (int c = warp_id; c < num_chunks; c += num_warps)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ int s = qi;
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx[c] = __float2half((float)s);
+ }
+ }
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation (one work unit per row × superblock) ────
+ const int total_units = rows_in_block * superblocks_per_row;
+
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 176;
+ const uint8_t* block = w_row + sb * 176;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qh = block + 16; // 32 bytes
+ const uint8_t* qs = block + 48; // 128 bytes
+
+ float row_acc = 0.0f;
+
+ // 8 sub-blocks of 32 elements each
+ #pragma unroll
+ for (int sub = 0; sub < 8; sub++)
+ {
+ int sc, m;
+ if (sub < 4)
+ {
+ sc = scales_raw[sub] & 0x3F;
+ m = scales_raw[sub + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sub + 4] & 0x0F) | ((scales_raw[sub - 4] >> 6) << 4);
+ m = (scales_raw[sub + 4] >> 4) | ((scales_raw[sub] >> 6) << 4);
+ }
+
+ int chunk_idx = sb * 8 + sub;
+ const int8_t* xq = s_xq + chunk_idx * 32;
+ const int pair_idx = sub >> 1;
+ const int nibble_half = sub & 1;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 8; g++)
+ {
+ // Pack four Q5_K weights in input memory order for this 32-value sub-block.
+ int pos0 = 4 * g;
+ int p0 = qs[pair_idx * 32 + pos0 + 0];
+ int p1 = qs[pair_idx * 32 + pos0 + 1];
+ int p2 = qs[pair_idx * 32 + pos0 + 2];
+ int p3 = qs[pair_idx * 32 + pos0 + 3];
+ int q0 = ((p0 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 0] >> sub) & 1) << 4);
+ int q1 = ((p1 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 1] >> sub) & 1) << 4);
+ int q2 = ((p2 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 2] >> sub) & 1) << 4);
+ int q3 = ((p3 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 3] >> sub) & 1) << 4);
+
+ int wpack = (q0 & 0xFF) | ((q1 & 0xFF) << 8)
+ | ((q2 & 0xFF) << 16) | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast(xq + 4 * g);
+
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(s_dx[chunk_idx]);
+ float sx = __half2float(s_sx[chunk_idx]);
+ row_acc += dx * (d * (float)sc * (float)dot - dmin * (float)m * sx);
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ // ── Stage 3: per-row reduction (same shape as Q4_K MMQ) ────────────────
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// Q6_K MMQ — 210 bytes per 256 values
+//
+// Q6_K math (per sub-block s, 16 elements each, 16 sub-blocks per superblock):
+// w[i] = d * sc_s * (q[i] - 32) for q[i] ∈ [0, 63] (6-bit)
+// x[i] ≈ dx_c * xq[i]
+// dot_s ≈ dx_c * (d*sc_s * Σ q[i]*xq[j_i] - 32 * d*sc_s * Σ xq[j_i])
+//
+// Q6_K has NO dmin term — much simpler than Q4_K/Q5_K.
+//
+// Sub-block size is 16 (vs 32 for Q4_K/Q5_K), so each 32-element input chunk
+// holds TWO Q6_K sub-blocks. Their scales differ, so we need separate sums of
+// x over the two halves of the chunk: sx_a (lanes 0..15) and sx_b (lanes 16..31).
+// We store both in s_sx2[c*2 + 0..1].
+//
+// Element layout within a half-superblock (128 elements covered by ql_half[64]
+// + qh_half[32] + sc_half[8]):
+// q1[l] = (ql_half[l] & 0x0F) | (((qh_half[l] >> 0) & 3) << 4) → x[+0..31]
+// q2[l] = (ql_half[l + 32] & 0x0F) | (((qh_half[l] >> 2) & 3) << 4) → x[+32..63]
+// q3[l] = (ql_half[l] >> 4) | (((qh_half[l] >> 4) & 3) << 4) → x[+64..95]
+// q4[l] = (ql_half[l + 32] >> 4) | (((qh_half[l] >> 6) & 3) << 4) → x[+96..127]
+// Each q_quad ∈ {0..3} → ql_offset = (qq & 1) * 32, ql_shift = (qq & 2) << 1,
+// qh_shift = qq << 1. Sub-block scale = sc_half[q_quad*2 + (l < 16 ? 0 : 1)].
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q6_k_mmq(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32;
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx2]. s_sx2 is 2 halves per chunk
+ // (one per half-chunk) instead of 1, so the dynamic budget here is
+ // num_chunks * (32 + 2 + 4) bytes.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx2 = s_dx + num_chunks; // [a, b] per chunk
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+ const int num_warps = blockDim.x >> 5;
+
+ // ── Stage 1: input quantization with per-half-chunk sums ───────────────
+ // dx is per-32-element chunk. sx is split: sx_a = Σ xq[0..15], sx_b = Σ xq[16..31].
+ // Half-warp reductions: omit the offset=16 step so each lane in 0..15 holds
+ // the sum of x[chunk_base + 0..15] and each lane in 16..31 holds the sum
+ // of x[chunk_base + 16..31] independently.
+ for (int c = warp_id; c < num_chunks; c += num_warps)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ // Full warp max for dx (chunk-wide scale).
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ // Half-warp sum: stop at offset=8 so lanes 0..15 and 16..31 stay isolated.
+ int s = qi;
+ #pragma unroll
+ for (int offset = 8; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx2[c * 2 + 0] = __float2half((float)s);
+ }
+ if (lane == 16)
+ {
+ s_sx2[c * 2 + 1] = __float2half((float)s);
+ }
+ }
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation ─────────────────────────────────────────
+ // One work unit per (row, superblock); each unit walks 16 sub-blocks
+ // (2 halves × 4 q_quads × 2 sub-blocks-per-quad).
+ const int total_units = rows_in_block * superblocks_per_row;
+
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 210;
+ const uint8_t* block = w_row + sb * 210;
+ const uint8_t* ql = block;
+ const uint8_t* qh = block + 128;
+ const int8_t* scales = reinterpret_cast(block + 192);
+ float d = __half2float(*reinterpret_cast(block + 208));
+
+ float row_acc = 0.0f;
+
+ #pragma unroll
+ for (int half_idx = 0; half_idx < 2; half_idx++)
+ {
+ const uint8_t* ql_half = ql + half_idx * 64;
+ const uint8_t* qh_half = qh + half_idx * 32;
+ const int8_t* sc_half = scales + half_idx * 8;
+
+ #pragma unroll
+ for (int q_quad = 0; q_quad < 4; q_quad++)
+ {
+ int ql_offset = (q_quad & 1) * 32;
+ int ql_shift = (q_quad & 2) << 1; // 0 or 4
+ int qh_shift = q_quad << 1; // 0,2,4,6
+
+ int chunk_idx = sb * 8 + half_idx * 4 + q_quad;
+
+ // Two sub-blocks per (half, q_quad) — l ∈ [0,16) and l ∈ [16,32).
+ #pragma unroll
+ for (int isc = 0; isc < 2; isc++)
+ {
+ int sc = sc_half[q_quad * 2 + isc];
+
+ int dot = 0;
+ int l_base = isc * 16;
+ const int8_t* xq = s_xq + chunk_idx * 32 + l_base;
+
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ int wpack = 0;
+ #pragma unroll
+ for (int b = 0; b < 4; b++)
+ {
+ int l = l_base + g * 4 + b;
+ int q = ((ql_half[l + ql_offset] >> ql_shift) & 0x0F)
+ | (((qh_half[l] >> qh_shift) & 3) << 4);
+ wpack |= (q & 0xFF) << (b * 8);
+ }
+ int xpack = *reinterpret_cast(xq + 4 * g);
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(s_dx[chunk_idx]);
+ float sx = __half2float(s_sx2[chunk_idx * 2 + isc]);
+
+ // d * sc * (Σ q*xq - 32 * Σ xq) scaled by chunk dx.
+ row_acc += dx * d * (float)sc * ((float)dot - 32.0f * sx);
+ }
+ }
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ // ── Stage 3: per-row reduction ─────────────────────────────────────────
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// MMVQ-large variants — 1 row per CUDA block, 128 threads (4 warps).
+//
+// Tuned for large-k regime (k >= 1024, ≥4 super-blocks per row). Modeled on
+// llama.cpp's mul_mat_vec_q structure: each block computes
+// exactly one output row's dot product, all 128 threads cooperate on the same
+// accumulator (kept in registers, no per-row shmem), final reduction is a
+// per-warp __shfl_xor_sync sum followed by a small 4-warp fan-in via shmem.
+//
+// The 4-rows-per-block kernels above remain the right choice for small k
+// (≤3 super-blocks/row, e.g. SmolLM-135M k=576) where the input-quantization
+// amortization across rows is the dominant win and there's not enough work
+// per row to saturate 128 threads.
+//
+// Stage 1 (input quant) is structurally identical to the 4-row variant: 4
+// warps walk all num_chunks chunks via warp-stride, producing s_xq/s_dx/s_sx
+// (or s_sx2 for Q6_K) in shared memory.
+//
+// Stage 2 distributes one row's dot-product work across all 128 threads using
+// a "g-cell" decomposition: each (super-block, sub-block-or-pair, g) triple is
+// a single dp4a-pair work unit; each thread accumulates partial dp4a results
+// into a register float `acc`. The mins/origin term (which is constant per
+// (sb, sub) and would over-count if added per-cell) is added by exactly one
+// designated cell (g==0) per (sb, sub) — the rest contribute only the dot
+// product.
+//
+// Stage 3: warp-shuffle reduction within each warp, 4 partials staged into
+// __shared__ float[4], one final reduction in warp 0, write y[row] from
+// thread (0, 0).
+
+// 4 warps × 32 = 128 threads/block, mirroring llama.cpp's mul_mat_vec_q for ncols_dst=1.
+// Smaller blocks give more concurrent blocks per SM (occupancy-vs-per-block-work tradeoff
+// favors small blocks for GEMV where each row is shallow on its own).
+#define MMVQ_LARGE_THREADS 128
+#define MMVQ_LARGE_NWARPS 4
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_iq4_nl_mmvq_large(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int blocks_per_row = k / 32;
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ const uint8_t* w_row = weight + (size_t)row * blocks_per_row * 18;
+ float acc = 0.0f;
+
+ for (int b = tid; b < blocks_per_row; b += MMVQ_LARGE_THREADS)
+ {
+ const uint8_t* block = w_row + b * 18;
+ float d = __half2float(*reinterpret_cast(block));
+ const uint8_t* qs = block + 2;
+
+ float block_acc = 0.0f;
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = qs[j];
+ block_acc += d * (float)kvalues_iq4nl_mmq[packed & 0x0F] * __half2float(x[b * 32 + j]);
+ block_acc += d * (float)kvalues_iq4nl_mmq[packed >> 4] * __half2float(x[b * 32 + j + 16]);
+ }
+ acc += block_acc;
+ }
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
+
+ if (lane == 0)
+ s_warp_partials[warp_id] = acc;
+
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ float v = (lane < MMVQ_LARGE_NWARPS) ? s_warp_partials[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 2; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+ if (lane == 0)
+ y[row] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_iq4_xs_mmvq_large(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 136;
+ float acc = 0.0f;
+
+ const int total_units = superblocks_per_row * 8;
+ for (int unit = tid; unit < total_units; unit += MMVQ_LARGE_THREADS)
+ {
+ int sb = unit >> 3;
+ int ib = unit & 7;
+
+ const uint8_t* block = w_row + sb * 136;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+ const uint8_t* sub_qs = qs + ib * 16;
+ int x_off = sb * 256 + ib * 32;
+
+ float sub_acc = 0.0f;
+ #pragma unroll 16
+ for (int j = 0; j < 16; j++)
+ {
+ uint8_t packed = sub_qs[j];
+ sub_acc += dl * (float)kvalues_iq4nl_mmq[packed & 0x0F] * __half2float(x[x_off + j]);
+ sub_acc += dl * (float)kvalues_iq4nl_mmq[packed >> 4] * __half2float(x[x_off + j + 16]);
+ }
+ acc += sub_acc;
+ }
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
+
+ if (lane == 0)
+ s_warp_partials[warp_id] = acc;
+
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ float v = (lane < MMVQ_LARGE_NWARPS) ? s_warp_partials[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 2; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+ if (lane == 0)
+ y[row] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_q4_k_mmvq_large(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32;
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx]. See Q4_K MMQ above for layout.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx = s_dx + num_chunks;
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ // ── Stage 1: quantize x to INT8 per 32-element chunk ────────────────
+ for (int c = warp_id; c < num_chunks; c += MMVQ_LARGE_NWARPS)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ int s = qi;
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx[c] = __float2half((float)s);
+ }
+ }
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation across (sb, pair) units ───────────────
+ // Each (super-block, pair) is one work unit covering 8 g-cells (16 dp4a)
+ // with constant per-pair scales sc0/m0/sc1/m1. Total units per row =
+ // superblocks_per_row * 4. For k=4096 → 64 units; k=12288 (Qwen3-8B
+ // MlpDown) → 192 units. Stride loop over 128 threads.
+ const int total_units = superblocks_per_row * 4;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 144;
+ float acc = 0.0f;
+
+ for (int unit = tid; unit < total_units; unit += MMVQ_LARGE_THREADS)
+ {
+ int sb = unit >> 2; // unit / 4
+ int pair = unit & 3; // 0..3
+
+ const uint8_t* block = w_row + sb * 144;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qs = block + 16;
+
+ int sb_even = pair * 2;
+ int sb_odd = pair * 2 + 1;
+ int sc0, m0, sc1, m1;
+ if (sb_even < 4)
+ {
+ sc0 = scales_raw[sb_even] & 0x3F;
+ m0 = scales_raw[sb_even + 4] & 0x3F;
+ sc1 = scales_raw[sb_odd] & 0x3F;
+ m1 = scales_raw[sb_odd + 4] & 0x3F;
+ }
+ else
+ {
+ sc0 = (scales_raw[sb_even + 4] & 0x0F) | ((scales_raw[sb_even - 4] >> 6) << 4);
+ m0 = (scales_raw[sb_even + 4] >> 4) | ((scales_raw[sb_even] >> 6) << 4);
+ sc1 = (scales_raw[sb_odd + 4] & 0x0F) | ((scales_raw[sb_odd - 4] >> 6) << 4);
+ m1 = (scales_raw[sb_odd + 4] >> 4) | ((scales_raw[sb_odd] >> 6) << 4);
+ }
+
+ const uint8_t* pair_qs = qs + pair * 32;
+ int chunk_even = sb * 8 + sb_even;
+ int chunk_odd = sb * 8 + sb_odd;
+
+ const int8_t* xq_even = s_xq + chunk_even * 32;
+ const int8_t* xq_odd = s_xq + chunk_odd * 32;
+
+ int dot0 = 0;
+ int dot1 = 0;
+
+ #pragma unroll
+ for (int g = 0; g < 8; g++)
+ {
+ uint32_t qpacked = *reinterpret_cast(pair_qs + g * 4);
+ int lo = (int)(qpacked & 0x0F0F0F0F);
+ int hi = (int)((qpacked >> 4) & 0x0F0F0F0F);
+
+ int xq_e_packed = *reinterpret_cast(xq_even + g * 4);
+ int xq_o_packed = *reinterpret_cast(xq_odd + g * 4);
+
+ dot0 = __dp4a(lo, xq_e_packed, dot0);
+ dot1 = __dp4a(hi, xq_o_packed, dot1);
+ }
+
+ float dx_e = __half2float(s_dx[chunk_even]);
+ float dx_o = __half2float(s_dx[chunk_odd]);
+ float sx_e = __half2float(s_sx[chunk_even]);
+ float sx_o = __half2float(s_sx[chunk_odd]);
+
+ acc += dx_e * (d * (float)sc0 * (float)dot0 - dmin * (float)m0 * sx_e);
+ acc += dx_o * (d * (float)sc1 * (float)dot1 - dmin * (float)m1 * sx_o);
+ }
+
+ // ── Stage 3: warp-shfl reduction → 4-warp shmem fan-in → final sum ──
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
+
+ if (lane == 0)
+ s_warp_partials[warp_id] = acc;
+
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ float v = (lane < MMVQ_LARGE_NWARPS) ? s_warp_partials[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 2; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+ if (lane == 0)
+ y[row] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// Q2_K MMVQ-large — same 1-row-per-block structure, Q2_K weight decode.
+//
+// Q2_K has 16 sub-blocks per super-block (16 elements each, like Q6_K), with
+// 4-bit packed scale + 4-bit packed dmin per sub-block. Each 32-element input
+// chunk holds TWO Q2_K sub-blocks, so we use the Q6_K-style per-half-chunk
+// sums (s_sx2[c*2 + isc]) populated in Stage 1.
+//
+// Cell layout per super-block: 16 sub-blocks × 4 g = 64 cells/sb.
+// For k=4096 → 16 sb × 16 = 256 units / 128 threads = 2 units per thread.
+// Unit decomposition: slot ∈ [0, 16) decodes as q_quad = slot >> 1, isc = slot & 1.
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_q2_k_mmvq_large(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32;
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx2]. See Q2_K MMQ above for layout.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx2 = s_dx + num_chunks; // [a, b] per chunk
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ // ── Stage 1: input quantization with per-half-chunk sums (mirrors Q6_K) ─
+ for (int c = warp_id; c < num_chunks; c += MMVQ_LARGE_NWARPS)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ // Full warp max for dx (chunk-wide scale).
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ // Half-warp sum: stop at offset=8 so lanes 0..15 and 16..31 stay isolated.
+ int s = qi;
+ #pragma unroll
+ for (int offset = 8; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx2[c * 2 + 0] = __float2half((float)s);
+ }
+ if (lane == 16)
+ {
+ s_sx2[c * 2 + 1] = __float2half((float)s);
+ }
+ }
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation across (sb, slot) units ─────────────────
+ // Each (super-block, slot) is one Q2_K 16-element sub-block (4 dp4a). slot
+ // ∈ [0, 16) decodes as q_quad = slot >> 1, isc = slot & 1. Total units per
+ // row = superblocks_per_row * 16. For k=4096 → 256 units = 2 per thread.
+ const int total_units = superblocks_per_row * 16;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+ float acc = 0.0f;
+
+ for (int unit = tid; unit < total_units; unit += MMVQ_LARGE_THREADS)
+ {
+ int sb = unit >> 4; // unit / 16
+ int slot = unit & 15; // 0..15
+ int q_quad = slot >> 1; // 0..7
+ int isc = slot & 1; // 0..1
+
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ int sub = q_quad * 2 + isc; // sub-block index 0..15
+ int sc = scales[sub] & 0x0F; // 4-bit scale
+ int dm = (scales[sub] >> 4) & 0x0F; // 4-bit dmin coef
+
+ const uint8_t* chunk_qs = qs + q_quad * 8; // 8 bytes = 2 sub-blocks
+ const uint8_t* sub_qs = chunk_qs + isc * 4; // 4 bytes for this sub-block
+ int chunk_idx = sb * 8 + q_quad;
+ int l_base = isc * 16;
+ const int8_t* xq = s_xq + chunk_idx * 32 + l_base;
+
+ int dot = 0;
+ // 16 elements / 4 per dp4a = 4 dp4a calls. Each byte qs[g] holds 4
+ // consecutive 2-bit values at bit offsets 0,2,4,6.
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ uint32_t qbyte = sub_qs[g];
+ int q0 = (int)((qbyte >> 0) & 0x3);
+ int q1 = (int)((qbyte >> 2) & 0x3);
+ int q2v = (int)((qbyte >> 4) & 0x3);
+ int q3 = (int)((qbyte >> 6) & 0x3);
+ int wpack = (q0 & 0xFF)
+ | ((q1 & 0xFF) << 8)
+ | ((q2v & 0xFF) << 16)
+ | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast(xq + 4 * g);
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(s_dx[chunk_idx]);
+ float sx = __half2float(s_sx2[chunk_idx * 2 + isc]);
+
+ // Per-element identity:
+ // Σ (d*sc*q - dmin*dm) * x ≈ dx * (d*sc * Σ q*xq - dmin*dm * Σ xq)
+ acc += dx * (d * (float)sc * (float)dot - dmin * (float)dm * sx);
+ }
+
+ // ── Stage 3: warp-shfl reduction → 4-warp shmem fan-in → final sum ──
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
+
+ if (lane == 0)
+ s_warp_partials[warp_id] = acc;
+
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ float v = (lane < MMVQ_LARGE_NWARPS) ? s_warp_partials[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 2; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+ if (lane == 0)
+ y[row] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// Q5_K MMVQ-large — same 1-row-per-block structure, Q5_K weight decode.
+//
+// Cell layout per super-block: 8 sub-blocks × 8 g = 64 cells/sb.
+// Each sub-block = one 32-element chunk with its own scale and 5-bit quants.
+// For k=4096 → 16 sb × 64 = 1024 cells / 128 threads = 8 cells per thread.
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_q5_k_mmvq_large(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32;
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx]. See Q4_K MMQ above for layout.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx = s_dx + num_chunks;
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ // ── Stage 1: input quantization (identical to Q4_K MMVQ-large) ─────
+ for (int c = warp_id; c < num_chunks; c += MMVQ_LARGE_NWARPS)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ int s = qi;
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx[c] = __float2half((float)s);
+ }
+ }
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation across (sb, sub) units ──────────────
+ // Each (super-block, sub-block) is one work unit (8 g-cells, 8 dp4a, with
+ // constant per-sub scales sc/m). Total units per row = superblocks_per_row * 8.
+ // For k=4096 → 128 units exactly = 1 unit per thread; for k=12288 → 384 units = 3 each.
+ const int total_units = superblocks_per_row * 8;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 176;
+ float acc = 0.0f;
+
+ for (int unit = tid; unit < total_units; unit += MMVQ_LARGE_THREADS)
+ {
+ int sb = unit >> 3; // unit / 8
+ int sub = unit & 7; // 0..7
+
+ const uint8_t* block = w_row + sb * 176;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qh = block + 16;
+ const uint8_t* qs = block + 48;
+
+ int sc, m;
+ if (sub < 4)
+ {
+ sc = scales_raw[sub] & 0x3F;
+ m = scales_raw[sub + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sub + 4] & 0x0F) | ((scales_raw[sub - 4] >> 6) << 4);
+ m = (scales_raw[sub + 4] >> 4) | ((scales_raw[sub] >> 6) << 4);
+ }
+
+ int chunk_idx = sb * 8 + sub;
+ const int8_t* xq = s_xq + chunk_idx * 32;
+ const int pair_idx = sub >> 1;
+ const int nibble_half = sub & 1;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 8; g++)
+ {
+ int pos0 = 4 * g;
+ int p0 = qs[pair_idx * 32 + pos0 + 0];
+ int p1 = qs[pair_idx * 32 + pos0 + 1];
+ int p2 = qs[pair_idx * 32 + pos0 + 2];
+ int p3 = qs[pair_idx * 32 + pos0 + 3];
+ int q0 = ((p0 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 0] >> sub) & 1) << 4);
+ int q1 = ((p1 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 1] >> sub) & 1) << 4);
+ int q2 = ((p2 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 2] >> sub) & 1) << 4);
+ int q3 = ((p3 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 3] >> sub) & 1) << 4);
+
+ int wpack = (q0 & 0xFF) | ((q1 & 0xFF) << 8)
+ | ((q2 & 0xFF) << 16) | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast(xq + 4 * g);
+
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(s_dx[chunk_idx]);
+ float sx = __half2float(s_sx[chunk_idx]);
+ acc += dx * (d * (float)sc * (float)dot - dmin * (float)m * sx);
+ }
+
+ // ── Stage 3: warp-shfl reduction → 4-warp shmem fan-in → final sum ──
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
+
+ if (lane == 0)
+ s_warp_partials[warp_id] = acc;
+
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ float v = (lane < MMVQ_LARGE_NWARPS) ? s_warp_partials[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 2; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+ if (lane == 0)
+ y[row] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// Q6_K MMVQ-large — same 1-row-per-block structure, Q6_K weight decode.
+//
+// Q6_K has 16 sub-blocks per super-block (16 elements each, vs 32 for Q4/5_K),
+// no dmin term, and per-half-chunk x sums. The cell decomposition unrolls the
+// 4-row kernel's nested (half, q_quad, isc) loop into a flat (sb, slot, g)
+// indexing where slot ∈ [0, 16) covers the 16 sub-blocks (2 halves × 4 q_quads
+// × 2 isc) and g ∈ [0, 4) covers the 4 dp4a calls per sub-block.
+//
+// Cell layout per super-block: 16 sub-blocks × 4 g = 64 cells/sb.
+// For k=4096 → 16 sb × 64 = 1024 cells / 128 threads = 8 cells per thread.
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_q6_k_mmvq_large(
+ const uint8_t* __restrict__ weight,
+ const half* __restrict__ x,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+ const int num_chunks = k / 32;
+
+ // Dynamic shmem: [s_xq | s_dx | s_sx2]. See Q6_K MMQ above for layout.
+ extern __shared__ uint8_t s_dyn[];
+ int8_t* s_xq = reinterpret_cast(s_dyn);
+ half* s_dx = reinterpret_cast(s_xq + num_chunks * 32);
+ half* s_sx2 = s_dx + num_chunks; // [a, b] per chunk
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ // ── Stage 1: input quantization with per-half-chunk sums ───────────
+ for (int c = warp_id; c < num_chunks; c += MMVQ_LARGE_NWARPS)
+ {
+ float v = __half2float(x[c * 32 + lane]);
+ float a = fabsf(v);
+
+ // Full warp max for dx (chunk-wide scale).
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ {
+ float other = __shfl_xor_sync(0xFFFFFFFF, a, offset);
+ a = fmaxf(a, other);
+ }
+
+ float inv_scale = (a > 0.0f) ? (127.0f / a) : 0.0f;
+ int qi = __float2int_rn(v * inv_scale);
+ qi = qi > 127 ? 127 : (qi < -127 ? -127 : qi);
+ s_xq[c * 32 + lane] = (int8_t)qi;
+
+ // Half-warp sum: stop at offset=8 so lanes 0..15 and 16..31 stay isolated.
+ int s = qi;
+ #pragma unroll
+ for (int offset = 8; offset > 0; offset >>= 1)
+ s += __shfl_xor_sync(0xFFFFFFFF, s, offset);
+
+ if (lane == 0)
+ {
+ s_dx[c] = __float2half(a / 127.0f);
+ s_sx2[c * 2 + 0] = __float2half((float)s);
+ }
+ if (lane == 16)
+ {
+ s_sx2[c * 2 + 1] = __float2half((float)s);
+ }
+ }
+
+ __syncthreads();
+
+ // ── Stage 2: dp4a accumulation across (sb, slot) units ────────────
+ // Each (super-block, slot) is one Q6_K 16-element sub-block (4 dp4a).
+ // slot ∈ [0, 16) decodes as: half_idx = slot >> 3, q_quad = (slot >> 1) & 3,
+ // isc = slot & 1. Total units per row = superblocks_per_row * 16.
+ // For k=4096 → 256 units = 2 per thread; for k=12288 → 768 units = 6 each.
+ const int total_units = superblocks_per_row * 16;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 210;
+ float acc = 0.0f;
+
+ for (int unit = tid; unit < total_units; unit += MMVQ_LARGE_THREADS)
+ {
+ int sb = unit >> 4; // unit / 16
+ int slot = unit & 15; // 0..15
+
+ int half_idx = slot >> 3; // 0..1
+ int q_quad = (slot >> 1) & 3; // 0..3
+ int isc = slot & 1; // 0..1
+
+ const uint8_t* block = w_row + sb * 210;
+ const uint8_t* ql = block;
+ const uint8_t* qh = block + 128;
+ const int8_t* scales = reinterpret_cast(block + 192);
+ float d = __half2float(*reinterpret_cast(block + 208));
+
+ const uint8_t* ql_half = ql + half_idx * 64;
+ const uint8_t* qh_half = qh + half_idx * 32;
+ const int8_t* sc_half = scales + half_idx * 8;
+
+ int ql_offset = (q_quad & 1) * 32;
+ int ql_shift = (q_quad & 2) << 1; // 0 or 4
+ int qh_shift = q_quad << 1; // 0,2,4,6
+ int chunk_idx = sb * 8 + half_idx * 4 + q_quad;
+ int sc = sc_half[q_quad * 2 + isc];
+
+ int l_base = isc * 16;
+ const int8_t* xq = s_xq + chunk_idx * 32 + l_base;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ int wpack = 0;
+ #pragma unroll
+ for (int b = 0; b < 4; b++)
+ {
+ int l = l_base + g * 4 + b;
+ int q = ((ql_half[l + ql_offset] >> ql_shift) & 0x0F)
+ | (((qh_half[l] >> qh_shift) & 3) << 4);
+ wpack |= (q & 0xFF) << (b * 8);
+ }
+ int xpack = *reinterpret_cast(xq + 4 * g);
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(s_dx[chunk_idx]);
+ float sx = __half2float(s_sx2[chunk_idx * 2 + isc]);
+
+ acc += dx * d * (float)sc * ((float)dot - 32.0f * sx);
+ }
+
+ // ── Stage 3: warp-shfl reduction → 4-warp shmem fan-in → final sum ──
+ #pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1)
+ acc += __shfl_xor_sync(0xFFFFFFFF, acc, offset);
+
+ if (lane == 0)
+ s_warp_partials[warp_id] = acc;
+
+ __syncthreads();
+
+ if (warp_id == 0)
+ {
+ float v = (lane < MMVQ_LARGE_NWARPS) ? s_warp_partials[lane] : 0.0f;
+ #pragma unroll
+ for (int offset = 2; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+ if (lane == 0)
+ y[row] = __float2half(v);
+ }
+}
+
+// ============================================================================
+// PRE-QUANTIZED VARIANTS (`_preq`)
+//
+// These kernels skip Stage 1 entirely. The host launches `quantize_x_to_q8_1`
+// once per input vector to populate device-resident scratch buffers, then
+// passes those pointers into the GEMV. This eliminates the redundant Stage 1
+// work that the on-the-fly kernels run once per CUDA block (n times for
+// MMVQ-large, n/4 times for MMQ-4-rows).
+//
+// Scratch layout (matches the in-kernel s_xq/s_dx/s_sx[2] for Q6_K):
+// const int8_t* xq [num_chunks * 32]
+// const half* dx [num_chunks]
+// const half* sx2 [num_chunks * 2] per-half-chunk sums (lo, hi)
+//
+// Q4_K / Q5_K consume the full-chunk sum lazily as sx2[c*2+0] + sx2[c*2+1]
+// (one extra integer-as-FP add per chunk-use), so a single scratch layout
+// works for all three quant types and the pre-quant kernel writes per-half
+// sums identically to Q6_K's existing in-kernel Stage 1.
+// ============================================================================
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_iq4_nl_mmq_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ (void)sx2_in;
+
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int blocks_per_row = k / 32;
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+ __syncthreads();
+
+ const int total_units = rows_in_block * blocks_per_row;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / blocks_per_row;
+ int b = unit % blocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * blocks_per_row * 18;
+ const uint8_t* block = w_row + b * 18;
+ float d = __half2float(*reinterpret_cast(block));
+ float dx = __half2float(dx_in[b]);
+ const uint8_t* qs = block + 2;
+ const int8_t* xq = xq_in + b * 32;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ int wpack_lo = 0;
+ int wpack_hi = 0;
+ #pragma unroll
+ for (int j = 0; j < 4; j++)
+ {
+ uint8_t packed = qs[g * 4 + j];
+ wpack_lo |= ((int)kvalues_iq4nl_mmq[packed & 0x0F] & 0xFF) << (j * 8);
+ wpack_hi |= ((int)kvalues_iq4nl_mmq[packed >> 4] & 0xFF) << (j * 8);
+ }
+
+ int xpack_lo = *reinterpret_cast(xq + g * 4);
+ int xpack_hi = *reinterpret_cast(xq + 16 + g * 4);
+ dot = __dp4a(wpack_lo, xpack_lo, dot);
+ dot = __dp4a(wpack_hi, xpack_hi, dot);
+ }
+
+ s_acc[r * 256 + tid] += d * dx * (float)dot;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_iq4_xs_mmq_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ (void)sx2_in;
+
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+ __syncthreads();
+
+ const int total_units = rows_in_block * superblocks_per_row * 8;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int row_unit = unit / (superblocks_per_row * 8);
+ int rem = unit - row_unit * superblocks_per_row * 8;
+ int sb = rem >> 3;
+ int ib = rem & 7;
+ int row = row_base + row_unit;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 136;
+ const uint8_t* block = w_row + sb * 136;
+ float d = __half2float(*reinterpret_cast(block));
+ uint16_t scales_h = (uint16_t)block[2] | ((uint16_t)block[3] << 8);
+ const uint8_t* scales_l = block + 4;
+ const uint8_t* qs = block + 8;
+
+ int low = (scales_l[ib >> 1] >> (4 * (ib & 1))) & 0x0F;
+ int high = (scales_h >> (2 * ib)) & 0x03;
+ int ls = low | (high << 4);
+ float dl = d * (float)(ls - 32);
+
+ int chunk_idx = sb * 8 + ib;
+ float dx = __half2float(dx_in[chunk_idx]);
+ const uint8_t* sub_qs = qs + ib * 16;
+ const int8_t* xq = xq_in + chunk_idx * 32;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ int wpack_lo = 0;
+ int wpack_hi = 0;
+ #pragma unroll
+ for (int j = 0; j < 4; j++)
+ {
+ uint8_t packed = sub_qs[g * 4 + j];
+ wpack_lo |= ((int)kvalues_iq4nl_mmq[packed & 0x0F] & 0xFF) << (j * 8);
+ wpack_hi |= ((int)kvalues_iq4nl_mmq[packed >> 4] & 0xFF) << (j * 8);
+ }
+
+ int xpack_lo = *reinterpret_cast(xq + g * 4);
+ int xpack_hi = *reinterpret_cast(xq + 16 + g * 4);
+ dot = __dp4a(wpack_lo, xpack_lo, dot);
+ dot = __dp4a(wpack_hi, xpack_hi, dot);
+ }
+
+ s_acc[row_unit * 256 + tid] += dl * dx * (float)dot;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q2_k_mmq_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ // Stage 2: dp4a accumulation. Same shape as quantized_gemv_q2_k_mmq's Stage 2,
+ // except the per-chunk INT8 packs / dx / per-half-chunk sums come from device
+ // scratch populated upstream by quantize_x_to_q8_1 (matching Q6_K's layout).
+ const int total_units = rows_in_block * superblocks_per_row;
+
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ float row_acc = 0.0f;
+
+ // 8 chunks per super-block, each chunk covers 2 sub-blocks (isc=0,1).
+ #pragma unroll
+ for (int q_quad = 0; q_quad < 8; q_quad++)
+ {
+ int chunk_idx = sb * 8 + q_quad;
+ const uint8_t* chunk_qs = qs + q_quad * 8; // 8 bytes = 2 sub-blocks of 4 bytes
+
+ #pragma unroll
+ for (int isc = 0; isc < 2; isc++)
+ {
+ int sub = q_quad * 2 + isc; // sub-block index 0..15
+ int sc = scales[sub] & 0x0F; // 4-bit scale
+ int dm = (scales[sub] >> 4) & 0x0F; // 4-bit dmin coef
+
+ const uint8_t* sub_qs = chunk_qs + isc * 4; // 4 bytes for this sub-block
+ int l_base = isc * 16;
+ const int8_t* xq = xq_in + chunk_idx * 32 + l_base;
+
+ int dot = 0;
+ // 16 elements / 4 per dp4a = 4 dp4a calls. Each byte qs[g] holds 4
+ // consecutive 2-bit values at bit offsets 0,2,4,6.
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ uint32_t qbyte = sub_qs[g];
+ int q0 = (int)((qbyte >> 0) & 0x3);
+ int q1 = (int)((qbyte >> 2) & 0x3);
+ int q2v = (int)((qbyte >> 4) & 0x3);
+ int q3 = (int)((qbyte >> 6) & 0x3);
+ int wpack = (q0 & 0xFF)
+ | ((q1 & 0xFF) << 8)
+ | ((q2v & 0xFF) << 16)
+ | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast(xq + 4 * g);
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(dx_in[chunk_idx]);
+ float sx = __half2float(sx2_in[chunk_idx * 2 + isc]);
+
+ // Per-element identity:
+ // Σ (d*sc*q - dmin*dm) * x ≈ dx * (d*sc * Σ q*xq - dmin*dm * Σ xq)
+ row_acc += dx * (d * (float)sc * (float)dot - dmin * (float)dm * sx);
+ }
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ // Stage 3: per-row reduction (same shape as Q4_K/Q5_K/Q6_K _preq).
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q4_k_mmq_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ const int total_units = rows_in_block * superblocks_per_row;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 144;
+ const uint8_t* block = w_row + sb * 144;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qs = block + 16;
+
+ float row_acc = 0.0f;
+
+ #pragma unroll
+ for (int pair = 0; pair < 4; pair++)
+ {
+ int sb_even = pair * 2;
+ int sb_odd = pair * 2 + 1;
+
+ int sc0, m0, sc1, m1;
+ if (sb_even < 4)
+ {
+ sc0 = scales_raw[sb_even] & 0x3F;
+ m0 = scales_raw[sb_even + 4] & 0x3F;
+ sc1 = scales_raw[sb_odd] & 0x3F;
+ m1 = scales_raw[sb_odd + 4] & 0x3F;
+ }
+ else
+ {
+ sc0 = (scales_raw[sb_even + 4] & 0x0F) | ((scales_raw[sb_even - 4] >> 6) << 4);
+ m0 = (scales_raw[sb_even + 4] >> 4) | ((scales_raw[sb_even] >> 6) << 4);
+ sc1 = (scales_raw[sb_odd + 4] & 0x0F) | ((scales_raw[sb_odd - 4] >> 6) << 4);
+ m1 = (scales_raw[sb_odd + 4] >> 4) | ((scales_raw[sb_odd] >> 6) << 4);
+ }
+
+ const uint8_t* pair_qs = qs + pair * 32;
+ int chunk_even = sb * 8 + sb_even;
+ int chunk_odd = sb * 8 + sb_odd;
+
+ const int8_t* xq_even = xq_in + chunk_even * 32;
+ const int8_t* xq_odd = xq_in + chunk_odd * 32;
+
+ int dot0 = 0;
+ int dot1 = 0;
+
+ #pragma unroll
+ for (int g = 0; g < 8; g++)
+ {
+ uint32_t qpacked = *reinterpret_cast(pair_qs + g * 4);
+ int lo = (int)(qpacked & 0x0F0F0F0F);
+ int hi = (int)((qpacked >> 4) & 0x0F0F0F0F);
+
+ int xq_e_packed = *reinterpret_cast(xq_even + g * 4);
+ int xq_o_packed = *reinterpret_cast(xq_odd + g * 4);
+
+ dot0 = __dp4a(lo, xq_e_packed, dot0);
+ dot1 = __dp4a(hi, xq_o_packed, dot1);
+ }
+
+ float dx_e = __half2float(dx_in[chunk_even]);
+ float dx_o = __half2float(dx_in[chunk_odd]);
+ // Full-chunk sum = lo-half + hi-half (Q6_K stores both; Q4_K only needs the sum).
+ float sx_e = __half2float(sx2_in[chunk_even * 2 + 0]) + __half2float(sx2_in[chunk_even * 2 + 1]);
+ float sx_o = __half2float(sx2_in[chunk_odd * 2 + 0]) + __half2float(sx2_in[chunk_odd * 2 + 1]);
+
+ row_acc += dx_e * (d * (float)sc0 * (float)dot0 - dmin * (float)m0 * sx_e);
+ row_acc += dx_o * (d * (float)sc1 * (float)dot1 - dmin * (float)m1 * sx_o);
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q5_k_mmq_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ const int total_units = rows_in_block * superblocks_per_row;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 176;
+ const uint8_t* block = w_row + sb * 176;
+ float d = __half2float(*reinterpret_cast(block));
+ float dmin = __half2float(*reinterpret_cast(block + 2));
+ const uint8_t* scales_raw = block + 4;
+ const uint8_t* qh = block + 16;
+ const uint8_t* qs = block + 48;
+
+ float row_acc = 0.0f;
+
+ #pragma unroll
+ for (int sub = 0; sub < 8; sub++)
+ {
+ int sc, m;
+ if (sub < 4)
+ {
+ sc = scales_raw[sub] & 0x3F;
+ m = scales_raw[sub + 4] & 0x3F;
+ }
+ else
+ {
+ sc = (scales_raw[sub + 4] & 0x0F) | ((scales_raw[sub - 4] >> 6) << 4);
+ m = (scales_raw[sub + 4] >> 4) | ((scales_raw[sub] >> 6) << 4);
+ }
+
+ int chunk_idx = sb * 8 + sub;
+ const int8_t* xq = xq_in + chunk_idx * 32;
+ const int pair_idx = sub >> 1;
+ const int nibble_half = sub & 1;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 8; g++)
+ {
+ int pos0 = 4 * g;
+ int p0 = qs[pair_idx * 32 + pos0 + 0];
+ int p1 = qs[pair_idx * 32 + pos0 + 1];
+ int p2 = qs[pair_idx * 32 + pos0 + 2];
+ int p3 = qs[pair_idx * 32 + pos0 + 3];
+ int q0 = ((p0 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 0] >> sub) & 1) << 4);
+ int q1 = ((p1 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 1] >> sub) & 1) << 4);
+ int q2 = ((p2 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 2] >> sub) & 1) << 4);
+ int q3 = ((p3 >> (4 * nibble_half)) & 0x0F) | (((qh[pos0 + 3] >> sub) & 1) << 4);
+
+ int wpack = (q0 & 0xFF) | ((q1 & 0xFF) << 8)
+ | ((q2 & 0xFF) << 16) | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast(xq + 4 * g);
+
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(dx_in[chunk_idx]);
+ float sx = __half2float(sx2_in[chunk_idx * 2 + 0]) + __half2float(sx2_in[chunk_idx * 2 + 1]);
+ row_acc += dx * (d * (float)sc * (float)dot - dmin * (float)m * sx);
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+extern "C" __global__ void __launch_bounds__(256, 2) quantized_gemv_q6_k_mmq_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row_base = blockIdx.x * MMQ_ROWS_PER_BLOCK;
+ if (row_base >= n) return;
+ const int rows_in_block = (n - row_base) < MMQ_ROWS_PER_BLOCK
+ ? (n - row_base)
+ : MMQ_ROWS_PER_BLOCK;
+
+ const int superblocks_per_row = k / 256;
+
+ __shared__ float s_acc[MMQ_ROWS_PER_BLOCK * 256];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ #pragma unroll
+ for (int r = 0; r < MMQ_ROWS_PER_BLOCK; r++)
+ s_acc[r * 256 + tid] = 0.0f;
+
+ __syncthreads();
+
+ const int total_units = rows_in_block * superblocks_per_row;
+ for (int unit = tid; unit < total_units; unit += blockDim.x)
+ {
+ int r = unit / superblocks_per_row;
+ int sb = unit % superblocks_per_row;
+ int row = row_base + r;
+
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 210;
+ const uint8_t* block = w_row + sb * 210;
+ const uint8_t* ql = block;
+ const uint8_t* qh = block + 128;
+ const int8_t* scales = reinterpret_cast(block + 192);
+ float d = __half2float(*reinterpret_cast(block + 208));
+
+ float row_acc = 0.0f;
+
+ #pragma unroll
+ for (int half_idx = 0; half_idx < 2; half_idx++)
+ {
+ const uint8_t* ql_half = ql + half_idx * 64;
+ const uint8_t* qh_half = qh + half_idx * 32;
+ const int8_t* sc_half = scales + half_idx * 8;
+
+ #pragma unroll
+ for (int q_quad = 0; q_quad < 4; q_quad++)
+ {
+ int ql_offset = (q_quad & 1) * 32;
+ int ql_shift = (q_quad & 2) << 1;
+ int qh_shift = q_quad << 1;
+ int chunk_idx = sb * 8 + half_idx * 4 + q_quad;
+
+ #pragma unroll
+ for (int isc = 0; isc < 2; isc++)
+ {
+ int sc = sc_half[q_quad * 2 + isc];
+
+ int dot = 0;
+ int l_base = isc * 16;
+ const int8_t* xq = xq_in + chunk_idx * 32 + l_base;
+
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ int wpack = 0;
+ #pragma unroll
+ for (int b = 0; b < 4; b++)
+ {
+ int l = l_base + g * 4 + b;
+ int q = ((ql_half[l + ql_offset] >> ql_shift) & 0x0F)
+ | (((qh_half[l] >> qh_shift) & 3) << 4);
+ wpack |= (q & 0xFF) << (b * 8);
+ }
+ int xpack = *reinterpret_cast(xq + 4 * g);
+ dot = __dp4a(wpack, xpack, dot);
+ }
+
+ float dx = __half2float(dx_in[chunk_idx]);
+ float sx = __half2float(sx2_in[chunk_idx * 2 + isc]);
+ row_acc += dx * d * (float)sc * ((float)dot - 32.0f * sx);
+ }
+ }
+ }
+
+ s_acc[r * 256 + tid] += row_acc;
+ }
+ __syncthreads();
+
+ if (warp_id < rows_in_block)
+ {
+ float v = 0.0f;
+ #pragma unroll
+ for (int i = 0; i < 8; i++)
+ v += s_acc[warp_id * 256 + i * 32 + lane];
+
+ for (int offset = 16; offset > 0; offset >>= 1)
+ v += __shfl_xor_sync(0xFFFFFFFF, v, offset);
+
+ if (lane == 0)
+ y[row_base + warp_id] = __float2half(v);
+ }
+}
+
+// ────────────────────────────────────────────────────────────────────────────
+// MMVQ-large `_preq` variants — 1 row per CUDA block, 128 threads, no Stage 1.
+// Stage 2/3 identical to the on-the-fly variants except scratch comes from the
+// passed device pointers instead of __shared__ buffers populated in-kernel.
+// ────────────────────────────────────────────────────────────────────────────
+
+extern "C" __global__ void __launch_bounds__(MMVQ_LARGE_THREADS) quantized_gemv_q2_k_mmvq_large_preq(
+ const uint8_t* __restrict__ weight,
+ const int8_t* __restrict__ xq_in,
+ const half* __restrict__ dx_in,
+ const half* __restrict__ sx2_in,
+ half* __restrict__ y,
+ const int n,
+ const int k)
+{
+ const int row = blockIdx.x;
+ if (row >= n) return;
+
+ const int superblocks_per_row = k / 256;
+
+ __shared__ float s_warp_partials[MMVQ_LARGE_NWARPS];
+
+ const int tid = threadIdx.x;
+ const int warp_id = tid >> 5;
+ const int lane = tid & 31;
+
+ // Stage 2: same (sb, slot) layout as the on-the-fly Q2_K MMVQ-large kernel.
+ // slot ∈ [0, 16) decodes as q_quad = slot >> 1, isc = slot & 1.
+ const int total_units = superblocks_per_row * 16;
+ const uint8_t* w_row = weight + (size_t)row * superblocks_per_row * 84;
+ float acc = 0.0f;
+
+ for (int unit = tid; unit < total_units; unit += MMVQ_LARGE_THREADS)
+ {
+ int sb = unit >> 4;
+ int slot = unit & 15;
+ int q_quad = slot >> 1;
+ int isc = slot & 1;
+
+ const uint8_t* block = w_row + sb * 84;
+ const uint8_t* scales = block;
+ const uint8_t* qs = block + 16;
+ float d = __half2float(*reinterpret_cast(block + 80));
+ float dmin = __half2float(*reinterpret_cast(block + 82));
+
+ int sub = q_quad * 2 + isc;
+ int sc = scales[sub] & 0x0F;
+ int dm = (scales[sub] >> 4) & 0x0F;
+
+ const uint8_t* chunk_qs = qs + q_quad * 8;
+ const uint8_t* sub_qs = chunk_qs + isc * 4;
+ int chunk_idx = sb * 8 + q_quad;
+ int l_base = isc * 16;
+ const int8_t* xq = xq_in + chunk_idx * 32 + l_base;
+
+ int dot = 0;
+ #pragma unroll
+ for (int g = 0; g < 4; g++)
+ {
+ uint32_t qbyte = sub_qs[g];
+ int q0 = (int)((qbyte >> 0) & 0x3);
+ int q1 = (int)((qbyte >> 2) & 0x3);
+ int q2v = (int)((qbyte >> 4) & 0x3);
+ int q3 = (int)((qbyte >> 6) & 0x3);
+ int wpack = (q0 & 0xFF)
+ | ((q1 & 0xFF) << 8)
+ | ((q2v & 0xFF) << 16)
+ | ((q3 & 0xFF) << 24);
+ int xpack = *reinterpret_cast