loader(safetensors): port MLA / Mixtral / Qwen-MoE loaders onto ISafetensorsTensorSource

README.md

@@ -672,6 +672,9 @@ Both modes transparently reuse the embedded chat UI assets if `serveUi: true`. T
 
 ## News
 
+- **2026-04** — **Qwen-MoE support (Qwen1.5/2/3-MoE + shared experts)** — extends the Mixtral MoE plumbing to the HF Qwen-MoE naming convention (`mlp.gate` + `mlp.experts.{j}.{gate_proj,up_proj,down_proj}` instead of Mixtral's `block_sparse_moe.gate` + `experts.{j}.w1/w2/w3`), with optional shared-expert branch (Qwen1.5-MoE-A2.7B: `mlp.shared_expert.*` + optional `mlp.shared_expert_gate.weight` sigmoid scalar) and the `norm_topk_prob=false` raw-softmax gating used by Qwen1.5. New `Architecture.QwenMoe` enum variant dispatches `Qwen{2,3}MoeForCausalLM` / `model_type=qwen{2,3}_moe`. `MoeConfig` gains `NormTopKProb`, `SharedExpertIntermediateSize`, `HasSharedExpertGate`, `DecoderSparseStep`, and `MlpOnlyLayers` — the last two let Qwen3-MoE interleave dense MLP and MoE layers in the same model (`decoder_sparse_step=2` → layer 0 dense, layer 1 MoE). New `MoeSwiGluMlp.ExecuteWithSharedExpert` overload runs a parallel dense SwiGLU on every token and (optionally) multiplies it by `sigmoid(hidden . shared_expert_gate)` before adding to the routed top-k sum. Existing Mixtral `Execute` call-sites are untouched — the kernel change is additive. Verified end-to-end against the real `yujiepan/qwen3-moe-tiny-random` HF checkpoint (~20 MB, 2 layers × 8 experts × top-2, `decoder_sparse_step=2`, no shared expert): detection → load → 3-token forward → finite logits. Synthetic-fixture coverage for the shared-expert + sigmoid-gate + raw-softmax path (Qwen1.5-MoE convention). DeepSeek-V2/V3 multi-shared-expert (`n_shared_experts > 1`) + MLA attention remain out of scope
+- **2026-04** — **Mixtral-family MoE support** — dense-routing top-k Mixture-of-Experts for Mixtral-convention models (Mixtral, Qwen*-MoE without shared experts, Phi-3.5-MoE). New `MoeConfig` on `ModelConfig` (`NumExperts`, `NumExpertsPerTok`, `MoeIntermediateSize`), `Architecture.Mixtral` enum variant, `HfConfigExtractor` detects `num_local_experts` / `num_experts` + `num_experts_per_tok` and surfaces Phi-3.5's `moe_intermediate_size` override. `MoeSwiGluMlp` kernel: full softmax over experts → top-k partial max-scan (stable tiebreak: lower index wins, matching `torch.topk`) → renormalise by sum (Mixtral convention, NOT a second softmax) → per-expert SwiGLU MLP via existing `FusedOps.SwiGLU` → weighted sum. `TransformerModel.Forward` branches on `TransformerLayerWeights.Moe`; safetensors loader resolves `block_sparse_moe.gate` + `experts.{j}.w1/w2/w3`, F16/BF16 → F32 upcast at load time. Verified against real `yujiepan/mixtral-tiny-random` (config + detection) and a synthetic 2-layer, 4-expert, top-2 fixture (full forward pass). Out of scope: shared experts (DeepSeek-V3), Qwen-MoE `mlp.experts` naming adapter, fused GroupedGEMM, expert parallelism, real Mixtral-8x7B validation
+- **2026-04** — Safetensors loader for dense transformers — `ModelLoader.LoadFromSafetensors` + `TransformerModel.LoadFromSafetensors` ingest HuggingFace `model.safetensors` + `config.json` for Llama/Mistral/Phi/Qwen. `HfConfigExtractor` mirrors the GGUF extractor pattern over HF JSON fields (`hidden_size`, `num_hidden_layers`, `num_key_value_heads`, `rope_theta`, `tie_word_embeddings`, …). bf16 tensors are upcast into 64-byte-aligned scratch at load time; F32 tensors are zero-copy mmap views. `ModelLoader.Load(path)` auto-detects `.gguf` vs `.safetensors`. Verified end-to-end on `hf-internal-testing/tiny-random-LlamaForCausalLM`
 - **2026-04** — **First public release (v0.1.0-preview.1)** — dotLLM goes public. [NuGet packages](#nuget-packages) for all 10 libraries + `DotLLM.Cli` as a global `dotnet tool`. Self-contained single-file downloads for Windows / Linux / macOS (Apple Silicon) and experimental Native AOT builds for Linux / Windows attached to every [GitHub Release](https://github.com/kkokosa/dotLLM/releases). Companion website at [dotllm.dev](https://dotllm.dev/) ([#119](https://github.com/kkokosa/dotLLM/issues/119))
 - **2026-04** — **Wave 7**: CPU performance cleanup pass — `TopKSampler` replaces full `Array.Sort` with a hand-rolled size-K min-heap (`O(N log K)`, stack-resident scratch); `JsonSchemaConstraint` adds first-char bucketing to skip the ~160 MB of struct clones per mask build when the tracker rejects most leading characters, plus LRU eviction instead of the previous full-flush cache overflow; `Dequantize.Q5_0` gains an AVX2 path matching Q8_0's throughput (reuses `MatMulQ5_0.ExtractQ5HighBits` / `vpshufb` bit-extraction); `BpeTokenizer` pre-splits special tokens via the existing `Trie.TryMatchLongest` instead of the O(n × m) linear scan; `ComputeThreadPool` now pins the caller (inference) thread to the first candidate P-core on first `Dispatch`, eliminating the hybrid-CPU stall where pinned P-core workers idled at the barrier waiting for an E-core caller. New BenchmarkDotNet suites for TopK sampling, schema mask build, and special-token encode ([#109](https://github.com/kkokosa/dotLLM/issues/109))
 - **2026-04** — **Phase 7 begins**: Logprobs — OpenAI-compatible `logprobs: true` + `top_logprobs: N` (0-20) on `/v1/chat/completions` and `/v1/completions`. Per-token log-softmax captured before sampling, returned in both streaming SSE chunks and non-streaming responses. Chat UI gains opt-in logprobs visualization: color-coded token confidence (green/lime/yellow/orange/red), hover tooltips with top-K alternatives and probabilities, diagnostic cues for low confidence, ambiguity, and sampling effect. `DotLLM.Sample.Logprobs` console sample with ANSI-colored output ([#101](https://github.com/kkokosa/dotLLM/issues/101))
@@ -716,13 +719,13 @@ Both modes transparently reuse the embedded chat UI assets if `serveUi: true`. T
 | Phase | Description | Status |
 |-------|-------------|--------|
 | **1 — End-to-End Generation** | GGUF loading, dequantization, CPU ops, tokenizer, attention, forward pass, KV-cache, sampling | Done (9/9) |
-| **2 — Practical Local Inference** | Engine metrics, benchmarks, Q4_K_M, chat templates, streaming, multi-threading, more architectures | Done (10/10) |
+| **2 — Practical Local Inference** | Engine metrics, benchmarks, Q4_K_M, chat templates, streaming, multi-threading, more architectures, safetensors loader | Done (11/11) |
 | **3 — CPU Performance** | Decode dispatch, Q8_1 input, weight repacking, outer-product GEMM, tiled attention, fast exp, fusion, NUMA | In Progress (7/8) |
 | **4 — GPU Acceleration** | CUDA backend, CPU/GPU hybrid, KV-cache quantization | Done (3/3) |
 | **5 — Constrained Decoding & API** | JSON mode, JSON Schema, regex/CFG, tool calling, OpenAI API server, chat UI, prompt caching | Done (7/7) |
 | **6 — Improved Serving** | Warm-up, NativeAOT, paged KV-cache, speculative decoding | Done (4/4) |
 | **7 — Diagnostics & Interpretability** | Logprobs, hook system, logit lens, SAE integration, LoRA adapters | In Progress (1/5) |
-| **8 — Model Expansion** | MLA attention, ALiBi, SmolLM3, Gemma 4, Mixture of Experts | Planned (0/5) |
+| **8 — Model Expansion** | MLA attention, ALiBi, SmolLM3, Gemma 4, Mixture of Experts | In Progress (1/5) |
 | **9 — Production Serving** | Continuous batching, prefix sharing, advanced scheduling, rate limiting, metrics & tracing | Planned (0/5) |
 
 See [docs/ROADMAP.md](docs/ROADMAP.md) for detailed steps, dependencies, and milestones.

benchmarks/DotLLM.Benchmarks/Columns/ColumnHelpers.cs

@@ -1,4 +1,5 @@
 using BenchmarkDotNet.Running;
+using DotLLM.Benchmarks.Lora;
 
 namespace DotLLM.Benchmarks.Columns;
 
@@ -26,16 +27,62 @@ internal static class ColumnHelpers
     }
 
     /// <summary>
-    /// Returns the metrics key for a benchmark case. When <c>DOTLLM_BENCH_MODEL_PATH</c> is set,
-    /// uses the filename stem; otherwise falls back to the <see cref="BenchmarkModel"/> enum name.
+    /// Returns the metrics key for a benchmark case. Resolution order:
+    /// <list type="number">
+    /// <item><see cref="LoraMacroBenchmarks"/> cases — composite key from
+    /// (model-label, variant, scenario), matching what the bench writes.</item>
+    /// <item><c>DOTLLM_BENCH_MODEL_PATH</c> env var — filename stem.</item>
+    /// <item><see cref="BenchmarkModel"/> param — the enum name.</item>
+    /// </list>
     /// </summary>
     public static string? TryGetMetricsKey(BenchmarkCase benchmarkCase)
     {
+        // LoRA macro-bench cases write a composite key — match it here so the
+        // shared Prefill / Decode columns surface the right value per case
+        // even though it carries no BenchmarkModel parameter.
+        var loraKey = TryGetLoraMacroKey(benchmarkCase);
+        if (loraKey is not null)
+            return loraKey;
+
         var envPath = Environment.GetEnvironmentVariable("DOTLLM_BENCH_MODEL_PATH");
         if (!string.IsNullOrEmpty(envPath))
             return Path.GetFileNameWithoutExtension(envPath);
 
         var model = TryGetModel(benchmarkCase);
         return model?.ToString();
     }
+
+    /// <summary>
+    /// Composes the <see cref="LoraMacroBenchmarks"/> metrics key from a benchmark
+    /// case's parameters. Returns <c>null</c> when the case is not a LoRA macro-bench
+    /// case (i.e. doesn't carry both <see cref="LoraVariant"/> and <see cref="LoraScenario"/>).
+    /// </summary>
+    private static string? TryGetLoraMacroKey(BenchmarkCase benchmarkCase)
+    {
+        if (!benchmarkCase.HasParameters) return null;
+
+        LoraVariant? variant = null;
+        LoraScenario? scenario = null;
+        foreach (var item in benchmarkCase.Parameters.Items)
+        {
+            if (item.Value is LoraVariant v) variant = v;
+            else if (item.Value is LoraScenario s) scenario = s;
+        }
+        if (variant is null || scenario is null) return null;
+
+        // The fixture label is determined at runtime; we cannot recover it
+        // from BDN params. Probe the on-disk metrics dir for the first key
+        // matching the expected suffix — there will be one per (variant, scenario).
+        string suffix = $"_{variant.Value}_{scenario.Value}";
+        string dir = Path.Combine(Path.GetTempPath(), "dotllm-bdn-metrics");
+        if (!Directory.Exists(dir)) return null;
+
+        foreach (var file in Directory.EnumerateFiles(dir, "Lora_*.json"))
+        {
+            string stem = Path.GetFileNameWithoutExtension(file);
+            if (stem.EndsWith(suffix, StringComparison.Ordinal))
+                return stem;
+        }
+        return null;
+    }
 }

benchmarks/DotLLM.Benchmarks/Lora/LoraDeltaOverheadBenchmark.cs

@@ -0,0 +1,123 @@
+using System.Runtime.InteropServices;
+using BenchmarkDotNet.Attributes;
+using DotLLM.Core.Lora;
+using DotLLM.Cpu.Kernels;
+
+namespace DotLLM.Benchmarks.Lora;
+
+/// <summary>
+/// Phase 4d.3 — Measures LoRA delta overhead vs the bare base projection.
+/// Baseline = a single F32 GEMM at TinyLlama-1.1B q_proj shapes
+/// (hidden=2048, q_out=2048, seq=128 typical prefill chunk).
+/// LoRA path = baseline + <c>scale × (x · B) · A</c> at r=16.
+/// Target: &lt;5% overhead on the bare projection.
+/// </summary>
+/// <remarks>
+/// We benchmark at the kernel level (no model load) because the spec target
+/// is the additional cost of the delta itself, and a kernel bench is fully
+/// reproducible without checkpoint download. The macro-bench against a real
+/// TinyLlama checkpoint is tracked as a follow-up — once a public checkpoint
+/// path is wired into the bench harness, replace this file with a
+/// model-level forward-pass bench.
+/// </remarks>
+[MemoryDiagnoser]
+[ShortRunJob]
+public unsafe class LoraDeltaOverheadBenchmark
+{
+    /// <summary>Sequence length (prefill chunk size).</summary>
+    [Params(1, 128)]
+    public int SeqLen { get; set; }
+
+    /// <summary>LoRA rank.</summary>
+    [Params(16)]
+    public int Rank { get; set; }
+
+    // TinyLlama q_proj shape.
+    private const int HiddenSize = 2048;
+    private const int OutputDim = 2048;
+
+    private nint _xPtr;
+    private nint _yBasePtr;
+    private nint _yLoraPtr;
+    private nint _wPtr;     // base weight [OutputDim, HiddenSize]
+    private nint _bPtr;     // LoRA B [Rank, HiddenSize]
+    private nint _aPtr;     // LoRA A [OutputDim, Rank]
+
+    [GlobalSetup]
+    public void Setup()
+    {
+        var rng = new Random(123);
+
+        _xPtr = AllocAligned(SeqLen * HiddenSize);
+        _yBasePtr = AllocAligned(SeqLen * OutputDim);
+        _yLoraPtr = AllocAligned(SeqLen * OutputDim);
+        _wPtr = AllocAligned(OutputDim * HiddenSize);
+        _bPtr = AllocAligned(Rank * HiddenSize);
+        _aPtr = AllocAligned(OutputDim * Rank);
+
+        FillRandom((float*)_xPtr, SeqLen * HiddenSize, rng, 0.05f);
+        FillRandom((float*)_wPtr, OutputDim * HiddenSize, rng, 0.05f);
+        FillRandom((float*)_bPtr, Rank * HiddenSize, rng, 0.05f);
+        FillRandom((float*)_aPtr, OutputDim * Rank, rng, 0.05f);
+    }
+
+    [GlobalCleanup]
+    public void Cleanup()
+    {
+        FreeAligned(_xPtr);
+        FreeAligned(_yBasePtr);
+        FreeAligned(_yLoraPtr);
+        FreeAligned(_wPtr);
+        FreeAligned(_bPtr);
+        FreeAligned(_aPtr);
+    }
+
+    /// <summary>Baseline: only the base GEMM projection.</summary>
+    [Benchmark(Baseline = true)]
+    public void BaseProjectionOnly()
+    {
+        // C[N, M] = B[N, K] × A[M, K]^T, so y = x · w^T.
+        MatMul.GemmF32((float*)_wPtr, (float*)_xPtr, (float*)_yBasePtr,
+            OutputDim, HiddenSize, SeqLen);
+    }
+
+    /// <summary>Base GEMM + F32 LoRA delta (Phase 4a path).</summary>
+    [Benchmark]
+    public void BasePlusLoraF32()
+    {
+        MatMul.GemmF32((float*)_wPtr, (float*)_xPtr, (float*)_yLoraPtr,
+            OutputDim, HiddenSize, SeqLen);
+        LoraDelta.Apply(
+            (float*)_xPtr, (float*)_bPtr, (float*)_aPtr, (float*)_yLoraPtr,
+            SeqLen, HiddenSize, OutputDim, Rank, scale: 0.5f);
+    }
+
+    /// <summary>Base GEMM + F16 LoRA delta (Phase 4d.1 path).</summary>
+    [Benchmark]
+    public void BasePlusLoraF16()
+    {
+        MatMul.GemmF32((float*)_wPtr, (float*)_xPtr, (float*)_yLoraPtr,
+            OutputDim, HiddenSize, SeqLen);
+        // Reinterpret existing F32 buffers as F16 for the dispatch test —
+        // we measure dispatch + dequant overhead, not the math (the test
+        // suite already verifies numerical parity).
+        LoraDelta.Apply(
+            (float*)_xPtr, (void*)_bPtr, (void*)_aPtr, (float*)_yLoraPtr,
+            SeqLen, HiddenSize, OutputDim, Rank, scale: 0.5f,
+            LoraWeightDType.F16, LoraWeightDType.F16);
+    }
+
+    private static nint AllocAligned(long elementCount)
+        => (nint)NativeMemory.AlignedAlloc((nuint)(elementCount * sizeof(float)), 64);
+
+    private static void FreeAligned(nint p)
+    {
+        if (p != 0) NativeMemory.AlignedFree((void*)p);
+    }
+
+    private static void FillRandom(float* p, long n, Random rng, float scale)
+    {
+        for (long i = 0; i < n; i++)
+            p[i] = ((float)rng.NextDouble() * 2f - 1f) * scale;
+    }
+}

benchmarks/DotLLM.Benchmarks/Lora/LoraMacroBenchFixture.cs

@@ -0,0 +1,104 @@
+namespace DotLLM.Benchmarks.Lora;
+
+/// <summary>
+/// Resolves a local GGUF checkpoint for the LoRA macro-bench, in priority order:
+/// <list type="number">
+/// <item>The <c>DOTLLM_BENCH_MODEL_PATH</c> env var (manual override).</item>
+/// <item>A TinyLlama GGUF anywhere under <c>~/.dotllm/test-cache/</c>.</item>
+/// <item>Llama-3.2-1B-Instruct Q8_0 GGUF (the closest available stand-in).</item>
+/// <item>SmolLM-135M Q8_0 GGUF (smallest fallback).</item>
+/// </list>
+/// We deliberately do NOT trigger downloads here — Phase 4d.3 is a measurement
+/// follow-up, not a fixture provisioner, and the parent agent is offline-tolerant.
+/// </summary>
+internal static class LoraMacroBenchFixture
+{
+    /// <summary>
+    /// Searches the local test-cache for a usable checkpoint. Returns the
+    /// resolved path + a short label, or <c>null</c> + a skip reason.
+    /// </summary>
+    public static string? ResolveCheckpoint(out string label, out string? skipReason)
+    {
+        // 1. Manual override.
+        var envPath = Environment.GetEnvironmentVariable("DOTLLM_BENCH_MODEL_PATH");
+        if (!string.IsNullOrEmpty(envPath) && File.Exists(envPath))
+        {
+            label = Path.GetFileNameWithoutExtension(envPath);
+            skipReason = null;
+            return envPath;
+        }
+
+        string cacheRoot = Path.Combine(
+            Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
+            ".dotllm", "test-cache");
+
+        if (!Directory.Exists(cacheRoot))
+        {
+            label = string.Empty;
+            skipReason = $"test-cache directory does not exist: {cacheRoot}";
+            return null;
+        }
+
+        // 2. TinyLlama GGUF — preferred per Phase 4d.3 spec.
+        //    Probe common filename stems used by HF mirrors.
+        foreach (var pat in TinyLlamaPatterns)
+        {
+            foreach (var hit in Directory.EnumerateFiles(cacheRoot, pat, SearchOption.AllDirectories))
+            {
+                label = "TinyLlama";
+                skipReason = null;
+                return hit;
+            }
+        }
+
+        // 3. Llama-3.2-1B Q8_0 — same scale class, exercises the same forward
+        //    path on the same architecture family.
+        string l32 = Path.Combine(cacheRoot,
+            "bartowski", "Llama-3.2-1B-Instruct-GGUF", "Llama-3.2-1B-Instruct-Q8_0.gguf");
+        if (File.Exists(l32))
+        {
+            label = "Llama32_1B";
+            skipReason = null;
+            return l32;
+        }
+
+        // 4. SmolLM-135M Q8_0 — smallest fallback. Useful for CI smoke runs but
+        //    note its absolute tok/s aren't representative of TinyLlama-class
+        //    bandwidth pressure — the delta % vs base is still meaningful.
+        string smol = Path.Combine(cacheRoot,
+            "QuantFactory", "SmolLM-135M-GGUF", "SmolLM-135M.Q8_0.gguf");
+        if (File.Exists(smol))
+        {
+            label = "SmolLM_135M";
+            skipReason = null;
+            return smol;
+        }
+
+        // SmolLM2 alt path (we saw both Q8 variants on disk).
+        string smol2 = Path.Combine(cacheRoot,
+            "bartowski", "SmolLM2-135M-Instruct-GGUF", "SmolLM2-135M-Instruct-Q8_0.gguf");
+        if (File.Exists(smol2))
+        {
+            label = "SmolLM2_135M";
+            skipReason = null;
+            return smol2;
+        }
+
+        label = string.Empty;
+        skipReason =
+            "no usable GGUF checkpoint found under ~/.dotllm/test-cache/. "
+          + "Set DOTLLM_BENCH_MODEL_PATH=<path-to-gguf> or place a TinyLlama / "
+          + "Llama-3.2-1B / SmolLM-135M GGUF in the test-cache.";
+        return null;
+    }
+
+    private static readonly string[] TinyLlamaPatterns =
+    [
+        "*TinyLlama*Q8_0*.gguf",
+        "*tinyllama*q8_0*.gguf",
+        "*TinyLlama*Q4_K_M*.gguf",
+        "*tinyllama*q4_k_m*.gguf",
+        "*TinyLlama*.gguf",
+        "*tinyllama*.gguf",
+    ];
+}