feat(eval): shared perplexity harness with llama.cpp-comparable sliding-window mode

docs/superpowers/specs/2026-07-30-perplexity-harness-design.md

@@ -0,0 +1,110 @@
+# Perplexity harness — design
+
+**Issue:** jamesburton/dotLLM#231 · **Upstream driver:** kkokosa/dotLLM#416 · **Date:** 2026-07-30
+
+## Problem
+
+Upstream #416's investigation log ends on a blocker: *"Every remaining lever changes numerics and
+none of them can currently be evaluated."* Scale-granularity repacking (G=64/128, estimated
+1.31–1.56×) and every other numerics-changing CPU lever are gated on an evaluation harness that
+does not exist upstream.
+
+`upstream/main` has only `samples/DotLLM.Sample.Logprobs/Program.cs`. Our `dev` has perplexity
+scoring in ten files — but entirely as **duplicated private per-test helpers**. `StableLogProb` is
+copy-pasted verbatim; the same English corpus string appears in multiple files; two different
+scoring strategies have diverged without a shared definition. None of it is reachable as a gate for
+kernel work.
+
+So this is consolidation plus extension, not greenfield — but the consolidation half must be done
+from `dev`, because that is where the helpers live. See Sequencing.
+
+## The one real axis
+
+The existing helpers diverged into two shapes for a single reason: **whether the backend's
+`Forward` returns logits for every position or only the last one.**
+
+- CPU returns `[seqLen, vocab]`, so one teacher-forced prefill scores every next-token NLL — O(n).
+  (`BitNetAccuracyTests.Cpu_Perplexity_OnFixedPassage_IsSane`)
+- CUDA returns only the final row, so each target needs its own growing-prefix re-prefill — O(n²),
+  which is why those helpers carry a stride to keep the sweep brisk.
+  (`CudaFlashPrefillForwardHarness.PrefillGrowingPrefixPerplexity`, and its twin in
+  `CudaG3PrefillForwardHarness`)
+
+Everything else about them is identical. Modelling that axis explicitly — `ReturnsAllRows` on
+`IPerplexityModel` — is what lets one evaluator replace both without changing any number either
+currently produces.
+
+## Components
+
+**`DotLLM.Core.Evaluation`** (abstractions only, so `main` and `dev` can both reference the
+contract without dragging in the implementation):
+
+- `IPerplexityModel` — `VocabSize`, `MaxContextLength`, `ReturnsAllRows`, `Forward(tokens, positions)`.
+  Deliberately narrower than `IModel`: perplexity needs no sampling, no KV-cache lifetime
+  management, no streaming, and binding to the full interface would prevent scoring a bare backend
+  or a test double.
+- `PerplexityMode` — `TeacherForced` | `SlidingWindow`.
+- `PerplexityOptions(Mode, ContextLength, Stride, MaxTokens)`.
+- `PerplexityResult(Perplexity, MeanNegativeLogLikelihood, ScoredTokens, WindowCount)`.
+
+**`PerplexityEvaluator`** — strategy selected from `ReturnsAllRows`, not from the caller.
+
+**Corpus handling** — streamed and tokenized in chunks.
+
+**CLI verb** — context length, stride, corpus path, token cap.
+
+## Two modes, two purposes
+
+`TeacherForced` preserves the in-tree "G1 precedent" methodology exactly. It is **ratio-oriented**:
+the load-bearing signal is the OFF/ON perplexity ratio on identical tokens under a <1% gate, not
+the absolute value. Not comparable to published figures, and must not be presented as if it were.
+
+`SlidingWindow` is new and **absolute-value oriented**: windows of `ContextLength` advanced by
+`Stride`, scoring only tokens beyond the carried-over prefix so every scored token has full-length
+context. Matches llama.cpp's `--perplexity` methodology so figures compare directly to published
+numbers.
+
+## Memory constraint (from planned Track D)
+
+On Strix Halo's UMA, a large VRAM carve-out leaves host RAM scarce, and perplexity is the workload
+most punished by it — a long sequence of full-context prefills rather than a single load. A harness
+that mmaps weights host-side while the backend separately uploads them pays for the model twice
+against an already-halved budget.
+
+Therefore, as a design constraint rather than a later optimisation:
+
+- **The evaluator never loads weights.** It takes an already-constructed `IPerplexityModel`.
+- **The corpus is streamed and tokenized in chunks**, never materialized whole.
+
+Both cost nothing now and keep Track D a pure optimisation rather than a rewrite.
+
+## Verification
+
+1. **Consolidation is behaviour-preserving.** Migrated tests must produce *numerically identical*
+   results to their previous private helpers. A changed number means the consolidation altered
+   semantics — this is the primary regression gate, and it is the reason `TeacherForced` is
+   preserved verbatim rather than "improved" in passing.
+2. **`SlidingWindow` is genuinely comparable.** Validated against a published llama.cpp perplexity
+   figure on matching model, corpus, context length and stride, within a stated tolerance. Without
+   this the word "comparable" is unearned, and the harness would give upstream false confidence on
+   exactly the numerics-changing decisions it is meant to gate.
+3. **`ScoredTokens` is reported and checked.** A perplexity over a different token count is a
+   different measurement; cross-run comparison requires it to match.
+
+## Sequencing
+
+Built from `main` in a worktree (`issue/231-perplexity-harness`), PR targets upstream `main`, then
+merges to `dev`.
+
+Because the ten duplicated helpers exist only on `dev`, the split is:
+
+- **On `main` / this PR:** the contract, the evaluator, corpus handling, CLI verb, and validation
+  of `SlidingWindow` against llama.cpp. Self-contained and upstream-contributable.
+- **On `dev`, follow-up:** migrate the ten existing helpers onto the harness, gated on producing
+  identical numbers.
+
+## Out of scope
+
+Track D (iGPU optimisation of the harness under a large VRAM carve-out, including eliminating
+host+device double-loading) is deliberately deferred until this and #233 land. The constraints
+above exist so D is optimisation, not rework.

src/DotLLM.Cli/Commands/PerplexityCommand.cs

@@ -0,0 +1,245 @@
+using System.ComponentModel;
+using System.Diagnostics;
+using DotLLM.Core.Configuration;
+using DotLLM.Core.Evaluation;
+using DotLLM.Core.Models;
+using DotLLM.Engine;
+using DotLLM.Engine.Evaluation;
+using DotLLM.Models.Architectures;
+using DotLLM.Models.Evaluation;
+using DotLLM.Models.Gguf;
+using Spectre.Console;
+using Spectre.Console.Cli;
+
+namespace DotLLM.Cli.Commands;
+
+/// <summary>
+/// Computes perplexity over a text corpus: load → stream-tokenize → score.
+/// </summary>
+/// <remarks>
+/// Defaults to <see cref="PerplexityMode.SlidingWindow"/> with <c>stride = context</c> and
+/// <c>unscored prefix = context / 2 + 1</c> — non-overlapping chunks, each scoring the targets after
+/// its midpoint. That reproduces llama.cpp's <c>--perplexity</c> methodology, so the reported figure
+/// is directly comparable to published numbers for the same model, corpus and context.
+/// <para>Advance and scored span are separate knobs on purpose: llama.cpp advances by the whole
+/// window yet scores only part of it, so its scored ranges have gaps. A single "stride" cannot
+/// express that.</para>
+/// </remarks>
+internal sealed class PerplexityCommand : AsyncCommand<PerplexityCommand.Settings>
+{
+    /// <summary>
+    /// Delimiters accepted in a <c>--tokens-file</c>: whitespace plus the punctuation of a
+    /// JSON array, so a reference implementation's dump parses as-is.
+    /// </summary>
+    private static readonly char[] TokenIdSeparators = [' ', '\t', '\r', '\n', ',', '[', ']'];
+
+    public sealed class Settings : CommandSettings
+    {
+        [CommandArgument(0, "<model>")]
+        [Description("Path to a GGUF file or HuggingFace repo ID (e.g., QuantFactory/SmolLM-135M-GGUF).")]
+        public string Model { get; set; } = string.Empty;
+
+        [CommandOption("--corpus|-f")]
+        [Description("Path to a UTF-8 text corpus (e.g. wiki.test.raw).")]
+        public string Corpus { get; set; } = string.Empty;
+
+        [CommandOption("--context|-c")]
+        [Description("Context window in tokens. Clamped to the model's maximum sequence length.")]
+        [DefaultValue(512)]
+        public int Context { get; set; } = 512;
+
+        [CommandOption("--stride")]
+        [Description("Tokens advanced between window starts. 0 selects the context length (non-overlapping chunks, llama.cpp's default).")]
+        [DefaultValue(0)]
+        public int Stride { get; set; }
+
+        [CommandOption("--unscored-prefix")]
+        [Description("Leading tokens of each window used as context only. -1 selects context/2 + 1, which scores the same targets as llama.cpp.")]
+        [DefaultValue(-1)]
+        public int UnscoredPrefix { get; set; } = -1;
+
+        [CommandOption("--max-tokens|-n")]
+        [Description("Cap on corpus tokens consumed. 0 = unbounded.")]
+        [DefaultValue(0)]
+        public int MaxTokens { get; set; }
+
+        [CommandOption("--mode")]
+        [Description("Scoring mode: sliding-window (default, llama.cpp-comparable) or teacher-forced.")]
+        [DefaultValue("sliding-window")]
+        public string Mode { get; set; } = "sliding-window";
+
+        [CommandOption("--tokens-file")]
+        [Description("Read pre-tokenized whitespace-separated ids instead of tokenizing --corpus. Isolates scoring from tokenization when comparing against another implementation.")]
+        public string? TokensFile { get; set; }
+
+        [CommandOption("--dump-tokens")]
+        [Description("Write the tokenized corpus ids to this path, whitespace-separated, then continue. Diagnostic.")]
+        public string? DumpTokens { get; set; }
+
+        [CommandOption("--per-window")]
+        [Description("Print each window's perplexity. Use to localize a disagreement with another implementation to specific corpus content.")]
+        [DefaultValue(false)]
+        public bool PerWindow { get; set; }
+
+        [CommandOption("--bos")]
+        [Description("Substitute BOS at the start of each window. Match the model's add_bos setting: llama.cpp only does this when the tokenizer requests it.")]
+        [DefaultValue(false)]
+        public bool Bos { get; set; }
+
+        [CommandOption("--quant")]
+        [Description("Quantization to select when resolving a HuggingFace repo ID.")]
+        public string? Quant { get; set; }
+
+        [CommandOption("--threads")]
+        [Description("Compute threads. 0 = auto.")]
+        [DefaultValue(0)]
+        public int Threads { get; set; }
+    }
+
+    public override async Task<int> ExecuteAsync(CommandContext context, Settings settings)
+    {
+        // --tokens-file supplies the token stream directly, so it replaces --corpus rather than
+        // supplementing it. Requiring both would defeat the flag's purpose: scoring a reference
+        // implementation's exact ids to separate a tokenizer difference from a scoring one.
+        if (settings.TokensFile is not null)
+        {
+            if (!File.Exists(settings.TokensFile))
+            {
+                AnsiConsole.MarkupLine(
+                    $"[red]Tokens file not found: {Markup.Escape(settings.TokensFile)}[/]");
+                return 1;
+            }
+        }
+        else if (string.IsNullOrWhiteSpace(settings.Corpus))
+        {
+            AnsiConsole.MarkupLine("[red]--corpus is required (or --tokens-file).[/]");
+            return 1;
+        }
+        else if (!File.Exists(settings.Corpus))
+        {
+            AnsiConsole.MarkupLine($"[red]Corpus not found: {Markup.Escape(settings.Corpus)}[/]");
+            return 1;
+        }
+
+        if (!TryParseMode(settings.Mode, out PerplexityMode mode))
+        {
+            AnsiConsole.MarkupLine(
+                $"[red]Unknown --mode '{Markup.Escape(settings.Mode)}'. Expected 'sliding-window' or 'teacher-forced'.[/]");
+            return 1;
+        }
+
+        string? resolvedPath = GgufFileResolver.Resolve(settings.Model, settings.Quant);
+        if (resolvedPath is null)
+            return 1;
+
+        using GgufFile gguf = GgufFile.Open(resolvedPath);
+        ModelConfig config = GgufModelConfigExtractor.Extract(gguf.Metadata);
+        var tokenizer = GgufBpeTokenizerFactory.Load(gguf.Metadata);
+        using TransformerModel model = TransformerModel.LoadFromGguf(
+            gguf, config, new ThreadingConfig(settings.Threads));
+
+        int effectiveContext = Math.Min(settings.Context, config.MaxSequenceLength);
+        // Defaults reproduce llama.cpp: non-overlapping chunks, scoring the second half of each.
+        int effectiveStride = settings.Stride > 0 ? settings.Stride : effectiveContext;
+        // context/2 + 1, not context/2: llama.cpp scores targets (n_ctx/2, n_ctx), leaving the token
+        // at n_ctx/2 as context only. See PerplexityOptions.LlamaCppDefault.
+        int effectivePrefix = settings.UnscoredPrefix >= 0
+            ? settings.UnscoredPrefix
+            : Math.Min(effectiveContext - 1, Math.Max(1, effectiveContext / 2 + 1));
+
+        // Streamed, then buffered once: scoring needs random access across windows, but the file
+        // itself is never held in memory and the token list is bounded by --max-tokens.
+        var tokens = new List<int>();
+        if (settings.TokensFile is not null)
+        {
+            // Accept both bare whitespace-separated ids and the JSON-array form that reference
+            // tools print, so a dump can be pasted in without reformatting.
+            foreach (string part in File.ReadAllText(settings.TokensFile)
+                         .Split(TokenIdSeparators, StringSplitOptions.RemoveEmptyEntries))
+            {
+                tokens.Add(int.Parse(part));
+                if (settings.MaxTokens > 0 && tokens.Count >= settings.MaxTokens) break;
+            }
+        }
+        else
+        {
+            using var reader = new StreamReader(settings.Corpus);
+            foreach (int id in CorpusReader.StreamTokens(reader, tokenizer, settings.MaxTokens))
+                tokens.Add(id);
+        }
+
+        if (tokens.Count < 2)
+        {
+            AnsiConsole.MarkupLine($"[red]Corpus tokenized to {tokens.Count} tokens; at least 2 are required.[/]");
+            return 1;
+        }
+
+        if (settings.DumpTokens is not null)
+            File.WriteAllText(settings.DumpTokens, string.Join(' ', tokens));
+
+        var perplexityModel = new TransformerPerplexityModel(model, deviceId: -1);
+        int bosTokenId = settings.Bos ? tokenizer.BosTokenId : -1;
+        var options = new PerplexityOptions(
+            mode, effectiveContext, effectiveStride, settings.MaxTokens, effectivePrefix, bosTokenId);
+
+        var sw = Stopwatch.StartNew();
+        PerplexityResult result;
+        try
+        {
+            PerplexityEvaluator.WindowObserver? observer = settings.PerWindow
+                ? (i, ppl, n) => Console.WriteLine($"window {i}: ppl={ppl:F6} scored={n}")
+                : null;
+            result = PerplexityEvaluator.Evaluate(
+                perplexityModel,
+                System.Runtime.InteropServices.CollectionsMarshal.AsSpan(tokens),
+                options,
+                observer);
+        }
+        catch (ArgumentException ex)
+        {
+            AnsiConsole.MarkupLine($"[red]{Markup.Escape(ex.Message)}[/]");
+            return 1;
+        }
+        sw.Stop();
+
+        // Window geometry and scored-token count are reported alongside the figure deliberately:
+        // a perplexity without them is not comparable to anything.
+        var table = new Table().Border(TableBorder.Rounded);
+        table.AddColumn("Metric");
+        table.AddColumn(new TableColumn("Value").RightAligned());
+        // Printed as "PPL +/- err" in llama.cpp's own format so the two can be compared by eye.
+        // Without the error bar a reader has no way to tell a regression from sampling noise.
+        table.AddRow("Perplexity", $"{result.Perplexity:F4} +/- {result.StandardError:F5}");
+        table.AddRow("Mean NLL (nats)", $"{result.MeanNegativeLogLikelihood:F6}");
+        table.AddRow("Scored tokens", $"{result.ScoredTokens:N0}");
+        table.AddRow("Windows", $"{result.WindowCount:N0}");
+        table.AddRow("Mode", mode == PerplexityMode.SlidingWindow ? "sliding-window" : "teacher-forced");
+        table.AddRow("Context", $"{effectiveContext:N0}");
+        table.AddRow("Stride", $"{effectiveStride:N0}");
+        table.AddRow("Unscored prefix", $"{effectivePrefix:N0}");
+        table.AddRow("Corpus tokens", $"{tokens.Count:N0}");
+        table.AddRow("Elapsed", $"{sw.Elapsed.TotalSeconds:F2} s");
+        AnsiConsole.Write(table);
+
+        await Task.CompletedTask;
+        return 0;
+    }
+
+    private static bool TryParseMode(string value, out PerplexityMode mode)
+    {
+        switch (value.Trim().ToLowerInvariant())
+        {
+            case "sliding-window":
+            case "sliding":
+                mode = PerplexityMode.SlidingWindow;
+                return true;
+            case "teacher-forced":
+            case "teacher":
+                mode = PerplexityMode.TeacherForced;
+                return true;
+            default:
+                mode = default;
+                return false;
+        }
+    }
+}

src/DotLLM.Cli/Program.cs

@@ -48,6 +48,10 @@
         .WithDescription("Interactive multi-turn chat with a GGUF model.")
         .WithExample("chat", "QuantFactory/SmolLM-135M-GGUF", "--system", "You are a helpful assistant.");
 
+    config.AddCommand<PerplexityCommand>("perplexity")
+        .WithDescription("Compute perplexity over a text corpus.")
+        .WithExample("perplexity", "QuantFactory/SmolLM-135M-GGUF", "--corpus", "wiki.test.raw", "--context", "512", "--stride", "256");
+
     config.AddCommand<ServeCommand>("serve")
         .WithDescription("Launch API server with built-in web chat UI.")
         .WithExample("serve", "Qwen/Qwen3-0.6B-GGUF", "--port", "8080");