Merge pull request #12 from sharpninja/copilot/create-specflow-tests

Add tensor-based `traditional-local` model and benchmarkable local model comparison

Author: sharpninja

.gitignore

@@ -47,6 +47,7 @@ bld/
 
 # Visual Studio 2017 auto generated files
 Generated\ Files/
+*.feature.cs
 
 # MSTest test Results
 [Tt]est[Rr]esult*/

docs/README.md

@@ -7,6 +7,7 @@ BitNet b1.58 Sharp is a .NET 10 C# reference implementation of the paper-aligned
 - A paper-aligned BitNet core model in `/src/BitNetSharp.Core`
 - A decoder-only transformer implementation with `BitLinear`, `RmsNorm`, RoPE, causal attention, SwiGLU, and `BitNetTransformer`
 - Microsoft Agent Framework-oriented hosting in `/src/BitNetSharp.App`
+- BenchmarkDotNet-based local model comparison in `/src/BitNetSharp.App`
 - Default American English interaction behavior
 - Seeded transformer inspection and ternary weight summaries
 - GitBook-formatted project documentation in `/docs`
@@ -23,6 +24,7 @@ dotnet test /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/BitNet-b1.58
 ## Documentation map
 
 - [Architecture](architecture.md)
+- [Benchmarking and model comparison](benchmarking.md)
 - [Implementation plan](implementation-plan.md)
 - [Releases and packaging](releases-and-packaging.md)
 - [Usage](usage.md)

docs/SUMMARY.md

@@ -3,6 +3,7 @@
 - [BitNet b1.58 Sharp](README.md)
   - [Architecture](architecture.md)
   - [Implementation plan](implementation-plan.md)
+  - [Benchmarking and model comparison](benchmarking.md)
   - [Releases and packaging](releases-and-packaging.md)
   - [Usage](usage.md)
   - [Training and visualization](training-and-visualization.md)

docs/architecture.md

@@ -20,6 +20,14 @@ BitNet b1.58 Sharp targets the paper-aligned BitNet b1.58 decoder-only transform
 
 The app registers a local `IChatClient` implementation so the paper-aligned BitNet model can be hosted through Agent Framework conventions while remaining runnable as a standalone console application.
 
+The hosting layer now resolves multiple local model types behind the same agent wrapper:
+
+- the seeded paper-aligned BitNet model
+- a traditional local tensor-based comparison model trained on the default corpus with `System.Numerics.Tensors`
+- local command models described by JSON configuration files
+
+This lets BenchmarkDotNet measure host construction, querying, streaming, and local training through one shared path.
+
 ## Language and interaction model
 
 The built-in vocabulary and command output default to American English. That keeps prompts, diagnostics, and help text aligned with the requirement for a primary U.S. English interface.

docs/benchmarking.md

@@ -0,0 +1,63 @@
+# Benchmarking and model comparison
+
+## Overview
+
+`BitNetSharp.App` can now host more than one local model shape through the same Microsoft Agent Framework wrapper:
+
+- `bitnet-b1.58-sharp` for the paper-aligned seeded transformer
+- `traditional-local` for a local tensor-based comparison model trained on the default training corpus
+- an absolute path to a local command model JSON file for other locally available models
+
+The benchmark command uses BenchmarkDotNet to measure the same hosted-model operations that the SpecFlow scenarios exercise:
+
+- training a selected trainable model on the default dataset
+- generating a response for a prompt
+- streaming a response for a prompt
+- building the agent host
+
+## Run the built-in comparison benchmark
+
+```bash
+dotnet run --configuration Release --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- benchmark --model=bitnet-b1.58-sharp --compare-model=traditional-local --prompt="how are you hosted"
+```
+
+This runs the BenchmarkDotNet suite over both local models so their hosted response and host-construction costs can be compared directly.
+
+## Train the traditional local model
+
+```bash
+dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- train --model=traditional-local
+```
+
+The traditional local model trains over `BitNetTrainingCorpus.CreateDefaultExamples()` for 24 epochs using `System.Numerics.Tensors` softmax and dot-product primitives so it can be benchmarked and queried against the same dataset every time.
+
+## Compare another local model
+
+Pass the absolute path to a JSON file that describes how to execute a locally available model runner:
+
+```bash
+dotnet run --configuration Release --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- benchmark --model=/absolute/path/to/local-model.json --compare-model=traditional-local
+```
+
+Example configuration:
+
+```json
+{
+  "modelId": "my-local-model",
+  "displayName": "My local model",
+  "executablePath": "/absolute/path/to/model-runner",
+  "arguments": [
+    "--model",
+    "/absolute/path/to/model.bin"
+  ],
+  "promptTransport": "StandardInput",
+  "primaryLanguage": "en-US"
+}
+```
+
+### Prompt transport options
+
+- `StandardInput`: the prompt is written to the process standard input
+- `FinalArgument`: the prompt is appended as the final command-line argument
+
+This keeps model comparison local-only and avoids endpoint or API-key based integrations.

docs/usage.md

@@ -8,10 +8,11 @@ dotnet build /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/BitNet-b1.5
 
 ## Chat
 
-The chat command inspects the paper-aligned transformer and reports its top next-token predictions for the supplied prompt.
+The chat command can host the seeded paper-aligned transformer or another local comparison model and report that model's response for the supplied prompt.
 
 ```bash
 dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- chat "how are you hosted"
+dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- chat "how are you hosted" --model=traditional-local
 ```
 
 Optional verbosity:
@@ -25,14 +26,31 @@ dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src
 
 ```bash
 dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- host
+dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- host --model=traditional-local
 ```
 
-This command confirms that the application is wired for Microsoft Agent Framework hosting and reports the current language and verbosity configuration.
+This command confirms that the application is wired for Microsoft Agent Framework hosting and reports the selected model, language, and verbosity configuration.
 
 ## Transformer inspection
 
 ```bash
 dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- visualize
 ```
 
-This command prints the current paper-model configuration and an aggregated ternary weight histogram across the transformer's `BitLinear` projections.
+This command prints the current model summary. When the selected model is the paper-aligned BitNet transformer, it also prints the ternary weight histogram across the transformer's `BitLinear` projections.
+
+## Benchmark
+
+```bash
+dotnet run --configuration Release --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- benchmark --model=bitnet-b1.58-sharp --compare-model=traditional-local --prompt="how are you hosted"
+```
+
+This command runs BenchmarkDotNet over the same hosted-model operations covered by the SpecFlow scenarios so you can compare local models under one agent wrapper.
+
+## Train the traditional comparison model
+
+```bash
+dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- train --model=traditional-local
+```
+
+The paper-aligned transformer still reports that training is not implemented in this branch. The `traditional-local` model trains a small tensor-based local language model on the default corpus for 24 epochs so its training and query performance can be benchmarked on the same dataset.

src/BitNetSharp.App/BitNetAgentHost.cs

@@ -8,33 +8,44 @@ namespace BitNetSharp.App;
 
 public sealed record BitNetHostSummary(
     string AgentName,
+    string ModelId,
+    string DisplayName,
     string PrimaryLanguage,
     string HostingFramework,
     VerbosityLevel Verbosity);
 
 public static class BitNetAgentHost
 {
-    public static IHost Build(BitNetPaperModel model)
+    public static IHost Build(BitNetPaperModel model) => Build(new BitNetHostedAgentModel(model));
+
+    public static IHost Build(IHostedAgentModel model)
     {
         ArgumentNullException.ThrowIfNull(model);
 
         var builder = Host.CreateApplicationBuilder();
-        var chatClient = new BitNetChatClient(model);
+        var chatClient = new HostedModelChatClient(model);
 
         builder.Services.AddSingleton(model);
         builder.Services.AddSingleton<IChatClient>(chatClient);
         builder.Services.AddSingleton(new BitNetHostSummary(
-            "bitnet-b1.58-sharp",
-            model.Options.PrimaryLanguage,
+            model.AgentName,
+            model.ModelId,
+            model.DisplayName,
+            model.PrimaryLanguage,
             "Microsoft Agent Framework",
-            model.Options.Verbosity));
+            model.Verbosity));
 
         builder.AddAIAgent(
-                "bitnet-b1.58-sharp",
-                "Respond in clear American English using the paper-aligned BitNet b1.58 transformer diagnostics.",
+                model.AgentName,
+                model.SystemPrompt,
                 chatClient)
             .WithInMemorySessionStore();
 
+        if (model is BitNetHostedAgentModel bitNetModel)
+        {
+            builder.Services.AddSingleton(bitNetModel.Model);
+        }
+
         return builder.Build();
     }
 }

src/BitNetSharp.App/BitNetChatClient.cs

@@ -1,25 +1,24 @@
-using BitNetSharp.Core;
 using Microsoft.Extensions.AI;
 
 namespace BitNetSharp.App;
 
-public sealed class BitNetChatClient(BitNetPaperModel model) : IChatClient
+public sealed class HostedModelChatClient(IHostedAgentModel model) : IChatClient
 {
-    public Task<ChatResponse> GetResponseAsync(
+    public async Task<ChatResponse> GetResponseAsync(
         IEnumerable<ChatMessage> messages,
         ChatOptions? options = null,
         CancellationToken cancellationToken = default)
     {
         var prompt = messages.LastOrDefault(message => message.Role == ChatRole.User)?.Text ?? string.Empty;
-        var result = model.GenerateResponse(prompt, options?.MaxOutputTokens);
-        var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, result.ResponseText))
+        var result = await model.GetResponseAsync(prompt, options?.MaxOutputTokens, cancellationToken);
+        var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, result.Text))
         {
             ModelId = model.ModelId,
             FinishReason = ChatFinishReason.Stop,
             CreatedAt = DateTimeOffset.UtcNow
         };
 
-        return Task.FromResult(response);
+        return response;
     }
 
     public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(

src/BitNetSharp.App/BitNetHostedAgentModel.cs

@@ -0,0 +1,49 @@
+using BitNetSharp.Core;
+using BitNetSharp.Core.Quantization;
+
+namespace BitNetSharp.App;
+
+public sealed class BitNetHostedAgentModel(BitNetPaperModel model) : IHostedAgentModel, IInspectableHostedAgentModel
+{
+    public BitNetPaperModel Model { get; } = model ?? throw new ArgumentNullException(nameof(model));
+
+    public string AgentName => Model.ModelId;
+
+    public string ModelId => Model.ModelId;
+
+    public string DisplayName => "Paper-aligned BitNet b1.58 transformer";
+
+    public string PrimaryLanguage => Model.Options.PrimaryLanguage;
+
+    public VerbosityLevel Verbosity => Model.Options.Verbosity;
+
+    public string SystemPrompt => "Respond in clear American English using the paper-aligned BitNet b1.58 transformer diagnostics.";
+
+    public IReadOnlyList<string> DescribeModel() =>
+    [
+        DisplayName,
+        $"Model ID: {ModelId}",
+        $"Vocabulary size: {Model.Config.VocabSize}",
+        $"Layers: {Model.Config.LayerCount}",
+        $"Dimension: {Model.Config.Dimension}",
+        $"Hidden dimension: {Model.Config.HiddenDimension}",
+        $"Heads: {Model.Config.HeadCount}",
+        $"Max sequence length: {Model.Config.MaxSequenceLength}"
+    ];
+
+    public Task<HostedAgentModelResponse> GetResponseAsync(
+        string prompt,
+        int? maxOutputTokens = null,
+        CancellationToken cancellationToken = default)
+    {
+        cancellationToken.ThrowIfCancellationRequested();
+        var result = Model.GenerateResponse(prompt, maxOutputTokens);
+        return Task.FromResult(new HostedAgentModelResponse(result.ResponseText, result.Diagnostics));
+    }
+
+    public TernaryWeightStats GetTernaryWeightStats() => Model.GetTernaryWeightStats();
+
+    public void Dispose()
+    {
+    }
+}

src/BitNetSharp.App/BitNetSharp.App.csproj

@@ -5,6 +5,7 @@
   </ItemGroup>
 
   <ItemGroup>
+    <PackageReference Include="BenchmarkDotNet" Version="0.15.8" />
     <PackageReference Include="Microsoft.Agents.AI" Version="1.0.0-rc4" />
     <PackageReference Include="Microsoft.Agents.AI.Hosting" Version="1.0.0-preview.260311.1" />
   </ItemGroup>