Merge pull request #16 from sharpninja/copilot/add-datagen-synthetic-data-generator

Add DataGen synthetic dataset generation to the BitNet b1.58 CLI

Author: sharpninja

docs/README.md

@@ -8,24 +8,27 @@ BitNet b1.58 Sharp is a .NET 10 C# reference implementation of the paper-aligned
 - A decoder-only transformer implementation with `BitLinear`, `RmsNorm`, RoPE, causal attention, SwiGLU, and `BitNetTransformer`
 - Microsoft Agent Framework-oriented hosting in `/src/BitNetSharp.App`
 - BenchmarkDotNet-based local model comparison in `/src/BitNetSharp.App`
+- DataGen synthetic dataset generation from JSON seed examples
 - Default American English interaction behavior
 - Seeded transformer inspection and ternary weight summaries
 - GitBook-formatted project documentation in `/docs`
 
 ## Quick start
 
 ```bash
-dotnet build /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp.slnx
-dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- chat "hello"
-dotnet run --project /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/src/BitNetSharp.App/BitNetSharp.App.csproj -- visualize
-dotnet test /home/runner/work/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp/BitNet-b1.58-Sharp.slnx
+dotnet build BitNet-b1.58-Sharp.slnx
+dotnet run --project src/BitNetSharp.App/BitNetSharp.App.csproj -- chat "hello"
+dotnet run --project src/BitNetSharp.App/BitNetSharp.App.csproj -- datagen --domain "customer-support" --count 10 --seeds examples/seed-examples.json --output data/customer-support.jsonl
+dotnet run --project src/BitNetSharp.App/BitNetSharp.App.csproj -- visualize
+dotnet test BitNet-b1.58-Sharp.slnx
 ```
 
 ## Documentation map
 
 - [Architecture](architecture.md)
 - [Benchmarking and model comparison](benchmarking.md)
-- [Implementation plan](implementation-plan.md)
+- [DataGen guide](datagen-guide.md)
+- [Implementation plan](implementation-plan-v3.md)
 - [Releases and packaging](releases-and-packaging.md)
 - [Usage](usage.md)
 - [Training and visualization](training-and-visualization.md)

docs/SUMMARY.md

@@ -2,6 +2,7 @@
 
 - [BitNet b1.58 Sharp](README.md)
   - [Architecture](architecture.md)
+  - [DataGen guide](datagen-guide.md)
   - [Implementation plan v3 (active)](implementation-plan-v3.md)
   - [Implementation plan v2 (archived)](implementation-plan-v2.md)
   - [Implementation plan v1 (archived)](implementation-plan-v1.md)

docs/architecture.md

@@ -7,6 +7,7 @@ BitNet b1.58 Sharp targets the paper-aligned BitNet b1.58 decoder-only transform
 `/src/BitNetSharp.Core` contains the paper-model runtime and transformer building blocks:
 
 - `BitNetPaperModel` wraps tokenizer state, the seeded transformer, and next-token inspection output
+- `DataGenGenerator` expands JSON seed examples into synthetic JSONL records while recording BitNet model provenance
 - `VerbosityLevel` exposes exactly three interaction levels: `Quiet`, `Normal`, and `Verbose`
 - `BitLinear` implements absmean-scaled ternary projections with signed int8 activation quantization
 - `RmsNorm`, `RotaryPositionEmbedding`, `MultiHeadAttention`, `SwiGLUFeedForward`, `BitNetLayer`, and `BitNetTransformer` implement the decoder-only paper architecture
@@ -28,6 +29,8 @@ The hosting layer now resolves multiple local model types behind the same agent
 
 This lets BenchmarkDotNet measure host construction, querying, streaming, and local training through one shared path.
 
+The same app surface also exposes a `datagen` command that keeps synthetic data generation local to the repository checkout.
+
 ## Language and interaction model
 
 The built-in vocabulary and command output default to American English. That keeps prompts, diagnostics, and help text aligned with the requirement for a primary U.S. English interface.

docs/datagen-guide.md

@@ -0,0 +1,68 @@
+# DataGen guide
+
+DataGen is the repository's offline synthetic dataset bootstrapper for the paper-aligned BitNet b1.58 runtime. It takes a small set of seed examples, applies deterministic variation patterns, and uses the built-in BitNet transformer to condition each batch with lightweight next-token cues.
+
+## Generate a dataset
+
+```bash
+dotnet run --project src/BitNetSharp.App/BitNetSharp.App.csproj -- datagen \
+  --domain "medical-diagnosis" \
+  --count 50000 \
+  --seeds examples/seed-examples.json \
+  --output data/synthetic-medical.jsonl \
+  --lora medical-lora.bin
+```
+
+The command writes one JSON object per line so the output can flow directly into local fine-tuning or evaluation jobs.
+
+## Seed format
+
+Seed files are standard JSON arrays. Each object must include one instruction-like field and one response-like field. DataGen accepts the following aliases:
+
+- instruction: `instruction`, `prompt`, or `input`
+- response: `response`, `output`, or `answer`
+
+Example:
+
+```json
+[
+  {
+    "prompt": "Summarize the patient's main complaint and likely differential diagnosis.",
+    "response": "Restate the complaint, list the most likely causes, and flag any immediate safety concerns."
+  },
+  {
+    "instruction": "Explain what evidence should be gathered before choosing a treatment plan.",
+    "answer": "Collect history, exam findings, recent medications, and any contraindications before recommending next steps."
+  }
+]
+```
+
+## Output schema
+
+Each JSONL line includes:
+
+- `domain`
+- `instruction`
+- `response`
+- `seedInstruction`
+- `seedResponse`
+- `variation`
+- `generatorModel`
+- `loraAdapter`
+- `tags`
+
+The optional `--lora` argument is recorded in output metadata so runs can stay attributable even when adapter-conditioned execution is handled outside the CLI.
+
+## Quality controls
+
+- Start from diverse seeds that already match the tone and structure you need.
+- Generate a smaller preview set first, then inspect the JSONL output before scaling up.
+- Filter or deduplicate generated samples before fine-tuning if your target pipeline requires stricter curation.
+
+## Integration notes
+
+DataGen is intentionally local-first:
+
+- generation runs entirely offline
+- output stays in your working directory
+- the same built-in BitNet model ID is recorded with every example for traceability

docs/usage.md

@@ -55,6 +55,15 @@ dotnet run --configuration Release --project src/BitNetSharp.App/BitNetSharp.App
 
 This command runs the BenchmarkDotNet suite, evaluates both built-in models against the shared default training corpus/query script, and writes HTML, Markdown, and JSON comparison reports to the selected output directory.
 
+## DataGen
+
+```bash
+dotnet run --project src/BitNetSharp.App/BitNetSharp.App.csproj -- datagen --domain "medical-diagnosis" --count 25 --seeds examples/seed-examples.json --output data/synthetic-medical.jsonl
+dotnet run --project src/BitNetSharp.App/BitNetSharp.App.csproj -- datagen --domain "medical-diagnosis" --count 25 --seeds examples/seed-examples.json --output data/synthetic-medical.jsonl --lora medical-lora.bin
+```
+
+This command reads a JSON array of seed examples, expands them into synthetic instruction-response pairs, and writes JSONL output for downstream local fine-tuning or evaluation. See the [DataGen guide](datagen-guide.md) for accepted seed aliases and the output schema.
+
 ## Train the traditional comparison model
 
 ```bash

examples/seed-examples.json

@@ -0,0 +1,14 @@
+[
+  {
+    "prompt": "Summarize the patient's symptoms and identify the most urgent risk to rule out first.",
+    "response": "Restate the symptoms, note the highest-priority red flag, and recommend the next safe diagnostic step."
+  },
+  {
+    "instruction": "Explain how to gather missing context before recommending a treatment plan.",
+    "answer": "Review history, medications, allergies, exam findings, and contraindications before proposing options."
+  },
+  {
+    "input": "Create a concise follow-up instruction for documenting how the case should be monitored.",
+    "output": "Specify what should be monitored, how often it should be reviewed, and which changes require escalation."
+  }
+]

src/BitNetSharp.App/DataGenCommand.cs

@@ -0,0 +1,124 @@
+using BitNetSharp.Core;
+using System.Text.Json;
+
+namespace BitNetSharp.App;
+
+public sealed record DataGenCommandOptions(
+    string Domain,
+    int Count,
+    string SeedsPath,
+    string OutputPath,
+    string? LoraPath)
+{
+    public static DataGenCommandOptions Parse(string[] args)
+    {
+        ArgumentNullException.ThrowIfNull(args);
+
+        var domain = ReadRequiredOption(args, "--domain");
+        var seedsPath = ReadRequiredOption(args, "--seeds");
+        var outputPath = ReadRequiredOption(args, "--output");
+        var countValue = ReadRequiredOption(args, "--count");
+
+        if (!int.TryParse(countValue, out var count) || count <= 0)
+        {
+            throw new ArgumentException("The datagen count must be a positive integer.", nameof(args));
+        }
+
+        return new DataGenCommandOptions(
+            domain,
+            count,
+            Path.GetFullPath(seedsPath),
+            Path.GetFullPath(outputPath),
+            ReadOptionalOption(args, "--lora"));
+    }
+
+    private static string ReadRequiredOption(string[] args, string optionName)
+    {
+        var value = ReadOptionalOption(args, optionName);
+        if (value is null)
+        {
+            throw new ArgumentException($"Missing required option '{optionName}'.", nameof(args));
+        }
+
+        if (string.IsNullOrWhiteSpace(value))
+        {
+            throw new ArgumentException($"Option '{optionName}' requires a non-empty value.", nameof(args));
+        }
+
+        return value;
+    }
+
+    private static string? ReadOptionalOption(string[] args, string optionName)
+    {
+        var equalsPrefix = $"{optionName}=";
+        var missingValueDetected = false;
+        for (var index = 0; index < args.Length; index++)
+        {
+            var argument = args[index];
+            if (argument.StartsWith(equalsPrefix, StringComparison.OrdinalIgnoreCase))
+            {
+                return argument[equalsPrefix.Length..];
+            }
+
+            if (string.Equals(argument, optionName, StringComparison.OrdinalIgnoreCase))
+            {
+                var nextIndex = index + 1;
+                if (nextIndex >= args.Length)
+                {
+                    missingValueDetected = true;
+                    continue;
+                }
+
+                var nextArgument = args[nextIndex];
+                if (nextArgument.StartsWith("--", StringComparison.Ordinal))
+                {
+                    missingValueDetected = true;
+                    continue;
+                }
+
+                return nextArgument;
+            }
+        }
+
+        if (missingValueDetected)
+        {
+            throw new ArgumentException($"Option '{optionName}' requires a value.", nameof(args));
+        }
+
+        return null;
+    }
+}
+
+public static class DataGenCommand
+{
+    private static readonly JsonSerializerOptions OutputJsonOptions = new()
+    {
+        PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
+        WriteIndented = false
+    };
+
+    public static async Task<string> RunAsync(string[] args, VerbosityLevel verbosity, CancellationToken cancellationToken = default)
+    {
+        var options = DataGenCommandOptions.Parse(args);
+        var seeds = DataGenGenerator.LoadSeeds(options.SeedsPath);
+        var generator = new DataGenGenerator(BitNetBootstrap.CreatePaperModel(verbosity));
+        var outputDirectory = Path.GetDirectoryName(options.OutputPath);
+
+        if (!string.IsNullOrWhiteSpace(outputDirectory))
+        {
+            Directory.CreateDirectory(outputDirectory);
+        }
+
+        await using var stream = File.Create(options.OutputPath);
+        await using var writer = new StreamWriter(stream);
+
+        foreach (var example in generator.Generate(options.Domain, options.Count, seeds, options.LoraPath))
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            var line = JsonSerializer.Serialize(example, OutputJsonOptions);
+            await writer.WriteLineAsync(line.AsMemory(), cancellationToken);
+        }
+
+        return options.OutputPath;
+    }
+}

src/BitNetSharp.App/Program.cs

@@ -28,6 +28,13 @@
     return;
 }
 
+if (command == "datagen")
+{
+    var outputPath = await DataGenCommand.RunAsync(args, verbosity);
+    Console.WriteLine($"Saved synthetic dataset to {outputPath}");
+    return;
+}
+
 using var model = HostedAgentModelFactory.Create(modelSpecifier, verbosity);
 using var host = BitNetAgentHost.Build(model);
 var hostSummary = host.Services.GetRequiredService<BitNetHostSummary>();