DStream Providers

Sample providers for the DStream data streaming platform, demonstrating external provider development using published SDK packages.

🚀 Features

• 🔢 Counter Input Provider - Generates sample data for testing and development
• 📺 Console Output Provider - Displays streaming data with formatted output
• 🐳 Cross-platform OCI containers - Ready for production deployment
• 📦 Uses published NuGet packages - No local SDK dependencies required
• 🏗️ Complete build system - Makefiles, Dockerfiles, and GitHub Actions

🎯 Purpose

This repository demonstrates external provider development for the DStream ecosystem:

• ✅ Consumes published SDK packages from nuget.org
• ✅ Independent development - No dependency on SDK source code
• ✅ Production-ready examples - Complete with build, test, and deployment automation
• ✅ OCI container distribution - Cross-platform container images for deployment

📦 Quick Start

Prerequisites

• .NET 9.0 SDK
• Docker (for container builds)
• DStream CLI (from katasec/dstream)

Install SDK Package

dotnet add package Katasec.DStream.SDK.Core --version 0.1.1

Clone and Build

git clone https://github.com/katasec/dstream-providers.git
cd dstream-providers

# Build counter input provider
cd counter-input-provider
make build
make test

# Build console output provider  
cd ../console-output-provider
make build
make test

Test Providers Independently

# Test input provider (generates 3 counter values)
echo '{"interval": 1000, "max_count": 3}' | ./counter-input-provider/bin/Release/net9.0/osx-x64/counter-input-provider

# Test output provider (processes sample data)
echo '{"outputFormat": "simple"}' | ./console-output-provider/bin/Release/net9.0/osx-x64/console-output-provider

Use with DStream CLI

Create a dstream.hcl file:

task "counter-to-console" {
  type = "providers"
  
  input {
    provider_path = "./counter-input-provider/bin/Release/net9.0/osx-x64/counter-input-provider"
    config = {
      interval = 1000
      max_count = 10
    }
  }
  
  output {
    provider_path = "./console-output-provider/bin/Release/net9.0/osx-x64/console-output-provider"
    config = {
      outputFormat = "structured"
    }
  }
}

Run with DStream:

dstream run counter-to-console

🐳 OCI Artifact Usage

Providers are distributed as OCI artifacts (not Docker containers) via GitHub Container Registry:

# List available versions
./version.ps1 list

# Pull and inspect OCI artifacts using ORAS
oras pull ghcr.io/katasec/dstream-counter-input-provider:v0.1.0
oras pull ghcr.io/katasec/dstream-console-output-provider:v0.1.0

Use with DStream:

task "oci-demo" {
  type = "providers"
  
  input {
    provider_ref = "ghcr.io/katasec/dstream-counter-input-provider:v0.1.0"
    config = { interval = 1000, max_count = 5 }
  }
  
  output {
    provider_ref = "ghcr.io/katasec/dstream-console-output-provider:v0.1.0" 
    config = { outputFormat = "simple" }
  }
}

🏷️ Semantic Versioning

Providers follow semantic versioning (like Terraform and Pulumi):

# List current versions
./version.ps1

# Create new versions
./version.ps1 patch   # v0.1.0 → v0.1.1 (bug fixes)
./version.ps1 minor   # v0.1.0 → v0.2.0 (new features) 
./version.ps1 major   # v0.1.0 → v1.0.0 (breaking changes)

# Create custom version
./version.ps1 create v2.1.0

# Release both providers with single command
./release.ps1 patch                    # Auto-increment patch version
./release.ps1 -Tag v1.2.3             # Explicit version

📁 Project Structure

dstream-providers/
├── counter-input-provider/          # Input provider example
│   ├── Program.cs                   # Provider implementation
│   ├── CounterConfig.cs             # Configuration model
│   ├── CounterReader.cs             # Business logic
│   ├── Makefile                     # Build automation
│   ├── Dockerfile                   # Container build
│   └── *.csproj                     # Uses NuGet packages
├── console-output-provider/         # Output provider example
│   ├── Program.cs                   # Provider implementation  
│   ├── ConsoleConfig.cs             # Configuration model
│   ├── ConsoleWriter.cs             # Business logic
│   ├── Makefile                     # Build automation
│   ├── Dockerfile                   # Container build
│   └── *.csproj                     # Uses NuGet packages
├── .github/workflows/               # CI/CD automation
│   ├── build-and-test.yml          # Build and test workflow
│   └── publish-containers.yml      # Container publishing
├── docker-compose.yml              # Local development
└── README.md                       # This file

🔧 Development

Building All Providers

# Build all providers
make build-all

# Test all providers  
make test-all

# Build containers
make docker-build

# Run integration tests
make integration-test

Creating New Providers

1. Create new provider directory
2. Add package reference:

   <PackageReference Include="Katasec.DStream.SDK.Core" Version="0.1.1" />

3. Implement provider interface:

   public class MyProvider : ProviderBase<MyConfig>, IInputProvider
   {
       public async IAsyncEnumerable<Envelope> ReadAsync(IPluginContext ctx, CancellationToken ct)
       {
           // Your implementation
       }
   }

4. Bootstrap in Program.cs:

   await StdioProviderHost.RunInputProviderAsync<MyProvider, MyConfig>();

📚 Documentation

• DStream Main Repository: katasec/dstream
• DStream .NET SDK: katasec/dstream-dotnet-sdk
• NuGet Packages: Katasec.DStream.SDK.Core
• Provider Development Guide: Provider Development

🤝 Contributing

This repository demonstrates the external provider development pattern. Contributions welcome:

1. New provider examples - Show different use cases
2. Improved build automation - Better CI/CD, cross-platform builds
3. Documentation improvements - Help others understand the pattern
4. Bug fixes - Issues with examples or build processes

📄 License

MIT License - see LICENSE file for details.

---

Note: This repository demonstrates external provider development using published SDK packages. For SDK development itself, see katasec/dstream-dotnet-sdk.

DStream: Unix pipelines for data streaming 🚀