Surgewave Bootcamp

A hands-on, self-paced tutorial to master Surgewave. 36 units, 8 learning paths, 35 guided tours.

Goal

Learn to build, operate, and scale real-time data systems with Surgewave — from your first message to production-grade streaming pipelines.

What is Surgewave Bootcamp?

Surgewave Bootcamp is an interactive learning experience where you'll build real applications while learning Surgewave concepts. Each lesson includes:

• Concepts: Clear explanations of what you're learning
• Exercises: Hands-on coding tasks
• Starter Code: Incomplete code to fill in
• Completed Solutions: Reference implementations
• Guided Tours: 35 Shepherd.js interactive walkthroughs integrated into the DocFX learn site

Prerequisites

• .NET 10 SDK or later
• A code editor (VS Code, Visual Studio, Rider)
• Basic C# knowledge
• ~2-4 hours per unit
• For AI units (18, 19, 20, 17): Surgewave.AI NuGet packages

Getting Started

# Clone the repository
git clone https://github.com/Kuestenlogik/Surgewave.Bootcamp.git
cd Surgewave.Bootcamp

# Build the solution
dotnet build Kuestenlogik.Surgewave.Bootcamp.slnx

# Start with the introduction
cd units/unit-0/lesson-1

DocFX Learn Site

The Bootcamp includes a full DocFX-powered learn site with Shepherd.js guided tours:

# Build and serve the learn site on port 8081
.\build-learn.ps1 -Serve -Port 8081

Learning Paths

Choose from 8 structured paths based on your role and goals:

Path	Target Audience	Duration	Units
Developer Fundamentals	All developers (start here)	~30h	8 units
Stream Processing	Data engineers, backend devs	~12h	2 units
Data Integration	Integration/platform engineers	~24h	6 units
Pipelines & Workflows	Low-code builders, process automation	~10h	2 units
AI & Intelligent Systems	AI/ML engineers, LLM developers	~20h	4 units
Operations & Production	DevOps, SREs, platform engineers	~35h	9 units
Advanced Patterns	Senior developers, architects	~28h	6 units
Performance & Resilience	Performance engineers, SREs	~15h	3 units

Or complete all 36 units in order for comprehensive coverage.

View Detailed Learning Paths →

---

Curriculum

Unit 0: Introduction

Time: ~1 hour

Understand what Surgewave is and set up your development environment.

Lesson	Topic	What You'll Learn
0.1	What is Surgewave?	Message streaming, Surgewave vs alternatives
0.2	Project Setup	Install packages, create projects
0.3	Hello Surgewave	Connect to Surgewave and verify setup
0.4	Contributing to Surgewave	GitHub, bug reports, pull requests
0.5	Getting Support	Community, documentation, support plans

Unit 1: Surgewave Fundamentals [DEV]

Time: ~3 hours

Learn the core concepts of message streaming with Surgewave.

Lesson	Topic	What You'll Build
1.1	Producing Messages	Send your first messages
1.2	Consuming Messages	Receive and process messages
1.3	Topics & Partitions	Understand partitioning
1.4	Consumer Groups	Scale with multiple consumers
1.5	Batching & Message Sizing	Optimize throughput
1.6	Message Best Practices	Events vs commands, immutability

Unit 2: Surgewave Control UI Mastery [DEV]

Time: ~3 hours

Master the Surgewave Control UI dashboard for cluster management.

Lesson	Topic	What You'll Learn
2.1	Dashboard & Navigation	Home dashboard widgets, layout customization, theme toggle
2.2	Topic & Message Management	Topic CRUD, message browser, producer, bulk operations
2.3	Consumer Groups & Lag Monitoring	Group state, member assignment, lag charts, offset reset
2.4	Pipeline Management	Pipeline editor, chat integration, templates, versioning
2.5	Monitoring & Alerting	Metrics dashboards, alert rules, performance advisor
2.6	Security & Governance	ACLs, RBAC, quotas, data catalog, schema registry

Unit 3: Testing Surgewave [DEV]

Time: ~3 hours

Master testing patterns for Surgewave applications.

Lesson	Topic	What You'll Build
3.1	Unit Testing Basics	xUnit tests with embedded broker
3.2	Mocking & Isolation	Mock producers and consumers
3.3	Integration Testing	End-to-end tests with fixtures
3.4	Testcontainers	Docker-based testing patterns
3.5	Benchmarking	Measure performance with BenchmarkDotNet

Unit 4: Architecture & Design [ARCH]

Time: ~3 hours

Design event-driven architectures with Surgewave.

Lesson	Topic	What You'll Learn
4.1	Topic Design	Naming conventions, granularity
4.2	Partitioning Strategies	Key selection, hot partitions
4.3	Event Schema Design	Versioning, compatibility
4.4	Event-Driven Patterns	CQRS, Event Sourcing, Saga
4.5	Capacity Planning	Sizing, retention, throughput

Unit 5: Stream Processing [DEV]

Time: ~4 hours

Master real-time stream processing with Surgewave Streams.

Lesson	Topic	What You'll Learn
5.1	Streams Basics	StreamsBuilder, Stream, topology
5.2	Transformations	map, filter, flatMap, branch, merge
5.3	Aggregations & Windowing	groupBy, count, tumbling/hopping/session windows
5.4	Joins	Stream-stream, stream-table, table-table joins
5.5	State Stores	KeyValueStore, WindowStore, interactive queries

Unit 6: Advanced Consumer Patterns [DEV]

Time: ~3 hours

Master advanced consumer techniques.

Lesson	Topic	What You'll Learn
6.1	Manual Offset Management	Commit strategies, guarantees
6.2	Seek Operations	Offset/timestamp seeking
6.3	Pause & Resume	Flow control, backpressure
6.4	Custom Checkpointing	External stores, exactly-once
6.5	Partition Assignment	Custom assignors, rebalancing

Unit 7: Advanced Stream Processing [DEV]

Time: ~3 hours

Advanced Streams features for production workloads.

Lesson	Topic	What You'll Learn
7.1	Processor Lifecycle Hooks	OnInit, OnClose, graceful shutdown orchestration
7.2	Per-Processor Metrics	OTEL counters, histograms, state store metrics
7.3	Backpressure & Flow Control	Bounded buffers, watermarks, drop strategies
7.4	Foreign Key Table-Table Joins	FK joins, left joins, subscription tracking
7.5	Retry & Backoff Integration	Retry policies, backoff strategies, exhaustion handling

Unit 8: Reactive Integration [DEV]

Time: ~3 hours

Integrate Surgewave with reactive programming patterns.

Lesson	Topic	What You'll Learn
8.1	Reactive Basics	IObservable, IObserver, subscriptions
8.2	Surgewave as Observable	Wrapping consumers as streams
8.3	Operators	Select, Where, Buffer, Window
8.4	Backpressure	Flow control, buffering strategies
8.5	Reactive Pipelines	End-to-end reactive data flows

Unit 9: Data Integration [DEV]

Time: ~3 hours

Connect external data sources and sinks with Surgewave.

Lesson	Topic	What You'll Build
9.1	Connect Concepts	Source/sink connector architecture
9.2	File Connectors	CSV/JSON file ingestion and export
9.3	Database Connectors	SQL database integration
9.4	Change Data Capture	Track database changes in real-time
9.5	Custom Connectors	Build your own connectors

Unit 10: Schema Registry [DEV]

Time: ~3 hours

Manage message schemas and ensure compatibility.

Lesson	Topic	What You'll Learn
10.1	Schema Basics	Why schemas, schema-first design
10.2	JSON Schema	Validation, type safety
10.3	Schema Versioning	Evolution strategies, migrations
10.4	Compatibility Modes	Forward, backward, full compatibility
10.5	Schema Registry Integration	Centralized schema management

Unit 11: Kafka Migration

Time: ~2 hours

Migrate from Apache Kafka to Surgewave.

Lesson	Topic	What You'll Build
11.1	Kafka Compatibility	Run Kafka apps on Surgewave
11.2	API Wrapper	Zero-code migration
11.3	Hybrid Deployment	Run both systems
11.4	Production Cutover	Complete migration

Unit 12: Queue Semantics & AMQP

Time: ~3 hours

Use Surgewave as a traditional message queue with AMQP support.

Lesson	Topic	What You'll Learn
12.1	Log vs Queue — Two Worlds, One System	QueueView concepts, when to use queues
12.2	Configuring QueueView	Per-topic queue semantics, visibility timeout
12.3	AMQP Protocol Adapter	Connect RabbitMQ clients, AMQP 0-9-1
12.4	Visibility Timeout & Delivery Guarantees	At-least-once, dead letter queue
12.5	Priority Lanes	Priority queues, lane routing
12.6	Migrating from RabbitMQ	Drop-in replacement strategy

Unit 13: ASP.NET Core Integration [DEV]

Time: ~3 hours

Integrate Surgewave with ASP.NET Core applications.

Lesson	Topic	What You'll Build
13.1	Hosted Services	BackgroundService, IHostedService
13.2	Dependency Injection	Service registration, scopes
13.3	Health Checks	Broker connectivity, consumer lag
13.4	Middleware & Endpoints	Minimal API, correlation
13.5	Background Processing	Producer pools, graceful shutdown

Unit 14: Wire Protocol [DEV]

Time: ~4 hours

Understand the Kafka wire protocol and build custom clients.

Lesson	Topic	What You'll Learn
14.1	Protocol Basics	Request/response format, API keys, versioning
14.2	Message Format	Record batches, headers, compression, CRC
14.3	Building a Custom Client	Socket connection, serialization, parsing
14.4	Protocol Debugging	Wireshark, hex dumps, troubleshooting
14.5	Protocol Extensions	Surgewave native vs Kafka, performance differences

Unit 15: gRPC/REST Gateway [DEV]

Time: ~3 hours

Use Surgewave's HTTP/gRPC gateway for web clients.

Lesson	Topic	What You'll Learn
15.1	Gateway Architecture	REST vs gRPC, Confluent v3 API compatibility
15.2	REST Produce/Consume	HTTP endpoints for messages
15.3	WebSocket Streaming	Real-time bidirectional communication
15.4	gRPC Clients	Protobuf definitions, streaming RPCs
15.5	Gateway Security	Authentication, rate limiting, CORS

Unit 16: Visual Pipeline UI [DEV]

Time: ~3 hours

Build data pipelines visually with Surgewave's Pipeline Designer.

Lesson	Topic	What You'll Learn
16.1	Pipeline UI Overview	Visual DAG editor, when to use
16.2	Creating Pipelines	Drag-drop nodes, connections
16.3	Connector Nodes	Source/sink/transform configuration
16.4	DAG Patterns	Fan-out, fan-in, parallel processing
16.5	Pipeline Monitoring	Debugging, data flow visualization

Unit 17: Hands-On Pipeline & Workflow Building [DEV]

Time: ~5 hours

Build production-ready data processing and AI pipelines from scratch.

Lesson	Topic	What You'll Learn
17.1	Pipeline Anatomy	PipelineDefinition model, nodes, connections, status lifecycle
17.2	Node Types & Categories	Source, sink, transform, AI/ML, streams, workflow nodes
17.3	Building Your First Pipeline	Step-by-step: Generator -> SQL Transform -> Stdio
17.4	Building a RAG Pipeline	RAG chatbot from template and from scratch with 7 nodes
17.5	Building an Agent Pipeline	Agent with MCP tool access from template
17.6	Workflow Patterns	Human-in-the-loop, batching, routing, retry, stateful
17.7	Pipeline Chat & Streaming	Chat API, sessions, SSE streaming, ChatDrawer
17.8	Advanced Pipeline Features	Templates, seeder, versioning, hot-deploy, sub-pipelines
17.9	Debugging Pipelines	Live preview, breakpoints, step-through, error inspection

Unit 18: AI/LLM Integration [DEV]

Time: ~4 hours

Integrate AI and LLM capabilities with Surgewave.

Lesson	Topic	What You'll Learn
18.1	OpenAI Embeddings	Generate text embeddings via Surgewave connector
18.2	Ollama Local LLM	Private LLM inference without API costs
18.3	Vector Databases	Store and query embeddings with Qdrant
18.4	RAG Pipeline	Retrieval-Augmented Generation architecture
18.5	Semantic Search	Build semantic search applications

Unit 19: MCP Server & AI Agents [DEV]

Time: ~4 hours

Build AI agents with Model Context Protocol.

Lesson	Topic	What You'll Learn
19.1	MCP Protocol Basics	JSON-RPC, tools, resources, prompts
19.2	Surgewave MCP Server	Configure stdio and HTTP/SSE transports
19.3	MCP Tools	Implement Surgewave tools for AI agents
19.4	Claude Desktop Integration	Connect AI assistants to Surgewave
19.5	Custom AI Agents	Build agents with Surgewave's agent framework

Unit 20: AI Guardrails & Agent Memory [DEV]

Time: ~4 hours

Secure AI applications with guardrails and persistent agent memory.

Lesson	Topic	What You'll Learn
20.1	Content Safety	IGuardrail interface, GuardrailResult, severity levels
20.2	PII Detection	PiiDetector for emails, phones, SSNs, IBANs
20.3	Toxicity Filtering	ToxicityFilter with keyword blocklists
20.4	Prompt Injection	PromptInjectionDetector for LLM security
20.5	Content Policy & Pipeline	GuardrailPipeline, DI registration
20.6	Agent Memory	IAgentMemoryStore, MemoryEntry, AgentMemoryContext
20.7	Tool Result Caching	CachedAgentTool, InMemoryToolResultCache

Unit 21: Production Operations [OPS]

Time: ~3 hours

Deploy and operate Surgewave in production.

Lesson	Topic	What You'll Build
21.1	Clustering	Multi-broker setup
21.2	Replication	Data durability, ISR
21.3	Monitoring	Metrics, Prometheus/Grafana
21.4	Security	TLS, SASL, ACLs
21.5	Performance Tuning	Batching, compression
21.6	Admin API	Programmatic cluster management

Unit 22: Observability & Troubleshooting [OPS]

Time: ~3 hours

Monitor, debug, and troubleshoot Surgewave applications.

Lesson	Topic	What You'll Learn
22.1	Structured Logging	Log correlation, trace IDs, structured formats
22.2	Metrics Collection	Counters, gauges, histograms, Prometheus
22.3	Distributed Tracing	OpenTelemetry, span propagation
22.4	Debugging Techniques	Consumer lag, message inspection
22.5	Performance Profiling	Bottleneck identification, optimization

Unit 23: Cloud & Kubernetes Deployment [OPS]

Time: ~3 hours

Deploy Surgewave to cloud and container environments.

Lesson	Topic	What You'll Learn
23.1	Docker Basics	Dockerfile, multi-stage builds
23.2	Docker Compose	Multi-broker setup, networking
23.3	Kubernetes Fundamentals	Deployments, Services, ConfigMaps
23.4	Helm Charts	Packaging, values, templating
23.5	Cloud Providers	AWS, Azure, GCP considerations

Unit 24: Disaster Recovery [OPS]

Time: ~3 hours

Implement disaster recovery strategies for Surgewave clusters.

Lesson	Topic	What You'll Learn
24.1	Backup Strategies	Topic backup, snapshots, retention
24.2	Cross-DC Replication	MirrorMaker, async replication
24.3	Failover Procedures	Failure detection, manual/auto failover
24.4	Data Recovery	Restore from backup, point-in-time recovery
24.5	DR Testing	Chaos engineering, failover drills

Unit 25: Multi-Tenancy [ARCH]

Time: ~3 hours

Build multi-tenant Surgewave applications.

Lesson	Topic	What You'll Learn
25.1	Tenant Isolation	Isolation levels, shared vs dedicated
25.2	Topic Namespacing	Naming conventions, tenant prefixes
25.3	Quota Management	Rate limiting, fair scheduling
25.4	Access Control	Per-tenant ACLs, audit logging
25.5	Resource Management	Partition allocation, monitoring

Unit 26: Tiered Storage [OPS]

Time: ~3 hours

Optimize storage costs with hot/warm/cold data tiers.

Lesson	Topic	What You'll Learn
26.1	Tiered Storage Concepts	Hot/warm/cold tiers, data lifecycle
26.2	Local File System Tier	File-based cold storage
26.3	S3/MinIO Integration	Cloud object storage tier
26.4	Azure Blob Storage	Azure cloud storage integration
26.5	Retention Policies	Lifecycle management, cost optimization

Unit 27: Object Store & Zero-Disk Mode [OPS]

Time: ~3 hours

Cloud object stores as primary storage backends.

Lesson	Topic	What You'll Learn
27.1	Zero-Disk Architecture	Trade-offs, when to use, architecture overview
27.2	S3-Direct Storage	S3 config, MinIO, multipart upload, consistency
27.3	Azure Blob Storage	Container setup, managed identity, GRS
27.4	GCP Cloud Storage	Bucket setup, storage classes, cross-region replication
27.5	Hybrid Storage Strategies	Tiered policies, cost modeling, migration
27.6	Alternative Storage Engines	RocksDB, SQLite, DuckDB, LMDB, Parquet, NVMe, Arrow

Unit 28: Advanced Messaging

Time: ~3 hours

Build sophisticated applications with advanced message patterns.

Lesson	Topic	What You'll Build
28.1	Custom Serialization	JSON, Protobuf serializers
28.2	Headers & Metadata	Correlation IDs, tracing
28.3	Error Handling	Retries, dead letter queues
28.4	Transactions	Atomic multi-message operations
28.5	Exactly-Once Semantics	Idempotent producers
28.6	Compacted Topics	Log compaction, changelog topics
28.7	Ephemeral Topics	Ring-buffer, real-time distribution

Unit 29: Real-World Patterns [ARCH]

Time: ~4 hours

Apply Surgewave to real-world application scenarios.

Lesson	Topic	What You'll Build
29.1	E-Commerce Pipeline	Order processing, inventory updates
29.2	IoT Data Ingestion	Sensor data, time-series processing
29.3	Log Analytics	Centralized logging, aggregation
29.4	Event Sourcing	Event store, projections, snapshots
29.5	Microservices Communication	Async messaging, choreography

Unit 30: Large Message Handling [DEV]

Time: ~3 hours

Handle large messages efficiently in Surgewave.

Lesson	Topic	What You'll Learn
30.1	Message Size Limits	Configuration, trade-offs
30.2	Compression	Codecs, compression levels
30.3	Chunking	Message splitting, reassembly
30.4	Claim-Check Pattern	External storage, references
30.5	Streaming Large Data	Memory efficiency, backpressure

Unit 31: Message Deduplication [DEV]

Time: ~3 hours

Implement idempotent message processing.

Lesson	Topic	What You'll Learn
31.1	Deduplication Concepts	Why duplicates occur, detection strategies
31.2	In-Memory Deduplication	LRU caches, bloom filters
31.3	Persistent Deduplication	Database-backed stores, TTL
31.4	Idempotent Operations	Upserts, conditional writes
31.5	End-to-End Exactly-Once	Transactional patterns

Unit 32: Hybrid Cloud [ARCH]

Time: ~4 hours

Deploy Surgewave across multiple clouds and on-premise.

Lesson	Topic	What You'll Learn
32.1	Hybrid Cloud Concepts	Architecture patterns, use cases
32.2	Multi-Region Deployment	Cross-region Surgewave clusters
32.3	Cross-Cloud Replication	AWS ↔ Azure ↔ GCP
32.4	Geo-Routing	Latency optimization, locality
32.5	On-Premise + Cloud	Hybrid deployments, data sovereignty

Unit 33: Performance & Tuning [DEV/OPS]

Time: ~4 hours

Optimize Surgewave for maximum throughput and minimum latency.

Lesson	Topic	What You'll Learn
33.1	Performance Fundamentals	Throughput, latency, P50/P90/P99
33.2	Producer Optimization	Batching, compression, acks
33.3	Consumer Optimization	Fetch size, parallelism
33.4	Memory & GC Tuning	Memory pools, zero-copy, GC modes
33.5	Benchmarking	BenchmarkDotNet, profiling tools

Unit 34: Performance Engineering [DEV]

Time: ~4 hours

Master zero-allocation patterns, SIMD, pooling, and regression detection.

Lesson	Topic	What You'll Learn
34.1	Performance Architecture	Hot paths, metrics, Kafka vs Surgewave comparison
34.2	Zero-Allocation Patterns	Span, Memory, stackalloc, ArrayPool, ValueTask
34.3	SIMD & Hardware Acceleration	CRC32C intrinsics, Vector cosine similarity
34.4	Memory Pooling	BufferPool, RecyclableMemoryStream, object pooling
34.5	Hot-Path Optimizations	LINQ elimination, string interning, compression bypass
34.6	Regression Suite	RegressionDetector, baselines, CI workflow
34.7	BenchmarkDotNet	Surgewave's 7 benchmark projects, writing benchmarks

Unit 35: Chaos Testing & Resilience [DEV/OPS]

Time: ~4 hours

Build resilient systems with proactive chaos testing.

Lesson	Topic	What You'll Learn
35.1	Chaos Engineering Intro	Principles, motivation, ChaosEngine architecture
35.2	The ChaosEngine	Fault activation, deactivation, FaultScope targeting
35.3	Storage Fault Injection	ChaosLogSegment for disk, corruption, partial write faults
35.4	Transport Fault Injection	ChaosRaftTransport for network and election faults
35.5	Scenario Helpers	NetworkPartition, BrokerCrash, DiskFailure, Latency scenarios
35.6	ChaosCluster	Multi-broker integration tests with chaos
35.7	FaultSchedule	Time-delayed fault activation and deactivation

How to Use This Bootcamp

Learning Path

1. Read the README - Understand the lesson objectives
2. Review Concepts - Learn the theory in INSTRUCTIONS.md
3. Code the Exercise - Complete the start/ code
4. Check Your Work - Compare with completed/
5. Experiment - Try variations and edge cases

Folder Structure

lesson-X/
├── README.md           # Lesson overview and objectives
├── INSTRUCTIONS.md     # Detailed step-by-step guide
├── start/              # Your starting point (incomplete)
│   ├── Program.cs
│   └── *.csproj
└── completed/          # Reference solution
    ├── Program.cs
    └── *.csproj

Running Exercises

# Navigate to lesson
cd units/unit-1/lesson-1

# Start with the incomplete code
cd start
dotnet run

# Check completed solution
cd ../completed
dotnet run

FAQ

Do I need prior Kafka experience?

No! This bootcamp teaches streaming concepts from scratch. Kafka experience helps but isn't required.

Can I skip lessons?

Each unit builds on previous lessons. Within a unit, lessons are sequential. You can skip entire units if you're experienced.

How do I get help?

• Check the completed/ solution
• Open a GitHub Issue
• Review Surgewave Documentation

Can I contribute lessons?

Yes! See CONTRIBUTING.md for guidelines.

Guided Tours

The Bootcamp includes 35 Shepherd.js interactive guided tours -- one per unit. Tours provide step-by-step walkthroughs directly in the DocFX learn site, highlighting key concepts and UI elements.

Tour definitions live in units/tours/ as JSON files following the tour.schema.json schema.

To experience the tours, build and serve the learn site:

.\build-learn.ps1 -Serve -Port 8081
# Open http://localhost:8081 and click the tour button on any unit page

Dependencies

• Surgewave Core: All units reference the main Surgewave packages
• Surgewave.AI: Units 18, 19, 20, and 17 require Surgewave.AI packages (Agents, RAG, Guardrails, Documents, MCP)
• Surgewave.Connectors: Unit 9 and 17 reference Surgewave.Connectors packages

What's Next?

After completing the bootcamp:

1. Build a Project -- Apply your skills to a real problem
2. Explore Samples -- Check out Surgewave.Samples for 28 ready-to-run applications
3. Contribute -- Help improve Surgewave or this bootcamp
4. Browse Templates -- Use Surgewave.Templates for dotnet new project scaffolding

---

Ready to start? --> Begin with Unit 0: Introduction