🌊 Danube Messaging

A lightweight and scalable Cloud-Native Messaging Platform with Cloud Object Storage (S3/GCS/Azure)

Danube is an open-source distributed messaging broker platform inspired by Apache Pulsar, designed to be cloud-native and cost-effective. Built with a Write-Ahead Log (WAL) architecture and persistent object storage integration, Danube delivers sub-second dispatch with cloud economics.

🚀 Get Started with Danube

Quick Start with Docker Compose - Deploy a cluster in seconds:

Create a directory and download the required files:

mkdir danube-docker && cd danube-docker

Download the docker-compose and broker configuration file:

curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/docker-compose.yml

curl -O https://raw.githubusercontent.com/danube-messaging/danube/main/docker/danube_broker.yml

Start the danube cluster:

docker-compose up -d

This launches a complete Danube cluster with:

• 2 High-Availability Brokers for topics failover
• ETCD for distributed metadata management
• MinIO S3-Compatible Storage for cloud-ready persistence
• Automatic bucket creation and configuration

Test the setup:

Produce messages with reliable delivery

docker exec -it danube-cli danube-cli produce \
  --service-addr http://broker1:6650 \
  --topic "/default/persistent-topic" \
  --count 100 \
  --message "Persistent message" \
  --reliable

Consume messages from the topic

docker exec -it danube-cli danube-cli consume \
  --service-addr http://broker1:6650 \
  --topic "/default/persistent-topic" \
  --subscription "persistent-sub" \
  --sub-type exclusive

📦 Run with Docker (choose one):

• Minimal stack (brokers + dependencies): docker/README.md
• Stack with Admin UI (UI + gateway + Prometheus): docker/danube_with_ui/README.md

📖 Complete Docker Setup Guide →

Architecture

🏗️ Cluster & Broker Characteristics

• Stateless brokers: Metadata in ETCD and data in WAL/Object Storage
• Horizontal scaling: Add brokers in seconds with zero-downtime expansion
• Intelligent load balancing: Automatic topic placement and rebalancing across brokers
• Rolling upgrades: Restart or replace brokers with minimal disruption
• Security-ready: TLS/mTLS support in Admin and data paths
• Leader election & HA: Automatic failover and coordination via ETCD
• Multi-tenancy: Isolated namespaces with policy controls

🌩️ Write-Ahead Log + Cloud Persistence

• Cloud-Native by Design - Danube's architecture separates compute from storage
• Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob, MinIO
• Hot path optimization: Messages served from in-memory WAL cache
• Stream per subscription: WAL + cloud storage from selected offset
• Asynchronous background uploads to S3/GCS/Azure object storage
• Infinite retention without local disk constraints

🎯 Intelligent Load Management

• Automated rebalancing: Detects cluster imbalances and redistributes topics automatically
• Smart topic assignment: Places new topics on least-loaded brokers using configurable strategies
• Resource monitoring: Tracks CPU, memory, throughput, and backlog per broker in real-time
• Configurable policies: Conservative, balanced, or aggressive rebalancing based on workload
• Graceful topic migration: Moves topics between brokers without downtime

Core Capabilities

📨 Message Delivery

• Topics: Partitioned and non-partitioned with automatic load balancing
• Reliable Dispatch: At-least-once delivery with configurable storage backends
• Non-Reliable Dispatch: High-throughput, low-latency for real-time scenarios

🔄 Subscription Models

• Exclusive: Single consumer per subscription
• Shared: Load-balanced message distribution across consumers
• Failover: Automatic consumer failover with ordered delivery

📋 Schema Registry

• Centralized schema management: Single source of truth for message schemas across all topics
• Schema versioning: Automatic version tracking with compatibility enforcement
• Multiple formats: Bytes, String, Number, JSON Schema, Avro, Protobuf
• Validation & governance: Prevent invalid messages and ensure data quality

🤖 AI-Powered Administration

Danube features the AI-native messaging platform administration through the Model Context Protocol (MCP):

• Natural language cluster management: Manage your cluster by talking to AI assistants (Claude, Cursor, Windsurf)
• 32 intelligent tools: Full cluster operations accessible via AI - topics, schemas, brokers, diagnostics, metrics
• Automated troubleshooting: AI-guided workflows for consumer lag analysis, health checks, and performance optimization
• Multiple interfaces: CLI commands, Web UI, or AI conversation - your choice

Example: Ask Claude "What's the cluster balance?" or "Create a partitioned topic for analytics" and watch it happen.

Community & Clients

Official Clients

• Rust Client - Full-featured async client with examples
• Go Client - Production-ready client with examples

Community Contributions

Contributions in Python, Java, JavaScript, and other languages are welcome! Join our growing ecosystem.

Development & Contribution

Get involved - Danube is actively developed with new features added regularly.

🐛 Report Issues | 💡 Request Features | 📖 Development Guide

Project Structure

• danube-broker - Core messaging platform
• danube-persistent-storage - WAL and cloud storage integration
• danube-client - Async Rust client library
• danube-cli - Command-line producer/consumer tools
• danube-admin - Unified admin tool (CLI + AI/MCP + Web UI)