A fault-tolerant distributed message queue inspired by Kafka — written in Go. It uses Raft (via Dragonboat) for consensus, BoltDB for per-partition storage, and features consumer groups with sticky re-balancing.
- Topic/Partition model with persistent on-disk storage (BoltDB per partition)
- Consumer Groups with sticky partition assignments
- At-least-once delivery
- gRPC API for Producer/Consumer interactions
- Distributed replication using Dragonboat Raft
- IOnDiskStateMachine integration for crash-safe FSM state
- Heartbeats and Sticky Rebalancing for dynamic consumer group coordination
- Manual
AckandPollAPIs to control delivery semantics
- Raft Shards map 1:1 with queue partitions (one Raft group per partition)
- Storage engine is decoupled using interface-based abstractions
- Custom
MessageServiceorchestrates routing logic and partition calculations - No centralized metadata — each broker stores partition state locally
- Go 1.24
- Protobuf Compiler (
protoc) if you want to generate gRPC code
$ go build -o queue . && ./queuethis will start a single broker instance with grpc enabled on 0.0.0.0:8000, check out queue.proto for the proto definition
+----------------+
| Client |
| (Producer/Cons)|
+-------+--------+
|
v
+----------+----------+
| Broker | <-- Handles API (Send, Poll, Ack, etc.)
| (Multiple instances)| <-- Maps topic-partitions, maintains metadata
+----------+----------+
|
+------------------------+-------------------------+
| | |
v v v
+---------------+ +---------------+ +---------------+
| Partition 0 | | Partition 1 | ... | Partition N |
| (TopicX-P0) | | (TopicX-P1) | | (TopicY-PM) |
+------+--------+ +------+--------+ +------+--------+
| | |
| Raft Group (Shard) | Raft Group (Shard) | Raft Group (Shard)
| | |
+------+------+ +------+------+ +------+------+
| Broker A | | Broker B | | Broker C |
| (Leader) | | (Leader) | | (Leader) |
| BoltDB P0 | | BoltDB P1 | | BoltDB PM |
+------+------+ +------+------+ +------+------+
| | |
+------+------+ +------+------+ +------+------+
| Broker B | | Broker C | | Broker A |
| (Follower) | | (Follower) | | (Follower) |
+-------------+ +-------------+ +-------------+
<< Metadata Raft Group >>
<< Replicates metadata to all brokers >>
+---------------------------------------------------------------+
| Each broker has its own `metadata.bolt` file |
| - Topic definitions |
| - Partition assignments |
| - Consumer group membership & offsets |
| Metadata is kept in sync via a Raft shard across brokers |
+---------------------------------------------------------------+
Handles incoming gRPC and HTTP requests, and translates them into internal commands.
grpc/server.go: gRPC transport layerhttp/server.go: Optional HTTP transport layer
Main orchestration logic:
- Topic and partition creation
- Routing messages
- Interacting with FSMs and services
Acts as the glue between transports, services, and Raft.
Implements Dragonboat's IOnDiskStateMachine:
BrokerFSM: Tracks broker, topic, consumer metadataMessageFSM: Handles message persistence, ack, polling
Domain logic for various operations:
BrokerService: Broker registration, shard info for each partition, entire cluster metadataConsumerService: Consumer group registration, re-balancingTopicService: Topic creation and introspectionMessageService: Append, poll, ack, etc.
Each partition’s finite state machine (FSM) persists its data in its own BoltDB file. so in a n-node cluster every message in a partition is stored on its leader plus n-1 additional nodes. Dragonboat takes care of Raft log replication for these partition shards. Separately, all cluster metadata—topic definitions, consumer group state, and shard membership is kept in sync across every broker via a dedicated Raft shard.
Implements partition assignment strategies (currently hardcoded to sticky partition assignment):
- Assigns partitions to consumers
- Triggers rebalance on consumer join/leave
- Can be extended with more strategies (e.g., round-robin)
This system uses Viper for configuration loading.
- Flags (via
pflag) - Environment Variables
- All config types supported by viper
To simplify client-broker interaction, a gRPC proxy is introduced at the client side.
- The proxy determines the correct broker using metadata (Topic, Partition).
- Routing is abstracted from application logic.
- Built using a custom gRPC resolver + load balancer that:
- Fetches broker topology via lightweight HTTP endpoints.
- Performs refreshes periodically to keep metadata in sync.
- Enables clean and scalable partition-level request routing.
This ensures that the application client simply calls Poll, Ack, or Append, without needing to know broker details.
The queue system supports adding brokers dynamically to the cluster.
- New brokers can be added without leader restart
- New partitions can be assigned to the new brokers.
- Existing partitions are not rebalanced — no reshuffling or migration is performed.
This makes horizontal scaling straightforward without introducing complexity or data shuffling overhead.
