Skip to content

About

A highly performant, distributed Key-Value store built in C++20. Features a custom TCP protocol, 16-shard concurrent storage engine, strict crash consistency via a binary Write-Ahead Log (WAL), $O(1)$ LRU caching, Raft consensus for leader election, and a Consistent Hash Ring for horizontal sharding with auto-failure detection.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Mini Distributed Key-Value Store

A highly performant, distributed, partitioned, and persistent key-value store built in modern C++ (C++20).

Features

  • Custom Framed Protocol: Uses a length-prefixed TCP protocol with JSON payloads (nlohmann/json).
  • Concurrent Storage Engine: Thread-safe in-memory std::unordered_map with 16-shard fine-grained locking.
  • Write-Ahead Log (WAL): Ensures strict crash consistency with tunable durability (ASYNC, SYNC). Features magic headers, CRC checksums, and startup replay logic.
  • LRU Cache: A fast-path $O(1)$ memory cache for accelerating reads (GET).
  • Global Metrics: Lock-free (std::atomic) counters tracking request volumes, cache hits/misses, and storage ops.
  • Raft Consensus: Core Raft algorithm implementation handling elections, randomized timeouts, heartbeats, log replication, and majority-quorum commits.
  • Dynamic Sharding: A Consistent Hash Ring with virtual nodes that automatically maps keys to partitions.
  • Failure Detection: Seamless integration between Raft heartbeats and the hash ring. Unresponsive nodes are evicted dynamically, auto-healing the cluster.

Getting Started

Prerequisites

  • Docker & Docker Compose
  • run_dev.ps1 helper for local compiling

Build the Node Binary

We provide a helper script to build the project inside a fully POSIX-compliant Ubuntu 24.04 environment.

.\run_dev.ps1 "cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j4"

Run the 5-Node Cluster

We provide a docker-compose.yml to spin up the distributed topology instantly.

docker-compose up --build -d

Usage

Use the built-in CLI to interact with the cluster.

# Set a value on Node 1 (if the key hashes to another node, Node 1 will respond with REDIRECT)
.\run_dev.ps1 "./build/src/client/kv-cli -h 127.0.0.1 -p 7001 set mykey hello_world"

# Get a value
.\run_dev.ps1 "./build/src/client/kv-cli -h 127.0.0.1 -p 7001 get mykey"

# View Cache and Node Metrics
.\run_dev.ps1 "./build/src/client/kv-cli -h 127.0.0.1 -p 7001 stats"

Testing & Benchmarking

Run the GoogleTest suite:

.\run_dev.ps1 "cmake -DBUILD_TESTING=ON -S . -B build && cmake --build build -j4 && cd build && ctest"

Run the benchmark (Python required on host):

# Requires python with numpy installed
pip install numpy
python benchmark.py --host 127.0.0.1 --port 7001 --threads 10 --requests 1000

About

A highly performant, distributed Key-Value store built in C++20. Features a custom TCP protocol, 16-shard concurrent storage engine, strict crash consistency via a binary Write-Ahead Log (WAL), $O(1)$ LRU caching, Raft consensus for leader election, and a Consistent Hash Ring for horizontal sharding with auto-failure detection.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages