granite

=============================

中文

Introduction

Granite is a fully functional modern storage engine kernel developed in depth based on RocksDB. It is compatible with Redis data structures and protocol specifications, and can be used as a multi-form storage base. Its core is the granite::Storage module, which implements all the core capabilities required by a storage engine, rather than a simple encapsulation of RocksDB.

Granite adopts an abstract interface and multi-backend pluggable architecture, allowing upper-layer services to seamlessly switch between different storage forms without modifying code, including pure in-memory high-speed engine, disk persistence engine (RocksDB), remote RPC distributed engine, and hybrid hot-cold storage engine.

This engine can be used as an underlying core to build high-performance cache services, persistent KV databases, Redis-compatible services, distributed storage systems, message stream storage, and index storage applications.

Core Features

1. Unified Abstract Architecture

• Provides a standardized storage interface that uniformly supports four storage forms: in-memory, RocksDB, Remote, and hybrid storage.

• Upper-layer services are completely decoupled from the underlying storage implementation, enabling switching between storage forms without modifying business code.

2. Enterprise-Grade Storage Capabilities

• Supports multi-ColumnFamily structured storage, including eight column families: default, metadata, zset_score, pubsub, propagate, stream, search, and index, each corresponding to different data scenarios.

• Fully compatible with all Redis data structures, including String, Hash, List, Set, ZSet, Stream, PubSub, Search, and Index.

• Integrates BlobDB features to automatically optimize storage for large-value data.

• Supports multiple compression algorithms (Snappy, Zlib, LZ4, ZSTD), which can be flexibly configured according to scenarios.

• Built-in dual-cache system (LRU and HyperClockCache) to improve data access performance.

• Supports IO throttling, SST file cleanup throttling, and DB space limit protection to ensure stable system operation.

3. High Availability and Data Security

• Supports online Checkpoint snapshot function, which can generate real-time data snapshots for backup and recovery.

• Provides a complete backup and atomic recovery mechanism to ensure rapid data recovery in abnormal scenarios.

• Built-in full replication and incremental replication base, compatible with Redis PSYNC protocol, supporting master-slave synchronization.

• Supports WAL log iteration and replay, which can recover data or implement incremental synchronization based on logs.

• Implements automatic generation, management, and persistence of replication ID (replid) to ensure the consistency of the replication process.

4. Production-Grade Stability

• Implements a complete transaction model and batch write mechanism, supporting atomic execution of transactions.

• Unifies transaction context through Context, standardizes transaction operation processes, and avoids concurrency conflicts.

• Built-in read-write locks to ensure concurrent access security, replication security, and system shutdown security.

• Real-time statistics of Compaction and Flush operation times, as well as Keyspace Hit/Miss indicators, facilitating performance monitoring and problem troubleshooting.

• Supports real-time monitoring of DB size, automatically switching to read-only mode when the set limit is reached to prevent disk overflow.

5. SST Import and Export (Data Migration/Expansion)

• Supports cross-column-family atomic SST import to ensure data consistency during the import process.

• Provides full SST dump function, which can export data as SST files for offline backup.

• Adapts to offline data migration, data distribution, and node scaling scenarios, and can be directly connected to data synchronization and backup systems.

System Architecture

Upper-Layer Services (Redis Protocol / gRPC / HTTP / Custom Protocol)
          ⇵
Unified Abstract Interface Layer (RedisDBBase)
          ⇵
Engine Core Layer (granite::Storage)
          ⇵
Underlying Backends (RocksDB / Memory / Remote / TiKV)

The architectural design of Granite follows the interface-first principle, isolating upper-layer services from the underlying storage implementation through a unified abstract interface layer. Its core design goals include:

• Standardized Interface: Defines a unified storage protocol to ensure decoupling between upper-layer services and specific storage implementations.

• High Scalability: Based on abstract interfaces, it can easily extend multiple storage backends such as in-memory, remote, and distributed versions.

• Production-Grade Adaptation: Built-in features such as high availability, monitoring, statistics, throttling, and security to meet commercial production needs.

• Redis Compatibility: The interface design is consistent with Redis protocol specifications, and it can be directly used as the storage kernel for Redis protocol services.

Application Scenarios

• Underlying storage kernel for high-performance persistent Redis services.

• Core component of distributed cache and storage systems.

• Storage base for large-scale stream data (Stream).

• Underlying storage support for index and search services.

• Storage layer implementation for message PubSub systems.

• Core engine for hybrid hot-cold storage scenarios.

• Database kernel scenarios requiring backup and recovery, as well as high availability guarantees.

🛠️ Build

This project uses kmpkg for dependency management and build integration. kmpkg automatically handles third-party library downloads, dependency lookup, compilation flag configuration, etc., avoiding the need to manually maintain complex CMake configurations.

0. Prepare the Environment

• Linux (Ubuntu 20.04+ / CentOS 7+ recommended)

• CMake >= 3.25

• GCC >= 9.4 / Clang >= 12

• kmpkg installed ( see Installation Documentation)

1. Configure the Project (Optional)

• For complete dependencies, refer to kmpkg.json

• To update the dependency baseline, refer to kmpkg-configuration.json and modify the baseline of default-registry

• The baseline can be obtained through git log

• Optional: Users can manage dependencies manually to ensure that CMake's find_package can correctly locate the required libraries.

• For example, install dependencies in the system path or a custom path, and specify it via CMAKE_PREFIX_PATH

• Or declare the external dependency path in kmpkg to avoid duplicate downloads

2. Compile the Project

Execute the following commands in the project root directory:

cmake --preset=default
cmake --build build -j$(nproc)

Manual dependency management:

mkdir build
cd build
cmake ..
make -j$(nproc)

Note

--preset=default requires that the corresponding CMake Preset has been defined in the project root directory.

3. Run Tests (Optional)

Execute the following command in the project root directory:

ctest --test-dir build

Summary

As a modern storage engine kernel, Granite is built on and deeply optimized based on RocksDB. It has complete core storage capabilities, including multi-form storage support, high availability guarantees, data security, production-grade monitoring, and flexible scalability. Its architectural design ensures decoupling from upper-layer services, and it can be directly used as the underlying core component of commercial storage systems, Redis-compatible services, distributed databases, and cache systems.