xconfig

=============================

English|中文版

xconfig is a self-developed, high-performance, lock-free, atomic hot-update configuration management framework by Kumo, designed specifically for high-concurrency distributed services, storage engines, gateways, and foundational system components. Built on Protobuf's strong static type system, it natively supports seamless lossless bidirectional conversion between TOML / JSON / Protobuf. Powered by a double-buffered read-write separation architecture, it achieves completely lock-free, wait-free, thread-safe configuration reading with near-native pointer access performance under extreme high-concurrency workloads.

Features

• Full Bidirectional Format Conversion Supports complete lossless interconversion among TOML ↔ JSON ↔ Protobuf binary. Use human-readable TOML for static configuration files, Protobuf for compile-time strong type safety, and JSON for external API data interaction. Single configuration instance adapts all scenarios.

• Lock-Free High-Concurrency Reading (Wait-Free) Implemented with DoublyBufferedData double-buffer architecture. All runtime read operations perform zero locking, zero contention, maintaining performance equivalent to raw pointer access even under massive concurrent reads. Write operations use fine-grained mutex protection without blocking ongoing reads.

• Atomic Hot-Update & Consistent Memory View All configuration updates are fully atomic with no partial loading or intermediate inconsistent states. All concurrent readers always obtain a complete, consistent configuration snapshot, completely eliminating runtime logic errors caused by partial hot-reloading.

• Multi-Version Management & One-Click Rollback Automatically persists every valid configuration revision as an immutable historical snapshot. Provides version indexing, current version query, and arbitrary historical version switching. Supports online rollback without process restart, greatly improving operation & maintenance safety.

• Global Background Recycler & Tombstone Persistence ✨New in v0.5.0✨ A single global background thread periodically recycles old configuration versions according to configurable policies (soft/hard limits, minimum retention time). Recycled versions can be optionally persisted to disk as TOML/JSON tombstone files, enabling “soft delete” and offline recovery. Dedicated restore_file() APIs allow reloading any tombstone file as a new version, preserving original version number and timestamp.

• Comprehensive Validator System & Callback Mechanisms Equipped with a full-featured strong-type validator system for config field validation, including: numeric comparison operators (<, <=, >, >=, ==) for integer & floating types; full closed/open interval range validation ([min,max], [min,max), (min,max), (min,max]); forbidden out-of-range validation to restrict values from specific intervals; whitelist in_set & blacklist out_of_set validation based on flat hash set; full string validators (prefix, suffix, case-insensitive equal, regex match); repeated field vector size boundary validation; and custom functional field validators with callback support. Extended support for std::string lexicographical order comparison is also provided. Additionally, it includes independent customizable global cross-field validator hooks before configuration update, and post-update notification callbacks after successful switch, enabling external component refresh, connection pool rebuild, route reload and fully decoupled runtime hot-reload logic.

• Global Multi-Instance Configuration Registry Unified global singleton manager supports arbitrary multiple independent Protobuf message types simultaneously. Each configuration instance maintains isolated version chain, read-only field set, callbacks and internal state with complete cross-instance isolation. Zero-cost onboarding for new service configurations.

• Official Standard Protobuf FieldPath Fine-Grained Read-Only Protection Full native support for official Protobuf FieldPath specification for field access control. Covers singular fields, repeated scalar elements with array indices, nested message fields, and deep nested repeated paths such as contributors[0].email, save[1].seconds. Read-only constraint validation is integrated into the entire update pipeline automatically, rejecting illegal modifications internally without exposing internal implementation details.

• Native Deep Protobuf Integration & Type-Safe Conversion Only a plain Protobuf message definition is required. No extra boilerplate serialization/deserialization code. Fully supports string/file parsing and format exporting with complete type safety. Enhanced type-safe conversion system enables automatic safe widening conversion between numeric types (int → double allowed, no implicit unsafe narrowing), and a unified intermediate value wrapper (ProtoPrimerValue variant) supports all proto primitive & repeated types.

• Minimal Integration API & Strong Memory Safety Lightweight registration interface, simple singleton acquisition, unified configuration API across all message types. Internal configuration lifetime managed via std::shared_ptr to guarantee no dangling pointers, memory leaks or double-free issues. All validators perform pre-registration type & field existence check to avoid runtime invalid usage errors, with strict type boundary checks to disallow mismatched validator types on fields (e.g., numeric validator on string, string validator on numeric).

Architecture Highlights

• Read-Write Separation: Lock-free reads, fine-grained short-duration locks for writes, vastly outperforms traditional global-mutex configuration frameworks in high-concurrency scenarios.
• Atomic Pointer Switching: Configuration swap is a single atomic pointer replacement with zero intermediate states.
• Strong Static Typing: Full structural and type validation enforced by Protobuf schema, enhanced by pre-registration validator checks.
• Complete Encapsulation: Internal data and logic well-encapsulated; external interfaces only expose safe const accessors. Fully reused underlying utility functions eliminate duplicate code.
• Global Singleton Registry: Cross-module global configuration access, no instance passing required, accessible anywhere in the program.
• Simple & Reliable Background Recycler: Uses second‑granularity polling instead of condition variables or eventfd, reducing complexity and AI confusion while meeting all production requirements. Tombstone directory changes are picked up automatically on the next cycle.
• Full Boundary Checking: Built-in validation for field path out-of-bounds, invalid path segments, unsupported field types, null pointer protection, and enhanced error status with clear human-readable error messages for all validation failures.

Dependencies

• C++17 or newer
• Protobuf 3+
• nlohmann/json
• turbo (Kumo base library)
• CMake 3.20+

🛠️ Compilation and Construction

This project uses kmpkg for dependency management and build integration, which can automatically complete the download of third-party libraries, dependency resolution, and compiler parameter configuration, eliminating the need to manually maintain complex CMake configurations.

0. Environment Preparation

• Operating System: Linux (Ubuntu 20.04+ or CentOS 7+ recommended)
• Build Tool: CMake ≥ 3.31
• Compiler: GCC ≥ 9.4 or Clang ≥ 12
• Dependency Tool: kmpkg (must be installed correctly, refer to relevant installation documents)

Quick Installation of kmpkg

git clone https://github.com/kumose/kmpkgcore.git
cd kmpkgcore
./bootstrap-kmpkg.sh

Configure environment variables (can be written to ~/.bashrc for permanent effect; run source ~/.bashrc to take effect):

export KMPKG_HOME=/home/jeff/kmpkgcore
export PATH=$PATH:$KMPKG_HOME
export KMPKG_CMAKE=$KMPKG_HOME/scripts/buildsystems/kmpkg.cmake

1. Project Configuration (Optional)

• For complete dependency information, refer to the kmpkg.json file in the project root directory.
• To update the dependency baseline version, modify the baseline field under default-registry in the kmpkg-configuration.json file (the baseline can be obtained via the git log command).

2. Project Compilation

Execute the following commands in the project root directory to complete the compilation and construction:

cmake --preset=default
cmake --build build -j$(nproc)

• If you want to run benchmark, add -DKMCMAKE_BUILD_BENCHMARK=ON
• If you want to run examples, add -DKMCMAKE_BUILD_EXAMPLES=ON
• KMCMAKE_BUILD_TEST is ON by default. Add -DKMCMAKE_BUILD_TEST=OFF when installing.

Manual Dependency Management (Optional)

If you need to manage dependencies manually, you can compile using standard CMake commands:

mkdir build
cd build
cmake ..
make -j$(nproc)

Note: --preset=default requires that the corresponding CMake preset is defined in the project root directory; when managing dependencies manually, ensure that CMake's find_package can locate all required libraries.

3. Compilation Outputs

• libxconfig.a: Main static library
• libxconfig_testing.a: Test support library

4. Run Tests

ctest --test-dir build

Usage

1. Define Protobuf Config

syntax = "proto3";

message ServerConfig {
  string ip = 1;
  int32 port = 2;
  bool enable_ssl = 3;
  double timeout = 4;
}

message RedisStyleConfig {
  string bind = 1;
  int32 port = 2;

  message SaveRule {
    int32 seconds = 1;
    int32 writes = 2;
  }
  repeated SaveRule save = 5;
}

2. Framework Initialization & Basic Usage (Including New Validator APIs)

#include "xconfig/xconfig.h"
#include "config.pb.h"

using namespace xconfig;

int main() {
    auto& instance = XConfigInstance::instance();
    const std::string config_name = "server";

    ServerConfig default_proto;
    instance.initialize(config_name, default_proto);
    auto cfg = instance.get_config(config_name);

    // Numeric comparison validator (int32)
    cfg->greater_equal<int32_t>("port", 1024);
    cfg->less_equal<int32_t>("port", 65535);

    // Numeric range validator (double, [5.0, 60.0))
    cfg->in_range<double, true, false>("timeout", 5.0, 60.0);

    // String validators
    cfg->starts_with("ip", "192.168");
    cfg->regex_match("ip", "^((25[0-5]|2[0-4]\\d|[01]?\\d\\d?)\\.){3}(25[0-5]|2[0-4]\\d|[01]?\\d\\d?)$");
    cfg->ignore_case_equal("ip", "192.168.1.1");

    // Set validator (whitelist)
    std::vector<int32_t> allowed_ports = {80, 443, 8080, 8081};
    cfg->in_set<int32_t>("port", allowed_ports);

    // Custom global cross-field validator
    cfg->set_validator([](const google::protobuf::Message* msg) {
        const auto* config = static_cast<const ServerConfig*>(msg);
        if (config->enable_ssl() && config->port() != 443) {
            return turbo::invalid_argument_error("SSL enabled must use port 443");
        }
        return turbo::OkStatus();
    });

    // Post-update callback
    cfg->set_updater([](const google::protobuf::Message* old_msg, const google::protobuf::Message* new_msg) {
        // Refresh external components (e.g., connection pool, route) after update
        return turbo::OkStatus();
    });

    // Read-only field protection (FieldPath support)
    cfg->mark_read_only_field("port");
    cfg->mark_read_only_field("save[0].seconds"); // Nested repeated field

    // --- Tombstone persistence (optional, v0.5.0) ---
    instance.set_tombstone_dir("./tombstones");
    instance.set_tombstone_format_json();          // Use JSON format (default is TOML)
    instance.set_tombstone_check_interval(60);     // Scan every 60 seconds

    // --- Configure recycling policy ---
    cfg->set_max_versions(100);
    cfg->set_min_reserved_hours(24);
    cfg->set_hard_limit(500);

    // Parse TOML string
    std::string toml_content = R"(
ip = "192.168.1.1"
port = 8080
enable_ssl = false
timeout = 30.0
)";
    auto status = cfg->parse_toml_string(toml_content, true);

    // Get current config & export to different formats
    const auto* current = static_cast<const ServerConfig*>(cfg->get_config());
    auto toml = cfg->save_to_toml();
    auto json = cfg->save_to_json();
    auto pb_bin = cfg->save_to_pb();

    // Version management
    size_t current_ver = cfg->current_version();
    cfg->set_to_version(0); // Rollback to initial version

    // --- Restore a previously recycled version from tombstone ---
    auto files = instance.get_tombstone_files("server");
    if (!files.empty()) {
        cfg->restore_file(files[0], false); // Restore as a new version (not active)
    }

    return 0;
}

3. Complex FieldPath Read-Only Example

// Nested repeated field read-only
cfg->mark_read_only_field("save[0].seconds");
// Deep nested field read-only
cfg->mark_read_only_field("contributors[0].email");

See demos example

Supported Formats

Load Configuration

• parse_toml_string / parse_toml_file
• parse_json_string / parse_json_file
• parse_pb_string / parse_pb_file

Export Configuration

• save_to_toml()
• save_to_json()
• save_to_pb()

Core Advantages

• Extreme Runtime Performance Zero overhead read access suitable for million-QPS high-concurrency underlying components.
• Absolute Runtime Stability No dirty reads, no deadlocks, no partial updates, no invalid memory access.
• High Development Efficiency Protobuf schema serves as single source of truth; multi-format parsing, validation, versioning and access control all built-in.
• Operation & Maintenance Friendly On-the-fly format export, historical version rollback, tombstone‑based recovery, and explicit error diagnostics for all invalid operations.
• Production-Grade Completeness Full API unit test coverage, all edge cases validated, production ready for storage engine and middleware infrastructure.
• Universal Multi-Component Base Unified configuration foundation for entire project, supports unlimited independent configuration entities.

License

Apache License 2.0

Copyright (C) 2026 Kumo inc. and its affiliates. All Rights Reserved.