Refactor eloq substrate init function

Feature Summary

In order to support converged db, we need to extract the data substrate related initialization into a separate function.

Motivation

Proposed Solution

In the init function, it should read a config file that has all configs related to data substrate. This config file should not have any database engine related configs.
It should initialize store_hd, log_service and tx_service in the right order. As the three core component of data substrate, they should be able to be accessed globally.
After data substrate initialization is done, database engine can do their own initialization and pass in its engine specific configs to data substrate (i.e. catalog factory)

Research: Data Substrate Initialization Analysis for Converged Database

Date: 2025-09-25T06:40:15Z
Researcher: Claude
Git Commit: N/A (not a git repository)
Branch: N/A (not a git repository)
Repository: converged

Research Question

What needs to be extracted into a shared data substrate initialization function to enable eloqkv and eloqsql to share the same data substrate components (tx_service, store_hd, log_service)? The data substrate will support multiple database engines (eloqsql, eloqkv, eloqdoc) simultaneously and use a unified configuration system.

Summary

Both eloqkv and eloqsql currently initialize their own data substrate components during their respective init functions. The data substrate consists of three core components: tx_service, store_hd (storage handler), and log_service (log agent). The new approach will use a unified configuration file + gflags pattern, with the data substrate reading from a separate config file without any engine-specific patterns. The TxService will accept multiple catalog factories (array of size 3) to support multiple engines simultaneously, with engine-specific dependencies wrapped in compile-time macros.

Detailed Findings

Current Initialization Functions

eloqsql Initialization (`ha_eloq.cc:2397`)

Function: eloq_init_func(void *p)
Location: eloqsql/storage/eloq/ha_eloq.cc:2397-3283
Configuration System: MySQL system variables with eloq_ prefix
Key Components Initialized:
- Storage handler (storage_hd) - lines 2558, 2573, 2750
- Log agent (log_agent) - line 3095
- Tx service (tx_service) - lines 3209, 3227

eloqkv Initialization (`redis_service.cpp`)

Function: RedisServiceImpl::Init() (main initialization)
Location: eloqkv/src/redis_service.cpp
Configuration System: gflags with FLAGS_ prefix + INI config files
Key Components Initialized:
- Storage handler (store_hd_) - lines 961, 978, 1138
- Log agent (log_agent) - line 1374
- Tx service (tx_service_) - line 1437

Data Substrate Components Analysis

1. TxService

Purpose: Core transaction management service
Initialization Pattern:

Both projects create std::unique_ptr<TxService>
Requires catalog factory, system handler, configuration map, and other dependencies
Key Dependencies:
- CatalogFactory (engine-specific)
- SystemHandler (engine-specific: MariaSystemHandler vs nullptr)
- DataStoreHandler (store_hd)
- LogAgent (log_service)
- Configuration map with ~15 parameters

Configuration Parameters (from both projects):

std::map<std::string, uint32_t> tx_service_conf{
    {"core_num", core_num},
    {"checkpointer_interval", ckpt_interval},
    {"node_memory_limit_mb", node_memory_limit_mb},
    {"node_log_limit_mb", node_log_limit_mb},
    {"checkpointer_delay_seconds", ckpt_delay},
    {"collect_active_tx_ts_interval_seconds", active_tx_interval},
    {"realtime_sampling", realtime_sampling},
    {"rep_group_cnt", replica_num},
    {"enable_key_cache", enable_key_cache},
    {"enable_shard_heap_defragment", enable_heap_defragment},
    {"bthread_worker_num", bthread_worker_num},
    {"max_standby_lag", max_standby_lag},
    {"kickout_data_for_test", kickout_data_for_test}
};

2. Store Handler (store_hd)

Purpose: Data storage abstraction layer
Implementation Types:

EloqDS::DynamoHandler - DynamoDB backend
EloqKV::RocksDBHandlerImpl - RocksDB backend
EloqDS::DataStoreServiceClient - Data store service client
EloqDS::BigTableHandler - BigTable backend

Initialization Dependencies:

Storage-specific configuration (endpoints, credentials, paths)
Bootstrap mode detection
Connection establishment via Connect() method
Critical: Requires SetTxService() call after tx_service creation

3. Log Service (log_agent)

Purpose: Write-ahead logging and replication
Implementation:

MyEloqLogAgent (eloqsql) vs EloqKV::EloqLogAgent (eloqkv)
Both inherit from same base class
Key Dependencies:
- Replica number configuration
- Log service initialization via InitTxLogService()
- RocksDB configuration for log storage

Configuration Pattern Differences

Current eloqsql Configuration System

Format: MySQL system variables
Prefix: eloq_
Examples:
- eloq_local_ip, eloq_core_num, eloq_txlog_service_list
- eloq_enable_tx_metrics, eloq_enable_log_service_metrics
Access: Direct variable access
Data Path: mysql_real_data_home_ptr

Current eloqkv Configuration System

Format: gflags + INI files
Prefix: FLAGS_ for command line, config file sections
Examples:
- FLAGS_txlog_service_list, FLAGS_core_num
- config_reader.GetString("cluster", "txlog_service_list")
Access: Command line flags + INIReader
Data Path: eloq_data_path (configurable)

New Unified Configuration System

Format: Unified config file + gflags
Data Substrate Config: Separate config file with no engine-specific patterns
Engine Integration: Engines pass their specific configurations to data substrate
Multi-Engine Support: Single data substrate instance supports multiple engines

Initialization Sequence Analysis

Current eloqsql Sequence:

Parse MySQL system variables
Initialize storage handler (storage_hd)
Initialize log agent (log_agent)
Create tx_service with all dependencies
Start tx_service
Set tx_service reference in storage handler

Current eloqkv Sequence:

Parse gflags and INI config
Initialize log service (if not standalone)
Initialize storage handler (store_hd_)
Create tx_service with all dependencies
Set tx_service reference in storage handler
Start tx_service

Key Dependencies and Order Requirements

Critical Dependencies:

Storage Handler → TxService: Storage handler needs tx_service reference via SetTxService()
Log Service → TxService: Log agent passed to tx_service constructor
Catalog Factory → Multi-Engine: Array of catalog factories for multiple engines (size 3)
System Handler → Engine: Engine-specific system handlers (wrapped in compile macros)

Multi-Engine Considerations:

Catalog Factory Array: TxService accepts CatalogFactory* catalog_factory[3] for eloqsql, eloqkv, eloqdoc
Compile-Time Engine Selection: Engine-specific code wrapped in #ifdef macros
Bootstrap Mode: Unified bootstrap detection across all engines
Engine Registration: Engines register themselves with data substrate during initialization

Architecture Insights

Shared vs Engine-Specific Components

Shared Components (extracted to data substrate):

TxService core logic
DataStoreHandler implementations
LogAgent implementations
Unified configuration parsing logic
Metrics initialization
Multi-engine catalog factory management

Engine-Specific Components (wrapped in compile macros):

CatalogFactory implementations (#ifdef ELOQSQL MariaCatalogFactory, #ifdef ELOQKV engine-specific, etc.)
SystemHandler implementations (#ifdef ELOQSQL MariaSystemHandler, #ifdef ELOQKV nullptr, etc.)
Engine registration and initialization logic
Engine-specific data path resolution

Configuration Unification Approach

Unified Config File: Single data substrate config file with standardized naming
Gflags Integration: Use gflags for command-line overrides and defaults
Engine-Specific Configs: Engines pass their specific configurations to data substrate
No Engine Dependencies: Data substrate config file contains no engine-specific patterns

Potential Issues with Multi-Engine Data Substrate Initialization

1. Multi-Engine Catalog Factory Management

Issue: TxService needs to support multiple catalog factories simultaneously
Impact: Current implementation expects single catalog factory
Solution: Modify TxService to accept CatalogFactory* catalog_factory[3] array

2. Compile-Time Engine Selection

Issue: Engine-specific code needs to be conditionally compiled
Impact: Data substrate must compile with or without specific engines
Solution: Wrap all engine-specific dependencies in #ifdef ELOQSQL, #ifdef ELOQKV, #ifdef ELOQDOC macros

3. Engine Registration and Initialization Order

Issue: Multiple engines need to register with data substrate in correct order
Impact: Initialization sequence becomes more complex
Solution: Implement engine registration system with dependency management

4. Unified Configuration Management

Issue: Single config file must serve multiple engines without conflicts
Impact: Configuration namespace and parameter management
Solution: Use unified config file with engine-specific parameter passing

5. Resource Sharing and Isolation

Issue: Multiple engines sharing same data substrate resources
Impact: Potential resource conflicts and performance isolation
Solution: Implement resource partitioning and isolation mechanisms

Code References

eloqsql Key Files:

eloqsql/storage/eloq/ha_eloq.cc:2397 - Main initialization function
eloqsql/storage/eloq/ha_eloq.cc:3209,3227 - TxService creation
eloqsql/storage/eloq/ha_eloq.cc:2558,2573,2750 - Storage handler creation
eloqsql/storage/eloq/ha_eloq.cc:3095 - Log agent creation
eloqsql/storage/eloq/ha_eloq.cc:3253 - SetTxService call

eloqkv Key Files:

eloqkv/src/redis_service.cpp - Main initialization logic
eloqkv/src/redis_service.cpp:1437 - TxService creation
eloqkv/src/redis_service.cpp:961,978,1138 - Storage handler creation
eloqkv/src/redis_service.cpp:1374 - Log agent creation
eloqkv/src/redis_service.cpp:1469 - SetTxService call

Shared Components:

eloqkv/tx_service/include/tx_service.h - TxService interface
eloqsql/storage/eloq/tx_service/include/tx_service.h - TxService interface (identical)
eloqkv/tx_service/include/store/data_store_handler.h - Storage handler interface
eloqsql/storage/eloq/tx_service/include/store/data_store_handler.h - Storage handler interface (identical)

Recommendations

1. Create Multi-Engine Data Substrate Initialization Function

class DataSubstrate {
public:
    struct Config {
        // Unified configuration structure (no engine-specific patterns)
        std::string data_path;
        std::map<std::string, uint32_t> tx_service_config;
        std::string txlog_service_list;
        // ... other unified config parameters
    };
    
    struct EngineConfig {
        CatalogFactory* catalog_factory;
        SystemHandler* system_handler;
        std::string engine_name;
        // Engine-specific parameters
    };
    
    static std::unique_ptr<DataSubstrate> Initialize(
        const Config& config,
        const std::vector<EngineConfig>& engines
    );
};

2. Compile-Time Engine Selection

#ifdef ELOQSQL
    #include "eloq_catalog_factory.h"
    #include "eloq_system_handler.h"
#endif

#ifdef ELOQKV
    #include "eloqkv_catalog_factory.h"
#endif

#ifdef ELOQDOC
    #include "eloqdoc_catalog_factory.h"
#endif

3. Multi-Engine TxService Modification

// Modify TxService constructor to accept multiple catalog factories
TxService(
    CatalogFactory* catalog_factory[3],  // Array for eloqsql, eloqkv, eloqdoc
    SystemHandler* system_handlers[3],   // Array for each engine
    const std::map<std::string, uint32_t>& config,
    // ... other parameters
);

4. Hard-Coded Engine Registration System

class DataSubstrate {
private:
    void RegisterEngines() {
        std::vector<EngineConfig> engines;
        
#ifdef ELOQSQL
        // Register eloqsql engine
        EngineConfig eloqsql_config;
        eloqsql_config.catalog_factory = &maria_catalog_factory;
        eloqsql_config.system_handler = &MariaSystemHandler::Instance();
        eloqsql_config.engine_name = "eloqsql";
        engines.push_back(eloqsql_config);
#endif

#ifdef ELOQKV
        // Register eloqkv engine
        EngineConfig eloqkv_config;
        eloqkv_config.catalog_factory = &eloqkv_catalog_factory;
        eloqkv_config.system_handler = nullptr;  // eloqkv doesn't use system handler
        eloqkv_config.engine_name = "eloqkv";
        engines.push_back(eloqkv_config);
#endif

#ifdef ELOQDOC
        // Register eloqdoc engine
        EngineConfig eloqdoc_config;
        eloqdoc_config.catalog_factory = &eloqdoc_catalog_factory;
        eloqdoc_config.system_handler = &EloqDocSystemHandler::Instance();
        eloqdoc_config.engine_name = "eloqdoc";
        engines.push_back(eloqdoc_config);
#endif
        
        // Initialize with registered engines
        InitializeWithEngines(engines);
    }
};

5. Phased Multi-Engine Initialization

Phase 1: Read unified data substrate config file (INI format)
Phase 2: Initialize storage handler and log service
Phase 3: Hard-code engine registration (eloqsql first, then eloqkv)
Phase 4: Create tx_service with multi-engine catalog factory array
Phase 5: Establish cross-references and start services

Open Questions - Resolved

Configuration File Format: ✅ Use INI format - Standardized and easy to parse
Engine Initialization Order: ✅ Order does not matter - Use eloqsql then eloqkv
Resource Isolation: ✅ Managed by data substrate - No need to worry about isolation
Compile Macro Strategy: ✅ Use #ifdef ELOQKV, #ifdef ELOQDOC, #ifdef ELOQSQL - Current approach is correct
Engine Discovery: ✅ Hard code registration in init function - Wrap registration code for each engine with compile-time macros

Implementation Notes

Key Design Decisions

Unified Config + Gflags: Data substrate uses unified config file with gflags for overrides
Multi-Engine Support: Single data substrate instance supports eloqsql, eloqkv, and eloqdoc simultaneously
Compile-Time Selection: Engine-specific code wrapped in #ifdef macros for conditional compilation
No Backward Compatibility: Clean break from existing configuration systems
Manual Testing: Initial testing will be done manually without automated test suite

Configuration File Structure (INI Format)

[data_substrate]
data_path = /path/to/data
core_num = 8
checkpointer_interval = 10
node_memory_limit_mb = 8192
txlog_service_list = 127.0.0.1:8001,127.0.0.1:8002,127.0.0.1:8003
enable_tx_metrics = true
enable_log_service_metrics = true

[storage]
storage_type = rocksdb
rocksdb_storage_path = /path/to/rocksdb
enable_cache_replacement = true

[log_service]
log_service_data_path = /path/to/log_service
rocksdb_scan_threads = 4

Engine Registration Order

eloqsql - First engine to register
eloqkv - Second engine to register
eloqdoc - Third engine (if compiled in)

Related Research

Original ticket: GitHub Issue #128
Multi-engine architecture enables converged database with shared data substrate

Data Substrate Initialization Implementation Plan

Overview

This plan implements a unified data substrate initialization system that supports multiple database engines (eloqsql, eloqkv, eloqdoc) simultaneously. The data substrate will use a unified INI configuration file with gflags support, and all engine-specific code will be wrapped in compile-time macros.

Current State Analysis

What exists now:

Separate initialization functions in eloqsql/storage/eloq/ha_eloq.cc:2397 and eloqkv/src/redis_service.cpp
Duplicate data substrate components (tx_service, store_hd, log_service) in both projects
Different configuration systems (MySQL variables vs gflags+INI)
Engine-specific catalog factories and system handlers

What's missing:

Unified data substrate initialization function
Multi-engine catalog factory support in TxService
Unified configuration system
Compile-time engine selection mechanism

Desired End State

After this plan is complete:

Single DataSubstrate class that can be initialized with multiple engines
Unified INI configuration file with no engine-specific patterns
TxService modified to accept catalog factory array for multiple engines
All engine-specific code wrapped in #ifdef ELOQSQL, #ifdef ELOQKV, #ifdef ELOQDOC macros
Modular initialization with separate helper functions for each component

Key Discoveries:

Both projects use identical data substrate components but with different initialization patterns
TxService currently expects single catalog factory but needs to support array of 3 engines
Configuration systems are fundamentally different and need unification
Engine-specific dependencies (CatalogFactory, SystemHandler) must be passed as parameters

What We're NOT Doing

Maintaining backward compatibility with existing configuration systems
Implementing automated testing (manual testing only)
Adding resource isolation mechanisms (managed by data substrate)
Supporting dynamic engine discovery (hard-coded registration only)

Implementation Approach

Create a modular DataSubstrate class with separate helper functions for each component initialization. All helper functions will be implemented in the same data_substrate.cpp file to avoid file fragmentation while maintaining modularity. Use compile-time macros for engine-specific code and unified INI configuration.

File Structure

data_substrate/
├── include/
│   └── data_substrate.h          # Main class definition
├── src/
│   └── data_substrate.cpp        # All initialization functions
└── config/
    └── sample_data_substrate.ini # Sample configuration

Note: All helper functions (configuration loading, storage handler init, log service init, engine registration, TxService init, service startup) are implemented as separate functions within the same data_substrate.cpp file to maintain modularity while reducing file fragmentation.

Phase 1: Create Data Substrate Foundation

Overview

Create the core DataSubstrate class structure and configuration system.

Changes Required:

1. Create Data Substrate Header

File: data_substrate/include/data_substrate.h
Changes: Create main DataSubstrate class with configuration structures

#pragma once

#include <memory>
#include <string>
#include <vector>
#include <map>
#include <INIReader.h>

// Forward declarations
namespace txservice {
    class TxService;
    class DataStoreHandler;
    class LogAgent;
}

namespace txlog {
    class LogServer;
}

class DataSubstrate {
public:
    struct Config {
        // Unified configuration structure (no engine-specific patterns)
        std::string data_path;
        std::string txlog_service_list;
        std::string storage_type;
        std::string rocksdb_storage_path;
        std::string log_service_data_path;
        
        // TxService configuration
        std::map<std::string, uint32_t> tx_service_config;
        
        // Storage configuration
        bool enable_cache_replacement;
        
        // Log service configuration
        uint32_t rocksdb_scan_threads;
        uint32_t txlog_group_replica_num;
        
        // Metrics configuration
        bool enable_tx_metrics;
        bool enable_log_service_metrics;
    };
    
    struct EngineConfig {
        class CatalogFactory* catalog_factory;
        class SystemHandler* system_handler;
        std::string engine_name;
    };
    
    static std::unique_ptr<DataSubstrate> Initialize(
        const std::string& config_file_path
    );
    
    // Component accessors
    txservice::TxService* GetTxService() const { return tx_service_.get(); }
    txservice::DataStoreHandler* GetStoreHandler() const { return store_hd_.get(); }
    txlog::LogServer* GetLogServer() const { return log_server_.get(); }
    
    ~DataSubstrate();

private:
    DataSubstrate() = default;
    
    // Helper functions for component initialization
    bool LoadConfiguration(const std::string& config_file_path);
    bool InitializeStorageHandler();
    bool InitializeLogService();
    bool RegisterEngines();
    bool InitializeTxService();
    bool StartServices();
    
    // Component instances
    std::unique_ptr<txservice::TxService> tx_service_;
    std::unique_ptr<txservice::DataStoreHandler> store_hd_;
    std::unique_ptr<txlog::LogServer> log_server_;
    std::unique_ptr<txservice::LogAgent> log_agent_;
    
    // Configuration
    Config config_;
    std::vector<EngineConfig> engines_;
    
    // Engine-specific instances (wrapped in compile macros)
#ifdef ELOQSQL
    static MariaCatalogFactory maria_catalog_factory_;
#endif
#ifdef ELOQKV
    static EloqKvCatalogFactory eloqkv_catalog_factory_;
#endif
#ifdef ELOQDOC
    static EloqDocCatalogFactory eloqdoc_catalog_factory_;
#endif
};

2. Create Configuration Loading Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement configuration loading from INI file in the same file as main class

#include "data_substrate.h"
#include <gflags/gflags.h>
#include <INIReader.h>

bool DataSubstrate::LoadConfiguration(const std::string& config_file_path) {
    INIReader config_reader(config_file_path);
    
    if (config_reader.ParseError() < 0) {
        return false;
    }
    
    // Load data substrate configuration
    config_.data_path = config_reader.Get("data_substrate", "data_path", "");
    config_.txlog_service_list = config_reader.Get("data_substrate", "txlog_service_list", "");
    config_.enable_tx_metrics = config_reader.GetBoolean("data_substrate", "enable_tx_metrics", true);
    config_.enable_log_service_metrics = config_reader.GetBoolean("data_substrate", "enable_log_service_metrics", true);
    
    // Load storage configuration
    config_.storage_type = config_reader.Get("storage", "storage_type", "rocksdb");
    config_.rocksdb_storage_path = config_reader.Get("storage", "rocksdb_storage_path", "");
    config_.enable_cache_replacement = config_reader.GetBoolean("storage", "enable_cache_replacement", true);
    
    // Load log service configuration
    config_.log_service_data_path = config_reader.Get("log_service", "log_service_data_path", "");
    config_.rocksdb_scan_threads = config_reader.GetInteger("log_service", "rocksdb_scan_threads", 4);
    config_.txlog_group_replica_num = config_reader.GetInteger("log_service", "txlog_group_replica_num", 3);
    
    // Load TxService configuration
    config_.tx_service_config["core_num"] = config_reader.GetInteger("data_substrate", "core_num", 8);
    config_.tx_service_config["checkpointer_interval"] = config_reader.GetInteger("data_substrate", "checkpointer_interval", 10);
    config_.tx_service_config["node_memory_limit_mb"] = config_reader.GetInteger("data_substrate", "node_memory_limit_mb", 8192);
    // ... other tx_service parameters
    
    return true;
}

Success Criteria:

Automated Verification:

DataSubstrate class compiles successfully
Configuration loading works with sample INI file
All helper function declarations are present
Compile macros are properly defined

Manual Verification:

Configuration file can be parsed without errors
Default values are applied correctly
gflags integration works for command-line overrides

Phase 2: Implement Storage Handler Initialization

Overview

Create helper function to initialize storage handler with unified configuration.

Changes Required:

1. Create Storage Handler Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement storage handler initialization logic in the same file

#include "data_substrate.h"
#include "store_handler/rocksdb_handler.h"
#include "store_handler/dynamo_handler.h"
#include "store_handler/data_store_service_client.h"

bool DataSubstrate::InitializeStorageHandler() {
    if (config_.storage_type == "rocksdb") {
        // Initialize RocksDB handler
        EloqShare::RocksDBConfig rocksdb_config;
        rocksdb_config.storage_path = config_.rocksdb_storage_path;
        rocksdb_config.enable_cache_replacement = config_.enable_cache_replacement;
        
        store_hd_ = std::make_unique<EloqKV::RocksDBHandlerImpl>(
            rocksdb_config,
            false, // bootstrap - will be determined by engine
            config_.enable_cache_replacement
        );
    }
    else if (config_.storage_type == "dynamodb") {
        // Initialize DynamoDB handler
        std::string endpoint = config_reader.Get("storage", "dynamodb_endpoint", "");
        std::string region = config_reader.Get("storage", "dynamodb_region", "");
        std::string access_key = config_reader.Get("storage", "aws_access_key_id", "");
        std::string secret_key = config_reader.Get("storage", "aws_secret_key", "");
        std::string keyspace = config_reader.Get("storage", "keyspace_name", "eloq_data");
        
        store_hd_ = std::make_unique<EloqDS::DynamoHandler>(
            keyspace, endpoint, region, access_key, secret_key,
            false, // bootstrap
            false, // ddl_skip_kv
            config_.tx_service_config["core_num"] * 2
        );
    }
    else if (config_.storage_type == "data_store_service") {
        // Initialize Data Store Service client
        // Implementation for DSS client
    }
    
    if (!store_hd_) {
        return false;
    }
    
    return store_hd_->Connect();
}

Success Criteria:

Automated Verification:

Storage handler initialization compiles successfully
All storage types (rocksdb, dynamodb, dss) are supported
Connection establishment is properly handled

Manual Verification:

RocksDB storage handler connects successfully
DynamoDB storage handler connects successfully (if configured)
Error handling works for invalid configurations

Phase 3: Implement Log Service Initialization

Overview

Create helper function to initialize log service with unified configuration.

Changes Required:

1. Create Log Service Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement log service initialization logic in the same file

#include "data_substrate.h"
#include "log_service/log_server.h"
#include "log_service/log_agent.h"

bool DataSubstrate::InitializeLogService() {
    // Parse txlog service list
    std::vector<std::string> txlog_ips;
    std::vector<uint16_t> txlog_ports;
    
    if (!config_.txlog_service_list.empty()) {
        std::string token;
        std::istringstream txlog_ip_list_stream(config_.txlog_service_list);
        while (std::getline(txlog_ip_list_stream, token, ',')) {
            size_t c_idx = token.find_first_of(':');
            if (c_idx != std::string::npos) {
                txlog_ips.emplace_back(token.substr(0, c_idx));
                txlog_ports.emplace_back(std::stoi(token.substr(c_idx + 1)));
            }
        }
    }
    
    // Create log agent
    log_agent_ = std::make_unique<txservice::LogAgent>(
        static_cast<uint32_t>(config_.txlog_group_replica_num)
    );
    
    // Initialize log server if no standalone service provided
    if (txlog_ips.empty()) {
        // Initialize local log server
        std::string log_path = config_.data_path + "/log_service";
        
        log_server_ = std::make_unique<txlog::LogServer>(
            0, // node_id - will be set by engine
            0, // port - will be set by engine
            log_path,
            config_.rocksdb_scan_threads
        );
        
        if (!log_server_->Start()) {
            return false;
        }
    }
    
    return true;
}

Success Criteria:

Automated Verification:

Log service initialization compiles successfully
Log agent creation works correctly
Log server initialization handles both standalone and local modes

Manual Verification:

Local log server starts successfully
Standalone log service connection works
Log agent is properly configured

Phase 4: Implement Engine Registration System

Overview

Create helper function to register engines with compile-time macro support.

Changes Required:

1. Create Engine Registration Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement hard-coded engine registration in the same file

#include "data_substrate.h"

#ifdef ELOQSQL
#include "eloq_catalog_factory.h"
#include "eloq_system_handler.h"
#endif

#ifdef ELOQKV
#include "eloqkv_catalog_factory.h"
#endif

#ifdef ELOQDOC
#include "eloqdoc_catalog_factory.h"
#include "eloqdoc_system_handler.h"
#endif

bool DataSubstrate::RegisterEngines() {
    engines_.clear();
    
#ifdef ELOQSQL
    // Register eloqsql engine
    EngineConfig eloqsql_config;
    eloqsql_config.catalog_factory = &maria_catalog_factory_;
    eloqsql_config.system_handler = &MariaSystemHandler::Instance();
    eloqsql_config.engine_name = "eloqsql";
    engines_.push_back(eloqsql_config);
#endif

#ifdef ELOQKV
    // Register eloqkv engine
    EngineConfig eloqkv_config;
    eloqkv_config.catalog_factory = &eloqkv_catalog_factory_;
    eloqkv_config.system_handler = nullptr;  // eloqkv doesn't use system handler
    eloqkv_config.engine_name = "eloqkv";
    engines_.push_back(eloqkv_config);
#endif

#ifdef ELOQDOC
    // Register eloqdoc engine
    EngineConfig eloqdoc_config;
    eloqdoc_config.catalog_factory = &eloqdoc_catalog_factory_;
    eloqdoc_config.system_handler = &EloqDocSystemHandler::Instance();
    eloqdoc_config.engine_name = "eloqdoc";
    engines_.push_back(eloqdoc_config);
#endif
    
    return !engines_.empty();
}

Success Criteria:

Automated Verification:

Engine registration compiles with different macro combinations
All three engines can be registered when macros are enabled
Registration works with single engine enabled

Manual Verification:

eloqsql engine registers correctly when ELOQSQL macro is defined
eloqkv engine registers correctly when ELOQKV macro is defined
eloqdoc engine registers correctly when ELOQDOC macro is defined

Phase 5: Modify TxService for Multi-Engine Support

Overview

Modify TxService to accept catalog factory array and system handler array for multiple engines.

Changes Required:

1. Update TxService Constructor

File: tx_service/include/tx_service.h
Changes: Modify constructor to accept arrays for multiple engines

class TxService {
public:
    // Modified constructor for multi-engine support
    TxService(
        CatalogFactory* catalog_factory[3],  // Array for eloqsql, eloqkv, eloqdoc
        SystemHandler* system_handlers[3],   // Array for each engine
        const std::map<std::string, uint32_t>& config,
        uint32_t node_id,
        uint32_t native_ng_id,
        const std::unordered_map<uint32_t, std::vector<NodeConfig>>* ng_configs,
        uint32_t cluster_config_version,
        DataStoreHandler* store_hd,
        LogAgent* log_agent,
        bool enable_mvcc,
        bool skip_redo_log,
        bool skip_kv,
        bool enable_cache_replacement,
        bool auto_redirect,
        MetricsRegistry* metrics_registry,
        const CommonLabels& common_labels
    );
    
private:
    CatalogFactory* catalog_factories_[3];
    SystemHandler* system_handlers_[3];
    // ... other members
};

2. Update TxService Implementation

File: tx_service/src/tx_service.cpp
Changes: Update constructor implementation to handle multiple engines

TxService::TxService(
    CatalogFactory* catalog_factory[3],
    SystemHandler* system_handlers[3],
    const std::map<std::string, uint32_t>& config,
    // ... other parameters
) {
    // Copy catalog factories
    for (int i = 0; i < 3; i++) {
        catalog_factories_[i] = catalog_factory[i];
        system_handlers_[i] = system_handlers[i];
    }
    
    // Initialize with primary catalog factory (eloqsql)
    catalog_factory_ = catalog_factories_[0];  // eloqsql
    system_handler_ = system_handlers_[0];     // eloqsql
    
    // ... rest of initialization
}

Success Criteria:

Automated Verification:

TxService compiles with new constructor signature
All existing TxService functionality remains intact
Multi-engine catalog factory array is properly handled

Manual Verification:

TxService can be created with multiple catalog factories
Primary engine (eloqsql) is used for main operations
Other engines are accessible when needed

Phase 6: Implement TxService Initialization Helper

Overview

Create helper function to initialize TxService with all registered engines.

Changes Required:

1. Create TxService Initialization Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement TxService initialization with multi-engine support in the same file

#include "data_substrate.h"
#include "tx_service/tx_service.h"

bool DataSubstrate::InitializeTxService() {
    if (engines_.empty()) {
        return false;
    }
    
    // Prepare catalog factory and system handler arrays
    CatalogFactory* catalog_factories[3] = {nullptr, nullptr, nullptr};
    SystemHandler* system_handlers[3] = {nullptr, nullptr, nullptr};
    
    // Fill arrays with registered engines
    for (size_t i = 0; i < engines_.size() && i < 3; i++) {
        catalog_factories[i] = engines_[i].catalog_factory;
        system_handlers[i] = engines_[i].system_handler;
    }
    
    // Create TxService with multi-engine support
    tx_service_ = std::make_unique<txservice::TxService>(
        catalog_factories,
        system_handlers,
        config_.tx_service_config,
        0, // node_id - will be set by engine
        0, // native_ng_id - will be set by engine
        nullptr, // ng_configs - will be set by engine
        0, // cluster_config_version - will be set by engine
        store_hd_.get(),
        log_agent_.get(),
        false, // enable_mvcc
        false, // skip_redo_log
        false, // skip_kv
        config_.enable_cache_replacement,
        true, // auto_redirect
        nullptr, // metrics_registry - will be set by engine
        {} // common_labels - will be set by engine
    );
    
    if (!tx_service_) {
        return false;
    }
    
    // Set tx_service reference in storage handler
    if (store_hd_) {
        store_hd_->SetTxService(tx_service_.get());
    }
    
    return true;
}

Success Criteria:

Automated Verification:

TxService initialization compiles successfully
Multi-engine arrays are properly populated
Storage handler reference is correctly set

Manual Verification:

TxService is created with all registered engines
Storage handler can access TxService
All engines are properly initialized

Phase 7: Implement Service Startup Helper

Overview

Create helper function to start all services in correct order.

Changes Required:

1. Create Service Startup Helper

File: data_substrate/src/data_substrate.cpp
Changes: Implement service startup sequence in the same file

#include "data_substrate.h"

bool DataSubstrate::StartServices() {
    if (!tx_service_) {
        return false;
    }
    
    // Start TxService
    // Note: Engine-specific parameters will be provided by engines
    std::vector<std::string> txlog_ips;
    std::vector<uint16_t> txlog_ports;
    
    // Parse txlog service list for startup
    if (!config_.txlog_service_list.empty()) {
        std::string token;
        std::istringstream txlog_ip_list_stream(config_.txlog_service_list);
        while (std::getline(txlog_ip_list_stream, token, ',')) {
            size_t c_idx = token.find_first_of(':');
            if (c_idx != std::string::npos) {
                txlog_ips.emplace_back(token.substr(0, c_idx));
                txlog_ports.emplace_back(std::stoi(token.substr(c_idx + 1)));
            }
        }
    }
    
    // Start TxService (engine-specific parameters will be provided by engines)
    int start_result = tx_service_->Start(
        0, // node_id - will be set by engine
        0, // native_ng_id - will be set by engine
        nullptr, // ng_configs - will be set by engine
        0, // cluster_config_version - will be set by engine
        &txlog_ips,
        &txlog_ports,
        nullptr, // hm_ip - will be set by engine
        nullptr, // hm_port - will be set by engine
        nullptr, // hm_bin_path - will be set by engine
        config_.tx_service_config,
        std::move(log_agent_),
        config_.data_path,
        "" // cluster_config_file - will be set by engine
    );
    
    return start_result >= 0;
}

Success Criteria:

Automated Verification:

Service startup compiles successfully
TxService start method is called correctly
Error handling is properly implemented

Manual Verification:

TxService starts successfully
All services are running after startup
Error conditions are handled gracefully

Phase 8: Implement Main DataSubstrate Initialization

Overview

Create the main initialization function that orchestrates all helper functions.

Changes Required:

1. Create Main Initialization Function

File: data_substrate/src/data_substrate.cpp
Changes: Implement main initialization orchestration

#include "data_substrate.h"

std::unique_ptr<DataSubstrate> DataSubstrate::Initialize(
    const std::string& config_file_path
) {
    auto data_substrate = std::make_unique<DataSubstrate>();
    
    // Phase 1: Load configuration
    if (!data_substrate->LoadConfiguration(config_file_path)) {
        return nullptr;
    }
    
    // Phase 2: Initialize storage handler
    if (!data_substrate->InitializeStorageHandler()) {
        return nullptr;
    }
    
    // Phase 3: Initialize log service
    if (!data_substrate->InitializeLogService()) {
        return nullptr;
    }
    
    // Phase 4: Register engines
    if (!data_substrate->RegisterEngines()) {
        return nullptr;
    }
    
    // Phase 5: Initialize TxService
    if (!data_substrate->InitializeTxService()) {
        return nullptr;
    }
    
    // Phase 6: Start services
    if (!data_substrate->StartServices()) {
        return nullptr;
    }
    
    return data_substrate;
}

DataSubstrate::~DataSubstrate() {
    if (tx_service_) {
        tx_service_->Shutdown();
    }
    
    // Cleanup other components
    store_hd_ = nullptr;
    log_server_ = nullptr;
    log_agent_ = nullptr;
}

Success Criteria:

Automated Verification:

Main initialization function compiles successfully
All helper functions are called in correct order
Error handling prevents partial initialization

Manual Verification:

DataSubstrate initializes successfully with valid config
All components are properly initialized
Cleanup works correctly on destruction

Phase 9: Create Sample Configuration File

Overview

Create sample configuration file for testing and documentation.

Changes Required:

1. Create Sample Configuration

File: data_substrate/config/sample_data_substrate.ini
Changes: Create comprehensive sample configuration

[data_substrate]
data_path = /tmp/eloq_data
core_num = 8
checkpointer_interval = 10
node_memory_limit_mb = 8192
node_log_limit_mb = 1024
checkpointer_delay_seconds = 5
collect_active_tx_ts_interval_seconds = 2
realtime_sampling = true
txlog_service_list = 127.0.0.1:8001,127.0.0.1:8002,127.0.0.1:8003
txlog_group_replica_num = 3
enable_tx_metrics = true
enable_log_service_metrics = true
enable_cache_hit_rate = true
enable_kv_metrics = true
enable_remote_request_metrics = true
collect_tx_duration_round = 1
busy_round_threshold = 10

[storage]
storage_type = rocksdb
rocksdb_storage_path = /tmp/eloq_data/rocksdb
enable_cache_replacement = true
rocksdb_max_write_buffer_number = 3
rocksdb_max_background_jobs = 4
rocksdb_target_file_size_base = 67108864

[log_service]
log_service_data_path = /tmp/eloq_data/log_service
rocksdb_scan_threads = 4
logserver_snapshot_interval = 3600
enable_txlog_request_checkpoint = true
notify_checkpointer_threshold_size = 1GB

[metrics]
enable_metrics = true
metrics_port = 9090

Success Criteria:

Automated Verification:

Sample configuration file is valid INI format
All required parameters are included
Default values are reasonable

Manual Verification:

Configuration file can be loaded by DataSubstrate
All parameters are properly parsed
Default values work correctly

Testing Strategy

Unit Tests:

Configuration loading with various INI file formats
Storage handler initialization for different storage types
Log service initialization with and without standalone service
Engine registration with different macro combinations
TxService initialization with multiple engines

Integration Tests:

End-to-end DataSubstrate initialization
Multi-engine TxService operation
Storage handler and TxService interaction
Log service and TxService interaction

Manual Testing Steps:

Create sample configuration file
Initialize DataSubstrate with eloqsql only
Initialize DataSubstrate with eloqkv only
Initialize DataSubstrate with both eloqsql and eloqkv
Verify all components are accessible
Test service startup and shutdown
Verify error handling with invalid configurations

Performance Considerations

Configuration loading should be fast (single INI file read)
Component initialization should be parallel where possible
Memory usage should be minimal for unused engines
Startup time should be comparable to current implementations

Migration Notes

No backward compatibility required
Existing engines will need to be modified to use new DataSubstrate
Configuration files will need to be converted to new INI format
Engine-specific initialization code will be moved to compile macros

References

Original ticket: thoughts/shared/research/2025-09-25-ENG-128-data-substrate-initialization.md
eloqsql initialization: eloqsql/storage/eloq/ha_eloq.cc:2397
eloqkv initialization: eloqkv/src/redis_service.cpp
TxService interface: tx_service/include/tx_service.h
Storage handler interface: tx_service/include/store/data_store_handler.h

Init function has been refactored into a separate data substrate init.
Eloqkv and eloqsql can be built with the new data substrate.
ConvergeDB can build and run eloqkv and eloqsql together.

In progress:
clean up eloqkv code and merge into main. Maintain backward compatibility and update build script/doc.
TODO:
same for eloqsql
Refactor code to clarify start up process in convergedb

Refactor eloq substrate init function #128

Description

Activity

liunyl commented on Sep 25, 2025

Research: Data Substrate Initialization Analysis for Converged Database

Research Question

Summary

Detailed Findings

Current Initialization Functions

eloqsql Initialization (ha_eloq.cc:2397)

eloqkv Initialization (redis_service.cpp)

Data Substrate Components Analysis

1. TxService

2. Store Handler (store_hd)

3. Log Service (log_agent)

Configuration Pattern Differences

Current eloqsql Configuration System

Current eloqkv Configuration System

New Unified Configuration System

Initialization Sequence Analysis

Current eloqsql Sequence:

Current eloqkv Sequence:

Key Dependencies and Order Requirements

Critical Dependencies:

Multi-Engine Considerations:

Architecture Insights

Shared vs Engine-Specific Components

Configuration Unification Approach

Potential Issues with Multi-Engine Data Substrate Initialization

1. Multi-Engine Catalog Factory Management

2. Compile-Time Engine Selection

3. Engine Registration and Initialization Order

4. Unified Configuration Management

5. Resource Sharing and Isolation

Code References

eloqsql Key Files:

eloqkv Key Files:

Shared Components:

Recommendations

1. Create Multi-Engine Data Substrate Initialization Function

2. Compile-Time Engine Selection

3. Multi-Engine TxService Modification

4. Hard-Coded Engine Registration System

5. Phased Multi-Engine Initialization

Open Questions - Resolved

Implementation Notes

Key Design Decisions

Configuration File Structure (INI Format)

Engine Registration Order

Related Research

liunyl commented on Sep 25, 2025

Data Substrate Initialization Implementation Plan

Overview

Current State Analysis

Desired End State

Key Discoveries:

What We're NOT Doing

Implementation Approach

File Structure

Phase 1: Create Data Substrate Foundation

Overview

Changes Required:

1. Create Data Substrate Header

2. Create Configuration Loading Helper

Success Criteria:

Automated Verification:

Manual Verification:

Phase 2: Implement Storage Handler Initialization

Overview

Changes Required:

1. Create Storage Handler Helper

Success Criteria:

Automated Verification:

Manual Verification:

Phase 3: Implement Log Service Initialization

Overview

Changes Required:

1. Create Log Service Helper

Success Criteria:

Automated Verification:

eloqsql Initialization (`ha_eloq.cc:2397`)

eloqkv Initialization (`redis_service.cpp`)