Skip to content

[Feature]: For range partitioning, all keys in a partition are sharded onto the same ccshard. #396

Closed
5 of 5 issues completed
Closed
[Feature]: For range partitioning, all keys in a partition are sharded onto the same ccshard.#396
5 of 5 issues completed
Assignees
yangsw26
Labels
featureAdd new feature

Description

@yangsw26
yangsw26
opened on Feb 3, 2026

Currently, for range partitioning, keys within the same partition are evenly distributed across all ccshards. To shard all these keys onto the same ccshard, several issues need to be considered, including but not limited to the following:

  1. When to split a range.
    To avoid single-core bottlenecks, ranges need to be split promptly, rather than waiting for the checkpoint process to trigger a range split.
    [Possible solution]: Track the range size during the write transaction commit process, and trigger range splitting if the threshold is reached.

  2. Adjust the algorithm of the key shard code so that all keys in the same range are sharded onto the same core.

  3. Modify the execution logic of ScanSliceCc to remove result processing operations for multi-core parallel scanning mode, such as alignment of end keys and merging of scan results.

  4. Modify the execution logic of DataSync to remove processing operations on scan results in multi-core mode, such as merge operations.

  5. Modify the backfill logic in the LoadSlice operation.

  6. [Advance] During range splitting, to avoid a decrease in cache hit rate, cross-core data migration may be required in memory (currently, only cross-node memory data migration is possible).

Activity

  1. yangsw26
    self-assigned this
    on Feb 3, 2026
  2. yangsw26
    added
    featureAdd new feature
    on Feb 3, 2026
  3. yangsw26
    added sub-issues on Feb 11, 2026

  4. yangsw26 commented on Feb 11, 2026

    @yangsw26
    yangsw26
    on Feb 11, 2026
    CollaboratorAuthor
    • 1. Range size tracking.
    • 2. Construct a high-priority DataSync task for range splitting.
    • 3. Adapt the range key sharding to the new mode on the same core for the relevant requests.
    • 4. [Optional][Low priority] During the range splitting process, the keys of the new range are sent to the corresponding core.

  5. yangsw26 commented on Mar 9, 2026

    @yangsw26
    yangsw26
    on Mar 9, 2026
    CollaboratorAuthor

    Perform regression testing and fix bugs.

  6. yangsw26 commented on Mar 12, 2026

    @yangsw26
    yangsw26
    on Mar 12, 2026
    CollaboratorAuthor

    EloqDoc test passed.

  7. yangsw26 commented on Mar 12, 2026

    @yangsw26
    yangsw26
    on Mar 12, 2026
    CollaboratorAuthor

    In EloqSQL testing, the explain results showed mismatch.

    explain select fld3 from t2 ignore index (fld3) where fld3 = 'honeysuckle';
 id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
-1	SIMPLE	t2	ALL	NULL	NULL	NULL	NULL	1199	Using where
+1	SIMPLE	t2	ALL	NULL	NULL	NULL	NULL	1198	Using where
 explain select fld3 from t2 use index (fld1) where fld3 = 'honeysuckle';
 id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
-1	SIMPLE	t2	ALL	NULL	NULL	NULL	NULL	1199	Using where
+1	SIMPLE	t2	ALL	NULL	NULL	NULL	NULL	1198	Using where

  8. yangsw26 commented on Mar 12, 2026

    @yangsw26
    yangsw26
    on Mar 12, 2026
    CollaboratorAuthor
    [time 2026-03-12T11:29:06.081915] [level INFO] [thread 60862] [prewarm_task.cpp:167] Prewarm downloading: __sequence_table.335/data_0_0                     [time 2026-03-12T11:29:06.082566] [level INFO] [thread 60863] [data_store_service_config.cpp:923] SwitchToReadOnly, shard 0 status: ^B
[time 2026-03-12T11:29:06.085253] [level INFO] [thread 60863] [data_store_service_config.cpp:873] SwitchToClosed, shard 0 status: ^A                        [time 2026-03-12T11:29:06.085261] [level INFO] [thread 60862] [prewarm_task.cpp:167] Prewarm downloading: __sequence_table.561/data_1_0
[time 2026-03-12T11:29:06.088291] [level INFO] [thread 60862] [prewarm_task.cpp:92] Shard 1 prewarm aborted due to system shutdown. Pulled 241 files in 14.728s
[time 2026-03-12T11:29:06.091207] [level INFO] [thread 60863] [async_io_manager.cpp:1983] AppendManifest tbl=table_catalogs.0 offset=5267456 bytes=8192 payload=7856 root=1 ttl_root=4294967295 checksum=15643340416640275655 record size=7876
[time 2026-03-12T11:29:08.451525] [level INFO] [thread 60863] [eloq_store.cpp:1069] EloqStore is stopped.
[time 2026-03-12T11:29:08.455888] [level INFO] [thread 60863] [data_store_service.cpp:2197] SwitchReadOnlyToClosed success for DSS shard 0, shard_id_: 0, shard_status_: ^C
[time 2026-03-12T11:29:08.456146] [level INFO] [thread 60863] [data_store_service.cpp:2228] OpenDataStore with term 0 for single shard
[time 2026-03-12T11:29:08.456154] [level INFO] [thread 60863] [data_store_service.cpp:396] Connecting and starting data store for shard id:0, open_mode:2, create_db_if_missing:0, data_store_ is null:0
[time 2026-03-12T11:29:08.456163] [level INFO] [thread 60863] [eloq_store.cpp:214] ===Start eloqstore, term: 0
[time 2026-03-12T11:29:08.456459] [level INFO] [thread 60863] [eloq_store.cpp:439] Store path /tmp/build/de54191f/eloqsql_src/bld/mysql-test/var/mysqld.1/data//eloq_dss/eloqstore_data weight slots: 594087
[time 2026-03-12T11:29:08.456481] [level INFO] [thread 60863] [eloq_store.cpp:449] Constructed store_path LUT with 1 entries
[time 2026-03-12T11:29:08.462677] [level INFO] [thread 60863] [cloud_backend.cpp:1187] cloud_secret_key is set; disabling auto credentials
260312 11:29:08 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.6.10-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=0
max_threads=65537
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 25171106 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(my_print_stacktrace+0x44)[0x5c7db7d8f48e]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(handle_fatal_signal+0x45a)[0x5c7db724befa]
[time 2026-03-12T11:29:11.339188] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1873] Total memtables size: 0 MB
[time 2026-03-12T11:29:11.339268] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1874] Total SSTables size used in replay: 0 MB
[time 2026-03-12T11:29:11.339278] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1878] Total replay log size: 0 MB
??:0(__sigaction)[0x7ff51c39e520]
??:0(mi_malloc)[0x7ff51f1147aa]
??:0(aws_mem_acquire)[0x7ff51c25474c]
endpoints_ruleset.c:0(s_on_rule_element)[0x7ff51bd7cb00]
??:0(aws_json_const_iterate_array)[0x7ff51c26f6c6]
endpoints_ruleset.c:0(s_on_rule_element)[0x7ff51bd7c9a7]
??:0(aws_json_const_iterate_array)[0x7ff51c26f6c6]
endpoints_ruleset.c:0(s_on_rule_element)[0x7ff51bd7c9a7]
??:0(aws_json_const_iterate_array)[0x7ff51c26f6c6]
endpoints_ruleset.c:0(s_on_rule_element)[0x7ff51bd7c9a7]
??:0(aws_json_const_iterate_array)[0x7ff51c26f6c6]
endpoints_ruleset.c:0(s_on_rule_element)[0x7ff51bd7c9a7]                                                                                                    ??:0(aws_json_const_iterate_array)[0x7ff51c26f6c6]
??:0(aws_endpoints_ruleset_new_from_string)[0x7ff51bd7e14a]                                                                                                 ??:0(Aws::Crt::Endpoints::RuleEngine::RuleEngine(aws_byte_cursor const&, aws_byte_cursor const&, aws_allocator*))[0x7ff51c2fa8f0]                           
??:0(Aws::Endpoint::DefaultEndpointProvider<Aws::S3::S3ClientConfiguration, Aws::S3::Endpoint::S3BuiltInParameters, Aws::S3::Endpoint::S3ClientContextParameters>::DefaultEndpointProvider(char const*, unsigned long))[0x7ff51ee69219]
??:0(Aws::S3::S3Client::S3Client(std::shared_ptr<Aws::Auth::AWSCredentialsProvider> const&, Aws::Client::ClientConfiguration const&, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy, bool, Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION))[0x7ff51ee4f709]                                                            /home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZSt11make_uniqueIN3Aws2S38S3ClientEJRSt10shared_ptrINS0_4Auth22AWSCredentialsProviderEERKNS0_6Client19ClientConfigurationENS8_15AWSAuthV4Signer20PayloadSigningPolicyEbEENSt9_MakeUniqIT_E15__single_objectEDpOT0_+0x8d)[0x5c7db8893617]
bits/unique_ptr.h:962(std::_MakeUniq<Aws::S3::S3Client>::__single_object std::make_unique<Aws::S3::S3Client, std::shared_ptr<Aws::Auth::AWSCredentialsProvider>&, Aws::Client::ClientConfiguration const&, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy, bool>(std::shared_ptr<Aws::Auth::AWSCredentialsProvider>&, Aws::Client::ClientConfiguration const&, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy&&, bool&&))[0x5c7db888d6b1]                                   storage/cloud_backend.cpp:982(eloqstore::(anonymous namespace)::AwsCloudBackend::AwsCloudBackend(eloqstore::KvOptions const*, eloqstore::CloudPathInfo))[0x5c7db8891231]                                                                                                                                                bits/unique_ptr.h:962(std::_MakeUniq<eloqstore::(anonymous namespace)::AwsCloudBackend>::__single_object std::make_unique<eloqstore::(anonymous namespace)::AwsCloudBackend, eloqstore::KvOptions const*&, eloqstore::CloudPathInfo const&>(eloqstore::KvOptions const*&, eloqstore::CloudPathInfo const&))[0x5c7db8890994]                                                                                                                                                         storage/cloud_backend.cpp:1557(eloqstore::CreateBackend(eloqstore::KvOptions const*, eloqstore::CloudPathInfo const&))[0x5c7db8877505]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZSt11make_uniqueIN9eloqstore16AsyncHttpManagerEJRPKNS0_9KvOptionsERPNS0_19CloudStorageServiceEEENSt9_MakeUniqIT_E15__single_objectEDpOT0_+0x55)[0x5c7db887c827]
storage/object_store.cpp:207(eloqstore::AsyncHttpManager::AsyncHttpManager(eloqstore::KvOptions const*, eloqstore::CloudStorageService*))[0x5c7db8876cfe]
bits/unique_ptr.h:962(std::_MakeUniq<eloqstore::AsyncHttpManager>::__single_object std::make_unique<eloqstore::AsyncHttpManager, eloqstore::KvOptions const*&, eloqstore::CloudStorageService*&>(eloqstore::KvOptions const*&, eloqstore::CloudStorageService*&))[0x5c7db880f736]                                       storage/object_store.cpp:119(eloqstore::ObjectStore::ObjectStore(eloqstore::KvOptions const*, eloqstore::CloudStorageService*))[0x5c7db88283ab]
[time 2026-03-12T11:29:21.339449] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1873] Total memtables size: 0 MB
[time 2026-03-12T11:29:21.339506] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1874] Total SSTables size used in replay: 0 MB
[time 2026-03-12T11:29:21.339514] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1878] Total replay log size: 0 MB
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN9eloqstore14AsyncIoManager8InstanceEPKNS_9EloqStoreEj+0xea)[0x5c7db8802ec4]
src/async_io_manager.cpp:2569(eloqstore::CloudStoreMgr::CloudStoreMgr(eloqstore::KvOptions const*, unsigned int, eloqstore::CloudStorageService*))[0x5c7db87a6dbb]
bits/unique_ptr.h:962(std::_MakeUniq<eloqstore::CloudStoreMgr>::__single_object std::make_unique<eloqstore::CloudStoreMgr, eloqstore::KvOptions const*&, unsigned int&, eloqstore::CloudStorageService*>(eloqstore::KvOptions const*&, unsigned int&, eloqstore::CloudStorageService*&&))[0x5c7db87830c0]
src/async_io_manager.cpp:137(eloqstore::AsyncIoManager::Instance(eloqstore::EloqStore const*, unsigned int))[0x5c7db87771b4]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN6EloqDS18EloqStoreDataStore7StartDBEl+0x46)[0x5c7db7f8257c]
storage/shard.cpp:52(eloqstore::Shard::Shard(eloqstore::EloqStore const*, unsigned long, unsigned int))[0x5c7db7f20b11]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN6EloqDS16DataStoreService13OpenDataStoreEjOSt13unordered_setItSt4hashItESt8equal_toItESaItEEl+0x248)[0x5c7db7f295d8]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN6EloqDS22DataStoreServiceClient13OnLeaderStartEjlPj+0x255)[0x5c7db7eac3ab]
bits/unique_ptr.h:962(std::_MakeUniq<eloqstore::Shard>::__single_object std::make_unique<eloqstore::Shard, eloqstore::EloqStore*, unsigned long&, unsigned int&>(eloqstore::EloqStore*&&, unsigned long&, unsigned int&))[0x5c7db85600cc]
src/eloq_store.cpp:254(eloqstore::EloqStore::Start(unsigned long))[0x5c7db80cc434]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN9txservice6remote13CcNodeService13OnLeaderStartEPN6google8protobuf13RpcControllerEPKNS0_20OnLeaderStartRequestEPNS0_21OnLeaderStartResponseEPNS3_7ClosureE+0x633)[0x5c7db84ae1bf]
eloq_data_store_service/eloq_store_data_store.h:176(EloqDS::EloqStoreDataStore::StartDB(long))[0x5c7db86cdfd8]
[time 2026-03-12T11:29:31.339620] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1873] Total memtables size: 0 MB
[time 2026-03-12T11:29:31.339682] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1874] Total SSTables size used in replay: 0 MB
[time 2026-03-12T11:29:31.339691] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1878] Total replay log size: 0 MB
[time 2026-03-12T11:29:41.339838] [level INFO] [thread 60867] [log_state_rocksdb_cloud_impl.cpp:1873] Total memtables size: 0 MB

  9. yangsw26 commented on Mar 13, 2026

    @yangsw26
    yangsw26
    on Mar 13, 2026
    CollaboratorAuthor
    [time 2026-03-12T16:31:46.097608] [level INFO] [thread 77939] [log_state_rocksdb_cloud_impl.cpp:1878] Total replay log size: 17 MB
260312 16:31:48 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.6.10-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=3
max_threads=65537
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 25171106 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(my_print_stacktrace+0x44)[0x55e49069a48e]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(handle_fatal_signal+0x45a)[0x55e48fb56efa]
??:0(__sigaction)[0x7d7e55058520]
??:0(syscall)[0x7d7e5513490d]
bthread/sys_futex.h:41(bthread::futex_wait_private(void*, int, timespec const*))[0x7d7e55cf27b2]
bthread/butex.cpp:669(bthread::butex_wait(void*, int, timespec const*))[0x7d7e55cf2bb7]
bthread/condition_variable.cpp:104(bthread_cond_wait)[0x7d7e55cf5a65]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(+0x22c0952)[0x55e48ffa5952]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN9txservice5fault15RecoveryService8ShutdownEv+0x113)[0x55e490e4df59]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN9txservice7Sharder8ShutdownEv+0x128)[0x55e4909d294c]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN9txservice9TxService8ShutdownEv+0xcb)[0x55e490750117]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_ZN13DataSubstrate8ShutdownEv+0x109)[0x55e490743c31]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(+0x2299aab)[0x55e48ff7eaab]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_Z22ha_finalize_handlertonP13st_plugin_int+0x119)[0x55e48fb59704]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(+0x1ab662d)[0x55e48f79b62d]
[time 2026-03-12T16:31:56.083091] [level INFO] [thread 77933] [log_state_rocksdb_cloud_impl.cpp:90] CronJob LogPurger scheduling check at Thu Mar 12 16:30:56 2026
[time 2026-03-12T16:31:56.097748] [level INFO] [thread 77939] [log_state_rocksdb_cloud_impl.cpp:1873] Total memtables size: 17 MB
[time 2026-03-12T16:31:56.097781] [level INFO] [thread 77939] [log_state_rocksdb_cloud_impl.cpp:1874] Total SSTables size used in replay: 0 MB
[time 2026-03-12T16:31:56.097788] [level INFO] [thread 77939] [log_state_rocksdb_cloud_impl.cpp:1878] Total replay log size: 17 MB
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_Z15plugin_shutdownv+0x2d6)[0x55e48f79e5fb]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(+0x1939585)[0x55e48f61e585]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(_Z11mysqld_mainiPPc+0xd4d)[0x55e48f6270d0]
/home/eloq/workspace/eloqsql/bld/sql/mariadbd(main+0xd0)[0x55e48f619379]
[time 2026-03-12T16:32:06.097941] [level INFO] [thread 77939] [log_state_rocksdb_cloud_impl.cpp:1873] Total memtables size: 17 MB

  10. yangsw26 commented on Mar 13, 2026

    @yangsw26
    yangsw26
    on Mar 13, 2026
    CollaboratorAuthor
    #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff51fcab2 in bthread::futex_wait_private (timeout=0x0, expected=0, addr1=0x7fffd86e47c0) at /home/cz-studio/tools/eloq_brpc/brpc/src/bthread/sys_futex.h:40
#2  bthread::wait_pthread (pw=..., abstime=<optimized out>, abstime@entry=0x0) at /home/cz-studio/tools/eloq_brpc/brpc/src/bthread/butex.cpp:153
#3  0x00007ffff51fceb7 in bthread::butex_wait_from_pthread (abstime=0x0, expected_value=0, b=0x21700081340, g=<optimized out>) at /home/cz-studio/tools/eloq_brpc/brpc/src/bthread/butex.cpp:669
#4  bthread::butex_wait (arg=0x21700081340, expected_value=expected_value@entry=0, abstime=abstime@entry=0x0) at /home/cz-studio/tools/eloq_brpc/brpc/src/bthread/butex.cpp:702
#5  0x00007ffff51ffd65 in bthread_cond_wait(bthread_cond_t * __restrict__, bthread_mutex_t * __restrict__) (c=0x7fffd86e4e20, m=0x7fffd86e4e08) at /home/cz-studio/tools/eloq_brpc/brpc/src/bthread/condition_variable.cpp:103
#6  0x0000555557811054 in bthread::ConditionVariable::wait (this=0x7fffd86e4e20, lock=...) at /usr/local/include/bthread/condition_variable.h:59
#7  0x0000555557840fc7 in txservice::TxResult<std::pair<txservice::RecordStatus, unsigned long> >::Wait (this=0x7fffd86e4de8) at /home/cz-studio/studio/dev/eloqdata/eloqsql/data_substrate/tx_service/include/tx_req_result.h:355
#8  0x000055555782c831 in txservice::TemplateTxRequest<txservice::ReadTxRequest, std::pair<txservice::RecordStatus, unsigned long> >::Wait (this=0x7fffd86e4de0) at /home/cz-studio/studio/dev/eloqdata/eloqsql/data_substrate/tx_service/include/tx_request.h:147
#9  0x0000555558450b70 in txservice::LocalCcShards::DataSyncForRangePartition (this=0x21688501800, data_sync_task=std::shared_ptr<txservice::DataSyncTask> (use count 1, weak count 0) = {...}, worker_idx=0)
    at /home/cz-studio/studio/dev/eloqdata/eloqsql/data_substrate/tx_service/src/cc/local_cc_shards.cpp:3755
#10 0x000055555844dc98 in txservice::LocalCcShards::DataSyncWorker (this=0x21688501800, worker_idx=0) at /home/cz-studio/studio/dev/eloqdata/eloqsql/data_substrate/tx_service/src/cc/local_cc_shards.cpp:3112
#11 0x0000555558440f10 in operator() (__closure=0x216b1bb6668) at /home/cz-studio/studio/dev/eloqdata/eloqsql/data_substrate/tx_service/src/cc/local_cc_shards.cpp:318
#12 0x000055555846d365 in std::__invoke_impl<void, txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#13 0x000055555846d095 in std::__invoke<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> >(struct {...} &&) (__fn=...) at /usr/include/c++/11/bits/invoke.h:96
#14 0x000055555846ce1a in std::thread::_Invoker<std::tuple<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x216b1bb6668) at /usr/include/c++/11/bits/std_thread.h:259
#15 0x000055555846cc88 in std::thread::_Invoker<std::tuple<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> > >::operator()(void) (this=0x216b1bb6668) at /usr/include/c++/11/bits/std_thread.h:266
#16 0x000055555846ca82 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> > > >::_M_run(void) (this=0x216b1bb6660) at /usr/include/c++/11/bits/std_thread.h:211
#17 0x00007ffff48dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007ffff4494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007ffff45268d0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) f 9
#9  0x0000555558450b70 in txservice::LocalCcShards::DataSyncForRangePartition (this=0x21688501800, data_sync_task=<error reading variable: Cannot access memory at address 0x7fffd86e4940>, worker_idx=0)
    at /home/cz-studio/studio/dev/eloqdata/eloqsql/data_substrate/tx_service/src/cc/local_cc_shards.cpp:3755
3755            read_req.Wait();
(gdb) p range_id
$1 = 3830
(gdb) p catalog_ccm_name
Cannot access memory at address 0x55555a7e8fc0
(gdb) p table_name
Cannot access memory at address 0x7fffd86e49e8
(gdb) p range_id
$2 = 3830
(gdb) p during_split_range
$3 = false
(gdb)
    [time 2026-03-13T14:26:11.452754] [level INFO] [thread 123181] [tx_operation.cpp:4959] Split Flush transaction write clean log, range id 3830, txn: 4295013281
[time 2026-03-13T14:26:11.460649] [level INFO] [thread 123181] [background_write.cpp:100] begin compaction on table_ranges.23
[time 2026-03-13T14:26:11.460856] [level INFO] [thread 123181] [tx_operation.cpp:978] Update leader from log service: ng_id=0, node_id=0, term=3
[time 2026-03-13T14:26:11.460911] [level INFO] [thread 123181] [sharder.cpp:687] skip out-dated node group leader update, ng: 0, term: 3, cached term: 3
[time 2026-03-13T14:26:11.461046] [level INFO] [thread 123341] [local_cc_shards.cpp:5648] Split range on table ./test/t2 partition 3830 succeeded.
[time 2026-03-13T14:26:11.461370] [level INFO] [thread 123181] [template_cc_map.h:771] AcquireAllCc from txn: 4295013305, on table: __catalog, key: ./test/t2 is resumed from blocking. acquired_lock: 4, err_code: 0
[time 2026-03-13T14:26:11.463152] [level INFO] [thread 123181] [file_gc.cpp:664] ListCloudFiles got 4 files
[time 2026-03-13T14:26:11.463213] [level INFO] [thread 123181] [file_gc.cpp:482] skip file since file_id=307, least_not_archived_file_id=0
[time 2026-03-13T14:26:11.464342] [level INFO] [thread 123181] [background_write.cpp:289] finish compaction on table_ranges.27
[time 2026-03-13T14:26:11.465906] [level INFO] [thread 123181] [background_write.cpp:100] begin compaction on table_ranges.15
[time 2026-03-13T14:26:11.469985] [level INFO] [thread 123180] [background_write.cpp:100] begin compaction on table_ranges.26
[time 2026-03-13T14:26:11.470244] [level INFO] [thread 123341] [local_cc_shards.cpp:5589] Splitting table ./test/t2*$$idx range 1284 into 3 ranges. New range ids 1285,1286, txn: 43790
[time 2026-03-13T14:26:11.470633] [level INFO] [thread 123341] [local_cc_shards.cpp:5607] Begin to do range split flush, range_id:1284
[time 2026-03-13T14:26:11.470736] [level INFO] [thread 123180] [tx_operation.cpp:4262] Split Flush transaction prepare acquire all, range id 1284, txn: 43790
[time 2026-03-13T14:26:11.470856] [level INFO] [thread 123180] [tx_operation.cpp:1900] All acquire all requests finished for tx 43790, fail count: 0
[time 2026-03-13T14:26:11.470919] [level INFO] [thread 123180] [tx_operation.cpp:4348] Split Flush transaction write prepare log, range id 1284, txn: 43790
[time 2026-03-13T14:26:11.480819] [level INFO] [thread 123180] [tx_operation.cpp:978] Update leader from log service: ng_id=0, node_id=0, term=3
[time 2026-03-13T14:26:11.480878] [level INFO] [thread 123180] [sharder.cpp:687] skip out-dated node group leader update, ng: 0, term: 3, cached term: 3
[time 2026-03-13T14:26:11.480906] [level INFO] [thread 123180] [tx_operation.cpp:4409] Split Flush transaction install dirty range, range id 1284, txn: 43790
[time 2026-03-13T14:26:11.481007] [level INFO] [thread 123180] [tx_operation.cpp:4452] Split Flush transaction unlock cluster config, range id 1284, txn: 43790
[time 2026-03-13T14:26:11.481074] [level INFO] [thread 123180] [tx_operation.cpp:4548] Split Flush transaction data sync, range id 1284, txn: 43790

    For test: subselect_sj.test

    insert into t2 select b+10 from t2;
analyze table t1,t2;
Table   Op      Msg_type        Msg_text
test.t1 analyze status  Engine-independent statistics collected
test.t1 analyze status  OK
test.t2 analyze status  Engine-independent statistics collected
test.t2 analyze status  OK
explain
select a from t1, t2 where b between 1 and 2 and a in (select b from t2);
id      select_type     table   type    possible_keys   key     key_len ref     rows    Extra
1       PRIMARY t2      range   idx     idx     5       NULL    #       Using where; Using index
1       PRIMARY t1      ALL     NULL    NULL    NULL    NULL    #       Using where; Using join buffer (flat, BNL join)
1       PRIMARY t2      ref     idx     idx     5       test.t1.a       #       Using index; FirstMatch(t1)
explain
select a from t1 join t2 on b between 1 and 2 and a in (select b from t2);
id      select_type     table   type    possible_keys   key     key_len ref     rows    Extra
1       PRIMARY t2      range   idx     idx     5       NULL    #       Using where; Using index
1       PRIMARY t1      ALL     NULL    NULL    NULL    NULL    #       Using where; Using join buffer (flat, BNL join)
1       PRIMARY t2      ref     idx     idx     5       test.t1.a       #       Using index; FirstMatch(t1)
drop table t1,t2;

  11. yangsw26 commented on Mar 13, 2026

    @yangsw26
    yangsw26
    on Mar 13, 2026
    CollaboratorAuthor

    Deadlock caused by range splits and drop tables.

  12. yangsw26
    closed this as completedon Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureAdd new feature

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions