[Feature] DataStoreService supports multi data shards #111

Activity

1. lzxddz

   self-assigned this

   on Oct 21, 2025
2. lzxddz

   changed the title [-]DataStoreService supports multi data shards[/-] [+][Feature] DataStoreService supports multi data shards[/+] on Oct 21, 2025
3. lzxddz commented on Oct 22, 2025

   

   lzxddz

   on Oct 22, 2025

   CollaboratorAuthor

   More actions

   Prioritize doing another task eloqdata/eloqdoc#263
4. lzxddz

   added a parent issue on Oct 28, 2025

   • Bind DataStoreService shard nodes with NodeGroups of txservice #120
5. lzxddz

   changed the title [-][Feature] DataStoreService supports multi data shards[/-] [+][Feature] DataStoreService client supports multi data shards[/+] on Oct 28, 2025
6. lzxddz commented on Oct 28, 2025

   

   lzxddz

   on Oct 28, 2025

   CollaboratorAuthor

   More actions

   Design of client config:

   • mapping function of partition_id to bucket_id
   • {bucket_id->ng_id}
   • {ng_id->leader_node_id}
   • {node_id->node_info(host,port,...)}
7. lzxddz commented on Nov 4, 2025

   

   lzxddz

   on Nov 4, 2025

   CollaboratorAuthor

   More actions

   test eloqdoc crashed with three node datastore

   eloqdata/eloqdoc#285
8. lzxddz commented on Nov 5, 2025

   

   lzxddz

   on Nov 5, 2025

   Last edited by lzxddz

   CollaboratorAuthor

   More actions

   Test multi shard datastore on Eloqdoc using TPCC.
9. lzxddz commented on Nov 6, 2025

   

   lzxddz

   on Nov 6, 2025

   CollaboratorAuthor

   More actions

   Loaded 100 million with 3 nodes (replica num 1). Restart and check data, no miss.
10. lzxddz commented on Nov 6, 2025

    

    lzxddz

    on Nov 6, 2025

    CollaboratorAuthor

    More actions

    TODO: rebase main and remove key prefix of bucket_id in datastore (that is added in PR #129)
11. lzxddz

    linked a pull request that will close this issueSupport multi data store shards #141on Nov 10, 2025
12. lzxddz

    changed the title [-][Feature] DataStoreService client supports multi data shards[/-] [+][Feature] DataStoreService supports multi data shards[/+] on Nov 11, 2025
13. lzxddz commented on Nov 11, 2025

    

    lzxddz

    on Nov 11, 2025

    CollaboratorAuthor

    More actions

    Replace srand()/rand() (not thread-safe) with std::mt19937( thread-safe random generator) to distribute the buckets to NodeGroups.
14. lzxddz commented on Nov 12, 2025

    

    lzxddz

    on Nov 12, 2025

    CollaboratorAuthor

    More actions

    Fix eloqdoc/eloqsql/eloqkv failed to run CI test.
15. lzxddz commented on Nov 12, 2025

    

    lzxddz

    on Nov 12, 2025

    Last edited by lzxddz

    CollaboratorAuthor

    More actions

    Test EloqKV 3 nodes, crashed at scan after kill node-3 and restart node-3.

    ccm_scanner.h:1068
     assert(kv_is_drained.at(bucket_id) || kv_cache->Size() > 0);
    

    "kv_is_drained" not contain the "bucket_id";

    Also crashed at here:

    eloqkv: /home/lzx/eloqdata/eloqkv/tx_service/include/cc/template_cc_map.h:2540: bool txservice::TemplateCcMap<KeyT, ValueT, VersionedRecord, RangePartitioned>::Execute(txservice::ScanNextBatchCc&) [with KeyT = EloqKV::EloqKey; ValueT = EloqKV::RedisEloqObject; bool VersionedRecord = false; bool RangePartitioned = false]: Assertion `typed_cache->Size() == 0' failed.
    
16. lzxddz commented on Nov 13, 2025

    

    lzxddz

    on Nov 13, 2025

    CollaboratorAuthor

    More actions

    Test EloqKV 3 nodes, crashed at scan after kill node-3 and restart node-3.

    ccm_scanner.h:1068
     assert(kv_is_drained.at(bucket_id) || kv_cache->Size() > 0);
    

    "kv_is_drained" not contain the "bucket_id";

    Also crashed at here:

    eloqkv: /home/lzx/eloqdata/eloqkv/tx_service/include/cc/template_cc_map.h:2540: bool txservice::TemplateCcMap<KeyT, ValueT, VersionedRecord, RangePartitioned>::Execute(txservice::ScanNextBatchCc&) [with KeyT = EloqKV::EloqKey; ValueT = EloqKV::RedisEloqObject; bool VersionedRecord = false; bool RangePartitioned = false]: Assertion `typed_cache->Size() == 0' failed.
    

    Fixed by PR: eloqdata/tx_service#217
17. lzxddz commented on Nov 14, 2025

    

    lzxddz

    on Nov 14, 2025

    Last edited by lzxddz

    CollaboratorAuthor

    More actions

    For changed the random generator in InitRangeBuckets from srand()/rand() to std:mt19937 and change the count of buckets to 1024, several "mono_multi" test cases of EloqSql are not passed.
    (fault_inject_for_upsert_1, multi_nodes)

    The "expire.tcl" test case of EloqKv mutli-nodes is not passed, see issue eloqdata/eloqkv#281
18. lzxddz commented on Nov 17, 2025

    

    lzxddz

    on Nov 17, 2025

    CollaboratorAuthor

    More actions

    Fix some potential issues on shutdown data_shard: data_store_ should not be set to nullptr on shutdown data_shard (just call shutdown()), when re-open the data-shard, call initialize() to re-init the members of datastore.
19. lzxddz

    closed this as completedin #141on Nov 21, 2025