Skip to content

Allow sharding of key space on a single machine #26

@adamfowleruk

Description

@adamfowleruk

Please start with a use case description for a USER of this work
[Who] As a database admin
[What] I need to use all of the processing power of my DB server when storing/retrieving keys and performing queries
[Value] In order to achieve very high throughput on sets, gets, queries

Describe the solution you'd like
Wrapper ShardingKeyValueStore(KeyValueStore[] shards) that only passes on sets and gets for keys to the correct sharded KV store. It should also map/reduce the queries passed to it

Describe alternatives you've considered
Consider letting the Database class handle this, but it's more of a storage detail, and so belongs in a KeyValueStore implementation. The suggested approach also makes it easier to go from an unsharded store to a sharded one, and doesn't impose knowledge on sharding on underlying KV store implementations

Additional context
Requirements:-

  • The deduction step to go from std::size_t hash -> KV index must be blindingly fast.
  • Use C++ threads with incoming/outgoing work queues if possible (will make doing this later over the network easier)
  • Fixed number of shards for now
  • No support for resharding for now

Not requirements:-

  • Not multi-machine for now - still single server/embedded

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions