-
Notifications
You must be signed in to change notification settings - Fork 29
Allow sharding of key space on a single machine #26
Description
Please start with a use case description for a USER of this work
[Who] As a database admin
[What] I need to use all of the processing power of my DB server when storing/retrieving keys and performing queries
[Value] In order to achieve very high throughput on sets, gets, queries
Describe the solution you'd like
Wrapper ShardingKeyValueStore(KeyValueStore[] shards) that only passes on sets and gets for keys to the correct sharded KV store. It should also map/reduce the queries passed to it
Describe alternatives you've considered
Consider letting the Database class handle this, but it's more of a storage detail, and so belongs in a KeyValueStore implementation. The suggested approach also makes it easier to go from an unsharded store to a sharded one, and doesn't impose knowledge on sharding on underlying KV store implementations
Additional context
Requirements:-
- The deduction step to go from std::size_t hash -> KV index must be blindingly fast.
- Use C++ threads with incoming/outgoing work queues if possible (will make doing this later over the network easier)
- Fixed number of shards for now
- No support for resharding for now
Not requirements:-
- Not multi-machine for now - still single server/embedded