Skip to content

feat(shard): add shard management with region splitting#115

Merged
slow2342 merged 1 commit into
mainfrom
feat/shard-management
Jun 12, 2026
Merged

feat(shard): add shard management with region splitting#115
slow2342 merged 1 commit into
mainfrom
feat/shard-management

Conversation

@slow2342

Copy link
Copy Markdown
Owner

Summary

Implement region-based data partitioning with metadata tracking through Raft. Regions represent contiguous key ranges [start_key, end_key) with associated leader and replica placement. This lays the foundation for future multi-Raft-group architecture.

Changes

  • src/shard/region.rsRegion, RegionEpoch data structures with serde + rkyv dual serialization
  • src/shard/manager.rsShardManager in-memory index with find_region, apply_split, apply_update, load/save from RocksDB
  • src/shard/scheduler.rsSplitScheduler background task for size-based auto-split (not yet wired)
  • src/api/shard.rsShardService gRPC: GetRegion, ListRegions, SplitRegion
  • proto/shard.proto — AetherShard service definition
  • src/raft/mod.rs — RegionSplit/RegionUpdate variants in RaftRequest/RaftResponse
  • src/raft/state_machine.rs — ShardManager integration with apply handlers and snapshot support
  • src/storage/rocksdb.rs — New region column family
  • src/config.rs — ShardConfig (max_region_size_bytes, max_regions, auto_split)
  • src/main.rs — Wire ShardService and ShardManager

Design decisions

  • Single Raft group: splits are metadata-only operations, no data movement
  • Region metadata through Raft: consistent with KV/Lease/Auth mutation pattern
  • rkyv + serde: rkyv for storage hot path, serde for Raft bincode path
  • max_regions enforcement: checked during apply_split, preserved across snapshot restore

Testing

  • cargo build passes
  • cargo test passes
  • cargo clippy -- -D warnings clean
  • cargo fmt -- --check clean

Related issues

Refs #86

Implement region-based data partitioning with metadata tracking through
Raft. Regions represent contiguous key ranges [start_key, end_key) with
associated leader and replica placement.

- Add Region, RegionEpoch data structures with serde + rkyv derives
- Add ShardManager for in-memory region index with find_region, apply_split
- Add region column family to RocksDB for persistent region metadata
- Add RegionSplit/RegionUpdate variants to RaftRequest/RaftResponse
- Integrate ShardManager into state machine with snapshot support
- Add AetherShard gRPC service (GetRegion, ListRegions, SplitRegion)
- Add ShardConfig with max_regions enforcement
- Add SplitScheduler for auto-split (not yet wired)
@slow2342
slow2342 merged commit e295aaa into main Jun 12, 2026
5 checks passed
@slow2342
slow2342 deleted the feat/shard-management branch June 12, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant