Skip to content

Complete parallel tree flush and incremental persistence - #56

Draft
giancao wants to merge 17 commits into
mainfrom
task-paxos-kv
Draft

giancao wants to merge 17 commits into
mainfrom
task-paxos-kv

Conversation

@giancao

@giancao giancao commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Tree flush now forks from root to leaf and joins consolidated page results back to root through the existing stdexec flow. Persistent flush writes final pages in that flow, reuses unchanged Local/Chunk mapping segments, and commits the durable root after its dependencies finish. Ordinary flush does not issue persistent writes.

Restore hard MemTable and Tree capacity backpressure until persistence and actual owner release make space available. Complete accepted work through queue pressure using existing progress dispatchers, preserve WAL replay pairing, and separate normal page split targets from the 32 MiB physical safety bound. Remove the abandoned independent snapshot capture path. This branch also contains the previously prepared architecture documentation commits.

Validation:

  • 1863 native C++ tests and 134 Rust FFI tests passed; Rust fmt/workspace clippy and the normal C++ format/tree-lint commit hook passed.
  • The original 100k × 16 KiB workload passed three consecutive rounds in one 3-node host-network deployment: 1986, 2323, and 2317 ops/s, six successful snapshots per round, zero snapshot failures, elections, failed/incomplete requests, or verification failures.
  • At least 104 seconds of metadata-only idle left all nodes at the same durable/applied frontier, no dirty pages, MemTable payload, overlay, or pending checkpoints, and one retained WAL slot. All nodes exited normally without OOM.
  • Experiment resources were 12 GiB / 4 CPU per node, 32 MiB × 8 MemTables, and a widened 95% guard. This uses the original memory-backed KV/WAL fixture and does not claim the old 80% guard or physical disk throughput passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant