Repository navigation
Deploys no longer stall after a partition heals: Corrosion syncs once per commit instead of every page - #3674
Conversation
…ery store write synchronous The store's inode sync flag synced every page, so a healing Machine held Corrosion's one write connection for up to 51 s per commit and its own writes passed ployzd's 10 s limit. FULL syncs the WAL once per commit, before Corrosion broadcasts it, which keeps the power-cut guarantee. Stores an older daemon flagged are cleared at start. Machines now report store_syncs_commits, so the CLI warns about ZFS and Btrfs stores only on daemons that predate this.
Brings rows-agree.sh and rows_diff.py from #3214. Equal clocks can hide different values, so the check now waits for every Machine to hold the same rows in every replicated table.
getployz/corrosion#1, built by the fork's image workflow at 28f00b417271.
Clearing it at install left an older Corrosion, still serving while the new image pulled, writing with neither the flag nor synchronous=full. A failed pull kept it that way. Found in review.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Closes #2925.
Why
Deploys must keep working while part of a Cluster is partitioned and right after it heals, which is exactly when an operator deploys to recover. On main they stall for minutes or fail with
Corrosion I/O timed out. The cause is the inode sync flag (chattr +S) ployzd put on the Corrosion store so that a commit survives a power cut. It makes every SQLite page write synchronous. While a healing Machine ingests the other side's changes, one sync commit holds Corrosion's only write connection for up to a minute on a slow disk. ployzd's own writes on that Machine then pass its 10 s limit.SQLite's own
synchronous = FULLgives the same guarantee with one sync per commit. Corrosion hardcodesNORMAL, so our fork gains an optional[db] synchronous(getployz/corrosion#1). Nick approved that fork patch as Q161, superseding Q76's "no fork".Summary
ployz, image20261031-28f00b417271@sha256:384f1e20…):DbConfig.synchronous, defaultnormal, applied to write connections besidecache_size. Upstream behaviour is unchanged. Nothing has gone to superfly. An upstream offer is drafted for Nick.synchronous = "full", pins the image (also in the testkit Dockerfile), and clears +S from stores an older daemon flagged. The clear runs only after the new Corrosion runs. A failed image pull leaves the old Corrosion serving with +S intact.store_syncs_commits. It istrueunless the store is on tmpfs. The CLI's power-cut warning (server ls, andserver show'sstoreline) is skipped when it istrue. Older daemons and tmpfs still warn.docs/user/troubleshooting/deployments.mdsays so.rows-agree.sh,rows_diff.pyandpower-cut-replication.shcome from Draft for 0.3.5: rotate the Corrosion actor after a power cut instead of synchronous store writes #3214. They compare every replicated row, not just clocks. Draft for 0.3.5: rotate the Corrosion actor after a power cut instead of synchronous store writes #3214's actor rotation is not included.Evidence
All runs are Boat
--cloudwith 4 Machines. Manifestrevision= PR headbd2dd303b6.dashboard/.verify/run/sdk.logreadssdk: hit local eb07fd70 (5a3013cf42ed), an exact hit keyed by the SDK's source files:bd2dd303changes only ployzd, which the SDK doesn't include, so it isn'tstale main@. Doctor passed. Every saved or posted output went throughredactwith 0 leaks.Partition A/B, same Machines. Each side ran one writer of 150 transactions × 200 rows during a 2/2 underlay split. Then the split healed and 8 deploys followed. machine-3 sits on a slow node: a synchronous 4 KiB write takes about 118 ms there, against 0.2 to 0.3 ms on the others. The "main" column is this build with the store forced back to main's mode (+S on, no
synchronousline).Corrosion I/O timed outOn an earlier main run (0.3.11 binaries), the first deploy after the heal took 301 s. With a config-only batch cap it failed with exit 1 after 397 s, so that approach is dropped. Switching the cluster back to this PR's mode let machine-3 finish its stuck 120,000-row catch-up within about 2 minutes.
Plain SQLite on the same disks, 50 tx × 200 rows:
Durability (
scripts/verify/power-cut-replication.sh):machine-3 kept its Corrosion actor,rows agree on every Machine: 120024 rows,PASS.HOT_SERVICE=web, the first cut lands the momentservice restartreturns, which Draft for 0.3.5: rotate the Corrosion actor after a power cut instead of synchronous store writes #3214 measured losing 9 versions without +S):machine-4 kept its Corrosion actor,rows agree on every Machine: 120024 rows,PASS.normalgave 10 API commits → 0 fsync/fdatasync, andfullgave 10 commits → 10.Swarm (landing.md), each agent on its own Boat at
bd2dd303:updateall → downgrade one. Every deploy exited 0 (6 to 11 s, through Cloud and each Entry). Postgres held1000|5aa148…throughout, with 0 timeouts. Polling at 0.1 s, +S cleared only once the new FULL Corrosion ran. On downgrade it came back before the old image started, so at no moment were both protections off. A 0.3.11 Entry dropsstore_syncs_commitswhen it relays a newer Machine's record, so in a mixed clusterserver lsthrough it can still warn about an upgraded ZFS or Btrfs Server. That over-warns and never under-warns, and absence claims nothing by design.synchronous = "full"with no +S and reportedstore_syncs_commits: true, and no power-cut warning showed. A double cut on a non-db Machine leftrows-agree.shpassing, andserver lsandpsagreed through every Entry. A cut on the db Machine during the ack probe lost 0 ids (probe:acked 214995, lost []). Deploys during and after a partition exited 3, 3 and 0, asscaling.mdsays. Aftersystemctl restart ployz, a deploy exited 0 in 9 s with 0 timeouts and no +S. Unrelated to this PR, Cloud picked the partitioned Machine as the Entry (filed Cloud picks a partitioned Machine as the Entry, so a deploy lands every new replica on the cut-off side #3675), and the progress count is deploy live view counts Servers, not replicas #2082.Unit tests:
a_store_an_older_daemon_flagged_stops_writing_every_page_synchronously,one_unreadable_file_leaves_the_rest_of_the_store_cleared(run on ext4, where +S takes),a_server_whose_store_syncs_each_commit_is_not_warned_about_on_any_filesystem. In the fork:write_connections_use_the_configured_synchronousandsynchronous_defaults_to_normal_and_reads_full.Review: round 1 had Opus (no defects) and Codex. Codex found that clearing +S at install left the old NORMAL Corrosion unprotected during the image pull. ployzd has no writers in that window today, but the clear now waits for the new Corrosion anyway. Round 2 (Codex, full diff) found no BUG-class issues.
Design bets
store_syncs_commitsis an optional field: older readers ignore it, and absence claims nothing. The Corrosion config is local to the Machine and moves with the pinned image, and the old image tolerates the new line. Downgrading to 0.3.11 sets +S again, which is today's state. Compat impact: a mixed cluster can over-warn as described above, nothing more.Merge Danger
Door: two-way. Reverting restores +S at the next daemon start, and the store format is unchanged.
Blast Radius: every Machine's store.
Every Machine's Corrosion is recreated on the new image at upgrade. On a disk whose fsync is slow, commits now cost one sync each instead of none (with NORMAL) or one per page (with +S). The swarm measured deploys at 6 to 11 s.