[nick-potts]

Closes #3680.

Summary

When ployzd replaces its Corrosion container, it waits for the old one to stop before it starts the new one. A replacement happens on an image change at upgrade or downgrade, a config change, or when a kept container doesn't answer. ployzd stopped the old container with Docker's default 10 s grace. On SIGTERM, Corrosion's gossip endpoint waits up to 5 s for its QUIC connections to go idle, and peers keep them open. So every replacement waited the full 5 s, and ployzd isn't ready until the new Corrosion answers. #3680 measured ployzd down 6.0 to 7.0 s per upgrade step.

 ployzd start
   CorrosionConfig::start
     ensure container: Replace
-      docker stop ployz-corrosion            # default 10 s grace; Corrosion exits after ~5.1 s
+      docker stop -t 1 ployz-corrosion       # SIGKILL 1 s after SIGTERM
       create + start new Corrosion
     wait ready                               # ployzd signals ready to systemd after this

• ManagedService::with_stop_grace sets Docker's per-stop timeout. Only Corrosion uses it; unregistry keeps Docker's default.
• The grace covers every stop of Corrosion: replacement, the readiness fallback, cleanup on reset, and remove_retained.
• With a 1 s grace, Corrosion's graceful exit never finishes, so every replacement and reset now ends in SIGKILL.

Nothing durable rides on Corrosion's graceful exit:

• A killed Corrosion keeps every committed write. SQLite's WAL survives a process kill, and Deploys no longer stall after a partition heals: Corrosion syncs once per commit instead of every page #3674's synchronous = "full" also covers power loss.
• The graceful exit already drops queued incoming changes before its 5 s wait. Peers resend them through sync, as after a crash. Buffered partial changesets live in SQLite and are re-applied at start.
• Subscription DBs are derived state, and ployzd opens fresh subscriptions.
• On upgrade no API write can be in flight, because Corrosion starts before ployzd serves.

Downgrade: a daemon at 0.3.11 or older stops a newer Corrosion with its own code and the default grace, so a downgrade that replaces Corrosion still pauses ~5 s. 0.3.12 and this head pin the same Corrosion image and config, so moving between them replaces nothing and doesn't pause.

Evidence

Boat --cloud with 3 Servers, enrolled on the checkout and then installed with the genuine v0.3.11 release (efb602dc). Each step rolled one Server while the other daemons kept running.

• Manifest revision = PR head dd0ef1430d. The running head ployzd hashed 82fe77fbc17b, the same as the run's checkout build.
• dashboard/.verify/run/sdk.log: sdk: hit github run 41214907435 (3add01b13afe). That is an exact key hit on this checkout's SDK sources; this PR changes only ployzd, which the SDK doesn't include. No stale main@.
• Doctor passed. Everything posted here went through redact: 0 leaks.
• The lab used Kill a Boat clone's stale sshd before restarting ssh; add PLOYZ_VERIFY_BOAT_AVOID_NODES #3698's sshd fix, applied locally and not committed. It isn't part of this diff.

Pause. start→ready is ployzd's start to systemd READY, which comes after Corrosion answers. Old-Corrosion timings are from docker events on ployz-corrosion, with SIGTERM at +0.

Step	Corrosion replaced	Old Corrosion	ployzd start→ready
Restart on 0.3.11 (baseline), ×3	no	–	0.10 / 0.15 / 0.16 s
Upgrade 0.3.11 → head, ×6 (one under load)	yes, by head	SIGKILL at +1.01 s; new one starts at +1.15 to 1.46 s	1.84 to 2.30 s
head → 0.3.12 → head	no (same image and config)	–	0.13 / 0.16 s
Downgrade head → 0.3.11, ×3 (one under load)	yes, by 0.3.11 code	exits at +5.10 to 5.13 s, the old behaviour	5.82 to 5.94 s

#3680 measured 6.5 to 7.0 s for the same upgrade on 0.3.12, which is where the downgrade rows sit.

• The probe through ployz exec (management path) lost 0 of 77,248 acks across all those steps. Its longest gap, 6.71 s, was the downgrade on the db host (old code). Head upgrades on the db host showed 3.34 s and 4.06 s, against Upgrading or downgrading a Server pauses its daemon ~7 s while the old Corrosion container takes ~5 s to stop #3680's 6.94 s. The probe's gap includes its own ployz exec reconnect.
• The in-cluster Postgres writer lost 0 of 9,187 acks. Every gap stayed ≤0.53 s except 0.71 s and 1.32 s during one db-host head upgrade, with no session drop. The first db-host upgrade showed no gap over 0.5 s.

Kill under write load (the coordinator's condition). A loop on each Server inserted numbered rows into the replicated cluster table through its local Corrosion API. An id counted as acked only after the API answered rows_affected: 1. While all three loops and the ack probe ran:

• 3× docker kill -s KILL and 3× docker stop -t 1 on Corrosion, on the db host and on a second Server. The keeper had it running again within 0.5 to 2.1 s each time.
• One real replacement on the third Server: down to 0.3.11 and back to head.

Check	Result
Acked Corrosion writes	25,054 (7,953 / 9,383 / 7,718); 0 lost on every Server
Rows	the identical 25,054 load rows on every Server; rows-agree.sh: rows agree on every Machine: 25073 rows
PRAGMA integrity_check	ok ×3
Ack probe (Postgres via ployz exec)	59,823 acked, 0 lost, longest gap 3.0 s

Power cut under load (#3674's scripts/verify/power-cut-replication.sh on the db host: two cuts, the second 8 s after boot). The Corrosion load ran on every Server during it, plus a web Service with 3 replicas.

• machine-3 kept its Corrosion actor, and every proxy routed only In copies.
• The script's own rows check reported load rows that were still being written: I'd left the loops running. After stopping them: rows agree on every Machine: 47653 rows. 22,138 acked writes, 0 lost on every Server, integrity_check ok ×3.
• The cut also zeroed the unsynced tail of the cut Server's own ack log file (2,175 NUL bytes). Those entries were never counted; all 349 legible acks from it are present everywhere.

Not covered: each roll's other-daemon timestamps weren't recorded per step. The step helper touched only one Server's units.

Focused tests: cargo test -p ployzd --lib -- managed_service corrosion::service: 16 passed. cargo clippy -p ployzd --all-targets -D warnings: clean.

Review:

• Round 1 (Codex and Opus, full diff): no BUG-class finding. Opus asked to drop a test that only mirrored the setter, and to correct the durability comment; both were applied.
• Final diff dd0ef1430d (Codex and Opus): CLEAN, no BUG-class finding.
• Codex noted a longer-term option: our Corrosion fork could close the gossip endpoint before waiting for idle, so a stop wouldn't need SIGKILL.

Design bets

• Stable promise: no wire, RPC, row, schema or store-format change. The container spec and its digest are unchanged, so this change alone doesn't recreate Corrosion.
• 2, AP over C: a killed Corrosion converges through sync, the path a crash already takes. Shown above under kills and a power cut.
• 3, bounded imperative commands: Corrosion's keeper and replacement stay Machine-local loops over its container that extend Docker's restart policy. The kill shortens their bound and adds no background work.
• 9, lean on proven primitives: Docker's per-stop timeout bounds the stop. The 1 s grace deliberately skips Corrosion's graceful exit, whose remaining step only waits for gossip to go idle.

Merge Danger

Door: two-way. Reverting restores the default grace; no data or format changes.

Blast Radius: Corrosion's lifecycle on every Server.

Every Corrosion replacement and reset now ends in SIGKILL. The kill and power-cut runs above show no lost acked writes and identical rows.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 1fc8feaa-3f93-4308-a1b5-b09f89b80d20

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-31T17:23:36.157736Z	dd0ef14	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.