A Corrosion stopped through Docker while ployzd waits for a new one leaves the daemon down 270 s, then a restart (also 0.3.12) #3706

Summary

When ployzd starts a new Corrosion container (an upgrade or downgrade that changes the image, or any replacement) and that container is then stopped through Docker (docker kill / docker stop ployz-corrosion) before it answers, ployzd doesn't start it again. It polls readiness until its 270 s START_TIMEOUT runs out, exits 1, and systemd's restart then brings Corrosion up. The Server's daemon is unavailable for about 4.5 minutes.

docs/user/troubleshooting/servers.md says: "If it was stopped, as by docker stop ployz-corrosion, Ployz starts it again within a few seconds". That holds once ployzd is running, since its keeper restarts Corrosion in 0.4-2.2 s; I measured 6 kills that way. It doesn't hold inside ployzd's own startup wait.

Where

• soak-0313, manifest revision = bc82b8fd1a6c91ab5ed394355a4cb1192f333471; dashboard/.verify/run/sdk.log: sdk: hit local b3214485 (74ce0b973a8d), an exact content-key hit (the only later commit, Stop a replaced Corrosion after 1 s so an upgrade pauses ployzd ~1 s, not ~7 s #3697, changes ployzd alone). Boat --cloud, 3 Servers. Each step rolled one Server (verify-server "Roll one Server") while the other daemons kept running.
• Code: core/crates/ployzd/src/corrosion/service.rs, wait_ready_or_replace / wait_ready. A kept container gets KEPT_READY_TIMEOUT (60 s) and is then replaced. A fresh container gets only wait_ready, which loops every 0.5 s until bounded_start's 270 s, and never checks whether the container is still running.

Repro

1. Have a Server on 0.3.11 (ployzd efb602dc), so the next daemon replaces Corrosion's image.
2. On that Server, arm a watcher that runs docker kill -s KILL ployz-corrosion as soon as the replacement container is running.
3. Roll that Server to the sha's ployzd (stop the ployz units, swap the binaries, ployzd install --install-only, start).

docker events on ployz-corrosion, with t=0 at systemctl stop ployz:

+0.92s kill   old (20261024 image) sig=15        # ployzd replaces it (#3697: -t 1)
+1.93s kill   old sig=9 ; die exit=137 ; destroy
+2.03s create new (20261031 image) ; +2.09s start
+2.11s kill   new sig=9 ; +2.14s die exit=137    # the watcher
+273.1s start new                                  # only after ployzd's restart

Journal:

19:53:42.50 systemd: Starting ployz.service
19:58:12.66 ployzd: Corrosion API error: Corrosion did not start within 270 seconds; inspect `journalctl -u ployz -n 100 --no-pager`
19:58:12.67 systemd: ployz.service: Main process exited, code=exited, status=1/FAILURE
19:58:14.69 systemd: Scheduled restart job, restart counter is at 1.
19:58:15.41 ployzd: INFO started phase="participating" version="0.3.12"

Expected vs actual

• Expected: ployzd sees that the container it just started has stopped, starts it again within a few seconds (as its keeper does once it runs, and as docs/user/troubleshooting/servers.md says), and becomes ready about 2 s later.
• Actual: 270 s of activating, exit 1, then a systemd restart. Apps kept serving: HTTPS 200 throughout, Cloud deploys rc 0 afterwards, 0 acked writes lost from Postgres.

Scope

• A real crash doesn't trigger it. A host kill -9 of the new container's process during the same step was restarted by Docker's unless-stopped policy 0.18 s later, and ployzd was ready 2.85 s after start. Docker treats only a stop or kill through its API as manual, which the restart policy doesn't undo. So this needs an operator or another tool to stop Corrosion through Docker during that window.
• Reproduces on v0.3.12: the same step from 0.3.11 to the published v0.3.12 binaries (ployzd e9833c19) gave the same did not start within 270 seconds, a restart, and Corrosion back at +274.7 s. Not a regression of the sha. Stop a replaced Corrosion after 1 s so an upgrade pauses ployzd ~1 s, not ~7 s #3697 doesn't touch the readiness wait.

Blocks the 0.3.13 cut: no. It's pre-existing (also v0.3.12), needs a manual Docker stop inside a ~1 s window during startup, and apps and data are unaffected.