Repository navigation
Upgrading or downgrading a Server pauses its daemon ~7 s while the old Corrosion container takes ~5 s to stop #3680
Description
Found by soak-0312 (area upgrade) on the 0.3.12 freeze sha 17c069ea742a51480b050e95d650c57a6c4ca642. Measured cost, not a regression in what users rely on; filed so the pause can be shortened.
Run: Boat --cloud, 3 genuine v0.3.11 Servers (ployzd efb602dc) rolled one at a time to the sha; manifest revision 17c069ea742a51480b050e95d650c57a6c4ca642; dashboard/.verify/run/sdk.log: sdk: hit local ffae9cf7 (17c069ea742a). Resampled on a fresh 2-Server run.
What happens
#3674 pins a new Corrosion image, so the first daemon start after the upgrade (and any downgrade) replaces the Corrosion container. In every such step the old container keeps running about 5.2 s after ployzd rewrites the config, before it stops and the new one starts 0.1 s later (0.1 s poll). ployzd is unavailable 6.0-7.0 s per step, against 0.4-0.6 s for a plain 0.3.11 restart, which never replaces Corrosion. ManagedService::stop calls stop_container(name, None), so the old container gets Docker's default grace.
Per step (probe = verify-cluster probe, whose ployz exec goes through the db host; writer = in-cluster Postgres writer):
| step | probe longest gap | writer longest gap | Corrosion replaced | old Corrosion stop | ployzd down |
|---|---|---|---|---|---|
| 0.3.11 restart m1 / m2 / m3 (db host) | 3.63 / 2.59 / 1.33 s | 0.07 / 0.73 / 0.40 s | no | - | 0.4-0.6 s |
| upgrade m1 / m2 / m3 (db host) | 3.62 / 2.66 / 6.94 s | 0.66 / 0.06 / 0.22 s | yes | 5.2-5.3 s | 6.5-6.8 s |
| resample db host: restart / upgrade / downgrade / upgrade | 3.72 / 7.14 / 6.53 / 6.79 s | <0.5 s | no / yes / yes / yes | 5.1-5.2 s | 0.4 / 7.0 / 6.4 / 6.6 s |
Acked writes lost: 0/59,663 and 0/55,163 (probe), 0/6,180 and 0/6,566 (writer); HTTPS 919/919. Apps and Postgres were unaffected; docs/user/servers/manage-servers.md ("Your apps keep running while they do") held. The pause is on the management path through the Server being upgraded, once per Corrosion image change.
Ask
Shorten the old Corrosion's stop on replacement (e.g. a short explicit stop timeout once the store is synced per commit, or start the new one as soon as the old exits on SIGTERM) so an upgrade's management pause returns to the 0.3.11 level.
Blocks the 0.3.12 cut: no. 0 lost acks, apps unaffected, one-time per image change; coordinator was told.
Activity