Repository navigation
server ls keeps a healed Machine 'suspect' for 10+ minutes after a partition alone #2506
Description
After a network partition alone (no power cut) heals, the other Machines keep showing the healed Machine as suspect in server ls for more than 10 minutes. Meanwhile the routing layer treats it as healthy. This is a status misreport. Related to #1423, which saw this after a power cut. There, a partition with no power cut converged in 39 s.
Where
- A Boat
verify-clusterrun with 3 Machines (20261020T055636-d2ee370bee, head of feat(core): route on copy rotation and drive Deploys through it #2343 at bbd1395). Before the partition, the cluster had about a dozen Deploys and earlier partition/heal cycles on machine-3. - R3 (feat(core): route on copy rotation and drive Deploys through it #2343) doesn't change the membership states or
server lscode (ployzd/src/corrosion/admin.rs,handlers/server/show.rsanddomain/machine.rshave no suspect/membership lines in its diff). So this is expected on main too.
Timeline (UTC)
- 06:24:07:
verify-cluster partition on machine-3, during a Deploy. - About 06:24:44: the partition was healed. Corrosion logs
Rejoined the cluster, thenCurrent node is considered ACTIVEat 06:24:49. - 06:29:23 and 06:35:01:
server lsstill shows machine-3 assuspectfrom machine-1 and machine-2.
server ls membership (06:29:23)
machine-1 view: [{"n":"machine-1","membership":"up"},{"n":"machine-2","membership":"up"},{"n":"machine-3","membership":"suspect"}]
machine-2 view: [{"n":"machine-1","membership":"up"},{"n":"machine-2","membership":"up"},{"n":"machine-3","membership":"suspect"}]
machine-3 view: [{"n":"machine-1","membership":"up"},{"n":"machine-2","membership":"up"},{"n":"machine-3","membership":"up"}]
At 06:35:01 it was the same: ["up","up","suspect"].
ReadRoutes at the same time
The Machine Down set was [], so routing treated machine-3 as healthy. The next Deploy brought machine-3's new copy In and routed it, and all three Machines served.
Expected
Once the heal is done and Corrosion has rejoined and synced, server ls stops showing suspect in about the time #1423's partition-only case took (around 40 s). If membership really stays Suspect in Corrosion, server ls should say why instead of showing a stale state.
Evidence, kept locally: server ls dumps, the Corrosion log of machine-1 and machine-3, and post-heal journals (/tmp/swarm-R3/owner/lanes2-a/head/suspect-after-heal.txt, corrosion-machine-*.txt, journal-machine-*-postheal.txt).
Activity
nick-potts commentedon Oct 22, 2026
Same symptom at v0.3.0-beta.7 with no partition at all: a Corrosion crash-restart on one Server is enough (soak
corrosion-5, Boat run 11cefe48e2, 3 Servers, standalone).machine-2's overlay address answers ping from machine-1, the WireGuard handshake is recent, and Corrosion rows keep converging (identical table hashes on all three Servers), so only the SWIM view is stale.
docker exec ployz-corrosion corrosion cluster membership-stateson machine-1 shows the peer"state": "Suspect","incarnation": 0, and the corrosion log never records a suspect or refute event.An earlier occurrence on the same run (machine-1's Corrosion crash at 00:14, then machine-2 powered off at 00:27) spread: both machine-1 and machine-2 reported healthy machine-3 as
suspectfor 8+ minutes, while machine-3 saw everyoneup. It cleared only when machine-3 itself wentdownand back during a later partition.suspectisn't defined anywhere indocs/user/(server lsMEMBERSHIP column). Evidence: soak evidencecorrosion/c9-*,corrosion/c9b-suspect-repro.txt,corrosion/membership-watch.log.nick-potts commentedon Oct 26, 2026
Another trigger for the same slow heal, from soak-036 (main
6c849d95a3+ 0.3.6 bump, Boat, local Cloud withSERVERS=2 DAEMON=checkout):verify-cluster update <run> --daemon machine-2=0.3.5at 04:02 UTC (machine-2: 0.3.6 → 0.3.5, both daemons restart).updateexits 1 withmachine-1: expected 2 reachable Machines.ployz server lsshows machine-2suspect. machine-1's journal logsasked peers Corrosion no longer hears whether this Machine was forgotten listed=[machine-2]every minute.upand the log line had stopped.So this is not a version-mix break (deploys worked the whole time), just the same ~10 minute
suspectafter a daemon restart. Not checked on v0.3.5 ↔ v0.3.5.nick-potts commentedon Oct 28, 2026
soak-038 (main b1e048a + local 0.3.8 bump, Boat, local Cloud, mixed 0.3.7/0.3.8): a related case, but here the Server stuck at 'suspect' isn't the one that was faulted.
--sport 51820 -m length --length 1001:65535 -j DROP) ran on machine-3 (0.3.8) from 03:58 to 04:12. Corrosion marked machine-3 Down at 04:11:46 and Up again at 04:12:40-50.suspectfor 18+ minutes:server lsthrough Cloud and via--connectto each of them, at 04:18, 04:19 and 04:30. Through machine-2 itself all three wereup.asked peers Corrosion no longer hears whether this Machine was forgotten listed=[MachineName("machine-2")].Cluster Memberslisted every peer with RTTs, and deploys across all 3 Servers still succeeded.Blocks the 0.3.8 cut: no. It didn't block a deploy, and it's not a regression as far as I can tell.
nick-potts commentedon Oct 31, 2026
soak-0313 (freeze bc82b8f, Boat
--cloud, 3 Servers on this sha's ployzd, same Corrosion image as v0.3.12): the same stucksuspect, after a Corrosion SIGKILL this time, and again it names a Server that wasn't faulted.docker kill -s KILL ployz-corrosionran on machine-2. ployzd's keeper had it running again 0.7 s later. machine-1's Corrosion was not touched; it had last been killed at 19:05:18 and had recovered fully by 19:06:04.suspect, and it stays that way:server lsthrough each Entry at +100 s, then at 19:15:30, 19:15:48 and 19:16:24. Through machine-1 itself all three areup.MembershipStatesat 19:16:24 matches: machine-2 and machine-3 hold machine-1's actorSuspectat incarnation 0, and machine-1 holds both peersAlive. Ping over the overlay from machine-2 to machine-1 shows 0% loss. Nothing gets refuted, and nothing times out to Down. It cleared on its own between 19:16:24 and 19:18:04, so 7 to 8.5 min after the kill.docker stop ployz-corrosionon machine-2 at 19:07:31 left one Entry's read empty during the 5.5 s stop, and every view wasupby +8 s. Each case has one sample.Why it matters for 0.3.13: #3697 makes every Corrosion replacement end in SIGKILL, 1 s after SIGTERM, so a replacement now looks like the kill above, not the graceful stop. I didn't run it on v0.3.12 daemons. The kill happens at the Docker level, and v0.3.12 pins the same Corrosion image and config, so this should reproduce there too; this issue already shows it on 0.3.7 and 0.3.8. Blocks the 0.3.13 cut: no. It's a status misreport, with routing and deploys unaffected.