Repository navigation
lab topology?: after a heal, 'Container row … on Machine contains document for … on Machine <id>' fails the certificate and ingress passes #3660
Description
Found while reproducing #2925. Labelled lab topology?: seen once, on Boat.
Repro
4-Machine Boat --cloud run on origin/main 842e417 (daemon 0.3.11). Split the underlay 2/2 with ip6tables, deploy web (nginx) ten times during the split, then remove the rules.
Actual
About 8 s after the heal, one Machine's ployzd logged this 18 times within about 2 s:
failed to obtain certificates: invalid Corrosion protocol response: Container row <id> on Machine contains document for <id> on Machine <machine id>
failed to rebuild ingress projection: invalid Corrosion protocol response: Container row <id> on Machine contains document for <id> on Machine <machine id>
The row's machine column read empty while its document named a Machine. It looks like a partially merged row (cr-sqlite merges per column). It cleared on its own, and later deploys succeeded.
Question
Whether a reader should skip a row whose key columns are not merged yet, instead of failing the whole certificate pass and ingress projection. Not chased.
Activity
nick-potts commentedon Oct 31, 2026
More evidence from the #2925 A/B on the same 4 Boat Machines (PR #3674, with the store forced back to main's mode:
chattr +Son,synchronousNORMAL).After a 2/2 split healed, three deploys in a row failed with exit 1 on this error, about 60 to 80 s after the heal:
In main's mode the half-merged row persisted for at least 30 s, because the healing Machine's sync commits took up to 60.8 s. With PR #3674's
synchronous = "full", on the same Machines and the same load, it didn't appear. So this is more than a few seconds of log noise: with slow commits it fails deploys. Whether a reader should skip a half-merged row stays open.nick-potts commentedon Nov 5, 2026
Closing during a backlog triage: Seen once on a lab partition run; the partial row cleared on its own and nobody has reproduced it. Reopen it if you still hit this.