Skip to content

Cloud picks a partitioned Machine as the Entry, so a deploy lands every new replica on the cut-off side #3675

Open

Description

@nick-potts

Found by a verification swarm on PR #3674, which doesn't touch this path. Not yet confirmed on main.

Repro

  1. Boat --cloud cluster of 3 Machines. Postgres db on a Managed Volume on machine-3, nginx web with 3 replicas, one per Machine.
  2. scripts/verify-cluster partition on <run> machine-1, which cuts machine-1's underlay to both peers. The host and Cloud still reach it.
  3. ployz deploy web through Cloud.

Actual

Cloud used machine-1, the cut-off Machine, as the Entry. It saw machine-2 and machine-3 as Down and started all 3 new replicas on machine-1. machine-2 and machine-3 kept their old ones, so the two sides ran different versions until a deploy a minute after the heal reconciled them. The deploy exited 3 (2 Server observation gaps · 1 deployed · across 1 server).

A ployz exec probe into db, which ran on the reachable side, saw its longest gap (132 s) over the same window. That suggests exec routed through machine-1 too (not confirmed).

Expected

docs/user/services/scaling.md says the new replicas start "on your other servers". With most of the Cluster reachable, an Entry on the majority side would deploy there and leave the cut-off Machine as the observation gap.

Question

Whether Cloud should prefer an Entry that observes more of the Cluster as up. Bet 2 (AP over C) still lets the small side act on its own, so this is about which Entry Cloud picks, not about refusing the deploy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingseverity:lowMinor, cosmetic, copy

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions