Skip to content

fix(k3s): turning workers into masters works; faster failover - #66

Open
thatbagu wants to merge 4 commits into
mainfrom
fix/ha-conversion
Open

thatbagu wants to merge 4 commits into
mainfrom
fix/ha-conversion

Conversation

@thatbagu

@thatbagu thatbagu commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Found by converting a live 3-node cluster (one master, two workers) to three masters, following the docs, then killing the founding master.

Bugs

  1. A converted worker started a second, empty cluster. The founding master was the first by hostname. Here a worker (bulbasaur) sorts before the existing master (meowth), so it got --cluster-init, and everyone was told to join it.
    • New nodes.<name>.clusterInit = true picks the founder.
    • A founder with no etcd data now refuses to start while another master answers. It tries every name (address, hostname, .local) three times, and says what to set.
    • The guard's first version was fooled: the node's DNS was Pi-hole, inside the cluster, so the other master's name didn't resolve. Fixed and re-verified on hardware: it refused, and no second cluster was created.
  2. Joining by name needs cluster DNS. A converted node couldn't join (failed to get CA certs): without an address, nodes join the founder by name, and the LAN's DNS (Pi-hole) runs in the cluster. With several masters, a master without its fixed address now gets a warning on every build and deploy. It's a warning, not an error, so existing configs keep building.
  3. The documented procedure couldn't work as written. It said to convert one worker at a time, but two masters are (rightly) refused. The docs now say to set clusterInit and both workers first.
  4. Longhorn on a converted node kept its CSI driver registered to the deleted node object, so no volume could attach there. The docs now say to restart Longhorn's pods on it.

Failover

With the founding master dead (k3s and every pod killed), the API carried on through the others: etcd keeps its quorum. But apps took about 4.8 minutes to move: that's Kubernetes' default 300 s toleration for a dead node. New k3s.deadNodeTolerationSeconds, 60 by default, set on the API server.

Tests

In edge, all of which fail without the fix:

  • clusterInit picks the founder, and the others join it by address
  • the guard exists only on a multi-master founder
  • clusterInit on more than one master is refused
  • a master without an address is warned about
  • the 60 s toleration is set.

🤖 Generated with Claude Code

thatbagu and others added 4 commits November 21, 2026 18:06
The master that founds the cluster was the first by hostname, so turning
workers into masters next to an existing master sorted after them (a live
cluster: bulbasaur < meowth) made a worker start a second, empty cluster,
and the other nodes join it. nodes.<name>.clusterInit picks the founder
(the existing master); a founder without etcd data refuses to start while
another master already answers, naming the fix. The documented conversion
(one worker at a time) couldn't evaluate (two masters are refused): the
docs now say to set clusterInit and both workers first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r DNS

On a live cluster the guard let a worker found a second cluster: the
nodes' DNS was Pi-hole, inside the cluster, unreachable from the node
while its own k3s restarted, so "meowth" didn't resolve. Each other
master is now tried by address, hostname and <hostname>.local (mDNS),
three times.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a minute

From turning a live cluster's workers into masters and killing one:
- A converted worker couldn't join ("failed to get CA certs"): nodes join
  the founding master by name without an address, and the LAN's DNS was
  Pi-hole, inside the cluster, unreachable from a node whose k3s is down.
  With several masters, each now needs its address.
- With a master dead, the API carried on (etcd quorum), but apps took ~5
  minutes to move (Kubernetes' default toleration): k3s.deadNodeTolerationSeconds,
  60 by default, on the API server.
- Docs: restart Longhorn on a converted node (its driver stayed registered
  to the deleted node object, so no volume could attach there).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Existing multi-master configs keep building (a minor release); every
build and deploy warns until the masters have addresses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant