Repository navigation
Conversation
The master that founds the cluster was the first by hostname, so turning workers into masters next to an existing master sorted after them (a live cluster: bulbasaur < meowth) made a worker start a second, empty cluster, and the other nodes join it. nodes.<name>.clusterInit picks the founder (the existing master); a founder without etcd data refuses to start while another master already answers, naming the fix. The documented conversion (one worker at a time) couldn't evaluate (two masters are refused): the docs now say to set clusterInit and both workers first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r DNS On a live cluster the guard let a worker found a second cluster: the nodes' DNS was Pi-hole, inside the cluster, unreachable from the node while its own k3s restarted, so "meowth" didn't resolve. Each other master is now tried by address, hostname and <hostname>.local (mDNS), three times. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a minute
From turning a live cluster's workers into masters and killing one:
- A converted worker couldn't join ("failed to get CA certs"): nodes join
the founding master by name without an address, and the LAN's DNS was
Pi-hole, inside the cluster, unreachable from a node whose k3s is down.
With several masters, each now needs its address.
- With a master dead, the API carried on (etcd quorum), but apps took ~5
minutes to move (Kubernetes' default toleration): k3s.deadNodeTolerationSeconds,
60 by default, on the API server.
- Docs: restart Longhorn on a converted node (its driver stayed registered
to the deleted node object, so no volume could attach there).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Existing multi-master configs keep building (a minor release); every build and deploy warns until the masters have addresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by converting a live 3-node cluster (one master, two workers) to three masters, following the docs, then killing the founding master.
Bugs
bulbasaur) sorts before the existing master (meowth), so it got--cluster-init, and everyone was told to join it.nodes.<name>.clusterInit = truepicks the founder..local) three times, and says what to set.failed to get CA certs): without anaddress, nodes join the founder by name, and the LAN's DNS (Pi-hole) runs in the cluster. With several masters, a master without its fixedaddressnow gets a warning on every build and deploy. It's a warning, not an error, so existing configs keep building.clusterInitand both workers first.Failover
With the founding master dead (k3s and every pod killed), the API carried on through the others: etcd keeps its quorum. But apps took about 4.8 minutes to move: that's Kubernetes' default 300 s toleration for a dead node. New
k3s.deadNodeTolerationSeconds, 60 by default, set on the API server.Tests
In
edge, all of which fail without the fix:clusterInitpicks the founder, and the others join it by addressclusterIniton more than one master is refused🤖 Generated with Claude Code