Make the cluster boot reliably: AMD-V contention, boot retries, VirtualBox only - #2
Merged
Conversation
Two failure shapes turned out to be one fault. Either every running VM dies in the same second with Guru Meditation VERR_SVM_IN_USE, or a single node stays "running" but never configures eth1, so vagrant gives up waiting for SSH and the console shows systemd-networkd blocked behind an expedited RCU grace period that never completes. Both are KVM and VirtualBox contending for AMD-V. libvirtd is socket activated, so it can start on its own mid-run, pull in kvm_amd, and take SVM away from VMs that are already up. Measured: libvirtd active at 18:49:06 UTC, all six VMs gurued at 18:49:51 UTC. A vCPU that cannot cleanly enter SVM does not always die -- sometimes it just stops reporting RCU quiescent states, which is the softer hang. Also records what was ruled out, so the same dead ends do not get re-run: host load, x2APIC, kvmclock, the rcu_sched_clock_irq WARNING (it fires on healthy boots too), vmwgfx, and the VirtualBox version itself.
A node that wedges before sshd comes up cannot be reached with vagrant ssh, and the only evidence left was a screenshot of the last 40 console lines. Wire serial console output to logs/<node>-serial.log on the host, and have common.sh add console=ttyS0 to GRUB so the guest actually writes there. The file stays empty on a node's very first boot -- the provisioner has not run yet -- but is complete on every boot after. Switch the graphics controller to vboxvga. With vmsvga the Linux guest binds the VMware vmwgfx driver to VirtualBox's partial SVGA device, and vmwgfx logs that it is on an unsupported hypervisor and likely broken. These are headless k3s nodes, so a plain VGA text console is all they need. This is tidiness, not the boot fix -- see docs/TROUBLESHOOTING-vagrant.md. Raise boot_timeout to 600s so a slow but healthy boot is not failed at vagrant's 300s default.
Even with AMD-V left alone, a seven node cluster is seven chances for a node to lose its boot, and bare vagrant up abandons the whole run on the first one. scripts/up.sh walks the nodes in dependency order, keeping m1 first so etcd is initialised before the others join, and gives each node three attempts. A wedged or gurued guest never recovers on its own and a half provisioned node would fail to join, so a retry destroys and recreates rather than rebooting. On failure it dumps the tail of that node's serial log, which is usually enough to tell a host level abort from a guest problem. Accepts node names to act on a subset, and RETRIES to change the budget.
The AMD-V contention has a concrete source on this host: a vagrant-libvirt lab in ~/repos/vagrant-proxmox/test-lab. Bringing it up takes the virtualisation extensions away from VirtualBox, and every cluster VM dies where it stands. up.sh now reads the kvm_amd/kvm_intel reference count and stops before touching anything, naming the live guests so it is obvious what to shut down. Failing in a second beats failing twenty minutes into provisioning. Correct the troubleshooting doc accordingly. It previously said to blacklist kvm and kvm_amd outright, which on this host would break the libvirt lab -- wrong advice. The rule is one hypervisor per boot, whichever one is wanted, and blacklisting is only for machines that genuinely never use KVM.
The serial log stayed empty after a reboot even though grub.cfg contained console=ttyS0. The box ships GRUB_CMDLINE_LINUX_DEFAULT="autoinstall ds=nocloud-net;s=http://10.0.2.2:.../" and GRUB's parser reads that ';' as a statement separator, so everything appended after it is dropped before the kernel ever sees it. The guest booted with no console= at all. Move the arguments to GRUB_CMDLINE_LINUX, which carries no semicolon, and strip the inert copy the previous version left in _DEFAULT so already provisioned nodes are repaired rather than skipped. Stop hiding update-grub failures behind '|| true', and warn if console=ttyS0 is missing from grub.cfg afterwards instead of assuming it landed. Verified on k8s-w4: after reload, /proc/cmdline carries the console arguments and logs/k8s-w4-serial.log captures the boot from the first kernel line.
Two things kept a fresh cluster from serving a working Forail.
post-cluster-setup.sh minted the self-signed cert for forail.local while the
chart, the ingress and the Cypress config all use forail.lan -- so the cert
never matched the host it was served for. The chart already explains why .lan
is the right choice (Avahi/mDNS hijacks every .local lookup and bypasses
/etc/hosts), so follow it here too.
Add install-forail.sh, which encodes the rest. The published chart defaults to
task.privileged=false, which is correct to ship, but Forail runs project
updates and jobs through podman inside the task pod and podman cannot mount
its overlay storage without those privileges. Every job then dies at 0s with
[graphdriver] prior storage driver overlay failed:
mount /var/lib/containers/storage/overlay: permission denied
while the UI only ever shows a project stuck in "Pending". That cost a full
Cypress run to diagnose: 7 tests failed across happy_path, job_templates,
projects and workflow_templates, all of them downstream of a sync that could
never finish. With the flags on, the same suite is 24/24 specs and 101 passing.
The script is dev-only and says so; this is a throwaway VM lab on a host-only
network, not a template for a real deployment.
scripts/install-forail.sh is the documented way to put Forail into this cluster, it runs on a control-plane node, and it calls helm directly -- but nothing ever installed helm. On a long-lived cluster this went unnoticed because helm had been put there by hand; the first install into a freshly created cluster fails immediately with "sudo: helm: command not found".
libvirt and VirtualBox cannot share AMD-V on one host -- one hypervisor owns the virtualisation extensions per boot, and the loser's guests die with 'Guru Meditation VERR_SVM_IN_USE'. This lab is VirtualBox, so the libvirt provider block goes and the provider is pinned explicitly.
A live KVM guest was not the only way the cluster died. libvirtd merely being *active* was enough, twice, with no guest running at all -- it is socket-activated, so it starts itself. Check the daemon, not just the module refcount. Also stop calling 'vagrant global-status' here: that call was itself one of the things that woke libvirtd up.
The troubleshooting page told the reader to halt a libvirt lab before running the cluster. That was the wrong fix for a machine that does not need libvirt at all -- record masking libvirtd instead, and drop libvirt from the prerequisites and the contributing guide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ten commits, all of them about the same thing — this cluster used to fail to come up, and now it does not.
What is in here
Guru Meditation VERR_SVM_IN_USE, or wedge a vCPU sosystemd-networkdnever finishes and Vagrant times out waiting for SSH. libvirt is now off the host and out of this repo: no provider block, provider pinned tovirtualbox, andscripts/up.shrefuses to start whenlibvirtdis active — not only when a guest holds the module, because the daemon being active was enough to kill a running cluster twice with no guest at all.scripts/up.shbrings nodes up one at a time in dependency order and recreates any node whose boot wedges, instead of failing the whole run.install-forail.shinstalls with the flags this dev cluster actually needs, and helm is installed on the control-plane nodes — nothing did that before, soinstall-forail.shfailed on any freshly created cluster.docs/TROUBLESHOOTING-vagrant.mdrecords the root cause, the measured evidence, and the dead ends that were ruled out.Verified
Cluster destroyed and rebuilt from scratch on this branch:
Ready, zero retries, 13 minutesinstall-forail.sh, 8/8 podsRunning