Make the GLKVM a first-class, cluster-independent recovery console

[kelchm]

Context

The GL-RM1PE at 10.32.20.10 is the out-of-band console for the compute rack and the escalation path the talos-rollout skill depends on: "Keep its KVM or physical power control open", and "if the node remains stuck, use the KVM/physical power control to power-cycle it." Node boot hangs during Talos rollouts have already required a manual power cycle.

For a device in that role it is under-provisioned in three ways. Each is small on its own; together they mean the console is hardest to reach exactly when it is needed. They share one appliance and one apply/verify cycle, and they are ordered — the hostname has to settle before a certificate can name it — so they are tracked together.

Appliance state as of 2026-08-23: firmware rm10-1.9.0-release1, Linux 6.1.141 aarch64 Buildroot, 10.32.20.10/24 on eth0.

---

1. Canonical hostname → kvm-compute-rack

The name is currently inconsistent across three places, and authoritative in none:

Where	Value
docs/architecture.md	10.32.20.10 glkvm
Device hostname	glkvm
/etc/kvmd/meta.yaml server.host (shown in the PiKVM UI)	localhost.localdomain
DNS	no record — glkvm.home.kelch.io does not resolve

kvm-compute-rack matches the established ups-compute-rack pattern (role plus physical rack) and survives replacing the GL.iNet hardware, which glkvm would not.

• Add the kvm-compute-rack.home.kelch.io → 10.32.20.10 record to the UniFi resolver
• Set the device hostname and /etc/kvmd/meta.yaml server.host to match
• Update the docs/architecture.md host table and DNS plan, and devices/glkvm/README.md
• Convert the address from its DHCP fixed assignment to an on-device static. architecture.md already flags this: a recovery console should not depend on DHCP being healthy

Keep the directory devices/glkvm/ — it names the hardware, which is the correct scope for vendor-specific kvmd overrides.

2. Let's Encrypt certificate

Both HTTPS listeners — 443 (PiKVM, primary) and 8888 (vendor GLKVM UI) — serve the vendor's self-signed certificate through /etc/kvmd/nginx/ssl.conf:

subject = C=US, O=GLKVM, OU=GLKVM, CN=localhost
notBefore = Jan  1 00:00:14 1970 GMT
notAfter  = Dec 30 00:00:14 1979 GMT

Wrong CN, and a validity window that ended in 1979. Every visit is a click-through warning, which trains exactly the wrong reflex on the box used during incidents.

• Issue via DNS-01 against Cloudflare, reusing the existing SOPS-encrypted CF token model. nginx ships a certbot webroot context at /run/kvmd-certbot/webroot, but HTTP-01 needs inbound reachability this VLAN-20-only device does not have
• Use acme.sh — no ACME client is installed, but bash, curl, openssl, socat, python3, jq, and crond all ship in the firmware, so a pure-shell client works without adding a package manager
• Install to /etc/kvmd/user/ssl/server.{crt,key}, the path ssl.conf already reads. Both listeners pick it up with no nginx change
• Wire renewal to crond and verify it survives a reboot

Issue this independently; do not copy the cluster's cert-manager wildcard. docs/plans/20260620-nas-out-of-cluster-workloads.md sets this precedent for the NAS S3 endpoint — coupling a recovery host's TLS to the cluster undercuts the isolation that justifies the host. It applies with more force here, since this device exists to recover that cluster.

Persistence caveat: the rootfs is an overlay on /userdata/overlay/upper (ext4, 906 MB free). devices/glkvm/README.md already warns that firmware upgrades may replace /etc/kvmd files. Fold the certificate and renewal job into the existing documented rebase-and-reapply procedure.

3. Native tailnet membership

This is the item that actually changes the failure mode.

The lan-subnet-router Connector from #227 advertises 10.32.20.0/24, so the KVM is already reachable off-LAN — but only through a pod running inside k8s-prod. #227 records this as a known caveat: "this ingress runs inside the cluster it exposes." A console reachable only while the cluster is healthy is unavailable in precisely the scenario it exists for.

A native client makes remote console access independent of the cluster, of Cilium/BGP, and of Multus.

• tailscaled needs no install — /usr/bin/tailscale and /usr/bin/tailscaled ship in the firmware. Current state: tailscale status → Logged out, and no tailscaled process is running
• Enroll with its own tag, not tag:k8s. Per docs/plans/20260622-tailscale-operator.md, tag:k8s is owned by tag:k8s-operator and cannot be applied to a separately-authenticated node. Add a tagOwners entry for the new tag
• Do not advertise routes from this device — the Connector already owns 10.32.20.0/24, and a second advertiser invites route ambiguity
• Add a start script under /etc/init.d/ — none exists today, so the daemon will not come back after a reboot
• Verify reachability with the cluster's lan-subnet-router scaled to zero. That is the only test that proves the property this item is for

Acceptance criteria

• kvm-compute-rack.home.kelch.io resolves to 10.32.20.10, and the device and PiKVM UI both report that name
• The address is statically configured on the device, not DHCP-assigned
• Both 443 and 8888 serve a valid publicly-trusted certificate for kvm-compute-rack.home.kelch.io, with no browser warning
• Renewal is automated and verified across a reboot
• The device appears in the tailnet under its own tag and is reachable off-LAN while lan-subnet-router is scaled to zero
• devices/glkvm/README.md covers all three, including reapplication after a firmware upgrade
• docs/architecture.md matches the deployed state

Related

• feat: commit the live netboot.xyz Compose project and GLKVM console helpers #417 — commits the console and HID helpers for this device, currently untracked
• Tailscale operator: complete tailnet setup + merge #152 #227 — the Connector whose in-cluster placement motivates item 3
• docs/plans/20260620-nas-out-of-cluster-workloads.md — the independent-TLS precedent
• .agents/skills/talos-rollout/ — the operational path that depends on this console

Note

The HP EliteDesk UEFI drops HID when a mass-storage interface is present. kvmd-otgconf toggles MSD live on the GL-RM1PE; never run bare kvmd over SSH on this firmware. Do not use kvmd -M to validate config — on V1.9.1 release1 it starts a second daemon instead of doing a validation-only run.