Skip to content

test: a fresh cluster bootstrapping in a VM, and upgrading from the last release - #63

Open
thatbagu wants to merge 5 commits into
mainfrom
test/vm-bootstrap
Open

thatbagu wants to merge 5 commits into
mainfrom
test/vm-bootstrap

Conversation

@thatbagu

Copy link
Copy Markdown
Owner

Two VM tests from the readiness checklist.

vm-bootstrap: a fresh cluster bootstraps by itself

Argo CD syncs every layer from the rendered branch, which the VM serves itself over git's HTTP protocol. That's no internet, the internal CA, and digests stripped for the preloaded images. The layers are MetalLB, Traefik with the edge proxy, cert-manager, Longhorn, Pi-hole, the VPN and a web app.

It passes only when:

  • every Application is Synced and Healthy, with nothing touched by hand
  • the app answers HTTPS through the edge proxy and the gateway
  • Longhorn's one-time manual sync succeeded on its own.

The three bootstrap bugs found on real hardware are what it targets:

  • the sync started before Argo CD's AppProject existed
  • the gateway's health check stalled the bootstrap
  • the sync operation dropped CreateNamespace and retry.

Its 28 images reuse pins other groups already have, and nix run .#update keeps the vm-bootstrap group current. It needs an 8 GB VM, so it was run only in CI.

vm-upgrade: upgrading from the previous release

The previous release's render (v1.24.2, pinned by hash) is applied to a cluster for real. Then the current render is applied over it: CRDs for real, everything else as a server-side dry run, the way Argo CD would. That catches what only breaks on upgrade: fields that can't change on existing objects, and CRD versions still stored but no longer served.

Run locally: it passes. A sabotage run, with a changed Deployment selector, fails as it should: Deployment "pihole" … spec.selector … field is immutable, 1 of 200 objects rejected. CONTRIBUTING gets a "Releasing" step to bump the pin.

🤖 Generated with Claude Code

thatbagu and others added 5 commits November 21, 2026 15:48
…ast release

- vm-bootstrap: Argo CD syncs every layer (MetalLB, the gateway and edge
  proxy, cert-manager with the internal CA, Longhorn, Pi-hole, the VPN, a
  web app) from the rendered branch, which the VM serves itself over git's
  HTTP protocol; passes only when every Application is Synced and Healthy,
  the app answers through the gateway, and Longhorn's first sync went
  through by itself. The three bootstrap bugs found on real hardware
  (AppProject race, gateway health, the sync operation's options) are the
  kind it catches.
- vm-upgrade: the previous release's render applied for real, then the
  current one over it (server-side dry run, CRDs for real): catches fields
  that can't change on existing objects and dropped CRD versions. Checked
  by sabotage: a changed Deployment selector is rejected ("field is
  immutable").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k3s imports every preloaded image (29 here) before the kubelet starts,
which took over 5 minutes on CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Argo CD lists refs with go-git, which doesn't speak git's plain HTTP
protocol ("failed to list refs: unexpected EOF"). A small bridge to git
http-backend serves the smart one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… render

vm-bootstrap listed its 13 apps by hand, duplicating the render: a core
app added later would have been missing from its preloaded images. Groups
may now leave apps out, and get every app in their render (and are rerun
for any change to it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The render's directories are symlinks into the Nix store; committed as
such, Argo CD refuses the repository ("out-of-bounds symlinks").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant