e2e: fast VM bootstrap using cached preprovisioned VM images ('boxes'). - #769
Merged
Conversation
klihub
force-pushed
the
devel/e2e/e2e-box-cache
branch
from
August 26, 2026 17:50
6a05cd3 to
a0cd2f2
Compare
klihub
changed the title
kad
reviewed
Aug 26, 2026
klihub
force-pushed
the
devel/e2e/e2e-box-cache
branch
4 times, most recently
from
August 27, 2026 15:42
5c33511 to
fb17881
Compare
klihub
marked this pull request as ready for review
August 27, 2026 16:57
Collaborator
Author
|
@askervin @kad @fmuyassarov There were a couple of problems still lurking there (failure of the 2nd test ran after a VM has been bootstrapped from a cached/stored image snapshots), but those should be now fixed. I reran all e2e tests, and all tests passed on the first run (when the snapshot was saved) and all tests passed on the second run, when the VMs were recreated from the saved snapshots. So this is ready for review now from my point of view. |
klihub
force-pushed
the
devel/e2e/e2e-box-cache
branch
4 times, most recently
from
August 28, 2026 12:16
7a43554 to
5a7bcc8
Compare
Provisioning a VM installs Kubernetes, a container runtime, a CNI plugin and Helm, and initializes a single-node cluster with kubeadm. That is by far the slowest part of a test run, and it is repeated for every fresh output directory although its result only depends on the versions installed. These helpers name, find and produce a "vagrant package"d image of an already provisioned VM. Nothing uses them yet. The topology is part of the key. The hostname of the VM is derived from it, and kubeadm bakes the hostname into the name of the node, into the certificates and into etcd, so a box is only reusable for the topology it was made for. The key also has a hash of the files which provisioning uses, listed in BOX_RECIPE_FILES, so that editing any of them invalidates the boxes rather than silently reusing an image which lacks the change. Editing a test case, or one of the playbooks which deploy a plugin or install a custom kernel after a box has been packaged, does not. Every file in the list has to be readable, or naming a box fails. Renaming or moving one of them without updating the list would otherwise go unnoticed and produce boxes which do not correspond to their recipe. The check runs in vm-box-cache-enabled, outside the command substitutions through which the hash reaches the name of a box: error only exits the subshell of a substitution, so failing inside one would leave a box named after an empty hash. vagrant-qemu only implements "vagrant package" in recent versions, and older ones fail in a confusing way, declaring the action but not shipping the middlewares it uses. Detect the capability rather than requiring a version number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Set e2e_vm_cache=yes to reuse a "vagrant package"d image of an already provisioned VM, refresh to replace one, and no, the default, to keep provisioning from scratch as before. It is off by default because the image contains a live single-node cluster which has to survive being shut down, packaged and booted again on a possibly different host. When a box is used, the VM comes up with --no-provision. The playbook has already run into that box, and kubeadm init cannot run a second time. The only thing the playbook leaves on the host is the file with the kubeadm join command, which nothing reads. When a box is created, packaging shuts the VM down, so bring it back up afterwards. Failing to package is only a warning: the VM which was just provisioned is perfectly usable, the run just does not leave a box behind for the next one. The box replaces the distro box, so it also gets a name of its own. Reusing the name of the distro box would overwrite the downloaded distro image in ~/.vagrant.d/boxes. Also, we need to prevent provision for a VM which came from a cached box. Skipping provisioning with --no-provision on the vagrant up which creates the VM is not enough. Vagrant then keeps the machine flagged as not provisioned, and runs the provisioner on the next vagrant up. That next up comes from the vm-setup of the following test case, which does not pass --no-provision: run.sh runs once per test case, and vm-setup only decides about boxes when the output directory has no Vagrantfile yet. So keep the provisioner out of the Vagrantfile instead. The Vagrantfile is generated once per output directory, which is exactly the lifetime of the decision, and this also covers a vagrant up by hand or a make ssh, both of which would otherwise re-run the playbook and break the cluster. Finally, to avoid a comparable provisioning error, never provision a VM which an output directory got from a box. Recognize a VM which came from a packaged box by the box the Vagrantfile names, and never provision it, whatever this run was asked to do and whether the VM is already up or about to be re-imported. Keep the flag of an existing env file in sync as well. An env file written before the VM was known to come from a box has no flag, so a vagrant up from make up or make ssh would still provision it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
A VM which has just booted from a packaged box, or which was brought back up after being packaged, has a cluster which is still starting up. A test which creates pods before the cluster DNS is up can fail for reasons which have nothing to do with what it tests. Wait for the node to be ready first: kubectl wait fails immediately if it cannot reach the API server, rather than retrying, so waiting for the deployment alone would only report that the API server is not up yet. Both waits use vm-command, so they run at the end of vm-setup. Before the ssh configuration is written and the command output directory exists, vm-command cannot work. CLUSTER_READY_TIMEOUT gives 300 seconds for all of booting a large topology, starting the control plane and the CNI plugin, and getting the DNS deployment available. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
A cached box costs a couple of gigabytes as a file and about as much
again once vagrant has unpacked it, and nothing removes either half.
e2e_vm_cache={cleanup|nuke|drop} removes both halves of every cached
box, printing each of them as it goes, and exits without running any
test. It also picks up the temporary files which an interrupted packaging
leaves in the box cache. Boxes of the distro images are left alone: only
the ones whose name the framework gave them are removed.
The name prefix of those boxes becomes a variable of its own, so that the
cleanup and the code which names them cannot disagree about which boxes
belong to the framework.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
devel/e2e/e2e-box-cache
branch
from
August 28, 2026 12:24
5a7bcc8 to
f6d4278
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: this PR depends and is manually stacked on #768. Review that first.
Use vagrant's
vagrant packagesupport to dump, cache, and reuse provisioned VM images. Image names contain the runtime, the kubernetes version, the helm version, CNI version, and the playbook SHA1 for uniqueness and proper avoidance of reusing images when they should not be.Use
e2e_vm_cache=yesto create and reuse cached images. Usee2e_vm_cache=dropfor cleaning up any cached boxes.