feat: run benches in a libkrun microVM for predictable numbers - #15
Merged
Merged
Conversation
Host noise (background load, thermal throttling, neighbouring processes)
makes bench numbers drift run-to-run. Add an opt-in `--isolation microvm`
that runs the bench inside an ephemeral libkrun microVM — pinned vCPUs,
fixed RAM, clean rootfs — driven by the `krunvm` CLI. One code path on
macOS (Hypervisor.framework) and Linux (KVM); no libkrun FFI, no new
build-time dependency.
- New `sandbox` module: Isolation::{Local,Microvm}, resolves VmOpts, boots
an ephemeral guest, mounts the workdir 1:1 + a persistent build cache,
execs through /bin/sh with a normalized PATH, and deletes the VM on drop.
- Wired into both bench entry points via their single choke points:
`perf-vs` (both A/B sides run in matching guests) and `aatxe run` (the
shared run_runner all three language adapters funnel through).
- Surfaces only the knobs that move numbers: --isolation / --vm-cpus /
--vm-mem / --vm-image (env-backed, sane per-language image defaults).
- Plug-and-play: `make microvm-setup` installs krunvm, `make microvm-doctor`
checks readiness, `make perf-vs-vm` runs the A/B loop in a guest. Missing
krunvm fails with an install hint, not a cryptic spawn error.
Build caches (cargo/go/npm) persist under ~/.cache/aatxe/vm, isolated from
the host's own target/ (different arch). Unit-tested at the argv/quoting
level; the live boot path needs krunvm installed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Opt-in
--isolation microvmruns a bench inside an ephemeral libkrun microVM (via thekrunvmCLI) instead of on the host — pinned vCPUs, fixed RAM, clean rootfs — so numbers don't drift with host load. One code path on macOS (Hypervisor.framework) and Linux (KVM). No libkrun FFI, no new build-time dependency; we just shell out tokrunvm.Default stays
local— nothing changes unless you ask for the VM.Design
New
sandboxmodule —Isolation::{Local, Microvm}, resolvesVmOpts, boots an ephemeral guest, mounts the workdir 1:1 + a persistent build cache, execs through/bin/shwith a normalizedPATH, and deletes the VM on drop (no leaks on error/panic).Wired into both bench entry points through their single choke points:
aatxe perf-vs— both A/B sides run in matching guests, so the comparison stays fair.aatxe run— the sharedrun_runnerall three language adapters (TS/Go/Rust) funnel through, so they all get it for free.Only the knobs that move numbers are surfaced (everything else = libkrun defaults):
--isolationlocal--vm-cpus(AATXE_VM_CPUS)2--vm-mem(AATXE_VM_MEM)2048--vm-image(AATXE_VM_IMAGE)rust:1/golang:1/node:22Plug-and-play
Missing
krunvmfails with an install hint, not a cryptic spawn error. Build caches (cargo/go/npm) persist under~/.cache/aatxe/vm, isolated from the host's owntarget/(different arch), so only the first run pays the image-pull + cold-build cost.Testing
cargo fmt --check,clippy -D warnings, full suite green (112 tests, incl. 9 newsandboxunit tests on argv construction / shell-quoting / config resolution).runpath still captures/parses/writes and propagates env+cwd.Caveat
The live VM boot path (
krunvm create/start/delete) is unit-tested at the argv level but not exercised live —krunvmisn't installed on my box. Allkrunvminvocation is centralized insandbox.rs(create_argv/guest_argv), so if the installed version names a flag differently it's a one-line fix. Validate withmake microvm-setup→make perf-vs-vm.