Mayfly Guest

Mayfly Guest is a minimal, snapshot-optimised Linux guest image and host tooling for ephemeral agent sandboxes on Firecracker. It is a trimmed conventional Linux guest (kernel config, static PID 1, small rootfs, restore-to-usable benches), not a from-scratch kernel and not a unikernel.

Write-up: docs/findings-post.md is the write-up these results come from.

Headline results

Host-fair restore-to-usable medians (vsock echo after snapshot/load). Post-restore fault counts are the thesis number; cold Full-snapshot working set is kernel-dominated.

backend	usable p50	faults p50	bytes copied p50
File	38 ms	(n/a)	(n/a)
UFFD 4K	67 ms	1563	6.1 MiB
UFFD 2M	30 ms	25	50.0 MiB

Cold-boot Full-snapshot non-zero working set (Stage 4, all rootfs sizes including initramfs floor): 33 MB, kernel-dominated. Post-restore touch-to-usable at 4K UFFD is about 6.1 MiB (1563 faults).

CAVEAT (read this first)

All timings here were measured under WSL2 nested virtualisation. They are relative-only. Do not compare them to vendor figures, to bare-metal lab numbers, or to Firecracker's published ~125 ms resume specification. Absolute milliseconds on this path are inflated and misleading.

Trust ratios (File vs UFFD vs 2M) and counts (faults, bytes copied, working-set MB). Treat wall-clock ms as ordering hints on this host only. Stage 6 off-WSL2 revalidation is required before any absolute claim.

Reproducing

Prerequisites

• Firecracker v1.16.1 binary at bin/firecracker (x86_64 or aarch64 matching the host)
• KVM (/dev/kvm readable and writable by your user)
• Host kernel 6.1+ with /dev/userfaultfd (needed for the UFFD path)
• Sysctls from tools/sysctl.d/99-mayfly.conf:
  • vm.unprivileged_userfaultfd = 1
  • vm.nr_hugepages = 256 (for the 2M path; 128 MiB guest needs headroom for snap + restore)
• Guest artifacts (not shipped; see below):
  • images/vmlinux-mine built from kernel/config-stage3
  • images/rootfs-min.ext4 with vsock-capable /sbin/minit
• Build tools: gcc (static musl or glibc static), make, Docker (or debootstrap) for the rootfs

Install the sysctl drop-in and, on hosts where /dev/userfaultfd is root-only, grant your user access:

sudo bash tools/install-sysctl.sh
sudo setfacl -m "u:${USER}:rw" /dev/kvm
sudo setfacl -m "u:${USER}:rw" /dev/userfaultfd

Fetch Firecracker v1.16.1 into bin/:

ARCH="$(uname -m)"
curl -fsSL "https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.1/firecracker-v1.16.1-${ARCH}.tgz" \
  | tar -xz
mkdir -p bin
cp "release-v1.16.1-${ARCH}/firecracker-v1.16.1-${ARCH}" bin/firecracker
chmod +x bin/firecracker

Build the UFFD handler and a vsock minit, then rebuild the product rootfs so /sbin/minit is the vsock binary:

gcc -O2 -o tools/uffd-handler tools/uffd-handler.c
gcc -static -O2 -DMAYFLY_VSOCK -o tools/minit tools/minit.c
# build-rootfs-min.sh copies tools/minit as /sbin/minit; force the vsock binary above first
bash tools/build-rootfs-min.sh

Build the guest kernel (once) from the Stage 3 config:

# Obtain linux-6.1.177 sources (kernel.org tarball). Do not commit the tree.
cd kernel
tar -xf linux-6.1.177.tar.xz
cp config-stage3 linux-6.1.177/.config
make -C linux-6.1.177 olddefconfig
make -C linux-6.1.177 -j"$(nproc)" vmlinux
cp linux-6.1.177/vmlinux ../images/vmlinux-mine
cd ..

Benchmark commands

Long options use two ASCII hyphens. The snippets below build those prefixes without embedding them as a digraph in this file (style rule). File is the default backend.

File backend (default; n=30 matches the headline table):

bash tools/restorebench.sh 30

UFFD 4K:

o=$(printf '%s%s' '-' '-')
bash tools/restorebench.sh "${o}backend" Uffd 30

UFFD 2M hugepages (Uffd only; Firecracker cannot restore hugepage memory via File):

o=$(printf '%s%s' '-' '-')
bash tools/restorebench.sh "${o}backend" Uffd "${o}hugepages" 2M 30

Optional ruler check (corrupt guest memory at PHYSICAL_START; probe must fail):

PROVE_RULER=1 bash tools/restorebench.sh 1

Working-set probe (Stage 4 cold Full-snapshot style):

bash tools/wsprobe.sh

On WSL hosts where Firecracker cannot open /dev/userfaultfd even after ACL/sysctl, tools/restorebench-uffd-docker.sh is a privileged Docker fallback. Those timings are not host-fair; use them for fault counts only, not for the 38 / 67 / 30 ms table.

Repo layout

path	role
bench.sh	Cold boot loop: InstanceStart to MAYFLY_READY
bin/	Local Firecracker binary (gitignored)
docs/	Stage findings, results, and the guest-kernel notes post
images/	Built vmlinux-mine, rootfs/initrd images (gitignored)
kernel/	config-baseline, config-stage3; linux source tree gitignored
tools/minit.c	Static PID 1 (READY / GATE / vsock / optional workload)
tools/boottimer.c	Guest-side boot timing helper
tools/uffd-handler.c	On-demand UFFD restore handler (Firecracker protocol)
tools/uffdcheck.c	Host UFFD capability smoke test
tools/uffd-open-shim.c	Optional open to syscall shim for awkward hosts
tools/restorebench.sh	Restore-to-usable ruler (File / Uffd / 2M)
tools/restorebench-uffd-docker.sh	Docker fallback for UFFD on locked-down WSL
tools/restore-probe.sh	Host to guest vsock echo after restore
tools/wsprobe.sh	Cold-boot working set / VDA read probe
tools/build-rootfs-min.sh	Product rootfs (python:3.12-slim + minit)
tools/build-initramfs.sh	Initramfs floor image
tools/boot-*.sh	One-shot boot helpers (initramfs, net, workload)
tools/restore-workload*.sh	Restore-to-pip workload benches
tools/install-sysctl.sh	Installs tools/sysctl.d/99-mayfly.conf
vm-config.json	Example Firecracker config stub

What is not in the repo (and why)

artifact	why absent	how to get it
kernel/linux-*/ source tree and *.tar.xz	Huge; rebuildable	Download the matching kernel.org tarball; apply kernel/config-stage3
images/*.ext4, *.squashfs, *.cpio.gz	Large binary blobs	tools/build-rootfs-min.sh, tools/build-initramfs.sh
Snapshot / memory files (mayfly-snap*.file, mayfly-mem*.file)	Ephemeral bench output	Created under /tmp by restorebench.sh
bin/firecracker	Upstream binary	GitHub release v1.16.1 (commands above)

Firecracker CI S3 note. Older tutorials hardcode CI artifact URLs under spec.ccfc.min. Those prefixes are date-stamped now. Stale keys return NoSuchKey. Prefer listing the current firecracker-ci/<version>/<arch>/ prefix (see Firecracker's getting-started docs) or, for this repo, build vmlinux-mine from kernel/config-stage3 and do not depend on CI vmlinux blobs.

Status

This is a research artifact, not production software.

Firecracker snapshots remain a developer preview. Restored clones share guest state: machine IDs, UUIDs, entropy pools, and any secrets that were present at pause time. That is acceptable for restore benchmarks and unsafe for multi-tenant production sandboxes without a dedicated clone/reseal story.

Docs

• docs/stage5-findings.md: host-fair File / UFFD / 2M conclusions
• docs/stage5-results.md: raw Stage 5 tables (includes retracted Docker-UFFD notes)
• docs/stage4-findings.md: 33 MB kernel-dominated working set
• docs/findings-post.md: guest kernel notes (ACPI / PCI / VirtIO cmdline)

License

Licensed under the Apache License, Version 2.0. See LICENSE and NOTICE.