Conversation
The flake, its lock, the d2b module sources and the host binaries are declared label inputs, so a host-binary change moves the image's action key. Nix store, substituters, build-users-group, sandbox and HOME are configured inside the action, so hermeticity comes from nix's own configuration rather than the calling shell. The output is the bootable image - kernel, initrd and a qcow2 root disk with a pinned filesystem UUID and build clock - not a copy of the toplevel symlink tree, which Bazel rejects as a tree artifact. The manifest keeps the host-tool inventory and the per-check invocation fields the lane reproduces. The emulator comes from the existing pinned nix package extension rather than a new Bazel module, keeping emulator and guest at one nixpkgs revision. The recipe's closure upload retires: a network side effect in a build action would make the image uncacheable. The environment-variable handoff stays until the make recipe collapses, so the live lane keeps receiving Bazel-built binaries.
The reusable NixOS node configuration - the daemon host, the nested cloud-hypervisor host, the shared base config and the acceptance-unit list - moves byte-identically out of the runNixOSTest fixture library into a module the lane evaluates, so a fixture can be deleted when its check ports without taking its guest declaration with it. The library keeps only the driver-coupled half, which dies with the fixtures. The eight fixtures that used the shared node are repointed at the re-homed module with no re-export shim: each diff is the import binding plus the node-constructor qualifier, and no testScript line changes. The other four never used it and are untouched. Per-check invocation is read off the evaluated config rather than restated, so the image action and the harness cannot drift from what the guest is actually configured to be. Carried here so the history is not broken: the image action now imports the re-homed module, and the stale fixture-library entry is dropped from its declared inputs so editing a driver diagnostic no longer invalidates a cacheable image the guest does not depend on. Both edits are behaviour-preserving - the rebuilt image carries the identical toplevel, host-tool bundle and guest sizing.
Adds a harness crate that reproduces each check's emulator invocation from the evaluated node configuration, enforces the host virtualization preconditions before anything is copied or spawned, and refuses at boot any writable device that cannot carry an internal snapshot. The lane suite is its own, and the crate deliberately holds no test aggregate so the repository's test census does not force it into the main package suite. The image action now evaluates both node shapes and emits a fully resolved manifest, so the invocation is read off the evaluated config rather than restated anywhere. Five defects were found by running the guest, not by reading it. The node's device-option list is shell text that only the module's run script word-splits; the monitor socket path exceeded the kernel's sun_path limit on the long runfiles path; the copied root disk inherited read-only mode from the read-only graph output; the image stopped producing the kernel and initrd its own manifest names, which an already-passing unit test had been asserting; and the guest activation script used $$ expansions, which a Nix indented string leaves as literal double dollars, so the unit polled a service named after its own pid. The activation stall itself was the lane dropping the node's declared networking, leaving the guest without virtio_net, which the daemon refuses to start without. Monitor and spawn failures now carry the emulator's stderr and the assembled command line, and an activation timeout carries the guest's systemd ordering state. Each of those is what made the next defect findable.
…face The guest image imported nixpkgs' qemu-vm.nix alone, so it had no way to be commanded: no backdoor service, no console, no channel of any kind. It now imports the same test instrumentation the nix lane's guest ran, leaving testing.backdoor at its default. The launcher declares the virtio-serial console on the command line and binds the host socket before spawn, because backdoor.service requires dev-hvc0.device and a unit whose device is absent at boot is not reliably restarted once it appears. That channel lets the harness re-provide the legacy driver's guest control surface, so every check that has not been ported keeps executing its own assertions and keeps gating the lane. The surface is the driver's, down to its refusal strings and its log lines, and the diagnostics prelude is one file both surfaces read, so a ported check and an unported one report identically. A console rather than a forwarded ssh port because no fixture uses ssh, a forwarded port does not survive snapshot restore, and the console needs no sshd, host keys or key auth in the guest. Corrects a doc comment on reserve_loopback_port that claimed the lane forwards the guest's ssh port. It forwards no port and never did. pkgs is bound in guest-image.nix's let rather than read inside laneGuestModule: the module system's own pkgs is resolved by asking the configuration being built, so a module reaching for it during that fixpoint is infinite recursion.
The lane had two guest images and eleven checks, and nine of those checks
declare their own nodes.machine. Each image is now built from its own
fixture, so the node, the machine size, the drive layout and the evaluated
assertions all come out of one file - there is nowhere left for a second
copy of a check's node to drift. The pool groups images by the emulator
invocation they declare, is sized from the distinct-invocation count
against a budget the guest configurations declare, and refuses an
unsnapshottable drive before the pool is built rather than running a check
without restore.
This lands with the lane running and failing at one known blocker, and the
blocker is recorded rather than papered over. Every lane guest mounts the
host Nix store over 9p, and QEMU refuses an internal snapshot while a
VirtFS export is mounted:
the emulator monitor refused savevm 'lane-base': Error: Migration is
disabled when VirtFS export path '.../xchg' is mounted in the guest
The export cannot be dropped. Setting the manifest's shared directories to
[] and booting the built image directly reproduces a kernel panic at
activation - mount: /sysroot/nix/.ro-store: special device nix-store does
not exist - because the guest is booted from a host store that is not
there. Settled: the lane takes external qcow2 overlays per pool member
instead of HMP savevm, which QEMU permits with virtiofs mounted, leaves
every guest byte-identical to the one the fixtures were written against,
and keeps the rule that a check never runs without restore. R6's
"internal snapshots" wording no longer describes the mechanism and is
corrected in the plan's decision record.
Four defects in this wiring are fixed here, none of them visible without
running the build. The test rule's src named the genrule rather than its
output file, which failed every target at analysis. The genrule's
argument parsing had lost its substituters positional. The declared
source set was missing three trees the guest evaluation reaches by
relative path - the manpages and shell completions the d2b module tree
installs, and the packaging tree the observability modules read - and the
two shape images had masked all three by never importing those modules.
And the fixture capture imported a { pkgs, self }: function without
calling it, so every check looked like one that declared no assertions.
The lane boots a pool of guests, snapshots each member against every writable device, and runs each check against a member restored from that snapshot. Per-check selection works, and the nested-guest members are retired rather than restored. Full lane: 768.2s, all four invocation groups admitted against the declared budget with nothing dropped. Nine of the eleven checks pass. Two fail, and both are assertion timeouts inside the guest rather than restore failures - the restore had already completed and the check ran on the restored guest. device-worker-launch never reaches tpm-worker-ready, with Endpoint/tpm-ctrl-tpm0 reporting "phase":"Failed" while the GPU and volume provider are Ready. virtiofsd-volume-runtime never reaches volume-realized. Both ran in a four-group wave and neither was re-run alone, so contention sensitivity is unresolved. R13's "indistinguishable from the current driver" is not claimed: the diagnostics shape is right, but no side-by-side against the nix driver was produced. Snapshots are external qcow2 overlays, not internal. Internal snapshots are unavailable to a guest with a VirtFS export mounted: HMP savevm and QMP snapshot-save are both refused with "Migration is disabled when VirtFS export path ... is mounted in the guest", and blockdev-snapshot-internal-sync is accepted but writes a disk-only snapshot with VM_SIZE 0 B that the emulator then refuses to load. A disk-only restore leaves the guest's page cache, tmpfs and every service's state behind, so the lane resets the guest onto the restored disk; the reset is the price of a member being the fresh boot its check was written against. Six defects, all found by running the lane, each invisible until the one before it was fixed. Block node names exceeded the emulator's 31-byte limit, and one check's name alone exceeds it. target_for resolved only against the launch node, so a device could not be mapped back to its backing file once a layer sat on top. The dirty-layer node was dropped unconditionally after a detach that had already freed it. Re-attach put a device path in the id slot, which is not an id. The unplug wait matched a stale DEVICE_DELETED left in the monitor buffer, so a second restore proceeded with the device still attached. And a node created by blockdev-snapshot-sync is dropped the moment nothing references it, so the restore attached to a collected, read-only node instead of re-opening the layer. The pool does not buy wall clock. The reported 0.3-0.6s restore is host-side block-graph work plus issuing the reset; the guest's reboot onto the restored disk is awaited outside that timed region and costs about what a fresh boot costs. KTD5's speed rationale is not supported by what was measured here. The image action no longer escalates a user the nix daemon already trusts: a Bazel action has no terminal to answer a prompt on, and the first trust probe discarded nix store ping's output because it goes to stderr.
The package, its crate and Bazel target names, the binary, and the D2B_VM_HARNESS_* environment prefix are renamed to d2b-test-vm-harness and D2B_TEST_VM_HARNESS_*. The prefix is not cosmetic: it was observed set in both generated lane runners and read by the live harness process in /proc/environ, because an unread variable yields a default rather than an error, so a passing lane could not otherwise have distinguished a working rename from a cosmetic one. Not renamed, because they are a different concept that shares the prefix: d2b_vm_* are per-VM-instance Prometheus metric names in d2bd-runtime, and the same names key four Grafana dashboards. Renaming those would silently break live metrics and every dashboard, with nothing in CI catching it before a release. D2B_VM_CHECK in the Makefile selects VM checks and is not this package; the legacy lane's D2B_VM_SSH_KEY_* are per-VM as well. CHANGELOG.md is released history and docs/plans/ are prior artifacts. The lane's per-check table was not re-measured: the verification run was damaged by a concurrent cleanup that removed its working directories underneath running guests. The build, clippy and crate policy gates are green, the rename is complete across every tracked file, and the two failing checks are a separate known defect in the restore path - one that this table would not have exonerated.
ActiveGuest::restore detached and re-attached every writable device in QEMU's query-block order, which is not the order the launch attached them. The kernel names a block device by creation order, so on every restore a two-disk member's root disk and its option-list state disk swapped /dev/vda and /dev/vdb. The guest's volume-local markers anchor a volume root by (device, inode) and fail closed on mismatch, so after any restore every marker disagreed with the live tree - RootReplaced, then volume-layout-effect-failed forever, the Volume row never reaching Ready, and the waits timing out at 180s. The fresh boot wrote the markers while /var/lib/d2b was /dev/vda; the restored boot saw the same tree at /dev/vdb. daemon-smoke carried the same swap in its console log and passed only because it never waits on a Volume, and single-disk members cannot swap at all, which is why two checks failed while the lane looked mostly green. Restore now ranks each device by SnapshotPoint::launch_rank - declared drives in manifest order, then the node's own option drives - and orders them with reattachment_order, a stable sort, so detach, drop, freeze, overlay and re-attach all happen in that one order. Two unit tests pin the ranking and the seam against a deliberately mismatched report. The equivalence marker now reports each /sys/block/vd* name and size, so the gate compares the guest's device identity between a fresh boot and a boot from the restored disk. That is what makes a regression of this defect fail at the gate in milliseconds instead of as a 180s wait; the marker grew from 307-319 bytes to 350-375 and all eleven members matched in two post-fix runs. What the gate still does not prove is written in its doc comment: it compares the configuration surface as a booted guest reports it, not RAM or in-flight state, because the restore is disk-only. The stale monitor test fake no longer drops device_del's ack, so wait_for_event no longer blocks on an event it already consumed. The clear() it depends on is untouched - discarding events that predate the wait is what stopped a restore matching a stale DEVICE_DELETED. virtiofsd-volume-runtime goes from a 240.9s failure to 68-110s passing. device-worker-launch still fails, deterministically, at the same TPM assertion with its Volume now Ready - a second, independent cause this commit does not address - and one nested-guest member failed in one of two runs. Neither is papered over here.
The first of eleven ports, and the pattern the rest follow. The check's twenty-eight guest assertions move from the fixture's testScript into packages/d2b-test-vm-harness/src/checks/daemon_smoke.rs, in the same order, with the same commands and the same 30s and 180s bounds: 17 succeed, 1 wait_for_file, 4 diag_unit, 5 stage, plus the acceptance census as a set comparison and the two restart/cgroup-survival steps. GuestControl grew the stage and diag_unit primitives the fixture's diagnostics prelude used, so the log reads the way it read. Deleting the fixture is what forced the design. A fixture carried both a check's guest configuration and its testScript, and the lane identified a check by the check.py the image wrote for it - so a check whose assertions are Rust had no way to appear in the lane at all. The manifest's CheckRecord now carries an `assertions` field naming which side holds a check's assertions, the image action writes it, and the harness dispatches on it: Rust goes to the crate's table of ported checks, Python takes the unchanged check.py path. Both directions fail closed - an image that claims Rust with no module behind it fails the lane by name, and a manifest that claims Python with no script still fails on the read - so a half-finished port is a red lane, never a check that quietly ran nothing. daemon-smoke's guest configuration moves to host-integration-node.nix, next to the node shapes it was written against, so deleting the fixture loses no machine size, drive layout or device option. Counts the port must satisfy: 17 succeed, 1 wait_for_file, 4 diag_unit, 5 stage, before and after. The lane's own plan carried 78 em-dashes and failed the source-hygiene gate that make check-tier0 runs, which no other tracked document in the repository does; they are now the ASCII form its superseded sibling uses.
…r binds under A zone-native Device owner declared only the swtpm state and marker rows. It did not declare the per-guest runtime tree, and nothing else in the tree provisions it: a legacy VM gets /run/d2b/vms/<vm> from its path:vm-run:<vm> row, created by tmpfiles at activation, and a zone-native host creates no such row and has no tmpfiles rule for the parent. The broker's socket grant walks the path down to the worker's socket and refuses an absent ancestor - an absent ancestor yields no traverse grant, which the caller turns into "runner path ancestor is absent: /run/d2b/vms" and a refused spawn. That is why Process/swtpm-tpm0 and both Process/gpu-* rows were refused at launch while the one-shot flush was admitted: the flush is the one role whose posture binds no runtime socket, so it is the one role the grant never ran for. The state Volume was already Ready throughout, which is what ruled out the state directory as the cause. Same path and same id as the legacy row, so this adds one vocabulary for one directory rather than a second name for it. Separately: a refused forwarded invocation reached the journal as a closed-set code and nothing else. The detail is redacted operator-facing text by contract and existed only in the failure this discarded, so a provider's actual reason for refusing a spawn was unrecoverable from any log - which is why this took a direct guest probe to place rather than reading the daemon's own account of it. It is now written down where it is dropped, without changing the wire envelope the broker speaks.
…usal-diagnosis fixes
The console bounded its read while waiting for the guest's shell and then cleared the bound, so every command read afterwards ran against an unbounded socket. The guest's own `timeout` bounds the command's execution; the host's wait for the answer is a different wait, beginning when the command is sent and ending when the answer comes back, and that one had nothing. A guest that dies mid-command, or a console that simply goes quiet, then left the host reading a socket that would never carry another byte. The lane's group threads parked in that read with their emulators already gone, and the work directories whose cleanup hangs off that path were never removed - which is what a full lane run looks like when it finishes its checks and never reports. A guest with a nested guest inside it is the one shape that can stop answering quietly, because its inner guest takes the console with it. The host's wait is now the command's own bound plus a slack covering the guest's teardown of the command, the base64 framing and the status round-trip, and a console that does not answer inside it fails with a message naming the command that went unanswered. That is the difference between a check that failed and a lane that stopped producing results. A test pins it against a console whose peer never replies. Under test the slack is short so the test does not wait the production wait; what it pins is that a bound is applied at all, because without one it does not return. Also carries three diagnostics primitives added alongside the in-flight check ports - `diag_run`, `diag_file` and `announce`, the last of them standing in for a fixture `print`. They are part of that work and are not verified by this commit's own test run; the ports that use them are where they are exercised. The previous commit also swept in a staged fixture deletion belonging to that work. It is restored by the port commit that follows this one.
The check's assertions move from the fixture's testScript into packages/d2b-test-vm-harness/src/checks/bridge_isolation.rs, in the same order, with the same command text and the same in-guest commands: 32 succeed, 2 fail, and the three `bridge -d link show` reads whose output substrings are the isolation assertion (work-l1 must not be isolated, work-l10 and work-l11 must be). No bound and no stage changed: the fixture declared none, and this port declares none. The fixture declared a plain NixOS node - it never wanted the d2b daemon host - so its two packages and its stateVersion move to nix/test-support/host-integration-node.nix as d2bBridgeIsolationNode, next to the reusable nodes, and its guest is built from that declaration by name. The machine size, the disk and the emulator invocation are the QEMU VM module's defaults in both places, because the fixture declared none of them either. The fixture file itself was removed by 8ad9e01f1, whose broad stage took a deletion this port had staged. This commit is the half that makes the check run again, so the tree is red between that commit and this one. Counts the port must satisfy: 32 succeed, 2 fail, 3 output-substring assertions, before and after.
The check's assertions, and the five helper reads its testScript defined, move into packages/d2b-test-vm-harness/src/checks/guest_agent_cap_confinement.rs in the fixture's own order: 5 stage, 1 wait_for_unit (multi-user.target, 180s), 1 diag_unit (d2b-test-guest-agent.service, 60s), the two succeed calls that start the namespace and the agent, and twelve assertions over the live state - the agent's namespace inode equals the declared one, it does not share the host namespace, its effective set carries all three declared capabilities and nothing else, its main process is inside its own control group, starting it added no network capability to a process that was already in the host namespace, and its service left no capability-bearing process there. The helper reads the fixture's Python performed - readlink of a namespace, stat of a netns, CapEff out of /proc/<pid>/status, the host namespace's per-process capability table, and a unit's process identities - are functions of the module with the fixture's own command text: nine of them run once, and the control-group walk adds one pid read per process. The fixture declared a plain NixOS node, so its unprivileged user, its two units and its iproute2 move to nix/test-support/host-integration-node.nix as d2bGuestAgentCapConfinementNode. Its fixture file was removed by 8ad9e01f1's broad stage; this commit adds the Rust that replaces it. Counts the port must satisfy: 5 stage, 1 wait_for_unit, 1 diag_unit, 2 succeed, 12 assertions, plus the helper reads, before and after.
The check's assertions move into packages/d2b-test-vm-harness/src/checks/guest_shell_service.rs in the fixture's own order: 1 stage, 1 wait_for_unit (multi-user.target, 180s), 1 diag_unit (d2bd-guest.service, 120s), 1 succeed (the unit is active), 1 diag_wait (60s) over the journal line that says the Agent ComponentSession listener is bound - with the fixture's own row set (the unit's status dump) and its own journal source - and 1 fail over the bundle-validation failure line, which must not be there. The listener line alone would not prove a boot, which is why the fixture asserted the absence too, and so does the port. The fixture's node - the component-session and guest-broker modules with the enrolled key pair, the v3 bundle, the bundle-install unit and the AF_VSOCK device - moves to nix/test-support/host-integration-node.nix as d2bGuestShellServiceNode, with the two runCommand derivations it carried. Its fixture file was removed by 8ad9e01f1's broad stage; this commit adds the Rust that replaces it. Counts the port must satisfy: 1 stage, 1 wait_for_unit, 1 diag_unit, 1 diag_wait, 1 succeed, 1 fail, before and after.
The live broker privilege posture oracle moves into packages/d2b-test-vm-harness/src/checks/privilege_oracle.rs in the fixture's own order: 4 stage, 2 diag_unit (d2b-broker.socket 30s, d2bd.service 180s), nine succeed calls (the broker start, the main-pid wait, the rendered posture, the live status, cgroup and namespace reads, the kernel's cap_last_cap and the two identity reads) and the twelve assertions the fixture made: uid and gid equal the rendered User and Group, the uid is 0, CapBnd equals the rendered CapabilityBoundingSet and is not the full kernel mask, CapEff stays inside CapBnd, CapAmb equals the rendered AmbientCapabilities and is zero, NoNewPrivs equals the rendered NoNewPrivileges, seccomp is filter mode, and the cgroup path carries both the rendered Slice and d2b.slice. The fixture's two parsing helpers come with it, including their refusal of a capability this kernel does not number. The four lines the fixture printed about the posture are reported through the surface's own `announce`, so a run reads the way it read. The guest is the reusable daemon node the fixture already booted from the node module, so no guest configuration moved. Its fixture file was removed by 8ad9e01f1's broad stage; this commit adds the Rust that replaces it. Counts the port must satisfy: 4 stage, 2 diag_unit, 9 succeed, 12 assertions, before and after.
The live Wayland proxy AF_UNIX relay check moves into packages/d2b-test-vm-harness/src/checks/wayland_proxy.rs in the fixture's own order: 5 stage, 1 wait_for_unit (multi-user.target, 180s), eleven succeed calls (the test directory, the fake compositor written and started, the proxy started, its socket bound and its directory mode, the wl_display.get_registry request sent and then asserted byte-for-byte against what the compositor received, and both processes killed), and the fixture's two diagnostics steps, which this surface provides as `diag_file`: the file wait with the fixture's own rows, its own 30s bound and its own failure reporting, rather than a bare wait that would print nothing when it timed out. The fixture declared a plain NixOS node, so its user and its two packages move to nix/test-support/host-integration-node.nix as d2bWaylandProxyNode, resolving the proxy through `self.packages` exactly as the fixture did, so the guest runs the binary this build produces. Its fixture file was removed by 8ad9e01f1's broad stage; this commit adds the Rust that replaces it. Counts the port must satisfy: 5 stage, 1 wait_for_unit, 11 succeed, 2 diag_file, before and after.
The authenticated Resource operator and framework census moves into packages/d2b-test-vm-harness/src/checks/resource_operator_activation.rs in the fixture's own order: 11 stage names (two the fixture called and nine its diagnostics steps called), 2 wait_for_unit (nftables 180s, d2b-broker.socket 30s), 2 diag_unit (d2bd.service, 180s each), 2 wait_for_file (public.sock, 30s each), 6 diag_wait (the provider session line, the host row, the controller process, the controller pid, adoption across the restart, and the resync twenty seconds later), 3 diag_run (the debug surface's zone report, named row and human tree), 13 succeed, 2 fail (a row nobody declared and a user without the role) and 4 assertions: the controller pid is the same before and after the restart, the acceptance census equals the framework declaration, no declared unit is missing, and no provider-owned persistent unit exists. The fixture's two row builders (`live_rows` and `saved_rows`, with its jq projection) come with it as functions of the module. Its guest - the daemon node plus nftables, the acceptance provider artifact, the two zones, the two users and jq - is declared in nix/test-support/host-integration-node.nix as d2bResourceOperatorActivationNode. Its fixture file was removed by 8ad9e01f1's broad stage; this commit adds the Rust that replaces it. Counts the port must satisfy: 11 stage, 2 wait_for_unit, 2 diag_unit, 2 wait_for_file, 6 diag_wait, 3 diag_run, 13 succeed, 2 fail, 4 assertions, before and after.
The declared host posture contract moves into packages/d2b-test-vm-harness/src/checks/state_posture_contract.rs. This check's assertions are data-driven: the fixture read /etc/d2b/state-posture-contract.json and walked it, so the port carries the same walk and the same helper set - `substitute` for the four declaration tokens, `tree`, `level_path`, `stat_row`, `acl_entries` with the fixture's observation cache, `named_entry`, `permission_bits`, `level_exists`, `spawn_preflight_entries`, `run_as` and `probe` - and `check_level`, with the fixture's own mode policies (exact, group-traverse-minimum, preserve-existing), its declared-ACL and undeclared-named-entry checks, its mask-union check and its per-principal traverse/read/write probes, refusing in the fixture's own words. The fixed half of the script is fixed here too: 6 stage, 1 succeed for the declaration read plus the broker start and the eight principal id reads, 1 wait_for_unit (d2b-broker.socket, 30s), 2 diag_unit (d2bd.service 180s, d2b-broker.service 30s), 1 wait_for_file (public.sock, 30s), 2 diag_wait (store-view sync and vmm spawn, 300s each), 1 fail (the store-view open never failed) and the fixture's closing announce. Its guest - the writable-store shape with the two provider artifacts, the ComponentSession keys, the v3 bundle, the checked guest system, its store-view image and the installed artifacts - moves to nix/test-support/host-integration-node.nix as d2bStatePostureContractNode, with the `let` bindings that build it. Its fixture file was removed by 8ad9e01f1's broad stage; this commit adds the Rust that replaces it. Counts the port must satisfy: 6 stage, 1 wait_for_unit, 2 diag_unit, 1 wait_for_file, 2 diag_wait, 1 fail, the driver's succeed calls, and the per-level and per-principal assertions the declaration generates, before and after.
The Device-worker launch path (U17 slice 3) moves into packages/d2b-test-vm-harness/src/checks/device_worker_launch.rs in the fixture's own order: 9 stage, 1 wait_for_unit (d2b-broker.socket, 30s), 1 wait_for_file (public.sock, 30s), 1 diag_unit (d2bd.service, 180s), 6 diag_wait (rows-ingested 180, tpm-worker-ready 180, tpm-worker-process 60, tpm-sockets 60, tpm-flush-outcome 180, tpm-teardown 180), 19 succeed, 3 execute (the fixture's own `d2b` and process reads) and twenty assertion sites - the fixture's own `check(...)` calls, which run fifty-four times over the declared-row loops. No fail call exists in the fixture and none exists here, and the fixture's fourteen printed lines are reported through the surface's own `announce`, so a run reads the way it read. The Python the fixture built - the `d2b`/`list_json`/`flush_get` command builders, the `DECLARED_ROWS` and `BINDING_OWNER` tables and the row dumps its diagnostics print - are private ports in the module. The guest it boots - the daemon shape plus the swtpm and GPU device-worker artifacts, the crosvm stand-in, the Cloud Hypervisor configuration and the declared zones, Devices and provider rows - is declared in nix/test-support/host-integration-node.nix as d2bDeviceWorkerLaunchNode. This commit also repairs the lane's own check inventory: `_CHECKS` in bazel/checks/vm/BUILD.bazel carried `state-posture-contract` twice, which would have declared one `guest_image_state-posture-contract` target twice. The duplicate is dropped, so the inventory is the eleven checks again. Counts the port must satisfy: 9 stage, 1 wait_for_unit, 1 wait_for_file, 1 diag_unit, 6 diag_wait, 19 succeed, 3 execute, 20 assertion sites (54 invocations), 14 announce lines, before and after.
…fixture
The U11 midpoint Volume proof moves into
packages/d2b-test-vm-harness/src/checks/virtiofsd_volume_runtime.rs in the
fixture's own order: 2 stage ("boot" and the tag `dump_rows` stages),
2 wait_for_unit (nftables.service 180s, d2b-broker.socket 30s),
1 wait_for_file (public.sock, 30s), 1 diag_unit (d2bd.service, 180s),
7 diag_wait (volume-realized 180, binding-realized 120, worker-realized
120, endpoint-realized 120, serving-socket 60, virtiofsd-process 60,
binding-deleting 60), 6 succeed (the pre-delete list, the revision read,
the delete, the post-delete read, and the two calls inside the teardown
sampling loop) and 1 diag (the tag's row dump). No fail call exists in the
fixture and none exists here.
The two identities the fixture derived - the binding name from the
attachment tuple and the serving socket path from (zone, volume, guest) -
are its own frozen-v1 derivations and appear here as the constants they
resolved to; the row projection every wait carries, the fixture's
`chain_row_dumps` and its `dump_rows`, come with it as functions of the
module. The teardown window is sampled the way the fixture sampled it: the
three lists are read parent -> worker -> endpoint so a violation is an
invariant rather than a straddled read, and the samples are asserted over
afterwards for endpoint-first ordering and for no owned row outliving its
parent, in the fixture's own words.
Its guest - the daemon shape plus nftables, the acceptance host runtime,
the Volume acceptance artifact and its publisher key, the two zones and
their rows with the operator role binding - is declared in
nix/test-support/host-integration-node.nix as
d2bVirtiofsdVolumeRuntimeNode.
Counts the port must satisfy: 2 stage, 2 wait_for_unit, 1 wait_for_file,
1 diag_unit, 7 diag_wait, 1 diag, 6 succeed, 2 assertions plus the
non-convergence raise, before and after.
An independent op-stream verification of the seven daemon-shape ports -
the fixture's own evaluated testScript run against recording stubs,
against the ported Rust run against a recording guest-control surface,
comparing every command, bound, order, row label, explain pair and failure
message - found two defects in the ports.
guest-agent-cap-confinement: the host-namespace capability scan's shell
text was transcribed with two quote pairs the fixture did not have. The
fixture's command is `pid=${status#/proc/}; pid=${pid%/status};`, because
the `''$` in its indented string is the escape for `$` rather than a quote.
The ported text read `pid='${status#/proc/}'`, which the shell leaves as a
literal rather than expanding, so every row was skipped, both capability
tables came back empty, and the two aggregate assertions - no capability
gained by a host-namespace process, none left behind by the agent's
service - passed vacuously. The quotes are gone; the command is the
fixture's.
guest-shell-service: the listener wait's row carried the unit's name as
its label. The prelude's `unit_dumps` labels that dump `<unit> status`,
which is what the port asserts now, so a failure report names the row the
way the fixture's failure named it. Diagnostic-only, but the port's whole
point is that a failure reads the way it read.
Both checks' assertion counts are unchanged: 5 stage, 1 wait_for_unit,
1 diag_unit, 2 succeed, 12 assertions for the first, and 1 stage,
1 wait_for_unit, 1 diag_unit, 1 diag_wait, 1 succeed, 1 fail for the
second, before and after.
…t_filter Two defects in how the shipped lane is reached. `make test-host-integration` was still the nix recipe: it evaluated `getFlake(...).vmChecks` to discover check names, ran an Attic cache preflight and closure upload, staged a host-tool bundle for the flake to inject, and built each vmCheck with `nix build` under sudo - and only then built the host tools with Bazel. The recipe's own comment said it "retires with the nix lane", and it had not. It is now one `bazel test` invocation of the lane target under the committed `guest` profile, matching the shape the `perf` target already uses. The Attic preflight and closure upload go with it. The guest-image action declares its own substituters and preflights them itself, so the cache handling now lives with the build that needs it instead of in a second place that can drift out of step with it. `D2B_HOST_VM_CHECK` and `D2B_HOST_VM_JOBS` existed only inside the retired recipe and are gone; `D2B_VM_CHECK` still selects one named check. The x86_64-linux skip stays, reworded now that there are no vmChecks to skip, and a missing `/dev/kvm` now stops with a message instead of a note about a TCG fallback the lane has never had - R7 makes virtualization a precondition, and the recipe should say so rather than imply a slow path exists. The lane ignored Bazel's own `--test_filter`. `selection` read the target's `--check` argument and `D2B_VM_CHECK` and nothing else, so `bazel test --test_filter=<check>` looked like it selected one check and quietly ran all eleven - a pool of guests, and a verdict for a check nobody asked about. It now reads `TESTBRIDGE_TEST_ONLY` as a third source, after the two existing ones, and the first that names anything wins so an explicit request is never widened by a default that happens to be set. The two contributor documents that described the handoff are rewritten to describe the shipped lane: the declared-input guest build, the restored pool, the virtualization precondition, and the loss of the emulation fallback.
… and retire its fixture
The last of the eleven, and the one whose fixture was the largest: its
assertions move into
packages/d2b-test-vm-harness/src/checks/runtime_cloud_hypervisor_guest_preflight.rs
in the fixture's own order, with the fixture's own command text and
bounds. Counts, fixture and port alike: 32 succeed, 1 fail, 2
wait_for_file, 1 wait_for_unit, 3 diag_unit, 15 diag_wait, 1 diag, 9
explicit stage (plus the 18 stage names the diag_unit and diag_wait calls
carry), 1 guest sleep, no bare assertion, and no start_all (the lane boots
the guest). The fixture's diag_projection and its three row builders
(live_rows, saved_rows, summary_rows) come with it as functions of the
module.
Two tokens could not stay byte-identical, and they are the only two: the
fixture interpolated two nix store paths into one enrollment command
(${fixtureKeys}/host.key and ${fixtureKeys}/guest.pub), and a store path
is not addressable from the lane's Rust. The node installs the same two
files of the same fixtureKeys derivation at /etc/d2b/fixture-keys/host.key
and /etc/d2b/fixture-keys/guest.pub, and the command's two source paths
are those; every other byte of every command is the fixture's.
The guest is the writable-store shape plus the fixture's own let bindings
(the two provider artifacts, the ComponentSession key pair, the v3 guest
bundle, the Cloud Hypervisor configuration, the nested guest system whose
boot the check preflights, its store-view image and the artifacts), all
declared in nix/test-support/host-integration-node.nix as
d2bRuntimeCloudHypervisorPreflightNode.
Counts the port must satisfy: 32 succeed, 1 fail, 2 wait_for_file,
1 wait_for_unit, 3 diag_unit, 15 diag_wait, 1 diag, 9 stage plus the 18
the diagnostics carry, 1 sleep, before and after.
All eleven checks assert in Rust, so the nix VM orchestration has nothing left to run. The `vmChecks` output is removed, and with it the `D2B_HOST_TOOL_BUNDLE` / `D2B_CH_CONTROLLER_BUNDLE` environment handoff that only that output read: the guest-image action declares the same bundles as Bazel label inputs, which is what R2 and R3 asked for. The comment that stood over the output now says where the lane lives. The one file left under tests/host-integration/ is lib.nix, and it stays on purpose: the ported checks' guest declarations in nix/test-support/host-integration-node.nix still read it for their provider-artifact builders (mkAcceptanceProviderArtifact, mkRuntimeCloudHypervisorArtifact, mkVolumeProviderArtifact, mkDeviceWorkerProviderArtifact), so the directory survives for its builders rather than for fixtures. Also removed here: the deferred Gateway-isolation fixture, tests/host-integration/deferred/host-zone-gateway-isolation.nix. That check asserted that a Gateway relay credential never materializes on the host - not in /etc/d2b, /var/lib/d2b, /run/d2b, /var/log, the audit tree, the journal, coredumps, nor in any d2bd or broker process's environ, cmdline or fd table, with no relay socket established. **No retained check carries that assertion**, and the eleven checks were reviewed for it, so this is a removal rather than a consolidation: the tokens gateway, relay, canary and SharedAccessKey appear in none of the retained fixtures and no ported module sweeps host paths or process tables for a guest-held secret. The changelog fragment records the loss in those terms. Checked before removing the handoff: the two variables appeared only in the retired output's own evaluation, in the Makefile recipe that is being retired with it, and as a fallback read in tests/host-integration/lib.nix (read it back with getEnv: when the variable is absent, the builder resolves the same package through self.packages, which is the path every image build already takes). Nothing else in the tree reads them.
…, and the console-read fix
The host lane is one Bazel test target and the heavy-gate semaphore is gone, but the instruction, contributor, and reference documents still described the removed `D2B_HOST_TOOL_BUNDLE` / `D2B_CH_CONTROLLER_BUNDLE` handoff, the `vmChecks` output, the `runNixOSTest` fixtures, and the deleted semaphore as current. Update those sites to the shipped lane: the Bazel-built host binaries arrive at the guest-image action as declared Bazel label inputs (Nix still must not rebuild them), the lane is `//bazel/checks/vm:host_integration_lane_run`, every assertion is Rust, `D2B_VM_CHECK` / `--test_filter` selects one check, and `/dev/kvm` is a declared precondition with no emulation fallback. Record the semaphore's deletion where a document still pointed at it, and say that nothing replaces it.
…ice node The ported guest-shell-service node was written as a function that takes its module arguments, binds two fixture artifacts in a `let`, and then returns *another* function of the same arguments - so calling the node once yielded a function rather than a module, and the guest-image evaluation refused it: "module ... does not look like a module". The other eleven nodes are single-layer and unaffected, which is why only this check's image failed to build. The outer function already destructures `lib` and `pkgs`, so the inner arrow was redundant: dropping it makes the `let` bindings visible to the module body directly, and the node now evaluates to an attrset with its `imports`, which is what the module system wants. Found by bisecting the module list: the node alone reproduces the failure under `nix eval-config`, and forcing it with module arguments returns a lambda rather than a set.
`admit` reported a group the host could not hold alongside the others as "will run after a member finishes" and then `continue`d, dropping it from the list the runner spawns. The group never ran. The lane still reported its results and still exited green, so a host whose declared budget admitted fewer groups than the lane has invocations silently checked a subset of its own checks while the log named every one of them as part of the pool. Admission now returns the groups that fit and the groups that do not, and the runner runs the second list as a second wave once the first has finished - which is what the log already promised. The concurrency is unchanged: a wave still runs its groups together, and the second wave starts only when the first is done, so the host is never oversubscribed relative to the budget it declared. On a host where every group fits, the deferred list is empty and the run is identical to before.
…opying Every direct-boot check's guest carries the same kernel and the same initrd - the system closure rides in the initrd - and the image action copied both into each of the eleven images. Measured: all eleven kernels hash to 6c3ebc5d64ca and all eleven initrds to 7096b56ae40f, so the store was holding 358 MiB of byte-identical content and every image build was copying it again. The manifest already resolved an absolute path as itself, so the store path needed no new mechanism - only the manifest had to stop asking for a relative name. An image now carries its disk and its manifest and nothing else. The per-image disk is not part of this: the resulting qcow2 files are 1.2-6.8 MiB, so there was no large duplicated image to share. The closure itself was already shared - one change to a shared input costs one closure rebuild, not eleven - which is why a base-image overlay would have bought nothing here.
…ion bound A full lane run took 32.7 minutes and reported nine of eleven checks passing. The other half-hour was one check: state-posture-contract's guest, which had booted, reached systemd's local-filesystems target at 40 seconds, started udev's device-event manager, and then produced nothing further on its console for the remaining 1760 seconds of a 1800-second activation bound. The console tail said exactly where it stopped; the lane simply refused to act on it. A guest that is slow is still writing to its console. A guest that has stopped writing is stuck, and the rest of the activation bound will not change that. Both activation waits now watch the console's own length and fail with a distinct `ConsoleStalled` error - naming how long the console was quiet, and carrying the same tail a bounded failure carries - once it has produced nothing for a stall bound. The stall bound is separate from the activation bound on purpose, and defaults to 180s against the observed 17-20s activations. A loaded host genuinely takes longer to boot a guest, so the generous bound stays for a guest that is talking; a guest that went quiet at 40s is now described in about three minutes instead of half an hour. Both waits read it, so it is overridable per run: D2B_TEST_VM_HARNESS_CONSOLE_STALL_SECS. A test pins the distinction, including that output resuming clears the stall rather than latching.
The activation bound was 1800s against a measured healthy activation of 17-20s. That is ninety times the time a working guest takes, so it was not a bound so much as an abandonment, and it dominated the lane: a full run took 32.7 minutes and spent half of it inside this single wait, on a guest that reached systemd's local-filesystems target at 40 seconds, started udev's device-event manager, and never advanced again. 600s is thirty times the observed maximum, which still leaves a heavily loaded host room, and the number stays overridable per run through D2B_TEST_VM_HARNESS_ACTIVATION_TIMEOUT_SECS for a host that needs more. This is separate from the console-stall guard in the previous commit, and the two cover different failures. That guard handles a console that has gone silent; this guest's console kept writing - a store-sync heartbeat every thirty seconds - while never reporting activation, so the guard correctly declined to fire and the bound was what the run waited on. A guest that is alive and never arrives is not the same defect as a guest that stopped talking, and only the bound distinguishes them.
`lane_test`'s contract says "the target's result is never cacheable. A guest's verdict depends on what the host did while it ran, so a second invocation re-runs every selected check rather than replaying what the first one concluded, and `no-cache` is what says that to the Bazel graph." The tag list did not contain `no-cache`. It was `exclusive`, `local`, `no-remote-cache`, `no-remote-exec` and `no-sandbox` - which keep the lane off the remote cache and the remote executors, and say nothing about the local one. So the lane's verdicts were cached locally like any other test's. Found by trying to repeat a run: three consecutive invocations of `--test_filter=state-posture-contract` all reported "PASSED in 92.9s" without executing, the second and third served from the first. A green from an earlier run was being reported as a green from this one, on a test whose whole value is that it boots real guests and reports what happened to them.
… socket under
The `path:vm-run:<guest>` storage row declares a creator; nothing created
it. `/run/d2b/vms` did not exist on the host, and the broker's socket grant
refuses every worker whose runtime socket path has an absent ancestor -
which is every worker except the one-shot flush, because the flush binds no
runtime socket. That asymmetry is why this presented as a TPM fault.
Verified on the guest: `find /run/d2b/vms` now reports
`drwxrwx--T d2bd:d2b /run/d2b/vms`, mode 1770 and owner d2bd:d2b, matching
the storage row exactly. It belongs in host-daemon.nix rather than
host-broker.nix because `/run/d2b` and its ACLs are canonical there and
host-broker deliberately does not touch them.
With the directory in place the worker gets past the grant, and the refusal
reason the check could never see is now in the log - the detail is the
`forward_rendezvous` logging added earlier, which had never fired before
because the grant refused before it:
forwarded invocation refused with a reason code="handler-refused"
detail="spawn-process: handler-errored (spawn-process: swtpm-dir
hardening failed: swtpm-dir-state-not-provisioned)"
That is the next defect, and it is not this one. `device-worker-launch` still
fails, now on a different and much more specific ground.
The check gains a row dump for the two accounts the journal and the existing
row dumps could not give: the `vm-run` storage row, which the swtpm storage
rows filter (`test("swtpm")`) could not show by construction, and the
broker's audit log, which records the spawn decisions the journal only
summarises. Without them a guest with that row and a guest without it read
identically.
…it-corpus disposal The inputs are generated, so the disposal of the whole-tree Rust audit corpus changes their digests; they are regenerated rather than resolved by hand.
The case asserted the D2B_HOST_TOOL_BUNDLE staging the retired nix recipe did in the Makefile. Its intent - Nix realizes the guest closure around binaries Bazel built and does not rebuild them - is unchanged, and the declaration that now carries it is the guest-image action's host_tools list, so the case asserts that instead of a recipe that no longer runs.
The case asserted D2B_HOST_TOOL_BUNDLE staging in the Makefile recipe the Bazel lane retired. The intent - Nix realizes the guest closure around binaries Bazel built and does not rebuild them - is preserved by asserting the recipe no longer carries a handoff, and the shared host-tool builder still supplies the acceptance controller.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Makes the host-integration lane Bazel-owned end to end.
All eleven checks now assert in Rust. Every
runNixOSTestfixture and itstestScriptare gone, thevmChecksflake output is gone, andmake test-host-integrationis a singlebazel testinvocation of//bazel/checks/vm:host_integration_lane_run. Each guest image is a graphoutput keyed on declared inputs - the flake and its lock, the guest module
sources and the d2b host binaries all arrive as label inputs - and the lane
restores a pooled guest per check rather than booting one guest per check.
Lane behaviour changes worth knowing
--test_filteris honoured. The lane read only its own--checkargumentand
D2B_VM_CHECK, sobazel test --test_filter=<name>looked like itselected one check and quietly ran all eleven. It reads
TESTBRIDGE_TEST_ONLYnow, after the two existing sources, and the firstthat names anything wins.
_LANE_TAGSnever carried the
no-cachetag its own contract promised, so a secondinvocation of an unchanged graph returned the first one's pass.
17-20s, where it was 1800s. A full run was spending half of 32.7 minutes
inside that one wait. A console that goes silent for 180s now fails with a
distinct
ConsoleStalledrather than waiting the bound out.dropped. It was logged as "will run after a member finishes" and then
dropped, so a constrained host reported green over a subset.
each image: all eleven were byte-identical, so the store held 358 MiB of
duplicate content.
Full lane: 8.2 minutes, down from 32.7.
Product defects this surfaced
/run/d2b/vmsruntime dir, so thebroker's socket grant refused every worker binding a runtime socket. The
one-shot flush binds none, which is why it presented as a TPM fault.
its reason discarded at the point of refusal.
against an unbounded socket.
Known red, and why
device-worker-launchfails. The broker refuses the long-lived swtpmworker retryably forever because its per-Device state subdirectory is never
created: the resource-backed branch of
live_handlers.rs:3118derivespaths, refuses on absence, then returns
Ok(None)- skipping the createpath - while the branch below it does exactly that create. The fix is a
product decision in the broker and is deliberately not in this PR.
packages/xtask:buildbuddy_configandbazel/checks/nix:nix-unit-broker-profilesfail on this branch and pass onorigin/v3. The nix one asserts the retiredD2B_HOST_TOOL_BUNDLEMakerecipe and needs updating to the declared-input handoff. The buildbuddy one
is under investigation: bisect points at a lane-only commit that cannot
plausibly cause it.
Retired coverage
The deferred Gateway-isolation check is deleted and no retained check
carries its assertions - a Gateway relay credential's canary never
materializing on the host. This is a removal, not a consolidation; there is
no successor. Recorded in
changelog.d/v3.md.