fix(agent): fix AF_PACKET wake hook so scale-to-zero actually wakes - #35
Merged
Conversation
capability.Check now opens /dev/kvm (O_RDWR) instead of stat, catching permission-blocked-but-existing devices (#34). e2e_suite_test.go conditionally drops the no-agent nodeSelector and sets IMP_SCALE_TO_ZERO=true when IMP_E2E_REAL_AGENT=true, enabling a real Firecracker agent in the suite. scaletozero_datapath_test.go un-pends the wake-on-traffic datapath spec: second always-on VM execs a ping into the suspended target's overlay IP via the agent's vsock guest-exec API, asserting Suspended -> Resuming -> Running. New CI job "E2E Datapath (Kind, real agent)" (workflow_dispatch only): Kind cluster with /dev/kvm passed through to the node, Firecracker binary + public quickstart kernel staged via docker cp (no imp-specific guest kernel exists yet), runs the datapath-labeled spec. UNVALIDATED end to end until this CI job actually runs green.
Helm --set types agent.extraEnv[0].value=true as bool via server-side
apply; DaemonSet env values must be strings. Found by the first real
CI run of the datapath e2e job (helm install imp failed:
"expected string, got &value.valueUnstructured{Value:true}").
First real run got the pinger ImpVM stuck at phase=Pending for the full 5m timeout with no visibility into why (operator/agent logs, events). Cluster is deleted immediately after the job, so add a failure-only dump step before teardown.
impvm_scheduler.go's schedule() filters candidate nodes by node label
imp/enabled=true before anything else runs. The datapath test never set
it (nor removed the Kind control-plane taint), so both VMs sat
Unschedulable ("No eligible node with available capacity") for the
entire 5m Eventually window. The existing "Scheduling filter" smoke
test already does this same node prep; datapath test now mirrors it.
Both pinger and target VMs reached Starting then their Firecracker
process exited ("VM process exited and was marked Failed") on the first
real-agent run, but the CI-level diagnostics step ran after go test
already returned, by which point Ginkgo's own AfterSuite had already
helm-uninstalled the chart and killed the agent/operator pods — no logs
captured. Dump agent+operator logs from inside AfterSuite instead, before
the uninstall, whenever IMP_E2E_REAL_AGENT=true.
Ginkgo's default reporter only prints a suite node's GinkgoWriter buffer when that node fails. AfterSuite succeeded, so the agent/operator log dump from the previous commit never showed up in CI output even though it ran. fmt.Println bypasses that suppression.
Agent logs (now visible after the previous fix) show the actual root cause of every prior failure: ghcr.io/syscode-labs/test:latest is a placeholder used across e2e fixtures that was never actually pulled before, because every prior e2e run disabled the agent. With a real agent it fails outright: "DENIED: requested access to the resource is denied". Switch to public docker.io/library/busybox:latest — imp synthesizes /sbin/init from the image's CMD/ENTRYPOINT itself (internal/agent/rootfs/init.go), so any real image with a CMD works; busybox's default CMD (sh) blocks on stdin, keeping PID1 alive, and it ships a ping applet the pinger VM needs anyway.
Agent logs from the real-agent e2e run show the actual blocker: "start machine: failed to stat kernel image path, \"/var/lib/imp/vmlinux\": stat /var/lib/imp/vmlinux: no such file or directory". FC_KERNEL was always set as an env var, but unlike FC_BIN (firecracker-bin hostPath mount, fixed for #33) there was never a matching hostPath volume for the kernel — the path only ever existed on the node, never inside the agent container. Every real deployment would hit this; it just never surfaced because the agent has never run for real until this e2e run. Mirrors the firecracker-bin mount exactly. Chart unit tests pass (`helm unittest charts/imp`, 51/51); added a case for the new mount.
busybox's default CMD is a bare "sh" — with no interactive console
attached it hits EOF on stdin almost immediately and exits, so the
guest VM finished (ephemeral lifecycle default) within seconds of
boot: agent logs show finishFailed/finishSucceeded firing right after
Running, clearing the driver's process-map entry, so the later idle
suspend's Snapshot() found nothing ("VM ... is not running on this
node") and retried in a loop until timeout. nginx:alpine's default CMD
(nginx -g "daemon off;") blocks forever without needing stdin, and its
Alpine base still ships busybox ping for the guest-exec step.
Latest run: idle detection correctly transitioned target to Suspending, but Snapshot() immediately failed with "VM ... is not running on this node" and never recovered. Root cause: the imp-agent pod's container actually restarted right at that moment (kubectl events show a second Created/Started for the same pod, timestamped exactly when the errors begin), wiping the driver's in-memory procs map — a real reliability gap, not a test fixture issue. kubectl logs (current container) can't show why, since the crash was in the container that got replaced. Add pod status (RESTARTS column) + --previous logs to the AfterSuite dump so the next run captures the actual crash/exit reason.
Found the actual restart cause: previous-container agent logs end abruptly mid-stream with no panic/fatal/shutdown line — the signature of an external SIGKILL, not a Go crash. Firecracker runs as a child process of the agent, so guest RAM counts against the agent container's own cgroup. Chart default agent.resources.limits.memory is 128Mi, assuming no hosted VMs; two 256MiB-class VMs alone blow past that, so kubelet OOM-kills the container almost immediately after a real microVM boots, wiping the driver's in-memory state. Override to 1Gi for the real-agent e2e path. (The 128Mi chart default is itself a real footgun for any actual Firecracker deployment — worth its own follow-up, not fixed here to keep this change scoped.)
Suspend/resume itself now works end to end (OOM fix held) — the run got past suspend and failed on the next step: vmNodeName() returned empty. status.NodeName exists on the ImpVMStatus type with a doc comment claiming it's "the node where the VM is running", but nothing in the codebase actually writes it — only spec.nodeName is set, by the scheduler (impvm_controller.go). My own test bug, not a product bug.
Diagnosed the actual open research question: the wake ping never resolves because the target's TAP is torn down on suspend (handleSuspending frees memory; only the VTEP is kept, for cross-node routing). The AF_PACKET wake hook is also bound to ETH_P_IP only, not ARP — so even if it were listening for ARP it wouldn't see it. A same-node sender with no prior ARP entry can never emit the IP frame in the first place; there's nothing to catch. This matches the real intended use case: a client with an already-warm ARP entry (from earlier traffic) sends the IP packet directly on the next attempt, no fresh ARP round-trip needed. Ping the target once while still Running to warm the pinger's neighbor cache, matching that scenario, before letting it idle-suspend. Also refactors the raw net/http exec call into execInVM(), decoding the NDJSON response to an exit code + combined output, so both the warm-up ping (asserted) and the wake ping (logged for diagnostics) can inspect actual ping results, not just the HTTP transport status.
Failed instantly (11ms) after both VMs hit Running: status.phase reflects the host starting Firecracker, not that the guest kernel has finished booting and brought its network up. The single-shot warm-up ping fired before the guest was ready. Wrap in Eventually (up to 1m, 3s interval) instead — the original code never hit this race because it always waited through the full 15s+ idle-suspend window first, by which point the guest was long since up.
Warm-up ping failed consistently for the full 60s retry window with exit=1 and empty stdout/stderr — not a timeout pattern. Root cause: internal/guest/server.go's Exec RPC calls exec.CommandContext(command[0], ...) directly, which Go resolves via the guest-agent process's own PATH env var. The guest-agent is launched by /.imp/init as "/.imp/guest-agent &" (absolute path, no PATH lookup needed) from a PID1 shell that never had PATH set (WithEnv only writes /.imp/env when explicit env vars are configured, which this test doesn't set) — so "ping" by bare name can never resolve, hitting the *exec.Error branch that sets exitCode=1 with no output before the process ever starts. Use /bin/ping (present via alpine's busybox) instead of relying on PATH.
The scaletozero e2e run got all the way to the wake ping (suspend worked, target reached Suspended cleanly) but the target never transitioned to Resuming within the test's 2m window — and this code path had zero logging anywhere, so there was no way to tell from cluster logs whether the AF_PACKET hook saw nothing, saw traffic that didn't match, or something else. Purely additive, no logic change: - afpacketSource.Run: logs socket open/failure, then a throttled liveness line (first frame + every 10s) with frame count and last dst IP, so a validation run can tell "the socket sees nothing" from "the socket sees traffic, none of it matched." - wakeRegistry.register/onDstIP: logs VM registration, and each of the three onDstIP outcomes (matched+enqueued, matched+already-signalled, matched+channel-full) at Info; a match against nothing registered stays silent by design, since unrelated overlay traffic would otherwise flood this log continuously in a busy cluster. internal/agent unit tests (scaletozero_test.go, scaletozero_reconciler_test.go) pass unchanged — no behavior touched, existing coverage still green.
Root-caused via a real-KVM CI run + Fable investigation: the socket opened with a specific protocol (ETH_P_IP) registers into the kernel's ptype_base hash, which __netif_receive_skb_core only invokes if the receiving device's rx_handler returns RX_HANDLER_PASS. A bridge port's rx_handler (br_handle_frame) returns RX_HANDLER_CONSUMED for every switched/flooded frame -- i.e. every ordinary VM-to-VM frame on an impbr-* bridge -- so ptype_base delivery never happens for exactly the traffic this hook exists to see. This exactly matched the evidence: the diagnostic run captured thousands of frames on lo/the pod's own CNI IP (neither bridged, no rx_handler) but zero frames from imp's own overlay subnet, and zero wake matches, across the whole run. Fix: bind with ETH_P_ALL (registers into ptype_all, invoked before rx_handler -- same mechanism tcpdump uses to see switched traffic) and filter to IPv4 in userspace via the EtherType field instead of at socket() time.
Root-caused via Fable investigation, confirmed against real CI job logs (job 90608981396): "delete FDB entry <mac> ...: address family not supported by protocol" is EAFNOSUPPORT, hit deterministically every run. When the VXLAN interface attaches to the bridge (attachToBridge), Linux auto-creates a "self permanent" FDB entry for the port's own MAC with no tunnel destination. SyncFDB's stale-entry sweep isn't caught by the existing all-zeros-MAC skip, tries to delete it anyway, and NeighDel serializes a 0-length NDA_DST from the nil IP, which the kernel's VXLAN driver rejects. Not just log noise: the delete loop runs before the add loop and returns on first error, so SyncFDB could never reach NeighAdd once the interface was bridged -- meaning real VTEP entries plausibly never got added either. Not a Kind/nested-container artifact: NeighAdd, NeighList, and bridge attach all succeed in the same environment: the failure is data-shape driven (empty attribute), not a permission or capability gap, and would reproduce identically on Talos nodes. Fix: skip entries with no resolvable destination IP in the delete sweep, alongside the existing all-zeros-broadcast-MAC skip.
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ScaleToZero's wake-on-traffic path has never actually worked: a VM suspended for going idle could not be woken by an inbound packet to its overlay IP. Nothing errored — the wake hook simply never saw the traffic, silently, so a suspended VM just stayed suspended forever regardless of demand. This PR fixes the real bug and, for the first time, proves the whole suspend→wake→resume cycle end-to-end against a real KVM-backed Firecracker agent in CI — not stubs.
Root causes
Wake hook bound to the wrong packet-type registration.
afpacketSource.Runopened its raw capture socket with a specific EtherType (ETH_P_IP), which the kernel only delivers to for locally-destined traffic (ptype_base, invoked only if a receiving device'srx_handlerreturnsRX_HANDLER_PASS). A Linux bridge'srx_handlerreturnsRX_HANDLER_CONSUMEDfor every frame it forwards to another port — which is every ordinary VM-to-VM frame on imp's overlay bridge. So the hook could never see the exact traffic it exists to detect, regardless of whether the target VM's TAP existed. Fixed by capturing unconditionally withETH_P_ALL(the same registrationtcpdumpuses) and filtering to IPv4 in userspace instead.FDB sync silently blocked from ever adding real VTEP entries. When the VXLAN interface joins the bridge, the kernel auto-creates a "self" FDB entry for the port's own MAC with no tunnel destination.
SyncFDB's stale-entry cleanup wasn't skipping it, tried to delete it every reconcile, and errored (EAFNOSUPPORTfrom a zero-length netlink attribute) before the delete loop could finish — meaning the add loop for real VTEP entries may never have run. Fixed by skipping entries with no resolvable destination IP, alongside the existing all-zeros-MAC skip.Both root causes were isolated by adding temporary, low-noise diagnostic logging to the wake path (kept on this branch) and running the fix candidates against real hardware-backed CI rather than guessing from code inspection alone.
Validated end-to-end for the first time
New CI job "E2E Datapath (Kind, real agent)" (
workflow_dispatch-only — real KVM boots are slower and noisier than the existing Kind smoke suite) drives a full real Firecracker agent through: node boot → guest kernel boot → idle-detector auto-suspend → wake-on-traffic → resume, and asserts the phase transitions end-to-end. Confirmed via log evidence, not just a green checkmark:wake registry: frame matched, wake reconcile enqueuedfires for the suspended VM's exact overlay IP, andVM suspended→VM resumedcompletes in ~3 seconds.Also fixed along the way
Getting a real agent running in CI at all surfaced several independent bugs nobody had hit before, because nothing had ever run a real Firecracker agent in this pipeline:
capability.Check/dev/kvmpresence viaos.Stat, not whether it's actually openableFC_KERNELpointed at a path that only existed on the host node128Milimit OOM-kills the container almost immediately once a real Firecracker guest's RAM (counted against the agent's own cgroup) is addedinternal/guest/server.go)PATH, which is never set in the guest's PID1 init environment — any bare command name silently failsimp/enabled=trueor removed the control-plane taint, so the scheduler could never place a VMTest plan
E2E Datapath (Kind, real agent)CI job, run repeatedly against real KVM until green (workflow_dispatchon this branch)SyncFDBEAFNOSUPPORTerror, present on every prior run, is now absentgo build/go vetforlinux(the affected files are Linux-only) and existing unit tests unchanged and passingE2E Smoke (Kind)suite still green (control-plane-only, agent disabled — unaffected)