Forensics recorder: a dying boot names its own last instant (#1278) - #1281
Merged
Merged
Conversation
A boot can end in VZVirtualMachine.State.error with no [EXC] block, no tombstone, and an ordinary last serial line. Guest RAM is unreadable post-mortem, so a record only survives if it reaches serial BEFORE the death -- hence a lock-free per-core ring plus a drain that rides the console line that was going to be written anyway, off by default so default boots stay byte-identical. The host half was the cheap half nobody had: the runner set no VZVirtualMachineDelegate, so VZ's error object was discarded and every gate could only ever report the bare state rawValue. The recording rules now live in a pure-Swift VMPostmortem module (the VFWire/VSSH precedent) so `swift test` pins them without booting a VM, and the runner reports a stop verdict on every run, not only the failing ones. Two defect fixes the new host tests found, both in the recorder itself: a record whose line did not fit the output buffer was marked drained and lost (now left pending and counted, since the newest record is the most valuable one in a dying boot), and a never-written slot was inferred from all-zero fields, which silently swallowed a legitimate event at seq=0 with arg=0 -- the head of a trace. A `live` flag now marks written slots. Closes #1278 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
Owner
Author
|
CI failure root cause (one line): the new type echo stdin (the pipe source) to stdout — the right half of `a | type`
+ forensics last-words recorder: on|off|dump|reset (off by default)
dmesg system log viewer: last bytes of serial output (D12)Fix: insert that line in |
The forensics command's help entry grows the shell `help` listing by one line, so the M1.5 transcript gate (verify-transcript.sh, byte-exact diff against tests/transcript-console.txt) failed on CI while every local run I had made passed -- test-console is not part of `zig build test`, only of `just verify-portable`. The one added line is the intended diff; the fixture is regenerated from the captured mock transcript. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1278. Unblocks #1261.
What this is
Two recorders, because the problem had two halves and only one of them
had ever been attempted.
Guest side —
kernel/src/forensics.zig. #1261 spent a wholeinvestigation unable to say anything about a boot that died: guest RAM is
not readable post-mortem (VZ exposes no dump, and
serial_ring.zigisRAM-only), and no
[EXC]block appeared, so no handler was ever entered.The design follows from that single constraint — a record only survives
if it reaches serial before the death:
note(site, arg)is a per-core counter and one fixed BSS slot: noallocation, no lock, no console, no comparator write, so it is callable
from IRQ/SVC/lock-held contexts. Race-freedom by construction: each core
writes only its own slots, so there is nothing to synchronise.
notenever prints. Emission is deferred to the console write path, sothe next ordinary line the guest prints carries the pending records
with it. The kernel's "no console in IRQ context" rule is preserved
exactly: printing still only happens where it already happened.
forensics on, so default boots arebyte-identical (the gate asserts
enabled=0on a boot that armednothing).
(
arg= INTID), the core-0 rotation (arg= the task switched awayfrom — the half a post-hoc "who is current" dump can never recover),
the wake funnel (
arg= the task that became runnable), and adumpsubcommand to force the tail out.
Host side —
VMPostmortem+ the runner. The runner set noVZVirtualMachineDelegate, so VZ's error object was discarded and everygate could only report the bare
staterawValue. That is precisely whatmade
state=3unfalsifiable. The delegate is now installed, and therecording rules live in a pure-Swift module with zero Virtualization
imports (the
VFWire/VSSHprecedent), soswift testpins them on amachine that never boots a VM. The runner reports a stop verdict on
every run, not only the failing one.
What it found on the tree it was built for
Run against the nudge tree (PR #1255's commit) with the recorder armed,
the boot dies at
dui tile 2and now says why:That is the datum #1261 could not obtain.
code=1inVZErrorDomainisthe internal error — VZ aborted the VM itself, which is why there was no
guest exception, no tombstone, and an ordinary last serial line. The full
trace is written up on #1261.
Two defects the new host tests found (in the recorder itself)
drainedbefore formatting, so it was silently lost — the one way this
recorder could drop exactly the newest record a dying boot is read for.
Now left pending and counted.
swallows a legitimate event at
seq=0witharg=0— the head of atrace. A
liveflag now marks written slots.Verification
live-forensics(class B, live VZ)irq238 /rotate121 /shot1, dense per-core sequences,enabled=0on an unarmed boot, host verdict line presentswift test --package-path host/vm-runnerStopReasonTestszig build test --summary allzig fmt --check/inventory-gates --check/verify-issue-coordinationGaps I am not papering over
asserted by a gate. What is asserted live is that the verdict is
reported on every boot (
vm-stop: state=… reason=<none reported by VZ>),which fails if the delegate stops being installed; the formatting and
ordering rules are unit-tested; and the callback names are witnessed by
#selectorresolution, since every protocol member is@optionaland awrong spelling would compile silently and never be called.
it emits on the next ordinary print. On the diagnostic run that cost
nothing (the trace's last record is 47,057 lines before the end, and the
drain was demonstrably live during the storm), but a death immediately
after a drain would carry one line less.