Skip to content

#1261 probes: exception entry, spurious ack, timer re-arm, console-free sample - #1284

Merged
drawmeanelephant merged 1 commit into
mainfrom
agent/buffy/1261-probes
Sep 15, 2026
Merged

drawmeanelephant merged 1 commit into
mainfrom
agent/buffy/1261-probes

Conversation

@drawmeanelephant

@drawmeanelephant drawmeanelephant commented Sep 15, 2026

Copy link
Copy Markdown
Owner

What this is

The probe set #1261 needed to make a dying boot name its own last instant, plus the one defect the extended gate caught in the sampler before merge. Every probe is inert unless forensics on; the extended live-forensics spec pins every probe call site so deleting one fails the gate instead of silently zeroing the reading.

Closes #1261 — the card's deliverable was a settled delivered-vs-recorded verdict with the mechanism bounded in writing, which this supplies (below). The follow-on work (which guest action makes VZ abort, and can the abort be made observable) is filed as #1287.

The probes

probe site arg what it separates
entry exc_dispatch entry, before any GIC state is consumed kind cadence vs post-ack irq: "not delivered" vs "delivered but acked spurious"
spur spurious-ack early return in irq_dispatch intid a spurious storm was previously indistinguishable from silence
rearm timer.arm, before the comparator write programmed delta steady rearm + no entry ⇒ comparator fires but exception never taken; rearm gap ⇒ the source stopped
shot shell idle loop, self-draining wall-clock second the guest is executing with no console traffic to ride on

The verdict (measured, twice, on the nudge dying-boot scratch tree)

The PPIs are delivered AND recorded to the last second. The death is not guest-side.

  • entry cadence matches the 1 Hz timer to the end; samples on a 4.0 s cadence through the final second; spurious acks zero for the whole run; rearm args all exactly 24000000 (a 1 s comparator at 24 MHz).
  • Host side names the death: VZErrorDomain code=1 — Internal Virtualization error. The virtual machine stopped unexpectedly. VZ aborts the VM rather than the guest faulting — which is why no [EXC] block and no tombstone ever appeared.
  • The console-free shot sample is what makes the "still alive" claim sound: 4.000 s exact spacing across the whole run needs neither a delivered interrupt nor a console line to be printed.
  • New collateral finding: the counter itself is not monotonic for the first ~0.9 s under VZ on this host (a ~0.1 s backward step at t≈0.89 s, observed twice). Any trace sorting t= across the first second is misordered; the sampler saturates elapsed time instead of underflowing, and the comment at the guard records this.

The gate-caught defect (why this PR exists beyond doc comments)

The first sampler clock was timer.irq_ticks. The extended gate's census came back shot: 6624 on a ~30 s boot — the exact flood signature the wrong-clock bug produced in the scratch tree. Reading the run with VIRELAI_KEEP_RUN=1: irq_ticks is a shared counter (both cores' 1 Hz PPIs) and is reset by re-init paths (it went 1 → 0 during early boot), and the == 0 bypass fired on every sample until the first timer IRQ. The shipped clock is timer.cntpct() / timer.freq — monotonic by contract, exact seconds, and deterministic in host tests because the frequency is a parameter. Post-fix census on a 120 s boot: entry 245 / irq 238 / rearm 238 / rotate 121 / shot 32, shot args 4 → 8 → … → 124 exact, dense per-core sequences intact.

Verification (all observed this session)

gate result
live-forensics (class B, extended spec) PASS 1/1 — census above, host verdict line present
just verify-portable (incl. transcript gate, fmt, unit tests) PASS — all 495 tests
zig build (guest image) rc=0
coordination gate ok — 2 open claims, no overlaps

Not re-run here: the other class-B VZ shards (need VZ; CI runs them).

…ee sample

All inert unless `forensics on`; each is pinned by the extended
live-forensics gate so deleting a call site fails the gate instead of
silently zeroing the reading a dying boot is read for.

- `entry` at exc_dispatch entry, BEFORE any GIC state is consumed —
  cadence vs the post-ack `irq` probe separates "not delivered" from
  "delivered but acked spurious".
- `spur` on the spurious-ack early return in irq_dispatch — a spurious
  storm was previously indistinguishable from silence. Reported in the
  gate census, not required (a healthy boot acks zero).
- `rearm` in timer.arm, arg = the programmed comparator delta — a steady
  rearm cadence with no `entry` says the comparator fires but the
  exception is never taken; a rearm gap says the source stopped.
- `shot` from the shell idle loop, self-draining, one per 4 s of
  wall-clock time: proves the guest is still executing with no console
  traffic to ride on.

Clock choice is measured, not assumed. Two candidates were tried and
rejected with live evidence: `scheduler.tick_count` (per-quantum, not
per-second: 6,212 samples in one gate boot) and `timer.irq_ticks`
(shared by both cores' PPIs and reset by re-init paths: went 1 -> 0
during early boot and advanced ~2/s, flooding at ~47 samples/s — the
gate census `shot: 6624` caught it before merge). The shipped clock is
`timer.cntpct() / timer.freq`: monotonic-by-contract, exact seconds,
host-test-deterministic (freq is a parameter).

The counter itself is NOT monotonic for the first ~0.9 s under VZ on
this host (observed twice: a ~0.1 s backward step at t≈0.89 s), so the
elapsed computation saturates instead of underflowing, and any trace
that sorts `t=` across the first second of a boot is misordered — a
finding for #1261, recorded where the guard is.

Measured on the nudge dying-boot scratch tree: samples on a 4.0 s
cadence to the end of a 120 s boot, spurious acks zero, beat deltas
1.000-1.002 s, `entry` cadence matching the 1 Hz timer — the PPIs are
delivered AND recorded to the end. Full finding on issue #1261.

Gate: live-forensics PASS 1/1 — census entry 245 / irq 238 / rearm 238 /
rotate 121 / shot 32 on a 120 s boot, shot args 4->8->...->124 exact.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SMP boot dies silently under WM traffic: the extra core-0 switch is survivable, the death is not guest-side

1 participant