Skip to content

kdump on Apple silicon: crash kernel under a CTXR window, with retained DMA, pre-M4 NVMe after a crash - #1200

Merged
RyanTheTide merged 5 commits into
integration/aurorafrom
kdump/apple-crash-kernel
Nov 10, 2026
Merged

RyanTheTide merged 5 commits into
integration/aurorafrom
kdump/apple-crash-kernel

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

kdump on Apple silicon: the crash kernel runs under a CTXR window, and pre-M4 NVMe works after a crash.

iBoot never keeps a panic record that Linux leaves behind (lane CRASH, 11-08), so kdump is how a Linux panic survives the restart on these Macs. The kernel config already had everything (KEXEC, KEXEC_FILE, CRASH_DUMP, CRASH_RESERVE, PROC_VMCORE come from the Fedora base), so this PR needs no config change. These things stopped the crash kernel on our hardware.

1. arm64: kexec: start the crash kernel from the boot CPU under a CTXR window

Under a CTXR window, every core except the boot CPU can fetch code only from inside the window, and the crash kernel lives in ordinary memory. Because of that, #985 refused kexec and ignored crashkernel=. A crash kernel needs only one CPU, and the boot CPU isn't confined. With this change:

  • crashkernel= is reserved as usual. The window is reserved first, so the crash kernel always lands outside it.
  • A crash image can be loaded. Ordinary kexec is still refused.
  • When the panic is handled on a confined core, that core offers the crash kernel to CPU 0 and parks. CPU 0 waits for the offer in its stop handler instead of parking. One cmpxchg state makes taking the offer and withdrawing it exclusive. If CPU 0 never acknowledged the stop, or doesn't take the offer within 1 s, the offer is withdrawn and the panic restarts the machine as before.
  • The crash kernel sees the same aurora,ctxr-exec-window in its device tree. It runs from ordinary memory on the boot CPU, so it keeps the secondaries where they are and leaves the window alone, with one info line and no error.

2. nvme: apple: disable a controller handed over enabled on pre-M4 ANS

On M1/M2-class ANS the coprocessor keeps running across a crash. The crash kernel sees CPU_CONTROL.RUN set and only wakes RTKit, but the controller is still enabled on the crashed kernel's queues. The admin and NVMMU queue registers may only change while CC.EN is clear. So on the first reset of a controller this kernel didn't start, it now reads CC from the hardware and disables the controller before programming the queues, as the post-M4 handover path already does. On a normal boot, U-Boot leaves the controller disabled and nothing changes.

3. kexec: let a crash kernel start while devices keep DMA to retained memory

The retained-memory interlock (2164a24) refuses every kexec load and execution while a device still DMAs to memory that can't be quiesced. On the Neo, the radio and the display hold such memory for the whole boot, so even loading a crash kernel was refused (kexec file load refused by active interlock; first reason: Neo radio admission or retained DMA ownership).

The interlock protects a new kernel that would hand that memory out again. A crash kernel never does. It runs in the crash kernel reservation, which memblock sets aside at boot before any driver can give memory to a device, and it only reads the old kernel's memory. So crash images may now load and run while blockers are held. There is one exception: CMA-backed crash memory (crashkernel=,cma) is lent to the page allocator at run time, so blockers still refuse a crash kernel that has any. Ordinary kexec stays refused. When the crash kernel starts despite a blocker, the panic log names the first one. The KUnit cases follow the new crash policy: kexec-block 7/7 pass in QEMU.

4. nvme: apple: restart pre-M4 ANS left running by a crashed kernel

A crashed kernel never stops ANS, so on M1/M2 the crash kernel finds the coprocessor still running the old RTKit session. A wake cannot work there: a running session does not repeat HELLO and the endpoint map. In a crash kernel only, when CPU_CONTROL.RUN is set on the first reset, the driver now takes the old session over, disables the controller, shuts RTKit down (best effort) and clears RUN, so the normal cold path resets and boots ANS afresh. On the M1 Pro: crash kernel: ANS still running (CC=0x460001 CSTS=0x1), restarting it, then a fresh RTKit init, admin and I/O queues, and the disk. After this restart the controller comes up disabled, so commit 2 is not exercised on the M1 Pro; it stays for a session that does answer a wake.

5. watchdog: apple: restart a stalled crash kernel

The crash kernel found the crashed kernel's watchdog enabled, marked it running, and the watchdog core then pinged it forever, because a crash kernel never opens /dev/watchdog. A stalled crash kernel hung the M2 for good (recovered with the VDM tether). In a crash kernel the watchdog is now armed at its longest timeout (~178 s at 24 MHz) and left unpinged, whether or not the crashed kernel had it running. Userspace that opens it takes it over as usual. Since then both pre-M4 Macs came back by themselves from every stalled crash kernel.

Tested

  • Build: the fleet recipe (Fedora base + asahi.config, 145/0 settings survive, config identical to fleet31). W=1 on the touched files adds 0 warnings. checkpatch is clean.
  • QEMU virt, TCG, -cpu max at EL2/VHE, 4 CPUs, a simulated window ctxr_exec_window=256M@0x40000000, crashkernel=256M, and a small static /init that loads the same kernel with kexec_file_load(ON_CRASH) and then triggers sysrq-c on a chosen CPU:
    • Panic pinned to CPU 2: CPU 2 is confined to the CTXR window, CPU 0 starts the crash kernel. The crash kernel boots on physical CPU 0 and reads /proc/vmcore (1957003264 bytes).
    • Panic on CPU 0, with the window also on the crash kernel's command line: no hand-off. The crash kernel logs crash kernel: running on the boot CPU, outside the window at info level, brings up 1 CPU and reads /proc/vmcore.
    • QEMU doesn't enforce CTXR, so these runs check the hand-off logic and the crash kernel's window handling, not the hardware fetch restriction.

Hardware (kdump-utils 1.0.61 with the fedora-aurora#208 dracut pieces, sysrq-c, one-shot crashkernel=):

Mac Kernel Result
Neo (J700, CTXR window) kdump2 = commits 1-3, crashkernel=320M Panic on confined CPU 3: hand-off to CPU 0, vmcore 147 MiB saved, back by itself 37 s after the panic. Panic on CPU 0: vmcore 190 MiB in 7 s, back after 29 s.
M1 Pro (T6000) fleet33 + crash kernel = this branch at c2824e0 (+ a local debug commit that only moves the watchdog deadline), crashkernel=512M, crash initramfs with omit_drivers Panic on CPU 2: commit 4 restarted ANS, vmcore 168 MiB + logs saved, back by itself 42 s after the panic.
M1 Pro, M2 (T8112) same crash kernel, crash initramfs without omit_drivers No dump: nvme_apple never probed (udev stuck on live-coprocessor drivers, fixed in fedora-aurora#208); commit 5's watchdog restarted both after ~3 min.
M2 kdump2 (before commits 4-5) No dump, hung until a VDM reset.
Neo, 11-09 kdump7 = these 5 commits on integration 2362d9a (fleet42), crashkernel=320M Panic on confined CPU 3: hand-off to CPU 0, crash kernel watchdog armed, NVMe up, dump dir dated to the panic second (fedora-aurora#208 clock hook), vmcore-dmesg saved. No vmcore because of a test-only console=ttySAC0 on the crash command line (/dev/console: No such device). Open bug: the default boot after it answered ssh and then hung (black screen, power button needed), with no journal left. Cause not found yet.
M2, 11-09 fleet42 + crash kernel kdump7, crash initramfs with omit_drivers No dump: restarted by the crash kernel's watchdog after 190 s. The M1 Pro passes with the same pieces, so something M2-specific still stalls; suspects are the I2C controller of the USB-C port controllers (the M2's cd321x interrupt storm) and spi-apple. Next run uses a tighter omit list and progress markers.

Still to do: an M2 row that saves a dump, and finding the Neo hang after the 11-09 crash. Commit 3 changes the policy of the retained-DMA interlock (2164a24), which deliberately refused crash execution, so it needs that author's review. This stays a draft until both are done.

Crash-kernel noise, all failing fast: macsmc RTKit -62 (no SMC RTC, so fedora-aurora#208 sets the clock from CRASHTIME), and on the Neo AOP / t8140-bootstrap -12, display DARTs -22, GPU -5.

The packaging half (kdump-utils with nvme_apple in the crash kernel's initramfs, kdump.service, crashkernel=) goes to fedora-aurora separately.

🤖 Generated with Claude Code

…indow

A CTXR window confines every core but the boot CPU to code inside the
window, so kexec was refused outright and crashkernel= was ignored. A
crash kernel only needs one CPU, and the boot CPU is not confined.

Reserve the crash kernel as usual (the window is already reserved, so
it lands outside it) and allow loading a crash image. When the panic is
handled on a confined core, that core offers the crash kernel to CPU 0,
which waits for it in its stop handler instead of parking, and parks
itself. If CPU 0 never acknowledged the stop or does not take the offer
within a second, the offer is withdrawn and the panic restarts the
machine as before. Ordinary kexec stays refused.

The crash kernel finds the same aurora,ctxr-exec-window in its device
tree. It runs from ordinary memory on the boot CPU, so it keeps the
secondaries where they are and leaves the window alone.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
On M1/M2-class ANS the coprocessor keeps running when the kernel is
replaced without a shutdown, as when a crashed kernel starts the crash
kernel. The new kernel finds CPU_CONTROL.RUN set and only wakes RTKit,
but the NVMe controller is still enabled on the old kernel's queues.
The admin and NVMMU queue registers may only be written while CC.EN is
clear, so the controller keeps using the old queues and the new admin
commands time out.

On the first reset of a controller this kernel did not start, read CC
from the hardware and disable the controller before programming the
queues, as the post-M4 hand-over path already does.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
…mory

The retained-memory interlock refuses every load and every kexec while a
device still DMAs to memory that cannot be quiesced. That protects a new
kernel that would hand such memory out again. A crash kernel never does:
it runs in the crash kernel reservation, which memblock set aside at
boot before any driver could give memory to a device, and it only reads
the old kernel's memory. On the MacBook Neo the radio and display keep
such DMA for the whole boot, so the interlock made kdump impossible
there.

Let crash images load and run while blockers are held, unless the crash
kernel has CMA-backed memory (crashkernel=,cma), which is lent to the
page allocator at run time. Ordinary kexec stays refused. The panic log
names the first blocker when the crash kernel starts anyway. Update the
KUnit cases to the new crash policy.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
@RyanTheTide RyanTheTide changed the title kdump on Apple silicon: crash kernel under a CTXR window, pre-M4 NVMe after a crash kdump on Apple silicon: crash kernel under a CTXR window, with retained DMA, pre-M4 NVMe after a crash Nov 8, 2026
A crashed kernel starts the crash kernel without stopping ANS, so on
M1/M2-class machines the crash kernel finds the coprocessor still
running the old kernel's RTKit session. The driver then only sends a
wake, but a running session does not repeat HELLO and the endpoint map,
so the handshake cannot complete and the crash kernel has no disk to
save the dump on.

In a crash kernel, when the coprocessor is found running on the first
reset, take the old session over, disable the controller, shut RTKit
down and clear CPU_CONTROL.RUN, so that the normal cold path resets and
boots ANS afresh. If the old session does not answer, forget it and let
the PMGR reset stop it.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
When the crashed kernel had the watchdog running, the crash kernel finds
it enabled, marks it running and the watchdog core pings it until
userspace opens it, which a crash kernel never does. A crash kernel that
then stalls hangs the machine for good: on an M2 a stalled crash kernel
ran for minutes past a 170 s watchdog that the crashed kernel had armed.
When the crashed kernel had no watchdog running, nothing restarts a
stalled crash kernel either.

In a crash kernel, arm the watchdog at its longest timeout (about 178 s
at 24 MHz) and do not mark it running, so the core leaves it unpinged.
Saving a dump took 7 to 25 s on a J700 with crashkernel=320M; a crash
kernel that has not restarted the machine by the timeout is stuck.
Userspace that opens the watchdog takes it over as usual.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
@RyanTheTide
RyanTheTide marked this pull request as ready for review November 10, 2026 07:25
@RyanTheTide
RyanTheTide merged commit 97d2b7c into integration/aurora Nov 10, 2026
1 check passed
@RyanTheTide
RyanTheTide deleted the kdump/apple-crash-kernel branch November 16, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant