Repository navigation
kdump on Apple silicon: crash kernel under a CTXR window, with retained DMA, pre-M4 NVMe after a crash - #1200
Merged
Conversation
…indow A CTXR window confines every core but the boot CPU to code inside the window, so kexec was refused outright and crashkernel= was ignored. A crash kernel only needs one CPU, and the boot CPU is not confined. Reserve the crash kernel as usual (the window is already reserved, so it lands outside it) and allow loading a crash image. When the panic is handled on a confined core, that core offers the crash kernel to CPU 0, which waits for it in its stop handler instead of parking, and parks itself. If CPU 0 never acknowledged the stop or does not take the offer within a second, the offer is withdrawn and the panic restarts the machine as before. Ordinary kexec stays refused. The crash kernel finds the same aurora,ctxr-exec-window in its device tree. It runs from ordinary memory on the boot CPU, so it keeps the secondaries where they are and leaves the window alone. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
On M1/M2-class ANS the coprocessor keeps running when the kernel is replaced without a shutdown, as when a crashed kernel starts the crash kernel. The new kernel finds CPU_CONTROL.RUN set and only wakes RTKit, but the NVMe controller is still enabled on the old kernel's queues. The admin and NVMMU queue registers may only be written while CC.EN is clear, so the controller keeps using the old queues and the new admin commands time out. On the first reset of a controller this kernel did not start, read CC from the hardware and disable the controller before programming the queues, as the post-M4 hand-over path already does. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
…mory The retained-memory interlock refuses every load and every kexec while a device still DMAs to memory that cannot be quiesced. That protects a new kernel that would hand such memory out again. A crash kernel never does: it runs in the crash kernel reservation, which memblock set aside at boot before any driver could give memory to a device, and it only reads the old kernel's memory. On the MacBook Neo the radio and display keep such DMA for the whole boot, so the interlock made kdump impossible there. Let crash images load and run while blockers are held, unless the crash kernel has CMA-backed memory (crashkernel=,cma), which is lent to the page allocator at run time. Ordinary kexec stays refused. The panic log names the first blocker when the crash kernel starts anyway. Update the KUnit cases to the new crash policy. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
A crashed kernel starts the crash kernel without stopping ANS, so on M1/M2-class machines the crash kernel finds the coprocessor still running the old kernel's RTKit session. The driver then only sends a wake, but a running session does not repeat HELLO and the endpoint map, so the handshake cannot complete and the crash kernel has no disk to save the dump on. In a crash kernel, when the coprocessor is found running on the first reset, take the old session over, disable the controller, shut RTKit down and clear CPU_CONTROL.RUN, so that the normal cold path resets and boots ANS afresh. If the old session does not answer, forget it and let the PMGR reset stop it. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
When the crashed kernel had the watchdog running, the crash kernel finds it enabled, marks it running and the watchdog core pings it until userspace opens it, which a crash kernel never does. A crash kernel that then stalls hangs the machine for good: on an M2 a stalled crash kernel ran for minutes past a 170 s watchdog that the crashed kernel had armed. When the crashed kernel had no watchdog running, nothing restarts a stalled crash kernel either. In a crash kernel, arm the watchdog at its longest timeout (about 178 s at 24 MHz) and do not mark it running, so the core leaves it unpinged. Saving a dump took 7 to 25 s on a J700 with crashkernel=320M; a crash kernel that has not restarted the machine by the timeout is stuck. Userspace that opens the watchdog takes it over as usual. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
RyanTheTide
marked this pull request as ready for review
November 10, 2026 07:25
This was referenced Nov 11, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kdump on Apple silicon: the crash kernel runs under a CTXR window, and pre-M4 NVMe works after a crash.
iBoot never keeps a panic record that Linux leaves behind (lane CRASH, 11-08), so kdump is how a Linux panic survives the restart on these Macs. The kernel config already had everything (KEXEC, KEXEC_FILE, CRASH_DUMP, CRASH_RESERVE, PROC_VMCORE come from the Fedora base), so this PR needs no config change. These things stopped the crash kernel on our hardware.
1. arm64: kexec: start the crash kernel from the boot CPU under a CTXR window
Under a CTXR window, every core except the boot CPU can fetch code only from inside the window, and the crash kernel lives in ordinary memory. Because of that, #985 refused kexec and ignored
crashkernel=. A crash kernel needs only one CPU, and the boot CPU isn't confined. With this change:crashkernel=is reserved as usual. The window is reserved first, so the crash kernel always lands outside it.aurora,ctxr-exec-windowin its device tree. It runs from ordinary memory on the boot CPU, so it keeps the secondaries where they are and leaves the window alone, with one info line and no error.2. nvme: apple: disable a controller handed over enabled on pre-M4 ANS
On M1/M2-class ANS the coprocessor keeps running across a crash. The crash kernel sees CPU_CONTROL.RUN set and only wakes RTKit, but the controller is still enabled on the crashed kernel's queues. The admin and NVMMU queue registers may only change while CC.EN is clear. So on the first reset of a controller this kernel didn't start, it now reads CC from the hardware and disables the controller before programming the queues, as the post-M4 handover path already does. On a normal boot, U-Boot leaves the controller disabled and nothing changes.
3. kexec: let a crash kernel start while devices keep DMA to retained memory
The retained-memory interlock (2164a24) refuses every kexec load and execution while a device still DMAs to memory that can't be quiesced. On the Neo, the radio and the display hold such memory for the whole boot, so even loading a crash kernel was refused (
kexec file load refused by active interlock; first reason: Neo radio admission or retained DMA ownership).The interlock protects a new kernel that would hand that memory out again. A crash kernel never does. It runs in the crash kernel reservation, which memblock sets aside at boot before any driver can give memory to a device, and it only reads the old kernel's memory. So crash images may now load and run while blockers are held. There is one exception: CMA-backed crash memory (
crashkernel=,cma) is lent to the page allocator at run time, so blockers still refuse a crash kernel that has any. Ordinary kexec stays refused. When the crash kernel starts despite a blocker, the panic log names the first one. The KUnit cases follow the new crash policy:kexec-block7/7 pass in QEMU.4. nvme: apple: restart pre-M4 ANS left running by a crashed kernel
A crashed kernel never stops ANS, so on M1/M2 the crash kernel finds the coprocessor still running the old RTKit session. A wake cannot work there: a running session does not repeat HELLO and the endpoint map. In a crash kernel only, when CPU_CONTROL.RUN is set on the first reset, the driver now takes the old session over, disables the controller, shuts RTKit down (best effort) and clears RUN, so the normal cold path resets and boots ANS afresh. On the M1 Pro:
crash kernel: ANS still running (CC=0x460001 CSTS=0x1), restarting it, then a fresh RTKit init, admin and I/O queues, and the disk. After this restart the controller comes up disabled, so commit 2 is not exercised on the M1 Pro; it stays for a session that does answer a wake.5. watchdog: apple: restart a stalled crash kernel
The crash kernel found the crashed kernel's watchdog enabled, marked it running, and the watchdog core then pinged it forever, because a crash kernel never opens /dev/watchdog. A stalled crash kernel hung the M2 for good (recovered with the VDM tether). In a crash kernel the watchdog is now armed at its longest timeout (~178 s at 24 MHz) and left unpinged, whether or not the crashed kernel had it running. Userspace that opens it takes it over as usual. Since then both pre-M4 Macs came back by themselves from every stalled crash kernel.
Tested
-cpu maxat EL2/VHE, 4 CPUs, a simulated windowctxr_exec_window=256M@0x40000000,crashkernel=256M, and a small static /init that loads the same kernel withkexec_file_load(ON_CRASH)and then triggers sysrq-c on a chosen CPU:CPU 2 is confined to the CTXR window, CPU 0 starts the crash kernel. The crash kernel boots on physical CPU 0 and reads /proc/vmcore (1957003264 bytes).crash kernel: running on the boot CPU, outside the windowat info level, brings up 1 CPU and reads /proc/vmcore.Hardware (kdump-utils 1.0.61 with the fedora-aurora#208 dracut pieces, sysrq-c, one-shot
crashkernel=):crashkernel=320Mcrashkernel=512M, crash initramfs withomit_driversomit_driverscrashkernel=320Mconsole=ttySAC0on the crash command line (/dev/console: No such device). Open bug: the default boot after it answered ssh and then hung (black screen, power button needed), with no journal left. Cause not found yet.omit_driversStill to do: an M2 row that saves a dump, and finding the Neo hang after the 11-09 crash. Commit 3 changes the policy of the retained-DMA interlock (2164a24), which deliberately refused crash execution, so it needs that author's review. This stays a draft until both are done.
Crash-kernel noise, all failing fast: macsmc RTKit -62 (no SMC RTC, so fedora-aurora#208 sets the clock from
CRASHTIME), and on the Neo AOP / t8140-bootstrap -12, display DARTs -22, GPU -5.The packaging half (kdump-utils with nvme_apple in the crash kernel's initramfs, kdump.service, crashkernel=) goes to fedora-aurora separately.
🤖 Generated with Claude Code