Repository navigation
arm64: run Linux at EL2 with its code confined to a CTXR window - #985
Conversation
Add CONFIG_EXECMEM_CONFINE: an optional backing mode for execmem ranges flagged EXECMEM_CONFINE. Such ranges keep their virtual placement (so a module stays within branch range of the kernel) but draw their physical backing pages from a single fixed window registered by the architecture via execmem_confine_init(), rather than from arbitrary buddy pages. This lets a platform keep every run-time executable physical page (modules, BPF, kprobes, ftrace) inside a hardware-enforceable code region such as an Apple-silicon CTXR window, which is a prerequisite for running the kernel at that privilege level. A confined allocation reserves its virtual area with __get_vm_area_node() inside the range, pulls a physically-contiguous chunk from a gen_pool over the window, and maps it with vmap_pages_range(); the backing pages are published on the vm_struct so set_memory_rox() can mirror permission changes onto their linear alias. If the pool is exhausted the allocation fails, with no buddy fallback, mirroring a hardware execute-range fault. A verifier sweeps for any executable mapping that resolves outside the window (the page-table sweep itself is provided per-architecture); when the window is enforcing a violation is fatal. "ctxr_exec_selftest" adds a boot-time positive/negative self-check. Signed-off-by: Diljit Singh <hey@djdev.me>
On T8140/M4-class Apple-silicon machines the boot firmware arms a per-CPU
code text execute range (CTXR) on the secondary cores: at the privileged
exception level an instruction may only be fetched from inside a fixed
physical window. To run the kernel there rather than dropping it to EL1,
every executable physical page must live inside one window.
Add CONFIG_ARM64_CTXR_EXECMEM, the arm64 end of the EXECMEM_CONFINE
contract:
- ctxr_exec_reserve(), from arm64_memblock_init(), learns the window
from /chosen/aurora,ctxr-exec-window (emitted by the EL2 monitor) or
the ctxr_exec_window= command line (for VM bring-up), checks that the
kernel image fits, and reserves it from memblock.
- execmem_arch_setup() carves the non-image remainder into a gen_pool,
registers it with the confinement core, and flags the executable
ranges (module text, BPF, kprobes and, by inheritance, ftrace) with
EXECMEM_CONFINE. Module data is left on ordinary vmalloc backing.
- execmem_confine_arch_verify() provides the page-table sweep used by
the verifier: every privileged-executable leaf outside the window is
reported, standing in for the hardware fault when testing in a VM.
The option is inert unless a window is supplied at boot, so it is enabled
in asahi.config and is a no-op on machines that do not need it.
Signed-off-by: Diljit Singh <hey@djdev.me>
|
Merge train (MTL, 06/11): holding this one for now, not merged. On the current fleet nothing changes (no window is supplied, so
Smaller things: Since the asahi.config change puts this in every distro kernel, I'd like 1 and 2 fixed before it lands, so that whatever aura starts publishing a window doesn't hit them. Happy to re-run the train as soon as it's updated. |
…ndow Build the executable pool only from memory that was free before the window was reserved. Snapshot for_each_free_mem_range() over the window just before memblock_reserve(), so anything already reserved at that point -- the image, the device tree blob, an initrd, /reserved-memory and memreserve entries, EFI runtime regions and the UEFI memory map -- and any hole or no-map range stays out of the pool. Previously the pool was the window minus the image only, so a lowest-first executable allocation could overwrite a live DTB, an initrd or EFI runtime memory that a loader had placed inside the window. When no base is given and anchoring the window on the image would run past the end of DRAM (a loader placed the image within the window size of the top), slide the window down to end at DRAM's end, still covering the image, instead of panicking. List the pool's ranges in /sys/kernel/debug/execmem_confine. Signed-off-by: Diljit Singh <hey@djdev.me>
The kernel image and the code of every module take about 66 MiB, and RAM in the window that the pool doesn't use is lost to the rest of the system. 128 MiB holds them with room to spare. Signed-off-by: Diljit Singh <hey@djdev.me>
Picks up the bindgen fix (#992), without which the asahi profile does not build. Signed-off-by: Diljit Singh <hey@djdev.me>
Kernel text inside an armed CTXR window can't execute at EL1 on a core the window is enforced on, so the usual EL1 detour on entry faults. Once the boot CPU has found an armed window, cores entering after it switch to VHE in init_kernel_el and never run at EL1. Signed-off-by: Diljit Singh <hey@djdev.me>
Kernel text inside an armed CTXR window can't execute at EL1 on a core the window is enforced on, and the window isn't known when the boot CPU first enters. Apple cores are VHE only, so with CONFIG_ARM64_CTXR_EXECMEM they now switch to VHE in init_kernel_el on every entry, boot CPU included, and never run at EL1. This replaces the flag the boot CPU set for later cores. Signed-off-by: Diljit Singh <hey@djdev.me>
The profile had no dm-crypt, so a kernel built from it can't unlock an encrypted root. Add it with the crypto interfaces LUKS uses, and the modules an Aurora install's services load. Signed-off-by: Diljit Singh <hey@djdev.me>
The vDSO's code is part of the kernel image, which sits in the CTXR window, and EL0 can't execute there on a core the window is enforced on. A process calling into the vDSO on such a core takes a permission fault that the kernel finds nothing to fix, and retries it forever. Map a copy from ordinary memory into userspace instead. Signed-off-by: Diljit Singh <hey@djdev.me>
A core that enforces the CTXR window doesn't let EL0 execute from it either. The fetch takes a permission fault that the page tables don't explain, so the kernel retries it forever. The window is reserved, but free_reserved_area() can still release parts of it: __init memory, which is part of the image, and an initrd placed in the window. Those pages reach the page allocator and can become user pages. Keep them reserved, and have the verifier count any window page that isn't reserved as a violation. Signed-off-by: Diljit Singh <hey@djdev.me>
With CONFIG_ARM64_CTXR_EXECMEM every Apple core switched to VHE in init_kernel_el, whether or not it has CTXR and whether or not a window was supplied, so a kernel built with the option took a different entry path on M1 to M3. Only M4-generation and later cores have CTXR. Their part numbers start at 0x50, and every earlier core's is below that, so limit the direct switch to them. Earlier cores drop to EL1 and come back through finalise_el2 as before. Signed-off-by: Diljit Singh <hey@djdev.me>
The copy was made whenever CONFIG_ARM64_CTXR_EXECMEM was on. With no window there is nothing to keep the vDSO out of, so map the image's pages as before. Signed-off-by: Diljit Singh <hey@djdev.me>
…t run EFI runtime code runs at EL2 on whichever core efi_rts_wq is on. On a core that enforces the window, code outside it faults, and the fault handler dies with "execution of user memory" before the EFI fixup runs. So when a window is enforcing and any EFI_RUNTIME_SERVICES_CODE region is outside it, warn once and turn runtime services off for that boot. Misaligned code counts too: on a 16K kernel it makes the misaligned regions after it executable, wherever they are, and the verifier would then panic on them. The window is read from this file rather than through execmem_confine_*(), since execmem_init() is a later core_initcall than arm64_efi_rt_init(). Have the verifier sweep efi_mm as well, so its report covers the runtime mappings that stay on. Signed-off-by: Diljit Singh <hey@djdev.me>
On a core enforcing the window, an EL0 fetch from a window page takes a level 3 permission fault that the page tables don't explain. handle_mm_fault() finds nothing to fix, so the task retries forever: it stays runnable, with no panic and no hung-task report. The zero page stays in the window, so a program jumping into fresh executable memory before writing it hits this. When a window is enforcing and such a fault lands on a present, executable mapping of a window page, log the task and kill it. Every other fault takes the path it took before. Signed-off-by: Diljit Singh <hey@djdev.me>
…window The relocation code runs from an ordinary page, outside the window, and a crash dump can start on a core enforcing it, where it would hang during a panic. Fail the load cleanly instead, with one message, so nothing finds out only when it needs the second kernel. Signed-off-by: Diljit Singh <hey@djdev.me>
A kernel in a window runs at EL2 with no monitor below it, so the bootloader is what supplies the window. Say so, and document the entry path. Signed-off-by: Diljit Singh <hey@djdev.me>
ctxr_exec_enforce= was only read along with ctxr_exec_window=, so it did nothing to a window from the device tree. strstr() matched inside other words and took the first use of an option. A value that didn't parse gave a default window, and a window too small for the image panicked a machine that would otherwise boot. Use early_param(), which runs before arm64_memblock_init(). The last use of an option wins, and ctxr_exec_enforce= applies to a window from either place. A value that doesn't parse is ignored with a warning. So is a window from the command line that can't work, which leaves the device tree's window if there is one. A window from the device tree that can't work is still fatal when enforcing, since the kernel can't run on the confined cores outside it. Signed-off-by: Diljit Singh <hey@djdev.me>
The pool leaves out whatever was reserved before the window, but two paths still took window pages to the page allocator, and the verifier then panicked at boot. A reusable CMA region inside the window goes to the page allocator from cma_activate_area(). Don't activate a CMA area that overlaps the window, and keep its pages reserved. A reserved-memory node with a "reg" whose init fails is freed back to memblock from unflatten_device_tree(), after the window was reserved. That leaves a hole for memblock_free_all() to release, so reserve the window again before that. Record the window when it's reserved rather than when the pool is set up, because execmem_init() and CMA activation are both core_initcalls. Signed-off-by: Diljit Singh <hey@djdev.me>
__vmalloc_node_range() unpoisons a VM_ALLOC area once it's mapped and registers it with kmemleak, and vfree() unregisters it. A confined allocation goes around both, so on a debug kernel KASAN sees the area as poisoned and kmemleak doesn't know it. Do the same here. Signed-off-by: Diljit Singh <hey@djdev.me>
Describe the /chosen properties a bootloader uses to give the kernel its executable window. Signed-off-by: Diljit Singh <hey@djdev.me>
Add the command-line options to kernel-parameters.txt and describe how they treat bad values. Say that a confined machine loses CPU hotplug and kexec, and that hibernate is untested under a window. aura doesn't emit aurora,ctxr-exec-enforce, so it's no longer shown as part of what the bootloader emits. The Kconfig help said to say N if unsure while asahi.config turns the option on, so say who wants it. Signed-off-by: Diljit Singh <hey@djdev.me>
|
Thanks for the review. It covered the first version, so each item was checked against the current branch, head Already fixed before this round
Fixed now
Left as is: hibernate's With no window, the entry path on M1 to M3 parts takes the usual EL1 detour, traced in QEMU with a MIDR override rather than booted on a real M1 to M3, and the feature's code paths stay inert. This head isn't merged with |
…r not Whether a core can fetch from outside the window is decided by the hardware, not by aurora,ctxr-exec-enforce or ctxr_exec_enforce=. Those only decide whether a verifier finding is fatal. EFI runtime services, kexec and the EL0 fetch check keyed on enforcement, so a confined machine booted with enforcement off, now possible from the command line, would run EFI runtime code from efi_rts_wq on a confined core, allow a kexec that hangs, and leave a task faulting forever on the zero page. Key all three on the window being active. Assisted-by: LLM Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
empty_zero_page is in the image's .bss, so it is inside the window. Any anonymous mapping that is read before it is written maps it, including PROT_EXEC ones, userfaultfd's UFFDIO_ZEROPAGE and KSM's zero pages, and KVM can map it into a guest's stage 2 for untouched guest RAM. EL0 can't execute from the window on a confined core, so an unprivileged program that jumps into such memory is killed there and gets SIGILL anywhere else, and a guest fetching from it may fault to EL2 again and again. With a window reserved, allocate the zero pages from memblock, outside the window, and clear their MTE tags as cpu_enable_mte() does for empty_zero_page. The fault-path kill stays as a backstop. Assisted-by: LLM Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
ctxr_exec_selftest checked, once per boot, that a confined allocation lands in the window and that the page-table sweep notices an executable mapping placed outside it. /sys/kernel/debug/execmem_confine reran the sweep and printed the pool. Both were test aids for bring-up, and the selftest mapped a page RWX to do its second check. Remove them, and make execmem_confine_verify() static, since the boot-time check is now its only caller. The boot-time verifier stays, and so does the EL0 fetch kill, neither of which depends on what goes here. ctxr_exec_enforce= now only decides whether a verifier finding at late_initcall panics. Signed-off-by: Diljit Singh <hey@djdev.me>
5863e24 added dm-crypt, the crypto user APIs, the sound sequencer, Bluetooth RFCOMM and BNEP, INET_UDP_DIAG, joydev, uinput and HIDRAW to the asahi fragment. None of that is about the CTXR window. Drop it and keep CONFIG_ARM64_CTXR_EXECMEM=y with its comment. The fleet's Fedora Asahi base config already has every one of them, and four of these lines turned the base's built-in BLK_DEV_DM and CRYPTO_USER_API_{HASH,SKCIPHER,AEAD} into modules, so the fleet kernel gets those back as they were. Signed-off-by: Diljit Singh <hey@djdev.me>
integration/aurora appended CONFIG_EDAC_APPLE_SOC=m to the asahi fragment, where this branch appends CONFIG_ARM64_CTXR_EXECMEM=y. Keep both, the EDAC line first, so the branch's change to the fragment stays the CTXR block alone. Signed-off-by: Diljit Singh <hey@djdev.me>
|
J813/T8142 (M5 MacBook Air) boot result: Linux runs at EL2 with its code confined to the window. I merged this branch (head a1a2ac0) into the J813 integration stack as 6e2897803fd1 and built with dmesg
|
init_el2 read ctxr_stay_el2 with the MMU off on every entry, the boot CPU's first one included. That entry runs before ctxr_exec_reserve() writes the flag, and nothing cleaned the .mmuoff.data.read line to the PoC: primary_entry cleans only the idmap text and init_el2 the HYP text, and the stub's relocation syncs code to the PoU. So on an M1-M3 machine with no window, stale DRAM from an earlier ctxr_exec_window= boot, or garbage, could send the boot CPU down the direct VHE path, running finalise_el2_state before init_feature_override() and ignoring kvm-arm.mode= and the VHE overrides. The secondaries then read the same uncleaned line. The boot CPU is never confined, so it now always takes the stock entry: init_kernel_el reads the flag only when x22 is 1, which just the secondary entry points and cpu_resume set; primary_entry sets it to 0. ctxr_exec_reserve() writes the flag on every boot, 0 when no window is active, and cleans it to the PoC before any secondary is released, so a confined secondary or a resuming CPU reads a defined value and no stale line survives. Signed-off-by: Diljit Singh <hey@djdev.me>
A successful ctxr_exec_window= returned before the device-tree checks, so it never saw the EL2 test or the hold path, and nothing compared it with the firmware's window. At EL1 it would confine the kernel's text and still release the secondaries; a window unrelated to the one the firmware armed would place code where a confined core can't fetch it. Route both the same way: a window from either place needs the kernel at EL2, and a command-line window is accepted only when it lies inside the device-tree window, if there is one. Anything else keeps the secondaries offline, as a bad device-tree window already did. A command-line window with no device-tree window (VM bring-up) is unchanged except that it now requires EL2. Update the binding doc and kernel-parameters.txt. Signed-off-by: Diljit Singh <hey@djdev.me>
The backstop only walked a folio, so a VM_PFNMAP or struct-page-less VM_MIXEDMAP mapping was never checked, although the comment said the PFN might have no struct page. With the zero page now outside the window, a driver's remap_pfn_range() of an image buffer is the likeliest way a window page reaches EL0, and that is exactly a PFNMAP with no folio. Fall back to follow_pfnmap_start() for such a VMA when the folio walk finds nothing, and compare that PFN against the window as before. The doc's claim is corrected to match. Signed-off-by: Diljit Singh <hey@djdev.me>
unflatten_device_tree() runs fdt_scan_reserved_mem_late(), which frees a reserved-memory node whose late init fails and can punch a hole in the window. bootmem_init() then reserves the crash kernel, which could land in that hole and later be handed to the buddy as a window page through free_reserved_page(). Close the hole at the start of bootmem_init(), before arch_reserve_crashkernel(), instead of in arch_mm_preinit() after it. Signed-off-by: Diljit Singh <hey@djdev.me>
cma_activate_area() allocated each range's bitmap first and only then checked whether the area overlaps the executable window. A bitmap allocation failure jumps to cleanup, which frees the area's pages to the buddy unless the reserve-on-error bit is set, so an overlapping area could leak window pages into the allocator on that path. Do the overlap check, which sets that bit, before the bitmap loop. Signed-off-by: Diljit Singh <hey@djdev.me>
execmem_confine_alloc() drew the whole allocation from the window with a single gen_pool_alloc(), which needs that many physically contiguous bytes free. The pages are vmapped one by one, so contiguity isn't needed, and a fragmented window could fail a 2 MiB BPF pack or a large module's text with enough free pages to spare. Allocate a page at a time, record the pages, and return them to the pool individually on free and on every error path. While here, use kvmalloc_array() for the page array, which can be large, and warn if set_memory_nx()/set_memory_rw() fail on free rather than leaving a possible executable alias behind without a word. Signed-off-by: Diljit Singh <hey@djdev.me>
When a window is active __vdso_init() copies each vDSO code page into a fresh page. If one allocation failed it returned -ENOMEM leaving the page-list array and the copies already made behind. Free them on the error path, taking care not to free the image pages used when no window is active. Signed-off-by: Diljit Singh <hey@djdev.me>
…olds The bootloader places the static image in the window; the kernel doesn't. And the pool is built from the window's free ranges snapshotted when it was reserved, not from a "non-image remainder". Fix the Kconfig help, the documentation and the file comment. Signed-off-by: Diljit Singh <hey@djdev.me>
apple_cpuidle_init() open-coded the M1 to M3 compatibles that apple_cyc_ovrd_machines already lists. Match against that list and add only the M4-and-later parts that load the driver for its WFI state (T6050, T8140), so the two can't drift apart. Signed-off-by: Diljit Singh <hey@djdev.me>
|
First hardware boots with a real CTXR window: the Neo (J700, T8140), 07/11. Setup
Boot (both kernels)
Stress on
Issues
Not from this PR, seen on the way
From the Neo side this is ready. The J616s gets the same run next. 🤖 Generated with Claude Code |
bpf_jit_limit, the cap on unprivileged JIT memory, defaults to half of bpf_jit_alloc_exec_limit(), which arm64 reports as the vmalloc space (about 64 TiB). Under a CTXR window every JIT image comes from the window's executable pool instead, about 53 MiB on a J700. On that machine an unprivileged user (uid 65534, no capabilities) filled the pool with 1379 classic socket filters in 0.3 s, after which root could no longer load a 4 MiB module. Record how much of the window the pool gets when the window is reserved, which is before bpf_jit_charge_init() runs, and report that as the JIT allocation limit. bpf_jit_limit then defaults to half of the pool, and bpf_jit_limit_max to the pool. Without a window nothing changes. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
bpf_jit_limit only charges unprivileged programs. Privileged BPF (cgroup programs from the init system, tracing tools) can still fill the confined pool, and then no module loads. Add EXECMEM_CONFINE_BOUNDED: a confined range with it never takes the pool below a reserve, a quarter of the pool, kept for the other confined ranges. arm64 sets it on the BPF range. Hold confine_lock across the page loop so the check still holds when the last page is taken, and free the pages of a failed allocation under the same lock. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
|
Pushed two commits for the unprivileged JIT fill from my last comment. The head is now What the bug was: a local denial of service. It isn't an escape, since nothing runs outside the window. Commits
Both commits do nothing without a window. W=1 shows 0 warnings on the four objects they touch. Neo (J700), padded Stage 1,
QEMU (TCG, KASAN, No window, M1 Pro: a one-shot of the same tree passed. All 10 CPUs came up at EL2. The same Neo boot also went through u-boot#32's head 🤖 Generated with Claude Code |
|
Thanks for the run, the BPF fix and the merge. Five commits landed on
All QEMU only, 11 of 11 window scenarios. Your Still open: on your Neo through U-Boot and GRUB, 74 MiB of the window was already taken, against 17 MiB on sithglan's M5. If you still have that boot's dmesg or 🤖 Generated with Claude Code |
Brings aurora-wip up to the tested release branch: CTXR execmem (#985, #1079), KFIX speaker sleep and resume fixes (#1077, #1080), JPEG v2 (#1072), Jonathan's G17 and USB work (#1073-#1076), brcmfmac flowring count (#1081), ANE H16 startup restart (#1066) and the Touch ID per-prompt match (#904). Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com> Assisted-by: LLM
Linux at EL2 with no monitor on M4-generation and later Macs. All of its executable code stays in one physical window enrolled in CTXR. On a confined secondary, EL2 can only fetch from inside the window, EL0 can't execute there, and EL1 executing there faults to EL2. CPU0 isn't confined.
Merged at
273b9e8c846.f6b1cbf1bc1has booted on the Neo with a real 128 MiB window (below); Ryan's two commits after it are in this merge, and five more are in #1079. With no window (no/chosen/aurora,ctxr-exec-window, noctxr_exec_window=), the kernel behaves asintegration/auroradoes, entry path included.What's in it
The window and the pool (from before):
mm/execmem(CONFIG_EXECMEM_CONFINE) backs module, BPF, kprobe and ftrace text with pages from agen_poolover the window. There's no buddy fallback.arm64(CONFIG_ARM64_CTXR_EXECMEM) learns the window from/chosen/aurora,ctxr-exec-windoworctxr_exec_window=, checks that the image fits, and reserves it. The reservation is the last step ofarm64_memblock_init(), after the initrd andearly_init_fdt_scan_reserved_mem(), and the pool holds only memory that was free just before it. A verifier sweeps the page tables and panics on an escape when the window is enforcing.Brought over from the stack that booted, patch-ids unchanged:
72880a44365and73b99c17e60stay at EL2 on entry.5863e2492e8added dm-crypt and what an installed system's services load to the asahi profile.0941e6f6831takes that back out (below).207bc4d748emaps a copy of the vDSO from ordinary memory.4b630f74a37keeps window pages out of the page allocator.New in the first round, none of it booted on hardware:
fe8741d4f6dlimited the direct VHE entry to part numbers from 0x50.46c5ae7cd01(below) replaces that test.0412b237dc1copies the vDSO only when a window is active.d9f915b119dturns EFI runtime services off with one warning when a window is active and anyEFI_RUNTIME_SERVICES_CODEregion is outside it or misaligned. The verifier sweepsefi_mmtoo.a64c54e49a6kills a task, with an uncatchableSIGSEGVand a ratelimited log line, when an EL0 instruction permission fault lands on a present, executable mapping of a window page.0ab88ef01d7makesmachine_kexec_prepare()fail withEOPNOTSUPPwhile a window is active.2e6e70017cedocs.From the first review, not booted on hardware:
d9de654671eearly_param()parsing.ctxr_exec_enforce=applies to a device-tree window, options don't match inside other words, the last use wins, and a bad command-line value or window is ignored with a warning.d637de6296fkeeps reusable CMA and failed reserved-memory nodes out of the window.b3a9578538fKASAN and kmemleak for confined allocations.524f9114cf1the binding for/chosen/aurora,ctxr-exec-window(andaurora,ctxr-exec-enforce, since dropped).c4b5fab28dddocs and the Kconfig help.From Ryan's second round (
1c4ce70047b,20519af6fc3,a57207316f8,e82a8514a6f): an active window counts as confining whether or not it is enforcing, the zero page moves out of the window, the direct entry checkedID_AA64MMFR1_EL1.VH(since dropped), and the TLB is flushed before a confined chunk goes back to the pool.This round, not booted on hardware:
46c5ae7cd01keeps the boot CPU on the usual entry path. The direct VHE entry was keyed on the part number, so every M4-generation core took it on every entry, window or not, and CPU0 took it before the feature overrides were parsed. CPU0 isn't confined, so it now enters as it always has (EL1, thenfinalise_el2). Once it has reserved a window at EL2 it setsctxr_stay_el2(in.mmuoff.data.read, cleaned to the PoC), and only later entries that read it set take the direct path. With no window, every core takes the stock path, andkvm-arm.mode=applies to CPU0 again.91c85e7bf4fand52252329df0keep the secondaries offline when the device tree's window can't be used, instead of panicking (enforcing) or releasing them anyway (ctxr_exec_enforce=0). Unusable means a malformed property (not four cells, size 0, not page aligned, wraps), the kernel not at EL2, an image that doesn't fit, a window outside DRAM, a failed reservation, or no free page left for the pool. The kernel logs why and boots on CPU0;smp_prepare_cpus()doesn't prepare or mark present any secondary, so no release address is written. The empty-pool check runs at reservation, because on arm64 the pool is built atcore_initcall, aftersmp_init(). Enforcement now only decides whether a verifier finding is fatal. The one panic left is a failed allocation while building the pool, when confined CPUs are already running.6ebbafe2175applies the Apple vGIC (broken SEIS) workaround to T8140. Without it KVM at EL2 on the Neo uses one list register (GICv3 sysreg trapping enabled ([C])in the 2026-11-05 EL2 boot log) and SMP guests livelock, as on T6040 beforefcb377a2544. ICH_VTR_EL2 reads0x90580007on both. It only applies with the kernel at EL2.ddb50b42408offers cpuidle's CPU PD state (which writes CYC_OVRD) only on M1 to M3 compatibles, instead of denying a list. Every device tree in the tree gets the same states as before.a3043575d91updates the apple-dcpcpuidle-s2idlehost test, which replays the probe with mocks and failed CI at52252329df0; it now covers T8103, T8140, T6041 and T8142 (142 assertions pass locally).131e02f6757mergesintegration/aurora(b452c284930): it added T6050 to the cpuidle deny list, which conflicted withddb50b42408; the allowlist already covers it.Simplified with the owner's approval, not booted on hardware:
51301b05422(coordinator) corrects the doc: the boot CPU'sCTXR_Ddoes describe the window (aura read[0x1000667c000, 0x10017950000)on CPU0 on 11-05, the range a secondary enforces); only enforcement is off there. The reason given in91c85e7bf4f's message for not cross-checking is wrong in the same way.65da3475d81drops theID_AA64MMFR1_EL1.VHtest from the direct entry: only a boot CPU already running VHE at EL2 sets the flag.c89843af86ddropsaurora,ctxr-exec-enforceand windows without a base. A device-tree base of 0 is malformed, andctxr_exec_window=takes<size>@<base>, both required, so the default size and the anchor-on-the-image code go. Binding, doc andkernel-parameters.txtfollow.d1eba7c3ee2drops the uncalledexecmem_confine_window()andexecmem_confine_enforced().9657b6e15dbdropsctxr_exec_selftestand/sys/kernel/debug/execmem_confine. The boot-time verifier and the EL0 fetch kill stay.ctxr_exec_enforce=now does one thing: whether a boot-time verifier finding panics (default) or is only logged.0941e6f6831leaves onlyCONFIG_ARM64_CTXR_EXECMEM=yinasahi.config. The Fedora base config the fleet builds from already has every option the dropped lines set, and the lines had turned its built-inBLK_DEV_DMandCRYPTO_USER_API_{HASH,SKCIPHER,AEAD}into modules. The build tool'sasahiprofile (defconfig plusasahi.config) no longer getsDM_CRYPT; build a test kernel for a LUKS root with the fleet recipe.The device-tree window isn't cross-checked against
CTXR_D: aura reads it when it picks the window and reserves the rest of the armed range, and the registers don't exist on every Apple core, where reading them that early can't be recovered.From an independent review (round 3), not booted on hardware:
22420ee58a0the boot CPU never reads the entry flag. It readctxr_stay_el2with the MMU off on its first entry, before the flag was set and with the.mmuoff.data.readline never cleaned to the PoC, so stale DRAM from an earlierctxr_exec_window=boot could have sent an M1-M3 boot CPU down the direct-VHE path with no window, skipping the feature overrides. The flag is now read only on the secondary and resume entries (x22), andctxr_exec_reserve()writes it every boot (0 when no window) and cleans it before any secondary starts.13c755306c5a command-line window takes the same EL2 and hold checks as a device-tree one, and is accepted only if it lies inside the device-tree window when there is one.59a218c59f7the EL0 fetch kill now also handles aVM_PFNMAP/VM_MIXEDMAPwindow page with no struct page (a driver'sremap_pfn_range()), viafollow_pfnmap_start().c2f9d0cc1cethe CMA window-overlap check runs before the bitmap allocation, so a bitmap failure can't free an overlapping area's window pages.344bd8f2bdba confined allocation is drawn one window page at a time (the pages are vmapped individually), so a fragmented window can't fail a large pack with free pages to spare; the page array useskvmalloc_array, andset_memory_*failures on free are logged.7f54ed9a877the window is re-reserved at the start ofbootmem_init(), before the crash kernel is reserved, closing the hole a failed late reserved-memory node can leave.8eb1ce3ce98__vdso_init()frees the copies it made if a later page fails.9478b364d2ecorrects stale text (the bootloader places the image; the pool is the window's free ranges).f6b1cbf1bc1cpuidle's load check reuses the M1-to-M3 list instead of duplicating it.From Ryan's Neo run (his commits):
6315aea76ccsizes the BPF JIT limit from the pool, so unprivileged programs (half of that by default) can't use up what module text and privileged BPF need.273b9e8c846keeps a quarter of the confined pool for module text.Not in this merge: these five are in #1079 (QEMU only):
d56b6e6292freports the window and pool use in/sys/kernel/execmem_confine/(window_base,window_size,pool_size,pool_free,pool_largest_free, bytes), and calls bothset_memory_nx()andset_memory_rw()on free.25e007f8256skips thecrashkernel=reservation, with one warning, while a window is active, since kexec is refused.590be8c8957uses the device-tree window unlessctxr_exec_window=matches it exactly. A smaller command-line window inside it left the rest of the firmware's window to the page allocator, where EL0 can't execute on a confined core; under the simulator with the enforced window equal to the device tree's,/inithung on CPU2.c6a7e1f3ef7names the CTXR window in the oops when a kernel fetch outside it faults on a confined core, instead of "execute from non-executable memory".c3c11153cdathe EL0 backstop checks only a user-executable PFN-mapped page.Measured on hardware
f6b1cbf1bc1(anda1a2ac0for one boot), fleet recipe, through aura#77cda450dpadded to 128 MiB by aurora-installer#104, u-boot#3250d879acand GRUB. 6 CPUs at EL2, boot verify 1696 mappings and 0 violations, no regressions against the fleet kernel. Stress passed: 1200 module load/unload cycles,test_bpf1061/1061 on each CPU, 400 kprobes and kretprobes, function and function_graph tracers, ftrace direct, KVM 4000/4000 runs pinned to each CPU, s2idle with all of it live. Code outside the window faults on every secondary (ESR 0x8600000f) and runs on CPU0. Pool: 53 MiB in 2 ranges, 74 MiB of the window already taken.a1a2ac0ea1bmerged into the J813 stack, through aura#77 and the padded Stage 1. 10 CPUs at EL2, boot verify 576 mappings and 0 violations, pool 110 MiB with 17 MiB taken.Open: what takes 74 MiB of the window on the Neo's U-Boot and GRUB path, against 17 MiB on the M5's.
QEMU
For #1079 at
c3c11153cda(small test kernel, CTXR simulator enforcing on CPU1 and CPU2, guest system files clean before and after): good-128m, good-kvm, nowin-kvm, bad-cells, bad-small, bad-nvhe, cmd-128m, cmd-nobase, cmd-inside-dt and cmd-outside-dt (both now with the simulated window equal to the device tree's 128 MiB; the command line is ignored and all 3 CPUs come up and run module, kprobe, BPF and ftrace code), and the mismatch control all pass. W=1 is clean on the touched objects. The results below are fromf6b1cbf1bc1and earlier.QEMU 11.1.3 with
ctxr-execmem-test/sim/qemu-ctxr-sim.patch, which refuses EL2 fetches outside a window and EL0 fetches inside it on every CPU but the first, with the Neo's syndromes. It doesn't model the EL1 fault, AIC, or Apple's registers.virt,virtualization=on,gic-version=3, 3 CPUs, a small test config (16K pages, KVM, kprobes, ftrace, BPF JIT), this branch merged ontointegration/aurora9127278801d. The guest's system files were checked before and after the run. With the selftests gone, the simulator is what checks that module, kprobe and BPF code is in the window: a secondary running any of it from outside would have its fetch refused.Device-tree and command-line windows, simulated window
[0x40200000, +128 MiB)on CPUs 1 and 2, 24 cases, all pass:0x40201000; base+size wrapsmalformed aurora,ctxr-exec-window, 1 CPU, boots0x40400000(image below it)does not fit window, 1 CPU, boots[0x40000000, +4 GiB)with 3 GiB RAMnot within DRAM, 1 CPU, bootsno free space, 1 CPU, bootskvm-arm.mode=nvheat EL2the kernel is not at EL2, 1 CPU, bootsctxr_exec_enforce=0confinement disabledctxr_exec_enforce=0aurora,ctxr-exec-enforce = <0>VHE mode initialized successfully, a guest runs on CPUs 0, 1 and 2ctxr_exec_window=128M@0x40200000(no DT window)ctxr_exec_window=128M(no base)ctxr_exec_window=4M@0x40200000(no DT window)does not fit window, 1 CPU, bootsctxr_exec_window=128M@...withkvm-arm.mode=nvhethe kernel is not at EL2, 1 CPU, bootsctxr_exec_window=64M@0x40200000(inside)ctxr_exec_window=64M@0x48000000(outside)is not inside the device-tree window, 1 CPU, bootscpu1 EL2 fetch refused, CPUs 1 and 2 fail to come onlineNo panic and no EL2 fetch refused in any case but the control. The verify-guarded runs are split into batches because the build VM corrupted guest memory several times mid-run; each batch's result is from a guest that passed its file check before and after.
Entry path from
-d int: with a 128 MiB window only CPU0 goes through EL1 (1 HVC, 1 return from EL2 to EL1); with none, every CPU does (3 and 3). Same with a T8140 MIDR and without one.No window against
integration/aurora33bec4016c4, same config,-cpu max,midr=T81400x061, M40x053, M2 Pro0x035and not Apple: the same entry path (3 HVCs, 3 CPUs at EL2) and no difference at all in dmesg once timestamps and addresses are set aside. KVM with no window: initialised, guests run on all three CPUs.Poisoned flag (the blocker's regression test): a copy of the Image with
ctxr_stay_el2set to 1 in the file (Image+0xcd8808,.mmuoff.data.read), which is what stale DRAM would hold at the boot CPU's first entry, booted with no window:0x0350x061a1a2ac0ea1b(before the fix), controlEach row ran on a guest that passed its file check before and after.
W=1 on
head.o,smp.o,cpu_errata.o,vdso.o,efi.o,machine_kexec.o,arch/arm64/mm/{init,fault,ctxr_execmem}.o,mm/{execmem,execmem_confine,memblock,cma}.oandcpuidle-apple.o, fleet config: 0 warnings, 0 errors. (It caught one in a review build, an uninitialisedallocrangein the CMA change, now fixed.)The EL0 fetch backstop, with
59a218c59f7: a real window page mapped to EL0 byremap_pfn_range()(aVM_PFNMAP, the case the folio walk alone missed) is now killed withSIGSEGVon the confined CPUs 1 and 2, and the kernel logsEL0 fetch ... refused by the CTXR window; before this it stalled forever. The zero page, now outside the window, decodes andSIGILLs on every CPU.Test tools (
ctxr-execmem-test, outside the tree) no longer use the removed debugfs file:el0_window_checkand the harnessinit.cread the window from the boot log, andctxr_test.kooffers/dev/ctxr_test_window, a read-only executable PFN mapping of its own text, forel0_window_check's newexec-windowstep andfault_forge_check'swindow-pagecase (the old zero-page case no longer reaches the window). Under the simulator with a 128 MiB window, against this head:exec-windowis fetched on CPU0 and killed on CPUs 1 and 2, the forged fault on the window page kills, and the zero-page, no-exec, not-present and ordinary cases behave as before. For the Neo boot, loadctxr_test.kobuilt against that kernel before runningel0_window_check; without it the tool still runs its zero-page, anon and vDSO steps.Reservations in the window, on the fleet kernel below at EL2 with a 128 MiB window (
review_check.py, adapted to EL2 and<size>@<base>): an initrd straddling the window's end, reserved-memory, no-map and dynamic nodes plus/memreserve/inside it, everything inside a 256 MiB one, a reusable CMA area byregand bysizein the window (overlaps the executable window, not activated), and a CMA area whose init fails: 6 of 6 pass, each with boot verify 1180 mappings / 0 violations and 0 user pages in the window. (Without debugfs the harness can no longer list the pool's ranges; boot verify's count of window pages that reached the allocator stands in.)Earlier rounds' QEMU results (EFI runtime, EL0 kill, kexec refusal, reservations in the window, KASAN and kmemleak) are in this description's history; their code hasn't changed.
Build
Fleet recipe (Fedora Asahi
config+asahi.config+fedora.configaskernel-aurora.specmerges them, LLVM,LOCALVERSIONempty) atf6b1cbf1bc1, with debug info and BTF off for host disk: guest output/var/lib/aurora-build/outputs/kernel/linux-ctxr985-base-39df485e1a2b,Imagesha256b9a4c0ddbe80ee56d804efd62910e05990744f1520ffd4858ca956b7b249258a(28,215,808 bytes), 1991 modules, 119 DTBs, release7.3.0. The resolved config hasBLK_DEV_DM=y,DM_CRYPT=m,DM_INTEGRITY=m,CRYPTO_ESSIV=mandCRYPTO_USER_API_{HASH,SKCIPHER,AEAD}=yfrom the Fedora base. Guest system files verified clean before and after, and the build tool's own pass over it rebuilt nothing.integration/aurora's newCONFIG_EDAC_APPLE_SOC=mdoesn't resolve on the Fedora base (# CONFIG_EDAC is not set), sovalidate-config.pyrejects it and this kernel doesn't have that driver; that's independent of this PR.Host tests (CI) pass at this head.
integration/aurorahas since moved to68db611bf7e(audio, ANE and Bluetooth only); the branch still merges cleanly, and the fleet build above is the branch on33bec4016c4.KVM and Apple registers at EL2 on M4 and later
CONFIG_KVM=yin the Fedora base;asahi.configdoesn't touch it). apple-aic hands KVM its vGIC info and the maintenance and guest-timer FIQs for every AIC version; M4-generation device trees useapple,t8122-aic3,apple,t8140-aic3orapple,t8142-aic3. Nothing Apple-specific turns KVM off.apple_cpus(M1/M2). Nothing confirms the bit on M4 or later, so guests there get no PMU.s3_5_c15_c5_0):cpuidle-apple.cis the only user, now M1 to M3 only.s3_5_c15_c6_2: nothing in this branch touches it or sets a trap that would affect EL2's own access.Review notes
With no window, every piece is inert: the gates are
ctxr_stay_el2,ctxr_active,ctxr_hold_cpus,confine_readyandexecmem_confine_active(), and the explicitEXECMEM_MODULE_DATArange matches whatexecmem_init_missing()fills in.select PTDUMPandGENERIC_ALLOCATORare already set in the fleet config. Two changes apply without a window, on purpose:6ebbafe2175(only with the kernel at EL2 on T8140) andddb50b42408(same states on every in-tree device tree).History a reviewer may want squashed:
73b99c17e60andfe8741d4f6dare replaced by46c5ae7cd01, which goes back to the split72880a44365had;207bc4d748eand0412b237dc1belong together; so do91c85e7bf4fand52252329df0;5863e2492e8and0941e6f6831cancel out apart from the CTXR line.Against
integration/aurorathe branch is 27 files, +1440/−38, down from +1692/−38 at51301b05422.131e02f6757anda1a2ac0ea1bmergeintegration/aurorato resolve conflicts incpuidle-apple.c(T6050) andasahi.config(EDAC_APPLE_SOC).Note, not fixed here:
integration/aurora's newCONFIG_EDAC_APPLE_SOC=mdoesn't resolve on the Fedora base config (# CONFIG_EDAC is not set), sokernel-aurora.spec'svalidate-config.pyrejects the fleet config onintegration/auroraitself.Not in this PR
GetTimeand efivarfs mounts read-only (GetVariableworks); this tree'sasahi.configlacksBT_MTK7932_PCIE; unloadingaegis128oopses on arm64 because its exit path writes__ro_after_initdata.🤖 Generated with Claude Code