Skip to content

arm64: run Linux at EL2 with its code confined to a CTXR window - #985

Merged
RyanTheTide merged 50 commits into
integration/aurorafrom
feat/ctxr-execmem
Nov 7, 2026
Merged

RyanTheTide merged 50 commits into
integration/aurorafrom
feat/ctxr-execmem

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Linux at EL2 with no monitor on M4-generation and later Macs. All of its executable code stays in one physical window enrolled in CTXR. On a confined secondary, EL2 can only fetch from inside the window, EL0 can't execute there, and EL1 executing there faults to EL2. CPU0 isn't confined.

Merged at 273b9e8c846. f6b1cbf1bc1 has booted on the Neo with a real 128 MiB window (below); Ryan's two commits after it are in this merge, and five more are in #1079. With no window (no /chosen/aurora,ctxr-exec-window, no ctxr_exec_window=), the kernel behaves as integration/aurora does, entry path included.

What's in it

The window and the pool (from before):

  • mm/execmem (CONFIG_EXECMEM_CONFINE) backs module, BPF, kprobe and ftrace text with pages from a gen_pool over the window. There's no buddy fallback.
  • arm64 (CONFIG_ARM64_CTXR_EXECMEM) learns the window from /chosen/aurora,ctxr-exec-window or ctxr_exec_window=, checks that the image fits, and reserves it. The reservation is the last step of arm64_memblock_init(), after the initrd and early_init_fdt_scan_reserved_mem(), and the pool holds only memory that was free just before it. A verifier sweeps the page tables and panics on an escape when the window is enforcing.
  • The default size is 128 MiB. The image and every module's code take about 66 MiB, and window RAM the pool doesn't use is lost to the system.

Brought over from the stack that booted, patch-ids unchanged:

  • 72880a44365 and 73b99c17e60 stay at EL2 on entry.
  • 5863e2492e8 added dm-crypt and what an installed system's services load to the asahi profile. 0941e6f6831 takes that back out (below).
  • 207bc4d748e maps a copy of the vDSO from ordinary memory.
  • 4b630f74a37 keeps window pages out of the page allocator.

New in the first round, none of it booted on hardware:

  • fe8741d4f6d limited the direct VHE entry to part numbers from 0x50. 46c5ae7cd01 (below) replaces that test.
  • 0412b237dc1 copies the vDSO only when a window is active.
  • d9f915b119d turns EFI runtime services off with one warning when a window is active and any EFI_RUNTIME_SERVICES_CODE region is outside it or misaligned. The verifier sweeps efi_mm too.
  • a64c54e49a6 kills a task, with an uncatchable SIGSEGV and a ratelimited log line, when an EL0 instruction permission fault lands on a present, executable mapping of a window page.
  • 0ab88ef01d7 makes machine_kexec_prepare() fail with EOPNOTSUPP while a window is active.
  • 2e6e70017ce docs.

From the first review, not booted on hardware:

  • d9de654671e early_param() parsing. ctxr_exec_enforce= applies to a device-tree window, options don't match inside other words, the last use wins, and a bad command-line value or window is ignored with a warning.
  • d637de6296f keeps reusable CMA and failed reserved-memory nodes out of the window.
  • b3a9578538f KASAN and kmemleak for confined allocations.
  • 524f9114cf1 the binding for /chosen/aurora,ctxr-exec-window (and aurora,ctxr-exec-enforce, since dropped).
  • c4b5fab28dd docs and the Kconfig help.

From Ryan's second round (1c4ce70047b, 20519af6fc3, a57207316f8, e82a8514a6f): an active window counts as confining whether or not it is enforcing, the zero page moves out of the window, the direct entry checked ID_AA64MMFR1_EL1.VH (since dropped), and the TLB is flushed before a confined chunk goes back to the pool.

This round, not booted on hardware:

  • 46c5ae7cd01 keeps the boot CPU on the usual entry path. The direct VHE entry was keyed on the part number, so every M4-generation core took it on every entry, window or not, and CPU0 took it before the feature overrides were parsed. CPU0 isn't confined, so it now enters as it always has (EL1, then finalise_el2). Once it has reserved a window at EL2 it sets ctxr_stay_el2 (in .mmuoff.data.read, cleaned to the PoC), and only later entries that read it set take the direct path. With no window, every core takes the stock path, and kvm-arm.mode= applies to CPU0 again.
  • 91c85e7bf4f and 52252329df0 keep the secondaries offline when the device tree's window can't be used, instead of panicking (enforcing) or releasing them anyway (ctxr_exec_enforce=0). Unusable means a malformed property (not four cells, size 0, not page aligned, wraps), the kernel not at EL2, an image that doesn't fit, a window outside DRAM, a failed reservation, or no free page left for the pool. The kernel logs why and boots on CPU0; smp_prepare_cpus() doesn't prepare or mark present any secondary, so no release address is written. The empty-pool check runs at reservation, because on arm64 the pool is built at core_initcall, after smp_init(). Enforcement now only decides whether a verifier finding is fatal. The one panic left is a failed allocation while building the pool, when confined CPUs are already running.
  • 6ebbafe2175 applies the Apple vGIC (broken SEIS) workaround to T8140. Without it KVM at EL2 on the Neo uses one list register (GICv3 sysreg trapping enabled ([C]) in the 2026-11-05 EL2 boot log) and SMP guests livelock, as on T6040 before fcb377a2544. ICH_VTR_EL2 reads 0x90580007 on both. It only applies with the kernel at EL2.
  • ddb50b42408 offers cpuidle's CPU PD state (which writes CYC_OVRD) only on M1 to M3 compatibles, instead of denying a list. Every device tree in the tree gets the same states as before. a3043575d91 updates the apple-dcp cpuidle-s2idle host test, which replays the probe with mocks and failed CI at 52252329df0; it now covers T8103, T8140, T6041 and T8142 (142 assertions pass locally).
  • 131e02f6757 merges integration/aurora (b452c284930): it added T6050 to the cpuidle deny list, which conflicted with ddb50b42408; the allowlist already covers it.

Simplified with the owner's approval, not booted on hardware:

  • 51301b05422 (coordinator) corrects the doc: the boot CPU's CTXR_D does describe the window (aura read [0x1000667c000, 0x10017950000) on CPU0 on 11-05, the range a secondary enforces); only enforcement is off there. The reason given in 91c85e7bf4f's message for not cross-checking is wrong in the same way.
  • 65da3475d81 drops the ID_AA64MMFR1_EL1.VH test from the direct entry: only a boot CPU already running VHE at EL2 sets the flag.
  • c89843af86d drops aurora,ctxr-exec-enforce and windows without a base. A device-tree base of 0 is malformed, and ctxr_exec_window= takes <size>@<base>, both required, so the default size and the anchor-on-the-image code go. Binding, doc and kernel-parameters.txt follow.
  • d1eba7c3ee2 drops the uncalled execmem_confine_window() and execmem_confine_enforced().
  • 9657b6e15db drops ctxr_exec_selftest and /sys/kernel/debug/execmem_confine. The boot-time verifier and the EL0 fetch kill stay. ctxr_exec_enforce= now does one thing: whether a boot-time verifier finding panics (default) or is only logged.
  • 0941e6f6831 leaves only CONFIG_ARM64_CTXR_EXECMEM=y in asahi.config. The Fedora base config the fleet builds from already has every option the dropped lines set, and the lines had turned its built-in BLK_DEV_DM and CRYPTO_USER_API_{HASH,SKCIPHER,AEAD} into modules. The build tool's asahi profile (defconfig plus asahi.config) no longer gets DM_CRYPT; build a test kernel for a LUKS root with the fleet recipe.

The device-tree window isn't cross-checked against CTXR_D: aura reads it when it picks the window and reserves the rest of the armed range, and the registers don't exist on every Apple core, where reading them that early can't be recovered.

From an independent review (round 3), not booted on hardware:

  • 22420ee58a0 the boot CPU never reads the entry flag. It read ctxr_stay_el2 with the MMU off on its first entry, before the flag was set and with the .mmuoff.data.read line never cleaned to the PoC, so stale DRAM from an earlier ctxr_exec_window= boot could have sent an M1-M3 boot CPU down the direct-VHE path with no window, skipping the feature overrides. The flag is now read only on the secondary and resume entries (x22), and ctxr_exec_reserve() writes it every boot (0 when no window) and cleans it before any secondary starts.
  • 13c755306c5 a command-line window takes the same EL2 and hold checks as a device-tree one, and is accepted only if it lies inside the device-tree window when there is one.
  • 59a218c59f7 the EL0 fetch kill now also handles a VM_PFNMAP/VM_MIXEDMAP window page with no struct page (a driver's remap_pfn_range()), via follow_pfnmap_start().
  • c2f9d0cc1ce the CMA window-overlap check runs before the bitmap allocation, so a bitmap failure can't free an overlapping area's window pages.
  • 344bd8f2bdb a confined allocation is drawn one window page at a time (the pages are vmapped individually), so a fragmented window can't fail a large pack with free pages to spare; the page array uses kvmalloc_array, and set_memory_* failures on free are logged.
  • 7f54ed9a877 the window is re-reserved at the start of bootmem_init(), before the crash kernel is reserved, closing the hole a failed late reserved-memory node can leave.
  • 8eb1ce3ce98 __vdso_init() frees the copies it made if a later page fails.
  • 9478b364d2e corrects stale text (the bootloader places the image; the pool is the window's free ranges).
  • f6b1cbf1bc1 cpuidle's load check reuses the M1-to-M3 list instead of duplicating it.

From Ryan's Neo run (his commits):

  • 6315aea76cc sizes the BPF JIT limit from the pool, so unprivileged programs (half of that by default) can't use up what module text and privileged BPF need.
  • 273b9e8c846 keeps a quarter of the confined pool for module text.

Not in this merge: these five are in #1079 (QEMU only):

  • d56b6e6292f reports the window and pool use in /sys/kernel/execmem_confine/ (window_base, window_size, pool_size, pool_free, pool_largest_free, bytes), and calls both set_memory_nx() and set_memory_rw() on free.
  • 25e007f8256 skips the crashkernel= reservation, with one warning, while a window is active, since kexec is refused.
  • 590be8c8957 uses the device-tree window unless ctxr_exec_window= matches it exactly. A smaller command-line window inside it left the rest of the firmware's window to the page allocator, where EL0 can't execute on a confined core; under the simulator with the enforced window equal to the device tree's, /init hung on CPU2.
  • c6a7e1f3ef7 names the CTXR window in the oops when a kernel fetch outside it faults on a confined core, instead of "execute from non-executable memory".
  • c3c11153cda the EL0 backstop checks only a user-executable PFN-mapped page.

Measured on hardware

  • Neo (J700, T8140), Ryan, 2026-11-07 (comment): f6b1cbf1bc1 (and a1a2ac0 for one boot), fleet recipe, through aura#77 cda450d padded to 128 MiB by aurora-installer#104, u-boot#32 50d879ac and GRUB. 6 CPUs at EL2, boot verify 1696 mappings and 0 violations, no regressions against the fleet kernel. Stress passed: 1200 module load/unload cycles, test_bpf 1061/1061 on each CPU, 400 kprobes and kretprobes, function and function_graph tracers, ftrace direct, KVM 4000/4000 runs pinned to each CPU, s2idle with all of it live. Code outside the window faults on every secondary (ESR 0x8600000f) and runs on CPU0. Pool: 53 MiB in 2 ranges, 74 MiB of the window already taken.
  • M5 (J813, T8142), sithglan, 2026-11-06: a1a2ac0ea1b merged into the J813 stack, through aura#77 and the padded Stage 1. 10 CPUs at EL2, boot verify 576 mappings and 0 violations, pool 110 MiB with 17 MiB taken.

Open: what takes 74 MiB of the window on the Neo's U-Boot and GRUB path, against 17 MiB on the M5's.

QEMU

For #1079 at c3c11153cda (small test kernel, CTXR simulator enforcing on CPU1 and CPU2, guest system files clean before and after): good-128m, good-kvm, nowin-kvm, bad-cells, bad-small, bad-nvhe, cmd-128m, cmd-nobase, cmd-inside-dt and cmd-outside-dt (both now with the simulated window equal to the device tree's 128 MiB; the command line is ignored and all 3 CPUs come up and run module, kprobe, BPF and ftrace code), and the mismatch control all pass. W=1 is clean on the touched objects. The results below are from f6b1cbf1bc1 and earlier.

QEMU 11.1.3 with ctxr-execmem-test/sim/qemu-ctxr-sim.patch, which refuses EL2 fetches outside a window and EL0 fetches inside it on every CPU but the first, with the Neo's syndromes. It doesn't model the EL1 fault, AIC, or Apple's registers. virt,virtualization=on,gic-version=3, 3 CPUs, a small test config (16K pages, KVM, kprobes, ftrace, BPF JIT), this branch merged onto integration/aurora 9127278801d. The guest's system files were checked before and after the run. With the selftests gone, the simulator is what checks that module, kprobe and BPF code is in the window: a secondary running any of it from outside would have its fetch refused.

Device-tree and command-line windows, simulated window [0x40200000, +128 MiB) on CPUs 1 and 2, 24 cases, all pass:

Case Result
3 cells; base 0; size 0; base 0x40201000; base+size wraps malformed aurora,ctxr-exec-window, 1 CPU, boots
4 MiB; base 0x40400000 (image below it) does not fit window, 1 CPU, boots
[0x40000000, +4 GiB) with 3 GiB RAM not within DRAM, 1 CPU, boots
exactly the image no free space, 1 CPU, boots
QEMU without EL2; kvm-arm.mode=nvhe at EL2 the kernel is not at EL2, 1 CPU, boots
image doesn't fit, ctxr_exec_enforce=0 1 CPU, boots, no confinement disabled
128 MiB 3 CPUs at EL2, pool 114 MiB, boot verify 477 mappings / 0 violations, module, kprobe, ftrace, BPF and uprobe on CPU1
128 MiB, ctxr_exec_enforce=0 same, reporting
128 MiB with the old aurora,ctxr-exec-enforce = <0> ignored, enforcing
128 MiB, KVM VHE mode initialized successfully, a guest runs on CPUs 0, 1 and 2
ctxr_exec_window=128M@0x40200000 (no DT window) same as the device tree's 128 MiB
ctxr_exec_window=128M (no base) ignored with a warning, no window, 3 CPUs
ctxr_exec_window=4M@0x40200000 (no DT window) does not fit window, 1 CPU, boots
ctxr_exec_window=128M@... with kvm-arm.mode=nvhe the kernel is not at EL2, 1 CPU, boots
DT 128 MiB + ctxr_exec_window=64M@0x40200000 (inside) 64 MiB window, 3 CPUs, verify 0 violations
DT 128 MiB + ctxr_exec_window=64M@0x48000000 (outside) is not inside the device-tree window, 1 CPU, boots
control: window accepted, simulated window elsewhere cpu1 EL2 fetch refused, CPUs 1 and 2 fail to come online

No panic and no EL2 fetch refused in any case but the control. The verify-guarded runs are split into batches because the build VM corrupted guest memory several times mid-run; each batch's result is from a guest that passed its file check before and after.

Entry path from -d int: with a 128 MiB window only CPU0 goes through EL1 (1 HVC, 1 return from EL2 to EL1); with none, every CPU does (3 and 3). Same with a T8140 MIDR and without one.

No window against integration/aurora 33bec4016c4, same config, -cpu max,midr= T8140 0x061, M4 0x053, M2 Pro 0x035 and not Apple: the same entry path (3 HVCs, 3 CPUs at EL2) and no difference at all in dmesg once timestamps and addresses are set aside. KVM with no window: initialised, guests run on all three CPUs.

Poisoned flag (the blocker's regression test): a copy of the Image with ctxr_stay_el2 set to 1 in the file (Image+0xcd8808, .mmuoff.data.read), which is what stale DRAM would hold at the boot CPU's first entry, booted with no window:

Head M2 Pro 0x035 T8140 0x061 not Apple
this head 3 HVCs, 3 EL1 hops: stock on every CPU same same
a1a2ac0ea1b (before the fix), control 0 HVCs, 0 EL1 hops: every CPU, the boot CPU included, takes the direct VHE path with no window same same

Each row ran on a guest that passed its file check before and after.

W=1 on head.o, smp.o, cpu_errata.o, vdso.o, efi.o, machine_kexec.o, arch/arm64/mm/{init,fault,ctxr_execmem}.o, mm/{execmem,execmem_confine,memblock,cma}.o and cpuidle-apple.o, fleet config: 0 warnings, 0 errors. (It caught one in a review build, an uninitialised allocrange in the CMA change, now fixed.)

The EL0 fetch backstop, with 59a218c59f7: a real window page mapped to EL0 by remap_pfn_range() (a VM_PFNMAP, the case the folio walk alone missed) is now killed with SIGSEGV on the confined CPUs 1 and 2, and the kernel logs EL0 fetch ... refused by the CTXR window; before this it stalled forever. The zero page, now outside the window, decodes and SIGILLs on every CPU.

Test tools (ctxr-execmem-test, outside the tree) no longer use the removed debugfs file: el0_window_check and the harness init.c read the window from the boot log, and ctxr_test.ko offers /dev/ctxr_test_window, a read-only executable PFN mapping of its own text, for el0_window_check's new exec-window step and fault_forge_check's window-page case (the old zero-page case no longer reaches the window). Under the simulator with a 128 MiB window, against this head: exec-window is fetched on CPU0 and killed on CPUs 1 and 2, the forged fault on the window page kills, and the zero-page, no-exec, not-present and ordinary cases behave as before. For the Neo boot, load ctxr_test.ko built against that kernel before running el0_window_check; without it the tool still runs its zero-page, anon and vDSO steps.

Reservations in the window, on the fleet kernel below at EL2 with a 128 MiB window (review_check.py, adapted to EL2 and <size>@<base>): an initrd straddling the window's end, reserved-memory, no-map and dynamic nodes plus /memreserve/ inside it, everything inside a 256 MiB one, a reusable CMA area by reg and by size in the window (overlaps the executable window, not activated), and a CMA area whose init fails: 6 of 6 pass, each with boot verify 1180 mappings / 0 violations and 0 user pages in the window. (Without debugfs the harness can no longer list the pool's ranges; boot verify's count of window pages that reached the allocator stands in.)

Earlier rounds' QEMU results (EFI runtime, EL0 kill, kexec refusal, reservations in the window, KASAN and kmemleak) are in this description's history; their code hasn't changed.

Build

Fleet recipe (Fedora Asahi config + asahi.config + fedora.config as kernel-aurora.spec merges them, LLVM, LOCALVERSION empty) at f6b1cbf1bc1, with debug info and BTF off for host disk: guest output /var/lib/aurora-build/outputs/kernel/linux-ctxr985-base-39df485e1a2b, Image sha256 b9a4c0ddbe80ee56d804efd62910e05990744f1520ffd4858ca956b7b249258a (28,215,808 bytes), 1991 modules, 119 DTBs, release 7.3.0. The resolved config has BLK_DEV_DM=y, DM_CRYPT=m, DM_INTEGRITY=m, CRYPTO_ESSIV=m and CRYPTO_USER_API_{HASH,SKCIPHER,AEAD}=y from the Fedora base. Guest system files verified clean before and after, and the build tool's own pass over it rebuilt nothing. integration/aurora's new CONFIG_EDAC_APPLE_SOC=m doesn't resolve on the Fedora base (# CONFIG_EDAC is not set), so validate-config.py rejects it and this kernel doesn't have that driver; that's independent of this PR.

Host tests (CI) pass at this head. integration/aurora has since moved to 68db611bf7e (audio, ANE and Bluetooth only); the branch still merges cleanly, and the fleet build above is the branch on 33bec4016c4.

KVM and Apple registers at EL2 on M4 and later

  • KVM is in the fleet config (CONFIG_KVM=y in the Fedora base; asahi.config doesn't touch it). apple-aic hands KVM its vGIC info and the maintenance and guest-timer FIQs for every AIC version; M4-generation device trees use apple,t8122-aic3, apple,t8140-aic3 or apple,t8142-aic3. Nothing Apple-specific turns KVM off.
  • T8140: broken-SEIS workaround added (above). T6040 had it. T8132, T6041, T8142 and later aren't added: their ICH_VTR_EL2 hasn't been read, and T8132 P and T6041 have no MIDR defines.
  • The IMPDEF PMUv3 trap (HACR_EL2 bit 56) stays on apple_cpus (M1/M2). Nothing confirms the bit on M4 or later, so guests there get no PMU.
  • TSO, APFLG, AFP, PRSV and AMX are detected from AIDR_EL1 and a write test of ACTLR_EL1, not by part number, and ACTLR virtualization covers parts from M2 on. The Neo at EL2 already detected TSO, APFLG and ACTLR virtualization.
  • CYC_OVRD (s3_5_c15_c5_0): cpuidle-apple.c is the only user, now M1 to M3 only.
  • s3_5_c15_c6_2: nothing in this branch touches it or sets a trap that would affect EL2's own access.

Review notes

With no window, every piece is inert: the gates are ctxr_stay_el2, ctxr_active, ctxr_hold_cpus, confine_ready and execmem_confine_active(), and the explicit EXECMEM_MODULE_DATA range matches what execmem_init_missing() fills in. select PTDUMP and GENERIC_ALLOCATOR are already set in the fleet config. Two changes apply without a window, on purpose: 6ebbafe2175 (only with the kernel at EL2 on T8140) and ddb50b42408 (same states on every in-tree device tree).

History a reviewer may want squashed: 73b99c17e60 and fe8741d4f6d are replaced by 46c5ae7cd01, which goes back to the split 72880a44365 had; 207bc4d748e and 0412b237dc1 belong together; so do 91c85e7bf4f and 52252329df0; 5863e2492e8 and 0941e6f6831 cancel out apart from the CTXR line.

Against integration/aurora the branch is 27 files, +1440/−38, down from +1692/−38 at 51301b05422.

131e02f6757 and a1a2ac0ea1b merge integration/aurora to resolve conflicts in cpuidle-apple.c (T6050) and asahi.config (EDAC_APPLE_SOC).

Note, not fixed here: integration/aurora's new CONFIG_EDAC_APPLE_SOC=m doesn't resolve on the Fedora base config (# CONFIG_EDAC is not set), so kernel-aurora.spec's validate-config.py rejects the fleet config on integration/aurora itself.

Not in this PR

  • Protected KVM under a window, beyond keeping the secondaries offline.
  • CPU off and on at EL2. On plain spin-table, hotplug isn't offered.
  • Hibernate's code is untouched and untested under a window.
  • U-Boot keeping its own runtime code inside the window (arm: apple: keep Linux inside the CTXR executable window u-boot#32).
  • Seen on Ryan's run, not from this PR: U-Boot's runtime has no GetTime and efivarfs mounts read-only (GetVariable works); this tree's asahi.config lacks BT_MTK7932_PCIE; unloading aegis128 oopses on arm64 because its exit path writes __ro_after_init data.

🤖 Generated with Claude Code

Add CONFIG_EXECMEM_CONFINE: an optional backing mode for execmem ranges
flagged EXECMEM_CONFINE. Such ranges keep their virtual placement (so a
module stays within branch range of the kernel) but draw their physical
backing pages from a single fixed window registered by the architecture
via execmem_confine_init(), rather than from arbitrary buddy pages.

This lets a platform keep every run-time executable physical page
(modules, BPF, kprobes, ftrace) inside a hardware-enforceable code region
such as an Apple-silicon CTXR window, which is a prerequisite for running
the kernel at that privilege level.

A confined allocation reserves its virtual area with __get_vm_area_node()
inside the range, pulls a physically-contiguous chunk from a gen_pool over
the window, and maps it with vmap_pages_range(); the backing pages are
published on the vm_struct so set_memory_rox() can mirror permission
changes onto their linear alias. If the pool is exhausted the allocation
fails, with no buddy fallback, mirroring a hardware execute-range fault.

A verifier sweeps for any executable mapping that resolves outside the
window (the page-table sweep itself is provided per-architecture); when
the window is enforcing a violation is fatal. "ctxr_exec_selftest" adds
a boot-time positive/negative self-check.

Signed-off-by: Diljit Singh <hey@djdev.me>
On T8140/M4-class Apple-silicon machines the boot firmware arms a per-CPU
code text execute range (CTXR) on the secondary cores: at the privileged
exception level an instruction may only be fetched from inside a fixed
physical window. To run the kernel there rather than dropping it to EL1,
every executable physical page must live inside one window.

Add CONFIG_ARM64_CTXR_EXECMEM, the arm64 end of the EXECMEM_CONFINE
contract:

  - ctxr_exec_reserve(), from arm64_memblock_init(), learns the window
    from /chosen/aurora,ctxr-exec-window (emitted by the EL2 monitor) or
    the ctxr_exec_window= command line (for VM bring-up), checks that the
    kernel image fits, and reserves it from memblock.
  - execmem_arch_setup() carves the non-image remainder into a gen_pool,
    registers it with the confinement core, and flags the executable
    ranges (module text, BPF, kprobes and, by inheritance, ftrace) with
    EXECMEM_CONFINE. Module data is left on ordinary vmalloc backing.
  - execmem_confine_arch_verify() provides the page-table sweep used by
    the verifier: every privileged-executable leaf outside the window is
    reported, standing in for the hardware fault when testing in a VM.

The option is inert unless a window is supplied at boot, so it is enabled
in asahi.config and is a no-op on machines that do not need it.

Signed-off-by: Diljit Singh <hey@djdev.me>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Merge train (MTL, 06/11): holding this one for now, not merged.

On the current fleet nothing changes (no window is supplied, so ctxr_exec_execmem_setup() returns false and every range takes the old path). The problem is what happens the first time a window is supplied:

  1. Memory inside the window is handed out twice (arch/arm64/mm/ctxr_execmem.c around lines 136 and 188). The pool is the window minus the kernel image only. The initrd is reserved before ctxr_exec_reserve() runs, and with the default base (image rounded down) and a 1G size, an initrd or FDT placed right after the kernel falls inside the window. free_initrd_mem() later gives those pages to the buddy allocator while the gen_pool can still hand them out as module text. Reserved-memory, CMA and no-map carveouts from early_init_fdt_scan_reserved_mem() can land in the pool the same way and end up mapped executable. The window needs the reserved regions carved out (or overlap rejected), and no-map holes handled.
  2. Cmdline parsing (ctxr_execmem.c lines 57-75 and 109-111). ctxr_exec_enforce= is only read when the window also comes from the cmdline, so with a DT window the documented cmdline override does nothing. strstr() on boot_command_line also matches substrings and takes the first occurrence, and an enforcing window with a bad parse panics. early_param() would fix both.

Smaller things: execmem_confine.c around line 128 skips kasan_unpoison_vmalloc() and kmemleak registration (false positives on debug builds), kexec relocation pages and the hibernate trans_pgd page run outside init_mm and will fault under real CTXR, the Kconfig help says "If unsure, say N" while asahi.config turns it on, and /chosen/aurora,ctxr-exec-* has no binding doc.

Since the asahi.config change puts this in every distro kernel, I'd like 1 and 2 fixed before it lands, so that whatever aura starts publishing a window doesn't hit them. Happy to re-run the train as soon as it's updated.

…ndow

Build the executable pool only from memory that was free before the window
was reserved. Snapshot for_each_free_mem_range() over the window just before
memblock_reserve(), so anything already reserved at that point -- the image,
the device tree blob, an initrd, /reserved-memory and memreserve entries, EFI
runtime regions and the UEFI memory map -- and any hole or no-map range stays
out of the pool. Previously the pool was the window minus the image only, so a
lowest-first executable allocation could overwrite a live DTB, an initrd or EFI
runtime memory that a loader had placed inside the window.

When no base is given and anchoring the window on the image would run past the
end of DRAM (a loader placed the image within the window size of the top),
slide the window down to end at DRAM's end, still covering the image, instead
of panicking.

List the pool's ranges in /sys/kernel/debug/execmem_confine.

Signed-off-by: Diljit Singh <hey@djdev.me>
The kernel image and the code of every module take about 66 MiB, and RAM
in the window that the pool doesn't use is lost to the rest of the
system. 128 MiB holds them with room to spare.

Signed-off-by: Diljit Singh <hey@djdev.me>
Picks up the bindgen fix (#992), without which the asahi profile does not
build.

Signed-off-by: Diljit Singh <hey@djdev.me>
Kernel text inside an armed CTXR window can't execute at EL1 on a core the
window is enforced on, so the usual EL1 detour on entry faults. Once the boot
CPU has found an armed window, cores entering after it switch to VHE in
init_kernel_el and never run at EL1.

Signed-off-by: Diljit Singh <hey@djdev.me>
Kernel text inside an armed CTXR window can't execute at EL1 on a core the
window is enforced on, and the window isn't known when the boot CPU first
enters. Apple cores are VHE only, so with CONFIG_ARM64_CTXR_EXECMEM they now
switch to VHE in init_kernel_el on every entry, boot CPU included, and never
run at EL1. This replaces the flag the boot CPU set for later cores.

Signed-off-by: Diljit Singh <hey@djdev.me>
The profile had no dm-crypt, so a kernel built from it can't unlock an
encrypted root. Add it with the crypto interfaces LUKS uses, and the modules
an Aurora install's services load.

Signed-off-by: Diljit Singh <hey@djdev.me>
The vDSO's code is part of the kernel image, which sits in the CTXR window,
and EL0 can't execute there on a core the window is enforced on. A process
calling into the vDSO on such a core takes a permission fault that the kernel
finds nothing to fix, and retries it forever. Map a copy from ordinary memory
into userspace instead.

Signed-off-by: Diljit Singh <hey@djdev.me>
A core that enforces the CTXR window doesn't let EL0 execute from it
either. The fetch takes a permission fault that the page tables don't
explain, so the kernel retries it forever.

The window is reserved, but free_reserved_area() can still release parts
of it: __init memory, which is part of the image, and an initrd placed in
the window. Those pages reach the page allocator and can become user
pages. Keep them reserved, and have the verifier count any window page
that isn't reserved as a violation.

Signed-off-by: Diljit Singh <hey@djdev.me>
With CONFIG_ARM64_CTXR_EXECMEM every Apple core switched to VHE in
init_kernel_el, whether or not it has CTXR and whether or not a window
was supplied, so a kernel built with the option took a different entry
path on M1 to M3. Only M4-generation and later cores have CTXR. Their
part numbers start at 0x50, and every earlier core's is below that, so
limit the direct switch to them. Earlier cores drop to EL1 and come back
through finalise_el2 as before.

Signed-off-by: Diljit Singh <hey@djdev.me>
The copy was made whenever CONFIG_ARM64_CTXR_EXECMEM was on. With no
window there is nothing to keep the vDSO out of, so map the image's pages
as before.

Signed-off-by: Diljit Singh <hey@djdev.me>
…t run

EFI runtime code runs at EL2 on whichever core efi_rts_wq is on. On a
core that enforces the window, code outside it faults, and the fault
handler dies with "execution of user memory" before the EFI fixup runs.
So when a window is enforcing and any EFI_RUNTIME_SERVICES_CODE region is
outside it, warn once and turn runtime services off for that boot.
Misaligned code counts too: on a 16K kernel it makes the misaligned
regions after it executable, wherever they are, and the verifier would
then panic on them.

The window is read from this file rather than through
execmem_confine_*(), since execmem_init() is a later core_initcall than
arm64_efi_rt_init().

Have the verifier sweep efi_mm as well, so its report covers the runtime
mappings that stay on.

Signed-off-by: Diljit Singh <hey@djdev.me>
On a core enforcing the window, an EL0 fetch from a window page takes a
level 3 permission fault that the page tables don't explain.
handle_mm_fault() finds nothing to fix, so the task retries forever: it
stays runnable, with no panic and no hung-task report. The zero page
stays in the window, so a program jumping into fresh executable memory
before writing it hits this.

When a window is enforcing and such a fault lands on a present,
executable mapping of a window page, log the task and kill it. Every
other fault takes the path it took before.

Signed-off-by: Diljit Singh <hey@djdev.me>
…window

The relocation code runs from an ordinary page, outside the window, and a
crash dump can start on a core enforcing it, where it would hang during a
panic. Fail the load cleanly instead, with one message, so nothing finds
out only when it needs the second kernel.

Signed-off-by: Diljit Singh <hey@djdev.me>
A kernel in a window runs at EL2 with no monitor below it, so the
bootloader is what supplies the window. Say so, and document the entry
path.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr DjDeveloperr changed the title arm64: confine kernel executable code to a CTXR physical window arm64: run Linux at EL2 with its code confined to a CTXR window Nov 6, 2026
ctxr_exec_enforce= was only read along with ctxr_exec_window=, so it did
nothing to a window from the device tree. strstr() matched inside other
words and took the first use of an option. A value that didn't parse gave
a default window, and a window too small for the image panicked a machine
that would otherwise boot.

Use early_param(), which runs before arm64_memblock_init(). The last use
of an option wins, and ctxr_exec_enforce= applies to a window from either
place. A value that doesn't parse is ignored with a warning. So is a window
from the command line that can't work, which leaves the device tree's
window if there is one. A window from the device tree that can't work is
still fatal when enforcing, since the kernel can't run on the confined
cores outside it.

Signed-off-by: Diljit Singh <hey@djdev.me>
The pool leaves out whatever was reserved before the window, but two paths
still took window pages to the page allocator, and the verifier then
panicked at boot.

A reusable CMA region inside the window goes to the page allocator from
cma_activate_area(). Don't activate a CMA area that overlaps the window,
and keep its pages reserved.

A reserved-memory node with a "reg" whose init fails is freed back to
memblock from unflatten_device_tree(), after the window was reserved. That
leaves a hole for memblock_free_all() to release, so reserve the window
again before that.

Record the window when it's reserved rather than when the pool is set up,
because execmem_init() and CMA activation are both core_initcalls.

Signed-off-by: Diljit Singh <hey@djdev.me>
__vmalloc_node_range() unpoisons a VM_ALLOC area once it's mapped and
registers it with kmemleak, and vfree() unregisters it. A confined
allocation goes around both, so on a debug kernel KASAN sees the area as
poisoned and kmemleak doesn't know it. Do the same here.

Signed-off-by: Diljit Singh <hey@djdev.me>
Describe the /chosen properties a bootloader uses to give the kernel its
executable window.

Signed-off-by: Diljit Singh <hey@djdev.me>
Add the command-line options to kernel-parameters.txt and describe how
they treat bad values. Say that a confined machine loses CPU hotplug and
kexec, and that hibernate is untested under a window. aura doesn't emit
aurora,ctxr-exec-enforce, so it's no longer shown as part of what the
bootloader emits. The Kconfig help said to say N if unsure while
asahi.config turns the option on, so say who wants it.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Thanks for the review. It covered the first version, so each item was checked against the current branch, head c4b5fab28dd. All testing is under QEMU, which doesn't enforce the window. None of these fixes has run on hardware.

Already fixed before this round

  • Memory handed out twice: 0a325c1dc7c builds the pool from for_each_free_mem_range(), taken just before the window is reserved, as the last step of arm64_memblock_init(). That's after the initrd reservation and early_init_fdt_scan_reserved_mem(). With an initrd, FDT, reg node, no-map node, dynamic node and /memreserve/ all in the window, the pool's ranges exclude every one.
  • free_initrd_mem(): 4b630f74a37 makes free_reserved_area() skip window pages. An initrd reaching past the window's end frees only the part outside it. 0 violations, and 0 user pages in the window.
  • kexec relocation pages: 0ab88ef01d7 refuses kexec and crash kernels while a window is enforcing, with EOPNOTSUPP. All four load paths were tested.
  • The window is now 128 MiB, not 1G.

Fixed now

  • d637de6296f: CMA did still get through. A reusable CMA area in the window went to the page allocator, and the verifier panicked with 2048 pages. A CMA area overlapping the window now isn't activated. Testing also found a reserved-memory node whose init fails leaving a hole for memblock_free_all(), a 64-page panic. The window is now re-reserved before that.
  • d9de654671e: early_param() parsing. ctxr_exec_enforce= now applies to a device-tree window. No more substring matches, and the last use wins. A value that doesn't parse, or a command-line window that can't work, warns and doesn't panic. A device-tree window that can't work is still fatal when enforcing, because the kernel can't run on the confined cores outside it.
  • b3a9578538f: KASAN and kmemleak. Without it, BUG: KASAN: vmalloc-out-of-bounds in jit_fill_hole at boot and kmemleak: Not scanning unknown object on module load. With it, no reports and an empty scan. That debug kernel is defconfig at a 512 MiB window, not the asahi profile, because its image is 101 MiB.
  • 524f9114cf1: binding for /chosen/aurora,ctxr-exec-window and aurora,ctxr-exec-enforce, passes dt_binding_check (yamllint wasn't installed).
  • c4b5fab28dd: Kconfig help no longer says N while asahi.config turns the option on. The doc now states that CPU hotplug and kexec are gone on a confined machine.

Left as is: hibernate's trans_pgd page, by the owner's decision. The doc says it's untested under a window.

With no window, the entry path on M1 to M3 parts takes the usual EL1 detour, traced in QEMU with a MIDR override rather than booted on a real M1 to M3, and the feature's code paths stay inert. This head isn't merged with integration/aurora's latest (a620f41876d, J713 and J616c work), so that it stays the tree tested; it merges cleanly. Hardware proof with a window still has to come from a Neo boot before merge. Full results are in the PR body.

…r not

Whether a core can fetch from outside the window is decided by the
hardware, not by aurora,ctxr-exec-enforce or ctxr_exec_enforce=. Those
only decide whether a verifier finding is fatal. EFI runtime services,
kexec and the EL0 fetch check keyed on enforcement, so a confined
machine booted with enforcement off, now possible from the command line,
would run EFI runtime code from efi_rts_wq on a confined core, allow a
kexec that hangs, and leave a task faulting forever on the zero page.

Key all three on the window being active.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
empty_zero_page is in the image's .bss, so it is inside the window. Any
anonymous mapping that is read before it is written maps it, including
PROT_EXEC ones, userfaultfd's UFFDIO_ZEROPAGE and KSM's zero pages, and
KVM can map it into a guest's stage 2 for untouched guest RAM. EL0 can't
execute from the window on a confined core, so an unprivileged program
that jumps into such memory is killed there and gets SIGILL anywhere
else, and a guest fetching from it may fault to EL2 again and again.

With a window reserved, allocate the zero pages from memblock, outside
the window, and clear their MTE tags as cpu_enable_mte() does for
empty_zero_page. The fault-path kill stays as a backstop.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
ctxr_exec_selftest checked, once per boot, that a confined allocation
lands in the window and that the page-table sweep notices an executable
mapping placed outside it. /sys/kernel/debug/execmem_confine reran the
sweep and printed the pool. Both were test aids for bring-up, and the
selftest mapped a page RWX to do its second check. Remove them, and
make execmem_confine_verify() static, since the boot-time check is now
its only caller.

The boot-time verifier stays, and so does the EL0 fetch kill, neither
of which depends on what goes here. ctxr_exec_enforce= now only decides
whether a verifier finding at late_initcall panics.

Signed-off-by: Diljit Singh <hey@djdev.me>
5863e24 added dm-crypt, the crypto user APIs, the sound sequencer,
Bluetooth RFCOMM and BNEP, INET_UDP_DIAG, joydev, uinput and HIDRAW to
the asahi fragment. None of that is about the CTXR window. Drop it and
keep CONFIG_ARM64_CTXR_EXECMEM=y with its comment.

The fleet's Fedora Asahi base config already has every one of them,
and four of these lines turned the base's built-in BLK_DEV_DM and
CRYPTO_USER_API_{HASH,SKCIPHER,AEAD} into modules, so the fleet kernel
gets those back as they were.

Signed-off-by: Diljit Singh <hey@djdev.me>
integration/aurora appended CONFIG_EDAC_APPLE_SOC=m to the asahi
fragment, where this branch appends CONFIG_ARM64_CTXR_EXECMEM=y. Keep
both, the EDAC line first, so the branch's change to the fragment stays
the CTXR block alone.

Signed-off-by: Diljit Singh <hey@djdev.me>
@sithglan

sithglan commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

J813/T8142 (M5 MacBook Air) boot result: Linux runs at EL2 with its code confined to the window.

I merged this branch (head a1a2ac0) into the J813 integration stack as 6e2897803fd1 and built with CONFIG_ARM64_CTXR_EXECMEM=y, LOCALVERSION -aurora-j813-ctxr; the built t8142-j813.dtb is byte-identical to the desktop DTB already in the ESP. Booted it on the M5 through the monitor-free aura chain (aurora-silicon/aura#77) with the padded Stage 1 (aurora-silicon/aurora-installer#104).

dmesg

Linux version 7.3.0-aurora-j813-ctxr+ (sithglan@midi) (Debian clang version 22.1.8 (1+b2), Debian LLD 22.1.8) #13 SMP PREEMPT Fri Nov  6 23:09:46 CET 2026
ctxr-execmem: reserved exec window [0x0000010007600000, +128 MiB), enforcing
CPU features: detected: Virtualization Host Extensions
SMP: Total of 10 processors activated.
CPU: All CPU(s) started at EL2
execmem-confine: window [0x0000010007600000, 0x000001000f600000) (128 MiB), enforcing
ctxr-execmem: executable pool: 110 MiB usable in 1 range(s) of a 128 MiB window (17 MiB already reserved)
execmem-confine: boot verify: 576 executable mapping(s), 0 violation(s): PASS
  • 45 modules loaded after boot with no execmem/allocation errors beyond the pool message, so confined allocations are being backed from the window in practice.
  • All ten CPUs came up through spin-table and run at EL2; no maxcpus=1.
  • NVMe ext4 root, systemd running, no failed units; Wi-Fi up; aurora 1.0.0 with GPU firmware 25G76; AOP RTKit initialized and the audio devices are present.
  • /sys/power/state = freeze mem, /sys/power/mem_sleep = [s2idle]: this boot has no suspend-to-RAM (that's a separate series), and I haven't exercised the documented hibernate/kexec refusals beyond boot.

cc @Acelogic @DjDeveloperr @RyanTheTide @LiamRay10

init_el2 read ctxr_stay_el2 with the MMU off on every entry, the boot
CPU's first one included. That entry runs before ctxr_exec_reserve()
writes the flag, and nothing cleaned the .mmuoff.data.read line to the
PoC: primary_entry cleans only the idmap text and init_el2 the HYP
text, and the stub's relocation syncs code to the PoU. So on an M1-M3
machine with no window, stale DRAM from an earlier ctxr_exec_window=
boot, or garbage, could send the boot CPU down the direct VHE path,
running finalise_el2_state before init_feature_override() and ignoring
kvm-arm.mode= and the VHE overrides. The secondaries then read the same
uncleaned line.

The boot CPU is never confined, so it now always takes the stock entry:
init_kernel_el reads the flag only when x22 is 1, which just the
secondary entry points and cpu_resume set; primary_entry sets it to 0.
ctxr_exec_reserve() writes the flag on every boot, 0 when no window is
active, and cleans it to the PoC before any secondary is released, so a
confined secondary or a resuming CPU reads a defined value and no stale
line survives.

Signed-off-by: Diljit Singh <hey@djdev.me>
A successful ctxr_exec_window= returned before the device-tree checks,
so it never saw the EL2 test or the hold path, and nothing compared it
with the firmware's window. At EL1 it would confine the kernel's text
and still release the secondaries; a window unrelated to the one the
firmware armed would place code where a confined core can't fetch it.

Route both the same way: a window from either place needs the kernel at
EL2, and a command-line window is accepted only when it lies inside the
device-tree window, if there is one. Anything else keeps the secondaries
offline, as a bad device-tree window already did. A command-line window
with no device-tree window (VM bring-up) is unchanged except that it now
requires EL2. Update the binding doc and kernel-parameters.txt.

Signed-off-by: Diljit Singh <hey@djdev.me>
The backstop only walked a folio, so a VM_PFNMAP or struct-page-less
VM_MIXEDMAP mapping was never checked, although the comment said the PFN
might have no struct page. With the zero page now outside the window, a
driver's remap_pfn_range() of an image buffer is the likeliest way a
window page reaches EL0, and that is exactly a PFNMAP with no folio.

Fall back to follow_pfnmap_start() for such a VMA when the folio walk
finds nothing, and compare that PFN against the window as before. The
doc's claim is corrected to match.

Signed-off-by: Diljit Singh <hey@djdev.me>
unflatten_device_tree() runs fdt_scan_reserved_mem_late(), which frees a
reserved-memory node whose late init fails and can punch a hole in the
window. bootmem_init() then reserves the crash kernel, which could land
in that hole and later be handed to the buddy as a window page through
free_reserved_page(). Close the hole at the start of bootmem_init(),
before arch_reserve_crashkernel(), instead of in arch_mm_preinit() after
it.

Signed-off-by: Diljit Singh <hey@djdev.me>
cma_activate_area() allocated each range's bitmap first and only then
checked whether the area overlaps the executable window. A bitmap
allocation failure jumps to cleanup, which frees the area's pages to the
buddy unless the reserve-on-error bit is set, so an overlapping area
could leak window pages into the allocator on that path. Do the overlap
check, which sets that bit, before the bitmap loop.

Signed-off-by: Diljit Singh <hey@djdev.me>
execmem_confine_alloc() drew the whole allocation from the window with a
single gen_pool_alloc(), which needs that many physically contiguous
bytes free. The pages are vmapped one by one, so contiguity isn't
needed, and a fragmented window could fail a 2 MiB BPF pack or a large
module's text with enough free pages to spare. Allocate a page at a
time, record the pages, and return them to the pool individually on free
and on every error path.

While here, use kvmalloc_array() for the page array, which can be large,
and warn if set_memory_nx()/set_memory_rw() fail on free rather than
leaving a possible executable alias behind without a word.

Signed-off-by: Diljit Singh <hey@djdev.me>
When a window is active __vdso_init() copies each vDSO code page into a
fresh page. If one allocation failed it returned -ENOMEM leaving the
page-list array and the copies already made behind. Free them on the
error path, taking care not to free the image pages used when no window
is active.

Signed-off-by: Diljit Singh <hey@djdev.me>
…olds

The bootloader places the static image in the window; the kernel doesn't.
And the pool is built from the window's free ranges snapshotted when it
was reserved, not from a "non-image remainder". Fix the Kconfig help, the
documentation and the file comment.

Signed-off-by: Diljit Singh <hey@djdev.me>
apple_cpuidle_init() open-coded the M1 to M3 compatibles that
apple_cyc_ovrd_machines already lists. Match against that list and add
only the M4-and-later parts that load the driver for its WFI state
(T6050, T8140), so the two can't drift apart.

Signed-off-by: Diljit Singh <hey@djdev.me>
@RyanTheTide

Copy link
Copy Markdown
Contributor

First hardware boots with a real CTXR window: the Neo (J700, T8140), 07/11.

Setup

  • Stage 1: aura#77 cda450d, padded to 128 MiB by aurora-installer#104's aura-execute-window and enrolled with kmutil configure-boot --compress (kcbz 0, kcrz 153141248).
  • Stage 2 and U-Boot: m1n1/boot-ctxr.bin = aura#77 cda450d Stage 2 + the fleet j700 DTB + u-boot#32 50d879ac, then GRUB.
  • Kernels, both with CONFIG_ARM64_CTXR_EXECMEM=y:
    • a1a2ac0 (fleet recipe), first boot only.
    • f6b1cbf (this PR's head, fleet recipe), the full run.

Boot (both kernels)

  • ctxr-execmem: reserved exec window [0x10007200000, +128 MiB), enforcing.
  • Pool: 53 MiB usable in 2 ranges (74 MiB already reserved by the image).
  • boot verify: 1696 executable mapping(s), 0 violation(s): PASS. Kernel code and data both sit inside the window.
  • CPU: All CPU(s) started at EL2 on 6 CPUs, released via spin-table (no online files). KVM comes up with nested virt.
  • Up 36 s after the reboot out of Recovery.
  • Regression against the fleet kernel before the enrol:
    • 0 AVCs (enforcing), 0 failed units, 0 oopses.
    • 1 kernel warning (the existing ISAR2 override).
    • Wi-Fi connected, eDP up, GPU and Vulkan working, 19 efivars.
    • Touch ID kept its enrolment across the Stage 1 policy change: 1 identity restored, 0 seal failures.

Stress on f6b1cbf

  • Modules:
    • 1200 load/unload cycles over 1002 unused modules with 6 workers, 0 ENOMEM.
    • amdgpu, nouveau and radeon loaded and unloaded 5 times each.
    • A 4 MiB nop-sled module's text ran on every CPU; 100 load/unload cycles brought the pool back to baseline.
    • Static keys and livepatch work on all CPUs.
    • Filling the pool ends in a clean ENOMEM, and a BPF image is refused (524) while it is full. Everything already loaded keeps running, and the pool recovers exactly.
    • Fragmentation: with only 4 MiB free in the largest extent, a 16 MiB module now loads. On a1a2ac0 it was refused, so 344bd8f fixes this on hardware.
  • BPF:
    • test_bpf: 1061/1061 JITed, on each of the 6 CPUs.
    • 3000 small and 200 4k-insn programs run on every CPU, with and without blinding.
    • A 32-deep tail-call chain works.
    • 37 fentry programs on one trampoline, attached and detached 100 times.
    • Filling to bpf_jit_limit and filling the pool both stop cleanly and recover.
  • kprobes/ftrace:
    • 400 kprobes and 400 kretprobes, hit on all CPUs, insn slots from the pool.
    • function and function_graph tracers on all functions, 10 rounds each.
    • ftrace direct and direct-modify, 20 cycles each.
    • Pool unchanged afterwards.
  • KVM at EL2:
    • 20 VMs x 4 vCPUs x 50 runs pinned to each host CPU (CPU0 and every secondary): 4000/4000 each.
    • 64-vCPU VMs work.
  • s2idle: rtcwake -d rtc1 -m freeze -s 20 with big modules, 500 BPF programs, kprobes and static keys live. After resume, all of them still run on every CPU.
  • Refusals:
    • /dev/mem refuses the window, kernel code and RAM outside the window.
    • kexec_load, kexec_file_load and both crash variants return EOPNOTSUPP.
    • No CPU can be offlined.
  • Escape attempts (page outside the window, run by a kthread bound to each CPU):
    • Tried two ways: set_memory_x on a vmalloc page, and a PAGE_KERNEL_EXEC vmalloc page.
    • Positive control: a page from the pool runs on all 6 CPUs.
    • Every secondary takes an instruction abort at EL2 (ESR 0x8600000f), even though the PTE is kernel-executable. The worker oopses, and the machine keeps running on all CPUs.
    • CPU0 runs the code, as designed.

Issues

  1. Unprivileged users can exhaust the window.
    • As uid 65534 with no capabilities, 1379 classic socket filters (4096 insns each) filled the pool in 0.3 s. Root's insmod of a 4 MiB module then failed with ENOMEM.
    • It recovers when the sockets close.
    • The default bpf_jit_limit is 70092251463680 (vmalloc-sized), so nothing caps this against a ~50 MiB pool. Seccomp filters take the same path.
    • Suggest clamping the default bpf_jit_limit and bpf_jit_limit_max to a fraction of the pool while confinement is active.
  2. Nothing reports pool size or use. A read-only counter (size, free, largest free extent) would make an ENOMEM on a user's Mac diagnosable.
  3. Misleading oops text. An out-of-window kernel fetch on a secondary is reported as "execute from non-executable memory". Naming the window there would save someone a debugging session.

Not from this PR, seen on the way

  • EFI: GetTime returns EFI_UNSUPPORTED from U-Boot's runtime, and efivarfs mounts read-only. GetVariable works (500/500), and runtime services stay on.
  • Bluetooth: this tree's asahi.config lacks CONFIG_BT_MTK7932_PCIE, so there's no hci0. The fleet build adds it.
  • aegis128: unloading it oopses on arm64 on any kernel. Module exit calls crypto_aegis128_have_simd(), which writes __ro_after_init data.

From the Neo side this is ready. The J616s gets the same run next.

🤖 Generated with Claude Code

bpf_jit_limit, the cap on unprivileged JIT memory, defaults to half of
bpf_jit_alloc_exec_limit(), which arm64 reports as the vmalloc space
(about 64 TiB). Under a CTXR window every JIT image comes from the
window's executable pool instead, about 53 MiB on a J700. On that
machine an unprivileged user (uid 65534, no capabilities) filled the
pool with 1379 classic socket filters in 0.3 s, after which root could
no longer load a 4 MiB module.

Record how much of the window the pool gets when the window is reserved,
which is before bpf_jit_charge_init() runs, and report that as the JIT
allocation limit. bpf_jit_limit then defaults to half of the pool, and
bpf_jit_limit_max to the pool. Without a window nothing changes.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
bpf_jit_limit only charges unprivileged programs. Privileged BPF (cgroup
programs from the init system, tracing tools) can still fill the
confined pool, and then no module loads. Add EXECMEM_CONFINE_BOUNDED: a
confined range with it never takes the pool below a reserve, a quarter
of the pool, kept for the other confined ranges. arm64 sets it on the
BPF range.

Hold confine_lock across the page loop so the check still holds when
the last page is taken, and free the pages of a failed allocation under
the same lock.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
@RyanTheTide

Copy link
Copy Markdown
Contributor

Pushed two commits for the unprivileged JIT fill from my last comment. The head is now 273b9e8 (fast-forward from f6b1cbf).

What the bug was: a local denial of service. It isn't an escape, since nothing runs outside the window. bpf_jit_limit defaults to half of bpf_jit_alloc_exec_limit(), and arm64 returns the vmalloc size for that (70092251463680). Under a window every JIT image comes from the pool, so the limit never applied. A user with no capabilities filled the pool with classic socket filters. With BPF_JIT_ALWAYS_ON=y, every later JIT allocation then failed until those sockets closed. Root couldn't load modules, and seccomp and cgroup programs failed to load too, so units with SystemCallFilter=/IPAddress*= and bwrap/Flatpak/Chromium sandboxes wouldn't start.

Commits

  • 6315aea arm64: bpf: size the JIT limit from the CTXR pool
    • The window code records how many bytes the pool gets when it reserves the window, which happens before bpf_jit_charge_init().
    • bpf_jit_alloc_exec_limit() returns that number, so the existing default (half) now limits unprivileged programs. With no window it still returns the vmalloc size.
    • It uses the pool, not the window size. On the Neo the image already takes 74 of the 128 MiB. Half of the window would be 64 MiB, more than the 53 MiB pool, so it wouldn't stop the fill.
  • 273b9e8 mm/execmem: keep a quarter of the confined pool for module text
    • bpf_jit_limit doesn't count privileged programs, so root BPF (init-system cgroup programs, tracing) could still fill the pool.
    • New flag EXECMEM_CONFINE_BOUNDED, set on the arm64 BPF range: a BPF allocation never takes the pool below a quarter of its size.
    • confine_lock is now held across the page loop, so the check still holds when the last page is taken.

Both commits do nothing without a window. W=1 shows 0 warnings on the four objects they touch.

Neo (J700), padded Stage 1, 273b9e8, fleet config

  • Window enforcing, 6 CPUs at EL2, boot verify 1696 mappings / 0 violations.
  • Pool 53 MiB, bpf_jit_limit=28114944, 13728 KiB kept for module text.
  • Unprivileged fill (uid 65534): stops after 848 x 4096-insn filters (ENOTSUPP, JIT refused) with 18 MiB of the pool still free. While those filters are held, root loads a 4 MiB module and a BPF program.
  • Root fill: stops at the module reserve, and a 4 MiB module still loads.
  • After the sockets close, the pool returns to its starting use. No kernel-bug lines.
  • No regressions: 1200 module load/unload cycles with 0 ENOMEM, the module fill/run/recover set, and test_bpf 1061/1061 JIT'ed on all 6 CPUs.
  • The only FAIL is the old one: nothing reports pool use. I have a small sysfs patch for that (/sys/kernel/mm/execmem_confine/{window,pool,free,module_reserve}_bytes) but left it out to keep this minimal. Say if you want it.

QEMU (TCG, KASAN, ctxr_exec_window=256M, 107 MiB pool): the unprivileged fill stops at bpf_jit_limit=56131584. Root module and BPF loads succeed while it's held, a root fill stops at the reserve, and there are 0 KASAN reports.

No window, M1 Pro: a one-shot of the same tree passed. All 10 CPUs came up at EL2. bpf_jit_limit is unchanged, with 0 AVC, 0 failed units, and fewer kernel warnings than fleet16. Wi-Fi, GPU and Vulkan work, and modules load and unload.

The same Neo boot also went through u-boot#32's head bb6e1e3, as a one-shot trial. Results are on aurora-silicon/u-boot#32.

🤖 Generated with Claude Code

@RyanTheTide
RyanTheTide merged commit bf8f10d into integration/aurora Nov 7, 2026
1 check passed
@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Thanks for the run, the BPF fix and the merge. Five commits landed on feat/ctxr-execmem after the merge, so they're now in #1079:

  • /sys/kernel/execmem_confine/ reports the window and pool use (your item 2).
  • The oops names the CTXR window for an out-of-window fetch (your item 3).
  • crashkernel= is skipped under a window.
  • The device-tree window is used unless ctxr_exec_window= matches it. A smaller command-line window used to hand the rest of the firmware's window to the page allocator.
  • The EL0 backstop only checks user-executable PFN-mapped pages.

All QEMU only, 11 of 11 window scenarios. Your 6315aea replaced our BPF clamp.

Still open: on your Neo through U-Boot and GRUB, 74 MiB of the window was already taken, against 17 MiB on sithglan's M5. If you still have that boot's dmesg or /proc/iomem, it would show what's sitting there.

🤖 Generated with Claude Code

RyanTheTide added a commit that referenced this pull request Nov 7, 2026
Brings aurora-wip up to the tested release branch: CTXR execmem
(#985, #1079), KFIX speaker sleep and resume fixes (#1077, #1080),
JPEG v2 (#1072), Jonathan's G17 and USB work (#1073-#1076), brcmfmac
flowring count (#1081), ANE H16 startup restart (#1066) and the
Touch ID per-prompt match (#904).

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants