Skip to content

aura: remove the EL2 monitor and run Linux at EL2 in the CTXR window - #77

Merged
DjDeveloperr merged 26 commits into
mainfrom
feat/ctxr-exec-window
Nov 7, 2026
Merged

DjDeveloperr merged 26 commits into
mainfrom
feat/ctxr-exec-window

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

This removes aura's resident EL2 monitor, PSCI, NV2 and PV-KVM, and runs Linux at EL2 inside the CTXR window on confined Macs. It is the owner's decision for the M4-generation release. Non-confined Macs (M1 to M3, raw boot objects) boot as before.

The monitor-free build has not booted on any hardware. The next step is one Neo boot of the full set, and this should not merge before that.

Why

Every M4-and-later chip boots a Mach-O object with kcbz 0, which skips SPTM. iBoot then leaves the CTXR windows armed and locked on the secondaries, and at EL2 they can only execute inside them. The monitor worked around that by running the OS at EL1 under aura. A kernel built with CONFIG_ARM64_CTXR_EXECMEM (aurora-silicon/linux#985) keeps all its code in one physical window, so it runs at EL2 with nothing of aura under it.

Linux never runs at EL1 on any boot path in this tree, raw or Mach-O. aura enters the kernel directly at EL2, as it always did on M1 to M3, and on a confined Mac it only adds the window to the device tree.

What's gone

The monitor, PSCI, NV2 and PV-KVM are removed completely. The code is kept in a private Aurora Silicon repository, so nothing here depends on it.

Removed: src/monitor.c, src/psci.c, src/nv2.c, src/nv2_asm.S, src/pv_kvm.c and their headers, monitor_eret, the ERET gate in _v_lower_sync, monitor_active, next_stage.monitor, the SMC handling, the exception hooks, the /psci and /hypervisor nodes, aura-nv2, aura-pv, chosen.aura,nv2, chosen.aura,pv, the NV2_BOOT and NV2_TRACE build switches, docs/j700-nv2-pmu.md and their tests.

The branch now has main up to 671c0ce merged in. The merge was clean, and main added no new users of any of it. A grep of the whole tree finds none left in code, the Makefile, tests, docs or config. What remains is the architectural HCR_NV* definitions from upstream, el1_call, el1_ret, gl2_call and gl_call_wrapper (the proxy and GXF), and aura's own hypervisor in src/hv and proxyclient, which runs its guests at EL1 by design. kboot still has one EL1 branch, for aura booted as a guest under that hypervisor (the J616s HV test in docs/j616s-bringup.md).

M4 and later take one path

There is no chip list on the CTXR path. Each check reads that chip's registers.

  • The probe. src/ctxr_exec.c asks a secondary to run code from the heap. The machine is confined if that faults. A secondary that doesn't take the probe within 1 second (smp_call4()), or takes it and doesn't answer within 500 ms, stops the boot before any device is handed over, because kboot never hands off with a call pending. The panel says the CPU didn't take or didn't answer the probe. The guard is off on every path. A probe that couldn't be sent counts as not confined.
  • A locked RVBAR. smp_prepare_rvbar() forwards it with a branch on any chip, when it points at the boot object's Mach-O header 64 KiB below the raw image, or at an earlier aura image's vectors. iBoot locks it at the header on T6040, which doesn't take the vector from kcev. This is fbalazsy's fix from J616s: boot Linux from a Mach-O boot object, with a fixed layout for hibernation #66, cherry-picked with his authorship and then made chip-independent. Without it no T6040 secondary started in Mach-O mode, so the probe had nobody to ask. Anything else is refused as before.
  • Reservations. Every confined chip reserves CTXR_D outside the window and CTXR_A to CTXR_C. A window inside T8140's firmware RO range is left to that range. The monitor's 2026-10-31 Neo boots relied on all three being there, so the J700 tree gains no entry, and the test fixture now has them there. T8140's firmware RO range is reserved at either EL, as on main. CTRR_C and CTRR_D are reserved on M4 and later at EL2. At EL1, under aura's hypervisor, kboot says what it leaves to the host: CTRR_D on T8140, both elsewhere. All of it is in dt_reserve_secondary_ro().
  • The window check takes the firmware RO range where one is known, CTXR_A to CTXR_C, and CTRR_C and CTRR_D, on every chip.
  • Guarded reads. Every CTXR and CTRR read, CTXR_D included, goes through ctxr_range(), under an exception guard, so an unexpected UNDEF on a chip nobody has probed counts as not armed instead of ending the boot. A range whose LWR is 0 isn't armed, in the window check and in kboot alike. A chip with unlocked Apple system registers (M1 to M3) skips CTRR in the window check, as kboot does.

The SMP-RO reservations differ from main in three ways. tests/python/test_secondary_ro_reservations.py runs dt_reserve_secondary_ro() for T8140, another M4-or-later chip and M1 to M3, at EL1 and at EL2, and spells each one out:

  • T8140 at EL2 also reserves CTRR_D when it is armed. Main never did. It read 0 on the Neo on 2026-10-31, so the J700 tree shouldn't change.
  • Other M4-and-later chips reserve CTRR_C to UPR + 4K, like every other range. Main stopped at UPR, so a 48 KiB range was 0xb000 bytes instead of 0xc000.
  • A CTRR_C that reads as empty or wrapped gets no entry. Main added a zero-size or wrapped one.

One T8140-only piece is left on purpose. memory_fw_ro_range() answers only on T8140, because its fixed 48 KiB range at a measured place is what the MMU and the chainload placement use there. No other chip has been measured, and the generic path doesn't need it.

Things outside the CTXR mechanism keep their chip lists. T8140 refuses a handoff with missing CPUs where other chips boot on fewer. The idle bootargs keep the list of kernels that save WFI state themselves. T8140 adds arm64.nowfxt, which the linux#985 kernel already forces on Apple parts from M4 on.

System registers

apple_sysregs_unlocked keeps upstream's meaning, and every use has main's truth table. Upstream m1n1 29073cc renamed cyc_ovrd to it: it is false on M4 and later, and gates CYC_OVRD, RVBAR, the L2C error status and SPRR/GXF. The CPU features record is the same 20 bytes as on main, so every enrolled Stage 1 and current proxyclient keeps working.

At EL2 on M4 and later HACR_EL2, the Apple ACTLR bits and the per-core deep sleep enable can be set. CYC_OVRD is undefined there, and with a kcbz-0 object the GXF registers are writable, though aura doesn't use them. The flag's comment says so, but nothing gates on it. aura's hypervisor already writes HACR_EL2 and ACTLR_EL2 on every chip, and Linux finds the ACTLR features by AIDR and readback at EL2, so the device tree says nothing about them. The kernel reads no property about this, and I added none. Nothing writes the deep sleep enable for the OS, because a secondary in deep WFI with it set stopped a Neo within 2 seconds.

What a confined Mac loses compared with main

  • CPU hotplug. Spin-table has no CPU off, and linux#985 has no cpu_die for it, so offlining a CPU fails with EOPNOTSUPP. Suspend to RAM and hibernation take secondaries offline the same way.
  • PSCI SYSTEM_RESET. Restart needs macsmc-reboot or the Apple watchdog. The monitor answered SYSTEM_OFF with a reset too, so power-off never worked through it, and it needs macsmc-reboot now as before.
  • kexec and kdump. linux#985 refuses both under an enforcing window.
  • NV2 and PV-KVM. KVM at EL2 in the window hasn't been tried.

Two costs that stay, and are written down in the docs. aura can't tell whether the kernel was built with CONFIG_ARM64_CTXR_EXECMEM, and moves one without it all the same. On T8140, T6041, T8152 and the J713 the heap copy of a moved kernel stays reserved with the retained heap: about 1 MiB for U-Boot, about 40 MiB for a kernel booted with boot=. Freeing it would mean editing memreserve entries after fdt_pack().

Measured

  • On the Neo, with an earlier version of this branch and a 256 MiB window. Linux reached a login at EL2 on 6 CPUs from NVMe through U-Boot booti, on one boot.
  • On a J616s, in J616s: boot Linux from a Mach-O boot object, with a fixed layout for hibernation #66. That series, header branch included, ran Linux with its secondaries up and hibernated, under the monitor this PR removes.
  • On this Mac. Stage 1 (RELEASE=1 CHAINLOADING=1 BUILTIN_LOGO=aurora) and the ESP Stage 2 (the same with ESP_STAGE2=1) build from make clean with no new warnings. pytest over tests/python and the two T8122 files gives 597 passed and 47 skipped. format-check (clang-format 20.1.8) and rustfmt-check pass.
  • Stage 1 size with clang. aura-raw.ld aligns each section to 64 KiB, so Stage 1 built with Homebrew clang fits under the DCP log buffer only while .text ends below its next boundary. With clang 23.1.1 and Rust 1.98.1 it ends 68 bytes short. Nothing could be cut cleanly to win back real room: the C has no dead code left, and building the Rust with opt-level s would change code on every chip without a hardware test. So a new stage1-clang job in build.yml links both stages with Homebrew clang on macos-26, and fails CI before it fails on a Mac. Releases build with gcc. The runner's LLVM and Rust may not match these versions byte for byte, so the job may tell us the margin is gone before this Mac does.

Not measured

  • This code on any hardware, including the RVBAR forward on a chip other than T6040, the 128 MiB window and the refusal path.
  • Any chip other than T8140 in this mode. T8132, T6040, T6041 and T8142 come from reading the code and the trees.
  • Idle power in this mode, and restart and power-off without PSCI.

Conflicts with open PRs

🤖 Generated with Claude Code

@DjDeveloperr
DjDeveloperr force-pushed the feat/ctxr-exec-window branch from a4df466 to 412c0e7 Compare November 5, 2026 11:34
@DjDeveloperr DjDeveloperr changed the title kboot: emit the CTXR executable-window contract to the OS aura: boot a CTXR-confined Linux kernel at EL2 (no resident monitor) Nov 5, 2026
PV-KVM ran only beside the EL2 monitor, which this series removes. It was a
bounded discovery and RUN prototype, not a /dev/kvm backend, and the kernel
has no client for it. Remove the sources, the aura-pv option, the
chosen.aura,pv ESP line, the /hypervisor node and its contract test.

Signed-off-by: Diljit Singh <hey@djdev.me>
With a kcbz-0 boot object iBoot leaves the CTXR windows armed and locked on the
secondary CPUs, and at EL2 they can execute only inside them. Move the probe that
detects this out of the monitor into src/ctxr_exec.c, renamed ctxr_exec_confined(),
and run the OS inside the window on a confined machine. kboot moves the kernel
there, keeps spin-table, and publishes /chosen/aurora,ctxr-exec-window for
CONFIG_ARM64_CTXR_EXECMEM and U-Boot. A machine that is not confined never reads
a CTXR register and sees the tree it always saw.

With no usable window there is no fallback: kboot_prepare_dt() and kboot_boot()
refuse, print the reason on the console and the panel, and name the installer's
Recovery step. The boot then follows the existing failure path into the proxy.

The window is 128 MiB wherever the object armed more. The rest of CTXR_D stays
reserved, because EL0 cannot execute there on a confined core and the OS only
pools its window.

There is no chip gate, since each check goes by the registers:

- memory_fw_ro_range() answers only on T8140, so other chips check the window
  against CTXR A to C and the CTRR ranges kboot reserves.
- Other chips reserve windows A to C themselves. T8140's firmware RO range
  covers them.
- The J713 high-water interval and a described bootloader handoff reach past
  the window, so they go around it. Reserving them would leave the kernel no
  pool.
- The T8152 handoff moves the kernel too.

Only T8140 has run this path. This leaves the monitor as dead code, removed next.

Signed-off-by: Diljit Singh <hey@djdev.me>
Linux now runs at EL2 on a confined machine, inside the CTXR window, and a
confined machine with no window does not start the OS. So aura no longer stays
resident under the OS.

Remove src/monitor.c, monitor_eret, monitor_active and next_stage.monitor, and
the console gating that went with them. Remove src/psci.c, the SMC handling and
the /psci node, so the kernel always gets spin-table. Remove the park loop's
re-entry helpers, which only CPU_OFF used.

NV2 ran only under the monitor: nv2_active was set only with next_stage.monitor
and nv2_enter_os() was only called from monitor_enter_os(). Remove src/nv2.c,
src/nv2.h, src/nv2_asm.S, the aura-nv2 option, the chosen.aura,nv2 ESP line, the
NV2_BOOT and NV2_TRACE build switches, the ERET gate in _v_lower_sync and the
exception hooks. The lower-EL sync vector goes back to _v_sp0_sync.

el1_call, el1_ret, gl2_call and gl_call_wrapper stay for the proxy and GXF, and
src/hv is untouched. Unused architectural register definitions in arm_cpu_regs.h
stay too.

Signed-off-by: Diljit Singh <hey@djdev.me>
Replace the monitor's description with what aura does now, and say why an old
Stage 1 keeps its old Stage 2. tools/fill_stage1_config.py and the Stage 1
parsers already take m1n1/boot-ctxr.bin, so aura needs no change for that path;
pin it with a test.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr
DjDeveloperr force-pushed the feat/ctxr-exec-window branch from 2c5be1d to febf859 Compare November 6, 2026 06:31
@DjDeveloperr DjDeveloperr changed the title aura: boot a CTXR-confined Linux kernel at EL2 (no resident monitor) aura: remove the EL2 monitor and run Linux at EL2 in the CTXR window Nov 6, 2026
ctxr_exec_confined() counted any probe that did not return 1 as
confined, including one that was never delivered (smp_call1() failed)
or never finished (smp_wait() timed out). On main that only put the OS
at EL1 under the monitor. Without the monitor it is a refusal to boot:
an M1 to M3 Mac whose probed secondary is still busy would read an empty
CTXR_D and tell the user to enroll Stage 1 again.

When the probe does not run, say so and treat the secondaries as not
confined: on a Mac that really is confined, the worst case is that its
secondaries stay offline. Leave the probe page allocated in that case,
since the secondary may still run it.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Adversarial review and regression run (CTXR lane, 06/11). Leaving this open: it can't boot on our lab Neo yet, see the last section.

Pushed: f24438b ctxr_exec: a probe that never ran does not mean confined. ctxr_exec_confined() counted a probe that was never delivered (smp_call1() failed) or never finished (smp_wait() timed out) as confined. On main that only put the OS under the monitor; here it is a refusal, so an M1 to M3 Mac whose probed secondary is busy would read an empty CTXR_D and tell the user to re-enroll Stage 1. Now it logs and treats the secondaries as not confined (a confined Mac then at worst boots with its secondaries offline), and keeps the probe page since the secondary may still run it. New case in test_ctxr_exec_probe.py; that file, test_ctxr_exec_window.py and test_smp_handoff_gate.py pass (14).

Checked and fine

  • Window maths (2 MiB alignment, aura slot and _end, no overflow), the kernel memmove and its cache maintenance, and every reservation (CTXR_D minus the window, CTXR_A to C, retained heap, J713) leave the window out.
  • The refusal reaches console and panel, then the proxy, without hanging. An old unpadded Stage 1 that gets this Stage 2 refuses with "did not arm a large enough window".
  • Removal is complete: no references to the monitor, NV2, PV-KVM, PSCI, smp_reenter or _v_lower_sync left in src, src/hv, proxyclient or rust.
  • Non-confined Macs (M1 to M3, raw objects): same DT and boot as main (spin-table, no /psci or /hypervisor, which main only added under the monitor), no CTXR registers read. aura-nv2/aura-pv are now ignored; they did nothing there anyway.

Should fix

  1. No live secondary means "not confined", cached for the boot (ctxr_exec.c, the cpu == MAX_CPUS return). On T6040 in Mach-O mode, or any chip where smp_start_secondaries() fails (payload.c tolerates that), Linux boots at EL2 outside the window with no warning. Print a warning and don't cache an answer when nothing was probed; ideally also check whether CTXR_D reads as armed.
  2. The kernel move doesn't check what it overwrites (kboot_move_to_exec_window). The window check covers boot_args and the heap, not an initrd, the DT or a described handoff; the J713 handoff comment says it "may cover the running kernel and initramfs". Refuse if the initrd or DT overlaps the window, and check the ARM\x64 magic before trusting image_size.
  3. EFI runtime services will be off on every confined boot through U-Boot. The comment in kboot.c says U-Boot places itself and the kernel inside the window, but u-boot#32 only places the kernel; its runtime code stays at the top of RAM and linux#985 turns runtime services off. Fix the comment or the U-Boot side.
  4. What confined Macs lose vs main should be in the docs and measured on the Neo: CPU hotplug (spin-table can't offline a CPU), PSCI SYSTEM_OFF/RESET, kexec/kdump (linux#985 refuses them), cpuidle-psci deep WFI (check cpuidle-apple binds on T8140 and measure idle power: the J700 is fanless), NV2 and PV-KVM.
  5. A DT with no cpu-release-addr now fails the boot on confined chips. Check the T6041, T8132 and T8142 trees before claiming those chips.

Regression run: CI is green. Host build of this branch on macOS needs GNU make 4 (&: grouped target), as main does. Hardware: none of this Stage 2 has booted anywhere yet.

Why it can't boot on our Neo tonight. Both Linux installs on the lab Neo have the unpadded Mach-O Stage 1 enrolled (aurora/stage1.macho, 1,753,088 B), and aura's probe there reports confined secondaries, so CTXR_D is the DT, the trust cache and about 1.7 MB: this Stage 2 refuses. A chainloaded Stage 1 can't change that, because iBoot arms the window from the enrolled object. Testing needs a padded object enrolled through kmutil in Recovery, which is a persistent Stage 1 change and needs the owner password, so it waits for Ryan. Dj's 256 MiB boot was on his own Neo.

With no secondary running, ctxr_exec_confined() answered "not confined" and
cached that for the boot. On T6040 in Mach-O mode, or wherever
smp_start_secondaries() fails, the OS then started outside the window with
nothing in the log, and a secondary started later was never asked.

Warn once, and keep the question open for a later call. Read CTXR_D under an
exception guard, at EL2 only, and say so if it is armed. The OS still starts on
the boot CPU alone, because dt_set_cpus() drops the CPUs that aren't running.

A probe that was sent and didn't finish stays cached: asking again could wait out
the same timeout and lose another page.

Signed-off-by: Diljit Singh <hey@djdev.me>
The window checks cover boot_args and the heap, but not everything that stays in
place when the kernel moves. Refuse if the initrd or the device tree overlaps the
window, and check the arm64 magic before trusting image_size.

A described bootloader handoff may cover the running kernel and initramfs, and
going around it left whatever lives there for the kernel's pool to overwrite. It
now fails kboot_prepare_dt() if it overlaps the window, and is reserved whole
otherwise.

Signed-off-by: Diljit Singh <hey@djdev.me>
List what is gone compared with the monitor: CPU hotplug, PSCI SYSTEM_OFF and
SYSTEM_RESET, kexec and kdump, cpuidle-psci, NV2 and PV-KVM. cpuidle-psci never
had states to use on these trees, and cpuidle-apple binds on T8140. Idle power
hasn't been measured.

Record what the kernel's trees give each chip for cpu-release-addr, which all but
T6040's boot CPU have, and that T605x has no tree yet.

U-Boot's head relocates its image and runtime code into the window, so say that
in the kboot.c comment and the docs.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Thanks for f24438b and the review. Pushed on top of it as d1327ff8, still unmerged. This Stage 2 hasn't booted on any hardware yet.

  1. Fixed in de92e4a3. With no running secondary, aura now warns once and keeps no answer. It reads CTXR_D under an exception guard, at EL2 only, and says if it's armed. The OS still starts on the boot CPU alone, because dt_set_cpus() drops CPUs that aren't running. Your probe-never-ran path stays cached on purpose, because asking again could wait out the same timeout and lose another page.

  2. Fixed in eaf63992. kboot_move_to_exec_window checks the ARM\x64 magic before trusting image_size, and refuses if the initrd or DT overlaps the window. This also reverses the earlier handling of a described handoff, which went around the window and left whatever lives there for the pool to overwrite. Now an overlap fails kboot_prepare_dt(), and otherwise the range is reserved whole.

  3. Comment fixed in d1327ff8, but the finding doesn't look like it applies to u-boot#32's head. Head 80270d92c896 relocates U-Boot's image, which holds its runtime code, to the top of the window (37f6ad66d7), and allocates other EFI runtime code inside it (80270d92c8). linux#985 turns runtime services off only if it finds runtime code outside the window. This is from reading the code, not a boot. The U-Boot side is being checked separately, and the answer will go on u-boot#32.

  4. Documented in d1327ff8, measurement not done. The losses are in docs/macho-boot-object.md and the PR body: hotplug, PSCI SYSTEM_OFF and RESET, kexec and kdump, NV2, PV-KVM.

    • Hotplug: linux#985 has no cpu_die for spin-table, so offlining a CPU fails with EOPNOTSUPP. Suspend to RAM and hibernation use the same path. A parking patch (b27de088de in linux-s2r) ran once on a Neo, and it isn't in linux#985.
    • cpuidle-psci loses less than it sounds: no Apple tree has cpu-idle-states, so it had no states under the monitor either.
    • cpuidle-apple binds on T8140 through apple,t8140-cpuidle, enabled in the J700 tree, with both states plain WFI. That's from reading the driver, not a boot log.
    • Idle power isn't measured and needs a Neo. The J700 is fanless, so it has to be measured before release.
  5. Checked and documented in d1327ff8. In integration/aurora at 0943983df73, none of the three chips you asked about lacks it. T6041 (16 CPUs), T8132 (10) and T8142 (10) have spin-table and cpu-release-addr on every CPU, as does T8140 (6). T6040 has it on every CPU except cpu@10100, which the tree marks as the boot CPU, so it fails if iBoot boots it on any other core. T605x has no tree at all. This only means the trees don't rule those chips out, not that they boot.

Not addressed: T6040's RVBAR at the load base, which PR 66 carries. A T6040 in Mach-O mode still runs on one CPU, now with a warning. Your lab Neo also still needs a padded Stage 1 through Recovery before this Stage 2 will boot there.

Build note: both targets link with macOS make 3.81, but only from make clean. Changing only flags can skip regenerating build_cfg.h and silently give you the wrong stage.

DjDeveloperr and others added 4 commits November 6, 2026 11:19
Bring in main up to 671c0ce. Main adds no new monitor, PSCI, NV2 or PV-KVM users, so the removal stands as it was.

Signed-off-by: Diljit Singh <hey@djdev.me>
A Mach-O boot object puts the raw image 64 KiB after its load base. An
iBoot that does not take the vector address from the object may lock
the secondaries' RVBAR at the load base, which holds the Mach-O header,
and smp_prepare_rvbar() then refused every secondary on T6040. When the
locked address is exactly 64 KiB below the vectors and still holds the
Mach-O magic, replace that word with a branch to the vectors, and accept
the branch on the next secondary. Any other content or distance is
refused as before.

tests/python/test_smp_t6040_load_base.py runs the production function
against modeled MMIO: one write and an RW/RX remap on the first core,
none on the next, refusal of foreign content, another distance or chip.
The existing RVBAR harnesses define T6040 and SZ_16K.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
Signed-off-by: Diljit Singh <hey@djdev.me>
apple_sysregs_unlocked was false on M4 and later, and it gated several
unrelated things there. At EL2 on these chips HACR_EL2, the Apple ACTLR bits
and the per-core deep sleep enable can all be set. CYC_OVRD is the register
that is missing, and it is undefined from M4 on.

Each use now goes by what it needs:

- cyc_ovrd (A7 to M3): the CYC_OVRD writes in init_cpu(), deep_wfi(),
  cpu_sleep() and the hypervisor. The WFI bootargs go by its absence too,
  because without it aura can't turn WFI retention on. Their chip list,
  which says which kernels save state themselves, is unchanged.
- secondary_ctrr (M4 and later): the CTRR_C and CTRR_D reservations in
  kboot, and the CTRR check on the CTXR window.
- legacy_impl_regs (A7 to M3): the old flag's slot in the proxy ABI, for
  what M4 and later lock or have not been shown to have: setting RVBAR, the
  L2C, LSU, FED and MMU error status registers, and the hypervisor's
  AMX_CTL_EL2, VM keys and SPRR/GXF. The proxyclient's uses move to it too.
- apple_sysregs_unlocked now means what it says, at EL2, and is true on M4
  and later as well. Nothing gates on it. aura's hypervisor already writes
  HACR_EL2 and ACTLR_EL2 on every chip, and Linux finds the ACTLR features
  itself, so the device tree says nothing about them.

Nothing writes the deep sleep enable. A secondary in deep WFI with it set
stopped a Neo within 2 seconds. The truth table of every old use is
unchanged on every chip, except that an unknown CPU part no longer reads the
M4 CTRR registers.

The record grows from 20 to 24 bytes. The proxy refuses a client of another
size, and a test now checks the Python layout against the C struct.

Signed-off-by: Diljit Singh <hey@djdev.me>
The CTXR path at EL2 had T8140-only branches. Make it the same on every chip,
so that T6040 and T6041 boot the way the Neo does.

- smp_prepare_rvbar() forwards a locked RVBAR on any chip. It writes a branch
  where RVBAR points when that is the boot object's Mach-O header 64K below
  the raw image, which is where iBoot locks it on T6040 (the previous commit,
  from aura#66, did this for T6040 only), or the vector table of an earlier
  aura image. Anything else is refused, as before. A later stage at the same
  base accepts the header branch an earlier one wrote. The earlier-image
  check takes the inherited top as its bound when there is no firmware RO
  range.
- Every confined chip reserves CTXR_A to CTXR_C. One that lies inside the
  T8140 firmware RO range is left to that range's reservation, so the Neo's
  tree is unchanged.
- The CTRR reservations are one function. Where memory_fw_ro_range() knows
  the range (T8140) it is reserved and kept out of the linear map as before,
  and elsewhere CTRR_C is read. CTRR_D is reserved when it is armed, on every
  chip. CTRR_C now ends at UPR + 4K, like CTRR_D, CTXR_D and the T8140
  range, and is skipped when it reads as empty.
- The window check on every chip takes the firmware RO range when one is
  known, CTXR_A to CTXR_C, and CTRR_C and CTRR_D.
- chainload refuses an invalid firmware RO range by its status, not by chip.

memory_fw_ro_range() itself stays T8140-only. Its fixed 48 KiB range at a
measured place is what the MMU and chainload placement depend on there, and
no other chip has been measured.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Pushed 1d9e9f1b, a fast-forward. It hasn't booted on any hardware yet.

  • Main merged (671c0cef) with no conflicts. Main added no new users of the monitor. The monitor, PSCI, NV2 and PV-KVM are fully gone from this repo and kept privately. No boot path hands the OS over at EL1.
  • apple_sysregs_unlocked split by what each use needs:
    • CYC_OVRD has its own cyc_ovrd flag, set for A7 to M3 only, so nothing writes it on M4 or later.
    • The M4+ CTRR reservations go by secondary_ctrr.
    • Setting RVBAR and the other pre-M4 registers go by legacy_impl_regs, which takes the old flag's place in the proxy ABI. The record grows from 20 to 24 bytes, and a new test checks the Python layout against the C struct.
    • The idle bootargs go by the absence of CYC_OVRD.
    • apple_sysregs_unlocked now means HACR, ACTLR and S3_5_C15_C6_2 are writable. It's true on M4 and later at EL2, and nothing gates on it.
    • Every old use keeps its truth table, except that an unknown CPU part no longer reads the CTRR registers. Nothing writes S3_5_C15_C6_2.
  • No chip list on the CTXR path. fbalazsy's RVBAR forward from J616s: boot Linux from a Mach-O boot object, with a fixed layout for hibernation #66 (5e4658c23a) is cherry-picked with his authorship and made generic. The CTXR A to C and CTRR reservations and the window check read the same registers on every chip. memory_fw_ro_range() stays T8140-only, because no other chip has been measured.
  • Tests: pytest 593 passed, 47 skipped, 0 failed. format-check and rustfmt-check pass. Both stages build from make clean with no new warnings.

Still untested on hardware:

  • the 128 MiB window and the refusal path
  • the RVBAR forward on any chip other than T6040
  • restart and power-off without PSCI
  • idle power

The Neo boot is next.

🤖 Generated with Claude Code

@sithglan

sithglan commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

J813/T8142 (M5 MacBook Air) boot result: the monitor-free chain works on hardware.

Tested this PR's head (1d9e9f1) together with the aurora-silicon/linux#985 kernel (feat/ctxr-execmem merged into the J813 stack as 6e2897803fd1) and the padded Stage 1 from aurora-silicon/aurora-installer#104.

Object and enrollment

  • Padded Stage 1: 153,190,400 bytes, __TEXT_EXEC 153,075,712 (0x91fc000), kcrz 153,141,248 (0x920c000) = 16 MiB aura slot + 2 MiB - 16 KiB + 128 MiB window; sha256 44f3d89389b6261621c69d028f69804117cdcdd064272bd7eb886ae8de059b92.
  • kmutil inspect -z -B ... --show-mach-boot-properties reports kcbz: 0 and kcrz 153141248; enrolled with kmutil configure-boot -c <obj> -v <system volume> --compress from the stub's 1TR.
  • Stage 1 and Stage 2 both report 1.0.0b5-g1d9e9f1b.
  • Stage 2: raw 4,063,232 bytes; the ESP m1n1/boot-ctxr.bin is 4,063,514 bytes with the appended boot=/chosen.bootargs=, sha256 0c2b2bc255d444d12ff852691e21c281e0fa1723fa10a9b2684223b0cc27041e; it loads aurora/boot-ctxr/{Image.gz, t8142-j813-desktop.dtb, initramfs.cpio.gz} with no maxcpus=1.
  • Kernel image_size 18,219,008, so it fits the 128 MiB window.

dmesg (7.3.0-aurora-j813-ctxr+ #13)

Linux version 7.3.0-aurora-j813-ctxr+ (sithglan@midi) (Debian clang version 22.1.8 (1+b2), Debian LLD 22.1.8) #13 SMP PREEMPT Fri Nov  6 23:09:46 CET 2026
ctxr-execmem: reserved exec window [0x0000010007600000, +128 MiB), enforcing
CPU features: detected: Virtualization Host Extensions
SMP: Total of 10 processors activated.
CPU: All CPU(s) started at EL2
execmem-confine: window [0x0000010007600000, 0x000001000f600000) (128 MiB), enforcing
ctxr-execmem: executable pool: 110 MiB usable in 1 range(s) of a 128 MiB window (17 MiB already reserved)
execmem-confine: boot verify: 576 executable mapping(s), 0 violation(s): PASS

The Stage 2 move worked as intended: the pool reports the 17 MiB image already reserved and the verifier finds all 576 EL1-executable mappings inside the window. NVMe root, Wi-Fi, GPU (aurora 1.0.0, M5 GPU firmware 25G76), AOP/audio and systemd all come up; no failed units.

Testing note: aura-try is only read on the chainload= path, so it can't replace the image for this boot= object — the marker stays armed and is silently ignored. I test kernels by swapping aurora/boot/Image.gz on the ESP. If that's intended, one line in the docs would save the next tester the confusion.

cc @Acelogic @DjDeveloperr @RyanTheTide @LiamRay10

apple_sysregs_unlocked comes from upstream m1n1, where 29073cc renamed
cyc_ovrd to it: false on M4 and later, and gating CYC_OVRD, RVBAR, the L2C
error status and SPRR/GXF. 89f2ff7 gave the name a new meaning, true on M4
and later, with no readers. A later upstream merge that adds a use of it
would then run on M4 and UNDEF there, and the proxyclient's emulate_sprr
default would flip on M4.

Go back to upstream's flag. Every use has main's truth table again,
cyc_ovrd, secondary_ctrr and legacy_impl_regs are gone, and the CPU features
record is 20 bytes again, the size every enrolled Stage 1 and current
proxyclient expects. The comment on the flag says what M4 and later can still
set at EL2. A test checks the proxyclient's layout against the C struct.

The CTXR changes keep their keys: RVBAR forwarding goes by the lock bit, and
the window's CTRR check runs where kboot reads CTRR, on parts without
apple_sysregs_unlocked.

Signed-off-by: Diljit Singh <hey@djdev.me>
1d9e9f1 added reads of CTXR_A to CTXR_C, CTRR_C and CTRR_D on T8140, in the
window check and the reservations. Nothing has read them on a Neo since, and
an unexpected UNDEF there would kill the boot. ctxr_range() now reads one
range under an exception guard, the way ctxr_exec_warn_unprobed() already
read CTXR_D, and every new read goes through it. A range whose read faults,
or whose LWR is 0, or whose UPR is below its LWR, counts as not armed in the
window check as well as in both reservation functions.

T8140's firmware RO range is reserved at either EL again, as main did. The
HV guest at EL1 doesn't need otherwise: only the CTRR reads are left to the
host there.

The J700 fixture now has CTXR_A to CTXR_C and CTRR_C inside the firmware RO
range, where the monitor's 2026-10-31 Neo boots relied on them being. It
gets no new reservation, which the low-window test now checks.

Signed-off-by: Diljit Singh <hey@djdev.me>
ctxr_exec_confined() waited for the probe with smp_wait(), up to 300 s, with
GUARD_RETURN | GUARD_SILENT set for every CPU the whole time. On a timeout it
called the machine not confined, but the call stayed in the CPU's slot, so
kboot_boot() refused the handoff anyway, after kboot_prepare_dt() had handed
devices over.

Wait 500 ms with smp_wait_timed(), far more than two instructions or a fault
take. The guard comes off on every path, as before. When the call is still
pending, whether its acknowledgement or its completion timed out,
ctxr_exec_os_may_start() says which CPU holds it and refuses at once, before
kboot_prepare_dt() touches any device. A probe that couldn't be sent still
counts as not confined. Neither is sent again.

The stuck-probe test now models a completion timeout and an unacknowledged
call. It checks the wait is between 100 and 1000 ms, that the guard is off
after, and that the boot stops with no Recovery advice. The docs say what
happens.

Signed-off-by: Diljit Singh <hey@djdev.me>
…-off

- Aura does read a CTXR register on M1 to M3: CTXR_D, under an exception
  guard, when it has no secondary to probe.
- Power-off never worked under the monitor. main's psci.c answered both
  SYSTEM_OFF and SYSTEM_RESET with reboot().
- Aura can't tell whether a kernel was built with CONFIG_ARM64_CTXR_EXECMEM
  and moves one without it all the same.
- On T8140, T6041, T8152 and the J713 the heap copy of a moved kernel stays
  reserved with the retained heap. That is about 1 MiB for U-Boot and about
  40 MiB for a kernel booted with boot=. Releasing it would mean editing the
  memreserve entries after fdt_pack(), so it is only written down here and
  next to kboot_move_to_exec_window().
- Windows A to C on T8140 are left to the firmware RO range's reservation,
  which is what the monitor's 2026-10-31 Neo boots relied on. The text now
  says where that comes from instead of claiming the tree can't change.

Signed-off-by: Diljit Singh <hey@djdev.me>
Stage 1 built with Homebrew clang 23 and the documented flags stopped linking
("Code and data reach the DCP log buffer after stage 1!"). aura-raw.ld aligns
each section to 64 KiB, and 1d9e9f1's .text ended 216 bytes short of a
boundary. This round's 308 bytes moved .text past it, and with it everything
after, to 0x12aa90 against the 0x120000 limit. CI and release build with gcc,
so they never saw it.

Take back enough to end below the boundary again:

- one loop over the five ranges in the window check
- one loop for CTRR_C and CTRR_D in kboot, without main's warning for a
  CTRR_C larger than 48 KiB, which only logged
- the stuck-probe message printed once where the probe fails, instead of in
  a second block in ctxr_exec_os_may_start()

.text now ends 16 bytes short of the boundary, so the next few dozen bytes
of code on this toolchain will move it again.

Signed-off-by: Diljit Singh <hey@djdev.me>
A probe stops the boot in two ways. smp_call4() gives the secondary 1000 ms
to take the call, and then smp_wait_timed() gives it 500 ms to answer. The
panel only named the 500 ms. Say that the CPU didn't take or didn't answer the
probe, and put both times in the docs and next to the timeout.

Signed-off-by: Diljit Singh <hey@djdev.me>
The window check and kboot's CTXR_D reservation still read CTXR_D directly. A
confined machine has the register, but read it under the same guard as the
others anyway. A fault there now reads as nothing armed, so the window is
refused, instead of ending the boot.

ctxr_range() is declared noinline. clang folded the constant CTXR_D calls
into inline copies of the guarded read, which cost Stage 1 the room it has
left under the DCP log buffer.

The window harness faults one chosen register now, so a CTRR fault and a
CTXR_D fault can be told apart.

Signed-off-by: Diljit Singh <hey@djdev.me>
dt_reserve_secondary_ro() now holds T8140's firmware RO range, the EL1 case
and the CTRR loop that kboot_prepare_dt() used to chain, so one harness can
drive all of it. At EL1 it now says what it leaves to the host: CTRR_D on
T8140, whose firmware RO range it has reserved by then, and CTRR_C and
CTRR_D elsewhere. Before, it said the whole SMP-RO range right after
reserving part of it.

tests/python/test_secondary_ro_reservations.py runs it with the production
ctxr_range() for T8140, another M4-or-later chip and M1 to M3, at EL1 and
EL2. Against main the reservations differ in three ways, all since 1d9e9f1,
which the test spells out:

- T8140 at EL2 also reserves CTRR_D when it is armed. It read 0 on the Neo
  on 2026-10-31.
- Other M4-and-later chips reserve CTRR_C to UPR + 4K, like every other
  range, where main stopped at UPR: 0xc000 bytes instead of 0xb000 for a
  48 KiB range.
- A CTRR_C that reads as empty or wrapped gets no entry. Main added a
  zero-size or wrapped one.

Signed-off-by: Diljit Singh <hey@djdev.me>
aura-raw.ld puts each section on a 64 KiB boundary, and Stage 1 has to end
below the panel DCP's log buffer. Built with Homebrew clang 23.1.1 and Rust
1.98.1, Stage 1's .text now ends 68 bytes short of its next boundary. Past
it, the link fails. Releases build with gcc, so only someone building on a
Mac found out.

There is no clean way to win back real room here. The C has no dead code
left to cut, and building the Rust with opt-level "s" would change code on
every chip without a hardware test. So link both stages with clang on a
macos-26 runner, with GNU make 4 from Homebrew, which runs the grouped
config rule once under -j, and let CI fail first.

Signed-off-by: Diljit Singh <hey@djdev.me>
- The window check skips CTRR_C and CTRR_D on a chip with unlocked Apple
  system registers (M1 to M3), as kboot does, so "every chip is checked
  against all of them" now says so.
- The apple_sysregs_unlocked comment said SPRR/GXF is A7 to M3 only, against
  the boot object notes, which say GXF is writable with a kcbz-0 object on
  M4. The flag covers aura's own use of SPRR/GXF, in the hypervisor. The
  comment now says GXF is writable there too, and that aura doesn't use it.

Signed-off-by: Diljit Singh <hey@djdev.me>
A kernel outside the window now logs why and boots on the boot CPU instead
of panicking, U-Boot warns and starts a zboot kernel instead of refusing it,
and kexec is refused whenever a window is active.

Signed-off-by: Diljit Singh <hey@djdev.me>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Neo (J700) boot with this head: the padded cda450d Stage 1 is enrolled, and Stage 2 cda450d (in m1n1/boot-ctxr.bin) with u-boot#32 50d879ac started linux#985 at EL2 on all 6 CPUs inside a 128 MiB window. Boot verify passed, and the stress and escape run passed. Details: aurora-silicon/linux#985 (comment)

🤖 Generated with Claude Code

@DjDeveloperr

DjDeveloperr commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the Neo run on cda450d. Since my last comment, head is now d5615078, all fast-forward:

From an independent review (e8234f44):

  • apple_sysregs_unlocked keeps upstream's name and meaning, which upstream 29073cce gave it by renaming cyc_ovrd. That way an upstream merge can't flip it.
    • It's false from M4 on only because it gates CYC_OVRD, which is locked there, and the M1–M3-only setup that comes with it.
    • On M4 and later at EL2 only CYC_OVRD is locked. HACR_EL2, the ACTLR bits and the per-core deep-sleep enable S3_5_C15_C6_2 are all writable.
    • Nothing in aura, the kernel or U-Boot touches CYC_OVRD on M4 or later.
    • The extra flags from the split are gone, every use has main's truth table, and the proxy CPU-features record is back to 20 bytes.
  • Probe timing: it now waits 500 ms instead of 300 s. A secondary that doesn't take the probe, or doesn't answer it, stops the boot before any device is handed over, with the reason on the panel. The guard is cleared on every path.
  • Guarded reads: CTXR_A to C and CTRR are read through one guarded reader that treats LWR 0 as unarmed. T8140's firmware RO range is reserved at either EL again, as on main.

From the re-review (cda450df):

  • CTXR_D now goes through the same reader.
  • The stuck-probe text covers both limits: 1 s to take the call, 500 ms to answer it.
  • The EL1 log line says what's left to the host.
  • The SMP-RO logic is one function, with a matrix test of T8140, other M4 chips and M1 to M3 at EL1 and EL2. Its three differences from main are spelled out: an armed CTRR_D on T8140 at EL2, CTRR_C to UPR + 4K, and no zero-size entry.
  • A new stage1-clang CI job links both stages with clang. With clang, Stage 1's .text is 68 bytes under its 64 KiB step.

d5615078: the docs now follow the kernel as merged. A kernel outside the window boots on CPU0 instead of panicking, and kexec is refused whenever a window is active.

CI 10/10.

🤖 Generated with Claude Code

DjDeveloperr and others added 3 commits November 7, 2026 00:46
HACR_EL2, the ACTLR bits and the per-core deep-sleep enable are writable at EL2 there; the flag stays false only because it gates CYC_OVRD.

Signed-off-by: Diljit Singh <hey@djdev.me>
Aurora agent policy: remove assistant instruction files that are
not copies of .github/aurora/AGENTS.md on main, and keep the copies on policy
branches identical to it.
Signed-off-by: Diljit Singh <hey@djdev.me>
DjDeveloperr added a commit that referenced this pull request Nov 7, 2026
Take aura#77, which removes the resident EL2 monitor, PSCI, NV2 and
PV-KVM from the Linux boot and runs Linux at EL2 in the CTXR window.

Resolutions:
- monitor.c, pv_kvm.c: deleted, as in #77. This branch's NV2 already
  has its own EL2 setup and entry (nv2_el2_setup, nv2_enter_el1h) and
  used neither the monitor nor PV-KVM.
- nv2.c, nv2.h, nv2_asm.S: kept from this branch. They are built only
  with SPTM_VIRT=1.
- exception.c: #77, plus this branch's SPTM_VIRT hooks. The UNDEF
  reflection on the monitor's path goes, with monitor_active. The
  proxy's hvc #1 case is SPTM_VIRT-only as a whole.
- exception_asm.S: #77's, with no lower-EL ERET gate. Until a later
  commit puts the gate back for SPTM_VIRT, the guest's ERETs take
  nv2_eret's C path, which handles all of them.
- Makefile: #77's object list plus SPTM_VIRT. NV2_TRACE and
  NV2_TRACE_PRINT only mean something inside SPTM_VIRT.
- boot_options, payload.c, main.c, kboot.c, docs/boot-options.md and
  their tests: #77's. This branch only removed NV2 host mode from them,
  and #77 removes it along with PV-KVM.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr
DjDeveloperr merged commit 3616f9c into main Nov 7, 2026
10 checks passed
DjDeveloperr added a commit that referenced this pull request Nov 8, 2026
Takes main's 19 commits since #77 (1.0.0b6 to 1.0.0b9 and the M4
firmware gates). Only the linker scripts conflicted: main moved hv_vm
and hv_sprr into the .init slot as this branch did, which also puts
sptm_virt*.o there.

Signed-off-by: Diljit Singh <hey@djdev.me>
@RyanTheTide RyanTheTide mentioned this pull request Nov 9, 2026
@RyanTheTide
RyanTheTide deleted the feat/ctxr-exec-window branch November 16, 2026 05:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants