Skip to content

J616s: boot Linux from a Mach-O boot object, with a fixed layout for hibernation - #66

Closed
fbalazsy wants to merge 13 commits into
aurora-silicon:mainfrom
fbalazsy:j616s/hibernation-layout
Closed

fbalazsy wants to merge 13 commits into
aurora-silicon:mainfrom
fbalazsy:j616s/hibernation-layout

Conversation

@fbalazsy

@fbalazsy fbalazsy commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

This lets the M4 Pro MacBook Pro (J616s, T6040) boot Linux from a Mach-O boot object that carries the kernel, and gives Linux a memory layout that stays the same from boot to boot, so that hibernation (S4) can resume. Everything new is gated on T6040, and the hibernation layout also needs the hib=1 configuration line. Nothing changes on other chips, or on a J616s without hib=1.

What it does

Boot from the boot object:

  • boot_storage: declare rust_load_image() with its trial arguments. boot_storage.c still declared the four-argument prototype from before aura-try, so its call left the last two arguments undefined. This one is not J616s-specific.
  • payload: end the payload scan at a boot object's prelink information. When payloads are appended to aura.bin before it is wrapped, the scan ran into the __PRELINK_INFO XML that tools/macho_boot_object.py puts after the raw image, and set config_lost.
  • firmware: J616s reports 26.6.2 OS firmware as mBoot-18000.161.10.701.1. A J616s whose 26.6.2 OS was installed from Recovery reports this OS iBoot, which the table lists for 26.7. The 26.6.2 gates were then off, and the GPU firmware and the display handoff were refused.
  • payload, kboot: hand the J616s ANS session to Linux from an inline payload. Start ANS for an inline kernel as the J713 path does, keep it for Linux, and reserve the retained heap and the SART-allowed ANS buffers on T6040 as on T6041.
  • kboot: hand the PMP tunables and boot arguments over on J616s. The J616s tree has no pmp alias. Find the node by compatible and pass dram-channel-disable, pmc-msg-disabled and soc-chip-variant.
  • kboot: give Linux only the J616s RAM that iBoot left free. Build /memory from DRAM minus the ADT carveouts, as the J616s m4-loader does. Keep what is below this stage. Keep 8 MiB of heap headroom for the allocations made while the tree is being built, and refuse the boot if they need more.
  • smp: forward a T6040 RVBAR locked at the boot object's load base. If RVBAR is locked at the load base (the Mach-O header, 64 KiB below the vectors), replace the Mach-O magic word with a branch to the vectors once.
  • kboot: keep CTXR A to C out of Linux on T6040 under the monitor. T6040 has no firmware RO range that covers these windows, as T8140 has. Describe them as a no-map reserved-memory node.
  • payload: copy an inline initramfs out of the boot object under the monitor. The initramfs inside the boot object lies in CTXR_D. Linux frees it after unpacking, and a secondary that later executes from one of those pages aborts to EL2. Hand Linux a copy from the heap instead.

Hibernation:

  • nv2: enter CPU_ON with the L1 translation regime off. After PSCI CPU_OFF, the EL1 registers that an NV2 guest uses as its EL2 still held the OS's MMU-on state. The next CPU_ON was then fetched through stale tables and the core never came back. Hibernation takes every secondary down this way and brings it back. monitor_enter_os() already resets this on the plain EL1 path; nv2_enter_os() did not.

  • psci: halt with the watchdog stopped on SYSTEM_OFF. Linux powers off through macsmc and calls SYSTEM_OFF only when that did not take effect. The monitor answered with a reset, which boots straight back into the OS and resumes an image that was just written. It now halts the calling CPU, after stopping the primary watchdog. SYSTEM_RESET still resets.

  • kboot: fixed T6040 memory layout for hibernation (hib=1). Where iBoot loads the boot object, and what it places after it, moves between boots, so the reserved ranges, the heap and the RAM above it moved with it. With hib=1:

    • the heap restarts 160 MiB above the 32 MiB aligned start of RAM;
    • RAM up to 128 MiB above that is kept from Linux as the retained heap, and region-id-45 is handed back from that end on;
    • the device tree is copied, as the last step before the handoff, into a 2 MiB slot at the start of that heap.

    The layout is refused, with nothing changed, on other chips or when the object or the heap already lies above the fixed heap.

  • docs: booting the J616s from a boot object, and hib=1 in docs/macho-boot-object.md.

Booting it on a J616s

Build with make NV2_BOOT=1. That is the configuration tested on hardware; a plain build enters Linux at EL1, which has not been booted on the J616s with this series. Append to build/aura.bin, in this order:

  1. the J616s device tree;
  2. the initramfs (aura_initramfs, its 32-bit little-endian size, the cpio);
  3. the configuration lines, for example:
    chosen.bootargs=root=UUID=<root> rw resume=UUID=<swap fs> resume_offset=<offset> ...
    hib=1
    
  4. the kernel Image.

Then wrap it with tools/macho_boot_object.py and enroll it as in docs/macho-boot-object.md. Hibernation needs every boot to use the same boot object, because the image must be resumed by the kernel that wrote it.

Linux side

The Linux series is aurora-silicon/linux#906. Its aura-hibernate driver:

  • registers every /memreserve/ range and every non-reusable /reserved-memory range as nosave, but only when /chosen/aura,hibernate-layout (two u64: start and end of the RAM kept from Linux) is present. Without the property, hibernation and resume are refused.
  • uses /chosen/aura,hibernate-dt (u64, the device tree slot), when the kernel was started with its tree there, to read the current boot's firmware regions after a resume (the AOP log segment moves between boots).

The driver changes for the devices themselves (NVMe/ANS, DCP, SMC, Type-C, MTP) are in that series, not here.

Tests

  • New tests/python/test_t6040_hibernate_layout.py: runs kboot_set_hibernate_layout(), dt_t6040_memory_holes() and heapblock_raise_base() on a modeled J616s carveout map. It checks the same layout for two object placements, region-id-45 handed back from the fixed end, guard pages dropped, a heap past the end refused, the unchanged map without hib=1, and each refusal.
  • New tests/python/test_smp_t6040_load_base.py: the load-base RVBAR relay writes once, remaps RW then RX, accepts its branch on the next core, and refuses foreign content, another distance or another chip.
  • Existing host harnesses that copy the touched functions get the stubs and defines they now need.
  • Each commit builds with make NV2_BOOT=1 (clang) and with make USE_CLANG=0 ARCH=aarch64-linux-gnu- CHAINLOADING=1 (gcc). The -Wstack-usage=2048 warnings are the same as on main.
  • make format-check (clang-format 20.1.8) and make rustfmt-check pass.
  • pytest tests/python tests/test_t8122_gpu_reader.py tests/test_t8122_gpu_resources.py, without USE_CLANG: 609 passed, 8 skipped, 1 xfailed, 2 failed. The two test_loadobjs.py failures compare nm output formats and fail the same way on main on this host. With USE_CLANG=1: 610 passed, 9 skipped, 1 xfailed.

Hardware

MacBook Pro 16" M4 Pro (J616s), NV2_BOOT=1, configuration hib=1 (and tso=1, which this series does not change), swap file resume. Peripherals: a monitor on HDMI and on USB-C DisplayPort, and a USB-C hub with Ethernet.

Linux Cycle Mode Result
early Linux series (no SMC hand-over) first in a boot shutdown, power key resumed; SMC session lost (keyboard, trackpad, battery gone); audio back
later Linux series (HDMI converter not powered again) first in a boot shutdown, power key everything back except HDMI
later Linux series first in a boot shutdown, power key everything back: HDMI, USB-C DP, internal panel, Ethernet, Wi-Fi, audio, keyboard and trackpad
same second in the same boot reboot the Mac stayed off after writing the image, and the power key was needed; then resumed
Linux series as submitted (aurora-silicon/linux#906) first in a boot shutdown, power key everything back, including the GPU desktop and all four USB-C port controllers in place
same second in the same boot shutdown, power key everything back as above

The remaining device issues in these runs are on the Linux side. These runs used this series as an earlier branch on top of an earlier head of #33. The last two rows used this series on main before #65 (final #33, the probe on every chip from #62, the ANE cold-boot marker); the branch was then rebased onto current main (#65 only changes the debug UART console) and built and tested again, not booted again.

Not done / open

  • Hibernation in reboot mode left the Mac off. Linux then restarts through kernel_restart(), which ends in PSCI SYSTEM_RESET and aura's reboot(), or in the SMC reboot path. The Mac stayed off and needed the power key. We do not know why. Advice is welcome.
  • hib=1 is only implemented for T6040. The 160 MiB window and the 128 MiB heap span are sized from the placements and heap use seen on the J616s.
  • The EL1 entry without NV2 has not been booted on the J616s with this series.
  • The RVBAR relay is only taken when RVBAR is locked at the load base. These hardware runs do not show whether it was.
  • In the hardware runs the kernel command line also had retain_initrd, so the boots do not show on their own that the initramfs copy is needed.
  • KVM guests do not survive a hibernation cycle; this series does not try.

chainload.c declares and calls rust_load_image() with the trial path
buffer and its capacity that the Rust side takes since the one-shot
aura-try boot. boot_storage.c kept the old four-argument prototype, so
its call left the last two arguments to whatever the registers held.
Declare the same prototype and pass no trial buffer.

The host test's stub takes the same arguments.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
tools/macho_boot_object.py places __PRELINK_INFO right after the raw
image. When payloads are appended to the raw image before it is wrapped,
the scan reaches that XML after the last payload, reads it as an unknown
payload and sets config_lost. Treat "<?xml" as the end of the payloads,
as the empty words and the SEP firmware after the image already are.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
A J616s (T6040, board 6) whose 26.6.2 OS was installed from Recovery
reports the OS iBoot mBoot-18000.161.10.701.1, which the version table
lists for 26.7. The GPU and DCP firmware it loads are the 26.6.2 images,
but every 26.6.2 gate was off, so kboot refused the GPU firmware and the
display handoff. On this board, take that string as 26.6.2. The Linux
drivers still check the loaded images themselves.

The firmware detection host test defines the chip and board it reads.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
…yload

A kernel carried inline in the boot object does not go through
boot_storage_load(), so on T6040 nothing started ANS before kboot and
Linux found the session iBoot had left instead of one it could adopt.
Start it before kboot as the J713 path does, keep it running for Linux
after a successful handoff, and shut it down when the handoff fails.

The retained heap and the SART-allowed ANS buffers were reserved for
T6041 only. The M4 Pro MacBook Pro is T6040 with the same storage
handoff, so reserve them there as well.

The payload host test defines T6040.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
The J616s device tree has no pmp alias, so dt_set_pmp() found no node
and handed nothing over. Find the node by its compatible on T6040, pass
the optional PMP boot arguments the ADT carries (dram-channel-disable,
pmc-msg-disabled, soc-chip-variant) as apple,<name> properties for the
Linux PMP driver, and leave the node's status as the tree has it.

The tunables and arguments take about 38K of the tree, so give a T6040
tree 64K more room, as T6030 and T6031 get.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
On T6040, build /memory from the DRAM range minus every carveout in
/chosen/carveout-memory-map, as the J616s m4-loader does. region-id-24
is the usable RAM itself and stays in. region-id-45 holds this stage and
its heap, so only its part above the heap high water is left out. The
single 16K pages left between carveouts are guards and are dropped.

Keep the memory below this stage as T8152 does: it holds what iBoot
loaded before the boot object.

kboot reserves the retained heap up to its high water before it builds
the tree, and dt_set_memory() hands Linux what lies above the high water
it sees later. Run against a J616s ADT, the range between the two was
about 224 KiB that nothing reserved. Keep 8 MiB more of the heap and
refuse to boot if building the tree took more than that.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
A Mach-O boot object puts the raw image 64 KiB after its load base. An
iBoot that does not take the vector address from the object may lock
the secondaries' RVBAR at the load base, which holds the Mach-O header,
and smp_prepare_rvbar() then refused every secondary on T6040. When the
locked address is exactly 64 KiB below the vectors and still holds the
Mach-O magic, replace that word with a branch to the vectors, and accept
the branch on the next secondary. Any other content or distance is
refused as before.

tests/python/test_smp_t6040_load_base.py runs the production function
against modeled MMIO: one write and an RW/RX remap on the first core,
none on the next, refusal of foreign content, another distance or chip.
The existing RVBAR harnesses define T6040 and SZ_16K.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
When the monitor runs the OS at EL1, kboot reserves CTXR_D. On T8140
windows A to C lie inside the firmware RO range reserved anyway; T6040
has no such range, and the three 16K windows the secondaries enforce
were left in Linux's map. Read their bounds on the boot CPU, which
reports the same windows with enforcement off, and describe them as a
no-map /reserved-memory node when they are three contiguous aligned
pages inside RAM. Otherwise log the bounds and reserve nothing.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
…nitor

When the monitor runs the OS at EL1, the boot object lies in CTXR_D. An
initramfs appended to it is handed to Linux in place; Linux frees it
after unpacking, the pages reach the page allocator, and the first
secondary that executes from one of them aborts straight to EL2. On the
J616s this showed as a boot that reached systemd from the initramfs and
died on the first binaries read from disk.

Remember an initramfs found in the payload area (from _payload_start up
to top_of_kernel_data, where the heap begins), and before the device tree
is prepared, when the monitor is required, hand Linux a copy from the
heap instead. An initramfs loaded from storage that replaced it is left
alone, and the copy is taken once.

The payload host test stubs the monitor and the payload bounds.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
Under NV2 the OS runs its EL2 on the EL1 registers. After a PSCI CPU_OFF
the core parks again while those registers still hold the OS's MMU-on
state, its TTBRs and vectors, and nv2_enter_os() does not reset them as
monitor_enter_os() does on the plain EL1 path. The next CPU_ON entry was
then fetched through stale tables and the core never reached the OS.
Clear SCTLR_EL12 before entering, which is what the first CPU_ON finds.

Hibernation takes every secondary down and brings it back this way.
On the J616s the secondaries came back after the fix and did not before.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
Linux powers the Mac off through the system management controller and
calls PSCI SYSTEM_OFF only when that did not take effect. The monitor
answered it with a reset, which boots straight back into the OS and,
after a hibernation image was written, resumes that image at once
instead of staying off. Halt the calling CPU in WFI instead, with a
message on the console; Linux has stopped the other CPUs before it gets
here.

A watchdog still running, such as the one a trial boot arms, would reset
into the OS later anyway. Stop the primary watchdog first, through the
base wdt_disable() cached at boot, so nothing walks the ADT at runtime.

SYSTEM_RESET still resets.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
A hibernation image restores every saved page to the frame it came
from, so the boot that resumes it must give Linux the same RAM and the
same reserved ranges as the boot that wrote it. On the J616s they move:
where iBoot loads the boot object, and what it places after it, changes
between boots, and with it the heap holding the kernel, initramfs and
device tree copies, and the end of the RAM handed back above the heap.

With the payload configuration line hib=1 on T6040:

- The heap restarts at a fixed address, 160 MiB above the 32 MiB aligned
  start of RAM, above every place the object has been seen to reach.
  heapblock_raise_base() moves the next allocation up; earlier
  allocations stay where they are.
- Everything from the start of RAM to a fixed end 128 MiB above that is
  kept from Linux as the retained heap, and region-id-45 is handed back
  from that end on. The boot is refused if the heap comes within the
  usual 8 MiB margin of the end.
- The device tree Linux is started with is copied, as the last step
  before the handoff, into a 2 MiB slot at the start of the fixed heap,
  below everything the heap hands out. A kernel restored from an image
  was started with an earlier boot's tree; the slot is in the range the
  image does not save, so after a resume that kernel finds this boot's
  tree at the address it was started with. Firmware segments that move
  between boots, such as the AOP log, are only described there.
- /chosen/aura,hibernate-layout holds the start and end of the range
  kept from Linux and /chosen/aura,hibernate-dt the address of the slot.

The layout is refused, with a message and nothing changed, on other
chips, with an invalid RAM extent, or when the object or the heap
already lies above the fixed heap. Without hib=1 nothing changes.

tests/python/test_t6040_hibernate_layout.py runs the layout and the RAM
map builder on a modeled J616s carveout map: the same layout for two
object placements, region-id-45 handed back from the fixed end, guard
pages dropped, a heap past the end refused, the unchanged map without
hib=1, and each refusal. The SMP handoff gate harness stubs the
hibernation state it now reads in kboot_boot().

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
Describe the payload order for a J616s boot object that carries the
kernel, what kboot does for it under the monitor, and the fixed
hibernation layout with the two /chosen properties Linux reads.

Signed-off-by: František Balázsy <frantisekbalazsy@gmail.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Merge-train review (M4 Pro, review + host build only for now). Thanks for this, it's a big step for the boot-object path. Sign-offs are all present; the head and a trial merge with main both build clean (ESP_STAGE2=1 CHAINLOADING=1). Blocking items before we can trial it on our J616s:

  1. Gating is per chip, not J616s/boot-object. The kboot/payload changes test chip_id == T6040, so they also change today's raw-mode J616s Stage 2 and the J614s (merged J614s handoff): the carveout-map RAM holes (bails if /chosen/carveout-memory-map is missing), the extra 8 MiB of kept heap, the below-stage reservation, the new heap-growth bail, the PMP node fallback/boot args, nvme_init + ANS kept running in payload.c. Please gate them on the J616s boot-object/hib=1 path (or on apple,j616s at least).
  2. firmware.c: forcing mBoot-18000.161.10.701.1 to 26.6.2 on the J616s. The table lists that iBoot string for 26.7 too, so a J616s really on 26.7 would be handed over as 26.6.2. The version needs to come from something the boot reads (ADT/firmware image), not a fixed override.
  3. Host test regression: tests/python/test_smp_handoff_gate.py::test_pending_calls_block_dt_preparation_and_kernel_publication fails on this branch (passes on main): the new hib printf/bail use %lx with u64, and the test model's u64 is uint64_t (unsigned long long on macOS) -> -Werror,-Wformat.
  4. Conflict with main in src/kboot.c (the ANS SART reservation condition vs the J614s change): taking chip_id == T6040 covers both, but please rebase.

Non-blocking, worth addressing or noting in the PR:

  • psci.c SYSTEM_OFF now halts and stops the watchdog on every Mac running the monitor (J700 included), which defeats the trial-boot dead-man; please gate it.
  • nv2.c writes SCTLR_EL12 = 0 on every NV2 entry rather than the RES1-correct MMU-off value, without a TLB invalidate.
  • payload.c copies an inline initramfs to the heap on every monitor Mac; large initramfs could hit the heapblock limit.
  • kboot.c region-45 carve: if the heap high-water is above the region end, start > end gives overlapping /memory ranges (if (start >= end) continue;).
  • A comment mentions a "J616s m4-loader" that isn't in the tree; please reword.
  • The boot_storage.c rust_load_image() argument fix is a real bug fix on every chip; it would be great as its own small PR so it can land right away.

Once 1-4 are addressed we'll run it as an aura-try one-shot Stage 2 on our J616s (raw mode; we never write Stage 1), so the boot-object path itself stays yours to prove.

🤖 Generated with Claude Code

@fbalazsy

Copy link
Copy Markdown
Contributor Author

Closing as superseded: since #77 there is no monitor, so the Mach-O boot object, PSCI and NV2 changes here no longer apply. The J616s EL2 handoff (hib=1 fixed layout, S2R windows) follows as a new PR on current main, with your review points (per-board gating, no fixed firmware-version override, host test, rebase) taken into account. Thanks for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants