Skip to content

drm/apple-g5p: Bound the kexec handover and fix two lost-GPU races - #125

Closed
brkosebay wants to merge 79 commits into
OrchardLinux:mainfrom
brkosebay:j71t/gpu-handover
Closed

brkosebay wants to merge 79 commits into
OrchardLinux:mainfrom
brkosebay:j71t/gpu-handover

Conversation

@brkosebay

@brkosebay brkosebay commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 11 of 11 of the iPad 5 (J71t) series in this repository; merge after #120.

Problem

A warm restart could jump into the next kernel while the GPU still had DMA in flight, or hang in an unbounded wait for the firmware to stop. Two races could also lose the GPU: with two jobs in flight, the previous job's 3D completion could make a new job's 3D phase look retired, so its doorbell was never rung and the job was lost after a silent second; and when the firmware's ALIVE overtook the GPU cores' power-on request, the busy request was counted as a failed power-on and the boot fell back to software rendering.

Change

Bound and serialise the GPU handover, accept an already completed handover, keep halt and poweroff working after a quiesce failure, and flush the consoles before an emergency cold reset. Ring a render phase's doorbell unless the phase's own stamp shows it retired, as Apple's TA and 3D submit paths ring after every ring entry. When ALIVE overtakes a power-on request that is still in flight, let the first-ALIVE handler wait for the request's busy bit to clear within the power-on bound, as Apple's power-state code does, before any GPU register write.

Testing

  • Hardware: iPad 5 Wi-Fi (J71t). Hands-on testing on 2026-11-11 with an integrated build of this series, made before the final failure-path fixes, covered warm restart with the GPU in use.
  • Not covered on hardware: the cold-reset escape when the handover fails. The power-on race is rare (twice in a day of boots before the fix), so the fix is not yet shown on hardware.
  • 2026-11-12, doorbell fix: glmark2 terrain ran 20 of 20 and then 5 of 5 times without a lost job (before the fix, 2 of about 11 runs lost a 3D job). 20+ warm restarts with the GPU in use, several right after a glmark2 run, completed without a freeze.
  • Host: the Linux test suite passes at this branch's head, and the kernel and m1n1 patch series apply on apple-16k and apple-4k. At the top of the series the applied kernel tree differs from the built release image only where main took earlier PRs of this series in reworked form, by the closed perf: apple: A9 memory port byte counters #123, and by the new G5P sleep refusal (drivers/drm 0045, not yet run on the device).
  • Other boards that share these drivers or compatibles have not been tested.

Dependencies

Depends on #120.

This branch is stacked on the PRs it depends on and carries their commits; the change of this PR is the commit "drm/apple-g5p: Bound the kexec handover and fix two lost-GPU races", and the merge commits after it bring the branch onto current main.

This update keeps the branch's history: it merges current main and the updated branch below it into this branch. Nothing is forced.

🤖 Generated with Claude Code

Add the A9 (S8003) G5P render driver used by the merged PowerVR/G5P Mesa
winsys as kernel patches, enable it on J71t in the device tree, set
CONFIG_DRM_APPLE_G5P=y for apple-16k, and add an install-time recipe that
extracts the original META LDR (apple/agx-g5p-c0.ldr) from the device's own
iOS kernelcache into /lib/firmware. No firmware is committed.

The driver provides TYPED_V3 rendering, GEM buffers, VM/arena mappings,
PRIME import/export, syncobj, bounded retirement and recovery. It does not
expose the old compute or fixed-fixture ABI.

Hardware: the exact kernel this derives from (brkosebay/linux a9/gpu-pr1
407b7bf2cbbe) booted on a J71t with the merged Mesa, rendered labwc on the
GPU with KMS scanout at 60 Hz, and passed the G5P device rungs with zero GPU
events. This Orchard adaptation itself has not yet been built by orchard or
run on hardware.

Assisted-by: LLM
kernel/configs/apple-16k/config:
CONFIG_DRM_APPLE_G5P=y
These options provide g5p and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

kernel/configs/apple-4k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
CONFIG_APPLE_AOP_GYRO=m
CONFIG_APPLE_AOP_PRESSURE=m
These options provide aop and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

kernel/configs/apple-4k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
CONFIG_APPLE_AOP_GYRO=m
CONFIG_APPLE_AOP_PRESSURE=m
These options provide aop and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

kernel/configs/apple-4k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
The VXD decoder only exposed H.264, so HEVC video was decoded on the
CPU.

Prepare the driver for a second coded format, add the HEVC
command-stream builder and the stateless HEVC decoder, enable it on the
S8000 and T8010 variants, and require a proven firmware stop before
kexec. Adds the video_decode_hevc feature (working on J71t, untested on
the other boards that have the driver, missing elsewhere).

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
S8000/S8003 left the SoC and memory performance state at the boot
default, which bounds GPU throughput by memory bandwidth.

Describe the SoC performance state in the binding and DT, run the SoC
and memory at the high state while active, and restore the low state for
sleep.

Assisted-by: LLM
A Mac grants an iPad more than the standard USB current, but the charger
took the gadget's reported budget unchecked and the composite gadget
could draw the extra current while the bus was suspended.

Cap and log the USB host budget in the TriStar charger, draw no extra
Apple current while suspended, and let the initramfs and orchard USB
helpers opt J71t into the Mac current profile. Documented in
docs/bring-up/charging/j71t-mac-current.md, with tests.

Assisted-by: LLM
There was no way to measure memory bandwidth per port on A9, which is
needed to tell GPU work from memory stalls.

Add a binding, DT node and perf driver for the A9 AMC port monitor,
exposing CPU and GFX read/write byte counters, and enable it for
apple-16k.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, drivers/base and drivers/usb
are renumbered to follow main's.
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, drivers/base,
drivers/pmdomain and drivers/usb are renumbered to follow main's.
The A9 AOP driver on main reads apple,aop-target, apple,chip-id and
apple,chip-revision, and the motion sensor drivers read their factory
calibration, from properties the bootloader fills in. Nothing on main
fills them in; going by the driver's probe, the A9 AOP cannot start
without them (read from the code, not run). This is the m1n1 patch that
fills them in, from OrchardLinux#101, unchanged.

Assisted-by: LLM
From OrchardLinux#101, the part that main did not take with the sensors: the
retained AOP reload across kexec (checksummed shutdown log, power-state
observer, pr_flush export, warm_state attribute). The sensor parts that
main carries are left out of these patches; what remains applies on
top of main's AOP driver and its fresh-start order.

Assisted-by: LLM
From OrchardLinux#110, the part main did not take with the stable USB serial: the
platform layer embeds each supported board's pristine DTB and a
compatible-to-file map in the initrd. Before switching root, the initrd
copies the running board's tree to /run/orchard-kexec/pristine.dtb, or,
after a warm restart, the retained tree the previous kernel named with
orchard_warm.dtb=, and records the boot ID next to it. Warm restart
boots the next kernel with that tree, never with one rebuilt from an
updated /boot.

No feature table is added: whether warm restart is offered is decided
from the mechanisms it uses, not from a feature state.

Assisted-by: LLM
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, drivers/base,
drivers/pmdomain and drivers/usb are renumbered to follow main's.
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, devices/t8010, drivers/base,
drivers/media, drivers/pmdomain, drivers/power and drivers/usb are
renumbered to follow main's.

Main's versions apply first now, so 0011 in drivers/media keep their
change with a diff regenerated for the new context.
From OrchardLinux#101, the suspend precheck that main did not take with the
sensors. It belongs with suspend: it makes the AOP refuse a pre-sync
system sleep while its firmware is starting or stopping.

Assisted-by: LLM
From OrchardLinux#121, the patch main did not take with HEVC. It uses the VXD
sleep stop and sleep gate this branch adds, so it sits here rather
than with the earlier warm restart branch: a failed firmware stop keeps
the firmware, command buffers and power, and kexec takes the cold-reset
path before those pages become the next kernel's RAM.

Assisted-by: LLM
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, devices/s8003,
devices/t8010, drivers/base, drivers/media, drivers/pmdomain,
drivers/power and drivers/usb are renumbered to follow main's.

Main's versions apply first now, so 0011 in drivers/media keep their
change with a diff regenerated for the new context.
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, devices/s8003,
devices/t8010, drivers/base, drivers/media, drivers/pmdomain,
drivers/power and drivers/usb are renumbered to follow main's.

Main's versions apply first now, so 0011 in drivers/media keep their
change with a diff regenerated for the new context.
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, devices/s8003,
devices/t8010, drivers/base, drivers/media, drivers/pmdomain,
drivers/power, drivers/soc-apple and drivers/usb are renumbered to
follow main's.

Main's versions apply first now, so 0011 in drivers/media keep their
change with a diff regenerated for the new context.
Bring this branch onto current main. Main has since merged OrchardLinux#124, OrchardLinux#122,
OrchardLinux#118, OrchardLinux#121, OrchardLinux#110, OrchardLinux#127, OrchardLinux#101, OrchardLinux#126, OrchardLinux#108 and OrchardLinux#111, several in part or
reworked, and OrchardLinux#123 and OrchardLinux#128 were closed. Where main carries a topic,
main's version is kept; the left-out parts that warm restart and
suspend need come back in OrchardLinux#107 and OrchardLinux#117.

This branch's own patches in devices/s8000, devices/s8003,
devices/t8010, drivers/base, drivers/media, drivers/pmdomain,
drivers/power, drivers/soc-apple and drivers/usb are renumbered to
follow main's.

Main's versions apply first now, so 0011 in drivers/media keep their
change with a diff regenerated for the new context.
On J71t a completed suspend/resume cycle does not bring the GPU back,
yet the G5P sleep admission admits an idle GPU, so
/sys/power/suspend_ready reads "ready" and userspace that follows it
suspends into a GPU that does not come back.

drivers/drm 0045 adds a read-only "suspend" parameter to the G5P
driver, off by default. While it is off, the GPU's sleep admission
refuses before anything is shut, and suspend_ready reads
"not-ready <gpu>:resume-unsupported -95". Booting with
a9_gpu_drm.suspend=1 admits sleep exactly as before. Boards without
the G5P driver are not affected.

The patch takes the next free number in drivers/drm, after the
gpu-handover branch's 0042-0044, so every existing patch keeps its
number.

Assisted-by: LLM
Carry the G5P sleep refusal (drivers/drm 0045) from j71t/g5p-suspend.

Assisted-by: LLM
Bringing this branch onto main took main's devices/j71t/firmware.json,
which main rewrote with its AOP firmware, and lost the output the GPU
needs: lib/firmware/apple/agx-g5p-c0.ldr, extracted from the
kernelcache by agx-g5p-ldr-from-kernelcache@1 (src/orchard/recipe.py
still has the step). Without it the G5P driver finds no loader and the
GPU never starts ("autoboot stopped at validate (-2)").

Put the entry back after main's outputs, unchanged; the file is again
main's plus this one output.

Assisted-by: LLM
Carry the restored G5P firmware loader entry in
devices/j71t/firmware.json.
Carry the restored G5P firmware loader entry in
devices/j71t/firmware.json.
Carry the restored G5P firmware loader entry in
devices/j71t/firmware.json.
J71t's agx-g5p-c0.ldr output was lost when main's firmware.json was
merged in, while its recipe step stayed, and no test noticed: the GPU
simply never started. Assert that J71t ships the loader, and that every
step in orchard.recipe.STEPS is used by some board's firmware output,
which holds on main as well.

Assisted-by: LLM
Carry the firmware recipe test for the G5P loader.
Carry the firmware recipe test for the G5P loader.
Carry the firmware recipe test for the G5P loader.
henrikklee pushed a commit that referenced this pull request Nov 14, 2026
The iPad 5's GPU takes geometry, fragment and compute jobs through
powervr's UAPI now, so it can render once Mesa speaks its job format.

What a user gets with it:
- Programs each get their own address space on the GPU; up to two
  frames of one program are in the GPU at a time, a fragment job starts
  as soon as its geometry job is submitted.
- A job that hangs ends after 5 s together with the other jobs in
  flight. The firmware is stopped, the GPU's power domain goes off for
  at least 10 ms, and the next job boots the firmware again. Only the
  program whose job hung loses its GPU context; nothing stays frozen
  until reboot.
- The firmware is stopped a second after the last job and for sleep,
  and booted again on the next use.
- The parameter manager's memory (its pool, its tables, its free list)
  is the kernel's and out of a program's reach.

[Orchard: ported from orchard PR #74 with #119 and #125 by Berkay
Kösebay (apple-g5p render-user-layout.h, render-user-geometry.h,
render-user-msaa.h, render-user-bind.h, render-user-mmu.h,
render-user-pmset.h, render-user-wake.h, compute-user-layout.h,
drm-render-v3.inc, drm-render.inc, drm-compute-v1.inc), as the A9 chip
of the powervr driver's Apple firmware path. The work records, commands
and microsequences this writes are byte for byte what his headers
write for the same inputs (compared on the host for nine target sizes,
one and four samples). What was left out and where the backend differs
is recorded in docs/bring-up/gpu/a9-g5p.md.]

Built for apple-16k; not run on hardware.

Assisted-by: LLM
@henrikklee

Copy link
Copy Markdown
Contributor

Your A9 GPU work is now on main as part of the shared powervr driver, with you as author on the ported code: kernel in orchard #138, Mesa in OrchardLinux/mesa #8, desktop GPU check in omarchy-a #54, OpenGL (Zink) from our Mesa in omarchy-a #55. docs/bring-up/gpu/a9-g5p.md in orchard records what was ported, what was left out and why. This branch head is kept as tag contrib/brkosebay-j71t-gpu-handover. Closing as superseded; thank you.

@henrikklee henrikklee closed this Nov 14, 2026
henrikklee pushed a commit to brkosebay/hardware that referenced this pull request Nov 15, 2026
The powervr backend for the iPad 5's GPU carries one boot path of Berkay
Kösebay's apple-g5p driver. This page keeps the rest of what that driver
knows about booting the GPU: its alternate boot paths and why each
existed, the overclock, the firmware's own idle park, power cycling and
recovery, the two lost-GPU races of OrchardLinux#125, the register access rules,
timing values and register meanings, with what was measured on J71t. It
also maps each piece of his boot code to where it lives now.

[Orchard: written from orchard PR OrchardLinux#74, OrchardLinux#119, OrchardLinux#120 and OrchardLinux#125 (apple-g5p)
by Berkay Kösebay; the findings and measurements are his.]

Assisted-by: LLM
henrikklee pushed a commit to brkosebay/hardware that referenced this pull request Nov 15, 2026
Berkay Kösebay built a warm restart for the iPad 5 (J71t): Restart goes
from Linux straight into Linux through kexec, without DFU and a host.
On 2026-11-12 it ran 20+ times without a freeze, back up in 7 to 30
seconds. The PRs are not merged; this page keeps what they found so the
feature can be rebuilt without reading their history.

It records, per subsystem, the mechanism, the measurements and what
stayed open: the resident EL3 PSCI in m1n1 (A7 to A9 only), the ANS
NVMe hand-over behind the S8003 PCIe port, the GPIO, SPI and touch
quiesce, the watchdog reset guard, the GPU and VXD stops, the retained
AOP reload, the display and device-tree state carried across kexec, and
the userspace trigger. It also lists which chips the design can apply
to, the points raised in review, and every PR, commit and tag.

Provenance: the content is Berkay Kösebay's work from orchard OrchardLinux#102,
OrchardLinux#105, OrchardLinux#106, OrchardLinux#107, OrchardLinux#116, OrchardLinux#125 (hand-over parts) and OrchardLinux#117 (restart
parts), and omarchy-a OrchardLinux#38 and OrchardLinux#43. The branch heads are kept as tags
contrib/brkosebay-j71t-{psci,warm-storage,warm-peripherals,warm-kernel,
reset-guard,gpu-handover} in orchard and
contrib/brkosebay-j71t-{warm-restart,warm-restart-shared} in omarchy-a.
Written up from the PRs; nothing here was run again.

Assisted-by: LLM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants