Skip to content

drm: add GPU cooling v2 and J700 thermal bindings - #1065

Draft
RyanTheTide wants to merge 16 commits into
integration/aurorafrom
pwr/gpu-cooling-v2-20261107
Draft

RyanTheTide wants to merge 16 commits into
integration/aurorafrom
pwr/gpu-cooling-v2-20261107

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem. drm/asahi has no GPU cooling device, so thermal zones cannot hold the GPU back. Under sustained load the Neo (J700) GPU die peaks around 107.6 C, where macOS stays near 105.5 C.

Change. A thermal-framework cooling device for the G17P and G16S firmware layouts. Each cooling state sets a nonzero GPU power ceiling, combined with any other owner's request; startup and recovery keep the restrictive ceiling. The device is created only when the GPU node has #cooling-cells, and the firmware layout says where the power word lives. J700 binds its SMC outer thermal zones to the GPU in its board DT. The M5 firmware layout has cooling turned off: its power word sits at a different offset.

The cooling device is optional. A board with #cooling-cells whose selected firmware layout has no live power state (V25F84, and the packed M5-profile graphs, which includes the J700 26B123 build from #1997) now starts the GPU without a cooling device and logs GPU firmware layout has no live power state: no GPU cooling device. Before, admit() refused the probe with EOPNOTSUPP, so a missing optional feature meant no GPU. A build that inherits live_power but boots a packed graph passed admit() and then ran without a cooling device and without a log line. The decision now comes from the layout data the build actually selects, not from the board or chip. The GPU die limiter target is back at 110 C (the series had lowered it to 95 C with no measurement; see Still to run). The M5 session initializer gets its missing cooling: None (the 11-20 merge left boot_m5() without it, so the driver did not compile).

Testing. arm64 QEMU KUnit 5/5, the J700 compiled-DT policy test 17/17, t6040 calibration host test 5/5 after the merge with current integration/aurora, and no new W=1 or DT schema diagnostics against the base. 11-20 follow-up (45b5650, ab10dc1, 00e0a70): host cooling selftest PASS (cap policy + typed PPM fields); rustfmt clean on the changed hunks; #1065 head + #1997 head merge with no conflicts, and the probe path was read through on the merged tree; arm64 Rust build of drivers/gpu/drm/asahi is pending on the buildbox (CI rust-build is running).

Still to run.

  • Neo: 10 min vkmark loop. Pass: peak GPU die temperature at or below 105.5 C, /sys/class/thermal/cooling_device*/cur_state steps logged, fps loss recorded, no critical-trip shutdown.
  • The die limiter target is 110 C again (00e0a70). A lower target needs its own sustained-load row: peak temperature and fps against this 110 C run.
  • Neo on 26B123 with drm/asahi: T8140 G17P on macOS 27.2 firmware #1997 merged: drm/asahi probes, the log has the "no live power state" line, and the GPU has no cooling device. Cooling on 26B123 needs its RegionC and power-state words admitted, plus a readback row.
Detail: earlier gates and qualification plan
  • Host/KUnit/DT policy: production C policy and Rust typed-layout tests; arm64 QEMU KUnit 5/5; compiled J700 DT policy 17/17. Two fresh arm64 W=1 Images with identical configs.
  • W=1: 746 diagnostics on each side (677 compiler + 69 kernel-doc), 0 in touched GPU files, 0 new.
  • DT schema: dt_binding_check and dtbs_check for the GPU, thermal-zone and SMC hwmon schemas, plus 119 Apple DTBs per revision: 33 existing records on both sides, 0 new. dtschema 2026.9; an unrelated apple,tunable decoder crash in the global run is excluded from the pass claim.
  • 11-20 merge with integration/aurora 0abc2f3 (5 conflicts: asahi Makefile, g17/fw/mod.rs, g17/initdata.rs, g17/mod.rs, selftests Makefile); the t6040 calibration test now builds g17/fw/power.rs.
  • Qualification per board (Neo/G17P, J616s/G16S): board binding inventory (J616s has no GPU #cooling-cells yet); no-load state sweep 100/75/50/25/0 with readback; short load bursts under an independent temperature supervisor; gpubench comparison against matched baselines; 0 new boot warnings and 0 AVCs with SELinux enforcing; stop and fall back on any failed cap update, lost binding, GPU fault or AVC. The in-tree plan is Documentation/gpu/asahi-gpu-thermal.rst. The 25% floor stays provisional until achieved-power evidence exists.

Assisted-by: LLM (Claude:Opus 5.5)

RyanTheTide and others added 8 commits November 7, 2026 13:15
Expose ten existing SMC temperature channels as reporting thermal zones,
including SoC, charger and enclosure readings. The channels keep their
existing order and the CPU zones retain their current policy.

These zones have no trips or cooling maps. GPU cooling still needs a
qualified firmware ceiling and board thermal policy.

Check the compiled DTB by resolving zone phandles and channel indices to
SMC keys, including the six CPU references. Add an arm64 KUnit config for
the existing SMC parser, channel-preservation and fan-model tests.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Document the distinction between G17 firmware die-limiter configuration,
SMC reporting and a live thermal cooling ceiling. State the shared G16S
and G17P integration requirements and the remaining qualification work.

Include offline sensor checks and a staged owner test plan covering
actuator faults, transient response, sustained loads, frame times and
matched macOS traces. The reporting zones do not establish GPU cooling.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Define the firmware percent interface and require a qualified ABI before a board requesting cooling may initialize its GPU.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Add a per-device three-mirror cooling owner, a retained 25 percent qualification ceiling, recovery rearming and callback teardown. Exercise the production cap mapping with host and KUnit fake-publication tests.

The clean percent offsets still lack a qualified firmware-allocation base for either build. Keep both ABI capabilities absent and refuse GPU initialization on boards requiring cooling, before booting either processor. Do not issue speculative firmware writes or claim host publication is achieved power.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Bind all correlated SMC, charger and chassis zones to exact percent requests with 100 ms polling and independent critical shutdown. Use the existing power_allocator active-trip path for this non-power-actor device and lower the firmware fast-die target to 95 C without changing its release or gains.

J700 now requires qualified GPU cooling and therefore intentionally refuses GPU initialization while the ABI tables are unresolved. Use a stricter zero final request pending a validated nonzero endpoint; do not claim zero is a proven stop action. Extend compiled-DTB checks and document the incomplete status, short-burst owner qualification and rollback.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Replace the obsolete host mirror model with typed RegionC and separate
PowerState PPM publications. Serialize requester ceilings by their minimum,
clean write-back mappings before reporting publication, and retain requests
across recovery. Bound the unqualified low end at a provisional nonzero
quarter of initialized maximum power.

Admit the common G17P 26A428 and G16S 25G83 layout, leaving other builds
unadmitted. The G17 fast die limiter is already initialized before submission
with the conservative bring-up policy. Expose publication/readback diagnostics
and update the offline tests and bounded owner qualification plan.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Both admitted PowerState objects use write-combined CPU mappings. Keep
cache cleaning confined to the write-back RegionC allocation and complete
the PPM stores with the publication barrier.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Require board binding, bounded load transitions, matched gpubench parity, boot-log warning parity and enforcing SELinux audit evidence per board. Record the missing J616s GPU binding as a prerequisite for a separate board-policy change.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Receipt retry completed for head 5e23eb258cd056486e1cfb3910cc38e056070eac against exact base 68db611bf7e3a2010eaa87be2b4b7043e27f4019. The branch is unchanged and this PR remains draft.

Buildbox receipt Result
h-49576 Host policy + Rust layout/canary tests PASS; arm64 QEMU KUnit 5/5; compiled J700 policy 17/17; fresh candidate/base W=1 Images PASS
h-49636 GPU/thermal-zone/SMC dt_binding_check + dtbs_check, and direct validation of 119 Apple DTBs per revision, completed without validator crashes
h-49640 Input hashes verified against controller receipts; identical Image configs; 746 diagnostics each, 0 touched-GPU diagnostics, 0 new warnings; 33 existing structured DT validation records each, 0 new DT records or full-text diagnostics

The expanded warning comparison includes all 677 compiler diagnostics and 69 capitalized kernel-doc warnings per revision. Only build roots and source coordinates are normalized; multiplicity is retained.

The successful DT run uses unmodified dtschema 2026.9 with the three selected bindings and builtin types in its processed schema. The default global metadata has an unrelated apple,tunable decoder collision; that failure and the unsuccessful older-tool retry h-49609 are preserved. Existing DT validation diagnostics remain; a globally clean schema tree is not claimed. The first comparison h-49639 flagged a DTC continuation's shifted source coordinates; its receipt is preserved and h-49640 corrects that normalization.

SHA256 receipts:

  • Candidate Image: e7bc1febbde029aa80ebe0439b68c787102fc76e7471f1df1a08db9c48fefc7b
  • Candidate W=1 log: 2f643cf686ccc2f72eba06f730d0c71aec03f1234d670aeb72463f723d4a855f
  • Base W=1 log: bb1fa914eca2d32adee0d4f1cefe05b0b2891ad0e69d4d9c0b92e0cd04113210
  • Final parity JSON: cb661238e43ccba95dc78e1bc0abe7cdbe2083aacafdac7c9b2368699b292ec1

No target device was used. PWR's Neo/J616s qualification plan remains in the PR, including the J616s binding prerequisite, GPU-load transitions, gpubench parity, zero new boot warnings, zero SELinux AVCs and rollback evidence.

Picks up the tools/aurora-sep source-check fix (c60537c) and the
current release branch, so CI runs against today's base.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Neo hardware trial, 2026-11-08 02:06. Do not re-trial this head.

Setup: aura-try one-shot trial of the integration j700 DTB (72f34232, integration d03acec) plus this PR's DT, built from d03acec with a clean merge. The phandles of every existing node were kept. fdtdiff against 72f34232:

  • added: the 10 new thermal-zone subtrees;
  • changed: the gpu node only (#cooling-cells);
  • unchanged: aliases and all 14 display nodes.

The kernel was a one-shot 7.3.0-aurora-gpux2: fleet23 + this PR + #1096 + #1146.

Result: the Neo went dark during the trial boot and stayed off.

  • No ping or ssh for 18+ minutes.
  • Neither the aura 178 s trial watchdog nor the 12-minute trial guard brought it back. A hung boot would have been reset, so this points at an orderly power-off.
  • The likely cause is a critical trip in one of the new zones (hw_protection_shutdown).
  • The production Stage 2 is untouched and the trial marker is spent. The Neo waits for a power-button press, after which I will read journalctl -b -1 for the zone and temperature and post them here.

What I checked offline:

  • Sensor indexes are right. macsmc-hwmon keeps one channel per temperature-* node, even for keys that fail. In the merged DTB, index 6 = Tp0t, 11 = TCMz, 12/13/14 = Ts0A/Ts0I/Ts0M, 50 = Tp02, 52 = Tp06, 56 = Tp0E, 63 = Tpx1, 65 = Tpx3, matching the comments.
  • The trip ladders are the suspect.
  • None of these values are recovered board limits. The PR calls them provisional.

Requested changes before any further hardware trial:

  1. Drop the critical trips from all new zones until each sensor's range is validated against macOS on the same Neo. Critical shutdown stays with the existing CPU zones and SMC/PMP protection. Alternatively, align the SoC zones with the CPU zones' 110 C critical.
  2. Validate every new key's live value on a normal boot (hwmon), idle and under load, before it gets a cooling map.
  3. The kernel side is inert without the DT. On gpux1 with the production DTB: no cooling device, no regression, Enforcing, 0 AVC.

Minor: modpost warns that cooling_test.ko is missing MODULE_DESCRIPTION().

Assisted-by: LLM

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Merge-train hold (11-08): don't merge this as is. In GPUX's DT trial on the Neo its critical trips at 98 °C on SoC zones nobody has validated powered the Neo off; the post-mortem is on this PR. It needs validated zones and trip points, and a hardware trial that survives load, before it can go through the train.

The ten SMC thermal zones added for GPU cooling (Tp0t, TCMz, Ts0A/I/M,
Tp02/06/0E, Tpx1/3) each had a critical trip with a shutdown action.
Their sensors are referenced by index into the SMC hwmon channel list,
and none of them has been read on hardware against its key yet. If an
index points at the wrong channel, or a key reads a bogus value at
boot, the thermal core powers the machine off, and a powered-off Mac
recovers from neither the boot watchdog nor a trial fallback. A J700
went down that way during a trial boot of this tree.

Drop the critical trips and the critical-action properties from those
ten zones. The cap ladder and the hot trips stay, so the GPU cooling
policy is unchanged. The six CPU zones are untouched. Critical trips
come back per zone once its sensor has been checked on hardware with
the trips disabled; until then the SMC and the platform firmware keep
their own overtemperature protection.

The selftest now requires that these zones carry no critical trip, and
the documentation says why.

Assisted-by: LLM
Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

DT3: pushed df7f7f1. It removes the critical trips and critical-action from the ten new SMC zones and keeps the cap ladder and hot trips. The six CPU zones are unchanged. The built J700 DTB differs from the previous head only by those 10 + 10 removals, with the same 5 pre-existing W=1 warnings, and j700-thermal-dt.py passes 17/17. One correction to the commit message: the 02:06 Neo power-off during the trial of this tree is suspected to be a critical-trip shutdown, but that is not yet confirmed; the previous boot's journal will be read when the Neo is powered on. Next steps before any new trial: (1) on the normal production DTB, read every referenced SMC key (hwmon idx 6, 11-14, 50, 52, 56, 63, 65) at idle and under load and match each to its key; (2) only then trial this tree; (3) add the critical trips back zone by zone. All of this waits for the boot-image freeze to lift after the audible speaker run.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

DT3 follow-up: confirmed. The Neo's 02:06 boot of the previous head logged 'critical temperature reached' on smc-tp0e, smc-tpx1, charger and smc-tp06 about 7 s into boot, then a hardware-protection shutdown. A read-only hwmon read on the production DT at idle shows the same keys at 51-53 C (Tp06 53.1, Tp0E 51.4, Tpx1 53.1, Charger 53.1), well under the 95-98 C critical points. The sensor-to-key mapping matches (sysfs tempN = DT index N-1), so the trigger is most likely the first values the thermal core reads through the hwmon thermal-sensor path at registration, before the SMC keys give real readings, or in a different scale. With df7f7f1 those zones can no longer shut the machine down. Before any critical trip comes back, the thermal-sensor read path for these keys needs a fix or an explanation: log the first get_temp values at zone registration on a no-critical trial. Evidence: traces/gpu/20261108-neo-thermal-poweroff/.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Hardware data for the J700 zones, from a log-only trial (no trips, no cooling maps, the same 10 sensors), kernel 7.3.0-aurora-fleet32, with trace_event=thermal:thermal_temperature on from boot:

  • The first reads at zone registration (t = 5.8 s) are normal: Tp* 58-72 C, chassis 42-50 C.
  • During the boot load burst (t = 7-16 s) the hotspot keys really do swing high, for 1-3 one-second samples at a time. Tp0E, Tpx1 and the charger zone reach 108-113 C, Tp02 and Tp06 reach 110-112 C, Tp0t and Tpx3 reach 83-86 C. Between bursts they fall back to about 70 C, and they settle at 65-70 C by t = 18 s. The existing cpu-p* zones reach 103 C in the same bursts; their critical point is 110 C.

So the earlier hardware-protection power-off was caused by real short transients above the 95-98 C critical points, not by bad first reads. Proposal for the trips:

  • Keep critical trips off these keys until they are debounced: critical-samples >= 5 at 1 s polling (thermal: of: confirm critical over N samples). In this trace the longest run above 95 C is 3 samples.
  • Base passive and hot trips on sustained load, not on the boot burst.

Something to check in the sensor map: the charger zone (index 11) repeats the Tpx1/Tp0E values sample for sample, and Tpx3 repeats Tp0t. That looks like index or key aliasing.

Conflicts:
- asahi Makefile and the apple selftest Makefile: keep both sides' objects
  and tests.
- g17/fw/mod.rs: keep the power module next to the m5 module.
- g17/initdata.rs: keep the new m5/graph/outer_credits fields and the
  cooling owner's comment on live publication.
- g17/mod.rs: the firmware build is now picked once, after the GPU ID is
  read; admit the cooling layout right after that pick instead of picking
  it twice. Gpu keeps its cooling slot next to the new render_ids.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
g17/fw/initdata.rs now takes PowerState from g17/fw/power.rs, so the
host build of the T6040 wire builders failed with an unresolved
super::power import. Copy power.rs into the test crate.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
V26A428_M5 inherits from V26A428 and so took its live_power layout,
whose RegionC max_power_mw is at +0x84. The common M5 graph moves the
region C fields from +0x38 up by four, so that word is not the M5
build's. Mark the M5 build as not admitted: a GPU node that asks for
cooling on it fails the probe instead of writing the wrong word.

Also give the cooling KUnit module a MODULE_DESCRIPTION (modpost warns
without one).

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The merge with integration/aurora brought in boot_m5(), whose Firmware
initializer has no cooling field, so the driver does not compile. The
packed M5-profile graphs have no admitted live power layout and never
attach a cooling owner; initialize the field to None as boot() does.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
admit() refused the probe with EOPNOTSUPP when the GPU node has
#cooling-cells but the selected firmware has no live power layout. The
cooling device is optional: the firmware die limiter and the board's
SMC thermal zones protect the GPU without it, as they did before this
series. Refusing the probe turns a missing optional feature into no GPU.
Log it once and continue without a cooling device.

Decide from the firmware layout data actually selected, not from the
variant alone. The common layout names objects of the G17P and G16S
25G83 graphs. The packed M5-profile graphs (boot_m5) place RegionC and
the power state elsewhere and never attach, so a variant that inherits
live_power from V26A428 while its build selects a packed graph (the
J700 26B123 build in #1997) has no live power
state. Before, such a build passed admit() and then ran without a
cooling device and without a word in the log.

cooling_words() now reports an unbuilt power state as None, checked
before RegionC is resolved, and attach() warns and continues. A
malformed #cooling-cells, invalid mappings and a zero initialized
maximum still fail initialization.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
"arm64: dts: apple: add provisional J700 GPU outer protection" lowered
TEMP_TARGET from 110 C to 95 C. The value is shared by every board on
the G17P graph, sits in driver code rather than board data, and no
measurement backs it: the Neo J700 baseline under sustained GPU load
peaks at 107.7 C on the hottest GPU key and macOS holds about 105.5 C,
so a 95 C firmware target throttles the GPU well below both, at a
performance cost nobody has measured. The 110 C target matches the
driver before this series and the status-config words of the M5
firmware tables (+0xc8 11000, +0xcc 8000).

Restore 110 C and keep the release, gains and period. The cooling
device and the SMC zones remain the outer loop; a lower firmware target
needs its own sustained-load temperature and performance measurement.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant