Repository navigation
Description
Cause and fix (update 2026-11-15)
The 40-core 25G83 configuration starts the firmware with three power configuration words measured from a running
25F84: +0x0 = 1, +0x44 = 0xc and +0x9c = 338.0. They are state the firmware's performance limiter writes while
it runs, and +0x48 is that limiter's limit. Started with them, the 25G83 firmware lowers the limit on cold power-ups
and never raises it. Left at the 25G83 starting values (as the M4 Pro and the J616c run), the limit stays at 1000
through 6,000 short clients and 10 minutes of idle, at 97 to 98 fps. Fix: #1987. The sections below are the
original report. Its reading of +0x48 as max_pstate_scaled and its leads were wrong. The 16 ms idle-off
mitigation is not needed.
Summary
On a Mac Studio J575c (M4 Max, 40-core G17P, firmware 25G83) the GPU loses performance in steps after a few thousand
cold power-ups (the GPU powering on again after the 2 ms idle-off), and stays slow until a reboot. A ray tracing
benchmark (gpubench rt.scene medium, ray query) goes from 99 fps to 79 and then 46 fps. The image CRC stays
the same. Its raster pass goes from 4.0 to 12.7 ms. Nothing in the kernel log.
The steps follow a firmware-written field in the power configuration (bundle 0x5380, the start of
PowerConfig25G83). The host writes 1000 at +0x48. The firmware then lowers it by about 1 per 10 cold power-ups,
and it never goes up again:
+0x2c / +0x30 |
+0x48 (= +0x50) |
+0x9c (f32) |
rt.scene fps | |
|---|---|---|---|---|
| fresh boot | 9 / 10 | 990 | 1312.0 | 99 |
| after 3,000 short GPU clients | 4 / 4 | 400 | ~924 | 100 |
| after 6,000 | 0 / 1 | 90 | 338.0 | 46 |
(+0x14, an f32 that starts at 1.0, moves between about 0.25 and 1.0.)
What drains it, and what doesn't
Measured as the change of +0x48, from /sys/kernel/debug/aurora-g17/hardware_data:
| action | change |
|---|---|
300x deqp-vk -n dEQP-VK.api.smoke.triangle (one process each), idle_off_delay_ms 2 (default) |
-27, -63 |
the same with idle_off_delay_ms 1000 |
0 |
300x vulkaninfo --summary, 300x api.device_init.create_device (no GPU work) |
0 |
| one process, 6,000 small CTS cases (~8,200 power epochs, short gaps): no slowdown | (fps 99 -> 101) |
| 90 s of sustained load | 0 |
| idle desktop, 4 min | -13 |
idle desktop, 10 min, idle_off_delay_ms 2 (4,412 power epochs) |
-50 |
idle desktop, 10 min, idle_off_delay_ms 16 (3,644 power epochs) |
-21 |
So it is the cold power-ups with longer gaps, not devices, VMs or load. Once it is down, nothing found so far brings it back
(load, idle). Not every power-up counts (about 1 in 90 on the idle desktop, 1 in 10 for the short clients). At the
idle-desktop rate a desktop would reach the slow state in about 3 hours (estimated, not observed).
What the field is
The block at PowerConfig25G83 +0x00..0x54 lines up with the G13 HwDataA head (fw/initdata.rs, without
unk_3c_0): +0x2c / +0x30 = actual_pstate / tgt_pstate, +0x48 = max_pstate_scaled (the host writes
1000, i.e. 100 x the top state). So the firmware lowers the GPU's maximum performance state, by 1 (of 1000) about
every 10 cold power-ups, down to 100 (state 1), where it stays: with the cap at 100, 3.5 minutes of full rt.scene load run at
actual/tgt 1/1 and 46 fps throughout.
What does not raise it again (measured on integration/aurora fc05050):
- sustained load (above), or 3 minutes with no GPU client at all (display manager stopped, no power epochs): the
cap stays where it was; - writing 1000 to
+0x48/+0x50(a debugfs hack on a test kernel): the next read shows the old value again, so this
is a copy the firmware publishes, not an input; - an s2idle cycle.
macOS's ADT GPU node has DVFS parameters that do not appear anywhere in the 25G83 bundle the driver builds:
gpu-perf-boost-ce-step 95, gpu-perf-boost-min-util 75, gpu-perf-base-pstate 3, gpu-idleoff-standby-timer
3500 (gpu-perf-tgt-utilization 70, gpu-num-perf-states 15, gpu-power-sample-period 8 and
gpu-pwr-sample-period-aic-clks 200000 do; the last is +0x4/+0x8). On G13 the first three are HwDataA
perf_boost_ce_step, perf_boost_min_util and the base state. A governor running without them would be a lead.
Reproduce
Kernel integration/aurora d89d8c0 and be8d4cd (same result), Mesa aurora-wip 24162c1642, Vulkan CTS 1.4.7.1:
for i in $(seq 6000); do deqp-vk -n dEQP-VK.api.smoke.triangle --deqp-log-images=disable >/dev/null; done
# about 10 minutes; any GPU benchmark before and after shows the drop (here at 5,000-5,500)
sudo grep -E '^0x53(80|a0|c0):' /sys/kernel/debug/aurora-g17/hardware_dataLong Vulkan CTS runs hit it too. The ray tracing selection of #837 slows down about 5,000 cases in, and
ray_query.stress.geometry_shader.triangles then takes 10.2 s instead of 4 s and loses its device at the 10 s ray watchdog
("watchdog gave up ... at the head of its queue for 10192 ms; long-work renewals 4/4", from #1926).
Leads
- macOS's ADT has idle-off parameters for the GPU:
gpu-idleoff-min_conf= 0.6 (0x3f19999a) and
gpu-idleoff-standby-timer= 3500. The live 25G83 power configuration has 0.6 at+0x29e8/+0x2a50, but 3500
appears nowhere in it (G15 passesapple,idleoff-standby-timerto its firmware; G17 has no equivalent).hw/t6041.rs
also has+0x44= 0xc and+0x9c= 338.0 (0x43a90000) as constants. That is the value+0x9cends at here. - J614/J616 and J773s set
apple,idle-off-delay-ms = <16>. J575c does not, so it runs at the 2 ms default. That
slows the idle-desktop drain about 2.4x (table above) but does not stop it. - Not checked: whether J614c/J616c (same firmware) drain the same way.
Not tested: other boards, other firmware.
Written with AI help (Claude Code); I ran the tests on my Mac.
Activity