Repository navigation
Description
Summary
On a 14-inch MacBook Pro M4 Max (J614c, T6041) the AOP firmware crashes during speaker use, about once a day. After the crash the speakers stay silent until a reboot: apple_aop marks the firmware crashed and nothing restarts it. All six crashes so far came within 200 ms of the speaker stream stopping or restarting, never during steady playback.
Setup
- Kernel:
kernel-aurora7.3.0-40 (pin9125b314) with the J614c speaker enablement from my open speaker PR, plus the trackpad resume fix - Speaker protection:
speakerguardd-v2-1.0.0~alpha.1-0.48.20261113git04169c447065, PipeWire 1.6.9, WirePlumber 0.5.19 - Speaker path:
snd_soc_apple_aopspkrservice (the J616s backend inaop_spkr.rs) with the six SN012776 amps on i2c1
The six crashes
| # | Time | Seconds since boot | Just before the crash | Gap |
|---|---|---|---|---|
| 1 | 2026-11-16 01:10 | 543.9 | PipeWire hw:AppleJ614,DEV=2c (V/I sense capture) follower delay:4944 … resync; speakerguardd released and re-granted protection (heartbeat_fresh: false) |
100 ms after the re-grant |
| 2 | 2026-11-16 17:16 | 1277.6 | playback stopped (speakerguardd-session … play=0) |
33 ms |
| 3 | 2026-11-17 00:46 | 201.9 | playback started, protection granted | 165 ms |
| 4 | 2026-11-17 16:38 | 3499.8 | DEV=2c follower delay:4856 … resync, release and re-grant, a VA-API AV1 decode error in the browser at the same moment |
68 ms after the re-grant |
| 5 | 2026-11-17 17:20 | — | playback stopped (speakerguardd-session … play=0), no resync in the 15 s before |
33 ms |
| 6 | 2026-11-17 20:48 | — | a volume-key press (the feedback sound) while the speakers played; PipeWire then re-prepared DEV=1p 54 ms after the crash |
54 ms after the key press |
The two DEV=2c delays (4856 and 4944 frames) are longer than the 80 ms (3840-frame) sense ring, so the ring lapped. That stops the sense front-end and the stream restarts, adding a quick pw1/pwrd cycle just before crashes 1 and 4.
Crashes 2 and 5 are the clearest. In both, the firmware died 33 ms after the stream stopped, and the first speaker power request after it (pw1, from stop_locked) only came about 70 ms after the crash and failed with EINVAL. In crash 5 the type-0 mailbox history (last 32 messages) has nothing on endpoint 34: only steady traffic on endpoints 0x24/0x25 (and one 0x27). So, as far as the log shows, no spkr power call was in flight when the firmware died. What had changed was the playback DMA stopping in the trigger while the serializer was still in pwrd, which it stays in until stop() sends pw1 about 100 ms later. That points at the firmware not tolerating a running serializer with no host data, rather than at a power transition itself. The test kernel below will confirm or rule this out.
Crash 6 was on a test kernel that logs every spkr power request with its duration. The last request before it, a pw1/pwrd re-prepare, finished 880 ms earlier. Nothing was in flight. The volume-key press 54 ms before the crash made PipeWire stop and restart the speaker stream. Its snd_pcm_prepare came 54 ms after the crash, and its pw1 was the first call to fail. So again the firmware died with the playback DMA stopped and the serializer still in pwrd, waiting for the next prepare to take it down.
The same test kernel shows the firmware answering quickly: 30+ power requests, pw1 in 1.2–2 ms and pwrd in 0.4–0.6 ms. The risky window is therefore not the firmware calls. It is the time PipeWire leaves the stream stopped between its stop and its next prepare, about 50–80 ms here. On an otherwise idle sink PipeWire restarts the speaker stream 2–3 times for every short sound (33 re-prepares in 45 s of volume-key presses), so a few key presses open many such windows. With music playing, the same presses caused no restarts.
No crash happened during sleep. Two of the first four boots had no suspend at all before the crash.
Crash 4, kernel side
[ 3499.734566] pipewire[5483]: spa.alsa: hw:CARD=AppleJ614,DEV=2c: follower delay:4856 target:1024 thr:512 resample:0, resync (0 suppressed)
[ 3499.740610] kernel: snd-soc-macaudio sound: Speaker protection withdrawn: released by the owner
[ 3499.750006] kernel: snd-soc-macaudio sound: Speaker protection granted
[ 3499.818023] kernel: apple_aop 510c10000.aop: RTKit: co-processor has crashed
[ 3499.818252] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): Global IRQ status 0x00000001 seen.
[ 3499.818452] kernel: apple_aop 510c10000.aop: RTKit: Crash time: 1369833736322
[ 3499.823168] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): SCMController i2cscm0 [0x1120358] : global-errors = 0
[ 3499.823311] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): CH[0] : ToT=0, TMO=0, OVR=0, ABR=0, NF=0, UNS=0, ED=0, EB=0, ENR=0, EBA=0, EBA=0, EO=0
[ 3499.823606] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): SCMController i2cscm1 [0x11201b8] : global-errors = 0
[ 3499.823976] kernel: apple_aop 510c10000.aop: AOP firmware crashed
[ 3499.921014] kernel: apple_aop 510c10000.aop: RTKit: Device is crashed, cannot send message
[ 3499.921299] kernel: snd_soc_apple_aop audio.7.auto: ASoC error (-22): at snd_soc_dai_prepare() on spkr
...
[ 3500.401809] kernel: apple_aop 510c10000.aop: Too many inflight calls on endpoint 34
...
[ 3705.349940] kernel: snd_soc_apple_aop audio.7.auto: spkr pw1 failed: EIO
[ 3705.350780] kernel: snd_soc_apple_aop audio.7.auto: spkr idle failed: EIO
The crash log carries no panic string: the message section only has the I2C SCM controller listings, all with zero errors. The two mailbox histories are in the full log, which I can attach.
Crashes 1 to 3 have the same crash-log shape. In crash 2 the first errors after it are spkr pw1 failed: EINVAL and spkr idle failed: EINVAL, from the stream that had just stopped.
After the crash
apple_aopsetscrashedand logs "AOP firmware crashed". The comment atAop::crashedsays nothing restarts the firmware short of a reboot, and that is what happens.- Each later call fails with "Device is crashed, cannot send message", but seems to keep its slot on endpoint 34. After about eight calls every call fails with "Too many inflight calls on endpoint 34" instead. The result is the same, but the log no longer says the firmware is dead.
- Not the kernel's, noted for completeness: speakerguardd's PipeWire graph adds the speaker PCM as a required object. Once it cannot open, PipeWire fails to create its context and exits, so headphones and Bluetooth go silent too. I'll raise that with speakerguardd separately.
Questions
- Does the speaker service expect the serializer to leave
pwrdbefore the host stops feeding it? On J616s, did anyone see the firmware tolerate a stopped playback DMA for ~100 ms while still inpwrd? If not, the fix may be to sendpw1before the DMA stops (a nonatomic trigger, or a stop path that powers down first). - Is there a known ordering or pacing requirement for
pw1/pwrdon T6041's AOP firmware that differs from the J616s the backend was qualified on? The re-grant crashes (1, 4) also contain a stop followed by a start. - Would an AOP restart after a crash be acceptable in principle, so a crash costs a second of audio rather than a reboot? I tried the closest thing available today, an unbind and rebind of
apple_aopafter crash 5. The unbind only finished after PipeWire, WirePlumber andalsactlreleased the card (until then it sat insnd_card_disconnect_sync). The rebind was then refused by design: the crashed firmware cannot acknowledge the RTKit shutdown, soremove()retains the shared buffers and setsRETIRED, and probe answersan earlier instance could not be shut down; reboot before probing again(-ENODEV). A restart would need a way to stop the co-processor's DMA without its cooperation (a CPU or power-domain reset of the ASC), and a firmware image to boot again, as far as I know only iBoot loads it. Is either available on T6041?
Next
A possible driver-side fix: on the playback front-end's STOP trigger, take the serializer down to pw1 promptly, after muting the amps, from a work item, since the trigger is atomic. The serializer would then never sit in pwrd without host data while userspace decides what to do next. The next prepare already brings it back through pwrd. This changes when the amps lose their clocks, though, so I'd like your view before I try it. I'm also looking at why PipeWire restarts the stream so often, which would shrink the problem from the other side.
Activity
joske commentedon Nov 18, 2026
Same here on another J614c, twice in two days (two different kernels on #1664). Both times the crash came right after playback stopped:
speakerguardd-session … play=0co-processor has crashedThe
Ccstcrash-log section is byte-identical in both. After the crash the speakers stay dead until a reboot, as you describe.🤖 Generated with Claude Code
joske commentedon Nov 18, 2026
A follow-up, with a before/after and a quick reproduction.
Before speakerguardd v2 this never happened here. From 11 to 16 Nov the J614c speakers ran on a local test setup, for the last three days on the same kernel speaker path as now (
aop_spkr.rsis identical between those kernels and my current one), with legacy speakersafetyd unlocking the volume once at startup and a −12 dB ceiling. I watched YouTube on it every night, starting and stopping playback about as often as now, and the AOP never crashed. Since switching to speakerguardd v2 on 16 Nov it has crashed three times, and now I can trigger it in a few minutes.Reproduction (kernel based on #1664, volume slider at about a third, YouTube in Firefox, pausing and resuming): fresh boot, six clean start/stop cycles, then during the seventh stream:
So this one came during playback, 1.7 s after a withdraw and re-grant caused by the sense-capture resync, not right after a stop. My two earlier crashes were 24 and 37 ms after a stop. All three have the same
Ccstsection.The volume was low (the slider at about a third), so the level is unlikely to matter. What the old setup didn't do is the grant/withdraw cycle: it unlocked once and stayed unlocked across stream stops. Every crash I have follows either a stream stop or a withdraw and re-grant.
🤖 Generated with Claude Code
joske commentedon Nov 18, 2026
One more data point, which supports the stop path as the trigger.
With a silent stream kept on the speakers, the AOP survives:
On a fresh boot with that running, I paused, resumed and scrubbed YouTube as before for about 6 minutes. Not a single
play=0: the speaker stream never stopped. There were five sense-capture laps (DEV=2c follower delay:4480-8000 … resync), each with a protection withdraw and re-grant 8-80 ms later, and no crash. The last two came 2 s apart, the second withdelay:8000.Without it, each of the three boots before crashed: twice 12 and 37 ms after a logged stream stop, once 1.7 s after a withdraw and re-grant (the 09:35 one above).
So a withdraw/re-grant alone doesn't reliably crash it; the 09:35 crash may have needed something else in that window. The stop is what kills it here, which fits your plan to change the STOP trigger path.
For anyone else hitting this, the
pw-catline above works as a stopgap.🤖 Generated with Claude Code
RyanTheTide commentedon Nov 19, 2026
Thanks to both of you for the detailed logs. Here is a code read of the driver side, with no audio run. Point by point:
1. The "Too many inflight calls on endpoint 34" after the crash. That is a driver bug, and #2342 fixes it. Each EPIC call reserved a tag before rtkit refused its doorbell ("Device is crashed, cannot send message"). The call was then marked abandoned, so its tag would be kept until a reply came, and a crashed AOP never replies. Eight calls later the table was full. With the PR, every call after the crash fails at once with
ENODEVand reserves nothing, sospkr pw1/idlekeep reporting the crash instead of a full table. The crash itself and the power sequence are unchanged.2. What the logs point at. Eight of the nine crashes in the two reports came within 200 ms of a stream stop or restart (12-37 ms after a
play=0stop in the clearest ones). The silentpw-catstream that keeps the speaker PCM running survives about 6 minutes of the same pause/resume/scrub use with five sense-ring laps. Inaop_spkr.rsa front-end STOP only stops the ADMAC (snd_dmaengine_pcm_trigger). The serializer stays inpwrduntilbe_hw_free(stop(),pw1) or the nextbe_prepare(run(),pw1thenpwrd), and PipeWire leaves 50-100 ms or more between the two. So the window that matters is a running serializer with no host DMA, which is what @mouthmantra suspected. Nospkrcall was in flight in crash 5 and crash 6, and that fits.3. On the proposed fix (power down on STOP). The direction looks right to me. Some constraints from the current code:
be_prepare, which may already be sendingpw1/pwrdfor the restart.run()would have to flush or cancel that work under the service lock before it powers the serializer back up. Otherwise a latepw1from the work item lands after the restart'spwrdand stops a stream that is playing.nonatomic = 1on the DAI link). Then STOP can mute the amps (mute_codecs()) and sendpw1inline. The firmware answerspw1in about 1-2 ms in the timings above, so that fits in a trigger. With DPCM the front-end link has to be nonatomic as well.stop_lockedalready does that afterpw1), and speaker protection has to see the stop as a withdrawal it can reverse, asbe_hw_freedoes today.spkrboards get the same change, and it is decided by the service's presence, not by board.4. Restarting a crashed AOP. Today, no. The ASC would have to stop without the firmware's cooperation, and the AOP firmware image would have to be loaded again. As far as we know only iBoot loads it, so a reboot is the only recovery. That is why the driver should not drive the firmware into this state in the first place (point 3).
Not tested on hardware. No sound was played.
jgoyvaerts commentedon Nov 19, 2026
The same crash on a J616s (14-inch MacBook Pro, M4 Pro, T6040), so not only J614c. It points at the shared
aop_spkr.rsstop path, as in the code read above.Setup: our own 7.3.0 kernel (integration/aurora bf423bf plus local G17 GPU patches,
aop_spkr.rsunchanged), speakerguardd-v2 1.0.0alpha.1-41, PipeWire 1.6.10, WirePlumber 0.5.19.A game (CS2 under FEX) played on the speakers for 78 s with no sense resync and no withdraw/re-grant, then quit. The crash came 28 ms after the stream stop:
(UTC.) After that, speakerguardd loops "protection granted" -> "released: heartbeat_fresh: false" every 4 s, opens of the speaker and sense PCMs fail with EINVAL and then EIO, and the speakers stay silent until a reboot. The SCM i2c controllers report 0 errors in the crash dump, as in your logs.
The same boot had about a dozen earlier game start/quit cycles plus ~10 h of desktop audio without a crash. It is the first AOP crash in our last 15 boots.
🤖 Generated with Claude Code