Skip to content

J614c: AOP firmware crashes at speaker start/stop, speakers dead until reboot #2188

Open

Description

@mouthmantra

Summary

On a 14-inch MacBook Pro M4 Max (J614c, T6041) the AOP firmware crashes during speaker use, about once a day. After the crash the speakers stay silent until a reboot: apple_aop marks the firmware crashed and nothing restarts it. All six crashes so far came within 200 ms of the speaker stream stopping or restarting, never during steady playback.

Setup

  • Kernel: kernel-aurora 7.3.0-40 (pin 9125b314) with the J614c speaker enablement from my open speaker PR, plus the trackpad resume fix
  • Speaker protection: speakerguardd-v2-1.0.0~alpha.1-0.48.20261113git04169c447065, PipeWire 1.6.9, WirePlumber 0.5.19
  • Speaker path: snd_soc_apple_aop spkr service (the J616s backend in aop_spkr.rs) with the six SN012776 amps on i2c1

The six crashes

# Time Seconds since boot Just before the crash Gap
1 2026-11-16 01:10 543.9 PipeWire hw:AppleJ614,DEV=2c (V/I sense capture) follower delay:4944 … resync; speakerguardd released and re-granted protection (heartbeat_fresh: false) 100 ms after the re-grant
2 2026-11-16 17:16 1277.6 playback stopped (speakerguardd-session … play=0) 33 ms
3 2026-11-17 00:46 201.9 playback started, protection granted 165 ms
4 2026-11-17 16:38 3499.8 DEV=2c follower delay:4856 … resync, release and re-grant, a VA-API AV1 decode error in the browser at the same moment 68 ms after the re-grant
5 2026-11-17 17:20 — playback stopped (speakerguardd-session … play=0), no resync in the 15 s before 33 ms
6 2026-11-17 20:48 — a volume-key press (the feedback sound) while the speakers played; PipeWire then re-prepared DEV=1p 54 ms after the crash 54 ms after the key press

The two DEV=2c delays (4856 and 4944 frames) are longer than the 80 ms (3840-frame) sense ring, so the ring lapped. That stops the sense front-end and the stream restarts, adding a quick pw1/pwrd cycle just before crashes 1 and 4.

Crashes 2 and 5 are the clearest. In both, the firmware died 33 ms after the stream stopped, and the first speaker power request after it (pw1, from stop_locked) only came about 70 ms after the crash and failed with EINVAL. In crash 5 the type-0 mailbox history (last 32 messages) has nothing on endpoint 34: only steady traffic on endpoints 0x24/0x25 (and one 0x27). So, as far as the log shows, no spkr power call was in flight when the firmware died. What had changed was the playback DMA stopping in the trigger while the serializer was still in pwrd, which it stays in until stop() sends pw1 about 100 ms later. That points at the firmware not tolerating a running serializer with no host data, rather than at a power transition itself. The test kernel below will confirm or rule this out.

Crash 6 was on a test kernel that logs every spkr power request with its duration. The last request before it, a pw1/pwrd re-prepare, finished 880 ms earlier. Nothing was in flight. The volume-key press 54 ms before the crash made PipeWire stop and restart the speaker stream. Its snd_pcm_prepare came 54 ms after the crash, and its pw1 was the first call to fail. So again the firmware died with the playback DMA stopped and the serializer still in pwrd, waiting for the next prepare to take it down.

The same test kernel shows the firmware answering quickly: 30+ power requests, pw1 in 1.2–2 ms and pwrd in 0.4–0.6 ms. The risky window is therefore not the firmware calls. It is the time PipeWire leaves the stream stopped between its stop and its next prepare, about 50–80 ms here. On an otherwise idle sink PipeWire restarts the speaker stream 2–3 times for every short sound (33 re-prepares in 45 s of volume-key presses), so a few key presses open many such windows. With music playing, the same presses caused no restarts.

No crash happened during sleep. Two of the first four boots had no suspend at all before the crash.

Crash 4, kernel side

[ 3499.734566] pipewire[5483]: spa.alsa: hw:CARD=AppleJ614,DEV=2c: follower delay:4856 target:1024 thr:512 resample:0, resync (0 suppressed)
[ 3499.740610] kernel: snd-soc-macaudio sound: Speaker protection withdrawn: released by the owner
[ 3499.750006] kernel: snd-soc-macaudio sound: Speaker protection granted
[ 3499.818023] kernel: apple_aop 510c10000.aop: RTKit: co-processor has crashed
[ 3499.818252] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): Global IRQ status 0x00000001 seen.
[ 3499.818452] kernel: apple_aop 510c10000.aop: RTKit: Crash time: 1369833736322
[ 3499.823168] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): SCMController i2cscm0 [0x1120358] : global-errors = 0
[ 3499.823311] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1):    CH[0] : ToT=0, TMO=0, OVR=0, ABR=0, NF=0, UNS=0, ED=0, EB=0, ENR=0, EBA=0, EBA=0, EO=0
[ 3499.823606] kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): SCMController i2cscm1 [0x11201b8] : global-errors = 0
[ 3499.823976] kernel: apple_aop 510c10000.aop: AOP firmware crashed
[ 3499.921014] kernel: apple_aop 510c10000.aop: RTKit: Device is crashed, cannot send message
[ 3499.921299] kernel: snd_soc_apple_aop audio.7.auto: ASoC error (-22): at snd_soc_dai_prepare() on spkr
...
[ 3500.401809] kernel: apple_aop 510c10000.aop: Too many inflight calls on endpoint 34
...
[ 3705.349940] kernel: snd_soc_apple_aop audio.7.auto: spkr pw1 failed: EIO
[ 3705.350780] kernel: snd_soc_apple_aop audio.7.auto: spkr idle failed: EIO

The crash log carries no panic string: the message section only has the I2C SCM controller listings, all with zero errors. The two mailbox histories are in the full log, which I can attach.

Crashes 1 to 3 have the same crash-log shape. In crash 2 the first errors after it are spkr pw1 failed: EINVAL and spkr idle failed: EINVAL, from the stream that had just stopped.

After the crash

  1. apple_aop sets crashed and logs "AOP firmware crashed". The comment at Aop::crashed says nothing restarts the firmware short of a reboot, and that is what happens.
  2. Each later call fails with "Device is crashed, cannot send message", but seems to keep its slot on endpoint 34. After about eight calls every call fails with "Too many inflight calls on endpoint 34" instead. The result is the same, but the log no longer says the firmware is dead.
  3. Not the kernel's, noted for completeness: speakerguardd's PipeWire graph adds the speaker PCM as a required object. Once it cannot open, PipeWire fails to create its context and exits, so headphones and Bluetooth go silent too. I'll raise that with speakerguardd separately.

Questions

  • Does the speaker service expect the serializer to leave pwrd before the host stops feeding it? On J616s, did anyone see the firmware tolerate a stopped playback DMA for ~100 ms while still in pwrd? If not, the fix may be to send pw1 before the DMA stops (a nonatomic trigger, or a stop path that powers down first).
  • Is there a known ordering or pacing requirement for pw1/pwrd on T6041's AOP firmware that differs from the J616s the backend was qualified on? The re-grant crashes (1, 4) also contain a stop followed by a start.
  • Would an AOP restart after a crash be acceptable in principle, so a crash costs a second of audio rather than a reboot? I tried the closest thing available today, an unbind and rebind of apple_aop after crash 5. The unbind only finished after PipeWire, WirePlumber and alsactl released the card (until then it sat in snd_card_disconnect_sync). The rebind was then refused by design: the crashed firmware cannot acknowledge the RTKit shutdown, so remove() retains the shared buffers and sets RETIRED, and probe answers an earlier instance could not be shut down; reboot before probing again (-ENODEV). A restart would need a way to stop the co-processor's DMA without its cooperation (a CPU or power-domain reset of the ASC), and a firmware image to boot again, as far as I know only iBoot loads it. Is either available on T6041?

Next

A possible driver-side fix: on the playback front-end's STOP trigger, take the serializer down to pw1 promptly, after muting the amps, from a work item, since the trigger is atomic. The serializer would then never sit in pwrd without host data while userspace decides what to do next. The next prepare already brings it back through pwrd. This changes when the amps lose their clocks, though, so I'd like your view before I try it. I'm also looking at why PipeWire restarts the stream so often, which would shrink the problem from the other side.

Activity

  1. joske commented on Nov 18, 2026

    @joske
    Contributor

    Same here on another J614c, twice in two days (two different kernels on #1664). Both times the crash came right after playback stopped:

    Date speakerguardd-session … play=0 co-processor has crashed Gap
    16 Nov 23:35:38.315 23:35:38.339 24 ms
    18 Nov 09:23:41.973 09:23:42.010 37 ms

    The Ccst crash-log section is byte-identical in both. After the crash the speakers stay dead until a reboot, as you describe.

    🤖 Generated with Claude Code

  2. joske commented on Nov 18, 2026

    @joske
    Contributor

    A follow-up, with a before/after and a quick reproduction.

    Before speakerguardd v2 this never happened here. From 11 to 16 Nov the J614c speakers ran on a local test setup, for the last three days on the same kernel speaker path as now (aop_spkr.rs is identical between those kernels and my current one), with legacy speakersafetyd unlocking the volume once at startup and a −12 dB ceiling. I watched YouTube on it every night, starting and stopping playback about as often as now, and the AOP never crashed. Since switching to speakerguardd v2 on 16 Nov it has crashed three times, and now I can trigger it in a few minutes.

    Reproduction (kernel based on #1664, volume slider at about a third, YouTube in Firefox, pausing and resuming): fresh boot, six clean start/stop cycles, then during the seventh stream:

    09:34:56.066 kernel: Speaker protection granted
    09:35:07.297 pipewire: hw:CARD=AppleJ614,DEV=2c: follower delay:4456 target:1024 thr:512 resample:0, resync
    09:35:07.298 speakerguardd: protection released: Observation { heartbeat_fresh: false, ... }
    09:35:07.298 kernel: Speaker protection withdrawn: released by the owner
    09:35:07.306 kernel: Speaker protection granted (1 speaker starts refused before)
    09:35:08.983 kernel: apple_aop 510c10000.aop: RTKit: co-processor has crashed
    

    So this one came during playback, 1.7 s after a withdraw and re-grant caused by the sense-capture resync, not right after a stop. My two earlier crashes were 24 and 37 ms after a stop. All three have the same Ccst section.

    The volume was low (the slider at about a third), so the level is unlikely to matter. What the old setup didn't do is the grant/withdraw cycle: it unlocked once and stayed unlocked across stream stops. Every crash I have follows either a stream stop or a withdraw and re-grant.

    🤖 Generated with Claude Code

  3. joske commented on Nov 18, 2026

    @joske
    Contributor

    One more data point, which supports the stop path as the trigger.

    With a silent stream kept on the speakers, the AOP survives:

    pw-cat -p -a --format s16 --rate 48000 --channels 2 /dev/zero
    

    On a fresh boot with that running, I paused, resumed and scrubbed YouTube as before for about 6 minutes. Not a single play=0: the speaker stream never stopped. There were five sense-capture laps (DEV=2c follower delay:4480-8000 … resync), each with a protection withdraw and re-grant 8-80 ms later, and no crash. The last two came 2 s apart, the second with delay:8000.

    Without it, each of the three boots before crashed: twice 12 and 37 ms after a logged stream stop, once 1.7 s after a withdraw and re-grant (the 09:35 one above).

    So a withdraw/re-grant alone doesn't reliably crash it; the 09:35 crash may have needed something else in that window. The stop is what kills it here, which fits your plan to change the STOP trigger path.

    For anyone else hitting this, the pw-cat line above works as a stopgap.

    🤖 Generated with Claude Code

  4. RyanTheTide commented on Nov 19, 2026

    @RyanTheTide
    Contributor

    Thanks to both of you for the detailed logs. Here is a code read of the driver side, with no audio run. Point by point:

    1. The "Too many inflight calls on endpoint 34" after the crash. That is a driver bug, and #2342 fixes it. Each EPIC call reserved a tag before rtkit refused its doorbell ("Device is crashed, cannot send message"). The call was then marked abandoned, so its tag would be kept until a reply came, and a crashed AOP never replies. Eight calls later the table was full. With the PR, every call after the crash fails at once with ENODEV and reserves nothing, so spkr pw1/idle keep reporting the crash instead of a full table. The crash itself and the power sequence are unchanged.

    2. What the logs point at. Eight of the nine crashes in the two reports came within 200 ms of a stream stop or restart (12-37 ms after a play=0 stop in the clearest ones). The silent pw-cat stream that keeps the speaker PCM running survives about 6 minutes of the same pause/resume/scrub use with five sense-ring laps. In aop_spkr.rs a front-end STOP only stops the ADMAC (snd_dmaengine_pcm_trigger). The serializer stays in pwrd until be_hw_free (stop(), pw1) or the next be_prepare (run(), pw1 then pwrd), and PipeWire leaves 50-100 ms or more between the two. So the window that matters is a running serializer with no host DMA, which is what @mouthmantra suspected. No spkr call was in flight in crash 5 and crash 6, and that fits.

    3. On the proposed fix (power down on STOP). The direction looks right to me. Some constraints from the current code:

    • The trigger runs in atomic context. A work item from it races the next be_prepare, which may already be sending pw1/pwrd for the restart. run() would have to flush or cancel that work under the service lock before it powers the serializer back up. Otherwise a late pw1 from the work item lands after the restart's pwrd and stops a stream that is playing.
    • The other option is a nonatomic back-end trigger (nonatomic = 1 on the DAI link). Then STOP can mute the amps (mute_codecs()) and send pw1 inline. The firmware answers pw1 in about 1-2 ms in the timings above, so that fits in a trigger. With DPCM the front-end link has to be nonatomic as well.
    • The sense ring has to stop with it (stop_locked already does that after pw1), and speaker protection has to see the stop as a withdrawal it can reverse, as be_hw_free does today.
    • This lives in the shared AOP speaker backend, not in a J614c-only path. J616s and the other spkr boards get the same change, and it is decided by the service's presence, not by board.

    4. Restarting a crashed AOP. Today, no. The ASC would have to stop without the firmware's cooperation, and the AOP firmware image would have to be loaded again. As far as we know only iBoot loads it, so a reboot is the only recovery. That is why the driver should not drive the firmware into this state in the first place (point 3).

    Not tested on hardware. No sound was played.

  5. jgoyvaerts commented on Nov 19, 2026

    @jgoyvaerts
    Contributor

    The same crash on a J616s (14-inch MacBook Pro, M4 Pro, T6040), so not only J614c. It points at the shared aop_spkr.rs stop path, as in the code read above.

    Setup: our own 7.3.0 kernel (integration/aurora bf423bf plus local G17 GPU patches, aop_spkr.rs unchanged), speakerguardd-v2 1.0.0alpha.1-41, PipeWire 1.6.10, WirePlumber 0.5.19.

    A game (CS2 under FEX) played on the speakers for 78 s with no sense resync and no withdraw/re-grant, then quit. The crash came 28 ms after the stream stop:

    13:55:08.614 kernel: snd-soc-macaudio sound: Speaker protection granted
    13:56:26.103 speakerguardd-session: granted=1 links=1 carrier=1 play=0 report=ok
    13:56:26.131 kernel: apple_aop 510c10000.aop: RTKit: co-processor has crashed     [37663.79 s after boot]
    13:56:26.133 kernel: apple_aop 510c10000.aop: RTKit: Message (id=1): Global IRQ status 0x00000001 seen.
    13:56:26.167 kernel: apple_aop 510c10000.aop: AOP firmware crashed
    13:56:26.220 kernel: apple_aop 510c10000.aop: RTKit: Device is crashed, cannot send message
    13:56:26.220 kernel: snd_soc_apple_aop audio.6.auto: spkr pw1 failed: EINVAL
    13:56:26.222 kernel: snd_soc_apple_aop audio.6.auto: spkr idle failed: EINVAL
    13:56:26.277 kernel: snd-soc-macaudio sound: Speaker protection withdrawn: released by the owner
    13:57:33.090 kernel: apple_aop 510c10000.aop: Too many inflight calls on endpoint 34
    

    (UTC.) After that, speakerguardd loops "protection granted" -> "released: heartbeat_fresh: false" every 4 s, opens of the speaker and sense PCMs fail with EINVAL and then EIO, and the speakers stay silent until a reboot. The SCM i2c controllers report 0 errors in the crash dump, as in your logs.

    The same boot had about a dozen earlier game start/quit cycles plus ~10 h of desktop audio without a crash. It is the first AOP crash in our last 15 boots.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions