Skip to content

drm/asahi: G15G firmware asserts "DAG: Non Sequential Stamp Updates" and never resumes (J613/T8122, M3 Air, os-fw 14.7) #1935

Closed

Description

@yuffington

Mac and installed build

Model / board / SoC: MacBook Air (13-inch, M3, 2024) / apple,j613 / apple,t8122
Installed Aurora tag: none; kernel is linux-aurora 7.2.11.aurora14-1 (Omarchy 4.5.0-3 candidate repos)
uname -r: 7.2.11-14-1-ARCH  (built Sun Nov 1 05:27:41 UTC 2026)
Bootloader / packages:
  uboot-aurora 2026.07.aurora6-1
  m1n1-aurora 1.6.1.aurora22-1
  omarchy-aurora-firmware 20261014-7
  mesa-aurora 1:26.3.0_devel.20261031.231483.a1969111475-1
Distribution / local changes: Arch Linux ARM (Omarchy 4.5.0-3); no local kernel patches.
  /proc/cmdline has no asahi.* overrides; dmesg shows the board default cap:
  "performance cap 8 lowered to 2 (asahi.t8122_pstate_cap)"
GPU: asahi 292400000.gpu, M3 G15G, T8122 C0, "firmware-compat 14.8.3 (start experiment)"

Both firmware versions and provisioning

asahi,os-fw-version: 14.7
asahi,system-fw-version: unknown

The asahi,system-fw-version property is present and readable; its value is the
literal string unknown on this machine. Calibration/provisioning is not
related to this report and was not touched.

Reproduction, expected result, and actual result

Expected: the compositor keeps running.

Actual: the G15G co-processor firmware crashes during a Hyprland render job.
The driver reports EIO and marks the device lost; M3 firmware recovery state
shows halt_count=0 halted=0 resume=0, i.e. the firmware never resumes.
Hyprland then aborts on its GL assert, SDDM's replacement greeter cannot get a
renderer (No such device / EGL_NOT_INITIALIZED), and the display stays dead
until a hard power-off.

Shortest reproduction: none found. The failure is spontaneous under ordinary
desktop use (Hyprland, Chromium, terminals; no game, no screen recording, no
external display). It has now occurred three times on three different boots:

  • 2026-11-12 15:22:22 PST — uptime ~4 days, Hyprland pid 26643
  • 2026-11-14 13:03:49 PST — uptime 1 day 21 h, Hyprland pid 1050
  • 2026-11-16 10:19:02 PST — uptime 1 day 21 h 13 m, Hyprland pid 1013

Frequency: 3 failures in 3 consecutive boots / ~8 days of uptime. No kernel,
Mesa, firmware, or userspace update was installed between the incidents (last
full system upgrade in /var/log/pacman.log was 2026-11-08), so all three ran
the versions shown above. Last known-good build is unknown: journal retention
on this machine covers only these three boots, and all are affected.

Each time the journal goes from normal thermal summaries straight into
RTKit: co-processor has crashed with no preceding warning. Nothing was
suspended immediately beforehand and the SoC was at 44-48 C.

Graphics or session details

Compositor: Hyprland 0.56.2-7 (efb50993780079460b0cbed1363e2166a2de1d9f), Wayland via uwsm
Mesa: mesa-aurora 1:26.3.0_devel.20261031.231483.a1969111475-1 (devel snapshot)
Active renderer: "Renderer: Apple M3 (G15G C0)" (Aquamarine GL path)
Lock screen: none installed (hyprlock not present); none of the failures happened at lock/unlock
GPU profile / opt-in: stock; no asahi.* kernel arguments
Crash executable: /usr/bin/Hyprland, signal 6 (ABRT, SI_TKILL), command line "Hyprland --watchdog-fd 4"
Failing GPU work owner in all incidents: process Hyprland (VM=4), i.e. the compositor itself

The secondary Hyprland aborts are the crash reporter and the SDDM greeter; the
initiating failure is the GPU firmware crash. Core dumps were not attached
(per the reporting guidance).

Relevant error lines

Incident 1 — 2026-11-12 15:22:22 PST (journalctl -b -2 -k):

Nov 12 15:22:22.607842 asahi 292400000.gpu: RTKit: co-processor has crashed
Nov 12 15:22:22.608828 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x1fd
Nov 12 15:22:22.609345 asahi 292400000.gpu: RTKit: Version: RTKit: RTKit-2419.140.12.release - Client: not set
Nov 12 15:22:22.656659 asahi 292400000.gpu: M3 G15S: firmware crashed, retained crashlog bytes=16384
Nov 12 15:22:22.656816 asahi 292400000.gpu: M3 failed work owner: VM=4 process Hyprland pid 26643 thread Hyprland (error EIO, 2 commands in flight)
Nov 12 15:22:22.656958 asahi 292400000.gpu: M3: first GPU failure: EIO (GPU work pending: true)
Nov 12 15:22:22.657090 asahi 292400000.gpu: M3 G15G verdict: job-faulted (EIO): the firmware crashed, reported an error, or the GPU reported a fault while the job ran; see the lines around this one
Nov 12 15:22:22.670687 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x100e4b04000 draw=29556735 batch=1
Nov 12 15:22:22.670825 asahi 292400000.gpu: M3 fault pass slot=3 compact_vdm=0x80c00000
Nov 12 15:22:22.670959 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=509 read=511
Nov 12 15:22:22.671109 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=506 read=510
Nov 12 15:22:22.678398 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 completed_events=56988966 event_messages=57142127
Nov 12 15:22:22.678528 asahi: GPU work of a client failed (EIO, reported at m3/m3_submit.rs:101); the client sees a lost device
Nov 12 15:22:22.678550 asahi: M3 scheduler: submission failed EIO
Nov 12 15:22:22.678570 asahi 292400000.gpu: [File 7 Queue 2]: IOCTL: submit failed! (submission ID: 29556724 err: EIO)

Incident 2 — 2026-11-14 13:03:49 PST (journalctl -b -1 -k):

Nov 14 13:03:49.289534 asahi 292400000.gpu: RTKit: co-processor has crashed
Nov 14 13:03:49.290471 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x4ff
Nov 14 13:03:49.291001 asahi 292400000.gpu: RTKit: Version: RTKit: RTKit-2419.140.12.release - Client: not set
Nov 14 13:03:49.346718 asahi 292400000.gpu: M3 G15S: firmware crashed, retained crashlog bytes=16384
Nov 14 13:03:49.346855 asahi 292400000.gpu: M3 failed work owner: VM=4 process Hyprland pid 1050 thread Hyprland (error EIO, 2 commands in flight)
Nov 14 13:03:49.346990 asahi 292400000.gpu: M3: first GPU failure: EIO (GPU work pending: true)
Nov 14 13:03:49.347128 asahi 292400000.gpu: M3 G15G verdict: job-faulted (EIO): the firmware crashed, reported an error, or the GPU reported a fault while the job ran; see the lines around this one
Nov 14 13:03:49.359366 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x1004d74c000 draw=12779520 batch=1
Nov 14 13:03:49.359465 asahi 292400000.gpu: M3 fault pass slot=5 compact_vdm=0x80c00000
Nov 14 13:03:49.361649 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=1279 read=1
Nov 14 13:03:49.361756 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=1276 read=0
Nov 14 13:03:49.368716 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 completed_events=24575828 event_messages=24619486
Nov 14 13:03:49.369153 asahi: GPU work of a client failed (EIO, reported at m3/m3_submit.rs:101); the client sees a lost device
Nov 14 13:03:49.369171 asahi: M3 scheduler: submission failed EIO
Nov 14 13:03:49.369188 asahi 292400000.gpu: [File 9 Queue 2]: IOCTL: submit failed! (submission ID: 12779524 err: EIO)

Incident 3 — 2026-11-16 10:19:02 PST (journalctl -b -1 -k):

Nov 16 10:19:02.073530 asahi 292400000.gpu: RTKit: co-processor has crashed
Nov 16 10:19:02.074140 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x4ff
Nov 16 10:19:02.074339 asahi 292400000.gpu: RTKit: Version: RTKit: RTKit-2419.140.12.release - Client: not set
Nov 16 10:19:02.105194 asahi 292400000.gpu: M3 G15S: firmware crashed, retained crashlog bytes=16384
Nov 16 10:19:02.105268 asahi 292400000.gpu: M3 failed work owner: VM=4 process Hyprland pid 1013 thread Hyprland (error EIO, 2 commands in flight)
Nov 16 10:19:02.105344 asahi 292400000.gpu: M3: first GPU failure: EIO (GPU work pending: true)
Nov 16 10:19:02.105417 asahi 292400000.gpu: M3 G15G verdict: job-faulted (EIO): the firmware crashed, reported an error, or the GPU reported a fault while the job ran; see the lines around this one
Nov 16 10:19:02.112283 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x10059a54000 draw=12779520 batch=1
Nov 16 10:19:02.112356 asahi 292400000.gpu: M3 fault pass slot=5 compact_vdm=0x80c00000
Nov 16 10:19:02.112428 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=1279 read=1
Nov 16 10:19:02.112503 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=1276 read=0
Nov 16 10:19:02.116405 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 packet=[00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00] completed_events=24569887 event_messages=24612972
Nov 16 10:19:02.116492 asahi: GPU work of a client failed (EIO, reported at m3/m3_submit.rs:101); the client sees a lost device
Nov 16 10:19:02.116510 asahi: M3 scheduler: submission failed EIO
Nov 16 10:19:02.116517 asahi 292400000.gpu: [File 8 Queue 2]: IOCTL: submit failed! (submission ID: 12779527 err: EIO)

All three report the same assert, and the firmware never resumes in any of
them. Incident 3 is effectively a replay of incident 2: same stampValue 0x100,
same channelRingCommandIndex 0x4ff, same draw=12779520 (0xC30000), identical
TA/fragment stamp state (0x3d000000 / 0xfffffe00), and a submission ID only
three apart (12779524 vs 12779527); only the render root address differs
(0x1004d74c000 vs 0x10059a54000). Incident 1 differs in
channelRingCommandIndex (0x1fd) and the draw/submission IDs, but its
draw=29556735 is exactly 0xffffff + 12779520 — the next 24-bit rollover of
the same offset. That is consistent with the deterministic rollover described
in the comments below, and incident 3 shows the boundary is reproducible rather
than probabilistic.

Downstream, the greeter fails with:

DRM_IOCTL_ASAHI_GET_PARAMS failed: No such device
Failed to query AGX device parameters: No such device
[EGL] Command eglInitialize errored out with EGL_NOT_INITIALIZED (0x12289): DRI2: failed to create screen

Hyprland's own crash backtrace (trimmed) is the render-path assert after the
device is gone:

#6 | Hyprland(Render::GL::CHyprOpenGLImpl::begin(...))
#7 | Hyprland(Render::GL::CHyprGLRenderer::beginRenderInternal(...))
#8 | Hyprland(Render::IHyprRenderer::beginRender(...))
#9 | Hyprland(Render::IHyprRenderer::renderMonitor(...))

Related issues, pull requests, or code examples

Workarounds tried: none; nothing was changed between the incidents, and no
update was installed between them.

What was not tested

  • No reproduction on demand; no bisect, and no last-known-good build (journal
    retention covers only the three affected boots).
  • No software-rendering or older-kernel/mesa session to isolate firmware vs.
    Mesa command stream. In particular, drm/asahi: m3: reseed completion stamps before rollover #1937 has not been tested on this
    hardware.
  • Only the internal panel (eDP-1) was in use; no external display, dock, or
    hotplug was involved.
  • No suspend/resume test around the failure. None of the incidents was preceded
    by suspend; one boot reached the sleep targets only after the compositor died.
  • The retained 16 KiB firmware crashlog was not captured (debugfs is root-only);
    no other Mac was tested.

Activity

  1. DjDeveloperr commented on Nov 14, 2026

    @DjDeveloperr
    Contributor

    I traced both reported draw counts to the M3 fragment-stamp rollover: 12,779,520 and 29,556,735 are successive boundaries of the old helper's 0xffffff period. At each boundary it advances 0xffffff00 → 0x100 by 0x200, matching the firmware's non-sequential-stamp error. The same helper is still present in current integration.

    I've opened draft fix #1937, crediting @yuffington's report. It drains active work before reseeding render/compute stamps and preserves host counters and queue state. All nine selected M3 host runs pass, with negative controls for the old rollover and unsafe live reseeding. Enabled-backend compilation is queued, and affected M3 Air hardware qualification remains pending; this is not yet a confirmed hardware fix.

  2. yuffington commented on Nov 16, 2026

    @yuffington
    Author

    Recurrence on the same pre-fix build — third failure, 2026-11-16.

    @DjDeveloperr: this is a data point for #1937, though not a test of it. The machine is still on linux-aurora 7.2.11.aurora14-1 (built 2026-11-01) with no package update installed since 2026-11-08, so the draft fix was never running here.

    Nov 16 10:19:02.074140 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x4ff
    Nov 16 10:19:02.112283 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x10059a54000 draw=12779520 batch=1
    Nov 16 10:19:02.112428 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=1279 read=1
    Nov 16 10:19:02.112503 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=1276 read=0
    Nov 16 10:19:02.116405 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 completed_events=24569887 event_messages=24612972
    

    Notable: this incident is nearly an exact replay of the 2026-11-14 one, not just the same class of failure. Same stampValue 0x100, same channelRingCommandIndex 0x4ff, same draw=12779520 (0xC30000), identical TA/fragment stamp state, and submission ID only three apart (12779524 vs 12779527). Only the render root differs (0x1004d74c000 vs 0x10059a54000). The 2026-11-12 incident had draw=29556735, which is exactly 0xffffff + 12779520 — the next 24-bit rollover of the same 0xC30000 offset.

    So the boundary looks deterministic/reproducible rather than probabilistic, which should make hardware qualification easier: if #1937 is correct, this machine should cross the same 0xC30000 boundary without the assert. Uptime at failure was 1 d 21 h 13 m; no suspend beforehand, SoC 44-48 C, work owner Hyprland (VM=4) again. I have updated the issue body with the full log block and details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions