Repository navigation
Description
Mac and installed build
Model / board / SoC: MacBook Air (13-inch, M3, 2024) / apple,j613 / apple,t8122
Installed Aurora tag: none; kernel is linux-aurora 7.2.11.aurora14-1 (Omarchy 4.5.0-3 candidate repos)
uname -r: 7.2.11-14-1-ARCH (built Sun Nov 1 05:27:41 UTC 2026)
Bootloader / packages:
uboot-aurora 2026.07.aurora6-1
m1n1-aurora 1.6.1.aurora22-1
omarchy-aurora-firmware 20261014-7
mesa-aurora 1:26.3.0_devel.20261031.231483.a1969111475-1
Distribution / local changes: Arch Linux ARM (Omarchy 4.5.0-3); no local kernel patches.
/proc/cmdline has no asahi.* overrides; dmesg shows the board default cap:
"performance cap 8 lowered to 2 (asahi.t8122_pstate_cap)"
GPU: asahi 292400000.gpu, M3 G15G, T8122 C0, "firmware-compat 14.8.3 (start experiment)"
Both firmware versions and provisioning
asahi,os-fw-version: 14.7
asahi,system-fw-version: unknown
The asahi,system-fw-version property is present and readable; its value is the
literal string unknown on this machine. Calibration/provisioning is not
related to this report and was not touched.
Reproduction, expected result, and actual result
Expected: the compositor keeps running.
Actual: the G15G co-processor firmware crashes during a Hyprland render job.
The driver reports EIO and marks the device lost; M3 firmware recovery state
shows halt_count=0 halted=0 resume=0, i.e. the firmware never resumes.
Hyprland then aborts on its GL assert, SDDM's replacement greeter cannot get a
renderer (No such device / EGL_NOT_INITIALIZED), and the display stays dead
until a hard power-off.
Shortest reproduction: none found. The failure is spontaneous under ordinary
desktop use (Hyprland, Chromium, terminals; no game, no screen recording, no
external display). It has now occurred three times on three different boots:
- 2026-11-12 15:22:22 PST — uptime ~4 days, Hyprland pid 26643
- 2026-11-14 13:03:49 PST — uptime 1 day 21 h, Hyprland pid 1050
- 2026-11-16 10:19:02 PST — uptime 1 day 21 h 13 m, Hyprland pid 1013
Frequency: 3 failures in 3 consecutive boots / ~8 days of uptime. No kernel,
Mesa, firmware, or userspace update was installed between the incidents (last
full system upgrade in /var/log/pacman.log was 2026-11-08), so all three ran
the versions shown above. Last known-good build is unknown: journal retention
on this machine covers only these three boots, and all are affected.
Each time the journal goes from normal thermal summaries straight into
RTKit: co-processor has crashed with no preceding warning. Nothing was
suspended immediately beforehand and the SoC was at 44-48 C.
Graphics or session details
Compositor: Hyprland 0.56.2-7 (efb50993780079460b0cbed1363e2166a2de1d9f), Wayland via uwsm
Mesa: mesa-aurora 1:26.3.0_devel.20261031.231483.a1969111475-1 (devel snapshot)
Active renderer: "Renderer: Apple M3 (G15G C0)" (Aquamarine GL path)
Lock screen: none installed (hyprlock not present); none of the failures happened at lock/unlock
GPU profile / opt-in: stock; no asahi.* kernel arguments
Crash executable: /usr/bin/Hyprland, signal 6 (ABRT, SI_TKILL), command line "Hyprland --watchdog-fd 4"
Failing GPU work owner in all incidents: process Hyprland (VM=4), i.e. the compositor itself
The secondary Hyprland aborts are the crash reporter and the SDDM greeter; the
initiating failure is the GPU firmware crash. Core dumps were not attached
(per the reporting guidance).
Relevant error lines
Incident 1 — 2026-11-12 15:22:22 PST (journalctl -b -2 -k):
Nov 12 15:22:22.607842 asahi 292400000.gpu: RTKit: co-processor has crashed
Nov 12 15:22:22.608828 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x1fd
Nov 12 15:22:22.609345 asahi 292400000.gpu: RTKit: Version: RTKit: RTKit-2419.140.12.release - Client: not set
Nov 12 15:22:22.656659 asahi 292400000.gpu: M3 G15S: firmware crashed, retained crashlog bytes=16384
Nov 12 15:22:22.656816 asahi 292400000.gpu: M3 failed work owner: VM=4 process Hyprland pid 26643 thread Hyprland (error EIO, 2 commands in flight)
Nov 12 15:22:22.656958 asahi 292400000.gpu: M3: first GPU failure: EIO (GPU work pending: true)
Nov 12 15:22:22.657090 asahi 292400000.gpu: M3 G15G verdict: job-faulted (EIO): the firmware crashed, reported an error, or the GPU reported a fault while the job ran; see the lines around this one
Nov 12 15:22:22.670687 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x100e4b04000 draw=29556735 batch=1
Nov 12 15:22:22.670825 asahi 292400000.gpu: M3 fault pass slot=3 compact_vdm=0x80c00000
Nov 12 15:22:22.670959 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=509 read=511
Nov 12 15:22:22.671109 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=506 read=510
Nov 12 15:22:22.678398 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 completed_events=56988966 event_messages=57142127
Nov 12 15:22:22.678528 asahi: GPU work of a client failed (EIO, reported at m3/m3_submit.rs:101); the client sees a lost device
Nov 12 15:22:22.678550 asahi: M3 scheduler: submission failed EIO
Nov 12 15:22:22.678570 asahi 292400000.gpu: [File 7 Queue 2]: IOCTL: submit failed! (submission ID: 29556724 err: EIO)
Incident 2 — 2026-11-14 13:03:49 PST (journalctl -b -1 -k):
Nov 14 13:03:49.289534 asahi 292400000.gpu: RTKit: co-processor has crashed
Nov 14 13:03:49.290471 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x4ff
Nov 14 13:03:49.291001 asahi 292400000.gpu: RTKit: Version: RTKit: RTKit-2419.140.12.release - Client: not set
Nov 14 13:03:49.346718 asahi 292400000.gpu: M3 G15S: firmware crashed, retained crashlog bytes=16384
Nov 14 13:03:49.346855 asahi 292400000.gpu: M3 failed work owner: VM=4 process Hyprland pid 1050 thread Hyprland (error EIO, 2 commands in flight)
Nov 14 13:03:49.346990 asahi 292400000.gpu: M3: first GPU failure: EIO (GPU work pending: true)
Nov 14 13:03:49.347128 asahi 292400000.gpu: M3 G15G verdict: job-faulted (EIO): the firmware crashed, reported an error, or the GPU reported a fault while the job ran; see the lines around this one
Nov 14 13:03:49.359366 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x1004d74c000 draw=12779520 batch=1
Nov 14 13:03:49.359465 asahi 292400000.gpu: M3 fault pass slot=5 compact_vdm=0x80c00000
Nov 14 13:03:49.361649 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=1279 read=1
Nov 14 13:03:49.361756 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=1276 read=0
Nov 14 13:03:49.368716 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 completed_events=24575828 event_messages=24619486
Nov 14 13:03:49.369153 asahi: GPU work of a client failed (EIO, reported at m3/m3_submit.rs:101); the client sees a lost device
Nov 14 13:03:49.369171 asahi: M3 scheduler: submission failed EIO
Nov 14 13:03:49.369188 asahi 292400000.gpu: [File 9 Queue 2]: IOCTL: submit failed! (submission ID: 12779524 err: EIO)
Incident 3 — 2026-11-16 10:19:02 PST (journalctl -b -1 -k):
Nov 16 10:19:02.073530 asahi 292400000.gpu: RTKit: co-processor has crashed
Nov 16 10:19:02.074140 asahi 292400000.gpu: RTKit: Message (id=1): DAG: Non Sequential Stamp Updates seen entryIdx 0x1 roots.dag 0x0 stampIdx 0x1 stampValue 0x100 channel 0xfffffc200051bf4c channelRingCommandIndex 0x4ff
Nov 16 10:19:02.074339 asahi 292400000.gpu: RTKit: Version: RTKit: RTKit-2419.140.12.release - Client: not set
Nov 16 10:19:02.105194 asahi 292400000.gpu: M3 G15S: firmware crashed, retained crashlog bytes=16384
Nov 16 10:19:02.105268 asahi 292400000.gpu: M3 failed work owner: VM=4 process Hyprland pid 1013 thread Hyprland (error EIO, 2 commands in flight)
Nov 16 10:19:02.105344 asahi 292400000.gpu: M3: first GPU failure: EIO (GPU work pending: true)
Nov 16 10:19:02.105417 asahi 292400000.gpu: M3 G15G verdict: job-faulted (EIO): the firmware crashed, reported an error, or the GPU reported a fault while the job ran; see the lines around this one
Nov 16 10:19:02.112283 asahi 292400000.gpu: M3 fault render VM=4 slot=1 root=0x10059a54000 draw=12779520 batch=1
Nov 16 10:19:02.112356 asahi 292400000.gpu: M3 fault pass slot=5 compact_vdm=0x80c00000
Nov 16 10:19:02.112428 asahi 292400000.gpu: M3 TA stamps=0x3d000000/0x3d000000 done=1279 read=1
Nov 16 10:19:02.112503 asahi 292400000.gpu: M3 fragment stamps=0xfffffe00/0xfffffe00 done=1276 read=0
Nov 16 10:19:02.116405 asahi 292400000.gpu: M3 firmware recovery state: halt_count=0 halted=0 resume=0 info=0x0/0x0 packet=[00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00] completed_events=24569887 event_messages=24612972
Nov 16 10:19:02.116492 asahi: GPU work of a client failed (EIO, reported at m3/m3_submit.rs:101); the client sees a lost device
Nov 16 10:19:02.116510 asahi: M3 scheduler: submission failed EIO
Nov 16 10:19:02.116517 asahi 292400000.gpu: [File 8 Queue 2]: IOCTL: submit failed! (submission ID: 12779527 err: EIO)
All three report the same assert, and the firmware never resumes in any of
them. Incident 3 is effectively a replay of incident 2: same stampValue 0x100,
same channelRingCommandIndex 0x4ff, same draw=12779520 (0xC30000), identical
TA/fragment stamp state (0x3d000000 / 0xfffffe00), and a submission ID only
three apart (12779524 vs 12779527); only the render root address differs
(0x1004d74c000 vs 0x10059a54000). Incident 1 differs in
channelRingCommandIndex (0x1fd) and the draw/submission IDs, but its
draw=29556735 is exactly 0xffffff + 12779520 — the next 24-bit rollover of
the same offset. That is consistent with the deterministic rollover described
in the comments below, and incident 3 shows the boundary is reproducible rather
than probabilistic.
Downstream, the greeter fails with:
DRM_IOCTL_ASAHI_GET_PARAMS failed: No such device
Failed to query AGX device parameters: No such device
[EGL] Command eglInitialize errored out with EGL_NOT_INITIALIZED (0x12289): DRI2: failed to create screen
Hyprland's own crash backtrace (trimmed) is the render-path assert after the
device is gone:
#6 | Hyprland(Render::GL::CHyprOpenGLImpl::begin(...))
#7 | Hyprland(Render::GL::CHyprGLRenderer::beginRenderInternal(...))
#8 | Hyprland(Render::IHyprRenderer::beginRender(...))
#9 | Hyprland(Render::IHyprRenderer::renderMonitor(...))
Related issues, pull requests, or code examples
- No existing issue matches this signature: searching for
"Non Sequential Stamp Updates"and"firmware crashed"in this repository returns no
issues.G15Galso returns nothing. - drm/asahi: g13: defer freeing a completed command until its stamps land #1828 ("drm/asahi: g13: defer freeing a completed command until its stamps
land", merged 2026-11-13) fixes a same-class stamp-coalescing /
freed-descriptor-reuse wedge on the G13 legacy path, and notes the G17 path
has hold-until-covered logic from drm/asahi: g17: release pairs, queues and contexts held by stamps that never arrive #1823. The kernel here was built 2026-11-01,
so it predates drm/asahi: g13: defer freeing a completed command until its stamps land #1828. This is a possible relation, not a diagnosis: the G15G
path already has the stamps-owed logic and still hits a firmware assert, so
the failure may be a distinct G15 firmware issue. Mentioning it in case the
firmware assertion is the same stamp bookkeeping going wrong one layer lower. - drm/asahi: kill faulted G17 contexts before freeing their render work #505 ("drm/asahi: kill faulted G17 contexts before freeing their render work")
is related only in that it also handles a faulted render context. - Draft drm/asahi: m3: reseed completion stamps before rollover #1937 was not installed for any of the three incidents: the kernel here
was built 2026-11-01, and no package update has been installed since
2026-11-08. Incident 3 therefore neither confirms nor refutes that fix; it
only shows the pre-fix build still fails at the same boundary.
Workarounds tried: none; nothing was changed between the incidents, and no
update was installed between them.
What was not tested
- No reproduction on demand; no bisect, and no last-known-good build (journal
retention covers only the three affected boots). - No software-rendering or older-kernel/mesa session to isolate firmware vs.
Mesa command stream. In particular, drm/asahi: m3: reseed completion stamps before rollover #1937 has not been tested on this
hardware. - Only the internal panel (eDP-1) was in use; no external display, dock, or
hotplug was involved. - No suspend/resume test around the failure. None of the incidents was preceded
by suspend; one boot reached the sleep targets only after the compositor died. - The retained 16 KiB firmware crashlog was not captured (debugfs is root-only);
no other Mac was tested.
Activity
DjDeveloperr commentedon Nov 14, 2026
I traced both reported draw counts to the M3 fragment-stamp rollover: 12,779,520 and 29,556,735 are successive boundaries of the old helper's 0xffffff period. At each boundary it advances 0xffffff00 → 0x100 by 0x200, matching the firmware's non-sequential-stamp error. The same helper is still present in current integration.
I've opened draft fix #1937, crediting @yuffington's report. It drains active work before reseeding render/compute stamps and preserves host counters and queue state. All nine selected M3 host runs pass, with negative controls for the old rollover and unsafe live reseeding. Enabled-backend compilation is queued, and affected M3 Air hardware qualification remains pending; this is not yet a confirmed hardware fix.
yuffington commentedon Nov 16, 2026
Recurrence on the same pre-fix build — third failure, 2026-11-16.
@DjDeveloperr: this is a data point for #1937, though not a test of it. The machine is still on
linux-aurora 7.2.11.aurora14-1(built 2026-11-01) with no package update installed since 2026-11-08, so the draft fix was never running here.Notable: this incident is nearly an exact replay of the 2026-11-14 one, not just the same class of failure. Same
stampValue 0x100, samechannelRingCommandIndex 0x4ff, samedraw=12779520(0xC30000), identical TA/fragment stamp state, and submission ID only three apart (12779524 vs 12779527). Only the render root differs (0x1004d74c000vs0x10059a54000). The 2026-11-12 incident haddraw=29556735, which is exactly0xffffff + 12779520— the next 24-bit rollover of the same 0xC30000 offset.So the boundary looks deterministic/reproducible rather than probabilistic, which should make hardware qualification easier: if #1937 is correct, this machine should cross the same
0xC30000boundary without the assert. Uptime at failure was 1 d 21 h 13 m; no suspend beforehand, SoC 44-48 C, work owner Hyprland (VM=4) again. I have updated the issue body with the full log block and details.drm/asahi: m3: reseed completion stamps before rollover