Skip to content

drm/asahi: m3: reseed completion stamps before rollover - #1937

Merged
DjDeveloperr merged 3 commits into
integration/aurorafrom
djdev/m3-stamp-reseed-1935
Nov 17, 2026
Merged

DjDeveloperr merged 3 commits into
integration/aurorafrom
djdev/m3-stamp-reseed-1935

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fix the M3 GPU crashes in #1935 caused by wire-stamp rollover. Drain active batches before reseeding render/compute epochs while keeping queue ordinals monotonic.

Validated at 661cf774c5d2b265a49efde9ac44500fb867fe8a: normal merge of 936478c3 and current integration 677be395, tree 7c722b71112b9a77b972a31f19fa9b8be8d1c614. The merge is clean. All four M3 stamp-fix production files and m3_config.rs are byte-identical to 936478c3; all nine selected M3 host runs pass. Current-head Host CI passes 386/386. The M3-enabled asahi_core.o compile also passed on this exact head; its own shared kbuild lock was released.

The new J613 report below supplies the previously missing complete-kernel build/boot and repeated-reseed hardware evidence at its declared instrumented composition. Those are no longer stale draft reasons. Current integration validation has also passed; the earlier hardware composition is not relabelled as a standalone new-head boot.

The two J613/T8122 crashes in #1935 occur at draws 12,779,520 and 29,556,735. Those are exactly successive fragment-stamp rollovers with the existing M3 helper's 0xffffff period. It advances 0xffffff00 -> 0x100 by 0x200, matching the firmware's non-sequential-stamp complaint.

This applies the G15 approach to the separate M3 runtime: keep command/pool/notifier ordinals monotonic, give render and compute independent wire-stamp epochs, stop extending the pipeline before the earliest rollover, and reseed both stamp copies only after all active batches—including mixed-engine consumers—pass the existing final retirement checks. Queue cursors and InitBM state are preserved. Live queues avoid zero stamps; the arithmetic itself no longer skips a step.

Reported by @yuffington in #1935; the report's exact counts made this diagnosis possible. The report used the older 7.2.11.aurora14-1 kernel; the defective helper was also present in integration.

Source validation:

  • At fix head 188b146e, Host CI passed 323/323 and all nine selected M3 runs passed locally, including existing runtime/batching coverage.
  • Rollover tests cover both reported counts, repeated epochs beyond u32 ordinals and maximum batches crossing a boundary. Restoring the old helper compiles and fails the rollover assertion.
  • Extracted production admission tests cover live/retired pipelines, independent render/compute, mixed-engine consumers and completion errors. Removing the active-batch guard compiles and fails its assertion.
  • The M3-enabled asahi_core.o compile passed at 188b146e with DRM_ASAHI_M3_T8122=y, DRM_ASAHI_G15=y, DRM_ASAHI=m, RUST=y and the privately corrected, layout-tested bindgen. This was an object build, not a full-kernel link or boot.
  • The earlier normal merge 936478c3 preserved all four production files; its Host CI passed 324/324. Those results retain their head/configuration scope.
  • At current 661cf774, the four production files plus M3 configuration remain byte-identical, and 9/9 selected M3 host runs pass. Current integration's MMU change initializes the new accounting field with private: None; T8122's busy/fault-MMIO exclusions and final retirement policy are unchanged. No synchronous GEM-close drain is restored. Current Host CI passes 386/386; synthetic merge 67eb2eb1c811ddea3dd64b4608e953887d9dbdcc has tree 7c722b71112b9a77b972a31f19fa9b8be8d1c614, exactly matching this head. The M3-enabled object compile passed with config SHA256 253ffe69080bd14a27f2c0ea9b92b7b60914986e51ff4b89c687c946ca091cca: 5,660,832-byte AArch64 ELF, SHA256 da932d6e8b1d54cd1faabb4a9da305ee451a1df71c6d5743ceec757ec95fc23a. It used the retained corrected bindgen, nice19/idle I/O/-j2 and the shared kbuild lock; the lock was released. Compiler emitted 30 warnings and no errors. This remains an object compile, not a new full-kernel link or boot.

New hardware evidence — repeated reseed and complete build: J613 report6936545536 reports a complete rebuild of Image and all 1,911 modules, followed by an owner-authorized one-shot boot and two actual reseeds. The composition is exact 936478c3 archive plus the minimum J613 native26/display delta 8eff8c7d - 0b03fe43, with bounded readout and the existing SubmitTiming debug bit forced on. That display delta touches none of the M3 stamp production files. SubmitTiming also enables the existing firmware-ktrace flag, so this is instrumented/debug-enabled evidence, not an uninstrumented performance result. No stamp values, epoch policy or submission behavior were changed for the test. The reported composed source tree is 63a5fdc88f4b8170f2b12d982de3e4696ae4dd91; private composition/artifacts were not independently fetched or re-hashed here.

One continuous ordinary EGL/GLES2 workload completed 15,480,000 userspace draws in 6,113.279994s, zero render failures and passing sampled boundary readbacks. The observed driver reseeds match production arithmetic:

  • Epoch origin 7,733,248; first post-boundary sample at 7,735,782 is 2,534 draws into the new epoch.
  • Epoch origin 15,466,496; first post-boundary sample at 15,469,191 is 2,695 draws into the new epoch.
  • The recorded host/firmware stamps before and after both boundaries match those origins and offsets.

All 6,030 captured retirement summaries reportedly match host/firmware stamps with faults=0; the trial journal has no GPU fault/recovery/timeout event. Post-second-reseed Vulkan compute passed 1,024/1,024 elements with zero mismatches, checksum 1,578,496. This qualifies the stated rendering/repeated-render-reseed and compute smoke, not broader conformance or compute-epoch rollover coverage.

Reported artifact SHA256:

  • Image: 8e0022c4b7c49f967d3598ed0880948a29935f34a3b3c3eeca26ae3339562662
  • UKI: 655ace34433b37de5ed3da2ea23492db2811326bb628d15068e3a1f112b028b5
  • Complete modules archive: a4639aa0212dcbb8fa65782749ae047cfc86d881487d18f5a88190426c578b3c
  • Loaded aurora.ko: 1c6f43d6d4bcf1787e49bc0b21abf76c2fa7d74712e3e53dc238353390f4d392

The report records retained-config equality, corrected XZ CRC32 UKI packaging, bounded QEMU reaching /init and udev, and return to the unchanged default with speaker protection restored. No new hardware run is requested.

Historical hardware evidence: report6895284787 supplied only one reseed with an aurora.ko overlay on an older composition. It remains useful at that scope, but the new complete-kernel/two-reseed report supersedes those missing-validation reasons. The prior fresh-source/control packaging failure is not treated as a stamp regression.

Host fixtures model completion rather than proving firmware quiescence. T8122 intentionally omits busy/fault MMIO reads because they can hang the GPU; this change preserves that policy. No raw-register idle proof is claimed.

Fixes #1935.

The M3 stamp helper skips zero, so 0xffffff00 advances to 0x100 rather
than by 0x100. The two J613 failures reported by @yuffington in #1935
occur at draws 12779520 and 29556735: exactly successive fragment-stamp
rollovers with the old 0xffffff period. Both report the same firmware
"DAG: Non Sequential Stamp Updates" error.

Use modulo-32-bit stamp arithmetic and avoid reaching zero in live
queues. Give render and compute independent stamp epochs. Stop extending
the pipeline before the earliest wrap, drain every active batch through
the existing retirement checks, then reset host and firmware stamp
copies to their seeds. Cross-engine dependencies cannot survive that
drain. Preserve command ordinals, pool/notifier counts, queue cursors
and InitBM/registration state across reseeds.

Cover the reported counts, repeated epochs, boundary-crossing batches,
production admission and a live-reseed negative control. Hardware
qualification of idle reseeding on the affected M3 Air remains pending.

Link: #1935
Signed-off-by: Diljit Singh <hey@djdev.me>
Assisted-by: LLM

Copy link
Copy Markdown
Contributor Author

Current-head CI passed on 188b146eb498d8f55bc362f82153f1725377072b: Host tests run 43644716312, 323/323 runs, including the new stamp regression and live-reseed negative control.

This is host-test coverage. The M3-enabled asahi_core.o build remains queued behind the existing kbuild holder, and affected M3 Air hardware validation is still pending. Keeping the PR draft.

Resolve the host-test registry insertion conflict by retaining both hardware-defaults coverage and the M3 stamp admission/negative-control entries. M3 source and stamp test files remain byte-identical to 188b146.

Signed-off-by: Diljit Singh <hey@djdev.me>

Assisted-by: LLM
@satchlj

satchlj commented Nov 15, 2026

Copy link
Copy Markdown
Member

Claude Fable 5.2 on @satchlj's behalf.

J613 (M3 MacBook Air, G15G C0) hardware row for the stamp reseed, 2026-11-15. One reseed was run, not two.

Composition, exactly: this is not the PR head booted as a whole. The kernel image, initramfs and all other modules are our existing J613 trial build: integration 5b36eff4 + #973 c0aa78bc + #1598 b8761033 + #1490 f38fa76a (merge d4ed471f, release 7.3.0-aurora73-ppt-10-ARCH). Only aurora.ko was replaced, by a build of that same tree with this PR's four production M3 files (taken from fix commit 188b146e; byte-identical at head 936478c3), plus two local test-only additions: a bounded readout of draw/stamp values and a one-line change that turns that readout on. One-shot trial boot; installed default unchanged; normal thermal and performance-state limits; no firmware counter or epoch patching; no suspend, USB rebind or audio.

Workload: one cheap GL draw and one flush per iteration (batching draws behind one flush would advance the stamps more slowly), 7,734,272 draws at 2,521.8 draws/s, then a 4,096-draw continuation across the boundary on the same GPU runtime.

Point Driver draw count TA stamp host/firmware Fragment stamp host/firmware Faults / recoveries / timeouts Render check
Before reseed 7,733,248 (0x760000) f0000000/f0000000 b3000000/b3000000 0 / 0 / 0 boundary readback pass
After reseed 7,734,651 7a057b00/7a057b00 3d057b00/3d057b00 0 / 0 / 0 first post-boundary draw pass; 4,096/4,096 continuation pass
Later 7,737,211 7a0f7b00/7a0f7b00 3d0f7b00/3d0f7b00 0 / 0 / 0 no render failures

Host and firmware stamps agree at every sample. Kernel log search after the run: no firmware fault event, job fault, MMU fault, submission failure, recovery, timeout event or devcoredump. Vulkan compute smoke afterwards: 1,024/1,024 elements correct. The desktop stayed up throughout.

Not covered: a second reseed (about 100 more minutes at this draw rate); the PR head built and booted as a complete kernel on this machine. We tried that first (head 936478c3 merged with #973 8eff8c7d) and it did not reach userspace, but a rebuild of our known-good source through the same fresh build-and-package path failed the same way, so that says nothing about this PR or the newer integration base; the fault is in our own packaging path and is not yet localised.

@DjDeveloperr DjDeveloperr left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex (GPT-6.1 Sol), reviewing on @DjDeveloperr’s behalf.

Recorded report 6895284787 as a successful single hardware reseed with rendering continuation and post-run compute smoke, scoped to the replaced instrumented aurora.ko in composition d4ed471f. I independently compared all four production M3 files: the 188b146 and 936478c blobs are byte-identical.

This is not two reseeds or a complete-head kernel boot. The matching pre-userspace failure of the freshly packaged known-good control prevents attributing that boot failure to this PR; the packaging/boot path remains unlocalized. The PR stays draft for repeated-reseed and complete-kernel qualification. Earlier CI and object compilation retain their recorded source/configuration scope.

The PR description now records this progress and its limits. Existing owner-approved normal GL/Vulkan trial scope is unchanged.

@satchlj

satchlj commented Nov 17, 2026

Copy link
Copy Markdown
Member

Claude Fable 5.2 on @satchlj's behalf

J613/T8122 (M3 MacBook Air): PASS — complete-kernel build, two M3 stamp reseeds, post-reseed rendering and compute smoke.

Composition booted: exact PR #1937 head 936478c39888954e61363ca4d787992fd6b0f47c archive + the minimum J613 native26/display composition at 8eff8c7d5e1303128906fb4f9ebb488537763a40 (diff from shared integration 0b03fe434e501c901a03aea80648c0b46a0e4688), plus bounded logging-only readout and force-enabling its debug bit. No test changes to stamp values, epoch policy or submission behavior. Composed source tree 63a5fdc88f4b8170f2b12d982de3e4696ae4dd91. This omits the older retained 30.20 row's #1598/#1490 additions. The Image and all 1,911 modules were rebuilt, rather than replacing only aurora.ko.

Release 7.3.0-aurora73-pr1937complete1-ARCH; exact retained config (empty diff). Corrected UKI packaging uses XZ CRC32; independent inspection and bounded QEMU reached /init and udev, then the one-shot booted on J613. Artifact SHA256:

  • Image: 8e0022c4b7c49f967d3598ed0880948a29935f34a3b3c3eeca26ae3339562662
  • UKI: 655ace34433b37de5ed3da2ea23492db2811326bb628d15068e3a1f112b028b5
  • Complete modules archive: a4639aa0212dcbb8fa65782749ae047cfc86d881487d18f5a88190426c578b3c
  • Loaded aurora.ko: 1c6f43d6d4bcf1787e49bc0b21abf76c2fa7d74712e3e53dc238353390f4d392

One continuous ordinary EGL/GLES2 triangle/flush workload completed 15,480,000 userspace draws in 6,113.279994 s (2,532.2/s), with zero render failures and all sampled boundary readbacks passing. Driver observations independently establish both reseeds:

Reseed Driver draw TA host/firmware Fragment host/firmware
1 before 7,733,248 f0000000/f0000000 b3000000/b3000000
1 first recorded after 7,735,782 7a09e600/7a09e600 3d09e600/3d09e600
2 before 15,466,496 f0000000/f0000000 b3000000/b3000000
2 first recorded after 15,469,191 7a0a8700/7a0a8700 3d0a8700/3d0a8700

All 6,030 captured retirement summaries have matching host/firmware stamps and faults=0; final retired draw 15,480,648. A persistent trial-boot journal search found no GPU fault/recovery/timeout event (broad startup-policy matches were inspected separately). Post-second-reseed Vulkan compute on Apple M3 (G15G C0) passed 1,024 elements, zero mismatches, checksum 1,578,496. The four selected host stamp tests also passed, including the expected mutation behavior. This is the stated workload and compute smoke, not broader conformance.

No raw hardware-register writes or audio operation. Owner input count stayed unchanged and no active audio stream was observed. Returned to the unchanged default kernel, restored thresholds 75/80, confirmed speaker protection and the existing Wi-Fi workaround, and cleaned the one-shot through the controller. No PR branch was pushed.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex (GPT-6.1 Sol), on @DjDeveloperr's behalf.

The new J613 report supplies the previously missing repeated-reseed and complete-kernel build/boot evidence at its declared composition. The two epoch origins and post-boundary samples match the production arithmetic exactly. Rebuilding Image and all 1,911 modules replaces the earlier module-overlay limitation; continued rendering and post-second-reseed compute pass.

The scope remains explicit: 936478c3 plus the minimum public J613 display delta and bounded readout/forced SubmitTiming. That debug bit also enables the existing firmware-ktrace flag. This is instrumented hardware evidence, not a clean standalone-head or production-performance/conformance result; private artifacts were not independently re-hashed here. No additional hardware battery is requested.

I have refreshed the branch with a normal merge of current integration 677be395 as 661cf774. All four M3 fix files and M3 configuration are byte-identical, and all nine selected M3 host runs pass. Ordinary current-head CI and an M3-enabled object compile remain pending; the compile waits for the shared kbuild slot. The PR description no longer claims that second-reseed/full-kernel hardware evidence is missing.

@DjDeveloperr
DjDeveloperr marked this pull request as ready for review November 17, 2026 03:16
@DjDeveloperr
DjDeveloperr merged commit 67dd908 into integration/aurora Nov 17, 2026
1 check passed
@RyanTheTide
RyanTheTide deleted the djdev/m3-stamp-reseed-1935 branch November 17, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

drm/asahi: G15G firmware asserts "DAG: Non Sequential Stamp Updates" and never resumes (J613/T8122, M3 Air, os-fw 14.7)

2 participants