Skip to content

nx2: survive corrupted GPU commands, lock the kind map, fix swapchain recreation - #329

Open
failaip12 wants to merge 9 commits into
Eglantine-Community:nx2from
failaip12:fix/nx2-freeze-crashes
Open

failaip12 wants to merge 9 commits into
Eglantine-Community:nx2from
failaip12:fix/nx2-freeze-crashes

Conversation

@failaip12

@failaip12 failaip12 commented Nov 2, 2026 •

Copy link
Copy Markdown

Mario Kart World crashed about every 45 minutes. 11 of the 14 recorded crash stacks were in code fed by garbage GPU commands, and a 1x comparison turned up a race in the GPU memory manager. This PR fixes the race and two swapchain crashes, and stops garbage commands from taking the emulator down. Upstream's latest release most likely removes the garbage at its source, so on this base the guards are a safety net.

Changes

  • Guards: invalid syncpoint ids, GPU addresses that translate past the device space, memory walks past the GPU page tables and impossible render targets are now ignored. IsFullyMappedRange reports a rejected range as unmapped, not as fully mapped.
  • Swapchain: recreation retries transient errors (one Qt run aborted on VK_INCOMPLETE). If every retry fails, the next frame tries again, and a frame with no swapchain is dropped instead of crashing.
  • Race: MemoryManager::PageTableOp now takes the kind_map lock that every other use of the map already takes.
  • Diagnostics (separate commit): frame time percentiles, EDEN_NX2_FRAME_LOG, EDEN_NX2_TC_DUMP and slow zone logging.

Where the garbage commands came from

  • It's Eden's command parser getting out of step, not corrupted guest memory. The crashing render target size, 2147487260x2147485323, is exactly the command headers 0x80000E1C and 0x8000068B read as data. The syncpoint ids over 192 and the out-of-range device addresses fit the same pattern, but weren't traced to their source words.
  • The freezes follow the garbage, not the other way round. In the one session a probe caught, the parser desynced first. A channel then waited on a semaphore value it had read from misparsed words, and the 4 s present timeouts started 4 s later.
  • It needs a GPU-starved game. 1 of 17 probe sessions desynced, and only that one spent 2 minutes at 4-5 fps with the host GPU at 100% before it did.
  • Most likely cause: with the GPU behind, the game gets more than a frame ahead and rewrites command lists Eden hasn't read yet. Upstream found the same thing in Splatoon Raiders. That fits Mario Kart World's single catch but isn't proven for it, since the probe didn't record the first wrong step.

Upstream's fixes for it (in this base):

  • 0f14696044: when the GPU is behind at present, also wait for the presented frame.
  • cce6d44af2: execute each command list as it was submitted (EDEN_NX2_PB_SNAPSHOT=0 turns this off).
  • 6612817cb4: flush command lists the GPU writes (culling) before reading them.
  • 8f1435f8db: read GPU-written command lists from copies instead of draining

Known gaps:

  • A GPU-written command list is still read stale the first time its page is used, before the GPU thread has marked it.
  • Upstream's lazy backing of render-target memory (6bc1ef08c2) uses the guest-locked mapping call from the old 4 s freeze stack again. It looks rare.

Testing

RTX 3070, eden-cli, scripted 9-minute Grand Prix sessions.

Rebased on ce1b1de6f1 builds clean. Two sessions at 1x and one at 4K, all with EDEN_NX2_PB_CHECK=1: no crash, freeze, garbage, semaphore wait over 2 s or driver error, and no guard fired. p99 frame time was 23.7 and 21.8 ms at 1x and 29.2 ms at 4K, and the worst frame was one 1.8 s hitch as a race loaded. About 0.8% of command entries change between submit and execution. Almost all of them are the 3- and 5-word GPU-written culling lists.. So the copy doesn't seem to drop commands the CPU writes late.

Another rebase on 6221409 gave these results:

p99 23.2 ms, worst frame 288 ms.
0.68% of entries changed after submit, the same culling lists.
Clean, and it quit normally.

On the earlier base 926e7ac6af, where upstream means the same base with only the diagnostics commit:

Fault injected (each real crash's values) Upstream This PR
Invalid syncpoint id SEGV, same stack as the real crashes Survives
Huge render target GPU Xid 69, permanent freeze 60 fps
0x0 render target, 36232 layers Abort 60 fps
Garbage render target address SEGV 60 fps
Device address past the device space SEGV within 32 hits 300 hits, 60 fps
Swapchain VK_INCOMPLETE Abort Survives
Swapchain recreation failing every retry SEGV the first time Recovered 3 of 3
9-minute sessions Upstream This PR
1x, 3 each 2 crashed (the race) 0 crashed during play
4K, 3 each (~7.1 GB VRAM budget) All clean All clean
Race forced by a second writer, 3 each 1 crashed 0 crashed

Qt is tested and ran.

Still open

  • Whether garbage still happens on this base. If it never shows up in normal play, the guards never fire. The syncpoint and render target guards log when they do.
  • The crash at quit: an assert in k_page_table_base.h. It's older than these changes and not fixed here.
  • The late-course 4K stutter (VRAM thrash). It needs re-measuring on this base, because upstream now collects textures from 5 GiB.

Details are in NX2_NOTES.md rows 153-155.

Assisted-by: Claude Opus 5.5 (debugging, implementation, commit message, PR description)
Signed-off-by: @failaip12

The per-600-present log only had means and a maximum, which hid the stutter
being chased. Log p50/p95/p99 and frames over 18 ms, plus hitches over 50 ms
with timestamps to match against other threads. EDEN_NX2_FRAME_LOG writes one
line per present, EDEN_NX2_TC_DUMP shows what fills device memory, and
EDEN_NX2_GPU_PROFILE now logs zones over 10 ms with the last engine method.

Assisted-by: Claude Opus 5.5 (implementation, commit message draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
Mario Kart World crashed about every 45 minutes. 11 of the 14 recorded crash
stacks were in code fed by garbage GPU commands. Ignore invalid syncpoint ids,
GPU addresses that translate past the device space, walks past the GPU page
tables and impossible render targets instead of crashing on them, and retry
transient swapchain errors (a Qt run aborted once on VK_INCOMPLETE).

With each crash's values injected, upstream crashed, aborted or froze in all
six cases and this kept running at 60 fps. The garbage itself still happens.
See NX2_NOTES.md rows 153-154.

Assisted-by: Claude Opus 5.5 (debugging, implementation, commit message draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
PageTableOp changed kind_map without the mutex every other use of it takes,
while the service thread maps and the GPU thread unmaps. Upstream crashed in
RangeMap::InternalMap on the corrupted tree in 2 of 3 Mario Kart World
sessions at 1x, and in 1 of 3 with a second thread writing the map; with the
lock, 0 of 3 each time.

Assisted-by: Claude Opus 5.5 (debugging, implementation, commit message draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
MemoryOperation drops ranges past the address space without calling the
unmapped callback, so IsFullyMappedRange reported them as mapped. Check
the bounds there as IsContinuousRange does.

When swapchain recreation fails on every retry, Create has already
destroyed the old swapchain and reset the flags that trigger recreation,
so the next frame acquired from a null swapchain. Treat a missing
swapchain as needing recreation, and drop the frame if it is still
missing.

Assisted-by: Claude Opus 5.5 (review fixes, commit message draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
The frame log's GPU-done column was read after the present had waited
for fences, composited and paced, so a frame still on the GPU at present
was usually logged as done. With the present now also waiting for the
current frame when the GPU is behind, that hid the backlog the column is
meant to show. Read it before any wait.

Assisted-by: Claude Opus 5.5 (fix and commit message draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
Row 155: the garbage of rows 153-154 is a pushbuffer parse desync, most
likely the game rewriting command lists Eden had not read while the GPU
was behind. Upstream's present wait, submit-time copy and GPU-written
list flush cover it; the probe results, the remaining gaps and the
sessions on ce1b1de (also with the host GPU loaded) are recorded.

Assisted-by: Claude Opus 5.5 (investigation and notes draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
Upstream 8f1435f reads GPU-written command lists from host copies
instead of flushing them, so the note on how they are read was out of
date. Also records one clean 1x session on 6221409.

Assisted-by: Claude Opus 5.5 (notes and commit message draft)
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
@PastaLaPate PastaLaPate added enhancement New feature or request Mario Kart labels Nov 2, 2026
capy-ai Bot added a commit to zortos293/Lichii that referenced this pull request Nov 4, 2026
* Port Polish language code correction from Eglantine PR 505

Source: 551fa99, Djihads80/Eglantine#505. Encode pl rather than lp in the little-endian language identifier.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Port compute subgroup-size stage guard from Eglantine PR 475

Source: Djihads80/Eglantine#475 at 60a7c42. Attach required subgroup size only when compute appears in requiredSubgroupSizeStages, not merely when the extension and warp operations are present.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Port GPU command safety subset from Eglantine PR 329

Adapt 0f89b1cd0c, 0a41327ff3, and fd22c71df5 from Djihads80/Eglantine#329. Guard invalid command ranges and syncpoints, serialize small-page kind-map updates, and recover absent swapchains. Extend the range guard to IsGranularRange and cover invalid reads/counters with regression tests. Exclude optional profiling and diagnostics.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Port bounded Yoshi shader forms from Eglantine PR 123

Adapt shader-only changes from Djihads80/Eglantine#123 at dc4efdcad66dcc452899f69bf822692e36427898. Preserve newer fork lowering and video/service fixes. Add UPOPC inversion/reservation, atomic operations, bounded 1D surfaces and SUATOM, SHF.L.HI, and bounded uniform-array gathers with focused regressions.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Port selected lifecycle and Vulkan fixes from Eglantine PR 472

Source: Djihads80/Eglantine#472 at 747697f28b5b8f11a45288bc608dce52c65d76fb. Use stable syncpoint action IDs, snapshot armed waits before refreshing values, include task/mesh push-constant visibility and layout transitions, compare NVIDIA driver versions correctly, and reinitialize on every memory-layout change. Preserve command safety and lossless video admission. Exclude speculative shader/HID/IMU/vibration and platform changes.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Port video conversion and texture-view fixes from Eglantine PR 502

Adapt Djihads80/Eglantine#502 at 5a104e829aabefadd0f589fe108b11907d300c9a. Skip null image IDs for blacklisted buffer views, directly interleave limited-range planar YUV, copy complete odd-width NV12 chroma, and reject insufficient NV12 pitch. Refresh scaler color-range state on every fallback to avoid stale full-range configuration. Keep the existing same-size NV12 fast path, including full-range sources, because FFmpeg same-format scaling can leave chroma unwritten. Add actual-frame conversion and cached-scaler regression tests.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Fair-Tech added a commit to Fair-Tech/Eglantine that referenced this pull request Nov 5, 2026
Kept nx2's vi2 present pacing code (the PR's version of nx2_gsv.h only changed its frame log). Kept both sets of notes rows; added rescaled image counting inside nx2's debug_stats path.
rdodoa pushed a commit to rdodoa/Eglantine that referenced this pull request Nov 7, 2026
…glantine-Community#329)

The required-subgroup-size create info was built for graphics stages but
never chained, so drivers chose the subgroup size themselves. Mesa's Intel
driver runs mesh shaders 16 wide (it reports 16 to 32, with task and mesh
in requiredSubgroupSizeStages), which splits the guest's 32-thread warp
and breaks the votes, ballots, shuffles and broadcasts the translated
programs use.

Ocarina of Time draws its vegetation with task/mesh programs: one pair for
the depth prepass and another for the colour pass, which tests depth EQUAL
against a 2D-engine copy of the prepass depth. With split warps the two
programs placed the leaves differently, so the colour pass left leaf-shaped
holes that showed the sky behind them, moving every frame with the wind
("white flashing leaves").

Chain the create info for every graphics stage that uses warp-level
operations when the device can force the guest warp size for that stage,
as compute pipelines already do. Quad shuffles and FSWZADD are left out:
they only span four lanes, and forcing SIMD32 on Intel can spill.

Kokiri Forest by the ladder, Arc B390 (Mesa), Low accuracy: leaf holes
went from 7-10% of the lower screen in every frame to 0 in 40 frames.

Assisted-by: Claude Opus 5.5 (diagnosis, fix)
A-Man-With-A-Plan-dev pushed a commit to A-Man-With-A-Plan-dev/Eglantine that referenced this pull request Nov 7, 2026
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
Signed-off-by: failaip12 <86018517+failaip12@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Mario Kart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants