DK Bananza: GPU hangs come from stale command list reads on GPU-written pages #392

Tracking issue for the Donkey Kong Bananza GPU hangs, the stream corruption behind them, and the work still to do. It is written so the investigation can be resumed cold.

Symptoms

Bananza freezes during gameplay; fast, repeated inputs reproduce it most reliably (usually within 1 to 3 minutes). The log shows one of two signatures, and both turn out to be the same bug:

• Stream corruption: a channel suddenly parses garbage. You see Dropping method ... on unbound subchannel, Unimplemented engine ..., Invalid KeplerMemory register, Native compute method out of range, Native QMD failed: unsupported QMD version or queue, and unknown host method with a count such as 2047.
• Semaphore hang: Channel 2 waits on semaphore 0x2161?1b000 (op 1, 0x2, now 0xffffffff) on device address 0x5cd1c000. Channel 1 is usually also stuck: waits on semaphore 0x210?4e0000 (op 4, N, now M) on device 0xe1000, with N far ahead of M.

Root cause (current understanding)

1. The semaphore hang is a consequence of the stream corruption. Device 0x5cd1c000 is the first page of a 0x40000 block the game maps with svc 0x80 at about 1.86 s, right before it creates its channels. Its GPU VA changes per run; its device address does not. Each frame, channels 1 and 2 run a fixed handshake on it:

   • Channel 1 resets 5 words to 0xffffffff.
   • Channel 1 releases +0 = 0. Channel 2 acquires +0 == 0.
   • Channel 2 releases +4 = 1, through a Native QMD release. Channel 1 acquires +4 == 1.
   • Channel 1 releases +0 = 2. Channel 2 acquires +0 == 2.
   • Channel 2 releases +4 = 3. Channel 1 acquires +4 == 3.

   Exactly one reset happens per cycle; about 12k resets happen during loading, before the cycles start. In the hangs, channel 1's stream got corrupted mid-cycle. It skipped its own releases and acquires, then waited on a garbage counter far ahead of the real value. Channel 2 then waits forever for a value that never comes.

2. The corruption comes from reading command memory the game had already refilled for a later frame. The game rewrites command memory after submitting it: present returns frames ahead, which Mario Kart World also does. nvhost_gpu.cpp handles this by snapshotting each entry's words at submit time and executing the snapshot. However, any entry on a 4 KiB page that the GPU ever wrote skips the snapshot: the !IsGpuWrittenCommands(...) term in the submitted condition in DmaPusher::Step. That bit is sticky (g_nx2_gpu_written_pages) and page-granular, so CPU-written command chunks that share a page with GPU-written data are read live. By parse time, the game has already overwritten them.

   • Evidence: in the trace run, the corruption at 152.23 s was unknown host method at get=0x210bf59c44.
     • Parsed entry -0 (0x210bf59c44+0x6) was read on the unsafe path; all 6 of its words differ from the submit snapshot.
     • Parsed entry -8 (0x210bf59c14+0xc) was also read unsafe; 11 of its 12 words differ.
     • The EDEN_NX2_CMD_WRITES recorder logged no emulator-side writes (DeviceMemoryManager::WriteBlock*) into them, so the guest CPU rewrote them.
   • The earlier observation of chunks in the 0x210bxxxxx/0x210cxxxxx rings being a fragment at parse time and a proper chunk later fits this cause too.

Fix under test (not yet verified)

Commit d3b4fb5274, behind the env var EDEN_NX2_EPOCH_SNAPSHOT=1:

• g_nx2_submit_epoch is incremented per submission.
• Each submission records CommandList::submit_epoch.
• Nx2MarkGpuWritten stores the current epoch per 4 KiB page in g_nx2_gpu_written_epoch (64 MiB of bss).
• IsGpuWrittenCommands(addr, size, since_epoch) returns true, so the entry is read live, only if the GPU wrote the page at or after the entry's submission. Otherwise the snapshot is used.

Risk: a GPU write recorded before the submit but not yet in guest memory when the snapshot was taken would make the snapshot stale. Watch for missing terrain, trees or culled objects, the case the GPU-written path was added for.

A test run was launched but has no result yet. Next steps:

1. Run with EDEN_NX2_EPOCH_SNAPSHOT=1 and fast inputs. Check for hangs or corruption, and for missing geometry.
2. If it holds, make it the default and remove the env gate. Consider a finer granularity, or drop the sticky bit in favour of the epoch.
3. If corruption persists, use the PB_TRACE dump: the parsed -N lines give each entry's read path and how many words differ from the snapshot.
4. Check the case of an entry submitted after a pending GPU write. One option is to compare the snapshot with live memory when the epoch says snapshot, and log the mismatches.

Branch and builds

• Branch: fix/bananza-terrain-full on jojo-chaechae/Eglantine. Local worktree: ~/edenpack-test/bananza-pr.
• Builds use local-build.yml on the self-hosted runner:

  gh workflow run local-build.yml -R jojo-chaechae/Eglantine --ref fix/bananza-terrain-full -f linux_pgo=true -f windows=false -f android=false -f release=false
  docker cp eglantine-runner:/home/runner/runner/_work/Eglantine/Eglantine/out/eglantine-jojo-<sha>-linux-x86_64-pgo.AppImage <dir>
  

• Run:

  : > ~/.local/share/eden/log/eden_log.txt
  ENV... <AppImage> -g ".../DKBananza-458752.nup"
  

  Copy the log out after each run, because the next launch truncates it.

Commits after 5c28733071, in order:

The diagnostic commits need squashing or dropping before anything goes upstream.

Useful env vars:

• EDEN_NX2_PB_TRACE=1: cheap ring of the last 64 parsed entries, dumped at the first unbound-subchannel method or unknown host method.
• EDEN_NX2_PB_CHECK
• EDEN_NX2_CMD_WRITES=1
• EDEN_NX2_SEM_WATCH_DEV=5cd1c000
• EDEN_NX2_SEM_WATCH=<gpu va>
• EDEN_NX2_PB_DUMP=<file>: the value is the output path. =1 writes a file literally named 1.

Logs (local)

Logs are in ~/edenpack-test/logs/b167/:

Notes are in ~/edenpack-test/SCRATCHPAD.md.

Other findings, not fixed

• KeplerMemory::ConsumeSinkImpl writes regs.reg_array[method] without a bounds check. Garbage streams hit it: Invalid KeplerMemory register asserts.
• Most split methods are legitimate and not part of this bug. Hardware carries method state across GP entries (open-gpu-doc dev_pbdma). The usual pattern is a macro 0xe50/0xe51 header with 5 parameters left, followed by a 5-word GPU-written record.
• Entry 0x210f6c41f0+0x146, word 176, is a per-frame pointer cycling 0bfeb000/0d233000/0e47b000. It was read live with a different frame's value. It may relate to jitter; it is probably the same snapshot-vs-live issue.
• Rubberbanding is deliberately parked.

Pending after this

• Contribute to draft Experimental Donkey Kong Bananza terrain rendering fixes #363: reset/pointer guard, F2I byte fix, FP16 texture fix, HMMA default, and the semaphore region set fix (2cfdf53510). Each upstream PR and comment needs a separate approval.

The human-written contributing guideline was not asserted for this issue.