Skip to content

A GPU reset that no call reports fails the device instead of hanging frame_begin - #98

Merged
tomaszbk merged 2 commits into
mainfrom
rhi/reset-without-timestamps
Nov 16, 2026
Merged

tomaszbk merged 2 commits into
mainfrom
rhi/reset-without-timestamps

Conversation

@5hif7y

@5hif7y 5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member

Closes #85.

Changes

After a Windows TDR, amdvlk reports the reset neither to vkQueueSubmit2 nor to
vkWaitSemaphores. The timeline reaches the frame's value, but the frame's timestamp queries are
never written. vk_collect_timestamps read them with VK_QUERY_RESULT_WAIT_BIT, which has no
timeout, so frame_begin spun in the driver forever. The GPU test that met it held the shared
device, and every later GPU test of the process waited for it.

  • vk_collect_timestamps reads the queries without WAIT. NOT_READY after the timeline passed
    means the submission ended without its work. The device is marked Lost, with the place the
    GPU stopped: "frame N ended without the work it was given: the GPU was reset under it".
  • frame_begin returns false once the collection failed the device.
  • This agrees with Give each GPU pass of a profile to the frame that submitted it, where it ran #10 (2181aac): the slot is not measured (timing_measured = false), as for
    any slot whose times the device could not read.
  • device_lose(d, reported = false) provokes such a reset for a test. The next frame's
    submission passes its semaphores, and its timeline is signalled. Its command list does not run
    and its queries are reset from the host (hostQueryReset, core in Vulkan 1.2). Metal, whose
    command buffers report every reset, and an immediate list report it as device_lose does.

+29/−5 outside tests (rhi only), +33 of test.

Tests

RX 580, amdvlk 31.0.21924.61, Windows 10.

  • New rhi_a_reset_no_call_reports_fails_the_device_at_once (rhi/tests/rhi_test.odin).
    • With the fix: it passes in 0.7 s, with the device Lost and "(provoked by device_lose)".
    • With WAIT put back: it hangs, and mirai test kills it at --timeout 60.
  • mirai test rhi: 31/31. mirai test rhi/tests: 29/29.
  • mirai test render with the four tests that used to wait past the timeout, by name: 4/4.
  • A real reset with this change, on a probe build (local #config knobs: one Ddgi_Trace
    dispatch of about 7 s, while the glass tests draw in another process):
    • Windows reports the TDR (event 4101).
    • gpu_page_requests_list_each_page_once_whatever_asks_for_it fails in seconds: "frame ended
      without the work it was given: the GPU was reset under it; the GPU stopped in ddgi, with the
      pipeline Ddgi_Trace".
    • The shared device is dropped, and the executable ends after 38 s with 3 of 4 passing.
    • Before this change the same run spun in vk_collect_timestamps until it was killed.
  • The full render suite was not run: other sessions use this GPU in parallel.

Not fixed

🤖 Generated with Claude Code

5hif7y and others added 2 commits November 15, 2026 10:29
After a Windows timeout detection and recovery, amdvlk reports the reset
neither at submission nor in vkWaitSemaphores: the timeline reaches the
value but the frame's timestamp queries are never written.
vk_collect_timestamps read them with VK_QUERY_RESULT_WAIT_BIT, which has
no timeout, so frame_begin spun in the driver forever. A GPU test that
hit this held the shared device, and every later GPU test of the process
waited for it until `mirai test` killed the executable.

The results are now read without WAIT. NOT_READY after the timeline
passed means the submission ended without its work: the device is marked
Lost with the place the GPU stopped, and frame_begin returns false.

Verified with a forced reset on an RX 580 (one ~7 s Ddgi_Trace dispatch
beside another GPU process). The test fails in seconds with "frame ended
without the work it was given: the GPU was reset under it; the GPU
stopped in ddgi, with the pipeline Ddgi_Trace", and the executable ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`device_lose(d, reported = false)` makes the next frame go as after a
reset that amdvlk reports nowhere: its submission passes its semaphores
and its timeline, its command list does not run and its timestamp
queries are reset from the host. Metal, whose command buffers report
every reset, and an immediate list report it as device_lose does.

rhi_a_reset_no_call_reports_fails_the_device_at_once covers the previous
commit: with WAIT back in vk_collect_timestamps it hangs until
`mirai test` kills it; with the fix the next frame on that slot fails at
once with the device Lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5hif7y 5hif7y self-assigned this Nov 15, 2026
@5hif7y
5hif7y requested a review from tomaszbk November 15, 2026 17:11
@tomaszbk
tomaszbk merged commit de362ee into main Nov 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A GPU reset that no call reports hangs frame_begin, and with it every later GPU test

2 participants