Repository navigation
A GPU reset that no call reports fails the device instead of hanging frame_begin - #98
Merged
Merged
Conversation
After a Windows timeout detection and recovery, amdvlk reports the reset neither at submission nor in vkWaitSemaphores: the timeline reaches the value but the frame's timestamp queries are never written. vk_collect_timestamps read them with VK_QUERY_RESULT_WAIT_BIT, which has no timeout, so frame_begin spun in the driver forever. A GPU test that hit this held the shared device, and every later GPU test of the process waited for it until `mirai test` killed the executable. The results are now read without WAIT. NOT_READY after the timeline passed means the submission ended without its work: the device is marked Lost with the place the GPU stopped, and frame_begin returns false. Verified with a forced reset on an RX 580 (one ~7 s Ddgi_Trace dispatch beside another GPU process). The test fails in seconds with "frame ended without the work it was given: the GPU was reset under it; the GPU stopped in ddgi, with the pipeline Ddgi_Trace", and the executable ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`device_lose(d, reported = false)` makes the next frame go as after a reset that amdvlk reports nowhere: its submission passes its semaphores and its timeline, its command list does not run and its timestamp queries are reset from the host. Metal, whose command buffers report every reset, and an immediate list report it as device_lose does. rhi_a_reset_no_call_reports_fails_the_device_at_once covers the previous commit: with WAIT back in vk_collect_timestamps it hangs until `mirai test` kills it; with the fix the next frame on that slot fails at once with the device Lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #85.
Changes
After a Windows TDR, amdvlk reports the reset neither to
vkQueueSubmit2nor tovkWaitSemaphores. The timeline reaches the frame's value, but the frame's timestamp queries arenever written.
vk_collect_timestampsread them withVK_QUERY_RESULT_WAIT_BIT, which has notimeout, so
frame_beginspun in the driver forever. The GPU test that met it held the shareddevice, and every later GPU test of the process waited for it.
vk_collect_timestampsreads the queries withoutWAIT.NOT_READYafter the timeline passedmeans the submission ended without its work. The device is marked
Lost, with the place theGPU stopped: "frame N ended without the work it was given: the GPU was reset under it".
frame_beginreturns false once the collection failed the device.timing_measured = false), as forany slot whose times the device could not read.
device_lose(d, reported = false)provokes such a reset for a test. The next frame'ssubmission passes its semaphores, and its timeline is signalled. Its command list does not run
and its queries are reset from the host (
hostQueryReset, core in Vulkan 1.2). Metal, whosecommand buffers report every reset, and an immediate list report it as
device_losedoes.+29/−5 outside tests (
rhionly), +33 of test.Tests
RX 580, amdvlk 31.0.21924.61, Windows 10.
rhi_a_reset_no_call_reports_fails_the_device_at_once(rhi/tests/rhi_test.odin).Lostand "(provoked by device_lose)".WAITput back: it hangs, andmirai testkills it at--timeout 60.mirai test rhi: 31/31.mirai test rhi/tests: 29/29.mirai test renderwith the four tests that used to wait past the timeout, by name: 4/4.#configknobs: oneDdgi_Tracedispatch of about 7 s, while the glass tests draw in another process):
gpu_page_requests_list_each_page_once_whatever_asks_for_itfails in seconds: "frame endedwithout the work it was given: the GPU was reset under it; the GPU stopped in ddgi, with the
pipeline Ddgi_Trace".
vk_collect_timestampsuntil it was killed.Not fixed
Ddgi_Tracedispatch runs 30 times its length in the full render suite and resets the GPU #86: what makes the reset happen, aDdgi_Tracedispatch 30 times its usual length.mirai test renderwith one thread crashes inrender_draws_the_same_image_with_a_generation_loaded_from_disk#87: the access violation of the single-thread run.🤖 Generated with Claude Code