opencl: retry D3D11 device enumeration on transient CL_DEVICE_NOT_FOUND

[jsphgei-dot]

Summary

• clGetDeviceIDsFromD3D11KHR() can occasionally return CL_DEVICE_NOT_FOUND on some AMD driver/OpenCL ICD combinations when another process is using the same D3D11 device at the same time (e.g. a second overlapping transcode). Right now that results in a hard Device creation failed: -19 / "No matching devices found", which breaks any hardware transcode combining an OpenCL filter (subtitle burn-in, tonemap, etc.) with AMF encoding when this happens.
• This adds a retry (5 attempts, 20ms apart) around both clGetDeviceIDsFromD3D11KHR() calls before treating CL_DEVICE_NOT_FOUND as a genuine "no devices support this" answer. If the issue doesn't come up, behavior is unchanged; on the rare path where it does, this just adds up to ~100ms.

Testing

• Saw this exact failure in real production Jellyfin transcode logs (AMD Radeon RX 6700 XT, Windows), hitting Device creation failed: -19 on subtitle-burn-in + AMF transcodes.
• Built and deployed this patch to that same server; the episode that was reliably failing now transcodes correctly with subtitles on.
• Wasn't able to reliably trigger the underlying driver race on demand in an isolated test (tried both single-shot and concurrent stress runs), so this is real-world validation rather than a clean before/after repro.

[gnattu]

I'm not against adding workarounds for gpu drivers but I want to limit scope as much as possible.

Can you make this change to be optionally opt-in instead, and change the jellyfin server implementation so that it opt-in this retry behavior, for amd on windows?

Also, please submit your change in a quilt patch like this: https://github.com/jellyfin/jellyfin-ffmpeg/tree/jellyfin/debian/patches

Instead of directly changing upstream code.

[nyanmisaka]

I haven't encountered this issue on GCN or RDNA cards over the past few years, and the clGetDeviceIDsFromD3D11KHR() documentation makes no mention of such redundant retry logic. I see no justification for it. Lock/unlock functions for D3D11 devices already exist in hwcontext_d3d11va, please check if utilizing them resolves the issue.

[nyanmisaka]

Here's the source code of AMD's clGetDeviceIDsFromD3D11KHR implementation. I didn't even see the need to add Lock and Unlock.

https://github.com/ROCm/clr/blob/8d601bff7bb2006f95f161ca93b1cf46310caf6f/opencl/amdocl/cl_d3d11.cpp#L34-L151

[jsphgei-dot]

Marking this as draft while I keep digging. This one was frustratingly persistent for the evening, happening frequently enough that I was troubleshooting it for hours the night before, but now that I've got proper debug logging in place to actually catch it, it's stopped reproducing, so I don't have a clean root cause yet and don't want to close this out prematurely.

Where things stand: the one real-world case that seemed to support this fix (a previously-failing episode working after deploying the patch) turned out to be inconclusive on closer look, not something I can actually attribute to this change. And nyanmisaka's pushback was fair, there's no confirmed mechanism for why this would help, and nothing I've found since supports treating CL_DEVICE_NOT_FOUND here as a genuine transient condition worth retrying on.

Still working the actual root cause. Will update this PR once I have something more solid.

[nyanmisaka]

It is known that the way OpenCL interop functions can conflict with Windows service installations and Remote Desktop usage. Such scenarios fall outside the scope of what we can fix.