fix: route Streamline's own export calls to the loader (DLSS-G crash on Wine/Proton)

[RobertRautenbach]

Summary

Under Wine/Proton, Streamline's own device dispatch table aliases the four vulkan-1 export detours, so sl.common calls them and receives the game's view. DLSS-G then faults about two seconds into the world, every launch. This sends callers that are Streamline itself to the loader on the three swapchain exports, leaving every other caller exactly where it is.

With this, DLSS-G runs: FG stats: 300 presents -> 600 frames displayed (x2.00) status=0, sustained, on Baldur's Gate 3's native Vulkan renderer under Proton.

The bug

On Wine, vulkan-1.dll is a pure forwarder into winevulkan, and winevulkan serves its device dispatch table from the same addresses it exports. I checked this byte-exactly rather than assuming it: vkGetSwapchainImagesKHR export RVA 0x1ac70 appears exactly once in the image, at .rdata 0x2eee8, immediately preceded by a pointer to the string "vkGetSwapchainImagesKHR". Same for vkAcquireNextImageKHR and vkQueuePresentKHR. The Khronos loader on Windows keeps lookup and dispatch apart, so GetProcAddress(vulkan-1, X) and vkGetDeviceProcAddr(dev, X) differ there and this never reproduces.

Because InstallVkHooks detours those exported bodies in place, and Streamline builds its table from GetProcAddress(vulkan-1, "vkGetDeviceProcAddr"), sl.common's m_ddt.GetSwapchainImagesKHR is h_GetSwapchainImagesKHRExport.

sl.common::Vulkan::getSwapChainBuffer issues the usual two-call count-then-fill query, from inside DLSS-G's cloneFakeBuffers. That re-enters the clone, which has already resized its real back-buffer vector but has not yet called resizeProxyBuffers, so the proxy vector is still empty. The copy loop computes count = (end - begin) / 8 = 0, takes its out-of-range fallback, and dereferences it:

sar  $0x3,%rcx        ; count = (end - begin) / 8   -> 0
cmp  %rcx,%rdx        ; index 0 vs count 0
jae  ...              ; taken
mov  %r14,%rcx        ; fallback object ... which is NULL
mov  0x28(%rcx),%rcx  ; <-- EXCEPTION_ACCESS_VIOLATION reading 0x28

Stack: sl.dlss_g+0x55C35 <- sl.interposer+0x366DB <- fgvk+0x289C. sl.log stops at cloneFakeBuffers: ENTRY - width=3840, height=2160, format=44, bufferCount=6, and fgvk.log's last line is export vkGetSwapchainImagesKHR from tid=328 routed to the game's (fake-buffer) view. Neither of cloneFakeBuffers' own failure exits logs, so nothing actually failed - the code observed its own half-built state.

ForceReal() cannot catch this. bg3.exe imports no vulkan-1.dll (I dumped its import and delay-import directories) and resolves through GIPA, so it never enters these hooks and no re-entry scope is ever pushed when Streamline calls them. The same holds on DLSS-G's pacer thread, where any thread-local guard is structurally zero.

Changes

• src/slmodule.h: IsStreamlineModule(), a pure predicate matching sl.*, nvngx_dlssg and NvLowLatencyVk on the module file name.
• src/tests/slmodule_test.cpp: 13 cases, wired into tests_main.cpp and the fgvk-tests target.
• src/vkhooks.cpp: CallerIsStreamline(_ReturnAddress()) resolves the calling module, and the three swapchain export hooks OR it into the existing ForceReal() test.
• BUILD.md: a paragraph in Script Extender coexistence, since this is the same "which caller gets which view" question.

Deliberately not a bare nvngx prefix. OptiScaler ships an nvngx.dll DLSS shim next to the exe; capturing it would put its hooks on the driver's present under the pacer thread, which is the deadlock class already fixed in f624005f. There is a test asserting it is excluded.

Verification

Proton (proton-cachyos, Wine 11), NVIDIA 610.57.04, RTX 5080, Baldur's Gate 3 at 3840x2160, native Vulkan, Streamline 2.12.0, alongside OptiScaler (DLSS SR + Neural Rendering) and BG3SE in-process.

• Before: access violation ~2s into the world, every launch, at the address above.
• After: sustained x2.00 status=0, no present failures, no watchdog stalls, clean exit.
• Third-party routing is untouched, and the log proves it rather than asserting it - export vkGetSwapchainImagesKHR from tid=328 routed to the game's (fake-buffer) view still appears, and export vkGetDeviceProcAddr(vkQueuePresentKHR) inside loader vkCreateDevice (third-party hook) -> game view still fires for OptiScaler. The OptiScaler menu and BG3SE mod UI both still draw.
• fgvk-tests: 27/27 pass, on Linux and on windows-latest/MSVC.

I could not test on Windows, and by construction there is nothing to test: the two addresses differ there, so CallerIsStreamline is only ever consulted for callers that already took the ForceReal path or are genuinely third parties. If you would rather it were impossible to reach on Windows at all, it is easy to gate.

Happy to attach the full fgvk.log / sl.log pairs from before and after, or to test any variant you would prefer - I have the environment set up.

---

Developed with AI assistance. I ran it, tested it in game and reviewed the change before submitting. Noting it because this touches a crash path you cannot reproduce, so the evidence above is what it rests on, not my say-so.