vk_native: honour the premultiplied flag on window-space layers (#1786)

[dfattal]

Fixes #1786.

Problem

vk_native stamps window-space layers (XrCompositionLayerWindowSpaceDXR) into the atlas through vk_hud_blend. Its colour blend was always straight alpha (SRC_ALPHA / ONE_MINUS_SRC_ALPHA), and it ignored the layer flags. A layer submitted with premultiplied bytes (SOURCE_ALPHA_BIT clear) came out as rgb·a², so translucent pixels and antialiased edges were too dark. #1784 now composites the HUD's alpha as well, so in a transparent session those panels show over the desktop, darkened.

D3D11 (comp_d3d11_renderer.cpp) and D3D12 (render_window_space_layer) already choose the blend from the flags: premultiplied when SOURCE_ALPHA_BIT is clear, straight when it is set. They deliberately do not use comp_layer_blend_mode() (#1599).

Fix

• vk_hud_blend builds a second pipeline, pipeline_premul, in the same instance. It is identical to the existing one except colour src ONE. Both share the render pass, the pipeline/descriptor layouts and the per-VkImage view/descriptor cache. The new vk_hud_blend_draw_no_layout_ex(..., bool premultiplied) picks the pipeline for each draw.
• vk_native's window-space pass uses the D3D11/D3D12 rule: premultiplied = !(flags & SOURCE_ALPHA_BIT).
• The alpha blend from vk_native: composite window-space alpha into the atlas (#1780) #1784 is unchanged in both pipelines (srcAlpha ONE, dstAlpha ONE_MINUS_SRC_ALPHA, A written when write_alpha).
• vk_hud_blend_init(), vk_hud_blend_draw() and vk_hud_blend_draw_no_layout() still use the straight pipeline, so the comp_multi_system.c callers (hud_blend, chrome_blend, shared_chrome_blend) behave exactly as before. The only change for them is one extra pipeline object at init.
• GL (comp_gl_compositor.cpp) had the same fixed glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) on its window-space pass. It now uses the same rule (one line).

Composes with #1783

This is designed to land next to #1783 (forget a destroyed image's cached HUD view). It uses one vk_hud_blend instance with two pipelines, not a second instance, so #1783's vk_compositor_forget_dead_window_space_images() still drains the only cache. Blend mode belongs to the layer, not the image, so the cache keys stay valid. The new _ex declaration sits before vk_hud_blend_draw_no_layout in the header, away from where #1783 adds vk_hud_blend_forget_image. I checked that this commit rebases cleanly onto the #1783 head.

Who is affected (submitter audit)

Submitter	Window-space flags	Effect of this PR
displayxr-common xr_window_space_hud.cpp (v2.24.0). Every in-tree test app's HUD (cube_handle_vk_macos, cube_zones_vk_macos, the windowspace_handle_* probes, GL/Metal cube apps)	SOURCE_ALPHA_BIT	none: still straight
displayxr-unity window-space UI (ps_fill_wsui_layer, shared by every graphics backend including Vulkan)	SOURCE_ALPHA_BIT	none: still straight
Any app submitting layerFlags == 0 on vk_native or GL	0	now premultiplied (matches D3D11/D3D12)

No in-tree or Unity window-space submitter sends flags == 0, so nothing that ships today changes appearance. The layerFlags = 0 // premultiplied bytes panels in the cube apps are Local2D layers (XRT_LAYER_LOCAL_2D), not window-space. This PR doesn't touch that path.

Parity notes (not changed here)

• Metal (comp_metal_compositor.m, window-space pass ~L3993): draws with projection_pipeline, which has blendingEnabled = NO. The HUD overwrites its rect instead of blending, which is a different bug from this one. proj_premult_pipeline/proj_straight_pipeline already exist, so a fix would be small, but it changes macOS HUD appearance and needs its own eyeball. Left for a follow-up.
• D3D12: already matches D3D11.
• Possible submitter mismatch, also not changed: hud_renderer_macos.mm draws into a kCGImageAlphaPremultipliedLast bitmap but xr_window_space_hud.cpp flags the layer SOURCE_ALPHA_BIT (straight). This is consistent across backends, so it is not a vk_native issue.

Testing

• macOS (Apple Silicon, MoltenVK): a Debug runtime build (cmake + ninja, same options as build_macos.sh, service off) compiles cleanly with no new warnings in the touched files. clang-format applied to the changed lines.
• Ran cube_handle_vk_macos against this build with the vk_native file-trigger capture ($TMPDIR/displayxr_atlas_trigger). The session started, rendered and captured normally. But the app's HUD is hidden by default and only toggles with Shift+Tab, which I could not send from the harness, so that run never submitted a window-space layer. It does not exercise this change.
• Confirmed this commit rebases cleanly onto vk_native: forget a destroyed window-space image's cached HUD view (#1782) #1783's head.

Not tested

• No run reached the window-space stamp, so neither pipeline has been seen drawing, and premultiplied pipeline creation on MoltenVK hasn't been run (it is lazy, on the first window-space layer).
• No flags == 0 window-space submitter exists to A/B the premultiplied path visually, on any platform.
• Linux and Windows Vulkan not run. GL change compiled only, not run.

🤖 Generated with Claude Code