vk_native: composite window-space alpha into the atlas (#1780)

[byungjul]

Fixes #1780.

Problem

In a transparent session on the Vulkan native compositor, a window-space layer (HUD) was visible only where 3D content already sat under it. On Linux the HUD showed inside the avatar's silhouette and was see-through everywhere else.

vk_hud_blend stamps the HUD into the atlas with RGB writes only (srcAlpha ZERO, dstAlpha ONE, no A in colorWriteMask). In a transparent session the atlas alpha decides where the window is see-through. Outside the silhouette it is 0, so the HUD's colour was written but stayed fully transparent.

Fix

• New vk_hud_blend_init_ex(blend, vk, fmt, write_alpha). With write_alpha the HUD is also composited "over" in alpha: srcAlpha ONE, dstAlpha ONE_MINUS_SRC_ALPHA, all channels written. This is the blend D3D11 already uses for window-space layers (comp_d3d11_renderer.cpp, DP compose-under-bg likely applied per-layer instead of post-composite — desktop bleeds through semi-transparent window-space HUD over opaque content #225).
• vk_native's window-space blend turns it on.
• vk_hud_blend_init() keeps the RGB-only blend, so comp_multi_system.c (hud_blend, chrome_blend, shared_chrome_blend) is unchanged.

Behaviour change: this applies to every vk_native session, not only transparent ones. Under "over" the alpha only grows (a = a_src + a_dst·(1 − a_src)), so an opaque atlas (alpha 1) is unchanged. I didn't key it on the session's transparency because that can switch on mid-session (lazy transparency) and this pipeline is built once. That affects Linux, and macOS through MoltenVK. Windows Vulkan apps use vk_native too. I only tested Linux.

Testing

Ubuntu 26.04, RTX 4090 Laptop, Leia DS1, Leia SR plug-in 2.7.6. App: the lenovo-avatar Unity player (transparent) with displayxr-unity window-space UI on Vulkan. Two HUD panels, opened with Shift+Tab.

Runtime	Result
v2.21.11	HUDs visible only where they overlap the avatar
v2.21.11 + this change	HUDs visible over the desktop too, and their clicks work; the background around the avatar stays transparent

The hardware test used v2.21.11 with this patch plus #1783 (the patches apply cleanly). The branch is on current main and builds there (build_linux.sh, Release, no new warnings).

🤖 Generated with Claude Code

[dfattal]

LGTM — small, correct, well-scoped.

• D3D11 parity confirmed: the new blend (colour SRC_ALPHA/INV_SRC_ALPHA, alpha ONE/INV_SRC_ALPHA, all channels written) matches comp_d3d11_renderer.cpp blend_alpha (#225) exactly.
• Unconditional enable is safe: alpha only grows under "over". The lazy-transparency alpha probe runs after the window-space stamp (comp_vk_native_compositor.c stamp → vk_lazy_transparency_frame), so a HUD can never create false alpha<1, and a HUD over a transparent region is now correctly counted as content. Opaque sessions present opaque. window_space_blend has one call site; comp_multi_system.c keeps the RGB-only blend.
• Output convention matches the present: a straight-alpha HUD source through this blend yields a valid premultiplied pixel, which is what the transparent target (COMPOSITE_ALPHA_PRE_MULTIPLIED) expects.

Non-blocking follow-up: vk_hud_blend always treats the source as straight alpha and ignores the layer's XR_COMPOSITION_LAYER_UNPREMULTIPLIED_ALPHA_BIT (OpenXR default is premultiplied), so a premultiplied HUD gets rgb·a² — darker translucent pixels. D3D11 has the same behaviour and RGB already did this, but it's now more visible over the desktop. Could pick src factor from the layer flags like comp_vk_native_renderer.c does for projection layers.

Not tested by me: macOS (MoltenVK) / Windows Vulkan transparent sessions — the monotone-alpha argument covers opaque ones.