vk_native: window-space layers always blend as straight alpha (premultiplied HUDs darken; D3D11 honours the flag)

[dfattal]

Summary

The Vulkan native compositor stamps window-space layers (XrCompositionLayerWindowSpaceDXR) into the atlas with a fixed straight-alpha blend (vk_hud_blend: colour SRC_ALPHA / ONE_MINUS_SRC_ALPHA) and ignores the layer's flags. A window-space layer submitted with premultiplied bytes (flags == 0, the convention the runtime's window-space submitters use) therefore has its colour multiplied by alpha twice: rgb·a². Semi-transparent HUD pixels and antialiased edges come out too dark.

D3D11 already does this right

comp_d3d11_renderer.cpp (window-space / Local2D quad path) chooses the blend from the flags:

bool is_premultiplied = (data->flags & XRT_LAYER_COMPOSITION_BLEND_TEXTURE_SOURCE_ALPHA_BIT) == 0;
OMSetBlendState(is_premultiplied ? r->blend_premul : r->blend_alpha, ...);

The comment there explains why window-space deliberately does not converge on comp_layer_blend_mode() (#1599): its submitters treat layerFlags == 0 as "premultiplied bytes, blend", e.g. cube_handle_d3d11_win's panels. vk_native should follow the same rule. It should not follow the Khronos three-way rule.

Why it's more visible now

Before #1784 the HUD stamp wrote RGB only, so in a transparent session a translucent panel over the desktop was invisible anyway. #1784 (fixing #1780) makes the stamp composite alpha "over" too. Translucent HUD panels now show over the desktop, darkened.

(Correction to my review on #1784: I wrote there that D3D11 has the same behaviour. It doesn't, as shown above.)

Expected

• vk_native window-space stamp: premultiplied (ONE / ONE_MINUS_SRC_ALPHA colour) when SOURCE_ALPHA_BIT is clear, straight when it is set. The alpha channel stays as vk_native: composite window-space alpha into the atlas (#1780) #1784 left it.
• Check that vk_native window-space submitters (Vulkan test apps, displayxr-unity window-space UI on Vulkan) produce bytes that match the flags they send.
• Check the Metal and GL window-space paths for the same parity.