From 0069135e4930a4eaab0beb4da1f99d5186372c53 Mon Sep 17 00:00:00 2001 From: dfattal Date: Thu, 8 Oct 2026 02:40:05 -0700 Subject: [PATCH 1/3] =?UTF-8?q?feat(lift):=20XR=5FDXR=5Flift=20v3=20API=20?= =?UTF-8?q?+=20DP=20contract=20=E2=80=94=20app=20rig=20viewpoints=20and=20?= =?UTF-8?q?auxiliary=20metric=20depth?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit API (XR_DXR_lift spec v2 -> 3, ADR-048 Addendum A): - An XrDisplayRigDXR / XrCameraRigDXR (XR_DXR_view_rig) chained on XrLiftOptionsDXR or XrWeaveRectLiftDXR drives the TRACKED lift viewpoints with the xrLocateViews rig math (the lifted rect is the screen). - EXPLICIT viewpoints are accepted on lifted weave rects. - XrLiftDepthRequestDXR (283) on XrLiftStreamCreateInfoDXR asks an SBS / NVIEW stream for its depth map as an auxiliary output; XrLiftDepthResultDXR (284) on XrLiftResultDXR returns it (texture + fence on first acquire / realloc, format, units, encoding, intrinsics, convergence depth, the viewpoint it refers to and a column-major depthToDisplay transform that puts the convergence depth on the screen plane); XrLiftDepthPropertiesDXR (285) on XrLiftPropertiesDXR reports the capability. DP contract (append-only, ADR-020, no ABI bump): - xrt_dp_lift_params += viewpoint_source, nominal_z_m (XRT_DP_LIFT_HAS_APP_RIG). - xrt_dp_lift_caps += aux_outputs, aux_depth_semantics; xrt_dp_lift_stream_info += aux_outputs, aux_depth_format; new struct xrt_dp_lift_depth (XRT_DP_LIFT_HAS_AUX_DEPTH). - D3D11 DP vtable slot 33: lift_get_depth (XRT_DP_D3D11_HAS_LIFT_DEPTH), called right after a successful lift_convert of a stream that asked for depth; helper xrt_display_processor_d3d11_lift_get_depth. Co-Authored-By: Claude Opus 5.5 --- src/external/openxr_includes/openxr/README.md | 2 +- .../openxr_includes/openxr/XR_DXR_lift.h | 162 +++++++++++++++++- .../include/xrt/xrt_display_processor_d3d11.h | 74 +++++++- src/xrt/include/xrt/xrt_dp_lift.h | 141 ++++++++++++++- 4 files changed, 370 insertions(+), 9 deletions(-) diff --git a/src/external/openxr_includes/openxr/README.md b/src/external/openxr_includes/openxr/README.md index fb62aa09d4..e1ad51643f 100644 --- a/src/external/openxr_includes/openxr/README.md +++ b/src/external/openxr_includes/openxr/README.md @@ -47,7 +47,7 @@ Rules: | 1004999250–259 | `XR_DXR_wayland_surface_binding` | 250 = `XR_TYPE_WAYLAND_SURFACE_BINDING_CREATE_INFO_DXR` (#757), 251 = `XR_TYPE_WAYLAND_SURFACE_GEOMETRY_DXR` (spec v2) — the app-declared surface size + output refresh, because a `wl_surface` has no intrinsic size and the compositor geometry service cannot bootstrap one. **250 was renumbered from 1004999210**, which collided with `XR_TYPE_DISPLAY_DESKTOP_POSITION_DXR` — the 210–219 decade was already `XR_DXR_display_info`'s. Safe to move: SPEC_VERSION 1, no shipped app chains it | | 1004999260–269 | `XR_DXR_depth_budget` | 260 = `XR_TYPE_REAR_DEPTH_BUDGET_DXR`, 261 = `XR_TYPE_CONTENT_BOUNDS_DXR`, 262 = `XR_TYPE_EVENT_DATA_REAR_DEPTH_BUDGET_STATE_CHANGED_DXR`, 263 = `XR_TYPE_CONTENT_MASK_DXR` (ADR-040). Was taken without a registry row — recorded retroactively in #1486 PR 2 | | 1004999270–279 | `XR_DXR_lift` | 270 = `XR_TYPE_LIFT_PROPERTIES_DXR`, 271 = `XR_TYPE_LIFT_STREAM_CREATE_INFO_DXR`, 272 = `XR_TYPE_LIFT_FRAME_SUBMIT_INFO_DXR`, 273 = `XR_TYPE_LIFT_OPTIONS_DXR`, 274 = `XR_TYPE_LIFT_RESULT_DXR`, 275 = `XR_TYPE_WEAVE_SUBMIT_LIFT_RECTS_DXR`, 276 = `XR_TYPE_WEAVE_RECT_LIFT_DXR`, 277 = `XR_TYPE_LIFT_BLOB_DXR` (spec v1, ADR-042). 278 = `XR_OBJECT_TYPE_LIFT_STREAM_DXR` — an `XrObjectType`, not an `XrStructureType`. 279 = `XR_LIFT_NOT_READY_DXR` — a **success**-class `XrResult` (positive) | -| 1004999280–289 | `XR_DXR_lift` (continued) | 280 = `XR_TYPE_LIFT_STREAM_STATS_DXR` (spec v1). A second decade because 270–279 is full. Spec v2 (ADR-048): 281 = `XR_TYPE_LIFT_VIEW_CONTROL_DXR`, 282 = `XR_TYPE_LIFT_RESULT_VIEWPOINTS_DXR` | +| 1004999280–289 | `XR_DXR_lift` (continued) | 280 = `XR_TYPE_LIFT_STREAM_STATS_DXR` (spec v1). A second decade because 270–279 is full. Spec v2 (ADR-048): 281 = `XR_TYPE_LIFT_VIEW_CONTROL_DXR`, 282 = `XR_TYPE_LIFT_RESULT_VIEWPOINTS_DXR`. Spec v3 (ADR-048 Addendum A): 283 = `XR_TYPE_LIFT_DEPTH_REQUEST_DXR`, 284 = `XR_TYPE_LIFT_DEPTH_RESULT_DXR`, 285 = `XR_TYPE_LIFT_DEPTH_PROPERTIES_DXR` | | 1004999290–299 | `XR_DXR_weave` (v14+ additions) | 290 = `XR_TYPE_WEAVE_SUBMIT_OVERLAY_UNCHANGED_DXR` (the v4 overlay atlas is unchanged since the previous submit, so a DP may reuse its lens prefilter of the 2D layer, spec v14, ADR-027 Amendment); 291 = `XR_TYPE_WEAVE_SUBMIT_OVERLAY_FILTER_DXR` (the caller's lens-filter strength for the 2D layer, 0..1, omitted = the DP's default, spec v15). The third weave decade (240–249 is full) | | 1004999300–309 | `XR_DXR_stereo_camera` | 300 = `XR_TYPE_STEREO_CAMERA_PROPERTIES_DXR`, 301 = `XR_TYPE_STEREO_CAMERA_CALIBRATION_DXR`, 302 = `XR_TYPE_STEREO_CAMERA_STREAM_CREATE_INFO_DXR`, 303 = `XR_TYPE_STEREO_CAMERA_FRAME_DXR`, 304 = `XR_TYPE_STEREO_CAMERA_STREAM_INFO_DXR`, 305 = `XR_TYPE_STEREO_CAMERA_STREAM_TRANSPORT_DXR`, 306 = `XR_TYPE_STEREO_CAMERA_STREAM_STATS_DXR`, 307 = `XR_TYPE_EVENT_DATA_STEREO_CAMERA_STATE_CHANGED_DXR`, 308 = `XR_TYPE_EVENT_DATA_STEREO_CAMERAS_CHANGED_DXR` (spec v1, ADR-043). 309 = `XR_OBJECT_TYPE_STEREO_CAMERA_STREAM_DXR` — an `XrObjectType`, not an `XrStructureType`. **Relocated from 290–299** on the rebase onto post-lift `main`: 290 had meanwhile been taken by `XR_TYPE_WEAVE_SUBMIT_OVERLAY_UNCHANGED_DXR` (weave spec v14). Safe to move: spec v1, draft PR, no shipped app chains it | | 1004999310–319 | `XR_DXR_stereo_camera` (continued) | 310 = `XR_STEREO_CAMERA_FRAME_NOT_READY_DXR` — a **success**-class `XrResult` (positive). A second decade because 300–309 is full. Spec v2 (R3, ADR-043 §7): 311 = `XR_TYPE_STEREO_CAMERA_CLIENT_INFO_DXR` (camera-only client declaration on `XrInstanceCreateInfo`); 312 = `XR_ERROR_STEREO_CAMERA_CONSENT_REFUSED_DXR`, 313 = `XR_ERROR_STEREO_CAMERA_DISABLED_DXR`, 314 = `XR_ERROR_STEREO_CAMERA_BUSY_DXR`, 316 = `XR_ERROR_STEREO_CAMERA_STREAM_ENDED_DXR` — **error**-class `XrResult`s (negated in the header); 315 = `XR_TYPE_EVENT_DATA_STEREO_CAMERA_STREAM_ENDED_DXR` | diff --git a/src/external/openxr_includes/openxr/XR_DXR_lift.h b/src/external/openxr_includes/openxr/XR_DXR_lift.h index 4dcd4b8430..7de7f79f4e 100644 --- a/src/external/openxr_includes/openxr/XR_DXR_lift.h +++ b/src/external/openxr_includes/openxr/XR_DXR_lift.h @@ -57,7 +57,13 @@ * viewpoints are relative to the lifted rect's / submitting window's centre * (was: the panel centre), XrLiftViewControlDXR (display-rig ipd / parallax * factors, axis mode, offset clamp, recentering) and the per-result echo - * XrLiftResultViewpointsDXR. + * XrLiftResultViewpointsDXR. 3 = app camera + metric depth (ADR-048 + * Addendum A): an XrDisplayRigDXR / XrCameraRigDXR (XR_DXR_view_rig) chained + * on XrLiftOptionsDXR or XrWeaveRectLiftDXR drives the TRACKED viewpoints with + * the xrLocateViews rig math; EXPLICIT viewpoints are accepted on lifted weave + * rects; SBS / NVIEW streams may return an auxiliary depth map from the same + * conversion (XrLiftDepthRequestDXR at stream creation, XrLiftDepthResultDXR + * on each result, XrLiftDepthPropertiesDXR on the properties). */ #ifndef XR_DXR_LIFT_H #define XR_DXR_LIFT_H 1 @@ -70,7 +76,7 @@ extern "C" { #endif #define XR_DXR_lift 1 -#define XR_DXR_lift_SPEC_VERSION 2 +#define XR_DXR_lift_SPEC_VERSION 3 #define XR_DXR_LIFT_EXTENSION_NAME "XR_DXR_lift" // Reserved 1004999270..279. Allocation registry: README.md in this directory. @@ -92,6 +98,10 @@ extern "C" { // Spec v2 (ADR-048). #define XR_TYPE_LIFT_VIEW_CONTROL_DXR ((XrStructureType)1004999281) #define XR_TYPE_LIFT_RESULT_VIEWPOINTS_DXR ((XrStructureType)1004999282) +// Spec v3 (ADR-048 Addendum A). +#define XR_TYPE_LIFT_DEPTH_REQUEST_DXR ((XrStructureType)1004999283) +#define XR_TYPE_LIFT_DEPTH_RESULT_DXR ((XrStructureType)1004999284) +#define XR_TYPE_LIFT_DEPTH_PROPERTIES_DXR ((XrStructureType)1004999285) //! Size of XrLiftPropertiesDXR::backend, NUL included. #define XR_LIFT_BACKEND_NAME_MAX_SIZE_DXR 32 @@ -133,6 +143,19 @@ typedef enum XrLiftDepthSemanticsDXR { XR_LIFT_DEPTH_SEMANTICS_MAX_ENUM_DXR = 0x7FFFFFFF } XrLiftDepthSemanticsDXR; +/*! + * How the samples of an auxiliary depth map encode depth (spec v3). Decode a + * sample @c s as d = valueScale * s + valueOffset (XrLiftDepthResultDXR); + * with INVERSE, depth = 1 / d. The decoded depth is in the result's units. + */ +typedef enum XrLiftDepthEncodingDXR { + //! The decoded value IS the depth (larger = farther). + XR_LIFT_DEPTH_ENCODING_LINEAR_DXR = 0, + //! The decoded value is INVERSE depth (disparity-like; larger = nearer). + XR_LIFT_DEPTH_ENCODING_INVERSE_DXR = 1, + XR_LIFT_DEPTH_ENCODING_MAX_ENUM_DXR = 0x7FFFFFFF +} XrLiftDepthEncodingDXR; + typedef enum XrLiftStateDXR { //! No module, or the module failed. supportedModes is 0. XR_LIFT_STATE_UNAVAILABLE_DXR = 0, @@ -248,6 +271,20 @@ typedef struct XrLiftPropertiesDXR { XrDuration typicalLatency; //!< submit→result, ns, as the module reports it; 0 = unknown } XrLiftPropertiesDXR; +/*! + * Auxiliary-depth capability (spec v3). Chain on XrLiftPropertiesDXR::next; + * filled by xrGetLiftPropertiesDXR. @c auxDepthSupported XR_TRUE = an SBS / + * NVIEW stream created with XrLiftDepthRequestDXR returns the depth map of the + * same conversion with each result; @c auxDepthSemantics is what that depth + * means (METRIC = metres). A runtime or module without it leaves XR_FALSE. + */ +typedef struct XrLiftDepthPropertiesDXR { + XrStructureType type; //!< XR_TYPE_LIFT_DEPTH_PROPERTIES_DXR + void* XR_MAY_ALIAS next; + XrBool32 auxDepthSupported; + XrLiftDepthSemanticsDXR auxDepthSemantics; +} XrLiftDepthPropertiesDXR; + /*! * One conversion stream. @c mode is ONE bit the runtime reported in * XrLiftPropertiesDXR::supportedModes. @c inputScale (0, 1] asks the module to @@ -262,6 +299,24 @@ typedef struct XrLiftStreamCreateInfoDXR { float inputScale; } XrLiftStreamCreateInfoDXR; +/*! + * Ask an SBS / NVIEW stream for its depth map as an auxiliary output (spec + * v3). Chain on XrLiftStreamCreateInfoDXR::next. XR_ERROR_VALIDATION_FAILURE + * on a DEPTH or GAUSSIANS stream (a DEPTH stream's result already is depth). + * Advisory: if the module cannot produce it (XrLiftDepthPropertiesDXR:: + * auxDepthSupported XR_FALSE) the stream is still created and every + * XrLiftDepthResultDXR reports @c depthValid XR_FALSE. + * + * @c preferredFormat is a DXGI_FORMAT hint (DXGI_FORMAT_R32_FLOAT or + * DXGI_FORMAT_R16_FLOAT; 0 = the module's choice). The result's @c format is + * authoritative. + */ +typedef struct XrLiftDepthRequestDXR { + XrStructureType type; //!< XR_TYPE_LIFT_DEPTH_REQUEST_DXR + const void* XR_MAY_ALIAS next; + int64_t preferredFormat; //!< DXGI_FORMAT hint; 0 = module's choice +} XrLiftDepthRequestDXR; + /*! * Per-frame conversion parameters. Chain on XrLiftFrameSubmitInfoDXR::next or * XrWeaveRectLiftDXR::next; omitted = the module's defaults (auto convergence, @@ -275,8 +330,11 @@ typedef struct XrLiftStreamCreateInfoDXR { * produces (2 for SBS; ignored for DEPTH / GAUSSIANS), ≤ maxViews. With * EXPLICIT viewpoints, @c viewpoints holds @c viewCount display-space positions * (metres; the runtime rebases them to the rect centre before the module sees - * them, spec v2); on the weave path viewpoints must be TRACKED. Chain - * XrLiftViewControlDXR on @c next to shape TRACKED viewpoints (spec v2). @c focalPx is the + * them, spec v2) — on the plain submit path and, since spec v3, on lifted + * weave rects too. Chain XrLiftViewControlDXR on @c next to shape TRACKED + * viewpoints (spec v2), and an XrDisplayRigDXR or XrCameraRigDXR + * (XR_DXR_view_rig) to derive them with the app's own rig (spec v3, see + * XrLiftResultViewpointsDXR). @c focalPx is the * submitted image's focal length in pixels of @c extent (for a photo, e.g. the * fx of an estimated intrinsics); <= 0 = unknown, the module assumes its * default field of view. Photo → Gaussians modules take it as input; DEPTH / @@ -341,6 +399,25 @@ typedef struct XrLiftViewControlDXR { * space (panel centre = 0) and @c rectSize the region's physical size. * @c viewpointCountOutput 0 = no viewpoints were known (the module used its * defaults), or a DEPTH stream. + * + * App rig (spec v3, ADR-048 Addendum A). An XrDisplayRigDXR or XrCameraRigDXR + * chained on XrLiftOptionsDXR (or directly on XrWeaveRectLiftDXR; direct wins; + * a camera rig wins over a display rig) makes the runtime derive the TRACKED + * viewpoints with the SAME math xrLocateViews runs for that rig, with the + * lifted rect as the screen: display rig = the processed eyes (ipdFactor, + * parallaxFactor) scaled by perspectiveFactor; camera rig = the physical + * viewer whose off-axis frustum onto the rect equals the camera rig's view + * frustum (convergenceDiopters, verticalFov, metersToVirtual; + * convergenceDiopters 0 = parallel cameras = one viewpoint for every view). + * The rig's @c pose and virtualDisplayHeight only place the app's virtual + * world and do not change the physical viewpoints. The rig replaces + * XrLiftViewControlDXR's ipdFactor / parallaxFactor; its axisMode, clamp and + * recentering still apply, to the viewer BEFORE the rig maps it — chain + * XrLiftViewControlDXR with XR_LIFT_AXIS_MODE_XYZ_DXR and recentering OFF for + * viewpoints identical to xrLocateViews'. The rig rides with the options, + * like EXPLICIT viewpoints: a submit carrying options without a rig clears + * it; a submit with no options keeps the stream's last one. Ignored for + * EXPLICIT viewpoints and for DEPTH / GAUSSIANS streams. */ typedef struct XrLiftResultViewpointsDXR { XrStructureType type; //!< XR_TYPE_LIFT_RESULT_VIEWPOINTS_DXR @@ -351,6 +428,79 @@ typedef struct XrLiftResultViewpointsDXR { XrExtent2Df rectSize; //!< metres } XrLiftResultViewpointsDXR; +/*! + * The auxiliary depth map of a result (spec v3, ADR-048 Addendum A). Chain on + * XrLiftResultDXR::next of an SBS / NVIEW stream created with + * XrLiftDepthRequestDXR; filled on every successful acquire. @c depthValid + * XR_FALSE = no depth for this result (a module without aux depth, or it + * declined this frame); every other field is then zero. + * + * Handles follow the main result's export rule: @c depthTexture (a D3D11 NT + * shared handle, runtime-owned, @c extent / @c format) and @c fence are + * handed out on the FIRST acquire that carries depth and again whenever the + * depth texture is reallocated; NULL otherwise. @c fence is the SAME fence + * object as XrLiftResultDXR::fence (a second handle to it; close both) and + * @c fenceValue equals XrLiftResultDXR::fenceValue: one wait covers the views + * and the depth. The caller finishes sampling before its next acquire. + * + * Samples: decode @c s as d = valueScale * s + valueOffset, then + * depth = d (LINEAR) or 1 / d (INVERSE), in @c units (METRIC = metres along + * the source camera's optical axis; RELATIVE = unitless, larger = farther). + * @c nearDepth / @c farDepth bound the decoded depth (0 = unknown). + * @c sameInference XR_TRUE = the depth comes from the very inference that + * produced this result's views; @c frameId always equals + * XrLiftResultDXR::frameId. + * + * Intrinsics (@c intrinsicsValid): the source camera the module assumed, in + * pixels of the depth map (u right, v down, origin at the top-left corner): + * @c focalLengthPx and @c principalPointPx. With metric depth they place every + * texel in the CAMERA's metric space: + * X = (u - cx) * depth / fx, Y = -(v - cy) * depth / fy, Z = -depth. + * + * Display alignment (@c transformValid): @c depthToDisplay maps a texel into + * DISPLAY space (metres, panel centre = 0, +x right, +y up, +z toward the + * viewer — the frame of XrLiftResultViewpointsDXR::rectCenter) the way the + * woven result presents it: the source image fills the lifted rect, seen from + * @c viewpoint (the midpoint of the viewpoints the views were synthesized + * for, rect-relative, as in XrLiftResultViewpointsDXR), and the depth + * @c convergenceDepth (decoded units) lies ON the screen plane. Column-major + * 4x4 (m[col * 4 + row]) applied to the homogeneous column + * (un * depth, vn * depth, depth, 1), where un = (u + 0.5) / extent.width and + * vn = (v + 0.5) / extent.height are the texel centre's normalised + * coordinates; the output's w is 1. depth = convergenceDepth lands on the + * rect, depth -> 0 converges on the viewpoint, and every texel stays on the + * ray from the viewpoint through its rect position — so content the app + * renders from XrLiftResultViewpointsDXR's eyes lines up with the lifted + * picture. The spec (§4.3) derives it. Valid only with METRIC units, a known + * @c convergenceDepth and a known rect; RELATIVE depth reports + * transformValid / intrinsicsValid XR_FALSE rather than inventing a scale. + */ +typedef struct XrLiftDepthResultDXR { + XrStructureType type; //!< XR_TYPE_LIFT_DEPTH_RESULT_DXR + void* XR_MAY_ALIAS next; + XrBool32 depthValid; + void* depthTexture; //!< shared HANDLE on first depth acquire / realloc, else NULL + void* fence; //!< shared fence HANDLE with depthTexture, else NULL + uint64_t fenceValue; //!< == XrLiftResultDXR::fenceValue + XrExtent2Di extent; //!< depth map size, texels + int64_t format; //!< DXGI_FORMAT (R32_FLOAT / R16_FLOAT) + XrLiftDepthSemanticsDXR units; + XrLiftDepthEncodingDXR encoding; + float valueScale; + float valueOffset; + float nearDepth; //!< decoded units; 0 = unknown + float farDepth; //!< decoded units; 0 = unknown + XrBool32 sameInference; + uint64_t frameId; //!< == XrLiftResultDXR::frameId + XrBool32 intrinsicsValid; + XrVector2f focalLengthPx; //!< fx, fy in depth-map pixels + XrVector2f principalPointPx; //!< cx, cy in depth-map pixels + float convergenceDepth; //!< decoded depth on the screen plane; <= 0 = unknown + XrVector3f viewpoint; //!< rect-relative metres (XrLiftResultViewpointsDXR frame) + XrBool32 transformValid; + float depthToDisplay[16]; //!< column-major; see above +} XrLiftDepthResultDXR; + /*! * One frame into a stream. NON-BLOCKING: the runtime snapshots the input's * @c extent (top-left sub-rect of @c inputTexture) into its own mailbox before @@ -389,7 +539,7 @@ typedef struct XrLiftFrameSubmitInfoDXR { */ typedef struct XrLiftResultDXR { XrStructureType type; //!< XR_TYPE_LIFT_RESULT_DXR - void* XR_MAY_ALIAS next; //!< chain XrLiftResultViewpointsDXR here (v2) + void* XR_MAY_ALIAS next; //!< chain XrLiftResultViewpointsDXR (v2) / XrLiftDepthResultDXR (v3) uint64_t frameId; //!< the xrSubmitLiftFrameDXR frame this converts XrTime sourceTime; //!< that frame's sourceTime, verbatim void* outputTexture; //!< shared HANDLE on first acquire / realloc, else NULL @@ -444,7 +594,7 @@ typedef struct XrLiftBlobDXR { */ typedef struct XrWeaveRectLiftDXR { XrStructureType type; //!< XR_TYPE_WEAVE_RECT_LIFT_DXR - const void* XR_MAY_ALIAS next; //!< chain XrLiftOptionsDXR / XrLiftViewControlDXR here + const void* XR_MAY_ALIAS next; //!< chain XrLiftOptionsDXR / XrLiftViewControlDXR / a view rig here uint32_t rectIndex; //!< index into XrWeaveSubmitRectsDXR::rects XrLiftStreamDXR stream; } XrWeaveRectLiftDXR; diff --git a/src/xrt/include/xrt/xrt_display_processor_d3d11.h b/src/xrt/include/xrt/xrt_display_processor_d3d11.h index dfd9eb7af8..0582b7d341 100644 --- a/src/xrt/include/xrt/xrt_display_processor_d3d11.h +++ b/src/xrt/include/xrt/xrt_display_processor_d3d11.h @@ -784,6 +784,42 @@ struct xrt_display_processor_d3d11 * @param panel_y Window client-area top edge, screen-relative device px. */ void (*set_present_origin)(struct xrt_display_processor_d3d11 *xdp, int32_t panel_x, int32_t panel_y); + + /*! + * The auxiliary DEPTH map of the conversion @ref lift_convert just + * returned for stream @p id (ADR-048 Addendum A, XR_DXR_lift v3). + * + * Called by the runtime on its lift thread, immediately after a + * successful lift_convert of an SBS / NVIEW stream that was created with + * XRT_DP_LIFT_AUX_DEPTH in xrt_dp_lift_stream_info::aux_outputs — never + * for any other stream, never before the stream's first conversion. + * Same threading, device and context as lift_convert. + * + * The depth should come from the SAME inference that produced the views + * (report it in xrt_dp_lift_depth::same_inference). @p out->resource is + * DP-owned and must stay valid until the next lift_convert on @p id — + * the runtime copies it into its own export texture before then, so a + * module may keep one depth texture per stream and overwrite it. + * + * @p out arrives zeroed with struct_size pre-set (xrt_dp_lift_depth_init); + * write only fields within it. + * + * Optional — an absent slot (older plug-in struct_size) or NULL ⟹ no + * auxiliary depth (XrLiftDepthResultDXR::depthValid stays XR_FALSE). + * Report the capability through xrt_dp_lift_caps::aux_outputs. Appended + * after @ref set_present_origin per ADR-020 (append-only within a major, + * no ABI bump), announced by XRT_DP_D3D11_HAS_LIFT_DEPTH. + * + * @param xdp Pointer to self. + * @param id The plug-in's stream id (lift_stream_create). + * @param d3d11_context The lift device's immediate context. + * @param[out] out Depth map + metadata (struct_size pre-set). + * @return false = no depth for this conversion. + */ + bool (*lift_get_depth)(struct xrt_display_processor_d3d11 *xdp, + uint64_t id, + void *d3d11_context, + struct xrt_dp_lift_depth *out); }; @@ -867,7 +903,8 @@ XRT_DP_ABI_ASSERT(offsetof(struct xrt_display_processor_d3d11, lift_convert_blob XRT_DP_ABI_ASSERT(offsetof(struct xrt_display_processor_d3d11, set_overlay_2d) == XRT_DP_D3D11_BASE_OFF + 30 * sizeof(void *), XRT_DP_ABI_MSG); XRT_DP_ABI_ASSERT(offsetof(struct xrt_display_processor_d3d11, set_overlay_2d_filter_strength) == XRT_DP_D3D11_BASE_OFF + 31 * sizeof(void *), XRT_DP_ABI_MSG); XRT_DP_ABI_ASSERT(offsetof(struct xrt_display_processor_d3d11, set_present_origin) == XRT_DP_D3D11_BASE_OFF + 32 * sizeof(void *), XRT_DP_ABI_MSG); -XRT_DP_ABI_ASSERT(sizeof(struct xrt_display_processor_d3d11) == XRT_DP_D3D11_BASE_OFF + 33 * sizeof(void *), XRT_DP_ABI_MSG); +XRT_DP_ABI_ASSERT(offsetof(struct xrt_display_processor_d3d11, lift_get_depth) == XRT_DP_D3D11_BASE_OFF + 33 * sizeof(void *), XRT_DP_ABI_MSG); +XRT_DP_ABI_ASSERT(sizeof(struct xrt_display_processor_d3d11) == XRT_DP_D3D11_BASE_OFF + 34 * sizeof(void *), XRT_DP_ABI_MSG); /*! * Defined when this header carries the five lift slots (lift_get_caps, @@ -879,6 +916,14 @@ XRT_DP_ABI_ASSERT(sizeof(struct xrt_display_processor_d3d11) */ #define XRT_DP_D3D11_HAS_LIFT 1 +/*! + * Defined when this header carries the lift_get_depth slot (ADR-048 + * Addendum A, XR_DXR_lift v3 auxiliary depth), so a plug-in built against an + * older runtime can #ifdef-guard it. Purely additive: no + * XRT_PLUGIN_API_VERSION_CURRENT bump (ADR-020). + */ +#define XRT_DP_D3D11_HAS_LIFT_DEPTH 1 + /*! * Defined when this header carries the get_background_preview slot, so a * plug-in built against an older runtime can #ifdef-guard its implementation @@ -1438,6 +1483,33 @@ xrt_display_processor_d3d11_has_lift(struct xrt_display_processor_d3d11 *xdp) xdp->lift_stream_destroy != NULL && xdp->lift_convert != NULL; } +/*! + * @copydoc xrt_display_processor_d3d11::lift_get_depth + * + * Helper: pre-sets @p out (xrt_dp_lift_depth_init) and returns false — no + * depth — when the slot is absent (older plug-in struct_size) or NULL, when + * the DP declines, or when it returned no resource. + * + * @public @memberof xrt_display_processor_d3d11 + */ +static inline bool +xrt_display_processor_d3d11_lift_get_depth(struct xrt_display_processor_d3d11 *xdp, + uint64_t id, + void *d3d11_context, + struct xrt_dp_lift_depth *out) +{ + xrt_dp_lift_depth_init(out); + if (!XRT_DP_HAS_SLOT(xdp, lift_get_depth) || xdp->lift_get_depth == NULL) { + return false; + } + if (!xdp->lift_get_depth(xdp, id, d3d11_context, out) || out->resource == NULL || out->width == 0 || + out->height == 0) { + xrt_dp_lift_depth_init(out); + return false; + } + return true; +} + /*! * @copydoc xrt_display_processor_d3d11::set_overlay_2d * diff --git a/src/xrt/include/xrt/xrt_dp_lift.h b/src/xrt/include/xrt/xrt_dp_lift.h index 48ee688caf..58e81771b6 100644 --- a/src/xrt/include/xrt/xrt_dp_lift.h +++ b/src/xrt/include/xrt/xrt_dp_lift.h @@ -8,7 +8,8 @@ * N-view synthesis, photo → Gaussian splats). The runtime reaches it through * optional appended slots on the per-API DP vtable — today only * @ref xrt_display_processor_d3d11 (lift_get_caps … lift_convert_blob, guarded - * by XRT_DP_D3D11_HAS_LIFT) — and exposes it to apps as XR_DXR_lift. + * by XRT_DP_D3D11_HAS_LIFT; plus lift_get_depth, guarded by + * XRT_DP_D3D11_HAS_LIFT_DEPTH) — and exposes it to apps as XR_DXR_lift. * * The split of labour (ADR-042, ADR-007): * - the PLUG-IN converts, synchronously, one frame per call. It never weaves @@ -88,8 +89,37 @@ struct xrt_dp_lift_caps uint32_t state; //!< 0 unavailable, 1 activating, 2 ready uint64_t typical_latency_ns; //!< submit→result as the module expects it; 0 = unknown char backend[32]; //!< NUL-terminated module name (informational) + + /* + * ── Auxiliary depth (ADR-048 Addendum A, XRT_DP_LIFT_HAS_AUX_DEPTH) ────── + * Appended; written only when struct_size covers them. + */ + + //! XRT_DP_LIFT_AUX_* bits the module can return alongside SBS / NVIEW + //! views (through xrt_display_processor_d3d11::lift_get_depth); 0 = none. + uint32_t aux_outputs; + //! Meaning of that auxiliary depth: XRT_DP_LIFT_DEPTH_RELATIVE / _METRIC. + uint32_t aux_depth_semantics; }; +/*! + * @name Auxiliary outputs (ADR-048 Addendum A) + * + * Announced by XRT_DP_LIFT_HAS_AUX_DEPTH (append-only, ADR-020: no ABI bump), + * together with the D3D11 slot lift_get_depth (XRT_DP_D3D11_HAS_LIFT_DEPTH). + * @{ + */ +#define XRT_DP_LIFT_HAS_AUX_DEPTH 1 + +//! xrt_dp_lift_caps::aux_outputs / xrt_dp_lift_stream_info::aux_outputs bit: +//! the depth map of the same inference that produced the views. +#define XRT_DP_LIFT_AUX_DEPTH 1u + +//! xrt_dp_lift_depth::encoding — values match XR_LIFT_DEPTH_ENCODING_*_DXR. +#define XRT_DP_LIFT_DEPTH_ENCODING_LINEAR 0u //!< decoded value = depth +#define XRT_DP_LIFT_DEPTH_ENCODING_INVERSE 1u //!< decoded value = 1 / depth +/*! @} */ + //! Stream creation parameters (runtime-filled). struct xrt_dp_lift_stream_info { @@ -97,6 +127,75 @@ struct xrt_dp_lift_stream_info uint32_t mode; //!< ONE XRT_DP_LIFT_MODE_* bit uint32_t content_hint; //!< 0 video, 1 photo float input_scale; //!< (0,1]: convert at reduced resolution; 1 = native + + /* + * ── Auxiliary outputs (ADR-048 Addendum A, XRT_DP_LIFT_HAS_AUX_DEPTH) ─── + * Appended; read only when struct_size covers them. + */ + + //! XRT_DP_LIFT_AUX_* bits the app asked for on this SBS / NVIEW stream + //! (never set for DEPTH / GAUSSIANS). A module that sees + //! XRT_DP_LIFT_AUX_DEPTH keeps each conversion's depth map for + //! lift_get_depth — ideally from the same inference as the views. + uint32_t aux_outputs; + //! DXGI_FORMAT hint for the depth map (R32_FLOAT / R16_FLOAT); 0 = any. + uint32_t aux_depth_format; +}; + +/*! + * The auxiliary depth map of the conversion lift_convert just returned + * (xrt_display_processor_d3d11::lift_get_depth). The runtime pre-sets + * @c struct_size (zeroing the rest); the DP writes only fields within it. + * + * Vendor-neutral by construction: everything a consumer needs to place the + * depth in metric space — the encoding of the samples, their unit, the + * camera the module assumed, and the depth it put on the screen plane — and + * nothing about how the module got there. The RUNTIME derives the display + * alignment (XrLiftDepthResultDXR::depthToDisplay) from these plus the + * viewpoints and rect it already owns; the plug-in only reports. + * + * Decoding (runtime and app): d = value_scale * sample + value_offset; depth = + * d (LINEAR) or 1 / d (INVERSE), in @c units. + */ +struct xrt_dp_lift_depth +{ + uint32_t struct_size; + //! DXGI_FORMAT of @c resource: DXGI_FORMAT_R32_FLOAT (41) or _R16_FLOAT (54). + uint32_t format; + //! ID3D11Resource* (a 2D texture) on the lift device, owned by the DP, + //! valid until the NEXT lift_convert on this stream (the runtime copies it + //! out before then). NULL = no depth for this conversion. + void *resource; + uint32_t width; //!< texels + uint32_t height; //!< texels + //! XRT_DP_LIFT_DEPTH_RELATIVE / XRT_DP_LIFT_DEPTH_METRIC (metres). + uint32_t units; + //! XRT_DP_LIFT_DEPTH_ENCODING_*. + uint32_t encoding; + float value_scale; //!< decode: d = value_scale * sample + value_offset (0 is read as 1) + float value_offset; //!< (see value_scale) + //! The input frame the depth refers to (w x h of the lift_convert call). + uint32_t source_width; + uint32_t source_height; + //! Pinhole intrinsics the module assumed, in DEPTH-MAP pixels (u right, + //! v down, origin top-left corner); focal <= 0 = unknown. + float focal_x_px; + float focal_y_px; + float principal_x_px; + float principal_y_px; + //! Decoded depth range of this map, in @c units; 0 = unknown. + float near_depth; + float far_depth; + //! The decoded depth (in @c units) the conversion put at ZERO disparity — + //! on the screen plane — after resolving xrt_dp_lift_params::convergence + //! (including AUTO). <= 0 = unknown (the runtime then reports no display + //! transform). + float convergence_depth; + //! Non-zero = produced by the SAME inference as the views the preceding + //! lift_convert returned (not a second pass, not a cached older frame). + uint32_t same_inference; + //! The module's own frame / inference counter, informational (logged). + uint64_t vendor_frame_id; }; /*! @@ -185,8 +284,35 @@ struct xrt_dp_lift_params //! XRT_DP_LIFT_VIEWPOINT_FRAME_*: origin of the viewpoints passed to //! lift_convert. Always RECT from a runtime that fills this field. uint32_t viewpoint_frame; + + /* + * ── App rig (ADR-048 Addendum A, XRT_DP_LIFT_HAS_APP_RIG) ────────────── + * Appended; read only when struct_size covers them. + */ + + //! XRT_DP_LIFT_VIEWPOINTS_*: where the viewpoints came from. With + //! DISPLAY_RIG / CAMERA_RIG they are the app's rig eyes (the xrLocateViews + //! math, the rect as the screen) — the same eyes the app renders its own + //! 3D from, so a module must reproduce them as given (no extra gain). + uint32_t viewpoint_source; + //! The reference viewing distance the runtime used (axis pinning, rig + //! nominal viewer), metres from the rect plane; 0 = unknown. + float nominal_z_m; }; +/*! + * @name App rig (ADR-048 Addendum A) — xrt_dp_lift_params appended fields + * Announced by XRT_DP_LIFT_HAS_APP_RIG (append-only, ADR-020: no ABI bump). + * @{ + */ +#define XRT_DP_LIFT_HAS_APP_RIG 1 + +#define XRT_DP_LIFT_VIEWPOINTS_TRACKED 0u //!< tracked eyes + the viewpoint policy +#define XRT_DP_LIFT_VIEWPOINTS_EXPLICIT 1u //!< the app's explicit viewpoints +#define XRT_DP_LIFT_VIEWPOINTS_DISPLAY_RIG 2u //!< tracked eyes through the app's display rig +#define XRT_DP_LIFT_VIEWPOINTS_CAMERA_RIG 3u //!< tracked eyes through the app's camera rig +/*! @} */ + /*! * Pre-set @p caps for a lift_get_caps call: zero it and stamp struct_size. */ @@ -200,6 +326,19 @@ xrt_dp_lift_caps_init(struct xrt_dp_lift_caps *caps) caps->struct_size = (uint32_t)sizeof(*caps); } +/*! + * Pre-set @p d for a lift_get_depth call: zero it and stamp struct_size. + */ +static inline void +xrt_dp_lift_depth_init(struct xrt_dp_lift_depth *d) +{ + uint8_t *p = (uint8_t *)d; + for (uint32_t i = 0; i < (uint32_t)sizeof(*d); i++) { + p[i] = 0; + } + d->struct_size = (uint32_t)sizeof(*d); +} + #ifdef __cplusplus } #endif From 016e5dfdf0771de7c12a4a9cc8554e323c4e1b0f Mon Sep 17 00:00:00 2001 From: dfattal Date: Thu, 8 Oct 2026 03:15:45 -0700 Subject: [PATCH 2/3] feat(lift): app rig drives lift viewpoints + auxiliary metric depth on the lift thread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R5 (#1860): an XrDisplayRigDXR / XrCameraRigDXR on a lift is mapped with the shared displayxr-common rig core xrLocateViews runs (u_lift_rig.{h,c}; aux_util now links displayxr::math_core). Display rig = dxr_display3d_compute_views eye_display with the rect as the screen (m2v 1); camera rig = the physical viewer whose Kooima frustum onto the rect equals the camera frustum (E = Z0 (l invd + z), Z0 = rect_h / 2 tan(vfov/2)). The rig replaces the policy's ipd / parallax; axis mask, ease-back and clamp still run on the viewer first. It rides with the options (cleared by options without a rig), on submits and on lifted weave rects. EXPLICIT viewpoints are accepted on lifted weave rects (staged per rect by a new lift_weave_rect_viewpoints IPC call: 8 rects of them do not fit IPC_BUF_SIZE). The DP is told viewpoint_source + nominal_z_m; the echo reports the rig eyes. R6 (#1861): SBS / NVIEW streams created with XrLiftDepthRequestDXR call the new DP slot lift_get_depth right after each successful lift_convert; the depth is copied into the same out ring slot (second keyed-mutex texture), then into a per-stream depth export texture under the views' fence (one fence value covers both). The runtime derives the intrinsics (METRIC only) and depthToDisplay (u_lift_depth.{h,c}: X = R + C + (P - C) d / dc, one column-major 4x4) from the viewpoints and rect it already owns; RELATIVE depth reports neither. New IPC call lift_get_depth_output; lift_stream_create carries aux_outputs / aux_depth_format; ipc_lift_result carries xrt_lift_depth_info. sim_display's fake lift returns a synthetic metric ramp (1..2 m) so the path runs without hardware (SIM_DISPLAY_FAKE_LIFT=1). Tests: tests_aux_lift_rig_depth — display rig == located eye_display / m2v after window rebase (and the same frusta), camera rig frusta == the camera3d core's, infinite convergence collapses the views, convergence depth lands on the rect, d -> 0 at the viewpoint, rays stay collinear, invalid inputs, decode. Co-Authored-By: Claude Opus 5.5 --- src/xrt/auxiliary/util/CMakeLists.txt | 9 + src/xrt/auxiliary/util/u_lift_depth.c | 72 +++ src/xrt/auxiliary/util/u_lift_depth.h | 78 ++++ src/xrt/auxiliary/util/u_lift_rig.c | 127 ++++++ src/xrt/auxiliary/util/u_lift_rig.h | 97 ++++ .../d3d11_service/comp_d3d11_service.cpp | 36 +- .../d3d11_service/comp_d3d11_service.h | 13 +- .../compositor/d3d11_service/d3d11_lift.cpp | 414 +++++++++++++++++- src/xrt/compositor/d3d11_service/d3d11_lift.h | 25 +- .../sim_display/sim_display_lift_d3d11.cpp | 100 +++++ .../sim_display/sim_display_lift_d3d11.h | 4 + .../sim_display_processor_d3d11.cpp | 10 + src/xrt/include/xrt/xrt_lift.h | 59 +++ src/xrt/ipc/client/ipc_client_compositor.c | 17 +- src/xrt/ipc/client/ipc_client_lift.c | 65 ++- src/xrt/ipc/client/ipc_client_lift.h | 23 +- src/xrt/ipc/server/ipc_server.h | 8 + src/xrt/ipc/server/ipc_server_handler.c | 102 ++++- src/xrt/ipc/shared/ipc_protocol.h | 31 +- src/xrt/ipc/shared/proto.json | 23 +- src/xrt/state_trackers/oxr/oxr_lift.c | 188 +++++++- src/xrt/state_trackers/oxr/oxr_objects.h | 2 + src/xrt/state_trackers/oxr/oxr_weave.c | 36 +- src/xrt/targets/cli/cli_cmd_lift.cpp | 4 +- tests/CMakeLists.txt | 5 + tests/tests_aux_lift_rig_depth.cpp | 243 ++++++++++ 26 files changed, 1732 insertions(+), 59 deletions(-) create mode 100644 src/xrt/auxiliary/util/u_lift_depth.c create mode 100644 src/xrt/auxiliary/util/u_lift_depth.h create mode 100644 src/xrt/auxiliary/util/u_lift_rig.c create mode 100644 src/xrt/auxiliary/util/u_lift_rig.h create mode 100644 tests/tests_aux_lift_rig_depth.cpp diff --git a/src/xrt/auxiliary/util/CMakeLists.txt b/src/xrt/auxiliary/util/CMakeLists.txt index 0fc069438b..aaaa09608d 100644 --- a/src/xrt/auxiliary/util/CMakeLists.txt +++ b/src/xrt/auxiliary/util/CMakeLists.txt @@ -26,8 +26,12 @@ add_library( u_bitwise.c u_cursor_depth.c u_cursor_depth.h + u_lift_depth.c + u_lift_depth.h u_lift_mailbox.c u_lift_mailbox.h + u_lift_rig.c + u_lift_rig.h u_lift_viewpoint.c u_lift_viewpoint.h u_snap_grid.c @@ -202,6 +206,11 @@ endif() # (compositors use mcp_capture, oxr uses mcp_server / mcp_log_ring). target_link_libraries(aux_util PUBLIC displayxr_mcp::mcp) +# XR_DXR_lift app rig (u_lift_rig.c, ADR-048 Addendum A) runs the shared +# displayxr-common rig core - the same functions xrLocateViews runs (aux_math +# FetchContents displayxr-common before this directory). +target_link_libraries(aux_util PRIVATE displayxr::math_core) + # CoreFoundation needed for CFRetain/CFRelease in u_handles.c (Metal handle ref counting). if(APPLE) target_link_libraries(aux_util PRIVATE "-framework CoreFoundation") diff --git a/src/xrt/auxiliary/util/u_lift_depth.c b/src/xrt/auxiliary/util/u_lift_depth.c new file mode 100644 index 0000000000..024cdb01bb --- /dev/null +++ b/src/xrt/auxiliary/util/u_lift_depth.c @@ -0,0 +1,72 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_lift auxiliary depth transform (ADR-048 Addendum A) - see + * u_lift_depth.h. + * @ingroup aux_util + */ + +#include "util/u_lift_depth.h" + +#include +#include + +static void +identity(float m[16]) +{ + memset(m, 0, 16 * sizeof(float)); + m[0] = m[5] = m[10] = m[15] = 1.0f; +} + +bool +u_lift_depth_to_display(const float viewpoint[3], + const float rect_center[3], + float rect_w, + float rect_h, + float convergence_depth, + float out_m[16]) +{ + identity(out_m); + if (viewpoint == NULL || rect_center == NULL || !(rect_w > 0.0f) || !(rect_h > 0.0f) || + !(convergence_depth > 0.0f) || !isfinite(rect_w) || !isfinite(rect_h) || !isfinite(convergence_depth) || + !(viewpoint[2] > 0.0f) || !isfinite(viewpoint[0]) || !isfinite(viewpoint[1]) || !isfinite(viewpoint[2])) { + return false; + } + const float inv = 1.0f / convergence_depth; + const float cx = viewpoint[0], cy = viewpoint[1], cz = viewpoint[2]; + memset(out_m, 0, 16 * sizeof(float)); + // col 0: coefficient of un*d + out_m[0] = rect_w * inv; + // col 1: coefficient of vn*d + out_m[5] = -rect_h * inv; + // col 2: coefficient of d + out_m[8] = (-0.5f * rect_w - cx) * inv; + out_m[9] = (0.5f * rect_h - cy) * inv; + out_m[10] = -cz * inv; + // col 3: constant + out_m[12] = rect_center[0] + cx; + out_m[13] = rect_center[1] + cy; + out_m[14] = rect_center[2] + cz; + out_m[15] = 1.0f; + return true; +} + +void +u_lift_depth_transform_point(const float m[16], float un, float vn, float d, float out_xyz[3]) +{ + const float v[4] = {un * d, vn * d, d, 1.0f}; + for (int r = 0; r < 3; r++) { + out_xyz[r] = m[0 * 4 + r] * v[0] + m[1 * 4 + r] * v[1] + m[2 * 4 + r] * v[2] + m[3 * 4 + r] * v[3]; + } +} + +float +u_lift_depth_decode(float sample, float scale, float offset, bool inverse) +{ + const float d = (scale != 0.0f ? scale : 1.0f) * sample + offset; + if (!inverse) { + return d; + } + return d != 0.0f ? 1.0f / d : 0.0f; +} diff --git a/src/xrt/auxiliary/util/u_lift_depth.h b/src/xrt/auxiliary/util/u_lift_depth.h new file mode 100644 index 0000000000..4601d7b211 --- /dev/null +++ b/src/xrt/auxiliary/util/u_lift_depth.h @@ -0,0 +1,78 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_lift auxiliary depth (ADR-048 Addendum A): the transform that + * places a lifted frame's depth map in display space, aligned with the + * viewpoints its views were synthesized for. + * + * Model (XR_DXR_lift.md §4.3). The woven result presents the source image on + * the lifted rect, seen from the viewpoint midpoint C (rect-relative), with + * the convergence depth dc on the screen plane. A texel at normalised + * coordinates (un, vn) (texel centre, u right, v down, [0, 1]) sits on the + * rect at + * P = (W * (un - 0.5), H * (0.5 - vn), 0) + * and a texel of decoded depth d lies on the ray from C through P, at the + * fraction d / dc of the way: + * X = R + C + (P - C) * d / dc + * (R = rect centre in display space). d = dc -> on the rect, d -> 0 -> at the + * viewpoint, d > dc -> behind the screen. X is linear in (un*d, vn*d, d, 1), + * so it is one 4x4 matrix — column-major, m[col * 4 + row], output w = 1: + * + * col 0 (un*d): ( W/dc, 0, 0, 0 ) + * col 1 (vn*d): ( 0, -H/dc, 0, 0 ) + * col 2 (d) : ( (-W/2 - Cx)/dc, (H/2 - Cy)/dc, -Cz/dc, 0 ) + * col 3 (1) : ( Rx + Cx, Ry + Cy, Rz + Cz, 1 ) + * + * Pure: no locks, no clock, no GPU. + * + * @ingroup aux_util + */ + +#pragma once + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/*! + * Build the depth-to-display transform (see the file comment). + * + * @param viewpoint C: the viewpoints' midpoint, rect-relative metres + * @param rect_center R: the rect centre in display space, metres + * @param rect_w W, metres + * @param rect_h H, metres + * @param convergence_depth dc, decoded depth on the screen plane + * @param out_m 16 floats, column-major + * @return false (out_m = identity) when W, H, dc or C.z is not positive / + * finite — the caller reports the transform invalid. + */ +bool +u_lift_depth_to_display(const float viewpoint[3], + const float rect_center[3], + float rect_w, + float rect_h, + float convergence_depth, + float out_m[16]); + +/*! + * Apply @p m to one texel: normalised texel centre (@p un, @p vn) at decoded + * depth @p d -> display-space point @p out_xyz. Reference for the spec and + * the tests (an app does the same multiply in its shader). + */ +void +u_lift_depth_transform_point(const float m[16], float un, float vn, float d, float out_xyz[3]); + +/*! + * Decode one sample: d = scale * s + offset (scale 0 is read as 1); INVERSE + * (@p inverse true) returns 1 / d, or 0 when d is 0. + */ +float +u_lift_depth_decode(float sample, float scale, float offset, bool inverse); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/auxiliary/util/u_lift_rig.c b/src/xrt/auxiliary/util/u_lift_rig.c new file mode 100644 index 0000000000..65c587ff4f --- /dev/null +++ b/src/xrt/auxiliary/util/u_lift_rig.c @@ -0,0 +1,127 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_lift app rig (ADR-048 Addendum A) - see u_lift_rig.h. + * @ingroup aux_util + */ + +#include "util/u_lift_rig.h" + +// displayxr-common's shared rig core: the same functions xrLocateViews runs. +#include "dxr_view_math.h" + +#include +#include + +#define U_LIFT_RIG_MAX_EYES 8 + +static float +clampf(float v, float lo, float hi, float dflt, bool *clamped) +{ + if (!isfinite(v)) { + *clamped = true; + return dflt; + } + if (v < lo) { + *clamped = true; + return lo; + } + if (v > hi) { + *clamped = true; + return hi; + } + return v; +} + +bool +u_lift_rig_sanitize(struct u_lift_rig *rig) +{ + bool clamped = false; + if (rig->type != U_LIFT_RIG_DISPLAY && rig->type != U_LIFT_RIG_CAMERA) { + memset(rig, 0, sizeof(*rig)); + rig->type = U_LIFT_RIG_NONE; + return false; + } + rig->ipd_factor = clampf(rig->ipd_factor, 0.0f, 1.0e4f, 1.0f, &clamped); + rig->parallax_factor = clampf(rig->parallax_factor, 0.0f, 1.0e4f, 1.0f, &clamped); + rig->perspective_factor = clampf(rig->perspective_factor, 0.1f, 10.0f, 1.0f, &clamped); + rig->inv_convergence_distance = clampf(rig->inv_convergence_distance, 0.0f, 20.0f, 0.0f, &clamped); + // tan of half of a [0.01, 3.13] rad full vertical FOV. + rig->half_tan_vfov = clampf(rig->half_tan_vfov, tanf(0.005f), tanf(1.565f), tanf(0.5f), &clamped); + if (!(rig->m2v > 0.0f)) { + rig->m2v = 1.0f; // 0 / unset = identity (XR_DXR_view_rig v3) + } else { + rig->m2v = clampf(rig->m2v, 0.0001f, 100000.0f, 1.0f, &clamped); + } + return clamped; +} + +bool +u_lift_rig_apply(const struct u_lift_rig *rig, + float nominal_z, + float rect_w, + float rect_h, + const float *in_xyz, + uint32_t count, + float *out_xyz) +{ + if (rig == NULL || in_xyz == NULL || out_xyz == NULL || count == 0 || count > U_LIFT_RIG_MAX_EYES) { + return false; + } + if (rig->type == U_LIFT_RIG_NONE) { + if (out_xyz != in_xyz) { + memmove(out_xyz, in_xyz, (size_t)count * 3 * sizeof(float)); + } + return true; + } + if (!(rect_w > 0.0f) || !(rect_h > 0.0f)) { + return false; + } + const float nz = nominal_z > 0.0f ? nominal_z : 0.5f; + const dxr_vec3 nominal = {0.0f, 0.0f, nz}; + const dxr_screen screen = {rect_w, rect_h}; + + dxr_vec3 eyes[U_LIFT_RIG_MAX_EYES]; + for (uint32_t i = 0; i < count; i++) { + eyes[i].x = in_xyz[3 * i + 0]; + eyes[i].y = in_xyz[3 * i + 1]; + eyes[i].z = in_xyz[3 * i + 2]; + } + + if (rig->type == U_LIFT_RIG_DISPLAY) { + dxr_display3d_tunables t = dxr_display3d_default_tunables(); + t.ipd_factor = rig->ipd_factor; + t.parallax_factor = rig->parallax_factor; + t.perspective_factor = rig->perspective_factor; + t.virtual_display_height = rect_h; // m2v = 1: eye_display stays in metres + dxr_display3d_view views[U_LIFT_RIG_MAX_EYES]; + dxr_display3d_compute_views(eyes, count, &nominal, &screen, &t, /*display_pose*/ NULL, + /*near_offset*/ 0.0f, /*far_offset*/ 0.0f, /*vulkan_flip_y*/ 0, views); + for (uint32_t i = 0; i < count; i++) { + out_xyz[3 * i + 0] = views[i].eye_display.x; + out_xyz[3 * i + 1] = views[i].eye_display.y; + out_xyz[3 * i + 2] = views[i].eye_display.z; + } + return true; + } + + // Camera rig: the camera's frustum, re-expressed as the physical viewer + // whose Kooima frustum onto the rect is the same (see the header). + dxr_camera3d_tunables t = dxr_camera3d_default_tunables(); + t.ipd_factor = rig->ipd_factor; + t.parallax_factor = rig->parallax_factor; + t.inv_convergence_distance = rig->inv_convergence_distance; + t.half_tan_vfov = rig->half_tan_vfov; + t.m2v = rig->m2v; + dxr_camera3d_view views[U_LIFT_RIG_MAX_EYES]; + dxr_camera3d_compute_views(eyes, count, &nominal, &screen, &t, /*camera_pose*/ NULL, 0.1f, 100.0f, views); + const float z0 = rect_h / (2.0f * t.half_tan_vfov); + const float k = z0 * t.inv_convergence_distance; // physical metres per world unit + for (uint32_t i = 0; i < count; i++) { + out_xyz[3 * i + 0] = k * views[i].eye_world.x; + out_xyz[3 * i + 1] = k * views[i].eye_world.y; + out_xyz[3 * i + 2] = z0 + k * views[i].eye_world.z; + } + return true; +} diff --git a/src/xrt/auxiliary/util/u_lift_rig.h b/src/xrt/auxiliary/util/u_lift_rig.h new file mode 100644 index 0000000000..0122063e3a --- /dev/null +++ b/src/xrt/auxiliary/util/u_lift_rig.h @@ -0,0 +1,97 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_lift app rig (ADR-048 Addendum A): map tracked eyes through + * an app's XR_DXR_view_rig display / camera rig into the PHYSICAL + * viewpoints a lift conversion synthesizes for. + * + * The lifted rect is the screen. The math is NOT re-implemented here: both + * rigs run through displayxr-common's shared core (dxr_view_math.h) — the very + * functions xrLocateViews runs (oxr_session.c, via displayxr_math_xrt) — and + * only the last step, expressing the rig's eye back in physical rect-relative + * metres, is local: + * + * - display rig: dxr_display3d_compute_views with the rect as the screen and + * virtual_display_height = the rect height (so m2v = 1). eye_display is + * then the processed eye (ipd / parallax factors, step 1a / 1b) scaled by + * the perspective factor — already physical metres. + * - camera rig: dxr_camera3d_compute_views (identity pose) gives the eye in + * world units, l = m2v * (processed - nominal). The camera's off-axis + * frustum (half-tangents ht * aspect, ht, sheared by l * invd) is EXACTLY + * the Kooima frustum onto the rect of the physical viewer + * E = Z0 * (l * invd + (0, 0, 1)), Z0 = rect_h / (2 * ht) + * — so E is the viewpoint (tests_aux_lift_rig_depth checks the frustum + * equality against the shared core). invd = 0 (convergence at infinity) + * collapses every eye to (0, 0, Z0): parallel cameras, no lift parallax. + * + * Pure: no locks, no clock, no GPU. + * + * @ingroup aux_util + */ + +#pragma once + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +#define U_LIFT_RIG_NONE 0u +#define U_LIFT_RIG_DISPLAY 1u +#define U_LIFT_RIG_CAMERA 2u + +/*! + * An app rig as the lift path consumes it (XrDisplayRigDXR / XrCameraRigDXR, + * boundary-converted the way xrLocateViews converts them: verticalFov -> + * half-tangent, convergenceDiopters = inverse convergence distance, 0 / unset + * metersToVirtual -> 1). The rig pose and virtualDisplayHeight do not appear: + * they only place the app's virtual world, never the physical viewpoints. + */ +struct u_lift_rig +{ + uint32_t type; //!< U_LIFT_RIG_* + float ipd_factor; //!< both rigs + float parallax_factor; //!< both rigs + float perspective_factor; //!< display rig + float inv_convergence_distance; //!< camera rig, 1 / world units; 0 = infinity + float half_tan_vfov; //!< camera rig, tan(verticalFov / 2) + float m2v; //!< camera rig, metres -> world units +}; + +/*! + * Clamp every field into the range xrLocateViews clamps it to (oxr_session.c + * view_rig_update_from_chain): factors [0, 1e4], perspective [0.1, 10], + * convergence [0, 20], half-tangent of a [0.01, 3.13] rad FOV, m2v + * [1e-4, 1e5] with <= 0 -> 1. Non-finite values take the neutral default. + * Returns true if anything was clamped. Unknown types become NONE. + */ +bool +u_lift_rig_sanitize(struct u_lift_rig *rig); + +/*! + * Map @p count rect-relative eyes (display axes, metres, +z toward the + * viewer; 3 floats each) through @p rig into physical rect-relative + * viewpoints (@p out may alias @p in). + * + * @param rig sanitized rig (NONE: @p out = @p in) + * @param nominal_z nominal viewer distance, metres (<= 0 = 0.5) + * @param rect_w lifted rect width, metres + * @param rect_h lifted rect height, metres + * @return false (and @p out untouched) when the rect size is unknown or + * @p count is 0 / above 8 — the caller keeps the unrigged eyes. + */ +bool +u_lift_rig_apply(const struct u_lift_rig *rig, + float nominal_z, + float rect_w, + float rect_h, + const float *in_xyz, + uint32_t count, + float *out_xyz); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/compositor/d3d11_service/comp_d3d11_service.cpp b/src/xrt/compositor/d3d11_service/comp_d3d11_service.cpp index 8eb68b2da7..de4cdf26b1 100644 --- a/src/xrt/compositor/d3d11_service/comp_d3d11_service.cpp +++ b/src/xrt/compositor/d3d11_service/comp_d3d11_service.cpp @@ -24651,7 +24651,9 @@ lift_weave_rect_batch(struct d3d11_service_system *sys, (void)d3d11_lift_submit_srv_locked(lift, owner, b->stream_id, in_srv, in_w, in_h, (uint32_t)x0, (uint32_t)y0, (uint32_t)(x1 - x0), (uint32_t)(y1 - y0), (int64_t)os_monotonic_get_ns(), b->has_params ? &b->params : nullptr, - b->has_view_control ? &b->view_control : nullptr, &frame, &frame_id); + b->viewpoint_floats > 0 ? b->viewpoints : nullptr, b->viewpoint_floats, + b->has_view_control ? &b->view_control : nullptr, + b->rig.type != XRT_LIFT_RIG_NONE ? &b->rig : nullptr, &frame, &frame_id); const float rx = (float)rect.offset.w, ry = (float)rect.offset.h; const float rw = (float)rect.extent.w, rh = (float)rect.extent.h; @@ -24755,10 +24757,12 @@ lift_weave_rects_nview(struct d3d11_service_system *sys, uint64_t frame_id = 0; struct xrt_lift_rect_frame frame; lift_rect_frame(sys, &c->base.base, &r, win_w, win_h, &frame); - (void)d3d11_lift_submit_srv_locked(lift, owner, b->stream_id, crop_srv, packed_w, packed_h, x0, y0, w, - h, (int64_t)os_monotonic_get_ns(), - b->has_params ? &b->params : nullptr, - b->has_view_control ? &b->view_control : nullptr, &frame, &frame_id); + (void)d3d11_lift_submit_srv_locked( + lift, owner, b->stream_id, crop_srv, packed_w, packed_h, x0, y0, w, h, + (int64_t)os_monotonic_get_ns(), b->has_params ? &b->params : nullptr, + b->viewpoint_floats > 0 ? b->viewpoints : nullptr, b->viewpoint_floats, + b->has_view_control ? &b->view_control : nullptr, + b->rig.type != XRT_LIFT_RIG_NONE ? &b->rig : nullptr, &frame, &frame_id); struct d3d11_lift_pin pin = {}; if (!d3d11_lift_pin_latest(lift, owner, b->stream_id, &pin) || pin.view_count == 0) { @@ -32020,6 +32024,7 @@ comp_d3d11_service_lift_submit(struct xrt_system_compositor *xsysc, const float *viewpoints, uint32_t viewpoint_floats, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, uint64_t *out_frame_id) { struct d3d11_lift *l = svc_lift(xsysc); @@ -32033,7 +32038,7 @@ comp_d3d11_service_lift_submit(struct xrt_system_compositor *xsysc, struct xrt_lift_rect_frame frame; lift_rect_frame(d3d11_service_system_from_xrt(xsysc), xc, nullptr, 0, 0, &frame); return d3d11_lift_submit_handle(l, owner, id, (HANDLE)handle, is_dxgi, w, h, source_time, params, viewpoints, - viewpoint_floats, view_control, &frame, out_frame_id); + viewpoint_floats, view_control, rig, &frame, out_frame_id); } extern "C" xrt_result_t @@ -32064,6 +32069,7 @@ comp_d3d11_service_lift_acquire(struct xrt_system_compositor *xsysc, memcpy(out->viewpoints, r.viewpoints, sizeof(out->viewpoints)); memcpy(out->rect_center, r.rect_center, sizeof(out->rect_center)); memcpy(out->rect_size, r.rect_size, sizeof(out->rect_size)); + out->depth = r.depth; return xret; } @@ -32100,6 +32106,24 @@ comp_d3d11_service_lift_export_fence(struct xrt_system_compositor *xsysc, return true; } +extern "C" bool +comp_d3d11_service_lift_export_depth(struct xrt_system_compositor *xsysc, + uint64_t owner, + uint64_t id, + xrt_graphics_buffer_handle_t *out_handle, + uint32_t *out_width, + uint32_t *out_height, + uint32_t *out_format) +{ + struct d3d11_lift *l = svc_lift(xsysc); + HANDLE h = nullptr; + if (l == nullptr || !d3d11_lift_export_depth(l, owner, id, &h, out_width, out_height, out_format)) { + return false; + } + *out_handle = (xrt_graphics_buffer_handle_t)h; + return true; +} + extern "C" xrt_result_t comp_d3d11_service_lift_acquire_blob(struct xrt_system_compositor *xsysc, uint64_t owner, diff --git a/src/xrt/compositor/d3d11_service/comp_d3d11_service.h b/src/xrt/compositor/d3d11_service/comp_d3d11_service.h index 6767a164d0..fddd5ff0b1 100644 --- a/src/xrt/compositor/d3d11_service/comp_d3d11_service.h +++ b/src/xrt/compositor/d3d11_service/comp_d3d11_service.h @@ -1273,7 +1273,7 @@ comp_d3d11_service_lift_release_owner(struct xrt_system_compositor *xsysc, uint6 * @p xc is the submitting session's compositor (NULL on a headless * connection): its window is the frame the viewpoints are rebased to * (ADR-048); without one, the panel centre. @p view_control NULL = the - * stream's last policy. + * stream's last policy. @p rig (XR_DXR_lift v3) rides with @p params. */ xrt_result_t comp_d3d11_service_lift_submit(struct xrt_system_compositor *xsysc, @@ -1289,6 +1289,7 @@ comp_d3d11_service_lift_submit(struct xrt_system_compositor *xsysc, const float *viewpoints, uint32_t viewpoint_floats, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, uint64_t *out_frame_id); //! xrAcquireLiftResultDXR. @p out_ready false = nothing newer (NOT READY). @@ -1314,6 +1315,16 @@ comp_d3d11_service_lift_export_fence(struct xrt_system_compositor *xsysc, uint64_t id, xrt_graphics_sync_handle_t *out_handle); +//! XR_DXR_lift v3: the stream's auxiliary depth export texture (service-owned). +bool +comp_d3d11_service_lift_export_depth(struct xrt_system_compositor *xsysc, + uint64_t owner, + uint64_t id, + xrt_graphics_buffer_handle_t *out_handle, + uint32_t *out_width, + uint32_t *out_height, + uint32_t *out_format); + /*! * xrAcquireLiftBlobDXR (two-call latch). On delivery @p out_bytes is a malloc'd * copy the caller frees. diff --git a/src/xrt/compositor/d3d11_service/d3d11_lift.cpp b/src/xrt/compositor/d3d11_service/d3d11_lift.cpp index 356d161260..56db5551da 100644 --- a/src/xrt/compositor/d3d11_service/d3d11_lift.cpp +++ b/src/xrt/compositor/d3d11_service/d3d11_lift.cpp @@ -14,7 +14,9 @@ #include "xrt/xrt_display_processor_d3d11.h" +#include "util/u_lift_depth.h" #include "util/u_lift_mailbox.h" +#include "util/u_lift_rig.h" #include "util/u_lift_viewpoint.h" #include "util/u_logging.h" #include "os/os_time.h" @@ -246,6 +248,8 @@ struct lift_in_slot //! from, as of this frame's submit. xrt_lift_view_control vc = {}; xrt_lift_rect_frame frame = {}; + //! XR_DXR_lift v3: the stream's app rig as of this frame (type NONE = none). + xrt_lift_rig rig = {}; }; //! One output ring slot: created on the LIFT device, opened on the SERVICE device. @@ -266,6 +270,16 @@ struct lift_out_slot float vps[3 * XRT_DP_LIFT_MAX_EXPLICIT_VIEWPOINTS] = {}; float rect_center[3] = {}; float rect_size[2] = {}; + //! XR_DXR_lift v3 auxiliary depth: a second texture of this slot (lift + //! device, keyed mutex, opened on the service device) + its metadata. + ID3D11Texture2D *d_lift_tex = nullptr; + IDXGIKeyedMutex *d_lift_km = nullptr; + HANDLE d_share = nullptr; + ID3D11Texture2D *d_svc_tex = nullptr; + IDXGIKeyedMutex *d_svc_km = nullptr; + uint32_t d_w = 0, d_h = 0, d_format = 0; + bool has_depth = false; //!< this slot's result carries depth + xrt_lift_depth_info depth = {}; //!< metadata (valid / realloc set at acquire) }; //! One letterbox profile readback (service device, producer thread only). @@ -291,6 +305,8 @@ struct lift_stream //! ADR-048: the sticky viewpoint policy (XrLiftViewControlDXR), set at //! stream create to the defaults; replaced by every submit that carries one. xrt_lift_view_control vc = {}; + //! XR_DXR_lift v3: the app rig, riding with the options (see submit_locked). + xrt_lift_rig rig = {}; uint32_t priority = U_LIFT_PRIORITY_NORMAL; //!< XrLiftPriorityDXR bool dead = false; //!< destroy requested; the lift thread reaps it @@ -307,6 +323,11 @@ struct lift_stream //! The last policy WARNed about (lift thread only): WARN on change, never per frame. xrt_lift_view_control logged_vc = {}; bool logged_vc_valid = false; + //! The last rig WARNed about (lift thread only). + xrt_lift_rig logged_rig = {}; + //! XR_DXR_lift v3: one-shot WARNs about the auxiliary depth (lift thread). + bool depth_logged = false; + bool depth_missing_logged = false; // Caller-input import cache (service device, producer thread only). HANDLE imp_handle = nullptr; @@ -323,6 +344,10 @@ struct lift_stream HANDLE exp_fence_handle = nullptr; uint64_t exp_fence_value = 0; uint32_t exp_w = 0, exp_h = 0, exp_format = 0; + //! XR_DXR_lift v3: the depth export texture (same fence as exp_tex). + ID3D11Texture2D *exp_depth_tex = nullptr; + HANDLE exp_depth_handle = nullptr; + uint32_t exp_depth_w = 0, exp_depth_h = 0, exp_depth_format = 0; // Blob latch (xrAcquireLiftBlobDXR two-call idiom). std::shared_ptr> blob_latched; @@ -510,6 +535,76 @@ out_slot_release(lift_out_slot &s) rel(s.lift_tex); s.w = s.h = s.format = s.view_count = 0; s.blob.reset(); + rel(s.d_svc_km); + rel(s.d_svc_tex); + close_handle(s.d_share); + rel(s.d_lift_km); + rel(s.d_lift_tex); + s.d_w = s.d_h = s.d_format = 0; + s.has_depth = false; +} + +/*! + * XR_DXR_lift v3: make out slot @p s's DEPTH texture @p w x @p h @p format — + * the views' pattern (created on the lift device with a keyed mutex, opened + * on the service device). Lift thread only. + */ +static bool +depth_slot_ensure(d3d11_lift *l, lift_out_slot &s, uint32_t w, uint32_t h, uint32_t format) +{ + if (s.d_lift_tex != nullptr && s.d_w == w && s.d_h == h && s.d_format == format) { + return true; + } + rel(s.d_svc_km); + rel(s.d_svc_tex); + close_handle(s.d_share); + rel(s.d_lift_km); + rel(s.d_lift_tex); + s.d_w = s.d_h = s.d_format = 0; + + D3D11_TEXTURE2D_DESC td = {}; + td.Width = w; + td.Height = h; + td.MipLevels = 1; + td.ArraySize = 1; + td.Format = (DXGI_FORMAT)format; + td.SampleDesc.Count = 1; + td.Usage = D3D11_USAGE_DEFAULT; + td.BindFlags = D3D11_BIND_SHADER_RESOURCE; + td.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX; + HRESULT hr = l->lift_device->CreateTexture2D(&td, nullptr, &s.d_lift_tex); + if (SUCCEEDED(hr)) { + hr = s.d_lift_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), (void **)&s.d_lift_km); + } + IDXGIResource1 *r1 = nullptr; + if (SUCCEEDED(hr)) { + hr = s.d_lift_tex->QueryInterface(__uuidof(IDXGIResource1), (void **)&r1); + } + if (SUCCEEDED(hr)) { + hr = r1->CreateSharedHandle(nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, nullptr, + &s.d_share); + } + rel(r1); + if (SUCCEEDED(hr)) { + hr = l->svc_device1->OpenSharedResource1(s.d_share, __uuidof(ID3D11Texture2D), (void **)&s.d_svc_tex); + } + if (SUCCEEDED(hr)) { + hr = s.d_svc_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), (void **)&s.d_svc_km); + } + if (FAILED(hr)) { + U_LOG_E("[lift] depth slot %ux%u fmt=%u create/share failed: 0x%08lx", w, h, format, (unsigned long)hr); + rel(s.d_svc_km); + rel(s.d_svc_tex); + close_handle(s.d_share); + rel(s.d_lift_km); + rel(s.d_lift_tex); + return false; + } + s.d_w = w; + s.d_h = h; + s.d_format = format; + U_LOG_I("[lift] depth slot %ux%u fmt=%u ready", w, h, format); + return true; } /*! @@ -657,6 +752,8 @@ stream_release_gpu(lift_stream &st) close_handle(st.exp_fence_handle); rel(st.exp_tex); close_handle(st.exp_handle); + rel(st.exp_depth_tex); + close_handle(st.exp_depth_handle); st.blob_latched.reset(); for (auto &r : st.lb_rb) { rel(r.staging); @@ -843,8 +940,31 @@ struct viewpoint_result uint32_t axis_mode = XRT_DP_LIFT_AXIS_X; float max_offset = 0.0f; uint32_t recenter_mode = U_LIFT_RECENTER_OFF; + uint32_t source = XRT_DP_LIFT_VIEWPOINTS_TRACKED; //!< XRT_DP_LIFT_VIEWPOINTS_* }; +//! Midpoint + first-to-last distance of @p n viewpoints (3 floats each). +static void +viewpoints_mid_baseline(const float *vps, uint32_t n, float mid[3], float *baseline) +{ + mid[0] = mid[1] = mid[2] = 0.0f; + *baseline = 0.0f; + if (n == 0) { + return; + } + for (uint32_t i = 0; i < n; i++) { + for (int a = 0; a < 3; a++) { + mid[a] += vps[3 * i + a] / (float)n; + } + } + if (n >= 2) { + const float *f = &vps[0]; + const float *e = &vps[3 * (n - 1)]; + const float dx = e[0] - f[0], dy = e[1] - f[1], dz = e[2] - f[2]; + *baseline = sqrtf(dx * dx + dy * dy + dz * dz); + } +} + /*! * Resolve the viewpoints of one conversion (lift thread, no lock held). * @@ -854,6 +974,10 @@ struct viewpoint_result * conversion, from the panel's predicted eyes (the lift DP has no tracker * session of its own) and then run through the stream's viewpoint policy * (u_lift_viewpoint.h): rig factors, axis mask, recenter filter, clamp. + * With an app rig (XR_DXR_lift v3, ADR-048 Addendum A) the rig replaces the + * policy's ipd / parallax factors: the policy (axis mask, recenter, clamp) + * runs on the viewer first, then the rig maps the eyes with the xrLocateViews + * math (u_lift_rig.h), the rect as the screen. * DEPTH / GAUSSIANS streams get none. */ static viewpoint_result @@ -861,6 +985,7 @@ resolve_viewpoints(d3d11_lift *l, lift_stream &st, uint32_t view_count, const xrt_lift_view_control &in_vc, + const xrt_lift_rig &in_rig, const xrt_lift_rect_frame &frame, float *vps, uint32_t vp_floats) @@ -878,17 +1003,8 @@ resolve_viewpoints(d3d11_lift *l, u_lift_viewpoint_rebase(vps, n, r.centre); r.floats = 3 * n; r.axis_mode = XRT_DP_LIFT_AXIS_XYZ; - for (uint32_t i = 0; i < n; i++) { - for (int a = 0; a < 3; a++) { - r.mid[a] += vps[3 * i + a] / (float)n; - } - } - if (n >= 2) { - const float *f = &vps[0]; - const float *e = &vps[3 * (n - 1)]; - const float dx = e[0] - f[0], dy = e[1] - f[1], dz = e[2] - f[2]; - r.baseline = sqrtf(dx * dx + dy * dy + dz * dz); - } + r.source = XRT_DP_LIFT_VIEWPOINTS_EXPLICIT; + viewpoints_mid_baseline(vps, n, r.mid, &r.baseline); return r; } if (l->eyes_fn == nullptr || st.info.mode == XRT_DP_LIFT_MODE_GAUSSIANS || @@ -918,6 +1034,47 @@ resolve_viewpoints(d3d11_lift *l, (unsigned long long)st.id, vc.ipd_factor, vc.parallax_factor, vc.axis_mode, vc.max_offset_m, recenter_str(vc.recenter_mode), vc.hold_s, vc.tau_s); } + // XR_DXR_lift v3 app rig (sanitized here, on the lift thread, like the policy). + u_lift_rig rig = {in_rig.type, + in_rig.ipd_factor, + in_rig.parallax_factor, + in_rig.perspective_factor, + in_rig.inv_convergence_distance, + in_rig.half_tan_vfov, + in_rig.m2v}; + u_lift_rig_sanitize(&rig); + const xrt_lift_rig eff_rig = {rig.type, + rig.ipd_factor, + rig.parallax_factor, + rig.perspective_factor, + rig.inv_convergence_distance, + rig.half_tan_vfov, + rig.m2v}; + if (memcmp(&eff_rig, &st.logged_rig, sizeof(eff_rig)) != 0) { + st.logged_rig = eff_rig; + if (rig.type == U_LIFT_RIG_DISPLAY) { + U_LOG_W( + "[lift] stream %llu app display rig: ipd=%.2f parallax=%.2f perspective=%.2f " + "(viewpoints = the xrLocateViews eyes, rect as the screen; ADR-048 Addendum A)", + (unsigned long long)st.id, rig.ipd_factor, rig.parallax_factor, rig.perspective_factor); + } else if (rig.type == U_LIFT_RIG_CAMERA) { + U_LOG_W( + "[lift] stream %llu app camera rig: ipd=%.2f parallax=%.2f convergence=%.3f /unit " + "half_tan_vfov=%.3f m2v=%.3f (viewpoints = the equivalent physical viewer; ADR-048 " + "Addendum A)", + (unsigned long long)st.id, rig.ipd_factor, rig.parallax_factor, + rig.inv_convergence_distance, rig.half_tan_vfov, rig.m2v); + } else { + U_LOG_W("[lift] stream %llu app rig cleared: tracked viewpoints + policy", + (unsigned long long)st.id); + } + } + if (rig.type != U_LIFT_RIG_NONE) { + // The rig owns the eye factors (step 1a / 1b): the policy keeps its + // axis mask, recentering and clamp, applied to the viewer. + vc.ipd_factor = 1.0f; + vc.parallax_factor = 1.0f; + } struct xrt_eye_positions eyes = {}; if (!l->eyes_fn(l->eyes_ud, &eyes) || !eyes.valid || eyes.count < 2) { @@ -940,10 +1097,104 @@ resolve_viewpoints(d3d11_lift *l, } u_lift_viewpoint_rebase(vps, n, r.centre); u_lift_viewpoint_apply(&vc, &st.recenter, l->nominal_z, os_monotonic_get_ns(), vps, n, vps, &r.baseline, r.mid); + if (rig.type != U_LIFT_RIG_NONE && u_lift_rig_apply(&rig, l->nominal_z, r.rect_w, r.rect_h, vps, n, vps)) { + r.source = rig.type == U_LIFT_RIG_CAMERA ? XRT_DP_LIFT_VIEWPOINTS_CAMERA_RIG + : XRT_DP_LIFT_VIEWPOINTS_DISPLAY_RIG; + viewpoints_mid_baseline(vps, n, r.mid, &r.baseline); + } r.floats = 3 * n; return r; } +/*! + * XR_DXR_lift v3 (ADR-048 Addendum A): fetch the auxiliary depth of the + * conversion lift_convert just returned for @p st, copy it into out slot + * @p o, and derive what the app is told (xrt_lift_depth_info) — including the + * display transform, from the viewpoints and rect the runtime already owns. + * Lift thread, no lock held, right after the views were copied into @p o. + */ +static void +lift_copy_depth(d3d11_lift *l, lift_stream &st, lift_out_slot &o, const viewpoint_result &vr) +{ + xrt_dp_lift_depth dd; + if (!xrt_display_processor_d3d11_lift_get_depth(l->dp, st.dp_id, l->lift_context, &dd)) { + if (!st.depth_missing_logged) { + st.depth_missing_logged = true; + U_LOG_W("[lift] stream %llu asked for auxiliary depth but the module returned none%s", + (unsigned long long)st.id, + XRT_DP_HAS_SLOT(l->dp, lift_get_depth) && l->dp->lift_get_depth != nullptr + ? "" + : " (the plug-in has no lift_get_depth slot)"); + } + return; + } + if (dd.format != DXGI_FORMAT_R32_FLOAT && dd.format != DXGI_FORMAT_R16_FLOAT) { + if (!st.depth_missing_logged) { + st.depth_missing_logged = true; + U_LOG_W("[lift] stream %llu: module depth format %u is not R32_FLOAT / R16_FLOAT — dropped", + (unsigned long long)st.id, dd.format); + } + return; + } + if (!depth_slot_ensure(l, o, dd.width, dd.height, dd.format)) { + return; + } + HRESULT hr = o.d_lift_km->AcquireSync(0, 100); + if (FAILED(hr) || hr == (HRESULT)WAIT_TIMEOUT) { + return; + } + D3D11_BOX box = {0, 0, 0, dd.width, dd.height, 1}; + l->lift_context->CopySubresourceRegion(o.d_lift_tex, 0, 0, 0, 0, (ID3D11Resource *)dd.resource, 0, &box); + o.d_lift_km->ReleaseSync(0); + l->lift_context->Flush(); + + xrt_lift_depth_info &di = o.depth; + di = xrt_lift_depth_info{}; + di.width = dd.width; + di.height = dd.height; + di.format = dd.format; + di.units = dd.units == XRT_DP_LIFT_DEPTH_METRIC ? XRT_DP_LIFT_DEPTH_METRIC : XRT_DP_LIFT_DEPTH_RELATIVE; + di.encoding = dd.encoding == XRT_DP_LIFT_DEPTH_ENCODING_INVERSE ? XRT_DP_LIFT_DEPTH_ENCODING_INVERSE + : XRT_DP_LIFT_DEPTH_ENCODING_LINEAR; + di.same_inference = dd.same_inference != 0 ? 1u : 0u; + di.value_scale = dd.value_scale != 0.0f && std::isfinite(dd.value_scale) ? dd.value_scale : 1.0f; + di.value_offset = std::isfinite(dd.value_offset) ? dd.value_offset : 0.0f; + di.near_depth = dd.near_depth > 0.0f ? dd.near_depth : 0.0f; + di.far_depth = dd.far_depth > 0.0f ? dd.far_depth : 0.0f; + const bool metric = di.units == XRT_DP_LIFT_DEPTH_METRIC; + // RELATIVE depth: no intrinsics, no transform — never a faked scale. + if (metric && dd.focal_x_px > 0.0f && dd.focal_y_px > 0.0f) { + di.intrinsics_valid = 1; + di.focal[0] = dd.focal_x_px; + di.focal[1] = dd.focal_y_px; + di.principal[0] = dd.principal_x_px; + di.principal[1] = dd.principal_y_px; + } + di.convergence_depth = dd.convergence_depth > 0.0f ? dd.convergence_depth : 0.0f; + // The depth refers to the source camera = the midpoint of the synthesized + // viewpoints (the straight-on viewer when none were known). + if (vr.floats >= 3) { + memcpy(di.viewpoint, vr.mid, sizeof(di.viewpoint)); + } else { + di.viewpoint[0] = di.viewpoint[1] = 0.0f; + di.viewpoint[2] = l->nominal_z; + } + di.transform_valid = metric && u_lift_depth_to_display(di.viewpoint, vr.centre, vr.rect_w, vr.rect_h, + di.convergence_depth, di.depth_to_display) + ? 1u + : 0u; + o.has_depth = true; + if (!st.depth_logged) { + st.depth_logged = true; + U_LOG_W( + "[lift] stream %llu auxiliary depth: %ux%u fmt=%u units=%s encoding=%s same_inference=%u " + "convergence=%.3f intrinsics=%s transform=%s (ADR-048 Addendum A)", + (unsigned long long)st.id, dd.width, dd.height, dd.format, metric ? "metric" : "relative", + di.encoding == XRT_DP_LIFT_DEPTH_ENCODING_INVERSE ? "inverse" : "linear", di.same_inference, + di.convergence_depth, di.intrinsics_valid ? "yes" : "no", di.transform_valid ? "yes" : "no"); + } +} + /*! * Convert the pending frame of @p st. Called with @p lk HELD; drops it around * every GPU / module call and returns with it held. @@ -966,12 +1217,14 @@ lift_convert_one(d3d11_lift *l, lift_stream &st, std::unique_lock &l memcpy(active, in.active, sizeof(active)); const xrt_lift_view_control in_vc = in.vc; const xrt_lift_rect_frame frame = in.frame; + const xrt_lift_rig in_rig = in.rig; const uint32_t w = meta.width, h = meta.height; lk.unlock(); - // ADR-048: rect-relative viewpoints + the stream's viewpoint policy, and - // what the DP is told about them (appended xrt_dp_lift_params fields). - const viewpoint_result vr = resolve_viewpoints(l, st, params.view_count, in_vc, frame, vps, vp_floats); + // ADR-048: rect-relative viewpoints + the stream's viewpoint policy (and, + // v3, the app rig), and what the DP is told about them (appended + // xrt_dp_lift_params fields). + const viewpoint_result vr = resolve_viewpoints(l, st, params.view_count, in_vc, in_rig, frame, vps, vp_floats); vp_floats = vr.floats; params.struct_size = (uint32_t)sizeof(params); params.rect_width_m = vr.rect_w; @@ -980,6 +1233,9 @@ lift_convert_one(d3d11_lift *l, lift_stream &st, std::unique_lock &l params.axis_mode = vr.axis_mode; params.max_offset_m = vr.max_offset; params.viewpoint_frame = XRT_DP_LIFT_VIEWPOINT_FRAME_RECT; + params.viewpoint_source = vr.source; + params.nominal_z_m = l->nominal_z; + const bool want_depth = (st.info.aux_outputs & XRT_DP_LIFT_AUX_DEPTH) != 0; bool ok = true; // 1. The module's own stream, created lazily (module contract: lift thread only). @@ -1091,6 +1347,13 @@ lift_convert_one(d3d11_lift *l, lift_stream &st, std::unique_lock &l o.rect_size[1] = vr.rect_h; } } + // XR_DXR_lift v3: the auxiliary depth of THIS conversion, copied into + // the same ring slot (the DP's texture is valid only until its next + // lift_convert on this stream). + o.has_depth = false; + if (ok && want_depth) { + lift_copy_depth(l, st, o, vr); + } } else if (ok && is_blob) { lift_out_slot &o = st.out[out_slot]; o.blob = blob_copy; @@ -1500,6 +1763,14 @@ d3d11_lift_stream_create(struct d3d11_lift *l, if (!(st->info.input_scale > 0.0f) || st->info.input_scale > 1.0f) { st->info.input_scale = 1.0f; } + // XR_DXR_lift v3: auxiliary depth rides SBS / NVIEW streams only (a + // caller's shorter struct carries none). + if (info->struct_size < offsetof(xrt_dp_lift_stream_info, aux_depth_format) + sizeof(uint32_t) || + (m != XRT_DP_LIFT_MODE_SBS && m != XRT_DP_LIFT_MODE_NVIEW)) { + st->info.aux_outputs = 0; + st->info.aux_depth_format = 0; + } + st->info.aux_outputs &= XRT_DP_LIFT_AUX_DEPTH; u_lift_mailbox_init(&st->mb); st->last_params.struct_size = (uint32_t)sizeof(st->last_params); st->last_params.convergence = -1.0f; @@ -1513,8 +1784,9 @@ d3d11_lift_stream_create(struct d3d11_lift *l, d.recenter_mode, d.hold_s, d.tau_s}; } *out_id = st->id; - U_LOG_W("[lift] stream %llu created (mode=%u hint=%u scale=%.2f, owner=%llu)", (unsigned long long)st->id, m, - info->content_hint, st->info.input_scale, (unsigned long long)owner); + U_LOG_W("[lift] stream %llu created (mode=%u hint=%u scale=%.2f aux=%u, owner=%llu)", + (unsigned long long)st->id, m, info->content_hint, st->info.input_scale, st->info.aux_outputs, + (unsigned long long)owner); l->streams[st->id] = std::move(st); l->live_streams++; l->cv.notify_all(); @@ -1816,6 +2088,7 @@ submit_locked(d3d11_lift *l, const float *viewpoints, uint32_t viewpoint_floats, const xrt_lift_view_control *view_control, + const xrt_lift_rig *rig, const xrt_lift_rect_frame *frame, uint32_t max_input_edge, bool letterbox, @@ -1887,9 +2160,15 @@ submit_locked(d3d11_lift *l, if (view_control != nullptr) { st->vc = *view_control; // sticky (sanitized on the lift thread) } + if (params != nullptr || rig != nullptr) { + // XR_DXR_lift v3: the rig rides with the options, like explicit + // viewpoints — options without a rig clear it. + st->rig = rig != nullptr ? *rig : xrt_lift_rig{}; + } lift_in_slot &in = st->in[slot]; in.params = st->last_params; in.vc = st->vc; + in.rig = st->rig; in.frame = frame != nullptr ? *frame : xrt_lift_rect_frame{}; if (capped && in.params.focal_px > 0.0f) { in.params.focal_px *= (float)dw / (float)cw; // focal is in INPUT pixels (a crop keeps the scale) @@ -1942,16 +2221,20 @@ d3d11_lift_submit_srv_locked(struct d3d11_lift *l, uint32_t h, int64_t source_time, const struct xrt_dp_lift_params *params, + const float *viewpoints, + uint32_t viewpoint_floats, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, const struct xrt_lift_rect_frame *frame, uint64_t *out_frame_id) { if (l == nullptr || !l->snap_ok || src == nullptr || out_frame_id == nullptr) { return XRT_ERROR_FEATURE_NOT_SUPPORTED; } - // Weave-rect snapshot: the service's size cap applies. - return submit_locked(l, owner, id, src, src_tw, src_th, x, y, w, h, source_time, params, nullptr, 0, - view_control, frame, l->max_input_edge, l->letterbox, out_frame_id); + // Weave-rect snapshot: the service's size cap applies. XR_DXR_lift v3: a + // lifted rect may carry EXPLICIT viewpoints and an app rig. + return submit_locked(l, owner, id, src, src_tw, src_th, x, y, w, h, source_time, params, viewpoints, + viewpoint_floats, view_control, rig, frame, l->max_input_edge, l->letterbox, out_frame_id); } xrt_result_t @@ -1967,6 +2250,7 @@ d3d11_lift_submit_handle(struct d3d11_lift *l, const float *viewpoints, uint32_t viewpoint_floats, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, const struct xrt_lift_rect_frame *frame, uint64_t *out_frame_id) { @@ -2062,7 +2346,7 @@ d3d11_lift_submit_handle(struct d3d11_lift *l, // An app's explicit frame: its size and content are the app's choice — // never capped, never letterbox-cropped. xret = submit_locked(l, owner, id, st->imp_srv, st->imp_w, st->imp_h, 0, 0, w, h, source_time, params, - viewpoints, viewpoint_floats, view_control, frame, /*max_input_edge*/ 0, + viewpoints, viewpoint_floats, view_control, rig, frame, /*max_input_edge*/ 0, /*letterbox*/ false, out_frame_id); } if (acquired) { @@ -2226,19 +2510,80 @@ d3d11_lift_acquire_result( o->h, o->format); } + // XR_DXR_lift v3: the auxiliary depth export texture, the same pattern + // (sized to the result's depth, re-created on change). A failure here only + // costs this result its depth, never the views. + bool depth_realloc = false; + bool depth_ok = false; + if (o->has_depth && o->d_svc_tex != nullptr) { + depth_ok = true; + if (st->exp_depth_tex == nullptr || st->exp_depth_w != o->d_w || st->exp_depth_h != o->d_h || + st->exp_depth_format != o->d_format) { + rel(st->exp_depth_tex); + close_handle(st->exp_depth_handle); + D3D11_TEXTURE2D_DESC td = {}; + td.Width = o->d_w; + td.Height = o->d_h; + td.MipLevels = 1; + td.ArraySize = 1; + td.Format = (DXGI_FORMAT)o->d_format; + td.SampleDesc.Count = 1; + td.Usage = D3D11_USAGE_DEFAULT; + td.BindFlags = D3D11_BIND_SHADER_RESOURCE; + td.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED; + HRESULT dhr = l->svc_device->CreateTexture2D(&td, nullptr, &st->exp_depth_tex); + IDXGIResource1 *r1 = nullptr; + if (SUCCEEDED(dhr)) { + dhr = st->exp_depth_tex->QueryInterface(__uuidof(IDXGIResource1), (void **)&r1); + } + if (SUCCEEDED(dhr)) { + dhr = r1->CreateSharedHandle(nullptr, + DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, + nullptr, &st->exp_depth_handle); + } + rel(r1); + if (FAILED(dhr)) { + U_LOG_E("[lift] depth export texture %ux%u fmt=%u create failed: 0x%08lx", o->d_w, + o->d_h, o->d_format, (unsigned long)dhr); + rel(st->exp_depth_tex); + close_handle(st->exp_depth_handle); + st->exp_depth_w = st->exp_depth_h = st->exp_depth_format = 0; + depth_ok = false; + } else { + st->exp_depth_w = o->d_w; + st->exp_depth_h = o->d_h; + st->exp_depth_format = o->d_format; + depth_realloc = true; + U_LOG_W("[lift] stream %llu depth export texture %ux%u fmt=%u ready", + (unsigned long long)id, o->d_w, o->d_h, o->d_format); + } + } + } + HRESULT hr = o->svc_km->AcquireSync(0, 4); if (FAILED(hr) || hr == (HRESULT)WAIT_TIMEOUT) { unpin(); return XRT_ERROR_WEAVE_REFUSED; } + if (depth_ok) { + HRESULT dhr = o->d_svc_km->AcquireSync(0, 4); + depth_ok = SUCCEEDED(dhr) && dhr != (HRESULT)WAIT_TIMEOUT; + } { std::lock_guard ctx_lock(*l->svc_ctx_mutex); l->svc_context->CopyResource(st->exp_tex, o->svc_tex); + if (depth_ok) { + l->svc_context->CopyResource(st->exp_depth_tex, o->d_svc_tex); + } + // One fence covers the views and the depth (XrLiftDepthResultDXR). st->exp_fence_value++; l->svc_context4->Signal(st->exp_fence, st->exp_fence_value); l->svc_context->Flush(); } o->svc_km->ReleaseSync(0); + if (depth_ok) { + o->d_svc_km->ReleaseSync(0); + } out->frame_id = meta.frame_id; out->source_time = meta.source_time; @@ -2253,6 +2598,11 @@ d3d11_lift_acquire_result( memcpy(out->viewpoints, o->vps, sizeof(out->viewpoints)); memcpy(out->rect_center, o->rect_center, sizeof(out->rect_center)); memcpy(out->rect_size, o->rect_size, sizeof(out->rect_size)); + if (depth_ok) { + out->depth = o->depth; + out->depth.valid = 1; + out->depth.realloc = depth_realloc ? 1u : 0u; + } *out_ready = true; unpin(); return XRT_SUCCESS; @@ -2297,6 +2647,30 @@ d3d11_lift_export_fence(struct d3d11_lift *l, uint64_t owner, uint64_t id, HANDL return true; } +bool +d3d11_lift_export_depth(struct d3d11_lift *l, + uint64_t owner, + uint64_t id, + HANDLE *out_handle, + uint32_t *out_w, + uint32_t *out_h, + uint32_t *out_format) +{ + if (l == nullptr) { + return false; + } + std::lock_guard g(l->mtx); + lift_stream *st = find_live(l, owner, id); + if (st == nullptr || st->exp_depth_handle == nullptr) { + return false; + } + *out_handle = st->exp_depth_handle; + *out_w = st->exp_depth_w; + *out_h = st->exp_depth_h; + *out_format = st->exp_depth_format; + return true; +} + xrt_result_t d3d11_lift_acquire_blob(struct d3d11_lift *l, uint64_t owner, diff --git a/src/xrt/compositor/d3d11_service/d3d11_lift.h b/src/xrt/compositor/d3d11_service/d3d11_lift.h index 5eb13497cd..bc93c22356 100644 --- a/src/xrt/compositor/d3d11_service/d3d11_lift.h +++ b/src/xrt/compositor/d3d11_service/d3d11_lift.h @@ -72,6 +72,8 @@ struct d3d11_lift_result_info float viewpoints[3 * XRT_LIFT_MAX_VIEWS]; float rect_center[3]; float rect_size[2]; + //! XR_DXR_lift v3: the auxiliary depth (valid = 0 when none). + struct xrt_lift_depth_info depth; }; //! One acquired blob result (xrAcquireLiftBlobDXR). @@ -168,7 +170,9 @@ d3d11_lift_stream_mode(struct d3d11_lift *lift, uint64_t owner, uint64_t id); * @p is_dxgi. @p params NULL = the stream's last parameters. * @p view_control NULL = the stream's last policy (initially the ADR-048 * defaults). @p frame = the submitting window (centre + size) the viewpoints - * are rebased to; NULL / invalid = the panel centre. + * are rebased to; NULL / invalid = the panel centre. @p rig (XR_DXR_lift v3) + * rides with the options: replaced (NULL = cleared) whenever @p params or + * @p rig is non-NULL, kept otherwise. */ xrt_result_t d3d11_lift_submit_handle(struct d3d11_lift *lift, @@ -183,6 +187,7 @@ d3d11_lift_submit_handle(struct d3d11_lift *lift, const float *viewpoints, uint32_t viewpoint_floats, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, const struct xrt_lift_rect_frame *frame, uint64_t *out_frame_id); @@ -194,7 +199,9 @@ d3d11_lift_submit_handle(struct d3d11_lift *lift, * (default 1920, 0 = off; u_lift_cap_dims) — the result is stretched back into * the rect, so consumers must sample it, never assume result size == rect size. * @p view_control / @p frame as for d3d11_lift_submit_handle (@p frame = the - * lifted rect's physical centre + size). + * lifted rect's physical centre + size). @p viewpoints / @p rig: a lifted + * rect's EXPLICIT viewpoints and app rig (XR_DXR_lift v3), same rules as + * d3d11_lift_submit_handle. */ xrt_result_t d3d11_lift_submit_srv_locked(struct d3d11_lift *lift, @@ -209,7 +216,10 @@ d3d11_lift_submit_srv_locked(struct d3d11_lift *lift, uint32_t h, int64_t source_time, const struct xrt_dp_lift_params *params, + const float *viewpoints, + uint32_t viewpoint_floats, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, const struct xrt_lift_rect_frame *frame, uint64_t *out_frame_id); @@ -242,6 +252,17 @@ d3d11_lift_export_output(struct d3d11_lift *lift, bool d3d11_lift_export_fence(struct d3d11_lift *lift, uint64_t owner, uint64_t id, HANDLE *out_handle); +//! XR_DXR_lift v3: the auxiliary depth export texture's NT handle +//! (service-owned; the IPC layer duplicates it). Signalled by the same fence. +bool +d3d11_lift_export_depth(struct d3d11_lift *lift, + uint64_t owner, + uint64_t id, + HANDLE *out_handle, + uint32_t *out_w, + uint32_t *out_h, + uint32_t *out_format); + /*! * xrAcquireLiftBlobDXR with the two-call latch (see XrLiftBlobDXR). On * delivery (@p capacity >= byte_count) @p out_bytes receives a malloc'd copy diff --git a/src/xrt/drivers/sim_display/sim_display_lift_d3d11.cpp b/src/xrt/drivers/sim_display/sim_display_lift_d3d11.cpp index 044f839312..e8fce0adf5 100644 --- a/src/xrt/drivers/sim_display/sim_display_lift_d3d11.cpp +++ b/src/xrt/drivers/sim_display/sim_display_lift_d3d11.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -74,6 +75,14 @@ struct fake_stream uint32_t out_w, out_h; DXGI_FORMAT out_fmt; std::vector blob; + // XR_DXR_lift v3 auxiliary depth (SBS / NVIEW streams that asked for it). + bool aux_depth; + ID3D11Texture2D *depth; + ID3D11RenderTargetView *depth_rtv; + uint32_t depth_w, depth_h; + bool depth_ready; //!< the last convert rendered a depth map + float convergence; //!< xrt_dp_lift_params::convergence of the last convert + uint64_t frames; }; struct sim_fake_lift @@ -117,6 +126,8 @@ stream_release(fake_stream &s) { safe_release(s.rtv); safe_release(s.out); + safe_release(s.depth_rtv); + safe_release(s.depth); s.blob.clear(); s.blob.shrink_to_fit(); s.used = false; @@ -227,6 +238,9 @@ sim_fake_lift_get_caps(struct sim_fake_lift *fl, struct xrt_dp_lift_caps *out) out->max_views = SIM_FAKE_LIFT_MAX_VIEWS; out->depth_semantics = XRT_DP_LIFT_DEPTH_RELATIVE; out->state = XRT_DP_LIFT_STATE_READY; + // The fake's auxiliary depth is a synthetic METRIC ramp (1 m .. 2 m). + out->aux_outputs = XRT_DP_LIFT_AUX_DEPTH; + out->aux_depth_semantics = XRT_DP_LIFT_DEPTH_METRIC; out->typical_latency_ns = (uint64_t)debug_get_num_option_sim_display_fake_lift_latency_ms() * 1000000ull; snprintf(out->backend, sizeof(out->backend), "sim_display-fake"); return true; @@ -249,6 +263,10 @@ sim_fake_lift_stream_create(struct sim_fake_lift *fl, const struct xrt_dp_lift_s s.used = true; s.id = ++fl->next_id; s.mode = m; + s.aux_depth = (m == XRT_DP_LIFT_MODE_SBS || m == XRT_DP_LIFT_MODE_NVIEW) && + info->struct_size >= offsetof(struct xrt_dp_lift_stream_info, aux_depth_format) + + sizeof(uint32_t) && + (info->aux_outputs & XRT_DP_LIFT_AUX_DEPTH) != 0; *out_id = s.id; return true; } @@ -297,6 +315,69 @@ ensure_out(struct sim_fake_lift *fl, fake_stream &s, uint32_t w, uint32_t h, DXG return true; } +static bool +fake_ensure_depth(struct sim_fake_lift *fl, fake_stream &s, uint32_t w, uint32_t h) +{ + if (s.depth != nullptr && s.depth_w == w && s.depth_h == h) { + return true; + } + safe_release(s.depth_rtv); + safe_release(s.depth); + D3D11_TEXTURE2D_DESC td = {}; + td.Width = w; + td.Height = h; + td.MipLevels = 1; + td.ArraySize = 1; + td.Format = DXGI_FORMAT_R32_FLOAT; + td.SampleDesc.Count = 1; + td.Usage = D3D11_USAGE_DEFAULT; + td.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE; + if (FAILED(fl->device->CreateTexture2D(&td, nullptr, &s.depth)) || + FAILED(fl->device->CreateRenderTargetView(s.depth, nullptr, &s.depth_rtv))) { + safe_release(s.depth_rtv); + safe_release(s.depth); + return false; + } + s.depth_w = w; + s.depth_h = h; + return true; +} + +extern "C" bool +sim_fake_lift_get_depth(struct sim_fake_lift *fl, uint64_t id, void *d3d11_context, struct xrt_dp_lift_depth *out) +{ + (void)d3d11_context; + if (fl == nullptr || out == nullptr || out->struct_size < sizeof(struct xrt_dp_lift_depth)) { + return false; + } + fake_stream *s = find_stream(fl, id); + if (s == nullptr || !s->aux_depth || !s->depth_ready || s->depth == nullptr) { + return false; + } + out->format = (uint32_t)DXGI_FORMAT_R32_FLOAT; + out->resource = s->depth; + out->width = s->depth_w; + out->height = s->depth_h; + out->units = XRT_DP_LIFT_DEPTH_METRIC; + out->encoding = XRT_DP_LIFT_DEPTH_ENCODING_LINEAR; + out->value_scale = 1.0f; // sample = v in [0, 1] -> 1 m (top) .. 2 m (bottom) + out->value_offset = 1.0f; + out->source_width = s->depth_w; + out->source_height = s->depth_h; + const float f = 0.78f * (float)(s->depth_w > s->depth_h ? s->depth_w : s->depth_h); + out->focal_x_px = f; + out->focal_y_px = f; + out->principal_x_px = 0.5f * (float)s->depth_w; + out->principal_y_px = 0.5f * (float)s->depth_h; + out->near_depth = 1.0f; + out->far_depth = 2.0f; + // Relative convergence c in [0, 1] over the 1..2 m range; AUTO = mid. + out->convergence_depth = s->convergence < 0.0f ? 1.5f : 1.0f + (s->convergence > 1.0f ? 1.0f : s->convergence); + out->same_inference = 1; + out->vendor_frame_id = s->frames; + return true; +} + extern "C" bool sim_fake_lift_convert(struct sim_fake_lift *fl, uint64_t id, @@ -371,6 +452,25 @@ sim_fake_lift_convert(struct sim_fake_lift *fl, ctx->Draw(4, 0); ID3D11ShaderResourceView *null_srv = nullptr; ctx->PSSetShaderResources(0, 1, &null_srv); + + // XR_DXR_lift v3: the auxiliary depth of the SAME (fake) inference — the + // ramp the DEPTH mode draws, at input size, R32F; decoded as 1 + v metres. + s->depth_ready = false; + if (s->aux_depth && !depth && fake_ensure_depth(fl, *s, w, h)) { + D3D11_VIEWPORT dvp = {}; + dvp.Width = (float)w; + dvp.Height = (float)h; + dvp.MaxDepth = 1.0f; + ctx->RSSetViewports(1, &dvp); + D3D11_RECT dsc = {0, 0, (LONG)w, (LONG)h}; + ctx->RSSetScissorRects(1, &dsc); + ctx->OMSetRenderTargets(1, &s->depth_rtv, nullptr); + ctx->PSSetShader(fl->ps_depth, nullptr, 0); + ctx->Draw(4, 0); + s->depth_ready = true; + s->convergence = p != nullptr ? p->convergence : -1.0f; + s->frames++; + } ID3D11RenderTargetView *null_rtv = nullptr; ctx->OMSetRenderTargets(1, &null_rtv, nullptr); safe_release(srv); diff --git a/src/xrt/drivers/sim_display/sim_display_lift_d3d11.h b/src/xrt/drivers/sim_display/sim_display_lift_d3d11.h index a38108463d..346e71bd63 100644 --- a/src/xrt/drivers/sim_display/sim_display_lift_d3d11.h +++ b/src/xrt/drivers/sim_display/sim_display_lift_d3d11.h @@ -68,6 +68,10 @@ sim_fake_lift_convert(struct sim_fake_lift *fl, uint32_t *out_h, uint32_t *out_format); +//! XR_DXR_lift v3: the synthetic metric depth of the last convert (aux depth). +bool +sim_fake_lift_get_depth(struct sim_fake_lift *fl, uint64_t id, void *d3d11_context, struct xrt_dp_lift_depth *out); + bool sim_fake_lift_convert_blob(struct sim_fake_lift *fl, uint64_t id, diff --git a/src/xrt/drivers/sim_display/sim_display_processor_d3d11.cpp b/src/xrt/drivers/sim_display/sim_display_processor_d3d11.cpp index f6999130c0..b23e8d197b 100644 --- a/src/xrt/drivers/sim_display/sim_display_processor_d3d11.cpp +++ b/src/xrt/drivers/sim_display/sim_display_processor_d3d11.cpp @@ -957,6 +957,15 @@ sim_dp_d3d11_lift_convert_blob(struct xrt_display_processor_d3d11 *xdp, out_format, out_bytes, out_size); } +static bool +sim_dp_d3d11_lift_get_depth(struct xrt_display_processor_d3d11 *xdp, + uint64_t id, + void *d3d11_context, + struct xrt_dp_lift_depth *out) +{ + return sim_fake_lift_get_depth(sim_dp_d3d11(xdp)->fake_lift, id, d3d11_context, out); +} + static xrt_result_t sim_display_processor_d3d11_create_bound(enum sim_display_output_mode mode, void *d3d11_device, @@ -1094,6 +1103,7 @@ sim_display_processor_d3d11_create_bound(enum sim_display_output_mode mode, sdp->base.lift_stream_destroy = sim_dp_d3d11_lift_stream_destroy; sdp->base.lift_convert = sim_dp_d3d11_lift_convert; sdp->base.lift_convert_blob = sim_dp_d3d11_lift_convert_blob; + sdp->base.lift_get_depth = sim_dp_d3d11_lift_get_depth; } } diff --git a/src/xrt/include/xrt/xrt_lift.h b/src/xrt/include/xrt/xrt_lift.h index 7a8e221ca1..11eaedc537 100644 --- a/src/xrt/include/xrt/xrt_lift.h +++ b/src/xrt/include/xrt/xrt_lift.h @@ -43,6 +43,55 @@ struct xrt_lift_view_control float tau_s; //!< <= 0 = default }; +//! xrt_lift_rig::type (ADR-048 Addendum A). +#define XRT_LIFT_RIG_NONE 0u +#define XRT_LIFT_RIG_DISPLAY 1u +#define XRT_LIFT_RIG_CAMERA 2u + +/*! + * An app rig chained on a lift (XrDisplayRigDXR / XrCameraRigDXR, + * XR_DXR_lift v3), boundary-converted like xrLocateViews' rig: verticalFov -> + * half-tangent, convergenceDiopters -> inverse convergence distance. The + * service sanitizes it (u_lift_rig_sanitize). Layout mirrors u_lift_rig. + */ +struct xrt_lift_rig +{ + uint32_t type; //!< XRT_LIFT_RIG_* + float ipd_factor; + float parallax_factor; + float perspective_factor; //!< display rig + float inv_convergence_distance; //!< camera rig + float half_tan_vfov; //!< camera rig + float m2v; //!< camera rig +}; + +/*! + * The auxiliary depth of one acquired result (XrLiftDepthResultDXR, + * XR_DXR_lift v3). Plain values; the texture travels separately. + */ +struct xrt_lift_depth_info +{ + uint32_t valid; //!< non-zero = this result carries depth + uint32_t realloc; //!< the depth export texture changed: re-export it + uint32_t width; + uint32_t height; + uint32_t format; //!< DXGI_FORMAT + uint32_t units; //!< XRT_DP_LIFT_DEPTH_RELATIVE / _METRIC + uint32_t encoding; //!< XRT_DP_LIFT_DEPTH_ENCODING_* + uint32_t same_inference; + float value_scale; + float value_offset; + float near_depth; + float far_depth; + uint32_t intrinsics_valid; + float focal[2]; //!< depth-map pixels + float principal[2]; //!< depth-map pixels + float convergence_depth; + float viewpoint[3]; //!< rect-relative metres + uint32_t transform_valid; + float depth_to_display[16]; //!< column-major +}; + /*! * The physical region a lift frame came from (ADR-048): its centre in display * space (metres, panel centre = 0, +y up) and its size. TRACKED and EXPLICIT @@ -74,6 +123,8 @@ struct xrt_lift_result float viewpoints[3 * XRT_LIFT_MAX_VIEWS]; float rect_center[3]; //!< display space, metres float rect_size[2]; //!< metres + //! XR_DXR_lift v3: the auxiliary depth of this result. + struct xrt_lift_depth_info depth; }; //! One acquired blob result. @@ -94,6 +145,14 @@ struct xrt_lift_weave_rect struct xrt_dp_lift_params params; bool has_view_control; //!< XrLiftViewControlDXR chained (ADR-048) struct xrt_lift_view_control view_control; + //! XR_DXR_lift v3: an app rig rode on this rect (type NONE = none). Like + //! the explicit viewpoints it rides with the options: applied whenever + //! has_params or a rig is present. + struct xrt_lift_rig rig; + //! XR_DXR_lift v3: EXPLICIT viewpoints on a lifted rect, display space + //! metres (3 floats each); 0 = tracked. + uint32_t viewpoint_floats; + float viewpoints[3 * XRT_LIFT_MAX_VIEWS]; }; //! One stream's counters + effective rate (XrLiftStreamStatsDXR). diff --git a/src/xrt/ipc/client/ipc_client_compositor.c b/src/xrt/ipc/client/ipc_client_compositor.c index 5148320aa9..87de843a68 100644 --- a/src/xrt/ipc/client/ipc_client_compositor.c +++ b/src/xrt/ipc/client/ipc_client_compositor.c @@ -3208,9 +3208,12 @@ comp_ipc_client_compositor_lift_stream_create(struct xrt_compositor *xc, uint32_t mode, uint32_t content_hint, float input_scale, + uint32_t aux_outputs, + uint32_t aux_depth_format, uint64_t *out_stream_id) { - return ipc_client_lift_stream_create(lift_conn(xc), mode, content_hint, input_scale, out_stream_id); + return ipc_client_lift_stream_create(lift_conn(xc), mode, content_hint, input_scale, aux_outputs, + aux_depth_format, out_stream_id); } xrt_result_t @@ -3231,10 +3234,11 @@ comp_ipc_client_compositor_lift_submit(struct xrt_compositor *xc, const float *viewpoints, uint32_t viewpoint_count, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, uint64_t *out_frame_id) { return ipc_client_lift_submit(lift_conn(xc), stream_id, handle, is_dxgi, width, height, source_time, params, - viewpoints, viewpoint_count, view_control, out_frame_id); + viewpoints, viewpoint_count, view_control, rig, out_frame_id); } xrt_result_t @@ -3255,6 +3259,15 @@ comp_ipc_client_compositor_lift_get_output(struct xrt_compositor *xc, return ipc_client_lift_get_output(lift_conn(xc), stream_id, out_have, NULL, NULL, NULL, out_handle); } +xrt_result_t +comp_ipc_client_compositor_lift_get_depth_output(struct xrt_compositor *xc, + uint64_t stream_id, + bool *out_have, + xrt_graphics_buffer_handle_t *out_handle) +{ + return ipc_client_lift_get_depth_output(lift_conn(xc), stream_id, out_have, NULL, NULL, NULL, out_handle); +} + xrt_result_t comp_ipc_client_compositor_lift_get_fence(struct xrt_compositor *xc, uint64_t stream_id, diff --git a/src/xrt/ipc/client/ipc_client_lift.c b/src/xrt/ipc/client/ipc_client_lift.c index 48ac8e2629..c3d7853b0e 100644 --- a/src/xrt/ipc/client/ipc_client_lift.c +++ b/src/xrt/ipc/client/ipc_client_lift.c @@ -65,13 +65,19 @@ ipc_client_lift_get_properties(struct ipc_connection *ipc_c, struct xrt_dp_lift_ } xrt_result_t -ipc_client_lift_stream_create( - struct ipc_connection *ipc_c, uint32_t mode, uint32_t content_hint, float input_scale, uint64_t *out_stream_id) +ipc_client_lift_stream_create(struct ipc_connection *ipc_c, + uint32_t mode, + uint32_t content_hint, + float input_scale, + uint32_t aux_outputs, + uint32_t aux_depth_format, + uint64_t *out_stream_id) { if (ipc_c == NULL || out_stream_id == NULL) { return XRT_ERROR_IPC_FAILURE; } - return ipc_call_lift_stream_create(ipc_c, mode, content_hint, input_scale, out_stream_id); + return ipc_call_lift_stream_create(ipc_c, mode, content_hint, input_scale, aux_outputs, aux_depth_format, + out_stream_id); } xrt_result_t @@ -95,6 +101,7 @@ ipc_client_lift_submit(struct ipc_connection *ipc_c, const float *viewpoints, uint32_t viewpoint_count, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, uint64_t *out_frame_id) { if (ipc_c == NULL || out_frame_id == NULL) { @@ -114,6 +121,9 @@ ipc_client_lift_submit(struct ipc_connection *ipc_c, args.params.has_view_control = 1; args.params.view_control = *view_control; } + if (rig != NULL) { + args.params.rig = *rig; // XR_DXR_lift v3; type NONE = none + } } xrt_graphics_buffer_handle_t handles[1] = {handle}; #if defined(XRT_GRAPHICS_BUFFER_HANDLE_IS_WIN32_HANDLE) @@ -159,9 +169,38 @@ ipc_client_lift_acquire(struct ipc_connection *ipc_c, memcpy(out_result->viewpoints, r.viewpoints, sizeof(out_result->viewpoints)); memcpy(out_result->rect_center, r.rect_center, sizeof(out_result->rect_center)); memcpy(out_result->rect_size, r.rect_size, sizeof(out_result->rect_size)); + out_result->depth = r.depth; return XRT_SUCCESS; } +xrt_result_t +ipc_client_lift_get_depth_output(struct ipc_connection *ipc_c, + uint64_t stream_id, + bool *out_have, + uint32_t *out_width, + uint32_t *out_height, + uint32_t *out_format, + xrt_graphics_buffer_handle_t *out_handle) +{ + if (ipc_c == NULL || out_have == NULL || out_handle == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + *out_have = false; + *out_handle = XRT_GRAPHICS_BUFFER_HANDLE_INVALID; + uint32_t w = 0, h = 0, f = 0; + xrt_result_t xret = ipc_call_lift_get_depth_output(ipc_c, stream_id, out_have, &w, &h, &f, out_handle, 1); + if (out_width != NULL) { + *out_width = w; + } + if (out_height != NULL) { + *out_height = h; + } + if (out_format != NULL) { + *out_format = f; + } + return xret; +} + xrt_result_t ipc_client_lift_get_output(struct ipc_connection *ipc_c, uint64_t stream_id, @@ -261,6 +300,24 @@ ipc_client_lift_weave_rects(struct ipc_connection *ipc_c, uint32_t count, const if (ipc_c == NULL || count > IPC_LIFT_WEAVE_RECTS_MAX || (count > 0 && rects == NULL)) { return XRT_ERROR_IPC_FAILURE; } + // XR_DXR_lift v3: EXPLICIT viewpoints of lifted rects do not fit the rects + // message; stage them first, one call per such rect. + for (uint32_t i = 0; i < count; i++) { + if (rects[i].viewpoint_floats < 3) { + continue; + } + struct ipc_arg_lift_rect_viewpoints va; + U_ZERO(&va); + va.stream_id = rects[i].stream_id; + uint32_t n = rects[i].viewpoint_floats / 3; + n = n > IPC_LIFT_MAX_VIEWS ? IPC_LIFT_MAX_VIEWS : n; + va.viewpoint_count = n; + memcpy(va.viewpoints, rects[i].viewpoints, (size_t)n * 3 * sizeof(float)); + xrt_result_t vx = ipc_call_lift_weave_rect_viewpoints(ipc_c, &va); + if (vx == XRT_ERROR_IPC_FAILURE) { + return vx; + } + } struct ipc_arg_lift_weave_rects args; U_ZERO(&args); args.count = count; @@ -275,6 +332,8 @@ ipc_client_lift_weave_rects(struct ipc_connection *ipc_c, uint32_t count, const args.rects[i].focal_px = rects[i].params.focal_px; args.rects[i].has_view_control = rects[i].has_view_control ? 1u : 0u; args.rects[i].view_control = rects[i].view_control; + args.rects[i].viewpoint_count = rects[i].viewpoint_floats / 3; + args.rects[i].rig = rects[i].rig; } return ipc_call_lift_weave_rects(ipc_c, &args); } diff --git a/src/xrt/ipc/client/ipc_client_lift.h b/src/xrt/ipc/client/ipc_client_lift.h index e2aafceb3c..fc42207d1b 100644 --- a/src/xrt/ipc/client/ipc_client_lift.h +++ b/src/xrt/ipc/client/ipc_client_lift.h @@ -29,9 +29,16 @@ struct ipc_connection; xrt_result_t ipc_client_lift_get_properties(struct ipc_connection *ipc_c, struct xrt_dp_lift_caps *out_caps); +//! @p aux_outputs / @p aux_depth_format: XR_DXR_lift v3 auxiliary depth request +//! (XRT_DP_LIFT_AUX_*, DXGI hint); 0 / 0 = none. xrt_result_t -ipc_client_lift_stream_create( - struct ipc_connection *ipc_c, uint32_t mode, uint32_t content_hint, float input_scale, uint64_t *out_stream_id); +ipc_client_lift_stream_create(struct ipc_connection *ipc_c, + uint32_t mode, + uint32_t content_hint, + float input_scale, + uint32_t aux_outputs, + uint32_t aux_depth_format, + uint64_t *out_stream_id); xrt_result_t ipc_client_lift_stream_destroy(struct ipc_connection *ipc_c, uint64_t stream_id); @@ -54,6 +61,7 @@ ipc_client_lift_submit(struct ipc_connection *ipc_c, const float *viewpoints, uint32_t viewpoint_count, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, uint64_t *out_frame_id); //! @p out_ready false = nothing newer than the last acquire (NOT READY). @@ -73,6 +81,17 @@ ipc_client_lift_get_output(struct ipc_connection *ipc_c, uint32_t *out_format, xrt_graphics_buffer_handle_t *out_handle); +//! XR_DXR_lift v3: the stream's auxiliary depth export texture (a NEW +//! handle, caller-owned) + its layout. +xrt_result_t +ipc_client_lift_get_depth_output(struct ipc_connection *ipc_c, + uint64_t stream_id, + bool *out_have, + uint32_t *out_width, + uint32_t *out_height, + uint32_t *out_format, + xrt_graphics_buffer_handle_t *out_handle); + //! The stream's export fence (a NEW handle, caller-owned). xrt_result_t ipc_client_lift_get_fence(struct ipc_connection *ipc_c, diff --git a/src/xrt/ipc/server/ipc_server.h b/src/xrt/ipc/server/ipc_server.h index 43d3aedaf0..71f4b88bf6 100644 --- a/src/xrt/ipc/server/ipc_server.h +++ b/src/xrt/ipc/server/ipc_server.h @@ -220,6 +220,14 @@ struct ipc_client_state */ uint64_t lift_owner; + /*! + * XR_DXR_lift v3: EXPLICIT viewpoints of lifted weave rects, staged by + * lift_weave_rect_viewpoints and consumed (cleared) by the next + * lift_weave_rects on this connection — they do not fit that message. + */ + struct ipc_arg_lift_rect_viewpoints lift_staged_vps[IPC_LIFT_WEAVE_RECTS_MAX]; + uint32_t lift_staged_vps_count; + /*! * XR_DXR_stereo_camera (ADR-043): this connection's stream-owner token, * taken from the camera manager on first stream create (0 = never used a diff --git a/src/xrt/ipc/server/ipc_server_handler.c b/src/xrt/ipc/server/ipc_server_handler.c index e8e176946f..8cad4fbe16 100644 --- a/src/xrt/ipc/server/ipc_server_handler.c +++ b/src/xrt/ipc/server/ipc_server_handler.c @@ -8352,6 +8352,8 @@ ipc_handle_lift_get_properties(volatile struct ipc_client_state *ics, struct ipc out_props->typical_latency_ns = caps.typical_latency_ns; memcpy(out_props->backend, caps.backend, sizeof(out_props->backend)); out_props->backend[sizeof(out_props->backend) - 1] = '\0'; + out_props->aux_outputs = caps.aux_outputs; + out_props->aux_depth_semantics = caps.aux_depth_semantics; } #endif // Anything else: modes 0, UNAVAILABLE — a successful answer, not an error. @@ -8363,6 +8365,8 @@ ipc_handle_lift_stream_create(volatile struct ipc_client_state *ics, uint32_t mode, uint32_t content_hint, float input_scale, + uint32_t aux_outputs, + uint32_t aux_depth_format, uint64_t *out_stream_id) { IPC_TRACE_MARKER(); @@ -8379,12 +8383,16 @@ ipc_handle_lift_stream_create(volatile struct ipc_client_state *ics, info.mode = mode; info.content_hint = content_hint; info.input_scale = input_scale; + info.aux_outputs = aux_outputs; + info.aux_depth_format = aux_depth_format; return comp_d3d11_service_lift_stream_create(xsysc, lift_owner(ics), &info, out_stream_id); } #else (void)mode; (void)content_hint; (void)input_scale; + (void)aux_outputs; + (void)aux_depth_format; #endif return XRT_ERROR_FEATURE_NOT_SUPPORTED; } @@ -8446,13 +8454,15 @@ ipc_handle_lift_submit_frame(volatile struct ipc_client_state *ics, uint32_t vp_count = args->params.viewpoint_count > IPC_LIFT_MAX_VIEWS ? IPC_LIFT_MAX_VIEWS : args->params.viewpoint_count; const bool has_vc = args->has_params && args->params.has_view_control != 0; + const bool has_rig = args->has_params && args->params.rig.type != XRT_LIFT_RIG_NONE; // The handle is the service's from here (closed or cached inside). The // session's compositor (NULL on a headless connection) names the window the // viewpoints are rebased to (ADR-048). return comp_d3d11_service_lift_submit( xsysc, ics->xc, lift_owner(ics), args->stream_id, in_handle, in_is_dxgi, args->width, args->height, args->source_time, args->has_params ? ¶ms : NULL, args->params.viewpoints, - args->has_params ? 3 * vp_count : 0, has_vc ? &args->params.view_control : NULL, out_frame_id); + args->has_params ? 3 * vp_count : 0, has_vc ? &args->params.view_control : NULL, + has_rig ? &args->params.rig : NULL, out_frame_id); #else (void)args; weave_submit_release_handles(handles, handle_count); @@ -8493,6 +8503,7 @@ ipc_handle_lift_acquire_result(volatile struct ipc_client_state *ics, memcpy(out_lift_result->viewpoints, r.viewpoints, sizeof(out_lift_result->viewpoints)); memcpy(out_lift_result->rect_center, r.rect_center, sizeof(out_lift_result->rect_center)); memcpy(out_lift_result->rect_size, r.rect_size, sizeof(out_lift_result->rect_size)); + out_lift_result->depth = r.depth; return xret; #else (void)stream_id; @@ -8541,6 +8552,47 @@ ipc_handle_lift_get_output(volatile struct ipc_client_state *ics, return XRT_SUCCESS; } +xrt_result_t +ipc_handle_lift_get_depth_output(volatile struct ipc_client_state *ics, + uint64_t stream_id, + bool *out_have_depth, + uint32_t *out_width, + uint32_t *out_height, + uint32_t *out_format, + uint32_t max_handle_count, + xrt_graphics_buffer_handle_t *out_handles, + uint32_t *out_handle_count) +{ + IPC_TRACE_MARKER(); + *out_have_depth = false; + *out_width = 0; + *out_height = 0; + *out_format = 0; + *out_handle_count = 0; + xrt_result_t auth = require_lift_client(ics, "lift_get_depth_output"); + if (auth != XRT_SUCCESS) { + return auth; + } + if (max_handle_count < 1) { + return XRT_SUCCESS; + } +#if defined(XRT_HAVE_D3D11_SERVICE_COMPOSITOR) + struct xrt_system_compositor *xsysc = lift_xsysc(ics); + xrt_graphics_buffer_handle_t h = XRT_GRAPHICS_BUFFER_HANDLE_INVALID; + if (xsysc != NULL && comp_d3d11_service_lift_export_depth(xsysc, lift_owner(ics), stream_id, &h, out_width, + out_height, out_format)) { + // Service-owned: the transport DuplicateHandle's it into the caller. + out_handles[0] = h; + *out_handle_count = 1; + *out_have_depth = true; + } +#else + (void)stream_id; + (void)out_handles; +#endif + return XRT_SUCCESS; +} + xrt_result_t ipc_handle_lift_get_fence(volatile struct ipc_client_state *ics, uint64_t stream_id, @@ -8633,6 +8685,37 @@ ipc_handle_lift_acquire_blob(volatile struct ipc_client_state *ics, uint64_t str return xret; } +// XR_DXR_lift v3 grew the lift messages (rig per rect; explicit viewpoints +// ride their own call precisely so these stay inside one pipe message). +static_assert(sizeof(struct ipc_lift_weave_rects_msg) <= IPC_BUF_SIZE, "lift_weave_rects message exceeds IPC_BUF_SIZE"); +static_assert(sizeof(struct ipc_lift_weave_rect_viewpoints_msg) <= IPC_BUF_SIZE, + "lift_weave_rect_viewpoints message exceeds IPC_BUF_SIZE"); +static_assert(sizeof(struct ipc_lift_submit_frame_msg) <= IPC_BUF_SIZE, + "lift_submit_frame message exceeds IPC_BUF_SIZE"); + +xrt_result_t +ipc_handle_lift_weave_rect_viewpoints(volatile struct ipc_client_state *ics, + const struct ipc_arg_lift_rect_viewpoints *args) +{ + IPC_TRACE_MARKER(); + xrt_result_t auth = require_present_owner(ics, "lift_weave_rect_viewpoints"); + if (auth != XRT_SUCCESS) { + return auth; + } + if (args->viewpoint_count == 0 || args->viewpoint_count > IPC_LIFT_MAX_VIEWS) { + return XRT_ERROR_IPC_FAILURE; // untrusted wire + } + if (ics->lift_staged_vps_count >= IPC_LIFT_WEAVE_RECTS_MAX) { + return XRT_ERROR_WEAVE_REFUSED; // more than one weave submit's worth: drop + } + // Staged until the next lift_weave_rects on this connection (which clears it). + struct ipc_arg_lift_rect_viewpoints *slot = + (struct ipc_arg_lift_rect_viewpoints *)&ics->lift_staged_vps[ics->lift_staged_vps_count]; + *slot = *args; + ics->lift_staged_vps_count++; + return XRT_SUCCESS; +} + xrt_result_t ipc_handle_lift_weave_rects(volatile struct ipc_client_state *ics, const struct ipc_arg_lift_weave_rects *args) { @@ -8663,7 +8746,24 @@ ipc_handle_lift_weave_rects(volatile struct ipc_client_state *ics, const struct rects[i].params.focal_px = w->focal_px; rects[i].has_view_control = w->has_view_control != 0; rects[i].view_control = w->view_control; + rects[i].rig = w->rig; + // XR_DXR_lift v3: this rect's EXPLICIT viewpoints, staged by + // lift_weave_rect_viewpoints just before (none staged = tracked). + if (w->viewpoint_count > 0) { + for (uint32_t k = 0; k < ics->lift_staged_vps_count; k++) { + const struct ipc_arg_lift_rect_viewpoints *sv = + (const struct ipc_arg_lift_rect_viewpoints *)&ics->lift_staged_vps[k]; + if (sv->stream_id == w->stream_id) { + uint32_t n = sv->viewpoint_count > XRT_LIFT_MAX_VIEWS ? XRT_LIFT_MAX_VIEWS + : sv->viewpoint_count; + rects[i].viewpoint_floats = 3 * n; + memcpy(rects[i].viewpoints, sv->viewpoints, (size_t)n * 3 * sizeof(float)); + break; + } + } + } } + ics->lift_staged_vps_count = 0; if (!comp_d3d11_service_lift_set_weave_rects(ics->xc, lift_owner(ics), args->count, rects)) { // A rect naming a stream this connection does not own (or a non-SBS/NVIEW // one): refused, non-fatal. The submit then weaves those rects as drawn. diff --git a/src/xrt/ipc/shared/ipc_protocol.h b/src/xrt/ipc/shared/ipc_protocol.h index 502143ac01..947d0dcda8 100644 --- a/src/xrt/ipc/shared/ipc_protocol.h +++ b/src/xrt/ipc/shared/ipc_protocol.h @@ -1046,9 +1046,11 @@ struct ipc_lift_properties uint32_t max_views; uint32_t depth_semantics; uint32_t state; - uint32_t reserved; + uint32_t aux_outputs; //!< XRT_DP_LIFT_AUX_* (XR_DXR_lift v3) uint64_t typical_latency_ns; char backend[32]; + uint32_t aux_depth_semantics; //!< XRT_DP_LIFT_DEPTH_* of the aux depth + uint32_t reserved; }; /*! @@ -1067,6 +1069,7 @@ struct ipc_lift_params float viewpoints[3 * IPC_LIFT_MAX_VIEWS]; //!< xyz, display space, metres uint32_t has_view_control; //!< XrLiftViewControlDXR chained (ADR-048) struct xrt_lift_view_control view_control; + struct xrt_lift_rig rig; //!< XR_DXR_lift v3 app rig; type NONE = none }; /*! @@ -1109,12 +1112,14 @@ struct ipc_lift_result float viewpoints[3 * IPC_LIFT_MAX_VIEWS]; float rect_center[3]; float rect_size[2]; + struct xrt_lift_depth_info depth; //!< XR_DXR_lift v3 auxiliary depth (handle via lift_get_depth_output) }; /*! - * One lift-flagged weave rect. Weave-path lifts always synthesize for the - * tracked eyes, so no viewpoints ride here (keeps 8 rects well inside - * IPC_BUF_SIZE). + * One lift-flagged weave rect. EXPLICIT viewpoints (XR_DXR_lift v3) do NOT + * ride here — 8 rects of them would overflow IPC_BUF_SIZE — but in one + * lift_weave_rect_viewpoints call per such rect, sent just before + * lift_weave_rects; @c viewpoint_count says whether this rect expects them. * * @ingroup ipc */ @@ -1130,6 +1135,24 @@ struct ipc_lift_weave_rect float focal_px; uint32_t has_view_control; //!< XrLiftViewControlDXR chained (ADR-048) struct xrt_lift_view_control view_control; + uint32_t viewpoint_count; //!< XR_DXR_lift v3: EXPLICIT viewpoints staged for it; 0 = tracked + struct xrt_lift_rig rig; //!< XR_DXR_lift v3 app rig; type NONE = none +}; + +/*! + * lift_weave_rect_viewpoints arguments (XR_DXR_lift v3): the EXPLICIT + * viewpoints of one lifted rect of the NEXT lift_weave_rects on the same + * connection, keyed by stream (display space, metres). Consumed — and the + * staging cleared — by that lift_weave_rects. + * + * @ingroup ipc + */ +struct ipc_arg_lift_rect_viewpoints +{ + uint64_t stream_id; + uint32_t viewpoint_count; //!< <= IPC_LIFT_MAX_VIEWS + uint32_t reserved; + float viewpoints[3 * IPC_LIFT_MAX_VIEWS]; }; /*! diff --git a/src/xrt/ipc/shared/proto.json b/src/xrt/ipc/shared/proto.json index 21f4e76dc7..ef7266079a 100644 --- a/src/xrt/ipc/shared/proto.json +++ b/src/xrt/ipc/shared/proto.json @@ -843,7 +843,9 @@ "in": [ {"name": "mode", "type": "uint32_t"}, {"name": "content_hint", "type": "uint32_t"}, - {"name": "input_scale", "type": "float"} + {"name": "input_scale", "type": "float"}, + {"name": "aux_outputs", "type": "uint32_t"}, + {"name": "aux_depth_format", "type": "uint32_t"} ], "out": [ {"name": "stream_id", "type": "uint64_t"} @@ -889,6 +891,19 @@ "out_handles": {"type": "xrt_graphics_buffer_handle_t"} }, + "lift_get_depth_output": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ], + "out": [ + {"name": "have_depth", "type": "bool"}, + {"name": "width", "type": "uint32_t"}, + {"name": "height", "type": "uint32_t"}, + {"name": "format", "type": "uint32_t"} + ], + "out_handles": {"type": "xrt_graphics_buffer_handle_t"} + }, + "lift_get_fence": { "in": [ {"name": "stream_id", "type": "uint64_t"} @@ -930,6 +945,12 @@ ] }, + "lift_weave_rect_viewpoints": { + "in": [ + {"name": "args", "type": "struct ipc_arg_lift_rect_viewpoints"} + ] + }, + "lift_weave_rects": { "in": [ {"name": "args", "type": "struct ipc_arg_lift_weave_rects"} diff --git a/src/xrt/state_trackers/oxr/oxr_lift.c b/src/xrt/state_trackers/oxr/oxr_lift.c index 7a9239f3f4..36df37147d 100644 --- a/src/xrt/state_trackers/oxr/oxr_lift.c +++ b/src/xrt/state_trackers/oxr/oxr_lift.c @@ -36,6 +36,7 @@ #include "xrt/xrt_lift.h" +#include #include #include @@ -45,8 +46,13 @@ xrt_result_t comp_ipc_client_compositor_lift_get_properties(struct xrt_compositor *xc, struct xrt_dp_lift_caps *out_caps); xrt_result_t -comp_ipc_client_compositor_lift_stream_create( - struct xrt_compositor *xc, uint32_t mode, uint32_t content_hint, float input_scale, uint64_t *out_stream_id); +comp_ipc_client_compositor_lift_stream_create(struct xrt_compositor *xc, + uint32_t mode, + uint32_t content_hint, + float input_scale, + uint32_t aux_outputs, + uint32_t aux_depth_format, + uint64_t *out_stream_id); xrt_result_t comp_ipc_client_compositor_lift_stream_destroy(struct xrt_compositor *xc, uint64_t stream_id); xrt_result_t @@ -61,6 +67,7 @@ comp_ipc_client_compositor_lift_submit(struct xrt_compositor *xc, const float *viewpoints, uint32_t viewpoint_count, const struct xrt_lift_view_control *view_control, + const struct xrt_lift_rig *rig, uint64_t *out_frame_id); xrt_result_t comp_ipc_client_compositor_lift_acquire(struct xrt_compositor *xc, @@ -73,6 +80,11 @@ comp_ipc_client_compositor_lift_get_output(struct xrt_compositor *xc, bool *out_have, xrt_graphics_buffer_handle_t *out_handle); xrt_result_t +comp_ipc_client_compositor_lift_get_depth_output(struct xrt_compositor *xc, + uint64_t stream_id, + bool *out_have, + xrt_graphics_buffer_handle_t *out_handle); +xrt_result_t comp_ipc_client_compositor_lift_get_fence(struct xrt_compositor *xc, uint64_t stream_id, bool *out_have, @@ -109,6 +121,9 @@ _Static_assert(XR_LIFT_MAX_VIEWS_DXR == XRT_LIFT_MAX_VIEWS, "lift view bound mis _Static_assert(XR_WEAVE_SUBMIT_MAX_LIFT_RECTS_DXR == XRT_LIFT_WEAVE_RECTS_MAX, "lift rect bound mismatch"); _Static_assert(XR_LIFT_BACKEND_NAME_MAX_SIZE_DXR == sizeof(((struct xrt_dp_lift_caps *)0)->backend), "backend name size mismatch"); +_Static_assert(XR_LIFT_DEPTH_ENCODING_LINEAR_DXR == XRT_DP_LIFT_DEPTH_ENCODING_LINEAR && + XR_LIFT_DEPTH_ENCODING_INVERSE_DXR == XRT_DP_LIFT_DEPTH_ENCODING_INVERSE, + "lift depth encoding mismatch"); //! Same rule as oxr_weave.c: IPC sessions carry no in-process native flag. static bool @@ -171,12 +186,8 @@ lift_validate_options(struct oxr_logger *log, const XrLiftOptionsDXR *o, bool we return oxr_error(log, XR_ERROR_VALIDATION_FAILURE, "XrLiftOptionsDXR::viewpointSource (%d) invalid", (int)o->viewpointSource); } + (void)weave_path; // XR_DXR_lift v3: EXPLICIT is valid on the weave path too if (o->viewpointSource == XR_LIFT_VIEWPOINT_SOURCE_EXPLICIT_DXR) { - if (weave_path) { - return oxr_error(log, XR_ERROR_VALIDATION_FAILURE, - "XrLiftOptionsDXR: EXPLICIT viewpoints are not allowed on a weave rect " - "(the weave path always synthesizes for the tracked eyes)"); - } if (o->viewCount == 0 || o->viewpoints == NULL) { return oxr_error(log, XR_ERROR_VALIDATION_FAILURE, "XrLiftOptionsDXR: EXPLICIT viewpoints need viewCount >= 1 and viewpoints"); @@ -212,6 +223,50 @@ oxr_lift_view_control_from_xr(struct oxr_logger *log, const XrLiftViewControlDXR return XR_SUCCESS; } +/*! + * XR_DXR_lift v3 (ADR-048 Addendum A): the app rig on a lift, if any — an + * XrCameraRigDXR or XrDisplayRigDXR chained on @p direct (wins) or on + * @p options (either may be NULL); a camera rig wins over a display rig, as in + * xrLocateViews. Boundary-converted exactly like xrLocateViews' rig + * (verticalFov -> half-tangent, convergenceDiopters = inverse distance, + * metersToVirtual 0 -> 1); the service clamps. Shared with oxr_weave.c. + * Returns false (out->type NONE) when none is chained. + */ +bool +oxr_lift_rig_from_chain(const void *direct, const void *options, struct xrt_lift_rig *out) +{ + memset(out, 0, sizeof(*out)); + out->type = XRT_LIFT_RIG_NONE; + const void *chains[2] = {direct, options}; + for (int c = 0; c < 2; c++) { + const XrBaseInStructure *head = (const XrBaseInStructure *)chains[c]; + if (head == NULL) { + continue; + } + const XrCameraRigDXR *crig = OXR_GET_INPUT_FROM_CHAIN(head, XR_TYPE_CAMERA_RIG_DXR, XrCameraRigDXR); + if (crig != NULL) { + out->type = XRT_LIFT_RIG_CAMERA; + out->ipd_factor = crig->ipdFactor; + out->parallax_factor = crig->parallaxFactor; + out->inv_convergence_distance = crig->convergenceDiopters; + out->half_tan_vfov = tanf(crig->verticalFov * 0.5f); + out->m2v = crig->metersToVirtual > 0.0f ? crig->metersToVirtual : 1.0f; + out->perspective_factor = 1.0f; + return true; + } + const XrDisplayRigDXR *drig = OXR_GET_INPUT_FROM_CHAIN(head, XR_TYPE_DISPLAY_RIG_DXR, XrDisplayRigDXR); + if (drig != NULL) { + out->type = XRT_LIFT_RIG_DISPLAY; + out->ipd_factor = drig->ipdFactor; + out->parallax_factor = drig->parallaxFactor; + out->perspective_factor = drig->perspectiveFactor; + out->m2v = 1.0f; + return true; + } + } + return false; +} + static XrResult oxr_lift_stream_destroy_cb(struct oxr_logger *log, struct oxr_handle_base *hb) { @@ -267,6 +322,16 @@ oxr_xrGetLiftPropertiesDXR(XrSession session, XrLiftPropertiesDXR *properties) memcpy(properties->backend, caps.backend, sizeof(properties->backend)); properties->backend[sizeof(properties->backend) - 1] = '\0'; properties->typicalLatency = (XrDuration)caps.typical_latency_ns; + + // Spec v3 (ADR-048 Addendum A): the auxiliary-depth capability. + XrLiftDepthPropertiesDXR *dp = + OXR_GET_OUTPUT_FROM_CHAIN(properties, XR_TYPE_LIFT_DEPTH_PROPERTIES_DXR, XrLiftDepthPropertiesDXR); + if (dp != NULL) { + dp->auxDepthSupported = (caps.aux_outputs & XRT_DP_LIFT_AUX_DEPTH) != 0 ? XR_TRUE : XR_FALSE; + dp->auxDepthSemantics = caps.aux_depth_semantics == XRT_DP_LIFT_DEPTH_METRIC + ? XR_LIFT_DEPTH_SEMANTICS_METRIC_DXR + : XR_LIFT_DEPTH_SEMANTICS_RELATIVE_DXR; + } return XR_SUCCESS; } @@ -303,6 +368,20 @@ oxr_xrCreateLiftStreamDXR(XrSession session, const XrLiftStreamCreateInfoDXR *cr "XrLiftStreamCreateInfoDXR::inputScale (%f) must be in [0, 1]", (double)createInfo->inputScale); } + // Spec v3 (ADR-048 Addendum A): auxiliary depth, SBS / NVIEW only. + const XrLiftDepthRequestDXR *dreq = + OXR_GET_INPUT_FROM_CHAIN(createInfo, XR_TYPE_LIFT_DEPTH_REQUEST_DXR, XrLiftDepthRequestDXR); + uint32_t aux_outputs = 0, aux_depth_format = 0; + if (dreq != NULL) { + if (mode != XRT_DP_LIFT_MODE_SBS && mode != XRT_DP_LIFT_MODE_NVIEW) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, + "XrLiftDepthRequestDXR: auxiliary depth is for SBS / NVIEW streams only"); + } + aux_outputs = XRT_DP_LIFT_AUX_DEPTH; + aux_depth_format = dreq->preferredFormat > 0 && dreq->preferredFormat <= UINT32_MAX + ? (uint32_t)dreq->preferredFormat + : 0u; + } if (!lift_session_is_ipc(sess)) { return oxr_error(&log, XR_ERROR_FEATURE_UNSUPPORTED, "xrCreateLiftStreamDXR: only available on the out-of-process (service) path"); @@ -311,7 +390,7 @@ oxr_xrCreateLiftStreamDXR(XrSession session, const XrLiftStreamCreateInfoDXR *cr uint64_t id = 0; xrt_result_t xret = comp_ipc_client_compositor_lift_stream_create( &sess->xcn->base, mode, (uint32_t)createInfo->contentHint, - createInfo->inputScale > 0.0f ? createInfo->inputScale : 1.0f, &id); + createInfo->inputScale > 0.0f ? createInfo->inputScale : 1.0f, aux_outputs, aux_depth_format, &id); XrResult r = lift_xret(&log, sess, xret, "xrCreateLiftStreamDXR"); if (r != XR_SUCCESS) { return r; @@ -322,6 +401,7 @@ oxr_xrCreateLiftStreamDXR(XrSession session, const XrLiftStreamCreateInfoDXR *cr st->sess = sess; st->id = id; st->mode = mode; + st->depth_requested = aux_outputs != 0; *stream = XRT_CAST_PTR_TO_OXR_HANDLE(XrLiftStreamDXR, st); return XR_SUCCESS; } @@ -366,6 +446,8 @@ oxr_xrSubmitLiftFrameDXR(XrLiftStreamDXR stream, const XrLiftFrameSubmitInfoDXR uint32_t vp_count = 0; struct xrt_lift_view_control vc; const struct xrt_lift_view_control *vcp = NULL; + struct xrt_lift_rig rig; + const struct xrt_lift_rig *rigp = NULL; const XrLiftOptionsDXR *opt = OXR_GET_INPUT_FROM_CHAIN(submitInfo, XR_TYPE_LIFT_OPTIONS_DXR, XrLiftOptionsDXR); if (opt != NULL) { XrResult vr = lift_validate_options(&log, opt, /*weave_path*/ false); @@ -383,6 +465,10 @@ oxr_xrSubmitLiftFrameDXR(XrLiftStreamDXR stream, const XrLiftFrameSubmitInfoDXR } vcp = &vc; } + // Spec v3: an app rig rides with the options (absent = cleared). + if (oxr_lift_rig_from_chain(NULL, opt, &rig)) { + rigp = &rig; + } if (opt->viewpointSource == XR_LIFT_VIEWPOINT_SOURCE_EXPLICIT_DXR) { vp_count = opt->viewCount; for (uint32_t i = 0; i < vp_count; i++) { @@ -396,10 +482,84 @@ oxr_xrSubmitLiftFrameDXR(XrLiftStreamDXR stream, const XrLiftFrameSubmitInfoDXR xrt_result_t xret = comp_ipc_client_compositor_lift_submit( &sess->xcn->base, st->id, (xrt_graphics_buffer_handle_t)(intptr_t)submitInfo->inputTexture, submitInfo->inputIsDxgi == XR_TRUE, (uint32_t)submitInfo->extent.width, (uint32_t)submitInfo->extent.height, - (int64_t)submitInfo->sourceTime, pp, vp_count > 0 ? vps : NULL, vp_count, vcp, frameId); + (int64_t)submitInfo->sourceTime, pp, vp_count > 0 ? vps : NULL, vp_count, vcp, rigp, frameId); return lift_xret(&log, sess, xret, "xrSubmitLiftFrameDXR"); } +//! Fill an XrLiftDepthResultDXR from an acquired result (+ handles on first +//! depth acquire / reallocation). +static XrResult +lift_fill_depth_result(struct oxr_logger *log, + struct oxr_session *sess, + struct oxr_lift_stream_dxr *st, + const struct xrt_lift_result *r, + XrLiftDepthResultDXR *d) +{ + XrStructureType type = d->type; + void *next = d->next; + memset(d, 0, sizeof(*d)); + d->type = type; + d->next = next; + const struct xrt_lift_depth_info *di = &r->depth; + if (!st->depth_requested || di->valid == 0) { + return XR_SUCCESS; // depthValid XR_FALSE, everything else zero + } + d->depthValid = XR_TRUE; + d->fenceValue = r->fence_value; + d->extent.width = (int32_t)di->width; + d->extent.height = (int32_t)di->height; + d->format = (int64_t)di->format; + d->units = di->units == XRT_DP_LIFT_DEPTH_METRIC ? XR_LIFT_DEPTH_SEMANTICS_METRIC_DXR + : XR_LIFT_DEPTH_SEMANTICS_RELATIVE_DXR; + d->encoding = di->encoding == XRT_DP_LIFT_DEPTH_ENCODING_INVERSE ? XR_LIFT_DEPTH_ENCODING_INVERSE_DXR + : XR_LIFT_DEPTH_ENCODING_LINEAR_DXR; + d->valueScale = di->value_scale; + d->valueOffset = di->value_offset; + d->nearDepth = di->near_depth; + d->farDepth = di->far_depth; + d->sameInference = di->same_inference != 0 ? XR_TRUE : XR_FALSE; + d->frameId = r->frame_id; + d->intrinsicsValid = di->intrinsics_valid != 0 ? XR_TRUE : XR_FALSE; + d->focalLengthPx.x = di->focal[0]; + d->focalLengthPx.y = di->focal[1]; + d->principalPointPx.x = di->principal[0]; + d->principalPointPx.y = di->principal[1]; + d->convergenceDepth = di->convergence_depth; + d->viewpoint.x = di->viewpoint[0]; + d->viewpoint.y = di->viewpoint[1]; + d->viewpoint.z = di->viewpoint[2]; + d->transformValid = di->transform_valid != 0 ? XR_TRUE : XR_FALSE; + memcpy(d->depthToDisplay, di->depth_to_display, sizeof(d->depthToDisplay)); + + // Handles: first depth acquire and every reallocation (the views' rule). + if (di->realloc != 0 || !st->depth_exported) { + bool have_tex = false, have_fence = false; + xrt_graphics_buffer_handle_t th = XRT_GRAPHICS_BUFFER_HANDLE_INVALID; + xrt_graphics_sync_handle_t fh = XRT_GRAPHICS_SYNC_HANDLE_INVALID; + xrt_result_t xret = + comp_ipc_client_compositor_lift_get_depth_output(&sess->xcn->base, st->id, &have_tex, &th); + XrResult xr = lift_xret(log, sess, xret, "xrAcquireLiftResultDXR (depth export)"); + if (xr != XR_SUCCESS) { + return xr; + } + xret = comp_ipc_client_compositor_lift_get_fence(&sess->xcn->base, st->id, &have_fence, &fh); + xr = lift_xret(log, sess, xret, "xrAcquireLiftResultDXR (depth fence export)"); + if (xr != XR_SUCCESS) { + return xr; + } + const bool got_tex = have_tex && th != XRT_GRAPHICS_BUFFER_HANDLE_INVALID; + const bool got_fence = have_fence && fh != XRT_GRAPHICS_SYNC_HANDLE_INVALID; + if (got_tex) { + d->depthTexture = (void *)(intptr_t)th; + } + if (got_fence) { + d->fence = (void *)(intptr_t)fh; + } + st->depth_exported = got_tex && got_fence; + } + return XR_SUCCESS; +} + XRAPI_ATTR XrResult XRAPI_CALL oxr_xrAcquireLiftResultDXR(XrLiftStreamDXR stream, XrLiftResultDXR *result) { @@ -484,6 +644,16 @@ oxr_xrAcquireLiftResultDXR(XrLiftStreamDXR stream, XrLiftResultDXR *result) // Latch only on a complete export (the #1427 rule): a miss retries. st->exported = got_tex && got_fence; } + + // Spec v3 (ADR-048 Addendum A): the auxiliary depth of this result. + XrLiftDepthResultDXR *dres = + OXR_GET_OUTPUT_FROM_CHAIN(result, XR_TYPE_LIFT_DEPTH_RESULT_DXR, XrLiftDepthResultDXR); + if (dres != NULL) { + xr = lift_fill_depth_result(&log, sess, st, &r, dres); + if (xr != XR_SUCCESS) { + return xr; + } + } return XR_SUCCESS; } diff --git a/src/xrt/state_trackers/oxr/oxr_objects.h b/src/xrt/state_trackers/oxr/oxr_objects.h index 97ed9d15e0..3bdeea3522 100644 --- a/src/xrt/state_trackers/oxr/oxr_objects.h +++ b/src/xrt/state_trackers/oxr/oxr_objects.h @@ -4481,6 +4481,8 @@ struct oxr_lift_stream_dxr uint64_t id; //!< service-side stream id uint32_t mode; //!< XRT_DP_LIFT_MODE_* (one bit) bool exported; //!< the caller holds the current export texture + fence + bool depth_requested; //!< XR_DXR_lift v3: created with XrLiftDepthRequestDXR + bool depth_exported; //!< the caller holds the current depth texture + fence }; #endif // OXR_HAVE_DXR_lift diff --git a/src/xrt/state_trackers/oxr/oxr_weave.c b/src/xrt/state_trackers/oxr/oxr_weave.c index b7fe0710cc..78c7d70f9d 100644 --- a/src/xrt/state_trackers/oxr/oxr_weave.c +++ b/src/xrt/state_trackers/oxr/oxr_weave.c @@ -111,6 +111,8 @@ comp_ipc_client_compositor_lift_weave_rects(struct xrt_compositor *xc, //! XrLiftViewControlDXR -> xrt (oxr_lift.c). XrResult oxr_lift_view_control_from_xr(struct oxr_logger *log, const XrLiftViewControlDXR *v, struct xrt_lift_view_control *out); +bool +oxr_lift_rig_from_chain(const void *direct, const void *options, struct xrt_lift_rig *out); #endif // Forward decls of the IPC-bridge wrappers (defined in ipc_client_compositor.c). @@ -793,12 +795,31 @@ oxr_xrWeaveSubmitDXR(XrSession session, const XrWeaveSubmitInfoDXR *submitInfo, lr[i].rect_index = e->rectIndex; const XrLiftOptionsDXR *o = OXR_GET_INPUT_FROM_CHAIN(e, XR_TYPE_LIFT_OPTIONS_DXR, XrLiftOptionsDXR); if (o != NULL) { - if (o->viewpointSource == XR_LIFT_VIEWPOINT_SOURCE_EXPLICIT_DXR || - o->viewCount > XR_LIFT_MAX_VIEWS_DXR) { - return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, - "xrWeaveSubmitDXR: lifts[%u] options: TRACKED viewpoints only on the " - "weave path, viewCount <= %u", - i, (uint32_t)XR_LIFT_MAX_VIEWS_DXR); + if (o->viewCount > XR_LIFT_MAX_VIEWS_DXR || + (o->viewpointSource != XR_LIFT_VIEWPOINT_SOURCE_TRACKED_DXR && + o->viewpointSource != XR_LIFT_VIEWPOINT_SOURCE_EXPLICIT_DXR)) { + return oxr_error( + &log, XR_ERROR_VALIDATION_FAILURE, + "xrWeaveSubmitDXR: lifts[%u] options: viewCount <= %u and a valid " + "viewpointSource", + i, (uint32_t)XR_LIFT_MAX_VIEWS_DXR); + } + // XR_DXR_lift v3: EXPLICIT viewpoints on a lifted rect, with the + // same validation as on a submit. + if (o->viewpointSource == XR_LIFT_VIEWPOINT_SOURCE_EXPLICIT_DXR) { + if (o->viewCount == 0 || o->viewpoints == NULL) { + return oxr_error( + &log, XR_ERROR_VALIDATION_FAILURE, + "xrWeaveSubmitDXR: lifts[%u] options: EXPLICIT viewpoints need " + "viewCount >= 1 and viewpoints", + i); + } + for (uint32_t k = 0; k < o->viewCount; k++) { + lr[i].viewpoints[3 * k + 0] = o->viewpoints[k].x; + lr[i].viewpoints[3 * k + 1] = o->viewpoints[k].y; + lr[i].viewpoints[3 * k + 2] = o->viewpoints[k].z; + } + lr[i].viewpoint_floats = 3 * o->viewCount; } lr[i].has_params = true; lr[i].params.struct_size = (uint32_t)sizeof(lr[i].params); @@ -823,6 +844,9 @@ oxr_xrWeaveSubmitDXR(XrSession session, const XrWeaveSubmitInfoDXR *submitInfo, } lr[i].has_view_control = true; } + // Spec v3 (ADR-048 Addendum A): an app rig, directly on the rect + // (wins) or on its XrLiftOptionsDXR. + (void)oxr_lift_rig_from_chain(e, o, &lr[i].rig); } xrt_result_t lx = comp_ipc_client_compositor_lift_weave_rects(&sess->xcn->base, lifts->liftCount, lr); if (lx == XRT_ERROR_IPC_FAILURE) { diff --git a/src/xrt/targets/cli/cli_cmd_lift.cpp b/src/xrt/targets/cli/cli_cmd_lift.cpp index 9ba24a5825..c1d5ec7bb0 100644 --- a/src/xrt/targets/cli/cli_cmd_lift.cpp +++ b/src/xrt/targets/cli/cli_cmd_lift.cpp @@ -600,7 +600,7 @@ cmd_probe(int argc, const char **argv) uint64_t sid = 0; xrt_result_t xret = ipc_client_lift_stream_create( &ipc_c, mode, mode == XRT_DP_LIFT_MODE_GAUSSIANS ? XRT_DP_LIFT_CONTENT_PHOTO : XRT_DP_LIFT_CONTENT_VIDEO, - 1.0f, &sid); + 1.0f, /*aux_outputs*/ 0, /*aux_depth_format*/ 0, &sid); if (xret != XRT_SUCCESS) { printf("lift_stream_create failed: %d\n", (int)xret); ipc_client_connection_fini(&ipc_c); @@ -647,7 +647,7 @@ cmd_probe(int argc, const char **argv) uint64_t frame_id = 0; xret = ipc_client_lift_submit(&ipc_c, sid, (xrt_graphics_buffer_handle_t)g.in_handle, false, (uint32_t)iw, - (uint32_t)ih, (int64_t)i, ¶ms, nullptr, 0, nullptr, &frame_id); + (uint32_t)ih, (int64_t)i, ¶ms, nullptr, 0, nullptr, nullptr, &frame_id); if (xret != XRT_SUCCESS) { printf("%6d submit refused (xrt_result=%d) — retrying next frame\n", i, (int)xret); continue; diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 0e836d98f4..928358c436 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -30,6 +30,7 @@ set(tests tests_weave_snap_grid tests_lift_mailbox tests_aux_lift_viewpoint + tests_aux_lift_rig_depth tests_space_overseer_rig tests_rig_composer tests_aux_x11_scale @@ -760,6 +761,10 @@ add_test(NAME tests_ipc_proto COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURC target_sources(tests_lift_mailbox PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display/sim_display_fake_ply.c) target_include_directories(tests_lift_mailbox PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display) +# XR_DXR_lift v3 (ADR-048 Addendum A): the app-rig mapping is checked against the +# shared displayxr-common rig core xrLocateViews runs. +target_link_libraries(tests_aux_lift_rig_depth PRIVATE displayxr::math_core) + # ADR-043: the sim_display FAKE stereo camera's scene is pinned host-side next to # the ring / conversion / disparity-probe tests (it compiles into the plug-in). target_sources(tests_stereo_camera PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.c) diff --git a/tests/tests_aux_lift_rig_depth.cpp b/tests/tests_aux_lift_rig_depth.cpp new file mode 100644 index 0000000000..9be84842cd --- /dev/null +++ b/tests/tests_aux_lift_rig_depth.cpp @@ -0,0 +1,243 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_lift v3 (ADR-048 Addendum A): the app-rig -> lift viewpoints + * mapping (u_lift_rig) against the shared xrLocateViews rig core, and + * the auxiliary-depth display transform (u_lift_depth). + */ + +#include "util/u_lift_depth.h" +#include "util/u_lift_rig.h" +#include "util/u_lift_viewpoint.h" + +#include "dxr_view_math.h" + +#include "catch_amalgamated.hpp" + +#include +#include + +using Catch::Approx; + +namespace { + +constexpr float kEps = 1e-4f; + +void +require_vec(const float *a, const float *b, float eps = kEps) +{ + CHECK(a[0] == Approx(b[0]).margin(eps)); + CHECK(a[1] == Approx(b[1]).margin(eps)); + CHECK(a[2] == Approx(b[2]).margin(eps)); +} + +// Rect-relative tracked eyes: viewer 4 cm right, 2 cm up, 0.6 m away, IPD 63 mm. +const float kEyes[6] = {0.04f - 0.0315f, 0.02f, 0.60f, 0.04f + 0.0315f, 0.02f, 0.60f}; +constexpr float kRectW = 0.30f; +constexpr float kRectH = 0.18f; +constexpr float kNominalZ = 0.5f; + +} // namespace + +TEST_CASE("lift_rig: NONE passes the eyes through", "[lift_rig]") +{ + u_lift_rig rig = {}; + rig.type = U_LIFT_RIG_NONE; + float out[6]; + REQUIRE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, kEyes, 2, out)); + require_vec(&out[0], &kEyes[0]); + require_vec(&out[3], &kEyes[3]); +} + +TEST_CASE("lift_rig: unknown rect or too many eyes is refused", "[lift_rig]") +{ + u_lift_rig rig = {U_LIFT_RIG_DISPLAY, 1.0f, 1.0f, 1.0f, 0.0f, 0.5f, 1.0f}; + float out[6]; + CHECK_FALSE(u_lift_rig_apply(&rig, kNominalZ, 0.0f, kRectH, kEyes, 2, out)); + CHECK_FALSE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, kEyes, 0, out)); + CHECK_FALSE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, kEyes, 9, out)); +} + +TEST_CASE("lift_rig: sanitize clamps like xrLocateViews", "[lift_rig]") +{ + u_lift_rig rig = {U_LIFT_RIG_CAMERA, -1.0f, 2.0e4f, 50.0f, 30.0f, NAN, 0.0f}; + CHECK(u_lift_rig_sanitize(&rig)); + CHECK(rig.ipd_factor == 0.0f); + CHECK(rig.parallax_factor == 1.0e4f); + CHECK(rig.perspective_factor == 10.0f); + CHECK(rig.inv_convergence_distance == 20.0f); + CHECK(rig.half_tan_vfov == Approx(std::tan(0.5f))); + CHECK(rig.m2v == 1.0f); // 0 / unset = identity + + u_lift_rig bad = {77u, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f}; + u_lift_rig_sanitize(&bad); + CHECK(bad.type == U_LIFT_RIG_NONE); +} + +TEST_CASE("lift_rig: display rig = the xrLocateViews eyes after window rebase", "[lift_rig]") +{ + // xrLocateViews in a window (ADR-012): raw display-space eyes, rebased to + // the window centre, through the shared display3d core with the window as + // the screen and the app's virtual display height (any m2v). The lift + // path with the same rig must land on the same physical eyes. + const float window_centre[3] = {-0.05f, 0.03f, 0.0f}; + float raw[6]; + for (int i = 0; i < 2; i++) { + for (int a = 0; a < 3; a++) { + raw[3 * i + a] = kEyes[3 * i + a] + window_centre[a]; + } + } + + const float ipd = 0.6f, parallax = 0.7f, persp = 1.3f, vdh = 2.4f; // m2v = vdh / rect_h + dxr_vec3 rebased[2]; + for (int i = 0; i < 2; i++) { + rebased[i] = {raw[3 * i] - window_centre[0], raw[3 * i + 1] - window_centre[1], + raw[3 * i + 2] - window_centre[2]}; + } + dxr_display3d_tunables t = dxr_display3d_default_tunables(); + t.ipd_factor = ipd; + t.parallax_factor = parallax; + t.perspective_factor = persp; + t.virtual_display_height = vdh; + const dxr_vec3 nominal = {0.0f, 0.0f, kNominalZ}; + const dxr_screen screen = {kRectW, kRectH}; + dxr_display3d_view located[2]; + dxr_display3d_compute_views(rebased, 2, &nominal, &screen, &t, nullptr, 0.0f, 0.0f, 0, located); + const float m2v = vdh / kRectH; + + // Lift: rebase (u_lift_viewpoint_rebase), policy as a pass-through + // (XYZ, recentering OFF, no clamp), then the rig. + float vps[6]; + memcpy(vps, raw, sizeof(vps)); + u_lift_viewpoint_rebase(vps, 2, window_centre); + u_lift_view_control vc; + u_lift_view_control_default(&vc); + vc.axis_mode = U_LIFT_AXIS_XYZ; + vc.recenter_mode = U_LIFT_RECENTER_OFF; + vc.max_offset_m = 0.0f; + u_lift_viewpoint_apply(&vc, nullptr, kNominalZ, 0, vps, 2, vps, nullptr, nullptr); + u_lift_rig rig = {U_LIFT_RIG_DISPLAY, ipd, parallax, persp, 0.0f, 0.5f, 1.0f}; + REQUIRE_FALSE(u_lift_rig_sanitize(&rig)); + REQUIRE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, vps, 2, vps)); + + for (int i = 0; i < 2; i++) { + const float expect[3] = {located[i].eye_display.x / m2v, located[i].eye_display.y / m2v, + located[i].eye_display.z / m2v}; + require_vec(&vps[3 * i], expect); + } + // And the frustum each eye sees onto the rect is the located one. + for (int i = 0; i < 2; i++) { + const dxr_fov f = + dxr_display3d_compute_fov({vps[3 * i], vps[3 * i + 1], vps[3 * i + 2]}, kRectW, kRectH); + CHECK(f.angle_left == Approx(located[i].fov.angle_left).margin(kEps)); + CHECK(f.angle_right == Approx(located[i].fov.angle_right).margin(kEps)); + CHECK(f.angle_up == Approx(located[i].fov.angle_up).margin(kEps)); + CHECK(f.angle_down == Approx(located[i].fov.angle_down).margin(kEps)); + } +} + +TEST_CASE("lift_rig: camera rig viewpoints see the camera rig's frustum", "[lift_rig]") +{ + const float ipd = 1.5f, parallax = 0.8f, invd = 0.4f, vfov = 0.9f, m2v = 2.0f; + u_lift_rig rig = {U_LIFT_RIG_CAMERA, ipd, parallax, 1.0f, invd, std::tan(0.5f * vfov), m2v}; + REQUIRE_FALSE(u_lift_rig_sanitize(&rig)); + float vps[6]; + REQUIRE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, kEyes, 2, vps)); + + // The camera rig as xrLocateViews computes it (shared camera3d core). + dxr_camera3d_tunables t = dxr_camera3d_default_tunables(); + t.ipd_factor = ipd; + t.parallax_factor = parallax; + t.inv_convergence_distance = invd; + t.half_tan_vfov = std::tan(0.5f * vfov); + t.m2v = m2v; + const dxr_vec3 eyes[2] = {{kEyes[0], kEyes[1], kEyes[2]}, {kEyes[3], kEyes[4], kEyes[5]}}; + const dxr_vec3 nominal = {0.0f, 0.0f, kNominalZ}; + const dxr_screen screen = {kRectW, kRectH}; + dxr_camera3d_view cam[2]; + dxr_camera3d_compute_views(eyes, 2, &nominal, &screen, &t, nullptr, 0.1f, 100.0f, cam); + + for (int i = 0; i < 2; i++) { + REQUIRE(vps[3 * i + 2] > 0.0f); + const dxr_fov f = + dxr_display3d_compute_fov({vps[3 * i], vps[3 * i + 1], vps[3 * i + 2]}, kRectW, kRectH); + CHECK(f.angle_left == Approx(cam[i].fov.angle_left).margin(kEps)); + CHECK(f.angle_right == Approx(cam[i].fov.angle_right).margin(kEps)); + CHECK(f.angle_up == Approx(cam[i].fov.angle_up).margin(kEps)); + CHECK(f.angle_down == Approx(cam[i].fov.angle_down).margin(kEps)); + } + // The equivalent viewer of the nominal (unperturbed) camera sits at + // rect_h / (2 tan(vfov / 2)) in front of the rect. + float nom[3] = {0.0f, 0.0f, kNominalZ}; + float nom_out[3]; + REQUIRE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, nom, 1, nom_out)); + CHECK(nom_out[2] == Approx(kRectH / (2.0f * std::tan(0.5f * vfov))).margin(kEps)); + CHECK(nom_out[0] == Approx(0.0f).margin(kEps)); +} + +TEST_CASE("lift_rig: camera rig at infinite convergence collapses the views", "[lift_rig]") +{ + u_lift_rig rig = {U_LIFT_RIG_CAMERA, 1.0f, 1.0f, 1.0f, 0.0f, std::tan(0.4f), 1.0f}; + float vps[6]; + REQUIRE(u_lift_rig_apply(&rig, kNominalZ, kRectW, kRectH, kEyes, 2, vps)); + require_vec(&vps[0], &vps[3]); + CHECK(vps[2] == Approx(kRectH / (2.0f * std::tan(0.4f))).margin(kEps)); +} + +TEST_CASE("lift_depth: convergence depth lands on the rect, 0 at the viewpoint", "[lift_depth]") +{ + const float c[3] = {0.03f, -0.01f, 0.55f}; // viewpoint, rect-relative + const float r[3] = {0.10f, 0.05f, 0.0f}; // rect centre, display space + const float w = 0.32f, h = 0.20f, dc = 2.5f; // metres, convergence depth + float m[16]; + REQUIRE(u_lift_depth_to_display(c, r, w, h, dc, m)); + + const float uv[5][2] = {{0.5f, 0.5f}, {0.0f, 0.0f}, {1.0f, 0.0f}, {0.25f, 0.75f}, {1.0f, 1.0f}}; + for (const auto &t : uv) { + float p[3]; + u_lift_depth_transform_point(m, t[0], t[1], dc, p); + const float on_rect[3] = {r[0] + w * (t[0] - 0.5f), r[1] + h * (0.5f - t[1]), r[2]}; + require_vec(p, on_rect); + + float e[3]; + u_lift_depth_transform_point(m, t[0], t[1], 0.0f, e); + const float eye[3] = {r[0] + c[0], r[1] + c[1], r[2] + c[2]}; + require_vec(e, eye); + + // Behind the screen (d > dc) stays on the ray eye -> rect point. + float b[3]; + u_lift_depth_transform_point(m, t[0], t[1], 2.0f * dc, b); + const float expect[3] = {eye[0] + 2.0f * (on_rect[0] - eye[0]), eye[1] + 2.0f * (on_rect[1] - eye[1]), + eye[2] + 2.0f * (on_rect[2] - eye[2])}; + require_vec(b, expect); + CHECK(b[2] < r[2]); // behind the screen plane + } + CHECK(m[3] == 0.0f); + CHECK(m[7] == 0.0f); + CHECK(m[11] == 0.0f); + CHECK(m[15] == 1.0f); // affine output, w = 1 +} + +TEST_CASE("lift_depth: invalid inputs report no transform", "[lift_depth]") +{ + const float c[3] = {0.0f, 0.0f, 0.5f}; + const float bad_c[3] = {0.0f, 0.0f, 0.0f}; + const float r[3] = {0.0f, 0.0f, 0.0f}; + float m[16]; + CHECK_FALSE(u_lift_depth_to_display(c, r, 0.0f, 0.2f, 1.0f, m)); + CHECK_FALSE(u_lift_depth_to_display(c, r, 0.3f, 0.2f, 0.0f, m)); + CHECK_FALSE(u_lift_depth_to_display(c, r, 0.3f, 0.2f, NAN, m)); + CHECK_FALSE(u_lift_depth_to_display(bad_c, r, 0.3f, 0.2f, 1.0f, m)); + CHECK(m[0] == 1.0f); // identity on failure + CHECK(m[5] == 1.0f); + CHECK(m[12] == 0.0f); +} + +TEST_CASE("lift_depth: decode linear and inverse samples", "[lift_depth]") +{ + CHECK(u_lift_depth_decode(0.5f, 1.0f, 1.0f, false) == Approx(1.5f)); + CHECK(u_lift_depth_decode(0.5f, 0.0f, 0.0f, false) == Approx(0.5f)); // scale 0 read as 1 + CHECK(u_lift_depth_decode(0.25f, 2.0f, 0.0f, true) == Approx(2.0f)); // 1 / (2 * 0.25) + CHECK(u_lift_depth_decode(0.0f, 1.0f, 0.0f, true) == 0.0f); +} From a46d671275110d70be38dc732b7b12e64303ced2 Mon Sep 17 00:00:00 2001 From: dfattal Date: Thu, 8 Oct 2026 03:15:46 -0700 Subject: [PATCH 3/3] docs(lift): XR_DXR_lift v3 spec (app rig, auxiliary depth, depthToDisplay math) + ADR-048 Addendum A MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - XR_DXR_lift.md: header table -> v3, §4.2 app rig (rig -> viewpoint mapping and the camera-rig frustum-equivalence proof), §4.3 auxiliary depth (request, result, decode, intrinsics, the depthToDisplay derivation + matrix table, DP side), §6 EXPLICIT on weave rects, §10 / §11 / §14 rows. - ADR-048 Addendum A: A1 the app rig is computed with the xrLocateViews core, A2 depth is an auxiliary same-inference output with a runtime-derived display transform; why a new slot instead of an out-struct on lift_convert. - xrt_plugin_iface.md lift section: appended params / caps / stream-info fields, struct xrt_dp_lift_depth, slot 33 lift_get_depth. - extensions index.json note. gen_adr_index --check, gen_extensions_index --check and check_doc_paths pass (ADR titles unchanged). Co-Authored-By: Claude Opus 5.5 --- docs/adr/ADR-048-lift-viewpoint-policy.md | 78 ++++++++++++++- docs/reference/xrt_plugin_iface.md | 27 ++++- docs/specs/extensions/XR_DXR_lift.md | 117 ++++++++++++++++++++-- docs/specs/extensions/index.json | 2 +- 4 files changed, 211 insertions(+), 13 deletions(-) diff --git a/docs/adr/ADR-048-lift-viewpoint-policy.md b/docs/adr/ADR-048-lift-viewpoint-policy.md index ed468dbef6..95f5a72868 100644 --- a/docs/adr/ADR-048-lift-viewpoint-policy.md +++ b/docs/adr/ADR-048-lift-viewpoint-policy.md @@ -1,6 +1,6 @@ # ADR-048: Lift viewpoint policy — window-relative viewpoints, rig gains, axis mode and ease-back are runtime policy -**Status:** Accepted (2026-10-08) · Phase 1 implemented · spec: +**Status:** Accepted (2026-10-08) · Phase 1 implemented · Addendum A (app rig + auxiliary depth, spec v3) implemented · spec: [XR_DXR_lift.md §4.1](../specs/extensions/XR_DXR_lift.md#41-viewpoint-frame-and-policy-spec-v2-adr-048) · extends [ADR-042](ADR-042-vendor-2d3d-conversion-supersedes-default.md) · applies [ADR-012](ADR-012-window-relative-kooima-projection.md) to lifted content · same split of labour as @@ -125,9 +125,79 @@ service's environment forces recentering off for every stream, for A/B compariso ## Later phases (not decided here) -- Accept a full `XrCameraRigDXR` / `XrDisplayRigDXR` on lift options so lifted frames and - app-rendered 3D share one camera computed with the `xrLocateViews` math. +- ~~Accept a full `XrCameraRigDXR` / `XrDisplayRigDXR` on lift options~~ — decided in + Addendum A (A1). - A metric, off-axis camera input on the vendor module (eye positions in metres relative to a screen plane of given size plus a depth scale), replacing today's dimensionless viewpoints. -- Metric depth as an auxiliary output aligned to the rigs. +- ~~Metric depth as an auxiliary output aligned to the rigs~~ — decided in Addendum A (A2). - Lift in-process (D3D12) for engine apps. + +## Addendum A (2026-10-08): the app's rig drives the viewpoints; depth is an auxiliary output + +**Status:** Accepted · XR_DXR_lift spec v3 · +spec: [XR_DXR_lift.md §4.2–4.3](../specs/extensions/XR_DXR_lift.md#42-app-rig-drives-the-viewpoints-spec-v3-adr-048-addendum-a) + +An app that mixes its own rendered 3D with lifted 2D content (a 3D scene around a lifted video +panel, a depth-aware cursor over lifted video) needs both to share one camera and one metric +space. Phase 1 gave lifted content window-relative viewpoints and a runtime-owned policy; this +addendum lets the app hand the runtime **its own rig**, and lets the module hand back **depth**. + +### A1. An app rig is computed with the `xrLocateViews` math — never re-implemented + +An `XrDisplayRigDXR` or `XrCameraRigDXR` (`XR_DXR_view_rig`) chained on `XrLiftOptionsDXR` or +`XrWeaveRectLiftDXR` makes the runtime derive the TRACKED lift viewpoints by running **the same +shared rig core `xrLocateViews` runs** (displayxr-common, `dxr_view_math.h`), with the lifted +rect as the screen. The display rig's eyes come straight out of `dxr_display3d_compute_views`. +The camera rig's eyes are the physical viewer whose Kooima frustum onto the rect equals the +camera's off-axis frustum (`E = Z0·(l·invd + ẑ)`, `Z0 = rect_h / 2·tan(vfov/2)`) — the only +local step, and it is checked against the shared core frustum by frustum in +`tests_aux_lift_rig_depth`. Why not a separate lift camera model: two code paths for "the eyes +this rig sees" would drift, and the whole point is that the app's 3D and the lifted pixels agree. + +The rig replaces the policy's ipd / parallax factors (the rig has its own); the axis mask, +ease-back and clamp still shape the viewer before the rig maps it, so the Phase 1 defaults keep +working for an app that only wants its rig's scale. An app that wants the located eyes exactly +chains XYZ + recentering OFF. The rig rides with the options (like explicit viewpoints), matching +`XR_DXR_view_rig`'s own per-call rule. EXPLICIT viewpoints are now accepted on lifted weave rects: +the Phase 1 refusal only existed because nothing could consume them there. + +The DP is told the viewpoints' origin (`viewpoint_source`: TRACKED / EXPLICIT / DISPLAY_RIG / +CAMERA_RIG) and the reference distance (`nominal_z_m`), appended to `xrt_dp_lift_params` +(`XRT_DP_LIFT_HAS_APP_RIG`). Rig viewpoints are the app's eyes, so the plug-in reproduces them as +given — no extra view gain. + +### A2. Depth is an auxiliary output of SBS / NVIEW streams, from the same inference + +A separate DEPTH stream runs a separate conversion, so its depth need not belong to the frame +whose views are woven. Instead an SBS / NVIEW stream created with `XrLiftDepthRequestDXR` +returns, with each result, the depth of **that** conversion (`XrLiftDepthResultDXR`): a texture +behind the views' own fence, its encoding and units, the module's intrinsics, the convergence +depth it placed on the screen, the viewpoint it refers to, and **`depthToDisplay`**. + +`depthToDisplay` is runtime policy, derived — not reported by the vendor: the source image fills +the lifted rect seen from the viewpoint midpoint `C`, and the convergence depth `dc` lies on the +screen plane, so a texel `(un, vn)` of depth `d` lands at `R + C + (P(un, vn) − C)·d/dc`. That is +linear in `(un·d, vn·d, d, 1)`: one 4×4. It is honest about what it is — the **display** geometry +of the woven result, the same perspective as the echoed eyes — and the intrinsics stay available +separately for the **camera** geometry. With RELATIVE depth the runtime reports neither, rather +than inventing a scale. + +DP contract (ADR-020, append-only, no ABI bump): `xrt_dp_lift_caps` += `aux_outputs`, +`aux_depth_semantics`; `xrt_dp_lift_stream_info` += `aux_outputs`, `aux_depth_format`; a new +`struct xrt_dp_lift_depth`; and D3D11 DP slot 33 **`lift_get_depth`** +(`XRT_DP_D3D11_HAS_LIFT_DEPTH`), called right after a successful `lift_convert` of a stream that +asked for depth. A slot rather than an out-struct on `lift_convert`: `lift_convert`'s signature +shipped and is frozen, and a separate call keeps depth optional per stream and per plug-in (an +absent slot reads as "no depth", which is exactly the old-plug-in case). + +### Consequences + +- An app renders its 3D with the echoed eyes (or its own rig) and the lifted picture lines up; + with metric depth it can occlude and point at lifted content in display metres. +- Full fidelity needs a metric, off-axis camera input in the vendor module (the module's own + camera today is an oblique shear around a convergence plane, not a pinhole); until then the + runtime's rig viewpoints reach the module through the same dimensionless translation as Phase 1. +- Depth quality and "same inference" are vendor capabilities, reported per result; the runtime + never synthesizes depth. +- Spec v3 (not folded into v2): v2 can ship on its own, and an app keys these features on + `SPEC_VERSION >= 3`. diff --git a/docs/reference/xrt_plugin_iface.md b/docs/reference/xrt_plugin_iface.md index b6b013ffb0..84767cfa97 100644 --- a/docs/reference/xrt_plugin_iface.md +++ b/docs/reference/xrt_plugin_iface.md @@ -511,7 +511,20 @@ struct xrt_dp_lift_params { uint32_t struct_size; float convergence; /* [0,1] re float baseline_m; /* first-to-last viewpoint distance; 0 = none */ uint32_t axis_mode; /* 1 X, 2 XY, 3 XYZ: components that follow the viewer */ float max_offset_m; /* x/y clamp the runtime applied; 0 none */ - uint32_t viewpoint_frame; /* 0 DISPLAY (panel centre), 1 RECT (lifted rect centre) */ }; + uint32_t viewpoint_frame; /* 0 DISPLAY (panel centre), 1 RECT (lifted rect centre) */ + /* appended, XRT_DP_LIFT_HAS_APP_RIG (ADR-048 Addendum A): */ + uint32_t viewpoint_source; /* 0 TRACKED, 1 EXPLICIT, 2 DISPLAY_RIG, 3 CAMERA_RIG */ + float nominal_z_m; }; +/* appended to caps / stream_info, XRT_DP_LIFT_HAS_AUX_DEPTH: caps.aux_outputs + aux_depth_semantics, + stream_info.aux_outputs (XRT_DP_LIFT_AUX_DEPTH) + aux_depth_format (DXGI hint) */ +struct xrt_dp_lift_depth { uint32_t struct_size; uint32_t format; /* R32_FLOAT / R16_FLOAT */ + void *resource; /* DP-owned, valid until the next lift_convert on the stream */ + uint32_t width, height, units /* 0 relative, 1 metric */, encoding /* 0 linear, 1 inverse */; + float value_scale, value_offset; /* d = scale * sample + offset */ + uint32_t source_width, source_height; + float focal_x_px, focal_y_px, principal_x_px, principal_y_px; /* depth-map px */ + float near_depth, far_depth, convergence_depth; /* decoded units; <=0 unknown */ + uint32_t same_inference; uint64_t vendor_frame_id; }; /* xrt_display_processor_d3d11, slots 25..29 */ bool (*lift_get_caps)(xdp, struct xrt_dp_lift_caps *out); @@ -525,6 +538,8 @@ bool (*lift_convert)(xdp, uint64_t id, void *d3d11_context, void *input_resource bool (*lift_convert_blob)(xdp, uint64_t id, void *d3d11_context, void *input_resource, uint32_t w, uint32_t h, const struct xrt_dp_lift_params *p, uint32_t *out_format /* 1 PLY_3DGS, 2 SOG */, const void **out_bytes /* valid until the next call */, size_t *out_size); +/* slot 33, XRT_DP_D3D11_HAS_LIFT_DEPTH: the depth of the conversion lift_convert just returned */ +bool (*lift_get_depth)(xdp, uint64_t id, void *d3d11_context, struct xrt_dp_lift_depth *out); /* xrt_plugin_iface, appended */ xrt_dp_factory_d3d11_fn_t create_dp_d3d11_lift; /* a DP that serves ONLY the lift slots */ @@ -559,6 +574,16 @@ xrt_dp_factory_d3d11_fn_t create_dp_d3d11_lift; /* a DP that serves ONLY the lif says which ones are live), and use `max_offset_m` + the rect size to map onto your module's range. A `struct_size` that stops before these fields means panel-centred viewpoints (an older runtime). +- **App-rig viewpoints (ADR-048 Addendum A).** With `viewpoint_source` DISPLAY_RIG / + CAMERA_RIG the viewpoints are the app's own rig eyes (the `xrLocateViews` math, the rect as the + screen) — the eyes the app renders its 3D from. Reproduce them as given: no extra view gain. +- **Auxiliary depth (ADR-048 Addendum A).** A stream whose `stream_info.aux_outputs` has + `XRT_DP_LIFT_AUX_DEPTH` gets a `lift_get_depth` call right after each successful + `lift_convert`. Return that conversion's depth — ideally from the same inference + (`same_inference`) — as an R32F / R16F texture you own until the next convert, with its + encoding, units, intrinsics and `convergence_depth` (the decoded depth you put at zero + disparity). The runtime copies it out and derives the display transform itself; report, don't + align. Advertise it in `caps.aux_outputs` / `aux_depth_semantics`. No slot ⟹ no depth. - **Output layout.** SBS = two views side by side; NVIEW = `view_count` views in one row, view 0 leftmost; DEPTH = one channel (any single-channel or RGBA format — say which in `out_format`). The runtime weaves SBS/NVIEW results itself on the ordinary weave path diff --git a/docs/specs/extensions/XR_DXR_lift.md b/docs/specs/extensions/XR_DXR_lift.md index 16fa61d7d6..f1951b6374 100644 --- a/docs/specs/extensions/XR_DXR_lift.md +++ b/docs/specs/extensions/XR_DXR_lift.md @@ -3,11 +3,11 @@ | Field | Value | |---|---| | **Extension Name** | `XR_DXR_lift` | -| **Spec Version** | 2 | +| **Spec Version** | 3 | | **Extension Type** | Instance extension (service path — Windows/D3D11; advertised on every desktop platform, where a service without a module reports `supportedModes = 0`) | | **Header** | `src/external/openxr_includes/openxr/XR_DXR_lift.h` (canonical; auto-syncs to `displayxr-extensions`) | -| **Status** | Provisional (`1004999270–282` block, pending Khronos registry) | -| **Decision record** | [ADR-042](../../adr/ADR-042-vendor-2d3d-conversion-supersedes-default.md); viewpoint policy: [ADR-048](../../adr/ADR-048-lift-viewpoint-policy.md) | +| **Status** | Provisional (`1004999270–285` block, pending Khronos registry) | +| **Decision record** | [ADR-042](../../adr/ADR-042-vendor-2d3d-conversion-supersedes-default.md); viewpoint policy, app rig and auxiliary depth: [ADR-048](../../adr/ADR-048-lift-viewpoint-policy.md) (+ Addendum A) | | **Plug-in contract** | `src/xrt/include/xrt/xrt_dp_lift.h` + the lift slots of `xrt_display_processor_d3d11` (`XRT_DP_D3D11_HAS_LIFT`), [`xrt_plugin_iface.md` § lift](../../reference/xrt_plugin_iface.md#turning-2d-into-3d-the-lift-slots-adr-042-xr_dxr_lift) | ## 1. What it is @@ -147,7 +147,7 @@ viewer facing the panel centre (the same window-relative rule ADR-012 sets for K viewpoints are given in display space as before and are rebased the same way; nothing else is applied to them. -**Policy** (`XrLiftViewControlDXR`, TRACKED only). Chain it on `XrLiftOptionsDXR` (submit or +**Policy** (`XrLiftViewControlDXR`, TRACKED only; with an app rig see §4.2). Chain it on `XrLiftOptionsDXR` (submit or weave rect) or directly on `XrWeaveRectLiftDXR` (the direct chain wins). It is **sticky per stream**: the last one sent applies until another is sent. Omitted, the defaults apply: @@ -191,6 +191,106 @@ separation after the ipd factor), `axis_mode` and `max_offset_m`. The plug-in on units — it normalises by `baseline_m` rather than a fixed eye distance, and honours every component it is sent. +### 4.2 App rig drives the viewpoints (spec v3, ADR-048 Addendum A) + +An app that renders its own 3D next to (or around) lifted content wants both to share **one +camera**. Chain the same `XR_DXR_view_rig` descriptor it chains on `xrLocateViews` — an +`XrDisplayRigDXR` or an `XrCameraRigDXR` — on `XrLiftOptionsDXR` (submit or weave rect) or +directly on `XrWeaveRectLiftDXR` (the direct chain wins; a camera rig wins over a display rig, +as in `xrLocateViews`). The runtime then derives the TRACKED viewpoints with **the same +functions `xrLocateViews` runs** (displayxr-common's shared rig core, `dxr_view_math.h`), with +the lifted rect as the screen: + +| Rig | Lift viewpoints (rect-relative metres) | +|---|---| +| `XrDisplayRigDXR` | `dxr_display3d_compute_views(...).eye_display` with the rect as the screen and `virtualDisplayHeight` = the rect height (m2v = 1): the eyes after `ipdFactor` / `parallaxFactor` (steps 1a / 1b), scaled by `perspectiveFactor`. With m2v ≠ 1 these are exactly the located `eye_display / m2v`. | +| `XrCameraRigDXR` | the physical viewer whose Kooima frustum onto the rect **equals** the camera rig's frustum: with `l = m2v · (processed eye − (0, 0, nominal))` (the shared camera3d core's eye, identity pose), `invd = convergenceDiopters`, `ht = tan(verticalFov / 2)` and `Z0 = rect_h / (2 ht)`: **E = Z0 · (l · invd + (0, 0, 1))**. Proof: Kooima's `tan_right = (W/2 − Ex)/Ez` reduces to the camera's `(ht·aspect − lx·invd) / (1 + lz·invd)`; likewise for the other three edges. `convergenceDiopters` 0 (parallel cameras) collapses every view onto `(0, 0, Z0)` — no lift parallax, which is what parallel cameras see of a flat image. | + +The rig `pose` and `virtualDisplayHeight` only place the app's virtual world; they do not move +the physical viewpoints. The rig **replaces** `XrLiftViewControlDXR`'s `ipdFactor` / +`parallaxFactor`; its axis mask, recentering and clamp still apply, to the viewer **before** the +rig maps it (order: rebase → axis mask → recenter → clamp → rig). For viewpoints identical to +`xrLocateViews`' (to float tolerance — `tests_aux_lift_rig_depth` checks both rigs against the +shared core), chain `XrLiftViewControlDXR` with `XR_LIFT_AXIS_MODE_XYZ_DXR` and recentering OFF. + +The rig **rides with the options**, like EXPLICIT viewpoints: a submit (or lifted rect) that +carries `XrLiftOptionsDXR` without a rig clears it; one that carries no options keeps the +stream's last rig. EXPLICIT viewpoints, DEPTH and GAUSSIANS streams ignore it. The echo +(`XrLiftResultViewpointsDXR`) reports the rig's viewpoints, so the app renders from the very eyes +the lifted views were synthesized for. The DP is told where they came from in +`xrt_dp_lift_params::viewpoint_source` (`DISPLAY_RIG` / `CAMERA_RIG`; `XRT_DP_LIFT_HAS_APP_RIG`) +and must reproduce them as given — no extra view gain — plus `nominal_z_m`. + +**EXPLICIT viewpoints on lifted weave rects** are accepted since v3, with the same validation +and the same rect rebase as on a submit. + +### 4.3 Auxiliary depth (spec v3, ADR-048 Addendum A) + +An SBS / NVIEW stream can return the **depth map of the same inference** that produced its +views, so app content can occlude, intersect and point at lifted video in one metric space +(ADR-046 cursor depth and ADR-040's rear budget reason about it the same way). + +- **Ask** at creation: chain `XrLiftDepthRequestDXR` on `XrLiftStreamCreateInfoDXR` + (`preferredFormat`: `DXGI_FORMAT_R32_FLOAT` / `_R16_FLOAT` / 0). On a DEPTH or GAUSSIANS stream + it is `XR_ERROR_VALIDATION_FAILURE`. **Capability**: `XrLiftDepthPropertiesDXR` chained on + `XrLiftPropertiesDXR` (`auxDepthSupported`, `auxDepthSemantics`). A request the module cannot + serve still creates the stream; results then report `depthValid = XR_FALSE`. +- **Receive**: chain `XrLiftDepthResultDXR` on `XrLiftResultDXR`. `depthTexture` + `fence` + follow §5.1's export rule (first acquire that carries depth, and every depth reallocation); + the fence is the SAME object as the views' fence with the SAME `fenceValue` — one wait covers + both. `frameId` equals the result's; `sameInference` says the module produced depth and views + in one pass. +- **Decode** a sample `s`: `d = valueScale · s + valueOffset`; `depth = d` (LINEAR) or `1 / d` + (INVERSE), in `units` (METRIC = metres along the source camera's axis; RELATIVE = unitless, + larger = farther). `nearDepth` / `farDepth` bound it (0 = unknown). +- **Intrinsics** (`intrinsicsValid`, METRIC only): the source camera the module assumed, in + depth-map pixels — `focalLengthPx`, `principalPointPx` — place a texel `(u, v)` in the + camera's metric frame: `X = (u − cx)·depth/fx`, `Y = −(v − cy)·depth/fy`, `Z = −depth`. + +**`depthToDisplay` — the display alignment.** The woven result presents the source image on the +lifted rect, seen from the viewpoint midpoint `C` (= `viewpoint`, rect-relative — the midpoint of +the echoed viewpoints, or the straight-on viewer `(0, 0, nominal)` when none were known), with the +module's convergence depth `dc` (= `convergenceDepth`, the decoded depth it put at zero +disparity) **on the screen plane**. So a texel at normalised coordinates `(un, vn)` — texel +centre, `un = (u + 0.5)/width`, `vn = (v + 0.5)/height`, v down — sits on the rect at + + P = (W·(un − ½), H·(½ − vn), 0) (rect-relative; W × H = the rect in metres) + +and a texel of depth `d` lies on the ray from the viewpoint through `P`, at the fraction `d / dc` +of the way: + + X = R + C + (P − C) · d / dc (R = rectCenter, display space) + +`d = dc` lands on the rect (z = the screen plane), `d → 0` converges on the viewpoint, `d > dc` is +behind the screen. `X` is linear in `(un·d, vn·d, d, 1)`, so it is one column-major 4×4 +(`m[col·4 + row]`, output `w` = 1): + +| column | multiplies | x | y | z | w | +|---|---|---|---|---|---| +| 0 | `un·d` | `W/dc` | 0 | 0 | 0 | +| 1 | `vn·d` | 0 | `−H/dc` | 0 | 0 | +| 2 | `d` | `(−W/2 − Cx)/dc` | `(H/2 − Cy)/dc` | `−Cz/dc` | 0 | +| 3 | 1 | `Rx + Cx` | `Ry + Cy` | `Rz + Cz` | 1 | + +Content an app renders from the echoed eyes (or from its own rig, §4.2) therefore lines up with +the lifted picture: both are perspective images of the same points from the same viewpoint. The +transform is the *display* geometry (strength 1); the intrinsics describe the *camera* geometry — +the two coincide only when the module's field of view equals the rect's angular size from `C`. +`transformValid` needs METRIC units, `dc > 0` and a known rect; **RELATIVE depth reports +`transformValid` / `intrinsicsValid` XR_FALSE rather than inventing a scale.** Reference +implementation and tests: `src/xrt/auxiliary/util/u_lift_depth.{h,c}`, +`tests/tests_aux_lift_rig_depth.cpp`. + +**DP side.** The runtime passes the request in `xrt_dp_lift_stream_info::aux_outputs` +(`XRT_DP_LIFT_AUX_DEPTH`, + `aux_depth_format`), and right after each successful `lift_convert` +of such a stream calls the appended D3D11 slot **`lift_get_depth`** (`XRT_DP_D3D11_HAS_LIFT_DEPTH`), +which fills `struct xrt_dp_lift_depth` (`XRT_DP_LIFT_HAS_AUX_DEPTH`): the DP-owned R32F / R16F +texture (valid until the next convert), units, encoding + scale / offset, source size, focal / +principal point, near / far, `convergence_depth`, `same_inference`, the module's frame id. The +runtime copies it into the result's ring slot and then the per-stream depth export texture, +and derives `depthToDisplay` itself. Capability: `xrt_dp_lift_caps::aux_outputs` / +`aux_depth_semantics`. An older plug-in (no slot) simply returns no depth. + ## 5. Acquiring results ### 5.1 Textures (DEPTH / SBS / NVIEW) @@ -246,7 +346,8 @@ xrWeaveSubmitDXR(session, &in, &out); "The content of weave rect `rectIndex` is 2D — lift it before weaving." An `XrRect2Di` has no `next`, so the association is by index into `XrWeaveSubmitRectsDXR::rects`; up to `XR_WEAVE_SUBMIT_MAX_LIFT_RECTS_DXR` (8) rects per submit. The stream must be an SBS or NVIEW -stream of the same session; options on the weave path must use TRACKED viewpoints. +stream of the same session. Since v3 options on the weave path may use EXPLICIT viewpoints +(§4.2), and a rig may be chained on the rect or its options. What the caller draws: @@ -373,7 +474,7 @@ Mirrors XR_DXR_weave §4b. | Result | When | Session still usable? | |---|---|---| | `XR_LIFT_NOT_READY_DXR` (success) | no result newer than the last acquired | yes | -| `XR_ERROR_VALIDATION_FAILURE` | a struct out of contract (mode not one bit, bad viewCount, EXPLICIT without viewpoints, EXPLICIT on a weave rect, wrong acquire for the mode, bad `rectIndex`) | yes — caller bug, nothing sent | +| `XR_ERROR_VALIDATION_FAILURE` | a struct out of contract (mode not one bit, bad viewCount, EXPLICIT without viewpoints, `XrLiftDepthRequestDXR` on a DEPTH / GAUSSIANS stream, wrong acquire for the mode, bad `rectIndex`) | yes — caller bug, nothing sent | | `XR_ERROR_FEATURE_UNSUPPORTED` | in-process session; mode not supported by a READY module; no module | yes — permanent for that request | | `XR_ERROR_LIMIT_REACHED` | `maxStreams` reached | yes | | `XR_ERROR_SIZE_INSUFFICIENT` | blob capacity too small (latch kept) | yes | @@ -389,7 +490,8 @@ Mirrors XR_DXR_weave §4b. | latest-wins mailbox, drops, frame ids | the model, its backend, licensing, warm-up (reported as ACTIVATING) | | output ring copies, export textures, fences | an output valid until its next call | | priority scheduling across streams | nothing about scheduling | -| tracked eyes → explicit viewpoints, rebased to the lifted rect, with the viewpoint policy applied (ADR-048) | honouring explicit viewpoints over any tracker of its own; unit translation only (normalise by `baseline_m`, honour every component sent) | +| tracked eyes → explicit viewpoints, rebased to the lifted rect, with the viewpoint policy applied (ADR-048) — or derived from the app's rig with the `xrLocateViews` math (v3) | honouring explicit viewpoints over any tracker of its own; unit translation only (normalise by `baseline_m`, honour every component sent; rig viewpoints reproduced as given) | +| copying the auxiliary depth out, exporting it, the display transform (`depthToDisplay`) | returning the depth of the same inference + its metadata (`lift_get_depth`) | | weaving SBS / N-view results (ADR-007) | never weaving a lift result | | timestamps, latency, stats | reporting `typical_latency_ns` | @@ -440,6 +542,7 @@ order: runtime → extensions auto-sync → consumers. |---|---| | 1 | Initial: properties + states, streams (DEPTH / SBS / NVIEW / GAUSSIANS), non-blocking latest-wins submit, texture acquire (weave-style handles + fence), blob acquire (two-call latch), weave-rect lift chain, per-stream priority scheduling + stats, `focalPx`. | | 2 | Viewpoint policy (ADR-048, §4.1): viewpoints relative to the lifted rect / submitting window centre (was: the panel centre), `XrLiftViewControlDXR` (ipd / parallax factors, axis mode, offset clamp, recentering; defaults X + EASE_BACK 1 s / 2 s), `XrLiftResultViewpointsDXR` echo. | +| 3 | App camera + metric depth (ADR-048 Addendum A, §4.2–4.3): `XrDisplayRigDXR` / `XrCameraRigDXR` on `XrLiftOptionsDXR` / `XrWeaveRectLiftDXR` drive the TRACKED viewpoints with the `xrLocateViews` rig math; EXPLICIT viewpoints accepted on lifted weave rects; auxiliary depth for SBS / NVIEW streams — `XrLiftDepthRequestDXR`, `XrLiftDepthResultDXR` (texture + shared fence, units, encoding, intrinsics, convergence depth, viewpoint, `depthToDisplay`), `XrLiftDepthPropertiesDXR`. A separate version from 2 because v2 can ship on its own (#1864): an app tests `SPEC_VERSION >= 3` for these. | ## Probing on a Windows box — gotchas (first N0 run, 2026-09-25) diff --git a/docs/specs/extensions/index.json b/docs/specs/extensions/index.json index b2a6390abd..664a0df62a 100644 --- a/docs/specs/extensions/index.json +++ b/docs/specs/extensions/index.json @@ -85,7 +85,7 @@ "XR_DXR_lift": { "group": "rendering", "title": "2D-to-3D Conversion", - "summary": "Asynchronous access to a vendor's 2D-to-3D conversion module (depth, stereo, N-view, or photo-to-Gaussian-splats): submit 2D frames, acquire the latest converted result, or flag a weave rect as 2D so the service lifts it before weaving; the runtime owns the viewpoint policy (window-relative look-around, rig factors, axis mode, ease-back)" + "summary": "Asynchronous access to a vendor's 2D-to-3D conversion module (depth, stereo, N-view, or photo-to-Gaussian-splats): submit 2D frames, acquire the latest converted result, or flag a weave rect as 2D so the service lifts it before weaving; the runtime owns the viewpoint policy (window-relative look-around, rig factors, axis mode, ease-back), can derive the viewpoints from the app's own view rig, and returns metric depth from the same inference as an auxiliary output aligned to display space" }, "XR_DXR_win32_window_binding": { "group": "windowing",