Add a desktop mirror to OVXR teleoperation

README.md

@@ -91,10 +91,10 @@ Third-party materials may carry their own licenses.
 ### Separately retrieved optional materials
 
 The optional `isaaccapture[ovxr]` extra retrieves OVXR and its OVRTX/OVStage
-dependencies.
+dependencies. The separate `isaaccapture[desktop-mirror]` extra retrieves OVUI.
 These packages are downloaded separately, rather than bundled in the Isaac Capture
 wheel. Their licenses apply independently of Isaac Capture's Apache-2.0 license.
-OVXR, OVRTX and OVStage are NVIDIA proprietary software.
+OVXR, OVRTX and OVStage are NVIDIA proprietary software. OVUI is Apache-2.0.
 See [package sources and license details](docs/source/getting_started/ovxr.rst#optional-packages-and-licenses).
 Review the applicable package licenses and terms before installing or using these
 optional dependencies.

docs/source/getting_started/ovxr.rst

@@ -37,6 +37,9 @@ from the Isaac Capture wheel. These packages retain their own licenses:
   `OVRTX <https://pypi.org/project/ovrtx/>`__ and
   `OVStage <https://pypi.org/project/ovstage/>`__ are NVIDIA proprietary software.
   Review the license included with each distribution before use.
+* `OVUI <https://pypi.org/project/ovui/>`__ is Apache-2.0 and is installed by
+  the independent ``desktop-mirror`` extra. Review its package metadata for
+  supported Python versions and platforms.
 
 CloudXR native libraries are bundled in Isaac Capture wheels and remain subject
 to their separate NVIDIA license and CloudXR EULA acceptance.
@@ -72,6 +75,25 @@ Use ``CUDAToolkit_ROOT`` to select a CUDA Toolkit installation and ``VULKAN_SDK`
 to help locate a Vulkan SDK. The GPU driver and Vulkan runtime loader are required
 on the host.
 
+Optional desktop mirror
+-----------------------
+
+The desktop mirror requires **Python 3.12 or newer**. Isaac Capture's core
+continues to support Python 3.11; on that version the extra does not install OVUI
+and the desktop mirror is unavailable. Use a Python 3.12+ environment and select
+both extras with ``isaaccapture[ovxr,desktop-mirror]`` in the installation
+requirement or the application's optional dependencies to include the OVUI
+mirror. The mirror requires CUDA 13 Python bindings, a compatible NVIDIA driver
+as described above, and a desktop display associated with a CUDA device.
+The native camera helper is required only when camera panels are also enabled.
+
+Initialize the renderer before starting the mirror. OVUI uses its default Vulkan
+backend; no ``OMNIUI_BACKEND`` setting is required. Capture does not change the
+process-wide backend. GPU images stay on their producing CUDA device, and OVUI
+copies them into its Vulkan image. The mirror requires the active OVUI backend
+to be Vulkan. Remove conflicting ``OMNIUI_BACKEND`` settings before OVUI starts.
+This selection does not change headset rendering.
+
 Use from another application
 ----------------------------
 
@@ -83,9 +105,11 @@ resolved its render-product paths. Capture does not depend on an Isaac Lab task
 or scene configuration. ``camera_paths`` is required and must contain both stereo
 cameras. Set ``up_axis="Y"`` for a Y-up Stage; the default is ``"Z"``.
 
-``CameraPanelConsumer`` implements the ``OvxrFrameConsumer`` lifecycle. Configure panels with a tuple of
+``CameraPanelConsumer`` and ``OwnedDesktopMirror`` implement the same
+``OvxrFrameConsumer`` lifecycle. Configure panels with a tuple of
 ``CameraPanelConfig`` values, each pairing a render-product path with its placement
-and image bound.
+and image bound. Configure the mirror's camera choices with ``MirrorProduct``
+values. Either consumer can be enabled independently.
 
 Consumers receive ``FrameOutputs`` on the XR owner thread. Use ``latest_color``
 to select an output, then ``copy_host_rgba`` or ``map_gpu`` to consume it without

pyproject.toml

@@ -54,6 +54,11 @@ ovxr = [
     "ovxr==0.1.0.5842",
     "cuda-bindings>=13,<14",
 ]
+desktop-mirror = [
+    # OVUI requires 3.12; core Capture continues to support 3.11.
+    "ovui==0.3.0; python_version >= '3.12'",
+    "cuda-bindings>=13,<14",
+]
 retargeters = [
     "dex-retargeting>=0.4.6,<0.6.0",
     "scipy>=1.15.0",

scripts/check_camera_quad_wheel.py

@@ -1,7 +1,7 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-"""Check optional camera-helper delivery in an actual project wheel.
+"""Check camera-helper and desktop-mirror delivery in an actual project wheel.
 
 Run against wheels built with BUILD_OVXR_CAMERA_QUADS ON and OFF, including
 ON -> OFF -> ON in the same build directory to detect stale packaged data.
@@ -26,6 +26,15 @@ def main():
         prefix + "libisaaccapture_camera_panels.so",
     }
     with ZipFile(args.wheel) as wheel:
+        mirror_modules = {
+            "isaaccapture/ovxr_session_manager/" + name + ".py"
+            for name in ("owned_mirror", "desktop_mirror", "gpu_output")
+        }
+        missing = mirror_modules.difference(wheel.namelist())
+        if missing:
+            raise RuntimeError(
+                f"Wheel is missing desktop mirror modules: {sorted(missing)}"
+            )
         members = {
             name
             for name in wheel.namelist()

src/compat/pyproject.toml

@@ -23,6 +23,7 @@ Homepage = "https://github.com/NVIDIA/IsaacCapture"
 dev = ["isaaccapture[dev]>=1.6.dev0,<1.9"]
 ui = ["isaaccapture[ui]>=1.6.dev0,<1.9"]
 ovxr = ["isaaccapture[ovxr]>=1.6.dev0,<1.9"]
+desktop-mirror = ["isaaccapture[desktop-mirror]>=1.6.dev0,<1.9"]
 retargeters = ["isaaccapture[retargeters]>=1.6.dev0,<1.9"]
 retargeters-lite = ["isaaccapture[retargeters-lite]>=1.6.dev0,<1.9"]
 grounding = ["isaaccapture[grounding]>=1.6.dev0,<1.9"]

src/core/python/CMakeLists.txt

@@ -84,6 +84,7 @@ add_custom_target(python_package ALL
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-dev.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-ui.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
+    COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-desktop-mirror.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-ovxr.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-cloudxr.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"
     COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_CURRENT_SOURCE_DIR}/requirements-retargeters.txt" "${CMAKE_BINARY_DIR}/python_package/$<CONFIG>/"

src/core/python/pyproject.toml.in

@@ -31,6 +31,7 @@ dynamic = ["dependencies", "optional-dependencies"]
 dependencies = {file = ["requirements.txt"]}
 optional-dependencies.dev = {file = ["requirements-dev.txt"]}
 optional-dependencies.ui = {file = ["requirements-ui.txt"]}
+optional-dependencies.desktop-mirror = {file = ["requirements-desktop-mirror.txt"]}
 optional-dependencies.ovxr = {file = ["requirements-ovxr.txt"]}
 optional-dependencies.retargeters = {file = ["requirements-retargeters.txt"]}
 optional-dependencies.retargeters-lite = {file = ["requirements-retargeters-lite.txt"]}

src/core/python/requirements-desktop-mirror.txt

@@ -0,0 +1,3 @@
+# Optional standalone desktop mirror (Python 3.12+)
+ovui==0.3.0; python_version >= "3.12"
+cuda-bindings>=13,<14

src/python/isaaccapture/ovxr_session_manager/__init__.py

@@ -13,6 +13,7 @@
     OvxrSession,
     StageOrdinalLedger,
 )
+from .desktop_mirror import DesktopMirrorConfig, DesktopMirrorStatus
 
 __all__ = [
     "OvxrSessionBinding",
@@ -21,6 +22,8 @@
     "OvxrFrameConsumer",
     "OvxrSession",
     "StageOrdinalLedger",
+    "DesktopMirrorConfig",
+    "DesktopMirrorStatus",
 ]
 
 from .camera_panels import CameraPanelConfig, CameraPanelConsumer

src/python/isaaccapture/ovxr_session_manager/desktop_mirror.py

@@ -0,0 +1,131 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Shared configuration, cadence and copy completion for the owned desktop mirror."""
+
+from __future__ import annotations
+
+import math
+from collections import deque
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class DesktopMirrorConfig:
+    """Configuration for the optional, application-side OVUI mirror."""
+
+    max_fps: float = 30.0
+    window_width: int = 960
+    title: str = "Isaac Teleop — render products"
+    initial_status_text: str = ""
+
+    def __post_init__(self) -> None:
+        if not math.isfinite(self.max_fps) or self.max_fps <= 0.0:
+            raise ValueError("desktop mirror max_fps must be finite and positive")
+        if self.window_width <= 0:
+            raise ValueError("desktop mirror window_width must be positive")
+        if not self.title:
+            raise ValueError("desktop mirror title must be non-empty")
+
+
+@dataclass(frozen=True)
+class DesktopMirrorStatus:
+    """Thread-safe snapshot of mirror lifecycle and latest-frame behavior."""
+
+    thread_id: int | None = None
+    window_opened: bool = False
+    closed: bool = False
+    error: str | None = None
+    presented_frames: int = 0
+    last_sequence: int = 0
+    main_thread_fps: float = 0.0
+    status_text: str = ""
+    render_thread_fps: float = 0.0
+
+
+class _CadenceSampler:
+    """Measure recent host and render cadence from monotonic samples."""
+
+    def __init__(self, window_s: float = 1.0) -> None:
+        self._window_ns = int(window_s * 1_000_000_000)
+        self._host_samples: deque[int] = deque()
+        self._render_samples: deque[int] = deque()
+
+    def record_host_frame(self, now_ns: int) -> None:
+        self._host_samples.append(now_ns)
+        while (
+            len(self._host_samples) > 2
+            and self._host_samples[1] <= now_ns - self._window_ns
+        ):
+            self._host_samples.popleft()
+
+    def record_render_frame(self, now_ns: int) -> None:
+        self._render_samples.append(now_ns)
+        while (
+            len(self._render_samples) > 2
+            and self._render_samples[1] <= now_ns - self._window_ns
+        ):
+            self._render_samples.popleft()
+
+    @staticmethod
+    def _event_rate(samples: deque[int] | list[int]) -> float:
+        if len(samples) < 2:
+            return 0.0
+        elapsed_s = (samples[-1] - samples[0]) / 1_000_000_000
+        return 0.0 if elapsed_s <= 0.0 else (len(samples) - 1) / elapsed_s
+
+    def rates(self, now_ns: int | None = None) -> tuple[float, float]:
+        def recent_rate(samples: deque[int]) -> float:
+            if now_ns is not None and (
+                not samples or now_ns - samples[-1] > self._window_ns
+            ):
+                return 0.0
+            return self._event_rate(samples)
+
+        return recent_rate(self._host_samples), recent_rate(self._render_samples)
+
+
+class _InputCopyCompletion:
+    """Fence OVUI input uploads, not its unrelated output-streaming API.
+
+    Confirm the source is reusable even after an upload exception. Only the UI
+    worker waits; OVXR skips captures when its optional slots are leased.
+    """
+
+    def __init__(self, device: int) -> None:
+        from cuda.bindings import runtime
+
+        self._cuda = runtime
+        self._device = device
+        display_device = self._check(runtime.cudaGetDevice())
+        try:
+            self._check(runtime.cudaSetDevice(device))
+            self._event = self._check(
+                runtime.cudaEventCreateWithFlags(runtime.cudaEventDisableTiming)
+            )
+        finally:
+            self._check(runtime.cudaSetDevice(display_device))
+
+    @staticmethod
+    def _check(result):
+        if int(result[0]):
+            raise RuntimeError(f"OVUI input-copy completion failed: CUDA {result[0]}")
+        return result[1] if len(result) > 1 else None
+
+    def wait(self) -> None:
+        # Vulkan may select its import device while performing its blocking copy.
+        display_device = self._check(self._cuda.cudaGetDevice())
+        try:
+            self._check(self._cuda.cudaSetDevice(self._device))
+            self._check(
+                self._cuda.cudaEventRecord(self._event, self._cuda.cudaStreamLegacy)
+            )
+            self._check(self._cuda.cudaEventSynchronize(self._event))
+        finally:
+            # Subsequent UI frames must retain the provider's import device.
+            self._check(self._cuda.cudaSetDevice(display_device))
+
+    def close(self) -> None:
+        if self._event is not None:
+            self._check(self._cuda.cudaEventDestroy(self._event))
+            self._event = None