Skip to content

Add the OV renderer lease and teleop-owned OVXR adapter - #8787

Draft
hougantc-nvda wants to merge 3 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-borrowed-renderer
Draft

hougantc-nvda wants to merge 3 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-borrowed-renderer

Conversation

@hougantc-nvda

@hougantc-nvda hougantc-nvda commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Connect kitless Isaac Lab teleoperation to Isaac Capture's OVXR session using the simulation's existing OVRTX renderer and OVStage. An explicit render lease lets Capture borrow those resources while Lab retains ownership and controls their lifetime.

The responsibilities are:

  • isaaclab_ov: owns the renderer, stage and scene publication. The generic lease serializes borrowed access without depending on XR or teleoperation.
  • isaaclab_teleop: implements the OVXR adapter in ov_libraries_xr.py and bootstrap in ovxr_session.py. It binds the environment, supplies XR render products, publishes scene/anchor updates and coordinates shutdown.
  • Isaac Capture: owns the OVXR session and render worker, and makes borrowed OpenXR handles available to the teleoperation input session.

Bootstrap reserves the lease before environment construction; the render worker starts afterward. Shutdown stops input consumers, closes the XR session and retires the lease before Lab destroys the renderer/stage. Failed retirement preserves resources for cleanup by their owning thread. The current lease supports one environment and a private rendering OVStage.

Lease/thread-transfer logic lives in render_lease.py; publication_buffers.py retains storage until native writes finish. Ordinary rendering stays in ovrtx_renderer.py, with existing public imports preserved. Without a lease, normal rendering continues; while resources are borrowed, conflicting camera rendering, scene writes and destruction are rejected.

See OVXR adapter ownership and lifecycle for details.

Dependencies: compatible Capture OVXR packages and the provider upgrade tracked in #8627. This PR leaves native-provider defaults unchanged. Until matching packages are released, use the source-build setup guide.

Type of change

  • New feature
  • Bug fix
  • Documentation update

Testing

Validation covers the cumulative stack, rebased on main fa569a5501a36051ac855769f11f8c76f0ac0194 with Capture 95ba799fca650dd003fb5a2400f9ed4b1ed3cb2d; it is not an independent full-suite result for this PR.

  • 1,062 cases across 51 selected Lab/Kit test files passed, with no failures or skips: ordinary rendering without XR/leases, publication/lifetime failures, GPU Newton, 52 Kit teleop startup cases, Kit headless and Scene UI/camera coverage.
  • Five IWER and physical-headset trials covered GR1T2 CPU/split-GPU Newton, G1 locomanipulation CPU, fixed-base G1 Newton and SO101 CPU. Humanoid PiP/mirror and shutdown were checked; SO101 used mirror only. Kit Franka XR passed stereo, motion, Reset, reconnect and shutdown.
  • Formatting, changelog, incremental warning-as-error Sphinx and the kitless dependency check passed. The clean Sphinx check was blocked by external-inventory DNS failures.

Open limits: cold attach over 300 seconds, intermittent GR1T2 first-Play recovery and slippery SO101 cubes remain unresolved; object retention/full task completion are not qualified. The separate Kit SDK audit has eight metadata mismatches. Full-repository, hosted-CI and arm64 coverage are not claimed. See #8791 for matched performance results and detailed validation limits.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

The warnings checkbox remains open: dependency warnings and the separate Kit metadata audit are unresolved.

@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Oct 23, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-borrowed-renderer branch from 9e86c66 to f63b4c2 Compare October 26, 2026 17:12
@hougantc-nvda hougantc-nvda changed the title Enable OVXR to use Isaac Lab's renderer and scene Expose the OV renderer and scene through a borrowed-renderer lease Oct 26, 2026
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 26, 2026
@hougantc-nvda hougantc-nvda changed the title Expose the OV renderer and scene through a borrowed-renderer lease Add the OV renderer lease and teleop-owned OVXR adapter Oct 26, 2026
Consolidate the reviewed PR boundary before rebasing onto current main.
Previous cumulative head: 12208c4

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-borrowed-renderer branch from 12208c4 to 6afe08b Compare October 27, 2026 16:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant