Skip to content

Add kitless teleoperation and replay with SO101 support - #8788

Draft
hougantc-nvda wants to merge 21 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-live-recorded-teleop
Draft

hougantc-nvda wants to merge 21 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-live-recorded-teleop

Conversation

@hougantc-nvda

@hougantc-nvda hougantc-nvda commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Connect live Capture tracking and recorded MCAP input to kitless XR, including Play, Pause, Reset and shutdown. Use the teleoperation-owned OVXR adapter and bootstrap supplied by #8787; keep input and target-frame integration in isaaclab_teleop. Add the SO101 CPU OvPhysX example and its public physics selector.

Include the prerequisites used by these entrypoints: Newton graph capture on a nonblocking stream for concurrent renderer/input uploads, lifecycle handling, and CloudXR profile documentation. The workspace NumPy override is constrained to >=2.3,<2.4 for the selected cmeel and Numba dependency requirements; this resolution constraint applies across the workspace.

Builds on #8787.

The provider upgrade is tracked in #8627. Main's native-provider defaults remain unchanged; the draft stack requires the documented scoped dependency preparation.

This draft targets main and includes the earlier PRs. Use the incremental comparison to review this PR's own changes.

Type of change

  • New feature
  • Bug fix
  • Documentation update

Testing

The cumulative stack at c862100202befb1f37e8055644a000aa47711762 passed 749 Lab checks across 32 affected test files, including the three initially skipped cross-GPU cases on explicit dual-GPU retry. Five Kit integration launch tests failed because Isaac Sim is not installed in this kitless environment; their Kit behavior remains unverified. Two stale test fixtures were corrected in their owning PRs: #8790 now supplies explicit environment paths to stage export, and #8791 supplies the device in its Newton manager mock. Both corrected files passed on retry. The full Newton manager suite passed 116 checks, and the opt-in native renderer ownership-transfer harness passed both cases. Capture's OVXR/mirror suite passed 191 checks.

#8787 was also tested at its own boundary: 100 checks passed, 6 device-dependent skips. Formatting/changelog checks passed on every branch; the cumulative warning-as-error Sphinx build and locked dependency check passed. An independent Claude static review found no blocking issue in the adapter/PR split; it was not an exhaustive concurrency audit.

Runtime verification uses Python 3.12, CUDA bindings 13.3.1, OVRTX 0.6.0.1791947691, OVStage 0.3.0.1791986420 and OvPhysX 0.7.0.74035321. The reused Capture wheel was built from 5d454411fd3d9eb1c5d65693e44d985f9e46fe54; all 177 installed runtime files match Capture 95ba799fca650dd003fb5a2400f9ed4b1ed3cb2d, whose intervening changes are metadata/documentation only. No wheel rebuild or fresh installation is claimed. See the source-build preparation guide.

GPU-backed Omen IWER checks passed on five warm-cache scenes: GR1T2 CPU OvPhysX, G1 locomanipulation CPU OvPhysX, SO101 CPU OvPhysX, fixed-base G1 split-GPU Newton and GR1T2 split-GPU Newton. Both reconstructed eyes and task objects were inspected; first Play, meaningful simulated input, Reset and post-reset motion worked. PiP was inspected on both humanoids, including all three fixed-G1 feeds. SO101 clutch/released hold and gripper controls worked. G1 reconnect passed. All five applications exited normally when their owned desktop mirror was closed.

The host desktop was locked in this run. Mirror window creation, presented-frame counts and close/shutdown passed, but its on-screen image remains unverified on this revision. IWER view/PiP inspection is separate from desktop-mirror inspection.

Bounded AV1 video intervals measured 52.1 decoded frames/s for GR1T2 CPU, 49.8 for G1 CPU, 68.2 for SO101, 30.9 for fixed G1 Newton and 33.5 for GR1T2 Newton. No sampled frames were dropped; fixed G1 reported two short video freezes totaling 0.462 s. These are received/decoded video measurements, not headset FPS or a controlled before/after benchmark. Fixed G1 remained around 65.7 physics steps/s against its 200 Hz target. The default active control loop is unpaced; some other tasks run faster than wall time. No pacing or fidelity default changed. No runtime traceback or CUDA error was observed.

Three cross-GPU cases initially skipped under the default test-device mask subsequently passed with both GPUs explicitly selected, including actual camera rendering, target-device rigid/geometry publication and producer-storage retention through failure/close.

Warnings from dependencies and split-GPU camera selection remain. Fresh physical-headset acceptance, cold-cache startup, hosted CI, arm64, cube retention and full task success remain unverified. The earlier cold-attach timeout and intermittent first-Play recovery remain open. Earlier unlocked-desktop mirror acceptance and physical-headset acceptance are historical evidence, not fresh visual qualification of this revision.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

The warnings checkbox remains open while the dependency upgrade and full runtime checks are pending. Local tests still report warnings from dependencies.

@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team infrastructure labels Oct 23, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from 7ffe1b4 to 900da07 Compare October 23, 2026 19:43
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Declare SO101 IK OVPhysX and USD asset selection without replacing parent-tuned Kit/Newton presets. Restore the compatible NumPy override and lock; document source-build prerequisites and first-use runtime compilation.

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Declare the versioned schema provider on the scoped OVXR extra and public index. Keep registered-task selector checks independent of the downstream teleop package.

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Reconcile the consuming isaaclab-ov Stage requirement before locking and refresh its editable metadata with ordinary uv. Keep default provider pins unchanged in product source.

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Build the real clutch without the unsupported orientation_offset argument. Seed its engage-relative full pose from nominal asset FK and restore those joints on each IK reset, avoiding orientation jumps or randomized home mismatch.

Exercise real graph outputs through movement, release, re-engage and reset; retain joint-position task randomization and physics gains.

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Pass the existing absolute-grip calibration to isaacteleop1.4 only when its constructor explicitly supports it. Keep Capture1.7 engage-relative full-home behavior unchanged.

Exercise selected API semantics through real graph outputs and retain reset, optional dependencies and physics behavior.

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from e933899 to 4d7c54f Compare October 26, 2026 17:12
@hougantc-nvda hougantc-nvda changed the title Add live teleoperation and MCAP replay through OVXR Add kitless teleoperation and replay with SO101 support Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant