[hougantc-nvda]

Description

Allow live OVXR teleoperation to place physics and rendering on different GPUs, with explicit scene-publication placement. This optional path supports GPU-heavy workloads; basic single-GPU and CPU-physics teleoperation remain available.

• Teleop CLI: --device selects physics, --ovrtx_active_cuda_gpus selects rendering and --geometry_publication_device selects transfer destinations. The latter two require live --ovxr; MCAP replay does not expose them.
• Renderer: publication_buffers.py copies foreign-GPU rigid transforms and deformable points into reusable destination buffers. It retains producer storage and exact streams until consumption, and drains pending work before failure/close cleanup.
• CloudXR: compositor/encoder selection is separate. Verify physical GPU identities because visible-device settings can remap ordinals; installing two GPUs alone does not split the workload.

No automatic CPU staging is enabled. PCIe peer transfers require synchronized value-integrity checks and a supported IOMMU configuration; host preparation belongs to an administrator with idle GPUs. Application startup does not change drivers, IOMMU or host services. Physics fidelity and pacing defaults are unchanged.

See GPU placement and peer-transfer requirements. The PR also versions release/source-build setup guides, pinning the tested Lab 592b218 / Capture 95ba799 runtime pair without committing developer-local wheel paths or provider overrides.

Builds on #8790. This draft targets main and includes its predecessors; review the incremental diff.

Dependencies: compatible Capture OVXR packages and the provider upgrade in #8627. Native-provider defaults are unchanged; use the source-build setup until matching packages and metadata are released.

Type of change

• New feature
• Bug fix
• Documentation update

Testing

Results below cover the cumulative stack, rebased on main fa569a5501a36051ac855769f11f8c76f0ac0194 with Capture 95ba799fca650dd003fb5a2400f9ed4b1ed3cb2d. They are not independent full-suite results for each PR. Subsequent source changes preserve the tested executable Python syntax.

• 1,062 cases across 51 selected Lab/Kit files passed, with no failures or skips: ordinary rendering without XR/leases, publication/lifetime failures, GPU Newton, 52 Kit teleop startup cases, Kit headless and Scene UI/camera integration.
• Five IWER and physical-headset trials covered GR1T2 CPU/split-GPU Newton, G1 locomanipulation CPU, fixed-base G1 Newton and SO101 CPU. Humanoid PiP/mirror and shutdown were checked; SO101 used mirror only. Kit Franka XR passed stereo, motion, Reset, reconnect and shutdown.

Formatting, changelog, incremental warning-as-error Sphinx and the kitless dependency check passed. The clean Sphinx check was blocked by external-inventory DNS failures.

Open limits: cold attach over 300 seconds, intermittent GR1T2 first-Play/Reset recovery and SO101 cube retention remain unresolved; full task completion is not qualified. The separate Kit SDK audit has eight metadata mismatches. Full-repository, hosted-CI and arm64 coverage are not claimed; the incomplete Claude follow-up does not establish an exhaustive concurrency review.

The scope corrections preserve all final runtime Python files. Eight camera tests, an intermediate G1 camera-resolution check, exact reconstruction of the prepared dependency metadata, resolver/locked-install dry runs and a 274-package dependency check pass. No runtime or performance matrix was repeated. NumPy/provider overrides remain explicit local premerge preparation.

Performance samples

Matched warm-cache IWER runs used the same client, flags and fixed motion inputs. Median decoded-video rates:

Scene	Baseline → candidate (frames/s)
GR1T2 CPU	50.793 → 50.926
G1 CPU	43.637 → 49.812
SO101 CPU	62.270 → 68.910
Fixed-base G1 Newton	31.063 → 32.993
GR1T2 Newton	32.964 → 35.984

Sampled dropped frames were zero. An earlier unseeded GR1 CPU decrease of 5.873% did not reproduce with fixed inputs; a final representative run after Kit/import corrections measured 51.620 versus 50.793 frames/s. These bounded samples show no reproduced regression in the matched cases, not statistical equivalence, headset FPS or physics-rate qualification.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with uv run isaaclab --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

The warnings checkbox remains open: dependency warnings and the separate Kit metadata audit are unresolved.

[hougantc-nvda]

Clarification: two intended first-time Teamfood paths

Path A — after merge: prepare the documentation now assuming the importer dependency lands, then #8627, then Lab #8787–#8791, with matching Capture release artifacts installed by Lab through uv. Pending dependency/release timing is expected, not a newly discovered code defect.

Path B — before merge: colleagues can already fetch the published cumulative Lab/Capture PR sources, build Capture locally, and apply the scoped dependency/lock preparation in the published source guide. Both routes need complete first-time setup/agent instructions. Local build outputs and these documented consuming-checkout metadata edits are valid temporary preparation; they do not need to wait for the merged packaging or a separate bootstrap implementation. The Teamfood document and both downloadable agent guides now describe these two routes.

Read item A below as the final installation integration/automation work, not a blanket gate preventing the basic premerge CPU examples. The unpublished tested renderer delta and I4H publication uncertainty remain separate optional-feature qualifications. No source push or merge is claimed by this clarification.

---

Teamfood: publications needed for first-time colleagues

@hougantc-nvda — publication checklist from the independent Teamfood reproduction. @r-schmitt — item A needs coordination with the provider-upgrade ownership in #8627; this is not a request to replace global/Kit defaults with the evaluation pins.

GitHub readback: this PR remains open at c13f861b782d86015defb785de068f27c8d086e1; #8627 remains open at 7ae0cf8ebeabc7aa63186130c62ab801f948cbec. Paired Capture #1296 remains at f9215fa344d32dd8cca33f0b3d16868ac63c71a1. Updating a guide or posting this checklist does not publish the missing implementation.

Already published — do not redo: canonical OvPhysX task selectors, the NumPy/setup recipe corrections, and SO101 Capture-constructor compatibility are in the c13 checkpoint. The current source/isaaclab_teleop/docs/OVXR_RUNTIME.md documents the corrected manual preparation. No new Capture runtime-code defect requiring a push has been established.

• A. Portable installation selection/preparation — IsaacLab, this stack coordinated with Upgrade public OVRTX, OvPhysX, and OVStage releases #8627. Owning files: root pyproject.toml, source/isaaclab_ov/pyproject.toml, uv.lock, and source/isaaclab_teleop/docs/OVXR_RUNTIME.md. Publish a scoped supported selection or reviewed, version-controlled preparation command for the ovxr extra, compatible providers, explicit physx-usd-schemas/public-index mapping, matching workspace OVStage requirement, and compatible NumPy/packaging resolution. These consuming-checkout edits are currently local; the published workspace still declares OVStage 0.2.1.385922. Do not commit the developer-local Capture wheel path or overwrite unrelated/default Kit provider choices. A reproducible build from exact public Capture source is valid: include CMake >=3.24, patchelf, Python/build prerequisites and wheel build/install steps. If promising a download, supply an accessible immutable artifact and checksum.
  Close with: actual public commit/recipe/artifact readback, paired source/module/native-provider origins, ordinary resolver/locked install and clean dependency check from the documented starting metadata; preinstalled packages must not conceal omissions.

• B. Tested direct-GPU transform-publication delta — IsaacLab renderer, this stack. Owning file: source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py. Current lines 1279–1310 still publish omni:xform directly from matrices. The preserved transform-publication-fix.patch (SHA-256 7881c2bac04ac574803139b0d734c5d705b27f168fe65c1ee3c6e8176a9c6c9c) routes rigid matrices through _GeometryPublicationBuffers on geometry_publication_device, alongside cloth points. Retain source buffers and their exact producer streams until transfer/OVStage consumption completes; drain those streams before freeing storage on failure/close. Review and split out the diagnostic CPU-staging branch as appropriate; accepted runs used ISAAC_LAB_OVRTX_HOST_STAGING=0, and CPU staging is not the accepted final path.
  Close with: public head/diff proving this minimal behavior or a verified equivalent, plus rigid/cloth foreign-GPU, producer-stream lifetime, reset and failure/close regression coverage. All accepted full-demo runs retained the local delta. A full unmodified-renderer application run under identity mode was not performed; this is the missing tested delta, not proof that every alternative fails.

• C. Supported launch/preflight — IsaacLab teleoperation entry point and guide. Existing flags in scripts/environments/teleoperation/teleop_se3_agent.py and the split-GPU section of OVXR_RUNTIME.md are already public. Complete the reusable configuration/preflight instead of distributing the host-specific patched review harness: select physics/rendering/publication/CloudXR compositor-encoder consistently and show physical UUID/PCI identities, accounting for ordinal remapping. Document that this tested renderer defaults to the simulation CUDA GPU; two cards alone do not imply split placement. Include a small bidirectional direct-copy integrity check with initialized destinations, synchronization and value comparisons—successful CUDA completion/peer-capability flags missed corruption.
  Close with: public entry-point/preflight commit and literal command acceptance. Unsupported translated-IOMMU PCIe paths need an actionable administrator prerequisite, never automatic driver/IOMMU/service changes. GPU-only identity/passthrough was verified with global IOMMU enabled and no reboot; the experimental root transaction is not a colleague setup step.

• D. I4H tablecloth — conditional, not a basic-robot blocker. Upstream source references are in isaac-for-healthcare/i4h-workflows: arena/i4h_arena/envcfg/tablecloth/{g1_spread_tablecloth_env_cfg.py,g1_spread_tablecloth_teleop_env_cfg.py,cloth_physics.py}, embodiments/tablecloth.py and the tablecloth asset modules. The actual kitless adaptation's publication status is unverified; no owning port PR/public commit has been established. If advertised, provide an immutable accessible port/registration/launcher, current-API coupled MJWarp/VBD setup, imports, Pink joint ordering, XR anchors/cameras and v0.9.0/7cf86a1 asset acquisition manifest. Reconcile the final tested source closure; do not publish an earlier snapshot blindly.
  Close with: verified exact port source/asset/launcher links and scene checks. Otherwise omit or qualify this optional example. No expired-lease access, new lease or bulk reconstruction is requested.

• E. Colleague-ready verification — after the publications. Extend meaningful coverage in source/isaaclab_ov/test/test_ovrtx_renderer_contract.py and applicable teleop selector/install/native-transfer tests. Reuse the existing lane/caches and preserve local preparation and original failures. Demonstrate literal published commands from declared metadata without hidden patches/private paths. Inspect both reconstructed IWER eyes, task object poses, meaningful actions, Reset, reconnect and shutdown. Report source/lock/artifact IDs and actual outcomes; no additional clone/environment/build is needed merely to claim a first-time test.

F. Open limits, not inferred fixes: >300 s cold renderer attach and intermittent GR1 first-Play/Reset recovery remain unexplained. IOMMU translation is a host/runtime compatibility condition, not an Ubuntu-26-only source defect. The bounded I4H 20→5-substep result improved control updates about 3.30→11.43 Hz; it is not headset FPS, 30 Hz qualification, autonomous task success or equivalent contact fidelity. Do not silently change shipped fidelity to five substeps.

This conveys the missing publications through the existing owners/stack. It does not authorize a merge, blanket provider upgrade, host changes, or claim that code has been pushed.