Mixed Reality interaction with Synth humanoids on Meta Quest. Physics-based hand tracking, room-scale environment setup, and passthrough rendering.
- Physics Hand Tracking — MuJoCo-bound hand bodies driven by OVR hand tracking. Push, grab, pull, and interact with the Synth using your real hands.
- Room Integration — MRUK-powered room setup with physics colliders for walls, floor, ceiling, and furniture anchors. The Synth interacts with your physical room layout.
- Smooth Locomotion — Controller-based walk and rotate with configurable speed.
- Passthrough Rendering — Occluder materials for room surfaces, PTRL (Passthrough Receive Light) shader for realistic lighting on the Synth.
- Ambient Light Estimation — Passthrough camera-based lighting estimation for harmonized rendering in mixed reality.
synth-vr is part of a three-package architecture for creating, training, and interacting with physics-simulated humanoids:
| Package | Role | |
|---|---|---|
| synth-core | Humanoid creation, MuJoCo physics, skill architecture | Required |
| synth-training | On-device reinforcement learning via TorchSharp SAC | Optional |
| synth-vr (this repo) | Mixed reality interaction on Meta Quest | — |
synth-core provides the physics body and motor system. synth-vr adds Meta Quest hand tracking, MRUK room integration, and passthrough rendering so the Synth lives in your physical space. Optionally add synth-training to enable on-device reinforcement learning — the Synth trains live on Quest while you interact with it.
- Unity 6000.x or later
- synth-core package
- MuJoCo Unity plugin (
org.mujoco) — via arghyasur1991/mujoco fork (synth-patchesbranch) - Meta XR SDK packages (v85+):
com.meta.xr.sdk.interaction.ovrcom.meta.xr.mrutilitykit
- synth-training — Add on-device reinforcement learning so the Synth trains directly on Quest while you interact with it in your room. Without this package, synth-vr provides physics interaction only (no learning).
Add to Packages/manifest.json:
{
"dependencies": {
"com.genesis.synth.vr": "https://github.com/arghyasur1991/synth-vr.git",
"com.genesis.synth": "https://github.com/arghyasur1991/synth-core.git",
"org.mujoco": "https://github.com/arghyasur1991/mujoco.git?path=unity#synth-patches"
}
}Meta XR SDK packages are installed separately via the Unity Package Manager or Meta's setup tools.
Use the built-in wizard to configure a VR scene: Synth > Setup > VR Scene Wizard
Open Meta > Tools > Building Blocks and add these to your scene:
| Block | Purpose |
|---|---|
| Camera Rig | OVRManager + OVRCameraRig with tracking space and hand/eye anchors |
| Hand Tracking (x2) | OVRHand + OVRSkeleton on left and right hand anchors |
| Passthrough | OVRPassthroughLayer for mixed reality passthrough |
| OVRInteractionComprehensive | Ray/grab/poke interactions (optional) |
The wizard validates Building Blocks, then configures everything else automatically:
- Conflicting objects — disables stray MainCamera, Ground/Plane, Haptics, and controller visuals
- MuJoCo scene — creates
MjScene,Global Settings(withMjGlobalSettings), andQuestPerformanceManager - Synth-VR components — adds
SceneMeshManager(room integration) andPlayerHandBodies(physics hands, auto-wired to OVR skeletons) - Lighting — creates a directional light if missing
- URP settings — applies Quest-optimized render pipeline settings (Render Scale 1.0, MSAA 4x, HDR off, Shadow Distance 50, SRP Batcher on)
- Permissions — fixes OculusProjectConfig for hand tracking, passthrough, scene support, and anchor support
Click Setup Everything to run all steps at once, or use per-section Fix buttons for granular control. All changes support Undo.
Add a single Synth to the scene via synth-core. Note: only one active Synth per scene is supported.
| Feature | OculusProjectConfig | OpenXR Feature | Android Permission |
|---|---|---|---|
| Hand Tracking | handTrackingSupport: 1 |
Hand Tracking Subsystem | Auto-injected |
| Passthrough | insightPassthroughEnabled: 1 |
Meta Quest: Camera | android.permission.CAMERA (auto) |
| Room Mesh (MRUK) | sceneSupport: 1 |
Meta Quest: Meshing / Planes | com.oculus.permission.USE_SCENE (auto) |
| Spatial Anchors | anchorSupport: 1 |
Meta Quest: Anchors | com.oculus.permission.USE_SCENE (auto) |
The wizard auto-fixes OculusProjectConfig settings. OpenXR features must be enabled manually in Project Settings > XR Plug-in Management > OpenXR.
synth-vr/
├── Runtime/
│ ├── Hands/ PlayerHandBodies (MuJoCo hand tracking)
│ ├── Locomotion/ PlayerLocomotion (smooth walk/rotate)
│ ├── SceneSetup/ SceneMeshManager (MRUK room integration)
│ ├── Performance/ QuestPerformanceManager (CPU/GPU levels)
│ ├── Lighting/ AmbientLightEstimator
│ ├── Shaders/ PTRLWithDepth
│ └── Resources/ InvisibleOccluder, PTRLHighlightsAndShadows materials
└── Editor/
├── VRSceneSetupWizard.cs (interactive scene setup)
└── XRSetupEditor.cs (XR plugin configuration)
| Component | Purpose | Where |
|---|---|---|
PlayerHandBodies |
Creates MuJoCo bodies from OVR hand tracking for physics interaction | OVRCameraRig |
SceneMeshManager |
Sets up room geometry, furniture anchors, and physics colliders from MRUK | SceneMesh object |
QuestPerformanceManager |
Requests CPU/GPU performance levels from Quest OS | Global Settings |
PlayerLocomotion |
Smooth walk and rotate via controller thumbstick | OVRCameraRig |
AmbientLightEstimator |
Estimates ambient lighting from passthrough camera for realistic rendering | Runtime (auto) |
- Voice interaction — Speech-to-intent pipeline for verbal commands and conversational interaction with the Synth
- Haptic feedback — Controller and hand haptics driven by MuJoCo contact forces
- Spatial audio — 3D audio anchored to the Synth for immersive presence
- Better light harmonization — Improved passthrough lighting estimation for more realistic Synth rendering in your room
- Better occlusion — Depth-accurate occlusion between virtual Synth and real-world objects
- Synth sees your world — Feed pre-scanned Gaussian splat of the room into Synth eye cameras so the agent perceives the real environment
Apache-2.0 — see LICENSE for details.