Go2W Sim2Sim is a MuJoCo-based simulation and policy playback workspace for the Go2W robot. The main executable is lab2mj, which loads the Go2W MJCF scene, runs ONNX/LibTorch policies, visualizes the robot and RayCaster camera, supports interactive perturbations, and records synchronized multi-view videos.
mujoco/C++/ C++ simulators and policy playback apps
mujoco/C++/mj_env.cpp lab2mj environment and video recording logic
mujoco/C++/sim2sim_env.cpp shared policy/control logic
robot/go2w_description/mjcf/ Go2W MJCF robot and scenes
robot/go2w_description/mjcf/scene_parkour.xml
default parkour scene used by lab2mj
policy/ policy checkpoints and deployment configs
utils/mujoco_thread/ MuJoCo window, input, camera, and recording backend
utils/mujoco_ray_caster/ external ray caster dependency
ros2/ ROS2 integration workspace
The C++ simulator currently expects a Linux desktop environment with OpenGL/GLFW support.
Required packages and libraries:
sudo apt-get install libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev \
libudev-dev joystick libjsoncpp-dev libopencv-devROS2-related dependencies are used by the real2sim target:
sudo apt-get install ros-${ROS_DISTRO}-rclcpp ros-${ROS_DISTRO}-std-msgs \
ros-${ROS_DISTRO}-sensor-msgs ros-${ROS_DISTRO}-cv-bridgeMuJoCo is expected to be installed under /opt/mujoco and discoverable via:
/opt/mujoco/lib/cmake
The CMake file uses:
find_package(mujoco REQUIRED PATHS /opt/mujoco/lib/cmake NO_DEFAULT_PATH)Install MuJoCo from source or a compatible release so that this CMake package exists.
The project uses an external ray caster utility under utils/mujoco_ray_caster/.
cd utils
git clone https://github.com/Albusgive/mujoco_ray_caster.gitThis path is ignored by git.
The default build uses ONNX Runtime:
option(USE_ONNX "Build with ONNX Runtime instead of LibTorch" ON)Set ONNXRUNTIME_ROOT if ONNX Runtime is not installed in a default search path:
export ONNXRUNTIME_ROOT=/path/to/onnxruntimeOr pass it during configure:
cmake -S . -B build_onnx -DONNXRUNTIME_ROOT=/path/to/onnxruntimeTo build with LibTorch instead:
cmake -S . -B build_libtorch -DUSE_ONNX=OFF -DTorch_DIR=/path/to/libtorch/share/cmake/TorchThe common build location used in this workspace is mujoco/C++/build_onnx.
cd mujoco/C++
cmake -S . -B build_onnx -DUSE_ONNX=ON
cmake --build build_onnx --target lab2mj -j2Other available targets include:
lab2mj main policy playback simulator
camera_test RayCaster camera test utility
camera_quat camera quaternion helper
real2sim ROS2-connected real-to-sim executable
cd mujoco/C++/build_onnx
./lab2mjlab2mj loads:
robot/go2w_description/mjcf/scene_parkour.xml
The scene path is compiled into the executable from mujoco/C++/CMakeLists.txt via MJCF_PATH.
The default scene_parkour.xml contains a multi-lane parkour course:
- Progressive isolated climb boxes from
0.10mto0.60m. - U-turn lane transitions.
- A
0.4m,30degramp connected to the continuous ditch section. - Continuous ditch gaps from
0.20mto0.60mwidth, with same-height recovery platforms. - Low-to-high jump boxes.
- Suspended overhead boxes with support posts.
- Light blue grid floor and a muted skybox for video-friendly rendering.
The scene includes the Go2W robot from:
robot/go2w_description/mjcf/go2w.xml
The robot has a fixed MJCF camera named:
RayCasterCamera
This camera is used both by RayCaster depth observations and RGB video recording.
Policy command controls:
w / s increase / decrease forward command
a / d increase / decrease lateral command
q / e increase / decrease yaw command
space zero velocity command
r reset current policy state
1..9 switch policy by index
Policy/split recording controls:
z start / stop multi-view policy video recording
x start split recording for current policy
v mark split recording step
c stop split recording
Simulation and camera controls:
backspace reset simulation
space pause / resume stepping at the MuJoCo thread level
= / + increase realtime factor
- decrease realtime factor
[ / ] switch MuJoCo fixed camera id
middle mouse click
toggle look-at tracking for selected body
F toggle relative-yaw tracking for selected body
F1 third-person over-shoulder forward view tracking base_link
F2 right-side view tracking base_link
V low-level current-view recording to mujoco_videos/current_view.mp4
Mouse controls:
left drag rotate camera
right drag pan camera
middle drag / scroll zoom camera
shift + drag alternate MuJoCo camera motion axis
double left click select body
double right click set camera target point
Perturbation controls follow MuJoCo simulate-style mapping:
Ctrl + left drag rotational perturbation / torque on selected body
Ctrl + right drag translational perturbation / external force on selected body
Two runtime camera presets are built into utils/mujoco_thread:
F1 base_link third-person over-shoulder view
relative yaw: 15 deg, elevation: -18 deg, distance: 3.2
F2 base_link right-side view
relative yaw: 90 deg, elevation: -5 deg, distance: 3.0
Both presets use relative-yaw tracking. The camera follows base_link position and keeps its yaw offset relative to the robot heading, without following the robot pitch or roll.
Press z in lab2mj to start or stop multi-view recording.
Recordings are saved under the repository root, independent of the current working directory:
policy_videos/<policy_name>_<timestamp>/
Example:
policy_videos/vtm_20260512_153000/
The output directory is ignored by git via:
/policy_videos/Each recording produces five synchronized MP4 files:
third_person.mp4
right_view.mp4
raycaster_rgb.mp4
raycaster_rgb_clean.mp4
policy_depth.mp4
Stream meanings:
third_person.mp4
Offscreen RGB render from the base_link third-person over-shoulder camera.
right_view.mp4
Offscreen RGB render from the base_link right-side camera.
raycaster_rgb.mp4
RGB render from the MJCF fixed camera RayCasterCamera, including current custom
draw overlays and MuJoCo camera visualization if enabled.
raycaster_rgb_clean.mp4
RGB render from RayCasterCamera with custom draw overlays disabled and
mjVIS_CAMERA temporarily hidden. This removes hit points and camera markers.
policy_depth.mp4
The noisy RayCaster depth image used by the visual policy.
The current window view is not used for z recording, so UI overlays and the left table are not included in these videos.
There is also a lower-level V shortcut from utils/mujoco_thread that records the current view to:
mujoco_videos/current_view.mp4
Use z for policy/debug recording and V only for quick current-window captures.
The environment draws RayCaster-related debug markers in MJ_ENV::draw().
Gamepad RB toggles sensor/camera visualization by controlling:
opt.flags[mjtVisFlag::mjVIS_CAMERA]It does not disable the custom draw() and draw_windows() paths globally. The clean RGB recording stream disables these only for that stream's offscreen render.
Gamepad support is implemented through utils/cpp_gamepad.
Main bindings in Sim2SimEnv:
left stick linear x/y command
right stick yaw command
A/B/X/Y switch policy presets
LB request current policy reset
RB toggle RayCaster sensor visualization
Menu reset current policy state
Install useful joystick tooling with:
sudo apt-get install joystick libudev-devPolicy paths are defined in mujoco/C++/CMakeLists.txt:
policy/history_1/
policy/history_5/
policy/vtm/
policy/vtm_lstm_sru/
policy/vtm_gru_sru/
policy/motion_tracking/
policy/raycam/
Visual policies such as vtm, vtm_lstm_sru, and vtm_gru_sru use RayCaster observations and refresh visual observation buffers when the policy changes or resets.
The following generated or local directories are ignored:
build/
.cache/
mujoco/C++/build/
mujoco/C++/build_onnx/
utils/mujoco_ray_caster/
split_records/
policy_videos/
ros2/build/
ros2/install/
ros2/log/
Build the main executable:
cd mujoco/C++
cmake --build build_onnx --target lab2mj -j2Validate the parkour MJCF scene:
/opt/mujoco/bin/compile robot/go2w_description/mjcf/scene_parkour.xml /tmp/scene_parkour.mjbRun from the build directory:
cd mujoco/C++/build_onnx
./lab2mj