OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation

🎉 Accepted to CORL 2026 🎉

Runyi Yu^1,2,*, Xiaoyi Lin^1,3,*, Ji Ma¹, Yinhuai Wang^2,✉, Koukou Luo^1,2, Jiahao Ji¹, Huayi Wang^1,4, Wenjia Wang^1,4, Runhan Zhang¹, Ping Tan², Ting Wu¹, Ruoli Dai¹, Qifeng Chen^2,✉, Lei Han^1,✉

¹Noitom Robotics   ²The Hong Kong University of Science and Technology   ³Wuhan University   ⁴The University of Hong Kong

^*Equal contributors    ^✉Corresponding authors

---

OmniContact is a contact-flow framework for long-horizon humanoid loco-manipulation. The system has two main pieces:

• CFgen generates task-space contact-flow references for skills such as carry, push, slide, relocate, kick, and chained meta-skills.
• CFtrack is the low-level policy that tracks either CFgen references or full .npz human-object interaction motions.

This repository provides two MuJoCo execution paths:

• deploy_omnicontact/run_skill_omnicontact.py: direct scripted execution for CFgen or NPZmotion tracking.
• deploy_omnicontact/deploy_omnicontact.py: interactive hot-switch execution with an Xbox joystick, designed to mirror the state switching pattern used by sim2real deployment.

📌 The CF-Track training code in Isaac Lab is available now in OmniContact.

⚙️ Setup

Create and activate the Python environment:

conda create -n omnicontact python=3.11 -y
conda activate omnicontact
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia -y
pip install numpy onnx onnxruntime mujoco pyyaml scipy pygame

Run commands from the repository root:

cd /path/to/OmniContact_sim2sim

The direct runner resolves model filenames under policy/omnicontact/model/. For example, --policy omnicontact_transformer.onnx loads policy/omnicontact/model/omnicontact_transformer.onnx.

▶️ Direct Runner

Use run_skill_omnicontact.py for scripted runs that start OmniContact immediately.

🧭 CFgen Reference

python deploy_omnicontact/run_skill_omnicontact.py \
  --reference-source CFgen \
  --policy omnicontact_transformer.onnx \
  --task carrybox \
  --init-pos 1.0 0.0 \
  --goal-pos 2.5 0.5

Supported Tasks

Family	Tasks
Locomotion	loco
Carry	carrybox
Push	pushbox, pushbox-in, pushbox-two
Slide	slidebox, slidebox-left, slidebox-right
Ball	relocateball, kickball

--task pushbox is treated as an alias for pushbox-in.

Policy Selection and Success Rates

Select a model with --policy. Both ONNX files are available in policy/omnicontact/model/:

• Transformer (default): --policy omnicontact_transformer.onnx
• MLP: --policy omnicontact_mlp.onnx

The table below compares their success rates on five CFgen tasks. The best result for each task is shown in bold.

Task	Threshold	Transformer	MLP
carrybox	0.2	99.0%	79.5%
pushbox-in	0.5	96.5%	94.5%
pushbox-two	0.5	96.5%	45.5%
slidebox	0.2	100.0%	99.5%
relocateball	0.2	98.0%	93.0%
Average	—	98.0%	82.4%
Training Time	—	4 GPU × 86.8 h	4 GPU × 48.9 h

Average is the unweighted mean across all five tasks.

🔗 Skill Chaining

python deploy_omnicontact/run_skill_omnicontact.py \
  --reference-source CFgen \
  --task-chaining relocate-kick \
  --init-pos -2.0 0.0 \
  --goal-pos 0.0 1.0

python deploy_omnicontact/run_skill_omnicontact.py \
  --reference-source CFgen \
  --task-chaining carry-push \
  --init-pos 1.0 0.0 \
  --goal-pos 2.0 0.5

Common chain presets include push-carry, carry-push, push-relocate, relocate-kick, carry-carry, carry-carry-carry, and carryheart.

Chain XML mapping

Chain	Scene XML
push-carry	g1_description/omnicontact_pushcarry_box.xml
carry-push	g1_description/omnicontact_pushcarry_box.xml
push-relocate	g1_description/omnicontact_pushrelocate_ball.xml
relocate-kick	g1_description/omnicontact_relocate_kick_ball.xml
carry-carry	g1_description/omnicontact_stack_2box.xml
carry-carry-carry	g1_description/omnicontact_stack_3box.xml
carryheart	g1_description/omnicontact_heart_10box.xml

Extra object initialization can be provided for chained tasks:

python deploy_omnicontact/run_skill_omnicontact.py \
  --reference-source CFgen \
  --policy omnicontact_transformer.onnx \
  --task-chaining push-carry \
  --init-pos 1.0 0.0 \
  --init-pos-extra 2.2 -0.8 \
  --goal-pos 2.5 0.5

🎞️ NPZ Motion Tracking

Use --reference-source NPZmotion to track a full .npz motion from data/.

python deploy_omnicontact/run_skill_omnicontact.py \
  --reference-source NPZmotion \
  --policy omnicontact_transformer.onnx \
  --npz-dir data/relocateball/relocateball_motion_3_with_contact.npz \
  --start-frame 0

The runner infers XML assets from common data folders. Pass --xml-path to override automatic XML selection.

NPZ XML mapping

NPZ path prefix	Scene XML
data/loco	g1_description/omnicontact_carry_box.xml
data/carrybox	g1_description/omnicontact_carry_box.xml
data/pushbox	g1_description/omnicontact_push_box_npz.xml
data/slidebox	g1_description/omnicontact_slide_box_npz.xml
data/relocateball	g1_description/omnicontact_relocate_ball.xml
data/kickball	g1_description/omnicontact_kick_ball_npz.xml

python deploy_omnicontact/run_skill_omnicontact.py \
  --reference-source NPZmotion \
  --xml-path g1_description/omnicontact_relocate_ball.xml \
  --policy omnicontact_transformer.onnx \
  --npz-dir data/relocateball/relocateball_motion_3_with_contact.npz

Useful runner options

Option	Meaning
--headless	Run without opening the MuJoCo GLFW viewer.
--max-steps N	Stop after N simulation steps. Values <=0 mean unlimited.
--stop-when-done	Stop when CFgen reaches the end and switches back to locomotion.
--start-frame N	Inclusive start frame for NPZ motion tracking.
--end-frame N	Exclusive end frame for NPZ motion tracking. -1 uses all remaining frames.
--no-reset-env	Disable startup environment reset.
--disable-replan	Disable CFgen replan support.

🎮 Interactive Hot-Switch Deploy

deploy_omnicontact/deploy_omnicontact.py keeps the full FSM alive and allows hot-switching between policies with an Xbox joystick:

Passive / default state -> DefaultPose -> LocoMode -> OmniContact

This mirrors the sim2real pattern: bring the robot to a stable default pose, switch into locomotion, then switch into the OmniContact policy at the desired moment without restarting the controller.

▶️ Example

python deploy_omnicontact/deploy_omnicontact.py \
  --task pushbox \
  --init-pos 1.0 1.0 \
  --goal-pos 3.0 0.0

deploy_omnicontact.py automatically selects the MuJoCo XML from --task. Pass --xml-path only when you want to override this automatic mapping.

Deploy XML mapping

Task	Scene XML
carrybox	g1_description/omnicontact_carry_box.xml
pushbox, pushbox-in, pushbox-two	g1_description/omnicontact_push_box.xml
slidebox, slidebox-left, slidebox-right	g1_description/omnicontact_slide_box.xml
relocateball	g1_description/omnicontact_relocate_ball.xml
kickball, kickbox	g1_description/omnicontact_kick_ball.xml
push-carry, carry-push	g1_description/omnicontact_pushcarry_box.xml
push-relocate	g1_description/omnicontact_pushrelocate_ball.xml
relocate-kick	g1_description/omnicontact_relocate_kick_ball.xml
carry-carry	g1_description/omnicontact_stack_2box.xml
carry-carry-carry	g1_description/omnicontact_stack_3box.xml
carryheart	g1_description/omnicontact_heart_10box.xml

🕹️ Xbox Joystick Controls

The joystick mapping is defined in common/joystick.py.

Input	Runtime switch
START	POS_RESET / DefaultPose
R1 + A	LocoMode
L1 + A	OmniContact policy
R1 + B	Skill cooldown / auxiliary skill mode
L3	Passive
SELECT	Stop the deploy loop

When switching to OmniContact with L1 + A, the script resets the active object and table references from --init-pos, --goal-pos, and --box-half-dims, then enters the OmniContact FSM state.

The interactive deploy visualizes:

• wrist, torso, and ankle reference mocaps;
• contact-state color changes;
• table/object references;
• ghost robot visualization when the loaded XML contains g1_ghost.xml.

If the ghost robot is not visible in the MuJoCo viewer, enable group 1 rendering, since the ghost geoms are assigned to visual group 1.

📌 CF-Track Training

The CF-Track training code in Isaac Lab is available in OmniContact.

📚 Citation

@misc{yu2026omnicontactchainingmetaskillscontact,
      title={OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation}, 
      author={Runyi Yu and Xiaoyi Lin and Ji Ma and Yinhuai Wang and Koukou Luo and Jiahao Ji and Huayi Wang and Wenjia Wang and Runhan Zhang and Ping Tan and Ting Wu and Ruoli Dai and Qifeng Chen and Lei Han},
      year={2026},
      eprint={2606.26201},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2606.26201}, 
}