Training a single policy to perform multiple manipulation tasks using vision + language instructions.
MultiTaskAgent/
├── src/
│ └── environments/
│ ├── base.py # Abstract simulator interface
│ └── pybullet_env.py # PyBullet implementation
├── data/ # Training datasets (generated)
├── configs/ # Model configurations
├── experiments/ # Training runs and checkpoints
├── notebooks/ # Jupyter notebooks for analysis
├── test_environment.py # Test script
└── requirements.txt
Quick Start:
cd /Users/priyanka/Documents/MultiTaskAgent
./install.shDetailed instructions: See SETUP.md
The installer automatically detects your platform and installs the appropriate simulator:
- Apple Silicon (M1/M2/M3): MuJoCo (recommended)
- x86_64: Choice of PyBullet or MuJoCo
Headless mode (no GUI, saves images):
python test_environment.py --mode headlessGUI mode (watch robot in 3D viewer):
python test_environment.py --mode gui- Pick: "pick up the [red|blue|green] block"
- Push: "push the [red|blue|green] block forward"
- Place: "place the [red|blue|green] block on the target"
The code is designed to be simulator-agnostic. To migrate:
- Implement
IsaacSimEnv(BaseRobotEnv)following the same interface - Replace
FrankaPickAndPlaceEnvwithIsaacSimEnvin training scripts - No changes needed to VLA model code!
See comments in base.py for Isaac Sim implementation pattern.