SymmGrid SERL builds on top of SERL using branched symmetries to accelerate learning in the Sample Efficient Reinforcement Learning frameowork. The latter, as originally stated, provides a set of libraries, env wrappers, and examples to train RL policies for robotic manipulation tasks. In this iteration, we include signficant additions and fixes.
Note: our work is still in progress and we have only tested in simulation, real robot peg-insertions, and cable-routing.
The following sections describe how to use SymmGrid and SERL. We will illustrate the usage with examples.
🎬: Original SERL video, SERL additional video on sample efficient RL.
Table of Contents
SymmGrid SERL has implemented branched and fractal symmetries with an optimized fractal_symmetry_replay_buffer to optimize SERL in simulation, real-robot peg-insert, and cable-routing. We are working to do object relocation and further optimize the code. We are still using ROS1 at this stage.
For people who use SERL for tasks involving controlling the gripper (e.g.,pick up objects), we strong recommend adding a small penalty to the gripper action change, as it will greatly improves the training speed. For detail, please refer to: PR #65.
Further, we also recommend providing interventions online during training in addition to loading the offline demos. If you have a Franka robot and SpaceMouse, this can be as easy as just touching the SpaceMouse during training.
We fixed a major issue in the intervention action frame. See release v0.1.1 Please update your code with the main branch.
-
Setup Conda Environment: create an environment with
conda create -n serl python=3.10
-
Install Jax as follows:
-
For CPU (not recommended):
pip install --upgrade "jax[cpu]" -
For GPU:
pip install --upgrade "jax[cuda12]==0.6.2" -
For TPU
pip install --upgrade "jax[tpu]" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html -
See the Jax Github page for more details on installing Jax.
-
-
Install the serl_launcher
cd serl_launcher pip install -e . pip install -r requirements.txt
-
Install the franka_sim
cd franka_sim pip install -e . pip install -r requirements.txt
-
Install the serl_robot_infra
cd serl_robot_infra pip install -e .
-
Install the demos
cd demos pip install -e .
SERL provides a set of common libraries for users to train RL policies for robotic manipulation tasks. The main structure of running the RL experiments involves having an actor node and a learner node, both of which interact with the robot gym environment. Both nodes run asynchronously, with data being sent from the actor to the learner node via the network using agentlace. The learner will periodically synchronize the policy with the actor. This design provides flexibility for parallel training and inference.
Table for code structure
| Code Directory | Description |
|---|---|
| serl_launcher | Main code for SERL |
| serl_launcher.agents | Agent Policies (e.g. DRQ, SAC, BC) |
| serl_launcher.wrappers | Gym env wrappers |
| serl_launcher.data | Replay buffer and data store |
| serl_launcher.vision | Vision related models and utils |
| franka_sim | Franka mujoco simulation gym environment |
| serl_robot_infra | Robot infra for running with real robots |
| serl_robot_infra.robot_servers | Flask server for sending commands to robot via ROS |
| serl_robot_infra.franka_env | Gym env for real franka robot |
We provide a simulated environment for trying out SERL with a franka robot.
Check out the Quick Start with SERL in Sim
- Training from state observation example
- Training from image observation example
- Training from image observation with 20 demo trajectories example
We provide a step-by-step guide to run RL policies with SERL on the real Franka robot.
Check out the Run with Franka Arm on Real Robot
We welcome contributions to this repository! Fork and submit a PR if you have any improvements to the codebase. Before submitting a PR, please run pre-commit run --all-files to ensure that the codebase is formatted correctly.
If you use this code for your research, please cite our paper:
@misc{vanderstelt2026SymmGrid,
title={Towards Accelerating Deep Reinforcement Learning via Branched Symmetries},
author={Ryan Vanderstelt, Cleiver Ruiz Martinez, Caeden Rosen, Blake Hull, and Juan Rojas},
year={2026},
eprint={____},
archivePrefix={arXiv},
primaryClass={cs.RO}
}