Implementation of R2-Dreamer (Morihira et al., ICLR 2026), a World Model agent that learns a latent dynamics model of the environment and trains its policy entirely in imagination, without a pixel decoder.
| CarRacing-v3 | Hopper-v5 | Walker Walk |
|---|---|---|
![]() Best reward: 941 |
![]() Best reward: 21711 |
![]() Best reward: 927 |
Instead of learning a policy directly from environment interactions (model-free), a World Model learns an internal model of the environment dynamics. The agent can then imagine future trajectories in latent space and optimize its policy on them, dramatically improving sample efficiency.
The training loop has three phases:
- Experience collection — the agent interacts with the real environment and stores transitions in a replay buffer.
- World model learning — the model learns to predict future latent states, rewards and episode continuations from past experience.
- Imagination — the actor-critic is trained on trajectories dreamed by the world model, without touching the real environment.
The core of the world model is the RSSM, which maintains a latent state with two components:
-
Deterministic
$h_t$ : a recurrent state updated by a Block-GRU. -
Stochastic
$z_t$ : 32 categorical variables with 16 classes each, sampled via Gumbel-Softmax.
The full latent state is
Sequence model (deterministic transition):
Prior (what the model predicts without observation):
Posterior (what actually happened, given the observation):
The world model is trained by minimizing the KL divergence between posterior and prior, plus reward and continuation prediction losses:
where
Standard DreamerV3 trains the encoder by reconstructing pixel observations through a CNN decoder. R2-Dreamer removes the decoder entirely and replaces it with a Barlow Twins redundancy-reduction loss on the latent representations:
where
This yields ~1.6x speedup over DreamerV3 with comparable or better performance, since the expensive CNN decoder is eliminated.
The actor and critic are trained entirely on imagined trajectories of length
Actor objective (maximize imagined returns + entropy):
Critic loss (TwoHot symlog distribution over 255 bins):
Return targets are normalized using exponential moving average of the 5th–95th percentile range, a key DreamerV3 trick for stable training across diverse reward scales.
| Technique | Description |
|---|---|
| Block-GRU | GRU with block-diagonal linear layers for memory efficiency |
| Gumbel-Softmax | Straight-through gradient estimator for discrete stochastic states |
| TwoHot encoding | Distributional value/reward prediction over 255 symlog-spaced bins |
| Unimix | 1% uniform mixture in categoricals to prevent distribution collapse |
| EMA target network | Slow-moving critic target ( |
| ReturnEMA normalization | Percentile-based return scaling for reward-agnostic training |
driving-world-model/
├── main.py # CLI entry point (train / evaluate)
├── pyproject.toml # Dependencies (uv)
├── configs/
│ └── hyperparameters.yml # All environment presets and hyperparameters
├── assets/ # GIFs for README
├── src/
│ ├── agent.py # Dreamer: world model + Barlow Twins loss + actor-critic
│ ├── rssm.py # RSSM: Block-GRU + observe/imagine/kl_loss
│ ├── networks.py # ConvEncoder, MLPHead, Projector, BlockLinear, ReturnEMA
│ ├── distributions.py # OneHotDist, TwoHot, BoundedNormal, symlog
│ ├── buffer.py # Replay buffer (TorchRL SliceSampler)
│ ├── envs.py # DMC/Gymnasium wrappers + ParallelEnv
│ └── tools.py # Logger, config loading, utilities
├── runs/ # Generated outputs (one folder per preset)
│ └── <preset_name>/
│ ├── config.yml # Frozen copy of the configuration used
│ ├── best_model.pt # Best model weights (by episodic return)
│ ├── checkpoint.pt # Full training state (resumable)
│ ├── training.log # Timestamped training log
│ └── graph.png # Reward curve plot
├── scripts/
│ └── generate_video_frames.py # Extract evaluation frames
├── DQN-Rainbow-Pixel-Control/ # Model-free baseline (separate project)
└── Report/ # Typst presentation
Training is automatically resumable: if checkpoint.pt exists, the agent restores its full state and continues from the last saved step.
- Python 3.11 or 3.12
- CUDA-capable GPU (recommended; ~8 GB VRAM sufficient for the 12M model)
- MuJoCo (installed automatically via
dm-control)
git clone https://github.com/pepert03/driving-world-model
cd driving-world-model
uv python install 3.11
uv venv --python 3.11
uv syncAll presets are defined in configs/hyperparameters.yml:
| Preset | Environment | Observation | Key hyperparameters |
|---|---|---|---|
walker_walk |
DMC Walker Walk | 64×64 pixels | batch=32, seq=32, horizon=8, lr=4e-5 |
hopper_hop |
DMC Hopper Hop | 64×64 pixels | batch=16, seq=64, horizon=15, lr=4e-5 |
car_racing2 |
CarRacing-v3 | 64×64 pixels | batch=32, seq=32, horizon=8, lr=4e-5 |
hopper_v5 |
Hopper-v5 (Gymnasium) | state vector | batch=16, seq=32, horizon=15, lr=1e-4 |
cheetah_run |
DMC Cheetah Run | 64×64 pixels | batch=16, seq=64, horizon=15, lr=4e-5 |
# Train on Walker Walk (DMC, pixel observations)
uv run python main.py walker_walk --train
# Train on CarRacing (Gymnasium)
uv run python main.py car_racing2 --train
# Train on Hopper (Gymnasium, state observations)
uv run python main.py hopper_v5 --trainLoads the best saved model and renders the agent:
uv run python main.py walker_walktensorboard --logdir runs/<preset_name>/tensorboard| Parameter | Value | Description |
|---|---|---|
| Deterministic state | 1024–2048 | Block-GRU hidden size |
| Stochastic state | 32 × 16 | Categorical variables × classes |
| KL free bits | 1.0 | Minimum KL before penalizing |
|
|
1.0 / 0.1 | KL balancing weights |
| Barlow |
0.0005 | Redundancy reduction coefficient |
| Barlow scale | 0.05 | Barlow Twins loss weight |
| Parameter | Value | Description |
|---|---|---|
| Imagination horizon |
8–15 | Rollout length in latent space |
| Discount horizon | 333 ( |
|
|
|
0.95 | GAE-style bootstrapping |
| Entropy coefficient |
3e-4 – 3e-3 | Action entropy regularization |
| Target EMA |
0.02 | Slow critic update rate |
| Parameter | Value | Description |
|---|---|---|
| Learning rate | 4e-5 – 3e-4 | Adam optimizer |
| Batch size | 16–32 | Sequences per update |
| Sequence length | 32–64 | Timesteps per sequence |
| Train ratio | 64–512 | Gradient steps per env step |
| Gradient clipping | 100–1000 | Max gradient norm |
- PyTorch >= 2.4, TorchRL >= 0.9
- MuJoCo >= 3.0, dm-control >= 1.0
- Gymnasium >= 1.0 (with Box2D for CarRacing)
- TensorBoard, NumPy, OpenCV
- R2-Dreamer: Redundancy-Reduced World Models (Morihira et al., ICLR 2026)
- DreamerV3: Mastering Diverse Domains through World Models (Hafner et al., 2023)
- World Models (Ha & Schmidhuber, 2018)
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction (Zbontar et al., 2021)


