A JAX/Equinox implementation of World Models (Ha & Schmidhuber, 2018) applied to the Gymnasium CarRacing-v3 environment.
(Visualization of Real Observation vs. VAE Reconstruction vs. RNN Dream)
The agent consists of three independent components trained sequentially:
-
Vision (V): A Convolutional VAE that compresses the 64x64x3 game frame into a 32-dimensional latent vector (
$z$ ). -
Memory (M): An MDN-RNN (LSTM + Mixture Density Network) that predicts the next latent state (
$z_{t+1}$ ) and reward ($r_{t+1}$ ) given the current state and action. This serves as the agent's "Dream" world. -
Controller (C): A simple linear model that maps the concatenated state
$[z_t, h_t]$ to actions. It is evolved using CMA-ES inside the RNN's dream environment.
.
├── src/ # Core model definitions (JAX/Equinox)
│ ├── vae.py # VAE architecture (Vision)
│ ├── rnn.py # MDN-RNN architecture (Memory)
│ └── controller.py # Linear Controller (Policy)
├── scripts/ # Helper tools
│ ├── data_collection/ # Distributed rollout collection
│ └── tools/ # Debugging & visualization
├── diagnostics/ # Debug outputs (filmstrips, grids)
├── train_dream.py # Evolution Strategy (CMA-ES) loop
├── train_rnn.py # World Model training script
├── process_data.py # Data preprocessing pipeline
└── test_agent.py # Final agent evaluation
This implementation introduces an asymmetric loss function that addresses the critical "Sim2Real gap" in World Models. The standard approach often leads to "optimism bias" where the RNN hallucinates safer outcomes than reality provides.
The asymmetric loss punishes overestimation of rewards (optimism) 5x more than underestimation (pessimism):
# Penalize "Optimism" (Pred > Actual) significantly more than "Pessimism"
asymmetric_weight = jnp.where(diff > 0, 5.0, 1.0)
loss_reward = jnp.mean(asymmetric_weight * (diff ** 2))This enables robust transfer from dream to reality, achieving human-level performance (843.0 score) without requiring massive datasets.
- Python 3.11+
- CUDA-enabled GPU (Recommended)
- Mac Users: You may need to install
swigto build Box2D (brew install swig).
Using uv (Recommended):
# Clone the repo
git clone https://github.com/Sha01in/world-models-jax.git
cd world-models-jax
# Initialize and sync environment
uv sync
# IMPORTANT: Install JAX with CUDA support (if you have an NVIDIA GPU)
# See: https://jax.readthedocs.io/en/latest/installation.html
uv pip install "jax[cuda12]"
# For Mac (Apple Silicon) users:
# uv pip install "jax-metal"Check if JAX can access your GPU:
python scripts/tools/check_gpu.pyOr using standard pip:
pip install -r requirements.txt
# IMPORTANT: Install JAX with CUDA support explicitly
pip install -U "jax[cuda12]"
# For Mac (Apple Silicon) users:
# pip install -U "jax-metal"To verify the entire pipeline runs correctly (without waiting hours for training), run the integration test:
python scripts/tools/run_pipeline_test.pyThe World Model is trained in a strict pipeline. Each step depends on the previous one.
Collect initial data to train the Vision model.
python collect_data.py
# Select Option 1: Random (Brownian Noise)Goal: ~2,000 - 5,000 episodes.
Note: Unlike the original paper which suggests 10,000 random episodes, I use a Curriculum Learning approach (see Section 4). I start with a smaller random dataset, train the agent, find where it fails, and then collect specific "failure" data. This is more efficient than random sampling.
Train the VAE to compress images.
python run_vae_training.pyEncode all collected images into latent vectors (
python process_data.pyTrain the MDN-RNN to predict the future.
python train_rnn.pyNote: This implementation uses an Asymmetric Loss to punish "Optimism" (predicting high rewards when crashing), which fixes the Sim2Real gap.
Evolve the controller inside the RNN.
python train_dream.pyRun the trained agent in the real environment.
python test_agent.pyTo achieve high scores (>800) without needing massive random datasets, I use an iterative data collection strategy:
- Recovery Data: (Option 3) Heuristic driver with noise to teach the RNN how to recover from bad states.
- Aggressive Data: (Option 4) Heuristic driver that drives too fast, teaching the RNN about friction limits.
- On-Policy Failures: (Option 5) Run your current agent, let it crash, and add that data to the training set.
Workflow:
- Collect ~2k Random Episodes -> Train V, M, C.
- Observe Failures (e.g., Agent spins out on sharp turns).
- Collect ~500 "On-Policy" or "Aggressive" episodes.
- Retrain M (RNN) with the new data.
- Retrain C (Controller).
- Repeat.
This Active Learning loop fixes the "Sim2Real Gap" (where the RNN hallucinates that driving on grass is safe) much faster than simply adding more random data.
To achieve the score of 843.0, I used the following dataset composition (~4,000 episodes total):
-
2,000 Random Episodes: Initial training of V and M.
-
500 Recovery Episodes: Heuristic driver with noise (teaching recovery).
-
500 Aggressive Episodes: Heuristic driver entering corners too fast (teaching friction limits).
-
500 On-Policy Failure Episodes: Critical Step. I ran the agent, let it crash (due to "optimism delusions"), and added this specific data to the training set.
-
Episode Score: 843.0 (Solved)
-
Behavior: Robust navigation of sharp turns; recovery from minor slips.
Note on Reproducibility: While the training scripts use fixed seeds for JAX operations (PRNGKey(0)), the data collection process (Sim2Real) involves real-time interaction with the Box2D physics engine, which can have non-deterministic elements across different hardware/OS. Exact score matching may vary, but the general learning curve should be consistent.
- Original Paper: World Models by David Ha and Juergen Schmidhuber.
- Environment: Gymnasium.
- Framework: JAX & Equinox.
This project demonstrates a Hybrid Architecture:
-
GPU-Accelerated Dreaming: The most significant advantage of JAX is in
train_dream.py. I usejax.vmapto simulate the RNN "dreams" for the entire population (256 agents) simultaneously on the GPU. This turns the Evolution Strategy evaluation which is usually a slow sequential process into a single efficient batched operation. -
CPU-Bound Reality: Since
CarRacing-v3is based on Box2D (CPU physics), the real environment interaction cannot be JIT-compiled. I handle this via:- Data Collection: Standard Python
multiprocessingto run parallel environments on CPU. - Inference: Using JAX on CPU (worker threads) or GPU (main agent) depending on the bottleneck.
- Data Collection: Standard Python
This approach leverages JAX where it excels (massive parallel simulation) while accommodating standard Gym environments.