Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
+ +
+ 
+ 
+ 
+ Abstract
++ Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the policy under a unified learning objective aimed at improving robustness. At the core of our method is a maximin optimization problem, which we solve by innovatively utilizing Stackelberg learning dynamics. We provide theoretical analysis to support our design and introduce computationally efficient implementations. We benchmark our algorithm on twelve noisy D4RL MuJoCo tasks and three stochastic Tokamak Control tasks, demonstrating its state-of-the-art performance. +
+ +Motivation
++ Existing offline MBRL methods learn a world model to maximize data likelihood, then freeze it (or lightly adapt it) while optimizing the policy. This causes an objective mismatch — the model is trained to explain the offline data, not to support good policy learning. In practice, this leaves policies brittle: even a modest amount of measurement noise at deployment time can sharply degrade the performance of strong offline RL baselines. +
+
+ Method
++ We formulate offline MBRL as a constrained maximin problem: the policy maximizes its return under the worst-case world model within an uncertainty set consistent with the offline data, while the world model is adversarially updated to minimize that return within a trust region. +
++ We solve this as a Stackelberg game — the policy is the leader, the world model is the follower that best-responds to the policy. This is the opposite direction from online MBRL, where the model is typically adapted to serve the leader's interest rather than oppose it. Using implicit differentiation through the follower's best response, we derive primal-dual Stackelberg update rules for \((\theta, \phi, \lambda)\), made practical at scale by: +
+-
+
- Approximating expensive second-order terms with the Fisher Information Matrix, using \(\mathbb{E}[\nabla^2 \log P_\phi] = -\,\mathbb{E}[\nabla \log P_\phi \, \nabla \log P_\phi^{\top}]\) to avoid an explicit Hessian. +
- The Woodbury matrix identity to invert the resulting low-rank matrix in time linear in the number of parameters rather than cubic. +
- A gradient-mask mechanism so off-policy replay data doesn't go stale as the model is updated jointly with the policy — a subtlety RAMBO's alternating-update scheme overlooks. +
The full algorithm, ROMBRL, is given in Appendix J of the paper.
+ +Theory
++ Theorem 3.1 bounds the performance gap between the policy learned by ROMBRL and the best possible comparator policy, assuming the true environment lies in the uncertainty set \(\Phi\) with probability at least \(1 - \delta/2\): +
++ where \(N\) is the offline dataset size, \(|\Phi|\) the covering number of the uncertainty set, \(C\) a concentrability coefficient, and \(\epsilon\) the uncertainty-set radius — so the suboptimality gap shrinks as the dataset grows, giving a formal robustness guarantee rather than a purely heuristic one. Theorems 3.2 and 3.3 instantiate this bound for tabular MDPs and for continuous MDPs with Gaussian world models respectively, characterizing how \(\epsilon\) should scale with dataset size and dimensionality. +
+ +Results
+D4RL MuJoCo, under 5% measurement noise at deployment. ROMBRL ranks first on 7/12 tasks and second on 4, with the best average score by a wide margin (Cohen's d ≥ 2 over every baseline).
+| Method | ROMBRL (ours) | CQL | EDAC | COMBO | RAMBO | MOBILE | RORL | TRACER | RFQI |
|---|---|---|---|---|---|---|---|---|---|
| Average Score | 77.7 (0.5) | 60.7 (1.2) | 53.7 (4.6) | 55.5 (3.6) | 55.8 (1.3) | 70.7 (2.4) | 62.3 (0.2) | 44.1 (4.8) | 30.9 (2.1) |
| Cohen's d vs. ROMBRL | – | 18.9 | 7.4 | 8.5 | 22.9 | 4.1 | 40.4 | 9.8 | 30.7 |
Robustness vs. clean-environment trade-off (OfflineRL-Kit protocol). ROMBRL matches the best clean-environment performance while losing almost nothing to noise.
+| Metric | ROMBRL (ours) | CQL | EDAC | COMBO | RAMBO | MOBILE | RORL |
|---|---|---|---|---|---|---|---|
| Standard env. | 92.8 | 80.4 | 93.0 | 89.3 | 82.7 | 95.9 | 89.5 |
| Noisy env. | 93.4 | 77.5 | 68.3 | 72.2 | 66.0 | 85.3 | 78.7 |
| Performance drop ↓ | -0.6% | 3.6% | 26.6% | 19.1% | 20.2% | 11.1% | 12.1% |
Tokamak Control (negative tracking error; higher is better). ROMBRL ranks first on 2/3 targets and second on the third, with the lowest variance across seeds.
+
+ | Tracking Target | ROMBRL (ours) | CQL | EDAC | COMBO | RAMBO | MOBILE | BAMCTS |
|---|---|---|---|---|---|---|---|
| βN | -70.9* (0.9) | -78.4 (3.1) | -63.4 (1.7) | -84.3 (7.6) | -121.1 (19.9) | -133.9 (10.1) | -111.3 (24.3) |
| Density | -60.0 (1.9) | -87.3 (12.5) | -112.5 (11.1) | -67.0* (3.1) | -81.3 (15.7) | -75.3 (4.3) | -79.6 (13.8) |
| Rotation | -10.6 (3.7) | -39.2* (10.1) | -95.4 (64.3) | -69.6 (25.9) | -300.3 (260.5) | -257.6 (153.7) | -305.6 (242.6) |
| Average Return | -47.1 (1.2) | -68.3* (6.8) | -90.4 (11.5) | -73.6 (5.8) | -167.6 (91.6) | -155.5 (47.7) | -165.5 (84.5) |
* marks the second-best result for each row.
+ +RWRL-style deployment perturbations (dropped/stuck observations, body mass, friction, joint damping). ROMBRL achieves the best score on every perturbation type and the smallest overall performance drop.
+| Metric | ROMBRL (ours) | CQL | MOBILE | RAMBO | RORL | TRACER | RFQI |
|---|---|---|---|---|---|---|---|
| Average score under perturbation | 51.9 (1.0) | 31.2 (1.0) | 45.7 (1.9) | 46.2 (0.2) | 39.3 (1.8) | 23.4 (1.6) | 16.3 (1.6) |
| Performance drop ↓ | 25.4% | 33.2% | 31.6% | 30.0% | 37.6% | 36.2% | 52.1% |
+ walker2d-medium: the full constrained Stackelberg update (Ours) substantially outperforms a naive alternating update and an unconstrained Stackelberg update — robustness comes from anticipating how the uncertainty-set boundary shifts with the policy, not just from anticipating the model.Poster
+ +
+ Citation
+@article{chen2025rombrl,
+ title = {Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning},
+ author = {Chen, Jiayu and Xu, Le and Venugopal, Aravind and Schneider, Jeff},
+ journal = {arXiv preprint arXiv:2505.13709},
+ year = {2025}
+}