CRONOS-RL is an empirical diagnostic study of whether learned representations of dataset regimes help offline reinforcement learning under heterogeneous behavior data. It does not claim a universally better algorithm or state-of-the-art performance.
Can learned representations of dataset regimes improve offline reinforcement learning under heterogeneous behavior data?
Offline datasets can combine trajectories generated by policies of different quality. A classifier may recover these dataset regimes while still learning shortcuts that are irrelevant—or harmful—to value learning and policy extraction. CRONOS-RL separates regime classification quality from downstream utility and uses semantic and non-semantic controls to diagnose that gap.
For Hopper, Walker2d, and HalfCheetah, we concatenate equal-purpose partitions derived from D4RL medium-replay-v2, medium-v2, and medium-expert-v2. The source regime is retained as a three-class diagnostic label. Raw D4RL files are not distributed here; see DATA.md.
An MLP receives only the environment state and produces a 32-dimensional embedding and three regime probabilities. The public feature pipeline is strict: transitions use f(s_t) for observation and independently use f(s_{t+1}) for next_observation.
Experiments compare Raw IQL; learned embeddings and probabilities; zero, uniform, fixed, and random controls; direct concatenation; and residual gated conditioning. The legacy ID GatedOracle2-IQL is displayed as GatedFixedCode2-IQL because it supplies fixed code 2 and is not a per-sample ground-truth oracle. Original CSV identifiers remain unchanged.
Experiments use three seeds on Hopper, Walker2d, and HalfCheetah. Historical runs used 100k-step main evaluations plus smaller diagnostics. Scores are D4RL-normalized and summarized by environment and seed. Configuration files expose the intended variants; exact historical scripts remain under legacy/ until numerical parity of the unified trainer is established.
- Regime classification accuracy is moderately high: approximately 0.72–0.79, with Macro-F1 approximately 0.72–0.78 across the three environments.
- Classification quality does not reliably translate into improved downstream policy performance.
- Learned regime conditioning does not consistently outperform Raw IQL.
- Fixed, uniform, zero, and random controls reveal important architectural and non-semantic effects.
- Results should be interpreted as a diagnostic study of representation failure, dataset shortcuts, and downstream utility—not as a universal performance improvement.
The central negative result is informative: moderate regime separability is insufficient evidence that conditioning improves offline RL. Some non-semantic controls match or exceed learned features, indicating that dimensionality, optimization, gating, or constant architectural biases can explain apparent gains. See RESULTS.md and LIMITATIONS.md.
src/cronosrl/ contains reusable dataset, encoder, network, IQL, training, and evaluation components. scripts/ provides CLI entry points, configs/ defines experiment families, and results/ contains compact summaries and plotting data. pretrained/ documents local encoder training; weights are excluded pending redistribution review. legacy/ documents historical provenance while machine-specific scripts remain ignored locally.
conda env create -f environment.yml
conda activate cronosrl
pip install -e .The original Conda environment uses Python 3.8 and PyTorch 2.3.1. Some other versions are recovered from notebook output; unverified packages are intentionally not tightly pinned. Legacy D4RL may require platform-specific MuJoCo setup, but synthetic tests do not import D4RL or start MuJoCo.
python scripts/prepare_mixed_regime_data.py --help
python scripts/train_regime_encoder.py --help
python scripts/train_iql.py --help
python scripts/generate_figures.py --help
pytest -qPrepare three local NPZ partitions, then:
python scripts/prepare_mixed_regime_data.py --medium-replay data/hopper-medium-replay-v2.npz --medium data/hopper-medium-v2.npz --medium-expert data/hopper-medium-expert-v2.npz --output data/hopper-mixed.npz
python scripts/train_regime_encoder.py --dataset data/hopper-mixed.npz --output-dir pretrained/regime_encoders/hopper
python scripts/generate_regime_features.py --dataset data/hopper-mixed.npz --checkpoint pretrained/regime_encoders/hopper/state_regime_encoder.pt --scaler pretrained/regime_encoders/hopper/state_regime_encoder_scaler.npz --output data/hopper-mixed-strict.npz
python scripts/train_iql.py --config configs/direct_conditioning.yaml --dataset data/hopper-mixed-strict.npz --output-dir outputs/hopper --run-name hopper_prob_seed0 --variant learned_probability --seed 0 --device auto
python scripts/generate_figures.py --list-methodsThe unified entry point performs real offline IQL updates for raw and strict learned_probability. It saves a JSON configuration, CSV loss log, and final checkpoint, but deliberately performs no environment evaluation. Historical numerical parity is still unverified; see REPRODUCIBILITY.md and EXPERIMENTS.md.
This study covers three MuJoCo environments, a specific three-way mixture, one encoder family, and IQL. It does not isolate every source of heterogeneity or establish causal explanations for all control effects. The legacy environment was only partially recoverable. See LIMITATIONS.md.
Datasets originate from D4RL and its MuJoCo task suite; users must obtain data and dependencies under their respective licenses. IQL follows the algorithmic formulation of Kostrikov et al. The repository license covers original project code only and does not relicense D4RL, Gym, MuJoCo, pretrained data sources, or third-party artifacts. See THIRD_PARTY_NOTICES.md and docs/DATA.md for scope and access guidance.
Citation metadata for Xiaorui Zhao is in CITATION.cff.
Original code and documentation authored for CRONOS-RL are released under the MIT License. The license does not relicense datasets, dependencies, pretrained artifacts, or other third-party materials; see THIRD_PARTY_NOTICES.md. Pretrained weights and raw D4RL data are excluded from the public Git set pending redistribution review.