Train models on custom dataset

[adamconkey]

I am interested in training these models on a completely different dataset (i.e. not using DM Control or MuJoCo). I have recorded simulation data from Isaac Sim and I would like to train the models in an offline mode such that I'm not iteratively improving the model for RL, I'm just interested to see how the latent dynamics model performs when I feed a known action sequence through it, and view the resulting predicted image sequence.

Any guidance on the quickest way to go about this? I haven't dug too deep into the code yet, but from a quick glance it wasn't obvious to me how to do this. I appreciate any tips, I will update this issue if I figure it out myself in case others have a similar use case.

[longfeizhang617]

It seems that your idea is similar to the world model , maybe you could focus it.