Regarding the issue with the training loss

[nblkStudy]

I hope you are doing well. I am writing to you regarding the TDMPC code you authored.
I have a few questions: I followed your instructions and ran the training command python src/train.py task=dog-run. After training for about 18 hours, I found that the performance did not seem to improve, and the loss even showed an increasing trend. Could you please tell me what might be causing this issue? Also, is there any way I can speed up the training process?I look forward to your reply.

[nicklashansen]

Hi @nblkStudy thanks for your interest in our work! The code should work as is, were you able to get a working policy? Maybe by training longer or rerunning with a different random seed? Based on the curves you provided it seems like it is still a bit early in training.

Regarding faster training: we released TD-MPC and this codebase about 4 years ago and RL has come a long way since then. I would suggest you give TD-MPC2 (https://github.com/nicklashansen/tdmpc2) or Newt (https://github.com/nicklashansen/newt) a try since they run significantly faster while also having better data efficiency + asymptotic performance. TD-MPC2 trains a working dog policy in ~1 day and Newt in <12 hours.