Training time cost

[Liwen-Xiao]

Hello! Great job! Could you tell me how long your training time is? On my machine, using a single 3090 GPU, the first epoch shows it will take around 8 hours to train. Is this similar to your experience?

[youngzhou1999]

Hi. Thanks for your interest.
The first epoch is relatively longer as it will process data for training, you can reduce the radius of map retrieval to fasten your training process.

[Liwen-Xiao]

Thank you for your reply! I changed the radius and the training cost decreased a lot. Thanks again!

[lon0862]

Hi, but I find my training cost is take around 8 hours in each epoch, not only the first epoch. Is this similar to your experience?

[Liwen-Xiao]

Hi, but I find my training cost is take around 8 hours in each epoch, not only the first epoch. Is this similar to your experience?

Yes, it is the same. I changed the local radius to a smaller one and solved the problem.

[lon0862]

Yes, it is the same. I changed the local radius to a smaller one and solved the problem.

Thanks, I do the same thing to solved it, too.

[lon0862]

@Liwen-Xiao Hi, I want to ask about the retrained result. I use 1 GPU, batch_size: 32, accumulate by 2 batches。Final only get val_minADE: 0.6525, val_minFDE: 0.9325, val_MR: 0.0880。And in preprocess data, I use local_radius:65。
I want to know have you get the result same as github checkpoints?

[Liwen-Xiao]

Hi, I also retrained the model. I got val_minFDE as 0.919, which is close to 0.913 as the author reported. I use 1 GPU, batch size is 32, and the epoch is 64. I get the best result at the 33rd epoch.

[lon0862]

Hi, I use the same parameter as you show. And I get the best result at the 31st epoch. which val_minADE is 0.649,
val_minFDE:0.924, val_MR:0.086. It is a little worse than yours and author reported, but I think it is acceptable. Thanks!

[lon0862]

But I still confuse about the quality score using, do you try it as paper says?

[weekiiy]

Do you have the same situation, I trained more than 40 epochs and the loss can not be reduced to around 8, how to solve this?

[Yurriki]

@Liwen-Xiao Sorry to bother you, but I would like to ask how long it takes to train for about 40 rounds on a single 3090 graphics card with the same experimental setup as yours