About training process

[lucky9-cyou]

Thanks for your awesome work. I would like to ask if you have encountered situations where the input tensor is nan during your training. When I started training, it could be stable for a while, but then the input tensor kept becoming nan.

[shajnnnbbd]

I'd like to ask if you've solved this problem yet?

Thanks for your awesome work. I would like to ask if you have encountered situations where the input tensor is nan during your training. When I started training, it could be stable for a while, but then the input tensor kept becoming nan.

[lucky9-cyou]

I'd like to ask if you've solved this problem yet?

Thanks for your awesome work. I would like to ask if you have encountered situations where the input tensor is nan during your training. When I started training, it could be stable for a while, but then the input tensor kept becoming nan.

No, it's caused by the number of keypoints. The number of kps is zero.

[shajnnnbbd]

I'd like to ask if you've solved this problem yet?

Thanks for your awesome work. I would like to ask if you have encountered situations where the input tensor is nan during your training. When I started training, it could be stable for a while, but then the input tensor kept becoming nan.

No, it's caused by the number of keypoints. The number of kps is zero.

Thank you for your reply.

[lucky9-cyou]

I'd like to ask if you've solved this problem yet?

Thanks for your awesome work. I would like to ask if you have encountered situations where the input tensor is nan during your training. When I started training, it could be stable for a while, but then the input tensor kept becoming nan.

No, it's caused by the number of keypoints. The number of kps is zero.

Thank you for your reply.

I tried using 1/9 of the data and training with a single card. It seems to stabilize after a few epochs, and no more NaNs appear.

[Eaphan]

Take it easy. It does not affect the training process.

[Kyekyekye]

In file "train.py"
after the line "loss, metrics = model(batch)"
replace all the NAN values in metrics with valid values (0 or 9999 or values from last iteration)

or just ignore it ;p