Geo_rec_loss Leads to Gradient Explosion and Zero Loss During Training

[cuiyujie0113]

I encountered an issue while trying to use geo_rec_loss in my training process. I modified the LiDAR-Diffusion/configs/autoencoder/kitti/autoencoder_c2_p4.yaml file by changing the geo_factor from 0 to 1. After encountering input shape mismatches, I adjusted the disc_in_channels parameter from 1 to 3 in the YAML file, which allowed the training to proceed. Then I changed False to True in self.geometry_converter = GeoConverter(curve_length, False, dataset_config).

However, I am now facing a problem where the geo_rec_loss becomes excessively large, leading to gradient explosion. After a short period of training, all the loss values, including geo_rec_loss, drop to 0.

Could you please provide guidance on how to address this issue?