Nan (or inf) is occured during training

[CharlesTao0926]

@ricky40403
Thanks for your hard and great work.
I copied the DSQConv.py into my project and used DSQConv instead of nn.conv2d except the first and the last layers. However during the training process, after several iterations I found that the value of alphaW is less than zero. It further caused the value of k is Nna. I used 0.5 as initial value of alphaW and set the initial learning rate to 0.001.
Would you please tell me the possible reasons of the above problem?
Thank you very much.

[ricky40403]

Hi, what is the batch size you set?
I remembered that it also happened if the batch size is small when I am doing the experiments.
I guess that it may be hard to find a good parameter before it converges.
Maybe you could try to increase the batch size, or increase the warmup iter and make the try.

[CharlesTao0926]

@ricky40403
Thanks for your reply. I have another question. How do you ensure that the value of alphaW, alphaB and alphaA is larger than zero? In the process of training, I encountered a situation where the value of alpha is less than 0 after the parameter update.
Thank you very much.

[wmkai]

@ricky40403
Thanks for your reply. I have another question. How do you ensure that the value of alphaW, alphaB and alphaA is larger than zero? In the process of training, I encountered a situation where the value of alpha is less than 0 after the parameter update.
Thank you very much.

Hi, I have the same problem. How do you fixed this problem finally?
Thank you very much.

[yuanjiechen]

@ricky40403
Thanks for your reply. I have another question. How do you ensure that the value of alphaW, alphaB and alphaA is larger than zero? In the process of training, I encountered a situation where the value of alpha is less than 0 after the parameter update.
Thank you very much.

Hi, I have the same problem. How do you fixed this problem finally? Thank you very much.

Set smaller learning rate than weights on lw, uw.