Loss is nan, stopping training

[JunLiangZ]

During the training process, the problem of loss being nan occurred. Why is this?

[jasscia18]

During the training process, the problem of loss being nan occurred. Why is this?

我也出现了这个问题，请问你解决了吗

[radarFudan]

Maybe try Float32 and reduce learning rate, BF16 can suffer from some stability issue.

[zhenyuZ-HUST]

保证--if_amp为False，似乎能解决这个问题。（Try setting --if_amp to False）

[sailor-z]

Hi,
if_amp = False doesn't work for me. I also tried using a small learning rate, but the problem still exists. Does anyone know how to handle it?

[BranStarkkk]

Hi, if_amp = False doesn't work for me. I also tried using a small learning rate, but the problem still exists. Does anyone know how to handle it?

I also have this problem, have you ever solved the problem?

[sailor-z]

Hi, if_amp = False doesn't work for me. I also tried using a small learning rate, but the problem still exists. Does anyone know how to handle it?

I also have this problem, have you ever solved the problem?

Not really. It seems all vision mambas have the same problem.

[CacatuaAlan]

set AMP=False may work or just set lower lr

[BranStarkkk]

Hi, if_amp = False doesn't work for me. I also tried using a small learning rate, but the problem still exists. Does anyone know how to handle it?

I also have this problem, have you ever solved the problem?

Not really. It seems all vision mambas have the same problem.

I just change the backbone Vim to other vision mamba model, and it works...
Its name is VMamba.

[sailor-z]

Hi, if_amp = False doesn't work for me. I also tried using a small learning rate, but the problem still exists. Does anyone know how to handle it?

I also have this problem, have you ever solved the problem?

Not really. It seems all vision mambas have the same problem.

I just change the backbone Vim to other vision mamba model, and it works... Its name is VMamba.

Thanks for the information! I'll look into it.

[mdchuc]

Got same problem, fixed by dividing the sum of forward/backward hidden states by 2 to make hidden states/residuals of all layers have similar magnitude. Check out the detail: #90

[Karn3003]

@mdchuc, do you have any idea why, in code, they are flipping the out_b across the dim=-1? Shouldn't it be dim = 1?