Skip to content

nan values in training #10

@cxym-yyn

Description

@cxym-yyn

Hello, thodan.
First of all, thanks for your sharing.
I'm trying to training model with 'python train.py --model='ycbv_custom'', but i got nan values in losses. Like:
step: 0 total_loss: 9.91135 obj_cls: 3.09097505 frag_cls: 4.15891361 frag_loc: 2.40971112
step: 100 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan
INFO:tensorflow:global_step/sec: 1.57567
step: 200 total_loss: nan obj_cls: nan frag_cls: nan frag_loc: nan
INFO:tensorflow:global_step/sec: 1.66842

And i check the input data with visualization, it works. I trying to fix for days, But I don't where the error is...
Can you give me some advice for that?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions