ValueError: Unknown format code 'f' for object of type 'str'

[xuebingbing121]

Hello！I successfully ran the first round, but encountered such a problem in the next round, what is the reason? Why can the first round succeed, but the following will not work? What is the reason for multiple rounds?Thank you so much！

[yzou2]

The code runs well in my desktop. The bug seems to caused by the string format of 'loss.data.cpu().numpy()'. But that's weird since 'loss.data.cpu().numpy()' should be a float number. You may try to comment the line to see if the code works.

[xuebingbing121]

Thank you for your reply. I commented it out. However, the actual memory is insufficient in operation, which may be a problem of my graphics card, so I set BATCH_SIZE to 1, but the result of retraining is very unsatisfactory as shown below.

So I restored BATCH_SIZE to 2 and tried to adjust the size of the picture, but it also showed some dimensional errors. I think it may be the reason that it does not match the label and the output size. So still in debugging, would you like to ask if you have better suggestions?

[yzou2]

You can keep the batch size as 2 while reduce the input image size. You can try to reduce the '--input-size' to smaller size (480, 960) Check #2 (comment)

[xuebingbing121]

OK thanhs, I will try, But the first round of training that can be run before seems to have new errors:

2020-07-02 14:17:20,950 Host Done 412/505 with speed: 0.26/s
2020-07-02 14:17:20,950 Host mean iou: 40.41%
2020-07-02 14:17:20,950 Host
[77.94 23.27 65.13 22.06 21.19 41.39 36.91 12.75 81.77 38.55 79.94 62.91
32.43 61.20 28.49 26.19 2.94 33.87 18.87]
2020-07-02 14:17:23,501 Host Done 413/505 with speed: 0.26/s
2020-07-02 14:17:23,501 Host mean iou: 40.44%
2020-07-02 14:17:23,501 Host
[77.97 23.27 65.15 22.06 21.19 41.36 36.94 12.74 81.80 39.29 79.93 62.91
32.42 61.17 28.28 26.19 2.94 33.87 18.86]
Traceback (most recent call last):
File "crst_seg.py", line 1012, in
main()
File "crst_seg.py", line 333, in main
label_2_id, valid_labels, args, logger)
File "crst_seg.py", line 506, in val
np.save('%s/%s.npy' % (save_prob_path, image_name), output)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/numpy/lib/npyio.py", line 491, in save
pickle_kwargs=pickle_kwargs)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/numpy/lib/format.py", line 584, in write_array
array.tofile(fp)
IOError: problem writing element 28636136 to file

I don't think i can handle this problem.I didn't change anything else. It worked before. What's wrong now?

[yzou2]

Maybe you don't have enough hard disk space.

On Thu, Jul 2, 2020 at 2:24 AM xuebingbing121 ***@***.***> wrote: OK thanhs, I will try, But the first round of training that can be run before seems to have new errors: 2020-07-02 14:17:20,950 Host Done 412/505 with speed: 0.26/s 2020-07-02 14:17:20,950 Host mean iou: 40.41% 2020-07-02 14:17:20,950 Host [77.94 23.27 65.13 22.06 21.19 41.39 36.91 12.75 81.77 38.55 79.94 62.91 32.43 61.20 28.49 26.19 2.94 33.87 18.87] 2020-07-02 14:17:23,501 Host Done 413/505 with speed: 0.26/s 2020-07-02 14:17:23,501 Host mean iou: 40.44% 2020-07-02 14:17:23,501 Host [77.97 23.27 65.15 22.06 21.19 41.36 36.94 12.74 81.80 39.29 79.93 62.91 32.42 61.17 28.28 26.19 2.94 33.87 18.86] Traceback (most recent call last): File "crst_seg.py", line 1012, in main() File "crst_seg.py", line 333, in main label_2_id, valid_labels, args, logger) File "crst_seg.py", line 506, in val np.save('%s/%s.npy' % (save_prob_path, image_name), output) File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/numpy/lib/npyio.py", line 491, in save pickle_kwargs=pickle_kwargs) File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/numpy/lib/format.py", line 584, in write_array array.tofile(fp) IOError: problem writing element 28636136 to file I don't think i can handle this problem.I didn't change anything else. It worked before. What's wrong now? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEY3Z54KQLFNNEOKFJS2HY3RZQRY3ANCNFSM4OL4TOCQ> .

-- null

[xuebingbing121]

Thanks！I have cleaned the hard disk and changed the input_size to 480*920, but still reported an error：

2020-07-02 15:25:33,507 Host Done 505/505 with speed: 0.34/s
2020-07-02 15:25:33,508 Host mean iou: 40.92%
2020-07-02 15:25:33,508 Host
[77.94 22.05 65.80 23.34 23.17 41.44 37.06 13.22 81.98 38.82 78.58 61.96
32.66 60.91 26.84 25.34 4.00 42.73 19.74]
2020-07-02 15:25:34,328 Host ###### Finish evaluating target domain train set in round 0! Time cost: 1489.27 seconds. ######
2020-07-02 15:25:34,333 Host ###### Start kc generation in round 0 ! ######
2020-07-02 15:28:33,083 Host Mining ids : [16 7]! 3 rarest ids: [16 7 18]!
2020-07-02 15:28:33,084 Host ###### Finish kc generation in round 0! Time cost: 178.75 seconds. ######
2020-07-02 15:28:33,084 Host ###### Start pseudo-label generation in round 0 ! ######
2020-07-02 15:42:33,872 Host ###### Finish pseudo-label generation in round 0! Time cost: 840.79 seconds. ######
2020-07-02 15:42:33,996 Host ###### Start model retraining dataset in round 0! ######
Traceback (most recent call last):
File "crst_seg.py", line 1012, in
main()
File "crst_seg.py", line 416, in main
train(mix_trainloader, model, device, interp, optimizer, tot_iter, round_idx, epoch, args, logger)
File "crst_seg.py", line 549, in train
pred = interp(model(images))
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/cai/CRST-master/deeplab/model.py", line 260, in forward
x = self.layer4(x)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/cai/CRST-master/deeplab/model.py", line 91, in forward
out = self.bn3(out)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
exponential_average_factor, self.eps)
File "/home/cai/anaconda3/envs/python27/lib/python2.7/site-packages/torch/nn/functional.py", line 1670, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 116.00 MiB (GPU 0; 7.79 GiB total capacity; 6.39 GiB already allocated; 92.44 MiB free; 235.29 MiB cached)

Should I continue to reduce the size of the picture？

[yzou2]

You can keep on trying until it works.

[xuebingbing121]

Ok, thank you so much! I resized the picture to 350*700 and the code run successfully!
But i have a question, where can i set the value of the probalitity and see its output？Please reply in a specific place, this is very important to me.Thanks a lot!

[yzou2]

The results can be seen in the *round_name/ folder. The output probability map will be generated on the fly and saved in *round_name/prob. But since it will consume lots of hard disk space, the /prob folder will be deleted every round. If you want to keep it in the end, you can disable rm_prob by setting '--rm-prob True'.

[xuebingbing121]

Thanks a lot ! But i look the *round_name/ folder, there are some pictures .
And i want to add dropout to deeplab v2. What shouod i do?

[yzou2]

The probability maps should be saved in the /prob folder. In addition, dropout can be added deeplab/model.py

[xuebingbing121]

Yes, i konw it can be added in deeplab/model.py. Do i only need to add a layer of "nn.dropout(x, p=0.5)" or do i need to set it in many places? Thank you and look forward to your reply.

[xuebingbing121]

And, if i want to see the effect of dropout, can the previous ''src_model.pth'' still be used? Do i need to retrain a model in the source domain?

[yzou2]

You can try to add dropout but you may need to do some network surgery to use the pretrained deeplab weights. But I think it's better to retrain the network on ImageNet. Probably you can find the dropout version pretrained deeplab v2 somewhere in github. If I remember correctly, adding dropout to deeplab is considered in Deeplab v3+. https://github.com/tensorflow/models/blob/master/research/deeplab/model.py

If you want to explore by yourself, a simple idea can be adding dropout to prediction layer.

[xuebingbing121]

Thank you very much but the open source code you provided to me is based on the tensorflow architecture. Can this be used with the pytorch architecture we use? How do I retrain on ImageNet? (I am a beginner and I am very honored to have your guidance)