Which version of the noise_index has been employed in your model?

[ppj567]

Hi, thank you for your great work!

I am a little bit confused that the noise_index adopted in NCR and DECL are different, while in BiCro it seems to take the noise_index in NCR in the file ./noise_index. However, in the training script, I cannot find the referred directory 'noise_index/coco_precomp_0.4.npy'.

BiCro/train_coco.sh

Line 23 in 0aee28b

--noise_file noise_index/coco_precomp_0.4.npy \

In fact, the noise_index is quite important in comparing with different methods. For instance, if DECL adopts the same noise_index as NCR, it will only show a tiny edge over NCR. This phenomenon could be found at their repo.

Therefore, in order to make a thorough comparison, could you share the noise_index adopted in BiCro? Thanks a lot.

[xu5zhao]

hi, We did use the same noise Settings as NCR, and we uploaded two files ('noise_index/f30k_precomp_0.4' and 'noise_index/coco_precomp_0.4.npy')

[ppj567]

Hi, NCR only provides the noise_index of 20% and 50% ratios in their repo. Did you mean that you adopted the noise_index of 20% ratio from NCR, and adopted that of 40% and 60% ratios from DECL?

[ppj567]

By the way, I trained BiCro on Flickr30K with the noise ratio of 50% (the noise_index is taken from DECL), and I obtained the following results (the best training epoch is 36), which are inferior to NCR and DECL. Since I did not change any part of BiCro, could you give some advise on how to reproduce a good result? Thanks a lot.

[ppj567]

hi, We did use the same noise Settings as NCR, and we uploaded two files ('noise_index/f30k_precomp_0.4' and 'noise_index/coco_precomp_0.4.npy')

Hi, it seems the rest two files are still missing ('noise_index/f30k_precomp_0.6.npy' and 'noise_index/coco_precomp_0.6.npy').

[xu5zhao]

Hi, NCR only provides the noise_index of 20% and 50% ratios in their repo. Did you mean that you adopted the noise_index of 20% ratio from NCR, and adopted that of 40% and 60% ratios from DECL?

We directly adopt the noise index of 20% ratios from NCR, and generate the noise index of 40% and 60% ratios by NCR method.

[xu5zhao]

hi, We did use the same noise Settings as NCR, and we uploaded two files ('noise_index/f30k_precomp_0.4' and 'noise_index/coco_precomp_0.4.npy')

Hi, it seems the rest two files are still missing ('noise_index/f30k_precomp_0.6.npy' and 'noise_index/coco_precomp_0.6.npy').

we uploaded two files ('noise_index/f30k_precomp_0.6' and 'noise_index/coco_precomp_0.6.npy')

[xu5zhao]

By the way, I trained BiCro on Flickr30K with the noise ratio of 50% (the noise_index is taken from DECL), and I obtained the following results (the best training epoch is 36), which are inferior to NCR and DECL. Since I did not change any part of BiCro, could you give some advise on how to reproduce a good result? Thanks a lot.

We haven't tried BiCro on Flickr30K with the noise ratio of 50% in the DECL setting way. Perhaps you can try setting the parameter '--fit_type gmm', It behaves steadily under "--p_threshold", default=0.5, type=float, help="clean probability threshold" based on our previous experience.

[ppj567]

Hi, I tried to reproduce the results of BiCro on Flickr30k with 40% noise rate based on your newly updated noise_index files. During training, I kept all the settings unchanged with yours, and obtained the following results. BiCro performs a little bit better than DECL, but there is still a gap between the reported numbers in the original paper. Should I do some further modifications so as to boost the performance, or run a couple of more times to find the best one? Thanks.

[ppj567]

BiCro/co_train.py

Line 339 in bd36643

c_y,n_y = net2.predict(batch_images_l, batch_text_l, batch_lengths_l)

I found something inconsistent with the original paper, that during the first 10 epochs of co-learning, the clean data is optimized using soft labels c_y rather than hard labels (i.e, 1). In order to reproduce the numbers reported in the paper, should I change soft labels into hard labels just as the paper stated?

[ppj567]

I ran multiple times with different seeds on Flickr30k with 40% noise rate and obtained very close results as follows:

I also tried to use hard labels for clean data just as the paper stated, and found that it will lead to much worse results (up to -6% decreases). Could the authors share the checkpoints of the 10-epoch warm-up models and the best co-learning models? Thanks a lot.

[xu5zhao]

The results in this paper are based on SGRAF, which includes two models: SGR and SAF. You seem to have only tested the SGR. SGRAF is more accurate, but requires training both models. I uploaded the checkpoints training in on Flickr30k with 40% noise rate.

[xu5zhao]

I updated the evaluation.py to fix the bug, noting that the path parameters for both models: --mode_path and --mode_path1 are required when testing the SGRAF model.

[ppj567]

The results in this paper are based on SGRAF, which includes two models: SGR and SAF. You seem to have only tested the SGR. SGRAF is more accurate, but requires training both models. I uploaded the checkpoints training in on Flickr30k with 40% noise rate.

I thought you followed the same training settings (e.g., optimizer, network architecture, all the hyperparameters, etc.) as NCR in this paper. Because NCR only adopts SGR as its backbone, and it involves 2 models for ensembling (2 co-learning SGR models). In contrast, at inference phase, this work will involve 4 models for ensembling (2 co-learning SGR models and 2 co-learning SAF models). Considering both NCR and DECL as previous works only adopt 2 models for ensembling, I am concerned that is it a fair comparison? Anyway, the SGR-only BiCro model does show a good performance over previous methods, which proves its effectiveness very well.