Problems about run your code

[xiny-gao]

Thanks for your work.
I have some problems when I download your code and run it.

I download the dataset(you list in the readme, the link is [https://drive.google.com/drive/folders/1QQPn8-lENxaJE-ebrcAJ7-HwqPegGCRw?usp=sharing]), and just use NELL dataset.
I run the code main in train mode. The parameters are here.
python main.py --dataset NELL-One
--data_path ./NELL
--few 5
--data_form Pre-Train
--prefix nell_recdap_iter100_lr1e-3_margin1
--device 0
--batch_size 64
--g_batch 1024
--epoch 100000
--eval_epoch 1000
--checkpoint_epoch 1000
--learning_rate 1e-3
--margin 1.0
--num_diffusion_iters 100

Then, I will get the dictionary state , some files in it.

So I use the file named state_dict in the above picture. Then I get the result here.
The test parameters are here.
python main.py --dataset NELL-One
--data_path ./NELL
--few 5
--data_form Pre-Train
--prefix nell_recdap_iter100_lr1e-3_margin1_2022_cuda:0_20250719143844
--state_dict_filename state_dict
--device 0
--batch_size 64
--g_batch 1024
--epoch 100000
--eval_epoch 1000
--checkpoint_epoch 1000
--learning_rate 1e-3
--margin 1.0
--num_diffusion_iters 100
--step test

I am not sure that the result is correct. In your paper, the MRR :0.505/ Hits@10: 0.528 /Hits@5: 0.506 / Hits@1: 0.493

Could you teach me how to run your code to get the correct number mentioned in your paper?
Thanks.

[hou27]

Hi,

As I mentioned in another issue, this repo is built on an older project that already had some reproducibility trouble. After I added the diffusion part, the results can still swing a bit, but I’ve never seen them drop as low as what you got.

To be sure, I ran the code again exactly the way you described. My numbers came out like the screenshot below. That means something in our setups must be different, though I’m not sure what yet. I’m using only the code that’s on GitHub—there are no extra files on my side.

I’ll keep digging on this problem.
Thanks a lot for testing the code and sharing your results.

[xiny-gao]

OK, Thank you for reply.
First, in conda environment, your requirement_recdap.txt notes that huggingface-hub==0.27.1; but when I tried this, some errors about dgl happens, when I set huggingface-hub==0.25.2, this error vanished.

Second, I want to know your the checkpoints in your state/state_check/state_dict, when I trained the model, after 4000-5000 epoch, the process is over because valid data maybe not good. So it just trained 4000-5000epoch, I want to know your training details.

If you could help, could you send your code about the picture in your reply througn mail, not download from your github? (gaoxy0724@163.com)

Thanks.

[dai-qi036]

Sorry to bother you. Thank you for the author's code and hard work, and thank you to the issue author for raising this question.

I have been running the code in this repository recently and encountered the exact same problem as you. The test results (MRR and Hits metrics) on the NELL dataset are very low, far below the performance reported in the paper. My results are basically the same as what you reproduced.

I would like to ask if the issue authorhas found a way to solve this reproduction problem since then? Or are there any hidden details in the training parameters or environment configuration? If it is convenient, could you please share your experience? Thank you very much!