Question about metric discrepancy in reproduction

[jiangjiang18890]

Dear author team,
I have reproduced the experiments following the experimental settings in the paper on the HM3D and HM3D-v0.2 datasets by calling the DeepSeek API online. However, there is some discrepancy between the metrics I obtained and those reported in the paper.
I would like to ask whether this discrepancy is due to experimental randomness, or whether I might have missed some details in parameter configuration or other steps.
Thank you very much for your help and clarification!

[Ryanzexuan]

Dear author team, I have reproduced the experiments following the experimental settings in the paper on the HM3D and HM3D-v0.2 datasets by calling the DeepSeek API online. However, there is some discrepancy between the metrics I obtained and those reported in the paper. I would like to ask whether this discrepancy is due to experimental randomness, or whether I might have missed some details in parameter configuration or other steps. Thank you very much for your help and clarification!

Hi, it's great to see that you've successfully reproduced the experiments! To ensure I'm aligned with your setup, could you please confirm if these results were obtained using the default parameters provided in the repository or are there any changes from the defaults? Thanks for your help.

[jiangjiang18890]

Hi!The results were reproduced based on the default parameters provided in the repository, with no additional modifications made to the hyperparameters or configurations. The experimental environment was also set up in accordance with the requirements specified in the README file to ensure consistency with the experimental conditions of the original paper.
It should be supplemented that regarding the list of similar objects output in the llm_answer.txt file, we did not use the existing txt file in the repository, but regenerated the relevant content by calling DeepSeek online. We would like to confirm with you: is it necessary to set the llm parameter to none in the configuration file?

Dear author team, I have reproduced the experiments following the experimental settings in the paper on the HM3D and HM3D-v0.2 datasets by calling the DeepSeek API online. However, there is some discrepancy between the metrics I obtained and those reported in the paper. I would like to ask whether this discrepancy is due to experimental randomness, or whether I might have missed some details in parameter configuration or other steps. Thank you very much for your help and clarification!

Hi, it's great to see that you've successfully reproduced the experiments! To ensure I'm aligned with your setup, could you please confirm if these results were obtained using the default parameters provided in the repository or are there any changes from the defaults? Thanks for your help.

Hi!The results were reproduced based on the default parameters provided in the repository, with no additional modifications made to the hyperparameters or configurations. The experimental environment was also set up in accordance with the requirements specified in the README file to ensure consistency with the experimental conditions of the original paper.
It should be supplemented that regarding the list of similar objects output in the llm_answer.txt file, we did not use the existing txt file in the repository, but regenerated the relevant content by calling DeepSeek online. We would like to confirm with you: is it necessary to set the llm parameter to none in the configuration file?

[Zager-Zhang]

Could you please try using the default llm_answer.txt provided in the repository to see if the performance gap is still significant?

The DeepSeek API is continuously being updated, and we haven't tested our existing prompts against their online API recently. Since we cannot control these backend updates, the generated results might differ from our original ones.

If you'd like to generate the answers yourself, we recommend trying the qwen3:8b model via Ollama as provided in our setup, which offers a much more stable and reproducible environment.

Let us know how it goes!