open-source evaluation code (FID, R-Precision, etc.)

[Heibaiii]

Dear authors,

Thank you for your great work. Would it be possible to also open-source the evaluation code for metrics like FID and R-Precision? Having it would greatly help reproduce the results. I understand if it takes extra effort, and thank you either way.

Best,

[BinLee26]

Our evaluation follows the same protocol and codebase as InterGen. The main difference is that we replace the original motion representation with the proprioception state representation used in our method. For implementation details, you may refer to the evaluation code in https://github.com/tr3e/InterGen

[Heibaiii]

Thanks for the clarification！

[Heibaiii]

Our evaluation follows the same protocol and codebase as InterGen. The main difference is that we replace the original motion representation with the proprioception state representation used in our method. For implementation details, you may refer to the evaluation code in https://github.com/tr3e/InterGen

Thank you very much for your prompt and helpful response regarding the evaluation code.

I have looked into the InterGen repository as you suggested. However, I noticed that the evaluation script requires an interCLIP.ckpt checkpoint. Since our method uses a different motion representation (proprioception state representation) instead of the original one, I would like to ask:

Did you retrain InterCLIP based on the proprioception state representation used in your method?

If so, would you be able to share the retrained checkpoint to help us precisely reproduce your reported results?

If the checkpoint cannot be shared, could you please provide the training code or some guidance on how to train InterCLIP with the new representation?

I fully understand if there are constraints regarding sharing the checkpoint. Any pointers you can provide would be greatly appreciated.

Thank you again for your time and for sharing your valuable work.

[BinLee26]

Yes, we retrained InterCLIP based on the proprioception state representation. Organizing the evaluation code into a clean and release-ready form is quite time-consuming, and unfortunately we are unable to dedicate sufficient time to it at the moment. However, we have provided several key modifications in the eval directory that differ from the original https://github.com/tr3e/InterGen implementation. We have also uploaded the evaluation checkpoint (step21000.ckpt) and the ground-truth test set(gt_data.tar.gz) to Hugging Face, which should facilitate reproducing the evaluation results reported in the paper. Hope this helps!

[Heibaiii]

Thank you very much for your prompt and helpful reply. Your guidance is greatly appreciated and will be very useful!

Best regards

[Heibaiii]

Dear authors,

Thank you very much for providing the test data and the key modifications for evaluation. I have tried to set up the evaluation pipeline based on your code and data, but I am not entirely sure whether my implementation is correct. I would greatly appreciate it if you could take a look and offer some guidance.

My current workflow is as follows:

①. I first use inference/multi-agent/ase/poselib/retarget_interhuman.py to retarget the InterHuman data into AMP‑format motions. This produces files named i_person1.npy and i_person2.npy for each data index i (ranging from 1 to 7779).

②. I then use these retargeted files (specifically, to obtain the initial poses for the two actors in _reset_actors() of humanoid_il_dm_eval.py) and run bash infer.sh with collect_dataset=True. For each two‑person motion segment, I collect at least 20 generated samples and save them as .pkl files.

③. Finally, I load both the ground‑truth data (using the gt_data you provided) and the generated samples via InterDPDataset in interdp.py, and then run eval.py from the InterGen repository to compute the metrics.

Could you please check whether this pipeline is correct? If any steps are wrong or missing, I would be very grateful for your advice.

In addition, I noticed a small discrepancy: the root_translation values in the retargeted files (i_person1.npy, i_person2.npy) are slightly different from those in the original files located at inference/multi-agent/data/1_person1.npy and 1_person2.npy in your repository. Would this minor mismatch affect the final evaluation results?

Thank you again for your time and for sharing your valuable work.

Best regards,

[BinLee26]

Your pipeline looks correct. The root_translation values will not affect the final evaluation results.

[Heibaiii]

Your pipeline looks correct. The root_translation values will not affect the final evaluation results.
Dear author,

Thank you again for your guidance. I have successfully set up the evaluation pipeline following your instructions. However, when I reused eval.py from InterGen to compute metrics, I found that the reproduced results differ significantly from those reported in your paper – even when evaluating only the ground-truth test set (to verify my eval implementation).

I tried adjusting two key parameters and observed the following（please note that all experiments and observations described below are based solely on the ground-truth test set (gt_data) you provided, and do not involve any generated samples）:

① batch_size: In InterGen, the default is 96. With batch_size=96, the results are much worse than reported (e.g., R-TOP3 ≈ 0.41, FID ≈ 0.48, while the paper reports R-TOP3 ≈ 0.72 and FID ≈ 0.004). However, when I set batch_size=16, the results improve significantly – MM-dist and R-Precision become close to the paper, but FID remains poor (≈ 0.12) and diversity is much higher (≈ 12) than expected.

② emb_scale: In metrics.py (line 4), the default is 6. With emb_scale=6, the FID is still poor (≈ 0.12) and diversity is high (≈ 12). When I set emb_scale=1, all metrics (MM-dist, R-Precision, FID, diversity) become roughly consistent with the paper.

Could you please clarify the correct values for these two parameters when evaluating on the InterHuman test set? Are there any other settings or preprocessing steps I might have missed? Your advice would be greatly appreciated.

Thank you for your time and for sharing your valuable work.

Best regards,

[BinLee26]

In our work, we set batch_size=16 and emb_scale=1.