How are the results in Tables obtained?

[mansooreh1]

Hello
How are the results in Tables 2 and 3 of your paper obtained?
Because when I run, for example:

python eval_smpl_short.py --resume_checkpoint checkpoints/diffusion.ckpt --resume_checkpoint_obj checkpoints/correction.ckpt --mode no_correction

the results I get are as follows:

112
global_mpjpe 0.14765596802213363
local_mpjpe 0.1244311052640634
body_translation 0.07383353186638228
obj_translation 0.12528914639885936
obj_rot_error 0.21166486587322186
penetrate 0.00946821

How can we reproduce the numbers reported in the tables?
I’m sorry if my repeated questions are bothering you — I really appreciate your time and help.

[Sirui-Xu]

Hello, no need to apologize — we’re happy to help address any concerns you have.

The command you ran corresponds to the results in Table 1. For Table 2, which uses a different dataset, please follow the instructions provided here:
🔗 Evaluate short-term generation on the skeleton-based dataset.

For Table 3, which evaluates long-term generation, you’ll need to use the --autoregressive flag. Please refer to the long-term evaluation section. Note that some functions there are currently misaligned — see this discussion and check out this corrected code version.

[mansooreh1]

Hello, thank you for your clear response.
What I mean is that the results in Table 1 (in previous message,I mistakenly said Tables 2 and 3) of the paper do not match the results obtained from running the code.
For example, the results I get are as follows:

112
global_mpjpe 0.14765596802213363
local_mpjpe 0.1244311052640634
body_translation 0.07383353186638228
obj_translation 0.12528914639885936
obj_rot_error 0.21166486587322186
penetrate 0.00946821

Even if these values are multiplied by 1000, they still differ from those reported in the paper.
For instance, here the global_mpjpe is 147, while in Table 1 of your paper it is 140.
Similarly, penetrate here is 94, but in the paper it is 164.
What causes these differences? Are the reported results obtained in a different way?

[Sirui-Xu]

Hello, just double checking — are you using our provided checkpoints or your own trained model?

[mansooreh1]

Hello,
I’m using the checkpoints that you provided.

[Sirui-Xu]

Hi,

I'm not sure if you're using exactly the same codebase, train/test split, or random seed. Usually, differences of this scale aren't due to bugs but rather to variations in settings. I recall that some others working on this project were able to reproduce the results. I'll find some time to verify this.

Best

[mansooreh1]

Hi, thanks so much