Evaluation on RealCam-I2V

[GeleiSG]

Thank you for your excellent work on RealCam-I2V and for making your code and pretrained models available to the community.

I have recently been working on evaluating your provided checkpoint of RealCam-I2V (256*256, 50k steps, 16 frames) on the RealEstate10K dataset. However, I am encountering a discrepancy between the results I am obtaining and the metrics reported in the paper. The error metrics I am calculating are several times higher than what is presented in your work, and also deviate substantially from the results of CameraCtrl and CamI2V.

Here's a summary of the situation:

Expected Results should be around (as per the paper):

RotErr: 0.3884, TransErr_rel: 1.2943, CamMC_rel: 1.4628, TransErr_abs: 2.2317, CamMC_abs: 2.3609

My Results:

RotErr: 3.316, TransErr_rel: 9.889, CamMC_rel: 11.634, TransErr_abs: 38.855, CamMC_abs: 40.298

Here is a brief overview of my evaluation setup and the steps I have followed:

Environment:

Frameworks: PyTorch 2.7.1

Hardware: NVIDIA A6000

Evaluation Command:

config_file=configs/inference/004_realcam-i2v_256x256.yaml
save_root=../test_results
suffix_name=256_RealCam-I2V
torchrun --standalone --nproc_per_node 3 main/trainer.py --test     --base $config_file --logdir $save_root --name $suffix_name
EXP_DIR=CamI2V/test_results/256_RealCam-I2V/images/test/ImageTextcfg7.5_CameraConditionTrue_CameraCfg1.0_eta1.0_guidanceRescale0.7_cfgScheduler=constant_steps25
python glomap_evaluation.py --exp_dir $EXP_DIR

I am using the metadata CamI2V_test_metadata_1k.pth and model weight 256_realcam-i2v_50k.safetensors released in the repo. Besides, to run the evaluation code smoothly, I made the following modifications:

1. In CameraControl/data/mix_dataset.py, line 28 in init(), add:

self.enable_per_frame_scale = enable_per_frame_scale
self.enable_scene_scale = enable_scene_scale

2. In configs/inference/004_realcam-i2v_256x256.yaml, line 164, 165, 169, 175, modify:

data_root: ../datasets/CamI2V_test_1k
metadata_path: ../CamI2V_test_metadata_1k.pth
resolution: [ 256, 256 ]
devices: 3

Also I checked some camera param conditions by hand in CamI2V/test_results/256_RealCam-I2V/images/test/ImageTextcfg7.5_CameraConditionTrue_CameraCfg1.0_eta1.0_guidanceRescale0.7_cfgScheduler=constant_steps25/camera_data, its perfectly the same as the one I generated when evaluating CamI2V, but getting results with big differences.

Could you please provide some clarification on the potential reasons for this discrepancy? I am wondering if there might be:

A specific version of the evaluation script that should be used?

Any particular data preprocessing steps that I might have missed?

Any guidance you could offer to help me reproduce the results reported in your paper would be greatly appreciated.

Thank you for your time and for your valuable contributions to the field.

[MuteApo]

Thank you for your detailed feedback! The repo do have missing parts after the code sanitizing. Previous CamI2V_test_metadata lacks per_frame_scale required for evaluating RealCam-I2V, producing erroneous results in your situation. I've uploaded RealCam-I2V_test_metadata (the same 1000 clips used in CamI2V but scaling factors) and updated the evalution scripts in the latest commit to enable correct evaluation for RealCam-I2V.

Your modification is generally perfect, but it will be better if you do sampling on 8 gpus because the random seeds will differ in the 3-gpu setting. Otherwise, glomap may output ridiculous predictions (outliers) due to numeric errors introduced by Ceres-solver or CuDSS. We remove such outliers by std filtering to get more stable results.

[GeleiSG]

Thank you for your detailed feedback! The repo do have missing parts after the code sanitizing. Previous CamI2V_test_metadata lacks per_frame_scale required for evaluating RealCam-I2V, producing erroneous results in your situation. I've uploaded RealCam-I2V_test_metadata (the same 1000 clips used in CamI2V but scaling factors) and updated the evalution scripts in the latest commit to enable correct evaluation for RealCam-I2V.

Your modification is generally perfect, but it will be better if you do sampling on 8 gpus because the random seeds will differ in the 3-gpu setting. Otherwise, glomap may output ridiculous predictions (outliers) due to numeric errors introduced by Ceres-solver or CuDSS. We remove such outliers by std filtering to get more stable results.

Thank you for the quick response! That looks like it should work. I'll try out the suggestion and report back.