Misalignment and Inverted Orientation of Object in Reconstructed GLB Output

[CHNxindong]

Dear Author,

Thanks for your great work and for releasing the code! I have a question regarding the following steps.

When I run the commands below:

1. python scripts/run_da3.py
   --image_dir ./data/example/images
   --output_dir ./da3_outputs/example

2. python run_inference_weighted.py
   --input_path ./data/example
   --mask_prompt stuffed_toy
   --image_names 0,1,2,3,4,5,6,7
   --da3_output ./da3_outputs/example/da3_output.npz
   --stage2_weight_source visibility
   --merge_da3_glb

I obtain the file result_merged_scene.glb. However, when I visualize this .glb file using the OHZI tool, the result appears as follows:

As shown, the reconstructed scene and the object are misaligned: the object appears to be oriented in the opposite direction.

I followed the instructions in the README.md, with only two minor modifications:

1. I copied the sam3d_objects/utils/visualization directory from the original SAM3D project (sam-3d-objects) into MV-SAM3D, because this folder is missing in MV-SAM3D and its absence caused an import error.

2. I renamed the image files in both data/example/images and data/example/stuffed_toy from 1–8.png to 0–7.png to match the zero-based indexing used by DA3 (since the DA3 output expects frames numbered from 0 to 7).

Could you please advise what might be causing this alignment/orientation issue? Thank you again for your great work!

[devinli123]

Thank you for your attention.

Regarding the two modifications you mentioned: the first one, your change is correct. I forgot to include this folder when I uploaded the project, and it will not cause such an error.

As for your second modification, I would like to clarify a detail: did you directly rename the 8.png to 0.png, or shift the sequence of PNGs numbered from 1 to 8 forward by one position to make them range from 0 to 7?

I suspect this issue might be caused by a mismatch between the order of input images here and the order of images in the DA3 results. Could you please check the processing of this part?

[CHNxindong]

Thank you for your attention.

Regarding the two modifications you mentioned: the first one, your change is correct. I forgot to include this folder when I uploaded the project, and it will not cause such an error.

As for your second modification, I would like to clarify a detail: did you directly rename the 8.png to 0.png, or shift the sequence of PNGs numbered from 1 to 8 forward by one position to make them range from 0 to 7?

I suspect this issue might be caused by a mismatch between the order of input images here and the order of images in the DA3 results. Could you please check the processing of this part?

Dear Author,

Thank you for your reply and for raising these important points. In response, I’d like to clarify the following:

1. File Renaming for Zero-Based Indexing:
   To align with zero-based indexing conventions in DA_3, I shifted the sequence of PNG files originally numbered from 1.png to 8.png forward by one position, renaming them to 0.png through 7.png. Specifically:

   • 1.png → 0.png
   • 2.png → 1.png
   • 3.png → 2.png
   • … and so on.

2. To confirm that there is no mismatch between the order of input images here and the order of images in the DA3 results, I generated a side-by-side visualization that concatenates, for each viewpoint: the original RGB image, the corresponding mask, and the DA_3 output. Each row in the visualization represents one viewpoint, clearly demonstrating consistent ordering across all components.

3. Moreover, I further provide you with the Object visualization: result.ply and result.glb
   (1) result.ply shown in SuperSplat: The model appears rotated by 180 degrees around the Z-axis. However, I did not apply any explicit transformation or rotation in this viewer

(2) result.glb viewed in the OHZI tool: In this viewer, the object appears correctly oriented, and the coordinate axes align as expected. However, its appearance, particularly in the collar (bow tie) and facial region of the toy, is suboptimal.

[jkim50104]

@CHNxindong I am getting a similar result. Did you figure out what causes the problem?

[jkim50104]

@CHNxindong Seems like the result closest to the example shown at the Readme is simple average version. I tried a few things to make the weighted version better with DA3, but didn't get any better result than simple average at different examples too.

[wuhoward]

The simple average version also seems better to me on my examples.