Worry about the adaptation to multi-objects?

[terencepp]

It's a great job to extent sam-3d objects to multi-view.

I am afraid that the layout prediction when introduce multi-view input, is it remain precise enough to support multi-objects make a scene? (like notebook/demo_multi_object.ipynb)

[devinli123]

Thanks for your interest and great question!

Short answer: Our multi-view extension does NOT affect the pose prediction accuracy.

Here's why:

In SAM-3D Objects, the pose (scale, rotation, translation) is predicted during Stage 1 via flow matching. Our multi-view fusion strategy works as follows:

• Shape: We average the velocity predictions from all views
• Pose: We only use View 0's velocity to update the pose

This means the pose iteration process is identical to single-view inference - it always follows View 0's prediction. The multi-view fusion only affects the shape and the color, not the object's pose in View 0's camera coordinate system.

Therefore, multi-object scene composition should work exactly as in the original implementation.

You can verify the pose accuracy using these visualization options:

• --overlay_pointmap: Overlays the reconstructed object on the input point cloud
• --merge_da3_glb: Merges the object with DA3's scene reconstruction

Example:

python run_inference_weighted.py \
    --input_path ./data/example \
    --mask_prompt your_object \
    --da3_output ./da3_outputs/example/da3_output.npz \
    --overlay_pointmap \
    --merge_da3_glb

Here's a visualization showing the object correctly positioned in the scene (the dog sitting on the desk):

[terencepp]

Thanks for your interest and great question!

Short answer: Our multi-view extension does NOT affect the pose prediction accuracy.

Here's why:

In SAM-3D Objects, the pose (scale, rotation, translation) is predicted during Stage 1 via flow matching. Our multi-view fusion strategy works as follows:

• Shape: We average the velocity predictions from all views
• Pose: We only use View 0's velocity to update the pose

This means the pose iteration process is identical to single-view inference - it always follows View 0's prediction. The multi-view fusion only affects the shape and the color, not the object's pose in View 0's camera coordinate system.

Therefore, multi-object scene composition should work exactly as in the original implementation.

You can verify the pose accuracy using these visualization options:

• --overlay_pointmap: Overlays the reconstructed object on the input point cloud
• --merge_da3_glb: Merges the object with DA3's scene reconstruction

Example:

python run_inference_weighted.py
--input_path ./data/example
--mask_prompt your_object
--da3_output ./da3_outputs/example/da3_output.npz
--overlay_pointmap
--merge_da3_glb
Here's a visualization showing the object correctly positioned in the scene (the dog sitting on the desk):

Thank you for your reply. Intuitively, predicting shape from multiview will change the shape, which should maybe not compatible with unchanged layout. However, the given example is too simple to support the opinion.

[devinli123]

Thank you for your reply. Intuitively, predicting shape from multiview will change the shape, which should maybe not compatible with unchanged layout. However, the given example is too simple to support the opinion.

Good point - shape and pose are indeed coupled. Here's my take:

1. The shape change from multi-view fusion is actually not drastic. Multi-view mainly refines regions unseen from View 0, complementing the original prediction with actual observations.
2. View 0's visible region typically provides sufficient information for pose estimation. Refinements to the backside geometry don't usually have huge impact on pose.

[terencepp]

Thank you for your reply. Intuitively, predicting shape from multiview will change the shape, which should maybe not compatible with unchanged layout. However, the given example is too simple to support the opinion.

Good point - shape and pose are indeed coupled. Here's my take:

1. The shape change from multi-view fusion is actually not drastic. Multi-view mainly refines regions unseen from View 0, complementing the original prediction with actual observations.
2. View 0's visible region typically provides sufficient information for pose estimation. Refinements to the backside geometry don't usually have huge impact on pose.

make sense. it's would be better if there are some examples with multiple objects (more than ten), thank you!

[devinli123]

make sense. it's would be better if there are some examples with multiple objects (more than ten), thank you!

Thanks for the suggestion! We plan to conduct multi-view experiments with multiple objects and will update the repository soon. You are welcome to try it out then!

[BengineerZ]

Any update on this? Very interested in multi-object tabletop reconstruction!

[xuyankun]

Great work!
I am wondering if one object is occluded/invisible in View 0 (or do not provide mask of this object in View 0), should I change another view as View 0, or next views can help fill up missing object in View 0?