Question about segmentation–reassembly dependency and input fragment alignment in GARF

[GiovanniTagliaferri]

Hi developers,

I am testing GARF on a custom dataset of real-world objects that I have personally scanned, and I would like to clarify two specific behaviors I encountered during my experiments.

First, regarding segmentation versus reconstruction quality: I observed that even when the fracture-aware pretraining module produces suboptimal segmentation (failing to accurately identify or highlight fracture surfaces on the fragments), the final reconstruction sometimes still places some fragments in the correct position, even though the fracture surface was not properly identified. I would like to understand whether this robustness is due to the reassembly module leveraging the rich 64-channel latent features extracted by the PTv3 backbone, rather than relying strictly on the binary output of the segmentation head.

Second, I noticed that if I pass the fragments to the model individually centered at the origin (0, 0, 0), the assembly performance degrades significantly and the model fails to reconstruct the object correctly. Since the framework integrates point cloud coordinates, normals, and scale information as pose-invariant shape priors in the position embedding, I would like to know whether centering the fragments individually breaks the reassembly process by removing necessary spatial cues, or if the model expects a specific initial spatial distribution of the input point clouds.

I would appreciate any guidance on the expected input format and further clarification on the internal dependencies between the segmentation labels and the flow-based reassembly module.

[JDScript]

Very good questions! We have actually observed similar behaviors during our own experiments.

Regarding the first question (segmentation vs. reconstruction quality)

1. In the first stage of the framework, segmentation mainly serves as a pretraining task. Its purpose is to help the encoder learn what fracture surfaces look like and to encourage the network to focus on fracture-related geometric features. The segmentation head itself is removed in stage two. If you lower the visualization threshold, you may still observe fracture predictions, but they are simply below the default 0.5 confidence threshold.

2. During the second stage, the features extracted by the PTv3 backbone are used together with the raw point coordinates and normals, which are encoded through positional encoding. In practice, this provides another representation of the overall geometric structure. These combined features still carry information related to fracture geometry and can therefore help the model during the reassembly stage.

In short, you can think of the PTv3 encoder as implicitly highlighting fracture-related features, even though the explicit segmentation head is no longer used in the second stage.

Regarding the second question (centering fragments at the origin)

1. This behavior is not what we would normally expect. In our experiments without ground-truth positions, we also place all fragments initially at the origin. If the reconstruction performance degrades significantly, it might be worth checking whether there are any data preprocessing issues, such as whether all fragments are properly normalized (e.g., the full shape scaled to a consistent range).

2. If the preprocessing is correct, the issue is again unlikely to be related to the initial positions themselves, since the pipeline places fragments at the origin by default. Instead, the behavior may be related to random seeds. During training and evaluation, we apply random transformations to the fragments, which distribute them randomly in space. Different seeds may therefore lead to different spatial configurations and potentially affect the reconstruction outcome.

Hope this helps! Let us know if you have further questions and thanks for your interest and time digging into our work!

[GiovanniTagliaferri]

Thank you very much for the quick response, it was very helpful and clear!

I would also like to take the opportunity to ask another question that came up during some inference tests on my real objects. I noticed that when an object contains one fragment that is significantly smaller than the others, and the segmentation is correct for all fragments, the reconstruction works properly if I run the inference with all fragments present.

However, if I remove one fragment (leaving a larger exposed fracture surface) the smaller fragment tends to be "attracted" to this larger surface instead of the correct one, even though in the previous scenario it was assembled correctly.

Have you ever observed something similar during your experiments?