Bad quality annotations from dataset.

[minsu1206]

Hi authors, thanks for releasing the project.

I’m unable to reproduce the table results now.

While inspecting the dataset,
I found
1. many redundant masks
2. noisy/possibly incorrect relation annotations
3. noisy/wrong label about instances
and all of them seem to make training unstable.

How did you handle this issue to obtain the reported numbers? Could you share some tips about any preprocessing/cleaning steps?

Here are some examples. (zoom for the detail. the numbers are from id. and they are used at indexing SAM masks.)

"relationships_affordances": [
{"s_id": 2, "subject_class": "light", "o_id": 1, "object_class": "ceiling", "predicates": "attached to"},
{"s_id": 6, "subject_class": "floor", "o_id": 12, "object_class": "rug", "predicates": "under, supporting"},
{"s_id": 12, "subject_class": "rug", "o_id": 6, "object_class": "floor", "predicates": "above, lying on"},
{"s_id": 13, "subject_class": "ottoman", "o_id": 12, "object_class": "rug", "predicates": "above, standing on"},
{"s_id": 16, "subject_class": "ottoman", "o_id": 12, "object_class": "rug", "predicates": "above, standing on"},
{"s_id": 17, "subject_class": "lamp", "o_id": 24, "object_class": "table", "predicates": "above, standing on"},
{"s_id": 18, "subject_class": "sofa", "o_id": 6, "object_class": "floor", "predicates": "above, standing on"},
{"s_id": 20, "subject_class": "door", "o_id": 9, "object_class": "door frame", "predicates": "within"},
{"s_id": 21, "subject_class": "door handle", "o_id": 20, "object_class": "door", "predicates": "attached to"},
{"s_id": 23, "subject_class": "pillow", "o_id": 18, "object_class": "sofa", "predicates": "above, lying on"},
{"s_id": 24, "subject_class": "table", "o_id": 6, "object_class": "floor", "predicates": "above, standing on"},
{"s_id": 26, "subject_class": "rug", "o_id": 6, "object_class": "floor", "predicates": "above, lying on"},
{"s_id": 28, "subject_class": "cabinet", "o_id": 6, "object_class": "floor", "predicates": "above, standing on"},
{"s_id": 31, "subject_class": "vase", "o_id": 28, "object_class": "cabinet", "predicates": "above, standing on"},
{"s_id": 32, "subject_class": "teapot", "o_id": 28, "object_class": "cabinet", "predicates": "above, standing on"},
{"s_id": 34, "subject_class": "tray", "o_id": 28, "object_class": "cabinet", "predicates": "above, standing on"},
{"s_id": 36, "subject_class": "cushion", "o_id": 18, "object_class": "sofa", "predicates": "above, lying on"},
{"s_id": 37, "subject_class": "pillow", "o_id": 18, "object_class": "sofa", "predicates": "above, lying on"},
{"s_id": 38, "subject_class": "lamp shade", "o_id": 17, "object_class": "lamp", "predicates": "part of"},
{"s_id": 39, "subject_class": "rug", "o_id": 6, "object_class": "floor", "predicates": "above, lying on"},
{"s_id": 40, "subject_class": "lamp switch", "o_id": 17, "object_class": "lamp", "predicates": "part of"}
]

"relationships_affordances": [
{"s_id":4,"subject_class":"carpet","o_id":3,"object_class":"floor","predicates":"above, lying on"},
{"s_id":3,"subject_class":"floor","o_id":4,"object_class":"carpet","predicates":"below, supporting"},
{"s_id":22,"subject_class":"ottoman","o_id":4,"object_class":"carpet","predicates":"above, standing on"},
{"s_id":24,"subject_class":"ottoman","o_id":4,"object_class":"carpet","predicates":"above, standing on"},
{"s_id":22,"subject_class":"ottoman","o_id":24,"object_class":"ottoman","predicates":"same type"},
{"s_id":31,"subject_class":"cabinet","o_id":3,"object_class":"floor","predicates":"above, standing on"},
{"s_id":30,"subject_class":"cabinet","o_id":31,"object_class":"cabinet","predicates":"same type"},
{"s_id":32,"subject_class":"cabinet","o_id":31,"object_class":"cabinet","predicates":"same type"},
{"s_id":33,"subject_class":"cabinet","o_id":31,"object_class":"cabinet","predicates":"same type"},
{"s_id":37,"subject_class":"vase","o_id":31,"object_class":"cabinet","predicates":"above, standing on"},
{"s_id":38,"subject_class":"bottle","o_id":31,"object_class":"cabinet","predicates":"above, standing on"},
{"s_id":9,"subject_class":"blinds","o_id":7,"object_class":"window","predicates":"covering"},
{"s_id":12,"subject_class":"cushion","o_id":28,"object_class":"cushion","predicates":"same type"},
{"s_id":28,"subject_class":"cushion","o_id":29,"object_class":"cushion","predicates":"same type"}
]

Thanks in advance.

[kochsebastian]

What is the issue exactly? You could use SemanticSAM to get less redundant masks. The generated LLM annotation look fine? But the instance class names indeed look weird. Are you passing in the entire image with SoM into the LLM? Thats what we did.
Which GPT model are you using?

[minsu1206]

I just followed given instruction and used datasets/preprocess_dataset_gpt.py (without any changes ; GPT-4o)
Issues are

(1) Redundant masks
This leads to redundant annotations.

In this case, 36, 38, 39-th masks are actually involved in 9-th mask (whole window) They are too-detailed sub-masks.
I'll try again with SemanticSAM.

(2) Strange LLM annotations ;
To be specific, most annotation shows wrong instance categories.
Therefore, some cases show wrong matched instance pairs, leading to wrong relation annotations.
e.g., If sofa is annotated as floor, wrong pair is generated : sofa and floor is considered as same type. This seems to be harmful for training.

[kochsebastian]

but the instance here look much cleaner than the ones you linked before. Some over-segmentation yes. I believe we omitted semantic-sam in this repo because of licensing issues.

[kochsebastian]

Once you installed semantic sam, you can run

python datasets/preprocess_dataset_gpt.py --data_dir [PATH] --sam_model semantic-sam

[minsu1206]

Hi. sorry for late reply

I am trying to reproduce the localization evaluation described in the paper, especially the following part from the experimental section:

“Localization. To localize target queries with RelationField, we split the language queries into nouns and verbs. First, the nouns are localized using the object field and computing the cosine-similarity to the nouns in the language query. Then, we refine the localization by combining the object prediction with the relationship embedding by rejecting all candidate predictions that do not have a relationship feature aligned with the verb from the query. For LERF and OpenNeRF ...”

I would like to clarify the exact implementation details of this localization pipeline.

At first, I tried the following approach:

1. Use the GT point cloud to isolate the subject points.
2. For all non-subject GT points, compute the relation feature with respect to the subject.
3. Then either:
   • apply thresholding to the relation score, or
   • perform softmax prediction over a set of verbs.

However, this did not work well. The IoU was very low, around 0.1–0.2.

After reading the paper more carefully, I changed the pipeline as follows:

1. First localize the target object using the OpenSeg/object field feature.

   • I compute cosine similarity between the NeRF point OpenSeg feature and the target object text embedding.
   • Then I apply thresholding to obtain an object candidate mask.

2. Then refine the candidate mask using the relation feature.

   • I compute cosine similarity between the relation feature and the relation/verb text embedding.
   • Then I apply thresholding to obtain a relation mask.

3. Finally, I take the intersection of the two masks: final_mask = object_mask & relation_mask

This gives much better results than my initial attempt, although the numbers are still not exactly the same as those reported in the paper.

Could you clarify whether this is the intended implementation of the localization procedure?

In particular, I would like to ask:

(A) Is the localization supposed to first find object candidates using the object/OpenSeg feature, and then refine them using the relation feature?

(B) Is the final prediction obtained by intersecting the object-based mask and the relation-based mask, or is there a different score-combination / candidate-rejection strategy?

(C) If you use thresholding, then, how are the thresholds for the object field and the relation field selected?

Any additional details about the exact localization protocol would be very helpful for reproducing the reported results.

Thank you in advance !!