A few questions about SOLD

[ugadiarov-la-phystech-edu]

Hello,
Thank you for sharing the code for your project. I would like to reproduce your results on Multi-Object-Fetch tasks and kindly ask for clarification on a few questions:

1. It is mentioned in the paper that you trained the SAVi model on a dataset of 1 million frames. Did you train an individual model for every task, or did you train one model for all Reach* tasks, one model for all Push* tasks, and one model for all Pick* tasks?
2. According to the video on the project’s site https://slot-latent-dynamics.github.io/ and Figure 1, the agent is tasked with moving the green cube to the target location in Push-Specific and Pick-Specific tasks. However, I did not find pre-defined MOF environments for these tasks. It looks like the most suitable candidates are PushRed_0to4Distractors_Dense-v1 and PickRed_0to4Distractors_Dense-v1, where the agent is tasked with moving the red cube, not the green one. Did you use custom environments PushGreen and PickGreen in your experiments instead of PushRed and PickRed?

[maltemosbach]

Hi,

Thanks for the questions and for your interest in reproducing our results.

SAVi Pretraining

We train a separate SAVi model for each task. The only exceptions are the advanced reaching tasks (Reach-Specific-Relative and Reach-Distinct-Groups), where we reuse the SAVi models from Reach-Specific and Reach-Distinct, respectively. In general, transferring SAVi models between tasks that share the same maximum number of objects, for example, from Reach-Specific to Reach-Distinct, tends to work well in our experience. Exploring this transferability more systematically is something we’ve left to future work to keep the experimental setup focused.

Task definitions in multi-object-fetch

Regarding the environments in the multi-object-fetch repository: in the latest version, I’ve updated the “specific” variants to use red as the target color for consistency across both reaching and manipulation tasks. This should make the task definitions easier to understand moving forward. To reproduce the original setup, you can simply switch the target color back to green in those environments here, where green should then be excluded when sampling distractor colors.

Please feel free to ask if anything else needs clarification!