Weird actions / Wrong environment interactions when replicating the results

[acivit]

Hello!

I modified the code to launch it with the VLM model: "Qwen3-VL-8B-Instruct-FP8".

I have results where the avatars perform actions. But when it comes to render them, they perform the actions in random locations or poses. For example, a simulated human wants to turn on the tv by sitting "inside" the fridge, or sometimes, it walks to a random place and does something weird, like painting in the air (image attached). Also, no objects appear in their hands. Nevertheless, the environment is loaded.

Is this because the simulation is not rendering the environment properly? Or is this an issue given by the limitations of the VLM?

Thanks!

[dannymcy]

Thank you for the question. There are a few factors that could be contributing to this behavior:

1. Motion set matching
   We use a predefined motion set derived from the Motion-X dataset. The LLM first proposes a free-form motion description, which we then match to the closest motion in this predefined pool using semantic similarity by another small language model. While this works well in most cases, it can occasionally result in a less relevant match. You can extend our predefined motion set by manually selecting more motion from the Motion-X dataset.

2. "Random" human location
   We have observed this issue as well. It stems from a bug in the path planning algorithm provided by Habitat-Lab 3.0. For example, when instructing a human to navigate to a sofa, the agent may sometimes end up on the opposite side of a wall. We implemented heuristic fixes in skill_utils.py to mitigate this, but the solution is not perfect. We chose not to further debug this deeply, as it originates from the simulator’s underlying planning algorithm.

3. No object in hand
   This behavior depends on the collaboration type:
   In Type 2 collaboration (robot object offering), the human performs only free-form motions and does not pick up objects.
   In Type 1 collaboration (rearrangement), the human does pick up and manipulate objects.

[dannymcy]

Hello!

I modified the code to launch it with the VLM model: "Qwen3-VL-8B-Instruct-FP8".

I have results where the avatars perform actions. But when it comes to render them, they perform the actions in random locations or poses. For example, a simulated human wants to turn on the tv by sitting "inside" the fridge, or sometimes, it walks to a random place and does something weird, like painting in the air (image attached). Also, no objects appear in their hands. Nevertheless, the environment is loaded.

Is this because the simulation is not rendering the environment properly? Or is this an issue given by the limitations of the VLM?

Thanks!

[acivit]

Thank you for the reply.

I used the type 2 collaboration, so it makes sense that the humans don't pick up objects.

Regardless, about the human location, it is not that the human goes to the sofa and ends up behind the wall.

For instance, in the following screenshot, the action should be "turn on tv" (the txt file is "turn_on_tv_clip1.txt"). There, the human walks in front of the fridge, sits down on nothing, and performs the motion.

I can also see that there is something weird, like the coffee machine and some pots are floating.

And in the following one, the action txt is "Laughing_loudly_and_sitting_at_the_same_time_1_clip1.txt". But the user goes and sits there:

Is that another bug from the simulator? Just to make sure.

Also, sometimes the third-person view is occluded by the wall. Is there any way to not visualise walls to avoid that issue?

[acivit]

And this one is the ultimatum:

Simultaneously_Dancing_with_both_hands_and_walking_clip1

I'm sorry for any inconvenience. I guess something must be wrong on my side for sure. And also, I wanted to congratulate you on the work. It does look amazing!

[dannymcy]

Thank you for the follow-up! I think issues you observed are due to the following:

1. Incorrect object grounding in human motion.
   For Type-2 collaboration, the human performs free-form motions that should be grounded to semantically relevant objects. In our demo, for example, the human “brushes teeth” in front of "a bathroom mirror". In your cases, the LLM appears to select incorrect interaction targets (e.g., “turn on TV” being associated with a fridge, and “laughing loudly and sitting” interacting with a TV or wooden desk instead of a sofa). This is a common failure mode in LLM-based planning.

2. Floating objects in a specific scene.
   We have also observed this issue in this particular scene and believe it is an internal problem with that specific environment from the Habitat HSSD dataset. We chose to retain this scene because it is a simple scene and representative of commonly used Habitat scenes. This issue does not appear in the other four larger and more complex scenes used in our experiments.

3. Upside-down human motion.
   We did not observe this issue on our end. It is likely caused by a problematic motion from the Motion-X dataset, where one axis may be inverted, resulting in an upside-down pose. We recommend removing this motion and replacing it with a similar one from Motion-X.