A few questions about RDT's inputs and outputs

[archworm19]

Hi, great work! I've been going through the code and am very impressed by your work. I'm considering trying out RDT2 on my franka setup and have a few questions:

1. I believe model outputs are relative to current pose of the end effector. Is that right? If so, what is xyz relative to the camera? It looks like maybe x=right, y=out, z=up. Is that right?

2. Can the pretrained model be used with a single arm without finetuning? What are the correct model inputs for this case? Left camera and 0s for right camera?

Thanks and keep up the great work!

[LBG21]

1. The model outputs relative action, and the action is relative to the end effector(the flange when you're using Franka). The actual direction is determined by the coordinate system of your own embodiment. 2. We haven't tested on single arm, but you can try it by setting the left camera to the background image in our code.

[omarrayyann]

@LBG21 where is the background image

[zcecere]

Thanks for the response! Ok, so actions are wrt to the end effector. What is the coordinate system? I just visualized your example data in hugging face and it looks like:

action[:, :10] is the right gripper
action[:, 10:] is the left gripper

It also looks like the coordinate system is the standard UMI coordinate system:
z: out
x: right
y: down
Is that right for your data?

Also, for the rotation (ex: action[:, 3:9]) is that the first 2 rows or columns of the rotation matrix?