Issues with MANUS fingertip data and poor finger extension in retargeting

[gray-wei]

Hi, Alex, Thanks for your great work and for sharing this project. I have a few questions regarding using MANUS data for retargeting to the Allegro Hand.

1. Using fingertip xyz vs joint angles:
   I noticed that MANUS states their fingertip positions (xyz) are highly accurate. Thus, I initially thought directly using these fingertip positions for retargeting would yield better results than using joint angles. However, in my experiments, the results were somewhat counterintuitive:

   • The thumb tracks very well.
   • But the other four fingers have poor extension capability, though their general direction is roughly correct.

   Could you please shed some light on why using fingertip xyz might not perform as well as expected here? Is it due to kinematic ambiguities or something related to optimization?

2. On data quality and sensitivity:
   I collected about 5000 samples using fingertip positions for training. After training the retargeting network, I tested it by feeding the same data back into inference to evaluate the output and online retargeting. Both of them have similar results:

   • The overall motion directions look roughly correct.
   • The thumb shows good tracking.
   • However, the other three fingers (index, middle, ring) show poor precision and cannot extend well, appearing much less refined than the thumb.

   I also found that training on different batches sometimes leads to different pairwise alignments: for example, one model aligns the index and thumb well, while another aligns the thumb and middle finger better.
   This makes me wonder if this approach is highly sensitive to the data distribution.

   • Should I pay extra attention during data collection to ensure more varied finger extensions, especially for the non-thumb fingers?

3. On simulation data:
   I also noticed that in many simulation setups, people simply sample joint angles uniformly. Given my observations, should we explicitly ensure that the synthetic dataset covers more fully extended and flexed poses to improve training quality?

Thanks a lot!!

[zhaohengyin]

It would be great if you can upload your URDF. I will look into it.

---

Regarding your points 1 & 2:
I am not sure which robot model you used.
If your robot structure does not support certain pinch poses, then the weight of certain loss functions should be tweaked smaller (e.g., pinch grasp loss), or we simply exclude the human pinch data in that case.
I will look into it and if this is the main issue, we will consider adding automatic identification & loss setup for this.

---

Regarding human data issue:
We used the following procedure.

For finger in [thumb, index, middle, ring]:
    move that finger to explore its fingertip moving range # duration: 20-30s

For pinch in [[thumb, index], [thumb, middle], [thumb, ring], [index, ring], ...]:
    perform_pinch_and_move(pair) # duration: 10-20s 
    # Detail: for pinch = (A,B): first pinch A, B. then move A and B together while maintaining pinch.

---

Regarding your point 3:
Our method first performs a very dense, uniform joint sampling (which can lead to some data imbalance in fingertip C-space) and then uses uniform voxel-downsampling over the generated robot point cloud. This approach can handle the data imbalance issue.

[gray-wei]

Thanks for your detailed reply — it was very helpful! I’ll be running some new experiments today following your suggested procedure and will update you with the results.

I’d also like to clarify a few points:

1. Initially, I used the Inspire Hand, but the results weren’t very promising. So I decided to reduce variables and switch to the Allegro Hand to better replicate your setup. All the issues I mentioned above are based on Allegro, using the provided URDF from the project.

2. Regarding the first question, I realize I might not have expressed myself clearly. When using the MANUS glove, I have the option to either directly use the provided fingertip xyz positions, or, like in your setup, retrieve the four joint angles per finger and then compute the fingertip positions through forward kinematics.
   However, since MANUS claims the fingertip xyz data is highly accurate, I was wondering:Why did you choose to use joint angles and apply FK instead of directly using the fingertip positions?
   I also noticed this additionally introduces manual measurement of finger segments. Is there a particular advantage in doing this?

[vhartman]

Hey! Thanks for the great project!

I was also wondering the same as @gray-wei inthe second question. From experience, the data that manus provides in xyz coordinates is quite good, (also for pinches from what I have seen so far). Is there a reason for the forward integration via the quaternions, and not directly using the xyz coordinates? (not angles as far as I can tell)

Is it related to wanting a different/uniform hand-size?