Accurate conversion Anny<->MHR

[leonardwetzel]

Thanks for your very interesting work regarding virtual try-on with Anny and MHR! I have been working with similar problems and feel like Anny works much better for fitting a certain body mesh to a given set of measurements, using the phenotype and local change parameters.

But I was not able to convert Anny to accurately to an MHR mesh, do you have done work in this direction by any chance?

[arkadiuss]

Hey @leonardwetzel! Thanks!

We've also built only MHR mesh to Anny mesh conversion (one-way). Agree - given the interpretability of Anny params it's much easier this way. With the abstract features it sounds harder, but maybe the similar trick works here.

Just an idea but maybe the approach from Anny regressor - with FD on 45 MHR shape params might be worth trying. Based on our previous MHR->Anny work for the loss I would try nearest-neighbour + bust/waist/hips measurements. My take is also that the basic pose matching should be done manually.

Might be a bit related: I'm working on pure Anny tuning and differentiable measurements. It let me to automatically find the connection between measurements and params - sensitivity Jacobian. Maybe the similar approach can bring some interpretability to MHR params - but I haven't tried it yet.

Also not sure what is the accuracy you need but maybe a workaround would be to generate MHR from questionnaire-style inputs (we only did this for Anny), or train an encoder-decoder directly from Anny mesh to MHR params. The idea here is from Bartol's regression and SHAPY.

[leonardwetzel]

Hey @arkadiuss thanks for your quick reply and thoughts! In the meantime I tried the following to test the capability of MHR identity space: Use the Anny provided SMPL-X topology output and fit an MHR mesh to it via the optimization provided by Meta:

https://github.com/facebookresearch/MHR/tree/main/tools/mhr_smpl_conversion

The mesh consisting of the target vertices in MHR topology (using barycentric interpolation from the source SMPL vertices) looks almost identical to the original Anny mesh -- however the optimization struggles to find well-approximating MHR parameters generating a similar mesh. After all MHR only has 20 body identity blendshapes while Anny has the phenotypes plus the over 100 local change parameters.

I actually tried directly tuning MHR to measurements before doing it with Anny and it led to really horrible results and humans looking like Aliens... I think the problem is that MHR identity parameters are PCA-based and not local, so you can not tune a specific body region without messing something else up.

[arkadiuss]

Got it, interesting. Clever idea with topology fitting and separating it from params. Tbh I focused more on Anny than MHR so I can only guess. But agree that MHR isn't as capable as other models (this research confirms). I hoped the even if they are PCA-based, they might be somehow anatomically interpretable - but you confirmed it's not. I'm planning some more work here on MHR around differentiable measurements so I will let you know if I get anything that might help.

Out of curiosity - why are you focusing on MHR while Anny seems to be more convenient to work with?

[leonardwetzel]

Interesting work, keep me updated!

I've been trying around with SAM 3D Body as the state of the art HMR algorithm but wasn't fully satisfied with the accuracy of the outputs. Turns out that it probably wasn't SAM 3D Body itself but rather the limitations of MHR. For Anny there's Multi-HMR but it seems like its technology is not state of the art and produces less accurate results (it's based on HMR2.0 which performs much worse according to the benchmarks).