Any plans to ship a MediaPipe↔GNM landmark correspondence file?

[adricostas]

First of all, thank you for this amazing work!

I was playing with it and was wondering if you plan to release an official MediaPipe landmark correspondence for GNM. As far as I can see, gnm/shape/data/landmarks/ currently ships head_sparse_68.txt for the FAN/iBUG-68 scheme, but in my experience MediaPipe detection is more accurate so it would be very helpful.

Thanks in advance!

[pundirmr]

@adricostas

Not official, but I built one for my own use case. Here it is if it's useful:

https://gist.github.com/pundirmr/ecf0c0bb5c93fc1a9534083a8b6d255e

It's a MediaPipe-468 → GNM correspondence in the same format as head_sparse_68.txt (i0 w0 i1 w1 i2 w2, barycentric), built against GNM v3.0 HEAD. Gist includes the file, a loader with a self-check, and the script that generates it so you can rebuild against a future GNM version.

Method: Umeyama similarity fit of MediaPipe's canonical_face_model.obj onto GNM via the shipped 68 landmarks (iBUG ordering — the mirrored variant is tried too and loses, 47.9 mm vs 18.8 mm rms), then closest-point projection onto the GNM surface constrained by GNM's own semantic vertex groups.

Accuracy vs the aligned canonical mesh: mean 2.59 mm, p95 5.09 mm, max 6.65 mm. Mouth corners (61/291) land within 0.1–0.3 mm, inner lips (13/14) 0.7–2.5 mm, eye outer corners (33/263) ~4.8 mm. Worst residuals are at the face-oval boundary, where the canonical mesh ends at the hairline and GNM keeps going over the skull.

One thing worth knowing if you roll your own: constrain the eyelid-margin landmarks to eye_sockets, split into upper/lower bands per eye. If you project them onto the orbital skin instead, the upper and lower lid land on the same triangle and the palpebral aperture collapses from ~6.3 mm to ~0.5 mm. Nothing errors, residuals stay small, and every obvious check still passes — but anything eye-related (blink detection, closure, ptosis) silently stops working. That's why the loader ships a self-check asserting those structures are still resolvable.

Note the loader has to read the file directly —> load_landmarks() dispatches off GNMLandmarksType, which only knows HEAD_SPARSE_68:

emb = np.loadtxt("head_dense_468.txt") idx, w = emb[:, ::2].astype(int), emb[:, 1::2] landmarks = (verts[idx] * w[..., None]).sum(axis=1) # (468, 3), any identity/expression/pose

Both inputs are Apache-2.0 so it's freely usable; please keep attribution to GNM and MediaPipe. Would be great to see something like this ship officially; happy for any of it to be used as a starting point.

[f-dy]

@adricostas

Not official, but I built one for my own use case. Here it is if it's useful:

https://gist.github.com/pundirmr/ecf0c0bb5c93fc1a9534083a8b6d255e

In that code, I would use the mapping from https://github.com/PeizhiYan/Mediapipe_2_Dlib_Landmarks/blob/main/mediapipe2dlib/mp2dlib.py which is more correct than MP68 (see figure below), with the bug fix from PeizhiYan/Mediapipe_2_Dlib_Landmarks#3.

Note than GNM anchors seem to have an ordering bug (see #75)