Train using masked mouth

[fnunnari]

If a subject is speaking, the movement of the mouth is heavily affecting the result of the facial expression classification.

Universal Ekman facial expressions were defined for still faces, not speaking. When detecting emotions in video chunks where people are speaking, this assumption breaks.

It would be useful to try creating a training set with the mouth masked-out (pixels to black), and try to do the prediction by analysing only the rest of the face.

• Mouth can be masked after MTCNN detection by drawing an ellipse according to the position of the mouth corners.
• Ellipses can be drawn using the PIL ImageDraw module: https://pillow.readthedocs.io/en/stable/reference/ImageDraw.html

Masking with an ellipse might not work when lips are funnelling.
So, more easily, we could even think of masking out everything above the nose-tip