A robust suite of scripts designed to extract relevant facial expression features from images and video recordings. This toolkit streamlines the extraction of facial landmarks, 3D face pose (rotation angles), Action Units (AUs) and emotions to support clinical and phenotypic digital health research.
This codebase has been actively utilized to process data for two major translational research studies:
- AMP SCZ (Accelerating Medicines Partnership® Schizophrenia): A major international public-private partnership led by the National Institute of Mental Health (NIMH) and the Foundation for the National Institutes of Health (FNIH) aiming to develop early treatments and predictive tools for schizophrenia. In this study, this code aided the automated feature analysis of open-ended interviews.
- PREDiCTOR (Phenotypes REimagined to Define Clinical Treatment and Outcome Research): An initiative by the Mount Sinai Health System and IBM Research leveraging artificial intelligence to create objective measures for psychiatric care. Here, the framework was deployed to analyze facial features during therapy sessions.
- Facial Landmark & Pose Estimation: Extracts 3D head posture (pitch, roll, yaw) derived from the Google MediaPipe FaceMesh topology (leveraging insights from Rasmus Jones' geometrical framework).
- Emotion & Action Unit Detection: Integrates Py-Feat alongside custom pipelines using the Residual Masking Network (ResMaskNet) model architecture.
- Tensor-Driven Processing: Fully optimized using PyTorch for fast inference.
pip install -r requirements.txtIf you utilize this toolkit or refer to its clinical application, please cite our published work:
- Bilgrami, Z. R., Castro, E., Agurto, C., et al. Collecting language, speech acoustics, and facial expression to predict psychosis and other clinical outcomes: strategies from the AMP® SCZ initiative. Schizophr 11, 31 (2025). https://doi.org/10.1038/s41537-025-00669-z