🎭 Multimodal Emotion Classification from Social Media Videos

Multimodal Machine Learning for Emotion Understanding

Early Fusion–based Emotion Classification using Visual, Audio, and Text Features

---

🧩 Project Summary

This repository documents a multimodal emotion classification project developed for the
SATRIA DATA 2025 – Big Data Challenge, where our team achieved
Top 20 out of ~300 teams in the preliminary round.

The project aims to automatically classify emotions from social media videos by integrating information from visual, audio, and text modalities.
I served as Team Leader & Machine Learning Engineer, focusing on multimodal fusion design, modeling, and evaluation.

---

🏗 Technical Methodology

The system follows a standard machine learning pipeline, from feature preparation to multimodal integration and evaluation.

📊 Modalities Used

Modality	Description	Role
Visual	Video-level visual features extracted from sampled frames	Emotion cues
Audio	Engineered audio features (e.g., MFCC statistics)	Prosody & tone
Text	Semantic text embeddings from captions/transcripts	Linguistic signals

Each modality was processed independently and aligned using a shared id.

🔗 Feature Fusion Strategy

• Early Fusion was applied by concatenating features from all modalities
• Enables the model to learn cross-modal interactions directly in feature space

---

👤 My Contributions

As Team Leader & Machine Learning Engineer, my responsibilities included:

🔹 Multimodal System Design

• Designed the end-to-end multimodal learning pipeline
• Defined the feature alignment and fusion strategy

🔹 Modeling & Optimization

• Implemented and experimented with early fusion architectures
• Trained and tuned models using classical ML and gradient boosting
• Optimized performance for class-imbalanced emotion categories

🔹 Evaluation & Coordination

• Led model evaluation using the official Macro-averaged F1-Score
• Coordinated team workflow and ensured compliance with competition rules

Feature extraction for individual modalities was handled collaboratively, while my focus was on cross-modal integration and performance optimization.

---

🧠 Modeling & Evaluation

Model Configuration

• Primary Model: XGBoost (XGBClassifier)
• Task: Multiclass emotion classification
• Number of Classes: 8
  (Proud, Trust, Joy, Surprise, Neutral, Sadness, Fear, Anger)

Training Strategy

• Stratified train–validation split
• Class imbalance handling using SMOTE
• Hyperparameter tuning with GridSearchCV

Evaluation Metric

• Macro-averaged F1-Score
  (Official competition metric ensuring balanced evaluation across all classes)

---

📂 Repository Structure

.
├── data/
│   ├── raw/
│   │   └── data_train_sample.csv
│   │
│   └── preprocessed/
│       ├── visual/
│       │   └── visual_features_sample.csv
│       ├── audio/
│       │   └── audio_features_sample.csv
│       └── text/
│           └── text_features_sample.csv
│
├── model/
│   └── best_model.h5
│
├── notebooks/
│   └── exploratory_and_modeling.ipynb
│
├── README.md

Note: This public repository contains sample data only. Full datasets and raw media files used during the competition are not included due to size and redistribution constraints.

---

📜 Academic & Portfolio Disclaimer

This project was developed as part of a national data science competition. The repository is intended for portfolio and demonstration purposes, focusing on methodological clarity rather than leaderboard replication.

---

📬 Contact

Natalio Michael Tumuahi
Team Leader & Machine Learning Engineer

📧 Email: nataliotumuahi@gmail.com
🔗 GitHub: https://github.com/natalio123
🔗 LinkedIn: (add your LinkedIn URL)