Introduction

Neuromarketing combines neuroscience, psychology, AI, and marketing to study consumer behavior and decision-making. It uses advanced technologies like EEG, HRV, eye-tracking, and facial expression analysis to measure emotional and cognitive responses, overcoming the limitations of traditional methods like surveys.

This repository contains the code and other analysis notebooks for the Multi-Motion project for predicting emotions using different bio-signals (FER, pupil, GSR, heart rate, EEG).

The focus of these notebooks is building a fusion model to combine the single modalities of the signals used.

Implemented different modeling approaches including Leave-One-Participant-Out (LOPO) and Leave-Two-Participants-Out (L2PO) cross-validaion, fusion models and modality-specific predictors (focusin on pupil, GSR and FER).

This project, a collaboration between CSEE, the Department of Psychology, and Essex Business School, aims to develop an AI-based tool for emotion recognition and attention detection to enhance neuromarketing research.

Objective

The primary objective is to develop an AI-based application capable of detecting and classifying emotions and attention levels using physiological and neurological data. Specific goals include:

• Designing an emotion and attention detection system.
• Leveraging multimodal data (e.g., EEG, ECG) for accurate classification.
• Supporting neuromarketing research and clinical psychology applications.

Steps to Set Up and Run the Project (Feature Extraction)

1. Clone the Repository
   Clone the repository to your local machine.

2. Download Survey Stimuli

   • Download the survey_stimuli folder from the Box link:
     Box Link
   • Create Files folder
   • Place the downloaded files in the following directory:
     Files/survey_stimuli
   • In a same way place all the data in Files folder
   • Like: MyFiles, required_files data inside Files folder.
   • Please Make sure the above steps are completed before running you code.
   • Do not push these files to the GitHub repository as they are too large and contains confidential data.

3. Run main.py
   Execute the main.py script to start the process.

4. Step 0: Imputation of Raw Files
   Compute the imputation of raw files using the appropriate option in the script.

5. Generate Pupil CSV Files
   Once the imputation is complete, generate the pupil CSV files by selecting option 5.

6. Compute Feature Matrix
   Finally, compute the feature matrix by selecting option 1.

Usage (Machine Learning)

Option 1: Run on Google Colab

1. Open the notebook in Colab.
2. Mount the Google Drive for the stored data.
3. Run all cells to reproduce the experiments.

Option 2: Run locally

1. Clone the repository.
2. Run the file either in IDE or bash.

python filename.py

Features

• LOPO and L2PO cross-validation
• Fusion models combining pupil, GSR and FER predictions
• Custom stacked SVR and LGBM model for modalities
• Analysing the regression coefficients
• Exporting results as files

Note

This project is part of the thesis submitted for the degree of Master of Science in Artificial Intelligence.