A Rigorous Empirical Evaluation of Machine Learning and Deep Learning Models for Question-Answer Classification

📌 Project Overview

This project presents a comprehensive empirical study on classifying Question-Answer (QA) pairs into ten predefined categories. The objective was to evaluate a wide spectrum of Machine Learning (ML) and Deep Learning (DL) models to find the optimal architecture for understanding the nuances of QA text.

We investigated four primary text representation techniques (BoW, TF-IDF, Word2Vec, GloVe) and trained various architectures ranging from classical baselines to complex Recurrent Neural Networks (RNNs, LSTMs, GRUs).

📄 Abstract

• Dataset: 339,998 Question-Answer pairs.
• Categories: 10 distinct classes.
• Best Model: Unidirectional GRU with GloVe embeddings.
• Top Accuracy: 72.68%.

The study highlights that while Long Short-Term Memory (LSTM) units are powerful, the Gated Recurrent Unit (GRU) provided a more streamlined and effective architecture for this specific task, outperforming both classical methods and other deep learning configurations.

🛠️ Methodology

1. Preprocessing Pipeline

We implemented a rigorous cleaning pipeline using NLTK:

• Tokenization & Lowercasing: Standardizing text format.
• Stopword Removal: Filtering out non-informative words.
• Lemmatization: Reducing words to their base roots (e.g., "running" -> "run").

2. Feature Representation

We evaluated four distinct feature extraction techniques:

• Sparse Representations: Bag-of-Words (BoW) and TF-IDF.
• Dense Embeddings: * Word2Vec (Skip-gram): Custom-trained on our corpus.
  • GloVe: Pre-trained global vectors (glove.6B.100d).

3. Model Architectures

• Classical Baselines: Logistic Regression, Multinomial Naive Bayes, Random Forest.
• Deep Neural Networks (DNN): Feedforward networks with ReLU activation and Dropout.
• Recurrent Networks:
  • Simple RNN (Uni/Bidirectional)
  • LSTM (Uni/Bidirectional)
  • GRU (Uni/Bidirectional)

📊 Key Results

The models were evaluated based on Accuracy, F1-Score, and Recall. The table below summarizes the performance of key configurations:

Model Architecture	Feature Representation	Accuracy
GRU (Unidirectional)	GloVe Embeddings	72.68%
GRU (Bidirectional)	Skip-gram	72.13%
LSTM (Bidirectional)	GloVe Embeddings	66.59%
Logistic Regression	TF-IDF	68.36%
Random Forest	TF-IDF	65.23%
Simple RNN	GloVe Embeddings	43.12%

💻 Installation & Requirements

To replicate this study, you will need Python installed along with the following libraries:

pip install tensorflow scikit-learn nltk numpy pandas matplotlib