Distributed IMDb Sentiment Analysis & Text Preprocessing Pipeline via PySpark

An enterprise-grade, distributed Natural Language Processing (NLP) pipeline designed to ingest, decontaminate, feature-engineer, and classify large-scale textual review datasets. Built on top of Apache PySpark, this repository provides a parallelized, data-cleansing framework that transitions raw, noisy web-scraped reviews into dense machine-learning vectors, executing a distributed binary classification workflow optimized for cluster deployments.

---

Architectural Blueprint

The pipeline establishes a strict linear DAG (Directed Acyclic Graph) workflow, processing unstructured review records across isolated nodes efficiently:

[ Raw IMDb CSV File ] ---> Ingested into Distributed PySpark DataFrames
|
v
+-------------------------------+
|  Text Decontamination Stage   |
|  * Multi-Pattern Regex Pass   | ---> Strips HTML tags, web URLs,
|  * Special Character Purging  |      and escape string blocks
+---------------+---------------+
|
v
+-------------------------------+
|  Linguistic Wrangling Stage   |
|  * Lowercase Normalization    | ---> Drops baseline vocabulary footprint,
|  * Stopword Filtering & Length|      isolates empty string fragments
+---------------+---------------+
|
v
+-------------------------------+
|   Distributed Vectorization   |
|  * Tokenization Splitting     | ---> Suppresses high-frequency background noise,
|  * HashingTF + IDF Scaling    |      accentuates rare semantic signals
+---------------+---------------+
|
v
+-------------------------------+
|  Parallelized Classifier Loop |
|  * Train/Test Evaluation Split| ---> Executes distributed optimization matrix,
|  * Binary Prediction Output   |      yields final ROC-AUC score
+-------------------------------+

1. Ingestion & High-Performance Cleansing

• Distributed Schema Parsing: Imports large-scale multi-column review entries natively into a partitioned PySpark DataFrame, managing data quality distributions programmatically.
• Web-Noise Decontamination: Deploys unified relational expressions (regexp_replace) to scrub raw inputs, parsing out nested HTML breaks (<br />), structural URL link references, layout markers, and complex special character matrices.

2. Feature Engineering & Lemmatization Pipeline

• Vocabulary Normalization: Transforms text inputs to lowercase to preserve spatial token symmetry, maps overall character length parameters, and excludes invalid null records.
• Stopword Pruning Loop: Leverages native StopWordsRemover models to isolate and discard recurrent functional words (e.g., 'the', 'a', 'is'), avoiding feature bloat and ensuring training focus rests entirely on structural linguistic indicators.
• Lemmatization Operations: Groups varying morphological inflection styles down to structural core bases (e.g., 'movies' vs. 'movie' -> 'movie'), keeping feature arrays compact and boosting downstream classifier accuracy.

3. Big Data Scaling & Model Classification

• Distributed HashingTF-IDF: Implements PySpark's execution of term frequency tracking coupled with Inverse Document Frequency adjustment weights, automatically lowering token parameters for cross-document background terms while boosting critical predictive triggers.
• Supervised Predictive Training: Partitions processed text frames into a deterministic train/test split configuration, running parallel classification passes directly across independent compute blocks.

---

Technical Stack & Libraries

• Core Execution Engine: Apache PySpark (v3.x+)
• Machine Learning Interface: PySpark MLlib (Feature, Classification, Evaluation namespaces)
• Natural Language Parsers: NLTK (Stopword and Tokenization corpus mappings)
• Analytical Workspace Environment: Jupyter Notebook / Google Colab
• Data Visualization Extensions: Matplotlib, Seaborn, and WordCloud

---

System Processing Sequence:

1. SparkSession Initialization: Establishes localized driver node orchestration vectors, configuring resource allocation boundaries.
2. Transform Operations Pass: Executes lazy-evaluation cleansing steps, processing text blocks down to normalized, tokenized words.
3. Model Validation Performance: Fits features through the TF-IDF array, computes matrix states, and prints classification reports directly to standard logging devices.

---

Summary of Inferences

• Analysis of review length shows that positive reviews are, on average, slightly longer (844 chars) and contain more words (122 words) than negative reviews (811 chars / 119 words).

• N-gram analysis is highly effective at finding predictive phrases. o Some interesting frequent trigrams of "positive" reviews include: "well worth watching", "seen long time", "one best movie", "one best film", "best movie ever", "based true story", "highly recommend movie" o Some interesting frequent trigrams of "negative" reviews include: "worst movie ever", "movie ever seen", "dont waste time", "one worst movie", "movie ive ever", "worst film ever", "doesnt make sense", "bad acting bad", "complete waste time"

• Importance of data cleaning & pre-processing:

Some interesting visualisations:

• Distribution of review length by sentiment

• Word cloud for positive review lemmas

• Word cloud for negative lemmas

Lemmatization: This is a crucial feature engineering step. Its importance is:

• Reduces vocabulary size: It consolidates words (e.g., "movie," "movies") into one base form ("movie"), making the feature vector smaller and more efficient.
• More accurate TF-IDF scores: By consolidating all variations of the word, it increases the frequency and TF-IDF score of the core lemma.
• Improves Accuracy: It helps the model generalize by treating different forms of a word as the same, strengthening the pattern between a phrase and its sentiment.

TF-IDF vectorisation:

• To Feed the model: PySpark ML model cannot process words/lemmas directly. It requires numerical representation.
• To give importance to the signal, not the noise: TF-IDF automatically punishes the noise (common) words (low IDF score) and boosts the signal (rare/unique) words (high IDF score).

PySpark Modelling:

• Test Set Area Under ROC (AUC): 87.57%