This project implements a spam detection system using machine learning techniques. It processes SMS messages and classifies them as either "spam" or "ham" (legitimate messages) using Natural Language Processing (NLP) and a Naive Bayes classifier.
- Text preprocessing and cleaning
- Feature extraction using TF-IDF
- Classification using Multinomial Naive Bayes
- Performance evaluation
The project uses the "spam.csv" dataset with the following structure:
- v1 (renamed to 'class'): Message classification (spam/ham)
- v2 (renamed to 'sms'): The actual SMS message content
- Python 3.x
- pandas
- nltk
- scikit-learn
- matplotlib (for visualization)
pip install pandas nltk scikit-learn matplotlibThe following NLTK resources are required:
nltk.download('stopwords')
nltk.download('punkt')- Ensure the "spam.csv" dataset is in the same directory as the script
- Run the script to:
- Load and preprocess the data
- Train the Naive Bayes model
- Evaluate the model's performance
- Loads data from "spam.csv" using latin-1 encoding
- Removes unnecessary columns
- Renames columns for clarity
- Removes duplicate entries
- Adds a 'Length' feature that represents the character count of each message
The clean_text() function performs several text preprocessing steps:
- Converts text to lowercase
- Tokenizes the text
- Removes non-alphanumeric characters
- Removes stopwords and punctuation
- Applies stemming to reduce words to their root form
- Uses TF-IDF (Term Frequency-Inverse Document Frequency) to convert text into numerical features
- Limits to the top 3000 most relevant features
- Splits data into training (80%) and testing (20%) sets
- Trains a Multinomial Naive Bayes classifier
- Evaluates the model using accuracy score
The model achieves high accuracy in distinguishing between spam and legitimate messages.
- Experiment with different classification algorithms
- Implement cross-validation
- Add model persistence to save and load trained models
- Create a simple API or UI for real-time spam detection
This project is licensed under the MIT License.