TweetEval Three-Class Sentiment Classification

An applied NLP experiment classifying tweets as negative, neutral, or positive with reproducible feature pipelines and class-level evaluation.

Project detail	Description
Academic context	DIGI405 Texts, Discourses and Data, University of Canterbury
Task	Three-class sentiment classification on CardiffNLP TweetEval
Selected model	Logistic Regression with 1,000 token unigrams and VADER lexicon features
Selection metric	Macro F1 on the unchanged validation split
Core tools	Python, Hugging Face Datasets, textplumber, scikit-learn, imbalanced-learn, VADER

Executive Summary

This project evaluates sentiment as a three-class problem rather than removing the difficult neutral category. Eight experiments compare model family and feature representation: smaller and larger unigram vocabularies, token bigrams, stop-word decisions, VADER sentiment lexicons, character n-grams, and two Decision Tree depths.

The training split is imbalanced, so it is deterministically under-sampled to 7,093 observations per class. The official validation split remains unchanged. Macro F1 is used for selection so strong performance on the largest class cannot hide weak performance elsewhere.

Evidence Status

The eight experiment results and selected-model metrics are taken from Sahil Mann's completed DIGI405 notebook and final report. The portfolio package does not recreate or redistribute TweetEval data locally.

The clean companion notebook preserves the documented experiment table and rebuilds the selected pipeline against CardiffNLP's official dataset when a reader chooses to run it. Reported scores are labelled as completed-project results rather than as a new execution.

Recorded Results

Model result	Score
Accuracy	0.626
Macro F1	0.606
Negative F1	0.499
Neutral F1	0.618
Positive F1	0.702

The selected fifth run narrowly outperformed the larger token-plus-character feature set. Positive sentiment was easiest to recognise, while negative sentiment was the weakest class. Neutral remained a substantive modelling challenge rather than a category to discard.

Reproducible Workflow

1. Download the public sentiment subset of cardiffnlp/tweet_eval at runtime.
2. Preserve TweetEval's official train and validation split.
3. Balance only the training data with RandomUnderSampler(random_state=0).
4. Clean text and preprocess with the course's textplumber pipeline.
5. Combine 1,000 token unigram counts with VADER lexicon counts.
6. Fit LogisticRegression(max_iter=5000, random_state=42).
7. Report accuracy, macro F1, class-level metrics, and a row-normalised confusion matrix.
8. Reconcile a fresh run with the metrics recorded during the completed project.

Repository Structure

.
|-- README.md
|-- requirements.txt
|-- assets/
|   |-- tweeteval_model_comparison_macro_f1.png
|   `-- tweeteval_selected_model_diagnostics.png
|-- notebooks/
|   `-- tweeteval_sentiment_classification.ipynb
`-- validation/
    |-- reported_model_runs.csv
    `-- fresh_run_reconciliation.csv          # generated on execution

Reproduce the Analysis

Use Python 3.12, matching the completed project environment.

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m spacy download en_core_web_sm
jupyter lab notebooks/tweeteval_sentiment_classification.ipynb

The first full run downloads TweetEval and may create a local SQLite feature cache. Neither raw tweet text nor the feature cache is committed.

Data and Responsible Use

Dataset: CardiffNLP TweetEval sentiment subset, introduced in the TweetEval benchmark paper.

The repository does not redistribute tweet text. Text is downloaded by the dataset library for computation and is never exported by the notebook. The model should not be used to infer a person's beliefs, wellbeing, or emotional state.

Limitations

• Random under-sampling improves balance but discards training observations.
• Short posts often rely on conversational context, irony, or shared knowledge absent from the text.
• Annotation conventions and platform language can drift over time.
• Dependency and tokenizer updates can cause small reproducibility differences, which the notebook reports explicitly.
• Benchmark performance does not establish suitability for consequential decisions.