ST5230 Assignment 1

Language Modeling and Representation Learning

This repository contains the implementation for ST5230 Applied Natural Language Processing – Assignment 1.
The project trains several language models on the IMDB dataset, compares their performance, studies embedding variants, and evaluates learned representations on a downstream sentiment classification task.

All models are implemented and trained using PyTorch.

---

Part I: Language Model Training

Code Location

Model implementations: src/models/ngram.py src/models/rnn_lm.py src/models/lstm_lm.py src/models/transformer_lm.py

Training script: src/training/train_lm.py

Evaluation script: src/training/evaluate.py

Configuration files: configs/rnn.yaml configs/lstm.yaml configs/transformer.yaml

---

Results

Example test perplexities:

Model	Test Perplexity
Bigram	866
Trigram	5753
RNN	150
LSTM	97
Transformer	80

Generated text samples are saved in: experiments/results/

Training logs are saved in: experiments/logs/

---

Part II: Embedding Variants

Code Location

Word2Vec training: src/embeddings/train_word2vector.py

Embedding matrix construction: src/embeddings/build_embedding_matrix.py

Embedding experiments are executed using modified config files in: configs/

---

Results

Example test perplexities:

LSTM

Embedding Type	Test PPL
Trainable	97
Self-trained Word2Vec	96
Pretrained GloVe	107

Transformer

Embedding Type	Test PPL
Trainable	89
Self-trained Word2Vec	98
Pretrained GloVe	106

---

Part III: Downstream Sentiment Classification

Code Location

Downstream classifier training: src/downstream/train_classifier.py

This script loads a trained Transformer language model and extracts representations for sentiment classification.

---

Results

Setting	Accuracy	F1
Mean pooling + freeze	0.784	0.783
Last hidden state + freeze	0.770	0.761
Mean pooling + fine-tune	0.822	0.827

---

Running the Code

Train a language model: python -m src.training.train_lm --config configs/lstm.yaml

Evaluate a trained model: python -m src.training.evaluate --config configs/lstm.yaml --ckpt experiments/checkpoints/lstm_best.pt

Train Word2Vec embeddings: python -m src.embeddings.train_word2vector

Run the downstream sentiment classifier: python -m src.downstream.train_classifier