Image Captioning

Last updated: 02/06/2022

Image Captioning

This project is developed from Udacity's Computer Vision Nanodegree

Project Summary

1. Use a hybrid CNN-RNN model to automatically generate descriptive text captions for images (MS COCO Dataset)
2. Train a ResNet encoder to extract and embed the semantic features from images and an LSTM decoder to output captions
3. Achieve BLEU score = 15.98 (1-gram score = 60.55) on validation dataset
4. Use the model to predict captions from custom input images

Future Work

1. Use Vision Transformer to serve as the encoder to extract features from the images
2. Use Transformer to serve as the decoder to generate captions

Dataset

I use the Microsoft Common Objects in Content (MS COCO) Dataset (Ver. 2014)

Link to the MS COCO Dataset: https://cocodataset.org/#home

MS COCO Dataset (https://cocodataset.org/#home)

How to Download MS COCO Dataset

This projects use the COCO API provided by the MS COCO. Instructions in detail can be found on their websites. Here is a brief instruction:

1. In your work directory, create a folder opt
2. In the opt folder, run the bash command git clone https://github.com/cocodataset/cocoapi.git
3. Download 2014 Train/Val annotations [241MB] from MS COCO download page
4. Extract the zip file annotation_trainval2014.zip into the opt/cocoapi folder
5. (Checkpoint) You should see a folder annotations inside the opt/cocoapi folder
6. Download 2014 Testing Image info [1MB] from MS COCO download page
7. Extract the zip file image_info_test2014.zip into the opt/cocoapi folder
8. (Checkpoint) You should see a file image_info_test2014.json inside the opt/cocoapi/annotations folder
9. In your work directory, create a folder images
10. Download 2014 Train images [83K/13GB] from MS COCO download page
11. Download 2014 Val images [41K/6GB] from MS COCO download page
12. Download 2014 Test images [41K/6GB] from MS COCO download page
13. Extract all 3 downloaded zip files (steps 10-12) into the images folder
14. (Checkpoint) You should see 3 folders (train2014, val2014, test2014) inside the images folder

Model

The model consists of an encoder an a decoder. The encoder extract semantic information from the input image to generate a feature vector.

A hybrid ResNet-LSTM model for image captioning

Encoder

I use a pre-trained ResNet-50 network to extract the features from an image. I removed the last fc layer, flattened the final output and pass through a dense layer to obtain a feature vector of size EMBED_SIZE

Decoder

I use an LSTM network as the decoder. I train the model from scratch. The dimension of the hidden units of the LSTM layer is HIDDEN_SIZE, which is a hyperparameter.

Hyperparameters

VOCAB_THRESHOLD = 5
BATCH_SIZE = 32
EMBED_SIZE = 512
HIDDEN_SIZE = 512
LR = 1e-3

Evaluation

I train the model for 5 epochs and use the loss on the validation dataset to choose the best model weights. I use the BLEU Score and the loss to evaluate the model performance.

The model achieves BLEU Score = 17.04, and an overall loss (cross entropy) of 2.3076

Inference

I used the ResNet-LSTM hybrid model to predict captions on the MS COCO Testing Data.

Good and bad captions generated by the model. Images are from MS COCO Test Dataset (2014).

References

1. Udacity's Computer Vision Nanodegree
2. COCO API: https://github.com/cocodataset/cocoapi
3. This notebook tells how to download the COCO Dataset https://colab.research.google.com/github/rammyram/image_captioning/blob/master/Image_Captioning.ipynb
4. Google's paper using LSTM for image captioning https://arxiv.org/pdf/1411.4555.pdf