In this project, we wish to build a system that automatically colorizes grayscale videos and historical films, in an effort to restore videos to their original colors. We propose a system that deals with two issues in particular: colorizing frames in a video, and optimizing this process across multiple frames. We wish to implement this using a pre-trained CNN model with optimizations for performance over video.
Here is a list of the functionality of each notebook in this repo:
-
scenedetect/SceneDetect.ipynb - Splitting large videos into individual scenes for parallel processing.
-
colorization/Caffe_Colorization_Notebook.ipynb - Implementing base paper's Image Colorization technique.
-
network/LSTMNetwork.ipynb - Encoder + LSTM + Fusion + Decoder based architecture for grayscale video colorization.
-
network/LSTMNetwork_4layers.ipynb - Experimenting on Encoder + LSTM + Fusion + Decoder based architecture using additional hidden layers.
-
network/Inference.ipynb - Plotting graphs for our loss and flicker metrics, and a comparison of our model performance on a sample video across different epochs and model architecture.
-
network/trained_models - Contains our best model from the appraoches we tried.
Create two virtual environments, to avoid dependency mismatches.
conda create -n sd3.7 python=3.7.4
conda activate sd3.7
Install dependencies from scenedetection/scenedetectenv_reqts.txt
pip install -r scenedetection/scenedetectenv_reqts.txt
Run the notebook : scenedetection/SceneDetect.ipynb
conda create -n cc3.6.3 python=3.6.3
conda activate cc3.6
Install dependencies from colorization/working_colorization_reqts.txt
pip install -r colorization/working_colorization_reqts.txt
Run the notebook : colorization/Caffe_Colorization_Notebook.ipynb
conda create -n nn_env python=3.6.3
conda activate nn_env
Install dependencies from network/requirements.txt
pip install -r network/requirements.txt
Run the notebook on Google Colab : network/LSTMNetwork.ipynb
If you're having trouble viewing the notebooks, copy the link to the .ipynb file into Jupyter Notebook Viewer!