My collection of notes on deeplearning.ai's Coursera specialisation. Currently being extended and improved! Please let me know if you have questions or feedback :)
The Deep Learning Specialisation is, in my view, very good for an engineering perspective on DL, surveying the fundamental mechanisms and building blocks that underly a lot of existing techniques, and whose case studies cover 'the classics' to start to demonstrate how to apply them.
What I don't think it covers, as someone who's more theoretically minded, is a thorough coverage of how the principles underlying these systems emerged, explaining why they are (un)reasonably good at what they do.
Initially, my gripes came from there not being a satisfying coverage of the maths, but this has developed into the deeper belief that the (understandable) the probabilistic lens to machine learning is the most enlightening way of framing what these algorithms are trying to do.
Therefore, these notes will attempt to:
- Cover the techniques presented in the specialisation
- Dissect underlying properties of the systems/algorithms that make them suitable for what they're trying to achieve
- Present the probabilistic justifications behind/meanings of certain techniques
I'm also using them as a means to practice reading research, to improve my critical engagement, though I'm not sure how that will emerge in the produced artefacts.
Note: These notes have currently been made to view on Obsidian with the TikzJax and LaTeX Suite community plugins. I will try to improve the structure s.t. they can be more readily used.
(Subject to change as I refactor)
Course 1: Neural Nets and Deep Learning Introduction ML Tasks Backpropagation
Course 2: Improving Deep Neural Nets Experimental Methodology Regularisation Optimisation Initialisation Optimisers BatchNorm Hyperparameter Tuning
Course 3: Structuring ML Projects Structuring ML Projects (I want to flesh this out too but I'm not as concerned with it right now)
Course 4: CNNs CNN Operations Convolutions + Properties Pooling The CNN Case Studies LeNet, AlexNet, VGG ResNets Inception Nets Application: Detection Application: Facial Recognition Application: Neural Style Transfer
Course 5: Sequence Models Intro to RNNs Variations on RNNs LSTM GRU BRNN Intro to Word Embeddings Case Studies GloVe Word2Vec Tasks in NLP (shove Linguistics in here?) Transformers Attention (Bahnadau) - Maybe ill-placed? Multi-Headed Self-Attention Transformer Architecture
These notes take inspiration from lots of sources which I think are very useful for overview. In all honestly, they're certainly each far better than this, and I'd thoroughly recommend you read them and work through them if you appreciate this style of thinking.
- The Deep Learning Specialisation (duh)
- Deep Learning - Goodfellow, Bengio, Courville
- Probabilistic Machine Learning - Murphy
- Pattern Recognition and Machine Learning - Bishop
- Cambridge CST IB Data Science - Wischik. I offer a personal shoutout, absolutely brilliant lecturer who first introduced me to this style of thinking, lectures can be found here
I also thank the researchers whose papers I've included in the repo for all their hard work in progressing the field, and inspiring us learners to also undertake the same duty!
Here are some improvements and additions I'm currently in the process of making:
Course 2: Improving DNNs - Better Explanations
- Loss Function Derivations (MSE, Ridge, Lasso, Cross-Entropy)
- Regularisation as:
- Optimisation
- Bayesian Inference
- Optimiser Algorithms Intuition (Adam, Momentum, RMSProp)
- Batchnorm + Dropout Effectiveness
Course 4: CNNs - Better Explanations
- Mathematical dive into the convolution operation and the invariances it affords
- LeNet, InceptionNet, ResNet, U-Net, YOLO case studies improved
- Refer properly to paper
- Explain limitations and how they were improved on
- Explicitly state implications for architectural design (and why these make sense)
- Clearer statement/presentation of algorithms
Course 5: Sequence Models - Better Explanations
- RNNs, LSTMs, GRUs - explain inductive biases and intuitions behind their working better
- Architectural comparisons
- Tikz diagrams
- Embeddings - Provide better linguistic insight about them about what they're achieving?
- Transformers - improve notes , I've got them handwritten and need to transcribe them.
(Possibly) LaTeX the notes rather than leaving them as md files.