Bayesian Neural Networks on CIFAR-10

MSc Deep Learning Coursework

Overview

This project compares two approximate Bayesian inference techniques for image classification on CIFAR-10 against a deterministic baseline:

1. Deterministic ResNet20 — standard ResNet20 (274K parameters) trained for 200 epochs with SGD, momentum 0.9, and step-decay learning rate schedule.
2. MC Dropout — same architecture with dropout (rate 0.1) inserted inside each residual block, kept active at inference to enable Monte Carlo sampling from an approximate posterior.
3. SWAG (Stochastic Weight Averaging Gaussian) — fits a low-rank-plus-diagonal Gaussian posterior from weight snapshots collected over 30 additional training epochs starting from the pre-trained deterministic model.

Key Results

Model	Test NLL	Test Accuracy	ECE
Deterministic ResNet20	0.5485	90.80%	0.0480
MC Dropout (S=30)	0.2785	91.49%	0.0276
SWAG (S=30)	0.3256	91.22%	0.0411

Both Bayesian methods improve calibration and NLL while maintaining comparable accuracy. MC Dropout achieves the best aggregate metrics with minimal overhead, while SWAG provides tighter calibration in the high-confidence region.

Repository Contents

• 01703570.ipynb — Full implementation notebook (data loading, model training, SWAG collection, evaluation, and reliability diagrams)
• report.pdf — Written report discussing experimental setup, results, calibration analysis, and practical findings
• 01703570.pdf — PDF export of the notebook

Setup

The notebook was developed to run on Google Colab (T4 GPU). Key dependencies:

• TensorFlow / Keras
• NumPy, Matplotlib
• CIFAR-10 (auto-downloaded via keras.datasets)