Model and Parameter Uncertainty in Bayesian Neural Networks

In this repository, we provide the code base for producing the results reported in the paper: Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty.

ABSTRACT

Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: Parameter and prediction uncertainties become easily available, facilitating rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, so far, there have been no scalable techniques capable of combining both model (structural) and parameter uncertainty. In this paper, we introduce the concept of model uncertainty in BNNs and hence make inference in the joint space of models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs. As expected, model uncertainties give higher, but more reliable uncertainty measures.

Additionally the research at different stages was presented via the following selected contributions:

Preprint of an early version of the work

Combining Model and Parameter Uncertainty in Bayesian Neural Networks. Hubin, Aliaksandr; Storvik, Geir Olve. 2019.

Scientific lectures and talks

1. Combining Model and Parameter Uncertainty in Bayesian Neural Networks. Hubin, Aliaksandr; Storvik, Geir Olve. 2019, Big Insight Lunch. Oslo, Norway (Invited).

2. Combining Model and Parameter Uncertainty in Bayesian Neural Networks. Hubin, Aliaksandr; Storvik, Geir Olve. 2019, Data Science Major Minsk. Minsk, Belarus. Video (ENG).

3. Combining Model and Parameter Uncertainty in Bayesian Neural Networks. Hubin, Aliaksandr; Storvik, Geir Olve. 2019, CDAM 2019. Minsk, Belarus.

4. Combining Model and Parameter Uncertainty in Bayesian Neural Networks. Hubin, Aliaksandr; Storvik, Geir Olve. 2019, Nordic Probabilistic AI School. Trondheim, Norway (Invited).

5. Combining Model and Parameter Uncertainty in Bayesian Neural Networks. Hubin, Aliaksandr; Storvik, Geir Olve. 2019, CRoNoS & MDA 2019. Limassol, Cyprus (Invited).

6. Combining model and parameter uncertainty in Bayesian neural networks. Storvik, Geir Olve; Hubin, Aliaksandr. 2019, CMStatistics 2019. London, UK (Invited).

7. Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty. Hubin, Aliaksandr; Storvik, Geir Olve. 2021, NORDSTAT 2021. Tromsø, Norway.

8. Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty. Hubin, Aliaksandr; Storvik, Geir Olve. 2021,World Meeting of the International Society for Bayesian Analysis. Online.

Developed by Aliaksandr Hubin and Geir Storvik