Effective dimension in PyTorch

Implementation test of the [Abbas' et al] paper available on arXiv. The initial implementation is available at amyami187/effective_dimension. The goal of this metric is to estimate the size of that model occupies in the model's parameters space and checks the generalisation error of the evaluated model.

About the local effective dimension (go check the implementation at amyami187/local_effective_dimension), its goal is the same as the global effective dimension but it only analyzes the training result and not the large number of parameters set and can be applied to Machine Learning models too.

⚠️ Important note : when the effective dimension is printed, more it is, better is the model generalization.

The NNGeometry package allow to efficiently compute the Fisher information matrix with a Kronecker-factored approximation technique to succeed calculations without storing all the Fisher matrix in memory.

Important : on the example notebook, some math explanation (to see the simplification in the paper) is available so you can go check that !

Table of contents

• Table of contents
• Setup
• Commands
• License

Setup

You need to have at least Python 3.8 installed on your machine to run the code.

Without environment

You can install the required packages with this command :

$ pip3 install numpy scipy, torch, nngeometry # at least
$ pip3 install torchvision # previous command + this one to run the MNIST example
$ pip3 install jupyterlab # previous commands + this one to run MNIST example with Jupyter Notebook

Note : for Windows systems, change pip3 to pip.

With environment

You can simply install the packages with this command (make sure that pipenv is already installed) :

$ pipenv install
$ pipenv shell
$ pip install torchvision # previous commands + this one to run the MNIST example
$ pip install jupyterlab # previous commands + this one to run MNIST example with Jupyter Notebook

Commands

Make sure that your environment is activated if you choose to use it.

• To run the script example :

$ python example/scripts/mnist.py

• To run the notebook example :

$ jupyter lab

And go notebooks/effective_dimension.ipynb or you can run it via Google Colab.

License

Project under the Apache-2.0 license (same as amyami187/effective_dimension and amyami187/local_effective_dimension repositories).