Parallelization of Local Learning Rules

Origin

This is a repository developed for the bachelor thesis of ikhovind and erlinssl. It is an extension of a machine learning article written by our supervisor Ole Christian Eidheim. In connection to said article, Eidheim developed an algorithm in Python, and the thesis concerns translating this algorithm into C++, optimizing it and finally GPU-parallelize it. The group then gathered data about the general calculations, and attempted to draw conclusion about the potentials of GPU-acceleration based on the results.

Filters produced by main:	Filters produced by threads-v1:	Filters produced by compute:

Results

We have implemented several different versions of the original Python algorithm, which are all spread across the different branches in this project. These are the main ones:

branch	description
main	Single threaded implementation of the algorithm in plain C++
threads-v1	Multi-threaded implementation parallelizing part of the samples being run, offering the greatest CPU speed but sacrificing accuracy, therefore finding fewer filters than other implementations
threads-v2	Multi-threaded implementation parallelizing the calculation of filter values, sacrifices no accuracy
arrayfire-revised	Implementation using the open source library ArrayFire, let's you choose whether to run on the CPU or on the GPU with CUDA or OpenCL; arrayfire-cpu/cuda/opencl branch out of this branch, and were mainly used to enforce the usage of each specific technology
compute	Implementation using Boost Compute, a thin wrapper over OpenCL; the fastest GPU-accelerated version
arrayfire-cifar	An attempt at using the CIFAR-10 dataset instead of MNIST; the team was not able to find filters using this iteration, but has left the branch here for further work purposes

The actual scientific results of the research is contained within the thesis itself, which is published on NTNU Open.

Dependencies

The dependencies of the program depends on the version used. All versions are dependent on matplotlib-cpp in order to display the final calculation. The ArrayFire branches are dependent on ArrayFire and compute is dependent on Boost Compute. The arrayfire-cifar branch also makes use of a CIFAR-10 dataset reader. Each dependency can be found as a git submodule in each relevant branch.

Running

If you want to run the program on your own system you will need:

1. C++20. The program can likely be easily rewritten to at least C++17 if not C++11 or lower, but out of the box it relies on C++20 functionality.

2. A copy of the MNIST dataset is needed; it should be named: train-images-idx3-ubyte and placed in a directory named data at the top level of this repository. See the configure_file() command in CMakeLists.txt for help

   1. If you wish to experiment with the arrayfire-cifar branch, you will naturally need the CIFAR-10 dataset instead; the extracted 'cifar-10-batches-bin' folder should be placed within the 'cifar-10' submodule directory

3. You will have to make sure you have Python 3 installed

4. Install NumPy and Matplotlib

5. If you want to run ArrayFire you will need to install ArrayFire

6. If you want to run compute you will need to install Boost Compute

A CMake file is provided, which should handle linking the given dependencies, given that you have them installed on your system.

When using ArrayFire, the desired backend is set in CMakeLists.txt as specified in the official documentation.

In order to run the actual script, you can build the program with CMake, here is an example on a Linux system, starting from the top level of this repository:

mkdir build
cd build
cmake ..
make
./filter_finder

This command will run the default experiments for the given branch. A specific experiment can be run by setting the parameters as command line arguments:

./filter_finder 1 0.5 1000 4 1000 5 0.1

The parameters are as follows, all 7 need to be set, otherwise all of them revert back to their standard values:

./filter_finder sigma lambda num_batches grid_size batch_size resolution learning_rate

The number of samples are decided by num_batches and batch_size, grid size is the square root of the maximum number of filters you want to find simultaneously, meaning that a value of 4 will create 4 ** 2 = 16 neurons, 5 will create 25 and so on. The rest of the parameters are described in Eidheim's original article.