Distributed, Arbitrary and Heterogeneous Learning
DAHL is a machine learning framework, developed as part of my thesis, which aims for the reduction of AI training electricity consumption.
Abstract: Artificial intelligence is transforming computing and our daily lives. Yet it also represents threats in terms of energy consumption and indirect ecological impact such as carbon emissions, resource depletion, clean water usage etc. Nowadays, the mainstream approach for training cutting- edge models is to distribute the workload across numerous homogeneous machines. Indeed, parallelizing over processing units with equivalent computing power is currently the best method to maximize the computation throughput. Nevertheless, it induces strong hardware constraints, requires costly investments, and is arguably not sustainable in the long term. With DAHL, our goal is to focus efforts on overcoming scientific and software limitations in supporting heterogeneous platforms, in place of hardware scaling methods. Current limitations are due to synchronization steps mandatory for updating model parameters. Those steps introduce waiting time that are exacerbated by heterogeneous platforms, representing longer run-time and energy loss. We believe that supporting heterogeneity in AI frameworks could help enhancing existing infrastructure and reusing existing machines.
The first step is to implement a machine learning framework that is able to handle heterogeneity at the machine level. We also want to support arbitrary execution of operations in order to develop energy-aware load balancing algorithms.
In a second time, we would like to expand our framework to multiple machines.
Under the hood, StarPU is used to support heterogeneous computing at a single machine level.
- dahl/ DAHL source
- datasets/ Datasets directory with download scripts
- doc/ Documentation, notes, slides
- experiments/ Experiment script folders
- nix/ Nix environment
- python-version/ Pytorch implementation of DAHL's CNN
- scripts/ Various scripts
- flake.nix Nix flake that provide a reproducible development environment
This project can be built with Nix
Enter the development environment
nix developCompiling
cd dahl
mkdir build
cd build
cmake ..
makeTry the tests to see if it works with ./tests/
The first available example is a CNN trained with the fashion mnist dataset
Download the dataset, see datasets/
cd ./datasets
./download_fashion_mnist.bashTrain the CNN
cd build
./basic_cnn ../fashion-mnist/train-images-idx3-ubyte ../fashion-mnist/train-labels-idx1-ubyteThanks tsoding for the Arena implementation.