Skip to content

README.md

Latest commit

 

History

History
60 lines (40 loc) · 2.7 KB

README.md

File metadata and controls

60 lines (40 loc) · 2.7 KB
Raw

Python MPI Terraform GCP

Distributed Random Forest with MPI & Terraform

A scalable, distributed Random Forest classifier built from scratch using the Message Passing Interface (MPI).

Architecture & Features

  • Custom Distributed Algorithm: A completely custom Random Forest implementation utilizing mpi4py for distributed training and inference.
  • Infrastructure as Code (IaC): Automated provisioning of GCP compute clusters (Master/Worker architecture) using Terraform.
  • Automated Bootstrapping: Bash orchestration scripts that handle SSH key distribution, dependency installation, data partitioning, and MPI hostfile generation.
  • Scaling Benchmarks: Includes automated test suites for both Strong Scaling (fixed data, increasing nodes) and Weak Scaling (increasing data, increasing nodes).

Quickstart

Prerequisites

  • Terraform CLI installed
  • Google Cloud CLI (gcloud) configured (gcloud auth application-default login)
  • Python 3.10+

Running the Pipeline

The primary entry point is the orchestration script, which spins up the infrastructure, distributes the dataset, trains the model via MPI, and aggregates the results before cleanly destroying the cloud resources.

chmod +x scripts/run_experiment.sh

# Usage: ./run_experiment.sh <GCP_PROJECT_ID> <NUM_VMS> <DATA_FRACTION>
./scripts/run_experiment.sh my-gcp-project 5 1.0

Benchmarking & Scaling Experiments

This project is designed to be benchmarked for distributed computing efficiency. You can easily replicate the scaling experiments:

Strong Scaling (Fixed workload, increasing compute resources):

./scripts/run_experiment.sh <PROJECT-ID> 5  1.0 | tee results/strong_5vms.log
./scripts/run_experiment.sh <PROJECT-ID> 10 1.0 | tee results/strong_10vms.log
./scripts/run_experiment.sh <PROJECT-ID> 20 1.0 | tee results/strong_20vms.log

Weak Scaling (Scaled workload, proportional compute resources):

./scripts/run_experiment.sh <PROJECT-ID> 5  0.25 | tee results/weak_5vms.log
./scripts/run_experiment.sh <PROJECT-ID> 10 0.50 | tee results/weak_10vms.log
./scripts/run_experiment.sh <PROJECT-ID> 20 1.00 | tee results/weak_20vms.log

For detailed insights, system architecture diagrams, and performance graphs, please see the Scaling Benchmarks Whitepaper.