🔧 Predictive Maintenance — End-to-End MLOps Pipeline

A production-grade Machine Learning system for predictive maintenance using time-series sensor data. Built as a graduation project for the Digital Egypt Pioneers Initiative (DEPI) — Microsoft Partnership Program.

---

📌 Project Overview

This project predicts equipment failure before it happens using historical time-series sensor readings. Instead of waiting for machines to break down (reactive maintenance), the system flags anomalies and failure risks in advance — saving time, money, and operational disruption.

The project goes beyond model training — it implements a full MLOps lifecycle including data pipelines, model serving, containerization, automated retraining, and production monitoring.

---

🏗️ Architecture

Raw Data
   │
   ▼
Data Cleaning & Preprocessing
   │
   ▼
Exploratory Data Analysis (EDA)
   │
   ▼
Model Training (XGBoost + Neural Network)
   │
   ▼
MLflow Experiment Tracking & Model Registry
   │
   ▼
REST API (FastAPI) for Real-Time Inference
   │
   ▼
Docker Containerization
   │
   ▼
Airflow DAG — Automated Retraining Pipeline
   │
   ▼
Prometheus — Model & System Monitoring
   │
   ▼
Data Drift Detection — Evidently AI

---

🚀 Features

Feature	Tool / Technology
Data Cleaning & EDA	Python, Pandas, Matplotlib, Seaborn
ML Model	XGBoost (best_xgb_model.pkl)
Deep Learning Model	Neural Network (Keras/TensorFlow)
Experiment Tracking	MLflow
Model Serving	FastAPI REST API
Containerization	Docker
Automated Retraining	Apache Airflow DAG
Model Monitoring	Prometheus
Data Drift Detection	Evidently AI
Unit Testing	pytest

---

📁 Project Structure

PM-time-series/
│
├── Data/                              # Raw and processed datasets
│
├── predictive_maintenance_EDA.ipynb   # Exploratory Data Analysis
├── predictive_maintenance_cleaning.ipynb  # Data cleaning pipeline
├── predictive maintenance_ml.ipynb    # XGBoost model training & evaluation
├── predictive maintenance_nn.ipynb    # Neural Network model training
│
├── train_pipeline.py                  # Full training pipeline script
├── api_inference.py                   # FastAPI REST API for model inference
├── data_drift.py                      # Data drift detection module
├── retrain_dag.py                     # Airflow DAG for automated retraining
│
├── test_train_pipeline.py             # Unit tests — training pipeline
├── test_api_inference.py              # Unit tests — API inference
├── test_data_drift.py                 # Unit tests — drift detection
│
├── best_xgb_model.pkl                 # Saved XGBoost model
├── scale_xgb.pkl                      # Saved feature scaler
├── feature_importances_xgb.png        # Feature importance visualization
├── predictions.log                    # Inference prediction logs
│
├── Dockerfile                         # Docker container configuration
├── prometheus.yml                     # Prometheus monitoring config
├── requirements.txt                   # Python dependencies
└── index.html                         # Project dashboard

---

⚙️ Getting Started

Prerequisites

• Python 3.8+
• Docker
• Apache Airflow
• MLflow

Installation

# Clone the repository
git clone https://github.com/Marwan-Makram/PM-time-series.git
cd PM-time-series

# Install dependencies
pip install -r requirements.txt

Run the Training Pipeline

python train_pipeline.py

Start the Inference API

python api_inference.py
# API will be available at http://localhost:8000

Run with Docker

# Build the Docker image
docker build -t pm-maintenance .

# Run the container
docker run -p 8000:8000 pm-maintenance

Track Experiments with MLflow

mlflow ui
# Open http://localhost:5000 to view experiments

Run Tests

pytest test_train_pipeline.py
pytest test_api_inference.py
pytest test_data_drift.py

---

📊 Model Performance

Model	Accuracy	Notes
XGBoost	Best performing	Optimized with hyperparameter tuning
Neural Network	Strong baseline	Multi-layer architecture

Feature importance visualization available in feature_importances_xgb.png

---

🔄 Automated Retraining (Airflow)

The retrain_dag.py defines an Apache Airflow DAG that:

1. Detects data drift using Evidently AI
2. Triggers model retraining when drift exceeds threshold
3. Evaluates new model vs production model
4. Promotes new model to production if performance improves
5. Logs all experiments to MLflow

---

📡 Monitoring (Prometheus)

The system exposes metrics via Prometheus including:

• Prediction latency
• Request count
• Model confidence scores
• Data drift scores

---

🧠 What I Learned

• Building end-to-end ML pipelines beyond just model training
• Serving ML models as REST APIs with FastAPI
• Containerizing ML applications with Docker
• Orchestrating ML workflows with Apache Airflow
• Tracking experiments and managing model versions with MLflow
• Detecting and handling data drift in production
• Writing unit tests for ML systems

---

👨‍💻 Author

Marwan Elsayed IT Support Engineer & Machine Learning Practitioner

• LinkedIn: linkedin.com/in/marwan-makram-683630b2
• GitHub: github.com/Marwan-Makram
• Email: marawanmakram159@gmail.com

---

📄 License

This project was developed as part of the Digital Egypt Pioneers Initiative (DEPI) — Microsoft Partnership Program graduation requirements.