IDS 568 - Milestone 3: Workflow Automation & Experiment Tracking

Author: Parth Patel | NetID: ppatel Course: IDS 568 - MLOps (Module 4)

---

Architecture

preprocess.py --> train.py --> model_validation.py
      |               |               |
 Airflow DAG    MLflow Tracking   Quality Gate
      |                               |
GitHub Actions CI/CD ----------> MLflow Model Registry

Stack: Apache Airflow | MLflow | scikit-learn | GitHub Actions | Python 3.11

---

Setup Instructions

1. Clone & Install

git clone https://github.com/ParthPatel0226/ids568-milestone3-ppatel.git
cd ids568-milestone3-ppatel
pip install -r requirements.txt

2. Run Training (standalone)

python train.py                     # single run, no registration
python train.py --register          # run + register to MLflow registry

3. Run MLflow UI

mlflow ui
# Open http://localhost:5000

4. Run Airflow Pipeline

# Initialize Airflow (first time only)
airflow db init
airflow users create --username admin --password admin \
  --firstname Admin --lastname User --role Admin \
  --email admin@example.com

# Start services (two separate terminals)
airflow webserver --port 8080
airflow scheduler

# Trigger DAG via UI at http://localhost:8080
# OR via CLI:
airflow dags trigger train_pipeline

5. Run Quality Gate Manually

python model_validation.py              # validates latest run
python model_validation.py <run_id>     # validates specific run

---

DAG Idempotency & Lineage Guarantees

• Idempotency: Each task re-run produces identical outputs. Preprocessing always re-generates the CSV from source data; training uses fixed random_state=42; MLflow creates a new run ID per execution so no state is ever mutated.

• Lineage: Every MLflow run logs data_hash, model_hash, all hyperparameters, and artifact paths - enabling full trace from production model back to the exact code version and data version used.

---

CI-Based Model Governance

GitHub Actions runs automatically on every push to main:

1. Installs all pinned dependencies from requirements.txt
2. Trains model with default hyperparameters
3. Runs model_validation.py - pipeline fails if accuracy < 0.85 or F1 < 0.85
4. Only models passing the quality gate are eligible for registry promotion

---

Experiment Tracking Methodology

• Experiment name: iris-classification
• Each run logs: model type, all hyperparameters, data version hash, accuracy, F1 score, model artifact, and model SHA256 hash
• Registry stages: None -> Staging -> Production
• Tags used: candidate, data_version, model_hash, dataset

---

Operational Notes

Retry Strategy

• Airflow tasks retry 2 times with a 5-minute delay between attempts
• on_failure_callback logs full context including DAG ID, task ID, execution time, and exception details

Rollback Procedure

from mlflow.tracking import MlflowClient
client = MlflowClient()

# Archive current production version
client.transition_model_version_stage("IrisClassifier", "2", "Archived")

# Restore previous stable version
client.transition_model_version_stage("IrisClassifier", "1", "Production")

Monitoring Recommendations

• Alert if production accuracy drops below 0.85
• Track input feature distribution drift on a weekly basis
• Review MLflow registry monthly to archive stale versions
• Set latency alerts if inference exceeds 100ms

---

Repository Structure

ids568-milestone3-ppatel/
|-- .github/
|   └-- workflows/
|       └-- train_and_validate.yml   # CI/CD pipeline
|-- dags/
|   └-- train_pipeline.py            # Airflow DAG definition
|-- screenshots/                     # MLflow UI evidence
|-- mlruns/                          # MLflow tracking data
|-- preprocess.py                    # Data preprocessing script
|-- train.py                         # Model training + MLflow logging
|-- model_validation.py              # Quality gate (threshold-based)
|-- requirements.txt                 # Pinned dependencies
|-- lineage_report.md                # Experiment analysis + justification
└-- README.md                        # This file

---

Quick Reference: Key Environment Variables

Variable	Default	Description
N_ESTIMATORS	100	Number of trees in Random Forest
MAX_DEPTH	5	Maximum tree depth
MODEL_TYPE	random_forest	Model type (random_forest or logistic)
MLFLOW_EXPERIMENT_NAME	iris-classification	MLflow experiment name
MLFLOW_TRACKING_URI	./mlruns	MLflow tracking server URI