🚀 MLflow on Kubernetes with ArgoCD — Beginner's Guide

A step-by-step tutorial to deploy MLflow on Minikube using GitOps with ArgoCD. No prior Kubernetes experience needed!

---

📚 Table of Contents

1. What Are We Building?
2. Key Concepts
3. Prerequisites
4. Project Structure
5. Step 1 — Start Minikube
6. Step 2 — Create Kubernetes Manifests
7. Step 3 — Deploy MLflow Stack
8. Step 4 — Install ArgoCD
9. Step 5 — Push to GitHub
10. Step 6 — Connect ArgoCD to GitHub
11. Step 7 — GitOps Workflow
12. Accessing the UIs
13. Troubleshooting
14. Glossary

---

What Are We Building?

┌─────────────────────────────────────────────────────────┐
│                     Your Laptop                         │
│                                                         │
│   ┌──────────┐    git push    ┌──────────────────────┐  │
│   │  VS Code │ ────────────► │  GitHub Repo         │  │
│   │ (edit    │               │  (mlflow-gitops)     │  │
│   │  YAMLs)  │               └──────────┬───────────┘  │
│   └──────────┘                          │ auto-sync     │
│                                         ▼               │
│   ┌─────────────────────────────────────────────────┐   │
│   │              Minikube (local K8s cluster)        │   │
│   │                                                  │   │
│   │   ┌──────────┐  ┌──────────┐  ┌─────────────┐  │   │
│   │   │  ArgoCD  │  │  MLflow  │  │  PostgreSQL  │  │   │
│   │   │ (GitOps) │  │   UI     │  │  (metadata) │  │   │
│   │   └──────────┘  └──────────┘  └─────────────┘  │   │
│   │                               ┌─────────────┐  │   │
│   │                               │    MinIO    │  │   │
│   │                               │ (artifacts) │  │   │
│   │                               └─────────────┘  │   │
│   └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

We are deploying a full MLflow machine learning platform on a local Kubernetes cluster (Minikube), managed by ArgoCD using the GitOps pattern. Any change you make to your YAML files and push to GitHub will automatically be applied to your cluster.

---

Key Concepts

🐳 What is Kubernetes (K8s)?

Kubernetes is a system that manages containers (like Docker) across multiple machines. Think of it as an operating system for your cloud. Instead of running docker run, you describe what you want in YAML files and Kubernetes makes it happen.

🖥️ What is Minikube?

Minikube runs a single-node Kubernetes cluster on your laptop. It's perfect for learning and development before deploying to a real cloud.

📦 What is a Pod?

A Pod is the smallest unit in Kubernetes — it's basically a running container (or group of containers). Like a single running instance of your app.

🔁 What is a Deployment?

A Deployment tells Kubernetes how to run your Pods — how many replicas, which image to use, resource limits, etc. If a Pod crashes, the Deployment restarts it automatically.

🌐 What is a Service?

A Service exposes your Pod to the network. Without a Service, your Pod is unreachable from outside. Think of it as a stable address for your Pod.

💾 What is a PersistentVolumeClaim (PVC)?

A PVC is a request for storage. It's how your database saves data even if the Pod restarts — like attaching a hard drive to your container.

🔐 What is a Secret?

A Secret stores sensitive data like passwords and API keys safely in Kubernetes, so you don't hardcode them in your app.

🔀 What is GitOps?

GitOps is a way of managing infrastructure where Git is the single source of truth. You describe your desired state in YAML files, push to Git, and a tool (ArgoCD) automatically applies changes to your cluster.

🔄 What is ArgoCD?

ArgoCD is a GitOps continuous delivery tool for Kubernetes. It watches your Git repo and automatically syncs any changes to your cluster.

🧪 What is MLflow?

MLflow is an open-source platform to manage the machine learning lifecycle — tracking experiments, packaging models, and deploying them.

---

Prerequisites

Install Required Tools

# Install Homebrew (macOS package manager) if you don't have it
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install Minikube (local Kubernetes)
brew install minikube

# Install kubectl (Kubernetes CLI)
brew install kubectl

# Install Helm (Kubernetes package manager)
brew install helm

# Install ArgoCD CLI
brew install argocd

# Install Git
brew install git

Verify installations

minikube version
kubectl version --client
helm version
argocd version --client
git --version

---

Project Structure

mlflow-gitops/
├── README.md                        ← You are here!
├── .gitignore
├── apps/
│   └── mlflow-app.yaml              ← ArgoCD Application definition
└── manifests/
    ├── 00-namespace.yaml            ← Creates the mlflow namespace
    ├── 01-postgres-secret.yaml      ← PostgreSQL credentials
    ├── 02-minio-secret.yaml         ← MinIO credentials
    ├── 03-postgres-pvc.yaml         ← PostgreSQL storage (5Gi)
    ├── 04-minio-pvc.yaml            ← MinIO storage (10Gi)
    ├── 05-postgres-deployment.yaml  ← PostgreSQL database
    ├── 06-minio-deployment.yaml     ← MinIO object storage
    ├── 07-mlflow-deployment.yaml    ← MLflow tracking server
    └── 08-ingress.yaml              ← External access rules

💡 Why numbered files? The numbers ensure files are applied in the correct order — namespace before secrets, secrets before deployments, etc.

---

Step 1 — Start Minikube

# Start minikube with enough resources
minikube start --cpus=4 --memory=8192 --disk-size=40g

# Enable the ingress addon (for external access)
minikube addons enable ingress

# Verify cluster is running
kubectl get nodes

Expected output:

NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   1m    v1.x.x

💡 What just happened? Minikube started a virtual machine on your laptop running a full Kubernetes cluster. The kubectl get nodes command shows the machines (nodes) in your cluster — you have one.

---

Step 2 — Create Kubernetes Manifests

Create the project structure:

mkdir -p mlflow-gitops/manifests mlflow-gitops/apps
cd mlflow-gitops

00-namespace.yaml

# A namespace is like a folder that groups related resources
apiVersion: v1
kind: Namespace
metadata:
  name: mlflow

01-postgres-secret.yaml

# Secrets store sensitive data encoded in base64
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
  namespace: mlflow
type: Opaque
stringData:                    # stringData lets you write plain text (K8s encodes it)
  POSTGRES_USER: mlflow
  POSTGRES_PASSWORD: mlflow123
  POSTGRES_DB: mlflow
  DATABASE_URL: postgresql://mlflow:mlflow123@postgres-service:5432/mlflow

02-minio-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: minio-secret
  namespace: mlflow
type: Opaque
stringData:
  MINIO_ROOT_USER: minioadmin
  MINIO_ROOT_PASSWORD: minioadmin123
  AWS_ACCESS_KEY_ID: minioadmin
  AWS_SECRET_ACCESS_KEY: minioadmin123

03-postgres-pvc.yaml

# PVC = request for persistent storage (survives pod restarts)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: mlflow
spec:
  accessModes:
    - ReadWriteOnce        # Only one pod can write at a time
  resources:
    requests:
      storage: 5Gi         # Request 5 gigabytes of storage

04-minio-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pvc
  namespace: mlflow
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

05-postgres-deployment.yaml

# Deployment = manages running pods
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: mlflow
spec:
  replicas: 1                    # Run 1 copy of this pod
  selector:
    matchLabels:
      app: postgres              # Match pods with this label
  template:
    metadata:
      labels:
        app: postgres            # Label applied to the pod
    spec:
      containers:
      - name: postgres
        image: postgres:15       # Official PostgreSQL image from Docker Hub
        ports:
        - containerPort: 5432
        envFrom:
        - secretRef:
            name: postgres-secret   # Inject all keys from the secret as env vars
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
        readinessProbe:             # K8s checks this before sending traffic
          exec:
            command: ["pg_isready", "-U", "mlflow"]
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:                 # Minimum resources guaranteed
            memory: "256Mi"
            cpu: "250m"
          limits:                   # Maximum resources allowed
            memory: "512Mi"
            cpu: "500m"
      volumes:
      - name: postgres-data
        persistentVolumeClaim:
          claimName: postgres-pvc   # Attach the PVC we created earlier
---
# Service = stable network address for the postgres pod
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
  namespace: mlflow
spec:
  selector:
    app: postgres             # Route traffic to pods with this label
  ports:
  - port: 5432
    targetPort: 5432

06-minio-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: mlflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: minio/minio:latest
        command:
        - minio
        - server
        - /data
        - --console-address
        - ":9001"
        ports:
        - containerPort: 9000    # MinIO API port
        - containerPort: 9001    # MinIO Console UI port
        envFrom:
        - secretRef:
            name: minio-secret
        volumeMounts:
        - name: minio-data
          mountPath: /data
        readinessProbe:
          httpGet:
            path: /minio/health/ready
            port: 9000
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
      volumes:
      - name: minio-data
        persistentVolumeClaim:
          claimName: minio-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
  namespace: mlflow
spec:
  selector:
    app: minio
  ports:
  - name: api
    port: 9000
    targetPort: 9000
  - name: console
    port: 9001
    targetPort: 9001
---
# Job = runs once to completion (creates the mlflow bucket in MinIO)
apiVersion: batch/v1
kind: Job
metadata:
  name: minio-create-bucket
  namespace: mlflow
spec:
  template:
    spec:
      restartPolicy: OnFailure
      initContainers:
      - name: wait-for-minio
        image: busybox
        command: ['sh', '-c', 'until nc -z minio-service 9000; do echo waiting for minio; sleep 2; done']
      containers:
      - name: create-bucket
        image: minio/mc:latest
        command:
        - sh
        - -c
        - |
          mc alias set myminio http://minio-service:9000 minioadmin minioadmin123
          mc mb myminio/mlflow --ignore-existing
          echo "Bucket created successfully"

07-mlflow-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow
  namespace: mlflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mlflow
  template:
    metadata:
      labels:
        app: mlflow
    spec:
      initContainers:            # These run BEFORE the main container starts
      - name: wait-for-postgres
        image: busybox
        command: ['sh', '-c', 'until nc -z postgres-service 5432; do echo waiting for postgres; sleep 2; done']
      - name: wait-for-minio
        image: busybox
        command: ['sh', '-c', 'until nc -z minio-service 9000; do echo waiting for minio; sleep 2; done']
      containers:
      - name: mlflow
        image: ghcr.io/mlflow/mlflow:v2.22.0   # Official MLflow image
        command:
        - mlflow
        - server
        - --host=0.0.0.0
        - --port=5000
        - --backend-store-uri=postgresql://mlflow:mlflow123@postgres-service:5432/mlflow
        - --default-artifact-root=s3://mlflow/
        - --serve-artifacts
        ports:
        - containerPort: 5000
        env:
        - name: MLFLOW_S3_ENDPOINT_URL
          value: http://minio-service:9000     # Tell MLflow to use MinIO as S3
        envFrom:
        - secretRef:
            name: minio-secret
        readinessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 15
          periodSeconds: 10
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: mlflow-service
  namespace: mlflow
spec:
  selector:
    app: mlflow
  ports:
  - port: 5000
    targetPort: 5000
  type: NodePort               # Exposes service on a port on the minikube node

08-ingress.yaml

# Ingress = routes external HTTP traffic to services by hostname
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mlflow-ingress
  namespace: mlflow
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: mlflow.local          # Access MLflow at this hostname
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mlflow-service
            port:
              number: 5000
  - host: minio.local           # Access MinIO console at this hostname
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: minio-service
            port:
              number: 9001

---

Step 3 — Deploy MLflow Stack

# Apply namespace and secrets first
kubectl apply -f manifests/00-namespace.yaml
kubectl apply -f manifests/01-postgres-secret.yaml
kubectl apply -f manifests/02-minio-secret.yaml
kubectl apply -f manifests/03-postgres-pvc.yaml
kubectl apply -f manifests/04-minio-pvc.yaml

# Deploy PostgreSQL and wait for it
kubectl apply -f manifests/05-postgres-deployment.yaml
kubectl wait --for=condition=ready pod -l app=postgres -n mlflow --timeout=120s

# Deploy MinIO and wait for it
kubectl apply -f manifests/06-minio-deployment.yaml
kubectl wait --for=condition=ready pod -l app=minio -n mlflow --timeout=120s

# Deploy MLflow
kubectl apply -f manifests/07-mlflow-deployment.yaml
kubectl wait --for=condition=ready pod -l app=mlflow -n mlflow --timeout=180s

# Apply ingress
kubectl apply -f manifests/08-ingress.yaml

# Verify everything is running
kubectl get all -n mlflow

---

Step 4 — Install ArgoCD

# Create ArgoCD namespace
kubectl create namespace argocd

# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for ArgoCD server to be ready
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-server \
  -n argocd --timeout=300s

# Get the auto-generated admin password
kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath="{.data.password}" | base64 -d && echo

💡 Save that password! You'll need it to log into the ArgoCD UI.

# In Terminal 1 — keep this running to access ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

Open https://localhost:8080 → login with admin / <your-password>

# Login via CLI (Terminal 2)
argocd login localhost:8080 \
  --username admin \
  --password <your-password> \
  --insecure

---

Step 5 — Push to GitHub

Create GitHub Repository

1. Go to https://github.com/new
2. Repository name: mlflow-gitops
3. Visibility: Public or Private
4. Click Create repository

Generate a Personal Access Token (PAT)

ArgoCD needs this to read your repo:

1. Go to https://github.com/settings/tokens/new
2. Note: argocd-mlflow
3. Expiration: 90 days
4. Scopes: check ✅ repo
5. Click Generate token → copy the ghp_... token

Push Your Code

cd mlflow-gitops

git init
git add .
git commit -m "feat: initial mlflow gitops setup"
git branch -M main
git remote add origin https://github.com/<your-username>/mlflow-gitops.git
git push -u origin main

---

Step 6 — Connect ArgoCD to GitHub

Create ArgoCD Application manifest

cat > apps/mlflow-app.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: mlflow
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/<your-username>/mlflow-gitops
    targetRevision: main
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: mlflow
  syncPolicy:
    automated:
      prune: true       # Remove resources deleted from Git
      selfHeal: true    # Fix manual changes made directly to cluster
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
EOF

Add Repo to ArgoCD

argocd repo add https://github.com/<your-username>/mlflow-gitops \
  --username <your-github-username> \
  --password <your-ghp-token>

# Verify connection
argocd repo list

Deploy via ArgoCD

kubectl apply -f apps/mlflow-app.yaml

# Watch sync status
argocd app get mlflow

# Manually trigger sync
argocd app sync mlflow

---

Step 7 — GitOps Workflow

This is the power of GitOps. Git is your source of truth.

Edit YAML → git push → ArgoCD detects change → Auto-applies to cluster

Example: Scale MLflow to 2 replicas

# Edit the deployment
vim manifests/07-mlflow-deployment.yaml
# Change replicas: 1 → replicas: 2

# Commit and push
git add manifests/07-mlflow-deployment.yaml
git commit -m "scale: increase mlflow replicas to 2"
git push

# ArgoCD auto-syncs within ~3 minutes
# Or trigger manually:
argocd app sync mlflow

Useful ArgoCD Commands

# List all apps
argocd app list

# Get app details and sync status
argocd app get mlflow

# Manually sync
argocd app sync mlflow

# View deployment history
argocd app history mlflow

# Rollback to a previous version
argocd app rollback mlflow <revision-number>

# Delete app and all its resources
argocd app delete mlflow

---

Accessing the UIs

Run each in a separate terminal:

# Terminal 1 — ArgoCD
kubectl port-forward svc/argocd-server 8080:443 -n argocd

# Terminal 2 — MLflow
kubectl port-forward svc/mlflow-service 5000:5000 -n mlflow

# Terminal 3 — MinIO
kubectl port-forward svc/minio-service 9001:9001 -n mlflow

Service	URL	Username	Password
ArgoCD	https://localhost:8080	admin	(generated)
MLflow	http://localhost:5000	—	—
MinIO	http://localhost:9001	minioadmin	minioadmin123

Test MLflow with Python

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-first-experiment")

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("epochs", 10)
    mlflow.log_metric("accuracy", 0.95)
    mlflow.log_metric("loss", 0.05)
    print("Run logged successfully!")

---

Troubleshooting

Pods stuck in ImagePullBackOff

# Check what's wrong
kubectl describe pod <pod-name> -n mlflow | grep -A 20 "Events:"

# Pre-pull images into minikube
eval $(minikube docker-env)
docker pull postgres:15
docker pull minio/minio:latest
docker pull ghcr.io/mlflow/mlflow:v2.22.0

Pods stuck in Pending

# Usually a resource issue — check node capacity
kubectl describe nodes minikube | grep -A 10 "Allocated resources"

# Restart minikube with more resources
minikube stop
minikube start --cpus=4 --memory=8192

ArgoCD shows OutOfSync

# Force a fresh sync
argocd app sync mlflow --force

Can't reach MLflow UI

# Check if pod is running
kubectl get pods -n mlflow -l app=mlflow

# Check pod logs
kubectl logs -n mlflow -l app=mlflow

# Re-run port-forward
kubectl port-forward svc/mlflow-service 5000:5000 -n mlflow

Reset everything and start fresh

kubectl delete namespace mlflow
kubectl delete namespace argocd
minikube stop
minikube delete
minikube start --cpus=4 --memory=8192

---

Glossary

Term	What it means
K8s	Short for Kubernetes (8 letters between K and s)
Pod	Smallest deployable unit — one or more containers
Deployment	Manages pods, handles restarts and scaling
Service	Stable network address for pods
Namespace	Virtual cluster to group resources
PVC	PersistentVolumeClaim — request for storage
Secret	Encrypted storage for passwords/keys
Ingress	Routes external HTTP traffic to services
ConfigMap	Non-sensitive config data for pods
Node	A machine (VM or physical) in the cluster
Cluster	Group of nodes managed by Kubernetes
Helm	Package manager for Kubernetes
ArgoCD	GitOps tool that syncs Git → Kubernetes
GitOps	Using Git as the source of truth for infrastructure
Minikube	Local single-node Kubernetes cluster
MLflow	ML lifecycle management platform
MinIO	S3-compatible object storage (for ML artifacts)
manifest	A YAML file describing a Kubernetes resource
kubectl	Command-line tool to interact with Kubernetes
port-forward	Tunnel from your laptop to a pod/service in the cluster

---

What's Next?

Once you're comfortable with this setup, explore:

• 🔐 Add authentication to MLflow using a reverse proxy
• 📊 Prometheus + Grafana for monitoring your cluster
• 🔒 Sealed Secrets to safely commit secrets to Git
• ☁️ Deploy to a real cloud (GKE, EKS, AKS) using the same manifests
• 🏗️ Kustomize or Helm to manage multiple environments (dev/staging/prod)
• 🔔 Webhook triggers so ArgoCD syncs instantly on every git push

---

Built with ❤️ for K8s beginners. Happy learning!