Skip to content

About

LLM 파인튜닝부터 vLLM 서빙, 성능 벤치마크까지 하나의 API로 통합한 End-to-End MLOps 플랫폼

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

LLM Ops Platform

LLM 파인튜닝부터 vLLM 서빙, 성능 벤치마크까지 하나의 API로 통합한 End-to-End MLOps 플랫폼. FastAPI Gateway, Celery 학습 파이프라인, MLflow 모델 레지스트리, Prometheus/Grafana 모니터링을 Docker Compose로 즉시 실행 가능. Terraform + Helm + ArgoCD 기반 프로덕션 배포와 GitLab CI/CD 파이프라인을 포함합니다.

Architecture

graph TB
    Client[Client / CI Pipeline] -->|REST API| GW

    subgraph "API Layer"
        GW[API Gateway<br/>FastAPI :18080]
    end

    subgraph "Compute Services"
        TW[Training Worker<br/>Celery]
        SM[Serving Manager<br/>Docker SDK]
        BR[Benchmark Runner]
    end

    subgraph "Model Serving"
        V1[vLLM Instance 1<br/>GPU]
        V2[vLLM Instance N<br/>GPU]
    end

    subgraph "Data Layer"
        ML[MLflow<br/>:25000]
        PG[(PostgreSQL<br/>:15432)]
        RD[(Redis<br/>:16379)]
        MN[MinIO<br/>:19000]
    end

    subgraph "Observability"
        PR[Prometheus<br/>:19090]
        GR[Grafana<br/>:13001]
        AM[Alertmanager]
        NE[Node Exporter]
    end

    GW --> TW & SM & BR
    TW -->|job queue| RD
    TW -->|experiment tracking| ML
    SM -->|container lifecycle| V1 & V2
    BR -->|load test| V1 & V2
    BR -->|results| ML
    ML -->|metadata| PG
    ML -->|artifacts| MN
    PR -->|scrape| GW & V1 & V2 & NE
    GR -->|query| PR
    AM -->|rules| PR
Loading

Request flow: Clients interact exclusively with the API Gateway. Training requests are dispatched to Celery workers via Redis. Serving requests launch vLLM containers through the Docker SDK. Benchmark requests exercise running vLLM endpoints and collect latency/throughput metrics. All experiment metadata, parameters, and artifacts are tracked in MLflow (backed by PostgreSQL and MinIO). Prometheus scrapes metrics from every component; Grafana renders dashboards and Alertmanager routes alerts.

Tech Stack

Component Technology Version Purpose
API Gateway FastAPI + Uvicorn 0.115 / 0.30 REST API routing, health checks, Prometheus metrics
Task Queue Celery + Redis 5.4 / 7 Async job orchestration for training and benchmarks
Model Training PyTorch + Transformers + PEFT + DeepSpeed 2.1+ / 4.44+ LoRA/QLoRA fine-tuning on GPU clusters
Model Serving vLLM latest High-throughput LLM inference with PagedAttention
Experiment Tracking MLflow 2.16 Model registry, parameter logging, artifact storage
Object Storage MinIO latest S3-compatible storage for model artifacts
Database PostgreSQL 16 MLflow backend store, metadata persistence
Monitoring Prometheus + Grafana 2.53 / 11.1 Metrics collection, dashboards, alerting
Alerting Alertmanager latest Alert routing (webhook, Slack, email)
Infrastructure Terraform + Helm 1.9 / 3.15 AWS EKS provisioning, Kubernetes deployment
GitOps ArgoCD -- Declarative Kubernetes deployment
Containers Docker + Docker Compose 24 / 3.9 Local development, CI/CD builds
CI/CD GitLab CI -- Lint, test, build, deploy pipeline

Quick Start

# 1. Clone the repository
git clone https://github.com/Jaeman-Lee/llm-ops-platform.git
cd llm-ops-platform

# 2. Create environment file
cp .env.example .env

# 3. Build and start the platform
docker compose up -d --build

Or use the automated setup script:

./scripts/setup.sh

Once running, open:

Service URL Credentials
API Gateway http://localhost:18080 --
Swagger UI http://localhost:18080/docs --
MLflow http://localhost:25000 --
Prometheus http://localhost:19090 --
Grafana http://localhost:13001 admin / admin
MinIO Console http://localhost:19001 minioadmin / minioadmin123

Seed Demo Data

# Populate MLflow with sample experiments, models, and benchmarks
./scripts/seed-mlflow.sh

API Reference

All endpoints are served under the API Gateway at http://localhost:18080.

Health and Monitoring

Method Path Description
GET /health Platform health check (gateway + dependencies)
GET /metrics Prometheus metrics endpoint

Model Registry

Method Path Description
GET /api/v1/models List all registered models
GET /api/v1/models/{name} Get model details by name
GET /api/v1/models/{name}/versions/{version} Get a specific model version with metrics
POST /api/v1/models/{name}/stage Transition model stage (Staging/Production/Archived)

Training

Method Path Description
POST /api/v1/training/jobs Submit a new training job (async, returns 202)
GET /api/v1/training/jobs List all training jobs and their status
GET /api/v1/training/jobs/{job_id} Get status of a specific training job
DELETE /api/v1/training/jobs/{job_id} Cancel a queued or running training job

Serving

Method Path Description
POST /api/v1/serving/endpoints Deploy a model as a vLLM serving endpoint
GET /api/v1/serving/endpoints List all active serving endpoints
GET /api/v1/serving/endpoints/{endpoint_id} Get status of a specific endpoint
DELETE /api/v1/serving/endpoints/{endpoint_id} Stop and remove a serving endpoint

Benchmarks

Method Path Description
POST /api/v1/benchmarks Start a benchmark run against a serving endpoint
GET /api/v1/benchmarks List all past benchmark runs
GET /api/v1/benchmarks/{benchmark_id} Get result of a specific benchmark run

Monitoring

Prometheus Metrics

The API Gateway exposes Prometheus metrics at /metrics, including:

  • http_requests_total -- counter of all HTTP requests by method, endpoint, and status code
  • http_request_duration_seconds -- histogram of request latency
  • training_jobs_total -- counter of training jobs by completion status
  • training_duration_seconds -- histogram of training job durations

Prometheus is configured to scrape the gateway, MLflow, node-exporter, and dynamically discovered vLLM endpoints (via file-based service discovery).

Grafana Dashboards

Grafana is pre-provisioned with 4 dashboards in monitoring/grafana/dashboards/:

  • Platform Overview -- request rate, error rate, latency percentiles across all services
  • GPU Monitoring -- per-GPU memory usage, compute utilization, temperature
  • Inference Performance -- tokens/second, time-to-first-token, P50/P95/P99 latency
  • Training Progress -- loss curves, resource usage, job status

Alert Rules

Alerting rules are defined in monitoring/prometheus/rules/ and cover:

  • GPU Alerts (gpu-alerts.yml) -- GPU memory exhaustion, thermal throttling, device errors
  • Serving Alerts (serving-alerts.yml) -- high latency, error rate spikes, endpoint unavailability
  • Training Alerts (training-alerts.yml) -- stuck jobs, repeated failures, queue depth warnings

Alertmanager routes critical alerts to webhooks (and optionally Slack/email). Configuration is in monitoring/alertmanager/alertmanager.yml.

Infrastructure

Local Development (Docker Compose)

The default docker-compose.yml starts all core services. The docker-compose.override.yml adds development conveniences such as hot-reload and debug logging for the gateway.

docker compose up -d --build    # build and start all services
docker compose ps               # check status
docker compose logs -f gateway  # tail gateway logs
docker compose down             # stop all services
docker compose down -v          # stop and remove volumes

Kubernetes (Helm)

The Helm chart in infrastructure/helm/llm-platform/ deploys the full platform to a Kubernetes cluster. Three values files are provided:

# Development (minimal resources, no autoscaling, no ingress)
helm install llm-platform infrastructure/helm/llm-platform \
  -f infrastructure/helm/llm-platform/values-dev.yaml \
  --namespace llm-platform --create-namespace

# Default (staging)
helm install llm-platform infrastructure/helm/llm-platform \
  --namespace llm-platform --create-namespace

# Production (HA, GPU tolerations, TLS ingress, fast-SSD storage)
helm install llm-platform infrastructure/helm/llm-platform \
  -f infrastructure/helm/llm-platform/values-prod.yaml \
  --namespace llm-platform --create-namespace

AWS Infrastructure (Terraform)

Terraform modules in infrastructure/terraform/ provision the underlying cloud resources:

  • EKS Cluster -- managed Kubernetes cluster with separate general and GPU node groups
  • GPU Nodes -- autoscaling group of g5.2xlarge instances with NVIDIA drivers
  • Monitoring -- Prometheus and Grafana deployed into the cluster
cd infrastructure/terraform
terraform init
terraform plan -var-file=environments/dev.tfvars     # staging
terraform plan -var-file=environments/prod.tfvars    # production

GitOps (ArgoCD)

ArgoCD application manifests are in infrastructure/argocd/ for declarative Kubernetes deployment.

CI/CD (GitLab)

The .gitlab-ci.yml pipeline automates the full delivery workflow:

  1. Lint -- Python (ruff), Dockerfiles (hadolint), Helm charts, Terraform validation
  2. Test -- Unit tests with pytest, integration tests with PostgreSQL and Redis service containers
  3. Build -- Docker images for gateway, training, serving, and tester services
  4. Deploy Staging -- Docker Compose deployment with health check verification
  5. Benchmark -- Automated performance validation against staging (p95 latency and error rate gates)
  6. Deploy Production -- Manual Helm upgrade to production Kubernetes cluster

Project Structure

llm-ops-platform/
├── .env.example                          # Environment variable template
├── .gitignore
├── .gitlab-ci.yml                        # CI/CD pipeline definition
├── CLAUDE.md                             # Claude Code agent instructions
├── docker-compose.yml                    # Core service definitions
├── docker-compose.override.yml           # Development overrides (hot-reload)
├── Makefile                              # Developer workflow commands
├── README.md
│
├── services/
│   ├── gateway/                          # API Gateway (FastAPI)
│   │   ├── Dockerfile                    # Multi-stage (base/development/production)
│   │   ├── requirements.txt
│   │   ├── main.py                       # App entry, middleware, health, metrics
│   │   ├── schemas.py                    # Pydantic request/response models
│   │   └── routers/
│   │       ├── training.py               # Training job endpoints
│   │       ├── serving.py                # Serving endpoint management (Docker SDK)
│   │       ├── models.py                 # MLflow model registry proxy
│   │       └── benchmark.py              # Benchmark execution endpoints
│   ├── training/                         # Training Worker (Celery)
│   │   ├── Dockerfile
│   │   ├── requirements.txt
│   │   ├── worker.py                     # Celery tasks, MLflow integration
│   │   └── pipeline.py                   # Training pipeline logic
│   ├── serving/                          # Serving Manager
│   │   ├── Dockerfile
│   │   ├── requirements.txt
│   │   └── manager.py                    # vLLM container lifecycle
│   └── tester/                           # Benchmark Runner
│       ├── Dockerfile
│       ├── requirements.txt
│       └── runner.py                     # Load test execution
│
├── infrastructure/
│   ├── helm/llm-platform/
│   │   ├── Chart.yaml
│   │   ├── values.yaml                   # Default values (staging)
│   │   ├── values-dev.yaml               # Development overrides
│   │   ├── values-prod.yaml              # Production overrides
│   │   └── templates/
│   │       ├── _helpers.tpl              # Template helpers, labels, selectors
│   │       ├── gateway-deployment.yaml   # Gateway Deployment + Service + SA + HPA
│   │       ├── vllm-deployment.yaml      # vLLM StatefulSet + Service + SA
│   │       ├── vllm-hpa.yaml             # GPU-based HPA
│   │       ├── mlflow-deployment.yaml    # MLflow Deployment + Service + SA
│   │       ├── training-job.yaml         # K8s Job template (GPU)
│   │       ├── configmap.yaml
│   │       ├── secret.yaml               # DB/MinIO/HF credentials
│   │       ├── ingress.yaml
│   │       └── servicemonitor.yaml       # Prometheus ServiceMonitor
│   ├── terraform/
│   │   ├── main.tf                       # Providers, EKS/GPU/monitoring modules
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   ├── environments/
│   │   │   ├── dev.tfvars                # Staging environment
│   │   │   └── prod.tfvars               # Production environment
│   │   └── modules/
│   │       ├── gpu-node/                 # GPU ASG provisioning
│   │       ├── k8s-cluster/              # EKS cluster
│   │       └── monitoring/               # Prometheus + Grafana
│   └── argocd/
│       ├── application.yaml
│       └── appproject.yaml
│
├── monitoring/
│   ├── prometheus/
│   │   ├── prometheus.yml                # Scrape configuration
│   │   └── rules/
│   │       ├── gpu-alerts.yml
│   │       ├── serving-alerts.yml
│   │       └── training-alerts.yml
│   ├── grafana/
│   │   ├── provisioning/
│   │   │   ├── datasources.yml
│   │   │   └── dashboards.yml
│   │   └── dashboards/
│   │       ├── platform-overview.json
│   │       ├── gpu-monitoring.json
│   │       ├── inference-performance.json
│   │       └── training-progress.json
│   └── alertmanager/
│       └── alertmanager.yml
│
├── scripts/
│   ├── setup.sh                          # Automated local setup
│   ├── seed-mlflow.sh                    # MLflow sample data
│   └── demo.sh                           # End-to-end demo walkthrough
│
└── docs/
    └── architecture.md                   # Detailed architecture document

Configuration

All runtime configuration is managed through environment variables. Copy .env.example to .env and update values:

cp .env.example .env
Variable Default Description
POSTGRES_PASSWORD changeme PostgreSQL password for the mlflow user
MINIO_ROOT_USER minioadmin MinIO root username
MINIO_ROOT_PASSWORD minioadmin123 MinIO root password
GF_ADMIN_PASSWORD admin Grafana admin password
VLLM_MODEL_NAME Qwen/Qwen2.5-7B Default model for vLLM serving
GATEWAY_PORT 18080 Host port for the API Gateway

License

This project is licensed under the MIT License. See LICENSE for details.

About

LLM 파인튜닝부터 vLLM 서빙, 성능 벤치마크까지 하나의 API로 통합한 End-to-End MLOps 플랫폼

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages