Skip to content
BranchesTags

Latest commit

 

History

15 Commits

Folders and files

NameName
Last commit message
Last commit date
clients
clients
 
 
config
config
 
 
docker
docker
 
 
docs
docs
 
 
functions
functions
 
 
scripts
scripts
 
 
workflows
workflows
 
 
.gitignore
.gitignore
 
 
CHANGELOG.md
CHANGELOG.md
 
 
HANDOVER.md
HANDOVER.md
 
 
README.md
README.md
 
 

Repository files navigation

# AI Platform

A self-hosted AI platform built with Docker, vLLM, LiteLLM, Open WebUI, and ComfyUI.

The project is designed to provide a production-ready local AI stack that can serve multiple model classes through a unified OpenAI-compatible API, while also supporting a separate image-generation pipeline based on ComfyUI.

The platform is configuration-driven:

- Docker Compose files are generated automatically
- LiteLLM configuration is generated automatically
- GPU allocation and scheduling are derived from model configuration
- ComfyUI model trees are generated from source models
- Workflow catalogs are generated from API workflow exports

All project scripts use `scripts/common.sh` to discover the project root automatically, which keeps the repository portable across different installation paths.

---

# Key Features

## LLM Platform

- Docker-based deployment
- vLLM inference server
- LiteLLM OpenAI-compatible gateway
- Open WebUI integration
- Multiple models served simultaneously
- Automatic model registration
- Automatic Docker Compose generation
- Automatic LiteLLM configuration generation
- Multi-GPU scheduling
- Runtime model overrides
- OpenAI-compatible endpoints
- Support for coding clients

## Image Platform

- Dedicated ComfyUI image platform
- Workflow-based image generation
- Workflow catalog generation
- Workflow routing
- Open WebUI image integration
- Markdown image preview inside Open WebUI
- Nginx image serving
- Source-model to ComfyUI model-tree generation
- Configuration-driven workflow organization

---

# High-Level Architecture

## LLM Stack

```text
Open WebUI
      │
      ▼
LiteLLM
      │
      ▼
vLLM
      │
      ▼
Loaded Models

Image Stack

Open WebUI
      │
      ▼
Ani Image Pipe
      │
      ▼
Workflow Router
      │
      ▼
ComfyUI
      │
      ▼
Generated Image
      │
      ▼
Nginx
      │
      ▼
Open WebUI Preview

The LLM platform exposes a single OpenAI-compatible endpoint regardless of how many inference services are active. The image platform is separate, but integrates into Open WebUI through a Function-based pipe.

Repository Layout

ai-platform/

├── backups/
├── config/
├── docker/
├── docs/
├── functions/
├── images/
├── logs/
├── models/
├── open-webui/
├── reports/
├── scripts/
└── workflows/

Important Subdirectories

config/

Main configuration files for model selection, overrides, and downloadable models.

docker/

Docker Compose files, environment templates, and deployment-specific configuration.

docs/

Project documentation, including model configuration, clients, troubleshooting, and the Open WebUI image pipeline.

functions/

Canonical source copy of Open WebUI Function code used by the project.

images/

Runtime ComfyUI data directory and generated image outputs.

models/

Local model storage, organized by model category.

scripts/

Deployment, generation, validation, routing, and platform control scripts.

workflows/

API-ready ComfyUI workflows, catalog files, router rules, and workflow metadata.

Services

Service Purpose Port
vLLM Model inference Internal
LiteLLM OpenAI-compatible gateway 4000
Open WebUI Web interface 8080
ComfyUI Image generation / editing 8188

All services communicate through the shared Docker network:

ai-platform

Current Models

Alias Model
qwen15 Qwen2.5-1.5B-Instruct
qwen7b Qwen2.5-7B-Instruct

Quick Start

1) Clone the repository

git clone https://github.com/ramin40686/self-hosted-ai-platform.git
cd self-hosted-ai-platform

2) Configure environment files

cp docker/.env.example docker/.env

Edit the local values as needed, including tokens, timezone, and ports.

3) Configure models

Edit:

config/enabled-models.conf

Optional overrides:

config/model-overrides.conf

4) Generate runtime configuration

./scripts/register-models.sh

5) Start the LLM platform

./scripts/start-ai-platform.sh

6) Start the image platform

./scripts/start-image-platform.sh

Installation

Requirements

Recommended:

  • NVIDIA GPU
  • NVIDIA Container Toolkit
  • Docker
  • Docker Compose
  • Git

Typical layout

  • Models are stored under models/
  • Runtime ComfyUI data is stored under images/comfyui/
  • Workflow exports are stored under workflows/Workflows_API/

Configuration

The platform is driven by configuration files.

Main model configuration

File Purpose
config/enabled-models.conf Enable or disable models
config/model-overrides.conf Runtime overrides
config/models-for-download.conf Models to download
config/model-stats.conf Local stats and benchmark data

Docker configuration

File Purpose
docker/.env Local Docker variables
docker/compose.inference.yml Generated inference services
docker/compose.litellm.yml LiteLLM deployment
docker/compose.openwebui.yml Open WebUI deployment
docker/compose.images.yml ComfyUI deployment

LLM Platform

The LLM platform consists of Open WebUI, LiteLLM, and one or more vLLM inference services.

Start

./scripts/start-ai-platform.sh

Stop

./scripts/stop-ai-platform.sh

Restart

./scripts/restart-ai-platform.sh

OpenAI-Compatible Endpoint

Base URL:

http://localhost:4000/v1

Example:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model":"qwen7b",
    "messages":[
      {
        "role":"user",
        "content":"Hello"
      }
    ]
  }'

Image Platform

The image platform is built around ComfyUI and a source-tree model layout.

Source models

Original downloads are stored in:

models/image/source/

Runtime model tree

generate-comfyui-model-tree.sh creates the ComfyUI runtime model tree by linking the source models into the container’s ComfyUI model folders.

Start

./scripts/start-image-platform.sh

Stop

./scripts/stop-image-platform.sh

Restart

./scripts/restart-image-platform.sh

Open WebUI Image Pipeline

Open WebUI includes a custom Function Pipe that routes image-generation prompts to ComfyUI workflows.

The implementation supports:

  • text-to-image generation
  • automatic workflow routing
  • workflow catalog lookup
  • image preview in Open WebUI
  • public image serving through nginx
  • Docker-native execution

Complete documentation:

docs/openwebui-image-pipeline.md

Canonical Function source:

functions/ani_image_pipe.py

Workflow Catalog

The image workflows are indexed through a generated catalog:

workflows/Workflows_API/catalog.yaml

The catalog is built automatically from the API workflows and validated before runtime.

Generate the catalog:

./scripts/build_workflow_catalog.sh

The runtime router uses the generated catalog instead of scanning workflow folders at startup.

Workflow Metadata

During catalog generation, an additional metadata report is produced for development and validation.

Default location:

reports/workflow-metadata.json

This file is used for inspection and debugging, not at runtime.

Workflow Development

API-ready ComfyUI workflows are stored under:

workflows/Workflows_API/

Typical workflow groups:

  • text_to_image/
  • image_edit/
  • image_upscaler/
  • remove_background/
  • guided_image_edit/
  • router/

Workflow development cycle:

  1. Create or export a ComfyUI API workflow
  2. Place it in the appropriate task directory
  3. Rebuild the catalog
  4. Validate the generated output
  5. Test the workflow through Open WebUI

Workflow-specific details are documented in:

docs/openwebui-image-pipeline.md
docs/image-generation.md

Scripts

Core Platform

Script Purpose
scripts/common.sh Shared runtime utilities
scripts/download-models.sh Download models from Hugging Face
scripts/register-models.sh Generate inference compose and LiteLLM config
scripts/measure-models.sh Benchmark supported models
scripts/gpu-report.sh GPU report

LLM Platform

Script Purpose
scripts/start-ai-platform.sh Start LLM services
scripts/stop-ai-platform.sh Stop LLM services
scripts/restart-ai-platform.sh Restart LLM services
scripts/wait-healthy.sh Wait until services are healthy

Image Platform

Script Purpose
scripts/start-image-platform.sh Start ComfyUI platform
scripts/stop-image-platform.sh Stop ComfyUI platform
scripts/restart-image-platform.sh Restart ComfyUI platform
scripts/generate-comfyui-model-tree.sh Build ComfyUI model tree

Workflow System

Script Purpose
scripts/build_workflow_catalog.sh Build and validate workflow catalog
scripts/extract_comfyui_workflow_catalog.py Extract workflow metadata
scripts/validate_workflow_catalog.py Validate workflow catalog
scripts/router_and_execute.py Runtime workflow orchestrator
scripts/run_workflow_api.py Execute ComfyUI workflows

Commands

List containers

docker ps

View GPU status

nvidia-smi

Check disk usage

df -h

Check model sizes

du -sh models/*

View logs

docker logs -f open-webui
docker logs -f litellm
docker logs -f comfyui

Rebuild the workflow catalog

./scripts/build_workflow_catalog.sh

Regenerate ComfyUI model tree

./scripts/start-image-platform.sh

Troubleshooting

A model does not start

Check:

  1. Docker logs
  2. GPU memory
  3. GPU_MEMORY_UTILIZATION
  4. MAX_MODEL_LEN
  5. Model-specific runtime overrides

A ComfyUI model does not appear

Check:

  1. Source model location in models/image/source/
  2. ComfyUI logs
  3. Generated runtime model tree
  4. The image platform startup script

A workflow is not found

Rebuild the catalog:

./scripts/build_workflow_catalog.sh

Then verify:

workflows/Workflows_API/catalog.yaml

Open WebUI image generation does not work

Check:

  1. The Open WebUI Function exists
  2. functions/ani_image_pipe.py matches the deployed Function
  3. The router returns valid JSON
  4. ComfyUI is reachable
  5. Nginx serves generated images

See:

docs/openwebui-image-pipeline.md

Supported Clients

The LiteLLM gateway exposes a standard OpenAI-compatible API.

Verified clients include:

  • Cursor
  • Continue
  • Cline
  • Aider
  • Qwen Code

All clients can connect to:

http://localhost:4000/v1

Documentation Index

General

  • CHANGELOG.md
  • HANDOVER.md
  • README.md

Model Management

  • docs/enabled-models.md
  • docs/model-overrides.md
  • docs/model-stats.md
  • docs/models-for-download.md
  • docs/CLIENTS.md
  • docs/troubleshooting.md

Image Platform

  • docs/image-generation.md
  • docs/flux-dev-inpainting.md
  • docs/flux-kontext-replace.md
  • docs/openwebui-image-pipeline.md

Workflow System

  • workflows/Workflows_API/catalog.yaml
  • workflows/Workflows_API/workflow-metadata.json
  • workflows/Workflows_API/router/

Roadmap

Completed

  • Docker infrastructure
  • vLLM
  • LiteLLM
  • Open WebUI
  • Multiple LLMs
  • Automatic model registry
  • Automatic Docker Compose generation
  • Automatic LiteLLM configuration
  • Multi-GPU scheduling
  • Coding clients setup
  • ComfyUI image platform
  • ComfyUI workflow library
  • Open WebUI image pipeline

Planned

  • Embedding server
  • Reranker
  • RAG pipeline
  • Whisper API
  • Vision models
  • Text-to-speech
  • Monitoring
  • Reverse proxy
  • Authentication
  • HTTPS
  • Backup automation

Image pipeline expansion

  • Image edit
  • Inpaint
  • Outpaint
  • Background removal
  • Image upscaler
  • Guided image editing
  • Automatic parameter mapping
  • Image upload support
  • Progress streaming
  • Workflow validation improvements

License

MIT

About

Self-hosted AI Platform powered by vLLM, LiteLLM and Open WebUI

Topics

aidockerdocker-composeinfrencelitellmllmopen-webuiopenai-apiopenai-compatibleqwenself-hostedvllm

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages