UNIFIED-GPU-RUNTIME

Heterogeneous GPU Orchestration for LLM Workloads
Run different AI models across mixed NVIDIA + AMD GPUs intelligently

GPU Combos • How It Works • Quick Start • Features • License

---

🎯 The Problem

Most multi-GPU solutions try to split one model across GPUs. This fails with mixed hardware.

Our solution: Split the workload, not the model weights.

❌ WRONG:  Model weights → split → GPU A + GPU B  (breaks with mixed hardware)
✅ RIGHT:  Tasks → route intelligently → GPU A runs Model X, GPU B runs Model Y

---

🖥️ Supported GPU Combinations

How Each Combination Works

Setup	How It Works	Best For
NVIDIA + NVIDIA	Both GPUs run CUDA. Larger GPU handles main LLM, smaller handles draft/embeddings	Speculative decoding, multi-model inference
NVIDIA + AMD	NVIDIA (CUDA) runs main LLM, AMD (ROCm) runs embeddings/vision/draft	Mixed vendor builds, cost optimization
AMD + AMD	Both GPUs run ROCm. Same as NVIDIA+NVIDIA but on AMD	All-AMD workstations
Any GPU + CPU	GPU handles inference, CPU as emergency fallback	Resilience, small models

Tested Configurations

Primary GPU	Secondary GPU	Status
RTX 5080 16GB	RTX 3080 10GB	✅ Tested
RTX 4090 24GB	RTX 3080 10GB	✅ Supported
RTX 4090 24GB	RX 7900 XTX 24GB	✅ Supported
RX 7900 XTX 24GB	RX 6800 16GB	✅ Supported
Any GPU	CPU Fallback	✅ Supported

What Each GPU Does

┌──────────────────────────────────────────────────────────────┐
│                    TASK SCHEDULER                             │
│         Routes tasks based on VRAM, load, affinity           │
└──────────────────────────────────────────────────────────────┘
                    │                       │
                    ▼                       ▼
          ┌─────────────────┐     ┌─────────────────┐
          │  PRIMARY GPU    │     │  SECONDARY GPU  │
          │  (Larger VRAM)  │     │  (Smaller VRAM) │
          │                 │     │                 │
          │  • Main LLM     │◄───►│  • Draft model  │
          │  • Reasoning    │     │  • Embeddings   │
          │  • Verification │     │  • Vision       │
          │  • Code gen     │     │  • Reranking    │
          └─────────────────┘     └─────────────────┘

GPU Compatibility

The runtime is GPU-agnostic. Your GPU support depends on the inference backend:

Backend	Supported GPUs
llama.cpp	NVIDIA (CUDA 5.0+), AMD (ROCm), Apple Silicon, Vulkan
Super-llama.cpp	Same as llama.cpp + enterprise optimizations

Compatible GPUs:

• NVIDIA: Any GPU with CUDA compute capability 5.0+ (GTX 900+, RTX series, Quadro, Tesla)
• AMD: GPUs supported by ROCm (RDNA/RDNA2/RDNA3, CDNA architectures)

Mixed configurations work:

• Different NVIDIA GPUs (e.g., RTX 5080 + RTX 3080)
• Different AMD GPUs (e.g., RX 7900 + RX 6800)
• NVIDIA + AMD together (each runs its own worker process)

---

⚡ How It Works

Speculative Decoding (2-3x Speedup)

Secondary GPU (Draft)              Primary GPU (Verify)
─────────────────────              ────────────────────

Small model generates          →   Large model verifies
5 tokens quickly                   all 5 in ONE pass

"The quick brown fox"          →   ✅ Accept 4/5 tokens
                                   🔄 Correct 1 token

Result: 4 verified tokens in ~1 forward pass

Task Routing

Task	Goes To	Why
Reasoning, Code	Primary GPU	Needs larger model
Draft tokens	Secondary GPU	Speed over quality
Embeddings	Secondary GPU	Smaller model sufficient
Vision encode	Secondary GPU	Dedicated vision model
Verification	Primary GPU	Quality critical

---

🏗️ System Architecture

Your App (HTTP: localhost:8000)
         │
         ▼
┌─────────────────────────────────┐
│      UNIFIED-RUNTIME            │
│   OpenAI-compatible API         │
│   Task Routing + Scheduling     │
└──────┬──────────────┬───────────┘
       │              │
   Named Pipe     Named Pipe
       │              │
       ▼              ▼
┌────────────┐  ┌────────────┐
│ Worker     │  │ Worker     │
│ (NVIDIA)   │  │ (AMD)      │
│ RTX 5080   │  │ RX 7900    │
│ llama.cpp  │  │ llama.cpp  │
└────────────┘  └────────────┘

---

🚀 Quick Start

1. Build

git clone https://github.com/GaloSerranoA/UNIFIED-GPU-RUNTIME.git
cd UNIFIED-GPU-RUNTIME

# Linux/macOS
mkdir build && cd build
cmake .. && make -j$(nproc)

# Windows
cmake -B build -G "Visual Studio 17 2022"
cmake --build build --config Release

2. Configure (configs/runtime.yaml)

gpus:
  - id: 0
    name: "RTX 5080"
    vram_mb: 16384
    role: "primary"

  - id: 1
    name: "RTX 3080"
    vram_mb: 10240
    role: "auxiliary"

speculative:
  enabled: true
  draft_model: "small-3b"
  verify_model: "large-8b"

3. Run

./build/unified-runtime --config configs/runtime.yaml

4. Use the API

curl -X POST http://localhost:8000/v1/inference \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quantum computing", "max_tokens": 500}'

---

✨ Features

Category	Features
Scheduling	GPU-aware routing, priority preemption, load balancing
Memory	Model ownership tracking, LRU eviction, migration support
Speculative	Draft/verify across GPUs, 2-3x speedup, adaptive draft length
Protocol	Binary TLV protocol, CRC32, multiplexed connections
Enterprise	Prometheus metrics, health probes, graceful shutdown
Platform	Windows, Linux, macOS

---

📋 Requirements

Requirement	Version
CMake	3.20+
C++	C++20
CUDA (NVIDIA)	12.0+
ROCm (AMD)	5.0+

Platform Requirements

1. Windows 10/11 or Linux (Ubuntu 20.04+, Fedora 36+)
2. Inference backend (choose one):
   • llama.cpp — community standard
   • Super-llama.cpp — enterprise fork with additional optimizations
3. GPU drivers:
   • CUDA build for NVIDIA GPUs
   • ROCm/HIP build for AMD GPUs
4. Visual Studio 2022 (for building on Windows)
5. GGUF model file(s) for inference

---

⚠️ What This Is NOT

• Not tensor parallelism (splitting weights across GPUs)
• Not a replacement for llama.cpp/vLLM
• Not cross-vendor weight splitting (NVIDIA+AMD sharing one model)

What we DO support: Different vendors running different complete models cooperatively.

---

👨‍💻 Author

Galo Serrano Abad — (https://github.com/GaloSerranoA)

---

📄 License

| License | | Apache 2.0 |

---

💖 Support

If you find this useful and want help please consider:

---

⭐ Star this repo if you find it useful!