Model Deployment Microservice Architecture

This repository demonstrates a complete, production-ready setup for serving Ultralytics YOLO models using a scalable microservice architecture (worker + FastAPI backend + Gradio frontend).

The project is a small production-style system built to serve ML/AI models efficiently. This repo can work with any model; you would need to integrate it into worker.py. The goal is to show how to structure a real, scalable inference pipeline using separate services for compute, API, and UI. A similar pattern is used in practical ML deployments.

Quick Start

Requirements

• Docker + Docker Compose should be installed
• (Optional) NVIDIA Container Toolkit for GPU inference

# Step 1 - Build and tag the image
docker build -t object-detection-app:latest .

# Step 2 - Start all three services using the pre-built image
docker compose -f docker-compose.yml up     # for CPU runtime
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up  # for GPU runtime

The application is available at: http://localhost:8080

Serving Ultralytics YOLO Models

This repository supports any Ultralytics YOLO model (YOLOv8, YOLOv9, YOLOv11, custom-trained .pt weights, etc.) without requiring code changes. The worker service automatically loads whatever model file you specify.

• Works out of the box for all Ultralytics-trained YOLO models.
• Set WORKER_MODEL_NAME in docker-compose.yml to any pretrained model Ultralytics provides.
• To serve a custom YOLO model trained with Ultralytics, mount the directory containing your .pt weights and point WORKER_MODEL_NAME to the file path inside the container.
• For example, if your model weights are stored at /home/ubuntu/models/my-custom-yolov11m.pt, update the worker service like this:

  worker:
    image: object-detection-app:latest
    command: ["worker"] 
    environment:
      WORKER_MODEL_NAME: "/app/my_custom_models/my-custom-yolov11m.pt"
      WORKER_NUM_PROCESSES: 2
    volumes:
      - shared_storage:/app/temp
      - /home/ubuntu/models:/app/my_custom_models

Performance Notes (preliminary)

The latency figures below represent the end-to-end client inference latency

YOLO Model Version	Size	Average Inference Latency
yolo11n.pt (Nano)	Smallest	≈ 23 ms
yolo11x.pt (Extra Large)	Largest	≈ 68 ms

Key Takeaway

The system maintains competitive latency, with the smallest model (yolo11n.pt) achieving sub-25ms response times consistently. This showcases that the architectural overhead is minimal, and model choice is the primary factor determining end-to-end speed. The architecture is optimized to prevent I/O or threading from becoming the bottleneck.

🏛️ System Architecture

Service	Role	Key Functionality
worker	Inference Engine (Compute)	GPU-accelerated processes Internal port 6000.
backend	API Gateway (Logic)	FastAPI service receives client requests and dispatches jobs to the worker. Exposes /predict API.
app	User Interface (Presentation)	Gradio web interface, accessible externally on host port 8080.

Deep Dive: High-Performance Worker Design

The worker service (worker.py) works in concurrent and non-blocking I/O, crucial for high-throughput inference:

1. Multi-Process Isolation: Multiple workers are spawned, each process loads its own instance of the model, and is pinned to the GPU. If using CPU, it is still beneficial since it prevents Python's Global Interpreter Lock (GIL) from bottlenecking compute tasks.

2. Asynchronous I/O: The main process runs a listener to accept network connections and a thread to handle incoming requests. This thread dispatches the task to the GPU in a non-blocking way. The server can continue accepting new requests simultaneously.

3. Inter-Process Communication (IPC): Simple queues are used to transfer requests and results between the network I/O threads (Main Process) and the GPU-bound workers (Child Processes). A dedicated router is responsible for consuming results and dispatching the signal to the right client.

Deep Dive: API Gateway (Backend)

The backend service (backend.py) works as an intermediary layer that can be scaled independently to utilize underlying workers to the fullest:

1. Non-Blocking Client: The /predict endpoint works in async, ensuring the FastAPI server remains fully responsive and non-blocking, maximizing throughput.
2. Extensibility: This layer is important since many features like rate limiting, call billing, user access, and many other operations can be managed here, while the GPU is exclusively for firing requests back-to-back. This architecture ensures the hefty GPU bill is fully utilized.

Deep Dive: Gradio Frontend (App)

The app service (app.py) is the UI component that handles all user interaction and visualization logic:

1. Orchestration: Manages the user-uploaded image, makes an HTTP POST request to the above endpoint, and receives the raw detection results.
2. Post-Processing & Visualization: The tool will draw the predictions from the model on the image and showcase it on the frontend. It will also show the RAW json.
3. Temporary File Management: A decorator that saves the image in shared storage so that the image is available to the GPU worker for inference. It also makes sure the image is deleted after the request is served, irrespective of any error. The decorator also keeps the code clean (which I am kinda obsessed with)

👋 Feedback & Connect

If this project helped you, inspired your own setup, or is working amazingly in your production, give me a shout-out on LinkedIn

Feel free to reach out. I’d love to hear about it.