Inference Mesh

Inference Mesh is a high-performance, edge-scalable PyTorch inference engine designed for executing open-weight LLMs locally. It features a blistering-fast Go API Gateway that orchestrates real-time Server-Sent Events (SSE) streaming from decoupled Python PyTorch gRPC workers.

Features

• High-Throughput Go Gateway: Achieves 438+ requests per second with sub-5µs JSON overhead.
• Decoupled PyTorch Workers: Scalable Python backend running isolated gRPC nodes.
• Hardware Agnostic: Defaults to optimized PyTorch CPU execution but seamlessly supports CUDA.
• Secure Next.js Frontend: Beautiful, responsive UI built on Turbopack with edge-middleware Supabase authentication.

Architecture

graph LR
    Client((Client)) --> NextJS[Next.js Frontend]
    NextJS -- "Supabase Auth" --> Supabase[(Supabase)]
    NextJS -- "Proxies Req" --> GoGateway[Go API Gateway]
    GoGateway -- "gRPC Streams" --> Worker1[Python Worker 1]
    GoGateway -- "gRPC Streams" --> Worker2[Python Worker 2]

Loading

1. Frontend (Next.js): Handles user interactions, Markdown rendering, and edge-level authentication via Supabase JWTs.
2. Gateway (Go): Validates requests, enforces token-bucket rate limits, and routes payloads to workers via gRPC.
3. Workers (Python): Loads HuggingFace models into VRAM/RAM and streams tokens back via gRPC.

Benchmark Results (Local CPU Execution)

Tested on an Intel i5-1135G7 (4 Cores, 32GB RAM) without CUDA acceleration. (For comprehensive methodology, P95 latency distributions, chaos testing results, and system capacity profiles, see the full Performance Benchmark Report).

Metric	Result
Gateway Routing Scalability	100+ concurrent connections (1.6 ms P95 Latency)
Worker Chaos Recovery	100% gateway recovery during 25s worker failure
Gemma 3 (270M) TTFT	553.2 ms (CPU FP32)
Middleware Pipeline Overhead	8.18 µs/op

Resume-Ready Metrics

• Built a production-ready multi-modal AI inference platform exposing OpenAI-compatible streaming APIs for open-weight LLMs using a containerized Go gateway and Python inference workers, architected for independent scaling through Kubernetes, HPAs, and Docker.
• Engineered a Go API gateway providing API Key authentication, token-bucket rate limiting, health-aware worker routing, persistent HTTP/2 gRPC communication, and Server-Sent Events (SSE), reducing Gateway→Worker connection establishment latency by 88% (4.43 ms → 0.51 ms) through persistent connection reuse.
• Validated production readiness using reproducible Go benchmarks, k6 load testing, pprof profiling, OpenTelemetry instrumentation, and chaos testing, achieving 454 ms P95 end-to-end production latency, 291 sustained requests/sec under 100 concurrent users, and deterministic recovery from worker failures with 0% request failures during public load testing.

Public Cloud Deployment

This project is fully containerized and deployable to edge/cloud platforms. The current reference deployment uses Vercel (Frontend) and Railway (Backend Go Gateway).

Warning

Known Infrastructure Limitations (Free Tier): Deploying the Python PyTorch Worker to standard Free Tier cloud environments (like Railway Free Tier or Hugging Face Spaces) has known limitations. Hugging Face Spaces load-balancers actively strip HTTP/2 gRPC headers, breaking Go→Python communication. Railway's Free Tier (1GB RAM limit) results in Out-Of-Memory (OOM) kills when PyTorch attempts to load standard open-weight LLMs into memory. For a fully functional end-to-end deployment, the Python Worker requires a dedicated GPU instance or a minimum of 8GB+ RAM.

1. Deploy the Backend Gateway (Railway)

1. Link your GitHub repository to Railway.
2. Railway will automatically detect the Dockerfile and deploy the Go Gateway.
3. Expose the gateway publicly via Railway networking to obtain your URL (e.g., https://inference-mesh-production.up.railway.app).

2. Deploy the Frontend (Vercel)

1. Import the repository into Vercel.
2. Set the Root Directory to frontend.
3. Under Environment Variables, add:
   • NEXT_PUBLIC_SUPABASE_URL = Your Supabase URL
   • NEXT_PUBLIC_SUPABASE_ANON_KEY = Your Supabase Anon Key
   • NEXT_PUBLIC_API_BASE_URL = The Railway Gateway URL you got in Step 1.
4. Deploy to obtain your public frontend URL.

Local Setup (Development)

1. Environment Configuration

Copy the .env.example file:

cp .env.example .env

Fill in your Supabase URL and keys, along with your HuggingFace token.

2. Run with Docker Compose

The easiest way to launch the entire stack locally is via Docker Compose.

# Starts the Go Gateway and Python PyTorch Worker
docker-compose -f docker-compose.prod.yml up -d --build

Note: The worker image installs PyTorch CPU wheels by default to save bandwidth and image size.

3. Run Frontend

cd frontend
npm install
npm run dev

Security

• Edge Authentication: Next.js middleware verifies Supabase sessions before granting access to /chat.
• API Protection: The Go Gateway is protected by internal API keys, ensuring isolated external-facing security.

Future Improvements

• Dynamic worker auto-scaling based on queue depth.
• Native Kubernetes Helm charts for multi-node deployments.
• vLLM or TGI integration for the Python workers.