LLM Gateway

A production-style gateway that sits in front of multiple LLM backends and adds routing, failover, a semantic cache, and rate limiting — the resilience + cost layer real LLM infra needs. Built on the OpenAI-compatible API, so it fronts any mix of local and cloud models.

Architecture

Client ──▶ Gateway ──┬─ (cache hit) ─────────▶ instant reply, no model call
                     ├─ local Ollama (qwen2.5:1.5b)   ← tried first (free, fast)
                     └─ cloud Groq (llama-3.3-70b)    ← automatic fallback if local is down

Features

• Health-based failover — routes to the local model first; if it's down, automatically falls back to the cloud backend. Reports which backend served each request.
• Semantic cache — embeds each query with local Ollama embeddings and returns a cached reply for similar questions (not just identical ones), skipping the model entirely. Milliseconds vs ~1 s, and fewer paid calls.
• Per-key rate limiting — sliding-window limit per X-API-Key (10 req / 60 s); returns 429 when exceeded, shielding backends from abuse and runaway cost.
• Graceful degradation — if the cache/embeddings fail, the gateway skips the cache and routes normally instead of crashing. An optimization must never take down the core request.
• Structured logging — backend served, cache hits, failovers.

Endpoints

• GET /health
• POST /chat — {"message": "..."} (+ optional X-API-Key header) → {"reply", "served_by", ...}

Run

ollama pull qwen2.5:1.5b
ollama pull nomic-embed-text
pip install "fastapi[standard]" httpx
export GROQ_API_KEY=...            # for the cloud fallback
fastapi dev gateway.py            # → http://127.0.0.1:8000/docs

Stack

FastAPI · httpx · Ollama (chat + embeddings) · Groq

Résumé line

Built an OpenAI-compatible LLM gateway with health-based failover across local + cloud backends, a semantic cache (local embeddings) that serves similar queries without a model call, per-key rate limiting, and graceful degradation of the cache layer.