One integration point for all your AI providers.
Automatic failover, free-tier aggregation, and capability-based routing.

---

Your application requests a capability (e.g. "chat completion"). ModelMesh picks the best available provider, rotates on failure, and chains free quotas across providers -- all behind a standard OpenAI SDK interface.

Install

Python:

pip install modelmesh-lite                # core (zero dependencies)
pip install modelmesh-lite[yaml]          # + YAML config support

TypeScript / Node.js:

npm install @nistrapa/modelmesh-core

Docker Proxy (any language):

# Option A: Pull pre-built image from GitHub Container Registry
docker pull ghcr.io/apartsinprojects/modelmesh:latest

# Option B: Build from source
git clone https://github.com/ApartsinProjects/ModelMesh.git
cd ModelMesh
cp .env.example .env   # add your API keys
docker compose up --build
# Proxy at http://localhost:8080 — speaks the OpenAI REST API

Quick Start

Set an API key and go:

export OPENAI_API_KEY="sk-..."

Python

import modelmesh

client = modelmesh.create("chat-completion")

response = client.chat.completions.create(
    model="chat-completion",          # virtual model name = capability pool
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

TypeScript

import { create } from "@nistrapa/modelmesh-core";

const client = create("chat-completion");

const response = await client.chat.completions.create({
    model: "chat-completion",
    messages: [{ role: "user", content: "Hello!" }],
});
console.log(response.choices[0].message.content);

How It Works

client.chat.completions.create(model="chat-completion", ...)
       |
       v
  +-----------+     +-----------+     +----------+
  |  Router   | --> |   Pool    | --> |  Model   | --> Provider API
  +-----------+     +-----------+     +----------+
  Resolves the       Groups models     Selects best     Sends request,
  capability to      that can do       active model     handles retry
  a pool             the task          (rotation policy) and failover

"chat-completion" resolves to a pool containing all models that support chat. The pool's rotation policy picks the best active model. If it fails, the router retries with backoff, then rotates to the next model. When a provider's free quota runs out, rotation automatically moves to the next provider.

Multi-Provider Failover

Add more API keys -- ModelMesh chains them automatically:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="AI..."

client = modelmesh.create("chat-completion")

# Inspect the providers behind the virtual model
print(client.describe())
# Pool "chat-completion" (strategy: stick-until-failure)
#   capability: generation.text-generation.chat-completion
#   → openai.gpt-4o [openai.llm.v1] (active)
#     openai.gpt-4o-mini [openai.llm.v1] (active)
#     anthropic.claude-sonnet-4 [anthropic.claude.v1] (active)
#     google.gemini-2.0-flash [google.gemini.v1] (active)

Same client.chat.completions.create() call -- but now if OpenAI is down or its quota is exhausted, the request routes to Anthropic, then Gemini.

YAML Configuration

For full control, use a configuration file:

# modelmesh.yaml
providers:
  openai.llm.v1:
    connector: openai.llm.v1
    config:
      api_key: "${secrets:OPENAI_API_KEY}"

  anthropic.claude.v1:
    connector: anthropic.claude.v1
    config:
      api_key: "${secrets:ANTHROPIC_API_KEY}"

models:
  openai.gpt-4o:
    provider: openai.llm.v1
    capabilities:
      - generation.text-generation.chat-completion

  anthropic.claude-sonnet-4:
    provider: anthropic.claude.v1
    capabilities:
      - generation.text-generation.chat-completion

pools:
  chat:
    capability: generation.text-generation.chat-completion
    strategy: stick-until-failure

client = modelmesh.create(config="modelmesh.yaml")

Why ModelMesh

Ten reasons to add ModelMesh to your next project.

#	Value	Feature	How It Delivers
1	Integrate in two minutes, scale the configuration as you grow	Progressive Configuration	Env vars for instant start. YAML for providers, pools, strategies, budgets, secrets. Programmatic for dynamic setups. All three compose seamlessly
2	One familiar API across every provider you will ever use	Uniform OpenAI-Compatible API	Same client.chat.completions.create() for OpenAI, Anthropic, Gemini, DeepSeek, Mistral, Ollama, or custom models. Chat, embeddings, TTS, STT, image generation. Swap providers in config, never in code
3	Chain free tiers so you never hit a quota wall	Free-Tier Aggregation	Set free API keys, call create("chat"). The library detects providers, pools them by capability, and rotates silently when a quota exhausts. Your code sees one provider; ModelMesh manages the rotation
4	Provider goes down, your app stays up	Resilient Routing	Multiple rotation strategies: cost-first, latency-first, round-robin, sticky, rate-limit-aware. On failure the router deactivates the model, selects the next candidate, and retries within the same request
5	Request capabilities, not model names	Capability Discovery	Ask for "chat-completion", not "gpt-4o". ModelMesh resolves to the best available model. New models appear, old ones deprecate, your code stays the same
6	Spending caps enforced before the overage, not after	Budget Enforcement	Real-time cost tracking per model and provider. Set daily or monthly limits in config. BudgetExceededError fires before the breaching request
7	One library for Python backend, TypeScript frontend, Docker proxy	Full-Stack Deployment	pip install, npm install, or docker run. Each exposes the same API with zero core dependencies. One config file drives all deployment modes
8	Test AI code like regular code	Mock Client and Testing	mock_client(responses=[...]) returns an identical API with zero network calls and millisecond execution. Typed exceptions carry structured metadata. client.explain() dry-runs routing decisions
9	Production-grade observability without extra plumbing	Observability Connectors	Pre-built sinks for console, file, JSON-log, Prometheus, and webhooks. Structured traces across routing, failover, and budget events. Plug in custom callbacks for existing dashboards
10	When pre-built doesn't fit, extend without forking	CDK	Base classes for providers, rotation policies, secret stores, storage backends, and observability sinks. Inherit, override what you need, ship as a reusable package

Documentation

Document	Description
System Concept	Architecture, design, and full feature overview
Model Capabilities	Capability hierarchy tree and predefined pools
System Configuration	Full YAML configuration reference
Connector Catalogue	All pre-shipped connectors with config schemas
Connector Interfaces	Interface definitions for all connector types
System Services	Runtime objects: Router, Pool, Model, State
Proxy Guide	Deploy as OpenAI-compatible proxy: Docker, CLI, config, browser access
AI Agent Integration	Guide for AI coding agents (Claude Code, Cursor, etc.) to integrate ModelMesh
Provider Keys Guide	How to obtain API keys for all 22 providers
Troubleshooting	Common issues and solutions

CDK (Connector Development Kit)

Document	Description
CDK Overview	Architecture and class hierarchy
Base Classes	Reference for all CDK base classes
Developer Guide	Tutorials: build your own connectors
Convenience Layer	QuickProvider and zero-config setup
Mixins	Cache, metrics, rate limiter, HTTP client

Samples

Collection	Description
Quickstart	12 progressive examples in Python and TypeScript
System Integration	Multi-provider, streaming, embeddings, cost optimization
CDK Tutorials	Build providers, rotation policies, and more
Custom Connectors	Full custom connector examples for all 6 types
Proxy Test	Vanilla JS browser test page for the OpenAI proxy

Running Samples

Python:

# Install the package first (editable mode for development)
pip install -e "./src/python[yaml]"

# Run any sample
python samples/quickstart/python/00_hello.py

TypeScript:

# Install workspace dependencies (from repo root)
npm install

# Run any sample with tsx
npx tsx samples/quickstart/typescript/00_hello.ts

Most quickstart and CDK samples use built-in mock providers and run without API keys. System integration samples require real provider API keys.

Development

# Clone the repository
git clone https://github.com/ApartsinProjects/ModelMesh.git
cd ModelMesh

# Install all dependencies
pip install -e "./src/python[yaml,dev]"
cd src/typescript && npm install && cd ../..

# Run the full test suite (1,879 tests)
./scripts/test-all.sh

# Or use make
make install
make test

See CONTRIBUTING.md for full setup guide and contribution guidelines.

Docker

# Pull pre-built image from GitHub Container Registry
docker pull ghcr.io/apartsinprojects/modelmesh:latest

# Run with your API keys
docker run -p 8080:8080 \
  -e OPENAI_API_KEY="sk-..." \
  ghcr.io/apartsinprojects/modelmesh:latest

# Or build from source with Docker Compose
cp .env.example .env                      # add your API keys
cp modelmesh.example.yaml modelmesh.yaml   # customize config (optional)
docker compose up --build

# Test the running proxy
curl http://localhost:8080/v1/models
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"text-generation","messages":[{"role":"user","content":"Hello!"}]}'

See the Proxy Guide for full configuration, CLI reference, and browser access.

Scripts

Script	Description
scripts/proxy-up.sh	Build and start the Docker proxy
scripts/proxy-down.sh	Stop the Docker proxy
scripts/proxy-test.sh	Smoke-test a running proxy
scripts/docker-build.sh	Build the Docker image
scripts/install-python.sh	Install Python package (dev or prod)
scripts/install-typescript.sh	Install TypeScript package
scripts/test-all.sh	Run full test suite (Python + TypeScript)
scripts/bump-version.sh	Bump version in both Python and TypeScript
scripts/check-licenses.sh	Verify license headers in source files

Starter Template

Get a new project running with ModelMesh in 30 seconds:

# Python
cp -r tools/starter-template/python/ my-project/ && cd my-project
pip install -r requirements.txt && python main.py

# TypeScript
cp -r tools/starter-template/typescript/ my-project/ && cd my-project
npm install && npx tsx main.ts

See tools/starter-template/ for details.

Security

See SECURITY.md for vulnerability reporting.

License

MIT

---

_{Created by Sasha Apartsin}