Unified LLM service access layer — Rust core, one API to access 290+ AI providers, 8 language bindings
This document compiles all of aimux's design decisions and benchmark conclusions, for use in README, blogs, and technical talks.
aimux is a Rust replacement for the Vercel AI SDK. It converges the HTTP APIs of 290+ AI providers into a single unified interface, uses a Rust core to deliver a 7-15x performance advantage, and covers eight language ecosystems — Node / Python / Go / Java / Kotlin / Swift / Flutter / C — through FFI bindings.
aimux does not do agent loops, RAG, or orchestration — it focuses solely on unifying service access. This is the fundamental difference between it and LangChain / Mastra: the former is an access layer, the latter is an orchestration layer.
| Metric | Value |
|---|---|
| Rust code | 144,500+ lines |
| AI providers | 329 (251 registry-backed OpenAI-compatible + 10 native protocols + 38 standalone/local/speech/image/video) |
| Modality traits | 8 (text/embedding/image/video/speech/transcription/reranking/search) |
| Test cassettes | 2,650 recorded replays |
| Test files | 118 |
| Rust crates | 7 core + 8 bindings |
| Language bindings | 8 (Node / Python / Go / Java / Kotlin / Swift / Flutter / C) |
| Type definitions | 79 (auto-generated by ts-rs) |
Provider access (RFC-0017 phase 4): all OpenAI-compatible providers are registry-backed —
provider-registry.jsonis the single source of truth, and every binding exposesprovider(name, ...)with a typedProviderName(enum/union/consts). The per-provider shell types (XxxConfig/XxxProvider) were retired in phase 4.
Environment: Linux x64, 32 cores, Node v24.18.0, Python 3.12.13 Method: same process, same mock server, fixed responses, N=200-300 iterations to compute statistics Full data: PERF-RESULTS.md
Clean numbers at the same abstraction layer (HTTP + JSON, no orchestration/schema validation/middleware).
| mean | P50 | P95 | P99 | RSS growth | |
|---|---|---|---|---|---|
| aimux (Node) | 0.101ms | 0.096 | 0.122 | 0.139 | +2MB |
| OpenAI Node SDK | 1.488ms | 1.500 | 1.637 | 1.923 | +17MB |
| aimux (Python) | 0.080ms | 0.075 | 0.108 | 0.129 | +0MB |
| OpenAI Python SDK | 0.595ms | 0.577 | 0.695 | 0.839 | +8MB |
- Node: aimux is 14.7x faster, uses 8.5x less memory
- Python: aimux is 7.5x faster, zero memory growth
| Scenario | SDK | rps | mean | P99 | RSS growth |
|---|---|---|---|---|---|
| 32 cores | aimux | 1512 | 0.66ms | 1.92ms | +23MB |
| AISDK | 563 | 1.78ms | 3.96ms | +103MB | |
| 1 core | aimux | 1497 | 0.67ms | 1.65ms | +21MB |
| AISDK | 473 | 2.11ms | 12.87ms | +60MB |
Key findings:
- When CPU-bound, aimux's performance does not drop (1512→1497 rps), while AISDK plummets (563→473 rps)
- aimux has no GC pauses: at 1 core, P99 stays stable at 1.65ms; AISDK's P99 spikes to 12.87ms (GC jitter)
- Python aimux's RSS does not grow by a single byte after 2000 requests
aimux: Node app → napi FFI → Rust core → reqwest → HTTP
AISDK: Node app → TS core → undici → HTTP
↑
V8 GC + Zod validation + middleware pipeline + telemetry
- The Rust core has zero GC, with predictable memory allocation patterns
- reqwest connection pool reuse (after RFC-0009 landed)
- Does not do Zod schema validation / fetch middleware / telemetry recording — these are the sources of AISDK's overhead
- PyO3 (Python binding) is lighter than napi (Node binding) — direct C API calls, without going through V8 wrapping
| Comparison | Multiple | Equivalent? | Notes |
|---|---|---|---|
| vs OpenAI Node SDK | 14.7x | ✅ Equivalent | Both are HTTP + JSON, no orchestration layer |
| vs OpenAI Python SDK | 7.5x | ✅ Equivalent | Same as above |
| vs Vercel AI SDK | ~11x | ❌ Not equivalent | AISDK includes Zod validation/middleware/telemetry; the 11x is inflated |
The comparison between aimux and the OpenAI official SDK is truly equivalent — both only do HTTP + JSON. The Vercel AI SDK additionally performs Zod schema validation, builds a typed object tree, runs a fetch middleware pipeline, and records telemetry on every request; these accumulate in the V8 heap and cause memory bloat.
| Trait | Capability | Example Providers |
|---|---|---|
LanguageModel |
Text generation + streaming + tool calling | OpenAI / Anthropic / Google / DeepSeek / 325 providers |
EmbeddingModel |
Vector embeddings | OpenAI / Cohere / Voyage / generic-compatible |
ImageModel |
Image generation | Black Forest Labs / Replicate / Fal / KlingAI |
VideoModel |
Video generation | Google Veo / Replicate |
SpeechModel |
Speech synthesis (TTS) | OpenAI / ElevenLabs / Cartesia |
TranscriptionModel |
Speech-to-text (STT) | OpenAI / Deepgram / AssemblyAI |
RerankingModel |
Reranking | Cohere / Voyage |
SearchModel |
Search | Tavily / Exa / Jina |
| Type | Count | Representatives |
|---|---|---|
| Native protocol | 11 | OpenAI, Anthropic, Google, Bedrock, Vertex, Azure, Cohere, Mistral, xAI, DeepSeek |
| OpenAI-compatible (registry) | 251 | Groq, Fireworks, Together, Perplexity, Ollama, OpenRouter, Alibaba Tongyi, Zhipu, Baidu, Tencent, iFlytek, Moonshot AI, SiliconFlow… |
| Voice/transcription | 7 | ElevenLabs, Deepgram, AssemblyAI, Cartesia… |
| Image/video | 8 | Black Forest Labs, Replicate, Fal, KlingAI… |
| Search | 11 | Tavily, Exa, Serper, Firecrawl… |
See rfc/0004-provider-inventory.md for the full list.
All registry providers are accessed via the unified provider(name, ...) entry point
— see API.md.
aimux's type design is already aligned with Vercel AI SDK V4, and the core structures are highly consistent:
| Dimension | Consistency | Notes |
|---|---|---|
GenerateResult |
✅ | Core fields consistent (content/finish_reason/usage/response) |
StreamPart |
✅ | aimux 18 variants vs V4 ~21 variants, core paths aligned |
Role |
✅ | Fully consistent (system/user/assistant/tool) |
FinishReason |
✅ | Basically consistent (unified + raw dual fields) |
ToolResult |
✅ | Fields fully aligned (result/is_error/preliminary/dynamic/tool_name) |
File variant |
✅ | Both GenerateContent/StreamPart have a File variant |
ToolChoice |
🟡 | Different format (bare string vs object), wrapper can convert |
| Naming | 🟡 | snake_case vs camelCase, wrapper unifies the mapping |
The 79 type definitions are auto-generated by Rust's ts-rs, ensuring the Rust core and TypeScript types are always in sync.
See type-comparison-aisdk.md for the full comparison.
aimux/
├── aimux-core # core abstractions: 8 traits + type definitions
│ ├── LanguageModel # object-safe, supports Box<dyn> for cross-provider swapping
│ ├── EmbeddingModel
│ ├── ImageModel / VideoModel
│ ├── SpeechModel / TranscriptionModel
│ └── RerankingModel / SearchModel
├── aimux-providers # 325 provider implementations
│ ├── 11 native protocols # standalone model + convert, handles provider-specific differences
│ ├── 251 OpenAI compatible # registry-backed: provider-registry.json + provider(name, ...) entry (RFC-0017 phase 4)
│ └── modalities/search # voice / image / video / search implementations
├── aimux-stream # SSE / NDJSON streaming parsing
├── aimux-provider-utils # One-exchange HTTP helpers, response handlers, API-key loading
├── aimux-ffi # C ABI (FFI infrastructure, shared by all bindings)
└── bindings/ # 6 language bindings
├── node/ # napi-rs v3 + typed TS wrapper
├── python/ # PyO3
├── swift/ # Swift Package
├── kotlin/ # Kotlin/JVM
├── flutter/ # dart:ffi
└── c/ # C ABI header file
// aimux-core: the trait is object-safe, supports Box<dyn LanguageModel>
pub trait LanguageModel: Send + Sync {
fn model_id(&self) -> &str;
fn do_generate(&self, prompt: ModelPrompt, options: CallOptions)
-> impl Future<Output = Result<GenerateResult, AiMuxError>>;
fn do_stream(&self, prompt: ModelPrompt, options: CallOptions)
-> impl Future<Output = Result<Stream, AiMuxError>>;
}This means Box<dyn LanguageModel> can be swapped between OpenAI / Anthropic / Google — switching providers only requires changing the provider construction, and the model usage stays exactly the same.
A thin wrapper does not lose differences. Each OpenAI-compatible service has subtle differences (some support top_k, some do not support tools, some have a different streaming usage format):
pub struct OpenAICompatProfile {
pub supports_top_k: bool, // Groq supports it, OpenAI does not
pub supports_tools: bool, // some services do not support it
pub supports_response_format: bool,
pub streaming_usage_format: UsageFormat, // streaming usage in data line vs chunk line
pub post_process_request: Option<fn(&mut serde_json::Value)>,
}Use a profile descriptor struct to express the differences, rather than writing an independent model for each provider — this is why the 145 compatible providers only need a thin wrapper.
All bindings communicate with the Rust core via JSON strings:
JS/Python call → JSON.stringify → serde_json::from_str → Rust core → serde_json::to_string → JSON.parse → JS/Python object
Benefit: the 6 bindings share the same Rust core, with completely consistent cross-language behavior. Cost: 5-6 serializations per call. But the serialization overhead is ~0.01-0.05ms, accounting for < 0.025% of a real LLM request (200-2000ms) — negligible.
2,650 real API response recordings, with no dependency on network or keys:
Testing: mock server replays fixed responses → deterministic testing
Development: recording mode → real API calls → stored as cassette
CI: full replay, zero network dependency
Covers scenarios such as tool calling, multi-turn dialogue, reasoning/thinking, and structured output. See rfc/0003-test-cassette.md for the full approach.
shared_client()connection pool sharing (providers no longer each establish their own connections)- TLS session reuse
- Full Jitter backoff lives in
aimux-core::retryas theget_delay_msdefault (server retry hints are honored exactly, not jittered; RFC-0031) - Fixed timeout
The provider layer does not directly depend on reqwest; it abstracts the HTTP client through a trait in aimux-provider-utils — reqwest does not leak into the provider layer, and can be replaced with hyper or other HTTP backends in the future.
| Dimension | aimux | AI SDK | Advantage |
|---|---|---|---|
| Performance (Node equivalent comparison) | 0.101ms | 1.488ms (OpenAI SDK) | aimux 14.7x |
| Performance (Python equivalent comparison) | 0.080ms | 0.595ms (OpenAI SDK) | aimux 7.5x |
| Provider coverage | 290+ | ~20 | aimux 14.5x |
| Modalities | 8 | 6 | aimux (more video/STT/reranking/search/file) |
| Memory (2000 req) | +0~2MB | +60~144MB | aimux |
| GC pauses | None (Rust) | V8 GC jitter | aimux |
| Language bindings | 8 | 1 (Node) | aimux |
| Type-safe DX | ✅ Complete (ts-rs 79 types, wrapper already exists) | Zod fully inferred | 🟢 Close |
| Agent loop | ❌ Not done (design decision) | stopWhen + execute |
AI SDK |
| Data model | Aligned with V4, interchangeable | V4 native | 🟢 Consistent |
aimux's positioning: a unified access layer, no orchestration. The Node binding already has complete types (ts-rs auto-generates 79 types + typed wrapper), and the agent loop is left to upper-layer frameworks (LangChain / Mastra).
Fundamental difference: aimux is an access layer, LangChain/Mastra is an orchestration layer.
Your app
└── Orchestration layer (LangChain / Mastra / custom loop)
└── Access layer (aimux) ← here
└── 325 AI providers
aimux does not compete with LangChain; instead it serves as the layer beneath LangChain — LangChain handles the agent loop / RAG / chain, while aimux handles unified access to 325 providers. Running the access layer in Rust delivers performance and memory behavior far exceeding a JS-implemented access layer.
| Dimension | aimux | rig / rust-genai |
|---|---|---|
| Provider count | 290+ | ~20-40 |
| Multi-language bindings | 8 | Rust only |
| Multimodal | 8 traits | Partial |
| Recorded testing | 2,650 cassettes | Few |
| Positioning | Pure access layer | Access + partial orchestration |
- Multi-provider aggregation: an app needs to access 5+ AI providers and doesn't want to write an adapter for each
- Performance-sensitive: high-concurrency API gateways / batch processing / real-time interaction, where GC pauses are unacceptable
- Multi-language stack: a Node + Python + mobile hybrid tech stack that wants a single unified AI access
- Cost control: needs to run AI services on low-spec servers with small memory footprint
- Provider replaceability: needs to switch / downgrade / mix LLM providers without changing business code
- Needs an agent loop: wants
stopWhen/execute/ multi-step reasoning loops — layer LangChain / Mastra on top of aimux - Only needs OpenAI: using the OpenAI official SDK directly is simpler
- Needs RAG / vector store orchestration: aimux only provides embedding calls and does not do RAG pipelines
npm install @arcships/aimuximport { openai, generateText, streamText } from '@arcships/aimux'
const model = await openai(process.env.OPENAI_API_KEY!, 'gpt-4o')
// non-streaming
const result = await generateText(model, 'What is Rust?')
console.log(result.text)
// streaming
const { stream } = await streamText(model, 'Write a haiku about Rust.')
for await (const part of stream) {
if (part.TextDelta) process.stdout.write(part.TextDelta.delta)
}
// switch provider: only change the provider
const deepseekModel = await deepseek(DEEPSEEK_API_KEY, 'deepseek-chat')
// model usage is exactly the samepip install arcships-aimuxfrom aimux import openai, generate_text, stream_text
model = openai("sk-...", "gpt-4o")
result = generate_text(model, "What is Rust?")
print(result["text"])
# streaming
result = stream_text(model, "Write a haiku about Rust.")
for part in result:
if "TextDelta" in part:
print(part["TextDelta"]["delta"], end="")use aimux_core::prelude::*;
use aimux_providers::{OpenAIConfig, OpenAIProvider};
#[tokio::main]
async fn main() -> Result<(), AiMuxError> {
let provider = OpenAIProvider::new(OpenAIConfig::new("sk-..."));
let model = provider.model("gpt-4o");
let result = generate_text(
&model,
"Explain Rust ownership in one sentence.",
GenerateTextOptions::default(),
).await?;
println!("{}", result.text);
Ok(())
}See API.md for the full API documentation.
One Rust core covers 7 language ecosystems via FFI:
- napi-rs (Node) — V8 native binding
- PyO3 (Python) — CPython native binding
- Swift Package (iOS/macOS)
- Kotlin/JVM (Android)
- dart:ffi (Flutter)
- C ABI (general FFI infrastructure)
Key engineering decision: JSON string boundary. Rust structs are not passed across the FFI; only JSON strings are passed. The benefit is that all bindings share the same Rust core with completely consistent cross-language behavior; the cost is 5-6 serializations, but the overhead is < 0.025%, negligible.
Not a single trick, but systematic design choices:
| Factor | aimux | Competitors | Impact |
|---|---|---|---|
| Language | Rust (zero GC) | TS/Python (GC jitter) | P99 stability |
| HTTP client | reqwest + connection pool | undici/httpx | Connection reuse |
| Serialization | serde_json (fastest in Rust) | JSON.parse + Zod | CPU overhead |
| Does not do | Zod validation / middleware / telemetry | Does | Extra CPU per request |
| Compilation | AOT compiled to native code | JIT | Cold start + steady state |
Don't write an independent model for each provider — that would explode. Use OpenAICompatProfile to describe the differences:
// one profile describes a compatible provider's differences
let profile = OpenAICompatProfile {
supports_top_k: true, // Groq supports top_k
supports_tools: true, // supports function calling
streaming_usage_format: UsageFormat::InDataLine,
post_process_request: Some(strip_unsupported_fields),
..Default::default()
};The 11 native protocols have independent models + convert (handling differences such as Anthropic message format / Google generateContent / Bedrock SigV4), while the 251 OpenAI-compatible providers are registry-backed (provider-registry.json + unified provider(name, ...) entry, RFC-0017 phase 4).
Tests do not depend on network or keys. Each CI run replays the full set of 2650 cassettes, guaranteeing regression safety for protocol conversion. The cassettes come from real API responses and cover scenarios such as tool calling, multi-turn dialogue, reasoning/thinking, and structured output.
aimux's type design directly targets Vercel AI SDK V4 provider types — the GenerateResult / StreamPart / Usage / ToolChoice structures are aligned. This is not a coincidence; it is a design goal. The 79 TypeScript types are auto-generated by Rust's ts-rs, ensuring the Rust core and TS types are always in sync.
- 329 providers integrated (10 native + 251 registry-backed OpenAI-compatible + voice/image/video/search)
- 8 modality traits (text/embedding/image/video/speech/transcription/reranking/search)
- 7 language bindings (Node/Python/Swift/Kotlin/Flutter/C/Rust)
- 2650 cassette recorded tests
- Performance benchmark: 14.7x (Node) / 7.5x (Python) lead
- Request-layer optimization (RFC-0009: connection pool + backoff + timeout + reqwest decoupling)
- Data model aligned with AI SDK V4
- C ABI full modality coverage (16 extern functions)
- Python wrapper defaulting (currently requires explicit
from aimux.wrapper import ..., to be changed to default export) - Unify wrappers across languages (camelCase naming + typed boundary)
- Performance benchmark dimension three: concurrency capability curve + memory growth chart
- More native protocols (Together / Fireworks / Anyscale specialization)
- Streaming TTFT (Time To First Token) benchmark
| Document | Content |
|---|---|
| README.md | Project homepage |
| docs/API.md | Full API documentation |
| docs/PERF-RESULTS.md | Full performance benchmark data |
| docs/aimux-vs-aisdk-node.md | Node.js experience comparison |
| docs/type-comparison-aisdk.md | Type comparison (V4 alignment) |
| docs/audit-001-feature-coverage.md | Feature audit report |
| docs/cross-lang-dx-plan.md | Cross-language DX unification plan |
| rfc/0001-multilang-bindings.md | Multi-language binding design |
| rfc/0004-provider-inventory.md | Provider inventory |
| rfc/0005-protocol-conversion.md | Protocol conversion design |
| rfc/0009-request-resilience.md | Request-layer optimization |
| rfc/0010-perf-benchmark-vs-aisdk.md | Performance comparison benchmark plan |
| Date | Version | Notes |
|---|---|---|
| 2026-07-30 | v1.0 | Initial version, compiling all design decisions + benchmark conclusions |