ollamux — an Ollama multiplexer: a mux, naturally.
(Previously published as omlx.)
A key-rotating reverse proxy for the Ollama Cloud API. Give it several API keys and it serves them combined behind one local endpoint: requests are spread across keys (round-robin, least-loaded first), and when a key is rate-limited, rejected, or invalid, the request silently retries on the next key before anything reaches your client.
ollama CLI ──┐
OpenAI SDK ──┼──> ollamux :11435 ──> https://ollama.com
curl ──────┘ (key 1..N, per-key concurrency slots, auto-failover)
# 1. Install
cargo install --path .
# 2. Add your keys (https://ollama.com/settings/keys)
mkdir -p ~/.config/ollamux
printf '%s\n' 'your-key-one...' 'your-key-two...' > ~/.config/ollamux/keys
chmod 600 ~/.config/ollamux/keys
# 3. Point any Ollama client at it
export OLLAMA_HOST=http://localhost:11435
ollama run gpt-oss:120b| Source | Install |
|---|---|
| AUR (source build) | paru -S ollamux |
| AUR (prebuilt) | paru -S ollamux-bin |
| Fedora/COPR | dnf copr enable j-stechmann/ollamux && dnf install ollamux |
| Debian (from release assets) | apt install ./ollamux_0.1.0-1_amd64.deb |
| Container | docker run -p 11435:11435 -e OLLAMUX_KEYS=… ghcr.io/j-stechmann/ollamux |
| From source | cargo install --locked ollamux (crates.io) or cargo install --path . |
Prebuilt static binaries (x86_64/aarch64 musl) are attached to each
GitHub release. Packager notes: packaging/README.md.
OpenAI-compatible clients use the same proxy with a base URL:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11435/v1", api_key="unused")One key per line in ~/.config/ollamux/keys (respects XDG_CONFIG_HOME),
or set OLLAMUX_KEYS (newline/comma-separated) to skip the file entirely.
Blank lines and # comments are allowed.
Optionally set a per-key concurrency limit with a :N suffix — the number of
cloud models that account may run at once (free=1, pro=3, max=10):
your-free-key...:1
your-pro-key...:3 # default when no suffix is given
your-max-key...:10
The proxy keeps at most N requests in flight per key (matching Ollama
Cloud's per-plan concurrency limits) and queues the rest briefly rather
than hammering upstream. Requests that wait too long get an honest 429.
| Path | Behavior |
|---|---|
/api/* |
Proxied to https://ollama.com with rotation |
/v1/* |
Proxied (OpenAI-compatible surface) with rotation |
/_keys |
Per-key health JSON (suffixes only, no secrets) |
/_usage |
Per-key usage JSON + pool-capacity aggregate (?refresh=1 forces a refresh, at most one fetch attempt per 5 s) |
/_health |
{"ok":…, "keys":…, "total_slots":…} |
Everything else answers 404 with a hint — this is not a local Ollama
server; it serves no models and ollama list against it shows the cloud
model list, not local models.
The full API surface is described in openapi.yaml
(OpenAPI 3.0): the proxied Ollama/OpenAI paths, the /_keys, /_usage,
/_health introspection endpoints, and ollamux's response headers and
error envelope.
Ollama Cloud publishes plan usage on the documented endpoint
GET https://ollama.com/api/balance (docs.ollama.com/api/balance), one
request per key. Legacy plans (session/weekly limits) answer with what
remains of each window's allowance, reset instants included:
{"included":{"session":{"remaining_percent":75,"resets_at":"2026-10-01T07:00:00Z"},
"weekly": {"remaining_percent":40,"resets_at":"2026-10-05T00:00:00Z"}},
"purchased":{"balance_usd":0}}Credit plans answer with USD amounts instead (included.balance_usd,
included.allowance_usd, included.period, purchased.balance_usd).
Rate limit: 10 requests per minute per user, shared across API keys —
the docs recommend polling once per minute, which ollamux's passive
60 s TTL respects for pools of up to ~10 keys. Larger pools fan out
one GET per key in parallel, so a single round can burst over the
shared per-user limit; a 429'd key keeps its last good row (an as_of
timestamp marks when that data was fetched) and sits out the next
round — or the server's Retry-After, capped at 5 min — so large
pools rotate the fetch budget across keys instead of losing data.
Transient failures (429, 5xx, network blips) carry the last good row;
persistent ones (unauthorized, endpoint gone, payload drift) surface
as per-key error strings, never a crash. (Proxied /api/usage
requests are NOT
special: they go through normal key rotation and reflect whichever key
served them — use /_usage for the per-account picture.)
curl -s localhost:11435/_usage | jq .
curl -s 'localhost:11435/_usage?refresh=1' | jq . # force a refreshForced refreshes are rate-limited to at most one upstream fetch attempt
per 5 s (a ?refresh=1 inside that window serves the cached snapshot —
stale keeps reflecting the 60 s TTL, not this guard). Note this guard
is independent of the TTL: a forced-refresh loop fetches up to 12
times/min per key, over the upstream rate limit even for a single key.
The guard counts
attempts, not successes: while the upstream is failing (failed rounds
keep the last good snapshot), polling loops back off instead of fanning
out on every request.
/_usage fans out to the balance endpoint with every configured key
(one GET per key, in parallel; keys in 429 backoff sit the round out)
and answers with one row per key (suffixes
only, never secrets): on legacy plans the used fraction of each window —
session/weekly in 0.0–1.0 (upstream's remaining percent, inverted),
one-decimal *_pct mirrors, and session_resets_at/weekly_resets_at
for countdowns — or, on credit-plan keys, included_usd/allowance_usd
and purchased_usd with no percent windows, or a per-key error
otherwise. Rows carried forward from a previous round (the key's latest
fetch failed transiently or was skipped in 429 backoff) carry as_of,
the unix time of their original fetch — absent means fresh this round.
Each row also carries the key's plan tier, inferred from
its per-key concurrency (KEY:N — free=1, pro=3, max=10; more
concurrency than normal is the next tier up). Responses are cached for
60 s (updated/age_s/stale fields tell you the age); /_keys embeds
the latest known usage per key from the same cache — it never triggers an
upstream call itself, and usage checks never touch key health (a 401
there is reported, not treated as a dead key).
The envelope's aggregate is the capacity-weighted mean of every
legacy-plan reporting key's usage fraction, weighted by its tier's cap —
free ×1, pro ×50, max ×250 (pro has 50× free's usage cap, max 5× pro's) —
so it reads as the fraction of the pool's combined capacity in use and
stays in 0.0–1.0 like the per-key numbers: a single-key pool reproduces
that key's own fraction, and applications built for the per-key range
need no adjustment. All keys contribute, including ones currently on
cooldown (usage fetching is health-blind) and ones whose latest fetch
failed transiently (carried rows keep contributing their last known good
fraction); a dead key (401/403) is
expected to fail its own usage fetch the same way and contributes
nothing (its weight is excluded from the denominator, so it never
dilutes the number). Windows nobody reported are null, never 0; when
every reporting key of a window carries the same resets_at, the
aggregate publishes it (session_resets_at/weekly_resets_at) so
consumers get one honest reset instant for the pool. Credit-plan rows
carry no window fractions and sit out the aggregate (their unit is USD,
not a fraction of an allowance).
{"updated":1756620000,"age_s":3,"stale":false,
"session_window":"plan session window (reset timestamps upstream; poll courtesy 60 s)",
"aggregate":{"session":0.807,"weekly":0.418,
"session_resets_at":"2026-10-09T00:00:00Z",
"unit":"pool capacity fraction"},
"keys":[
{"index":0,"suffix":"1234","ok":true,"tier":"free",
"session":0.037,"weekly":0.007,"session_pct":3.7,"weekly_pct":0.7,
"session_resets_at":"2026-10-09T00:00:00Z",
"weekly_resets_at":"2026-10-12T00:00:00Z"},
{"index":1,"suffix":"5678","ok":true,"tier":"max",
"session":0.81,"weekly":0.42,"session_pct":81.0,"weekly_pct":42.0,
"session_resets_at":"2026-10-09T00:00:00Z",
"weekly_resets_at":"2026-10-12T00:00:00Z",
"as_of":1756619955}]}(In the example, key 5678's latest fetch failed transiently — its data
was fetched 45 s before the snapshot completed, hence as_of; key
1234 is fresh and carries no as_of.)
ollamux --usage-aware # demote keys at/over 80% session usage
ollamux --usage-aware=90 # custom threshold (1–99)
# or: OLLAMUX_USAGE_AWARE=80 (the flag wins when both are set)When enabled, ollamux polls the balance endpoint every 60 s and orders candidate keys so that keys whose session usage is at/over the threshold are served last — demoted, never excluded: an over-quota key still takes requests when no fresh key has a free slot. Failed usage fetches keep the previous snapshot; with the feature off, routing behaves exactly as before.
Credit-plan keys are exempt from demotion: their balance responses carry
USD amounts, not a session fraction, so there is no percent to compare
against the threshold — under --usage-aware they are ordered like
unmeasured keys (demote only ever applies to legacy-plan percent data).
The demotion signal is the session window alone: a key whose weekly
window is exhausted but whose session window is under the threshold is
never demoted.
Ollama Cloud caches prompt prefixes server-side, per account (i.e. per
API key). ollamux's normal least-loaded routing would send successive
requests of one conversation to different keys and re-ingest the same
prefix every turn. With affinity, ollamux computes a conversation
identity from the request body (model + the leading system messages +
first user message for chat; model + first 8 KiB of the prompt for
/api/generate and /v1/completions) and pins it to the key that last
served it — as long as that key is healthy, has a free slot, and is not
demoted by --usage-aware. When the pinned key cannot serve right now,
the request is routed normally (no waiting on a busy key); a cache miss
just means the next success re-pins.
- Disable with
--no-affinityorOLLAMUX_NO_AFFINITY=1. - Responses served by a pool key carry
X-Ollamux-Affinity: hit|miss|off(off= disabled or the body gave no identity; proxy-generated failures where no key was selected — admission rejects, exhausted failover — omit the header).hitmeans a pin existed and was usable (healthy, under quota, free slot) at admission;misscovers first requests, evictions, and pins that fell through. - Helps append-only conversations (the common case: chat turns only extend the messages array). Clients that rewrite the system prompt or trim history mid-conversation simply get the old rotation behavior.
- The pin map holds 4096 conversations (LRU-evicted, in-memory only).
- 429 (rate limit): the key cools down (60 s, or the server's
Retry-After, capped at 5 min) and the request retries on the next key. - 401/403 Unauthorized (invalid key): the key is marked dead until restart; the request retries on the next key.
- 5xx / network errors: retry the next key; three consecutive failures put a key in cooldown. Successes reset the counter. (Merely admitting a request to a key never resets its strike counter — only confirmed upstream successes do.)
- Everything else (e.g. a 400 from a malformed request) is passed through untouched — that's your bug, not a key problem.
- Failover happens before the first response byte, so streaming responses (NDJSON and SSE) are never corrupted mid-flight by a key switch.
If every key is cooling down or dead, you get a clear JSON error instead of
a mysterious upstream one — see /_keys for per-key state.
- Logs: silent by default.
-vadds a per-request stderr line (retries=N,key=<suffix>), key cooldown/death events, startup banner, shutdown notices, and upstream error snippets; fatal errors always print. - Response headers include
X-Ollamux(ollamux/version — every response, including relayed upstream errors, is attributable),X-Ollamux-Key(which key served it),X-Ollamux-RetriesandX-Ollamux-Affinity(prompt-cache affinity:hit/miss/off; present only when a key served the request). SIGINT(ctrl-c) drains in-flight requests for up to 5 s, then exits; press again to force-quit.- Request bodies are buffered up to 16 MiB (needed for replay across
failover); larger bodies get
413. - Binds
127.0.0.1:11435by default (--addrto change). The default port is not 11434 so it can run alongside a real local Ollama.
- Usage introspection is read-only reporting; there is no token accounting
or historical dashboard —
/_usagemirrors what ollama.com exposes per account (the only derived figure is the capacity-weighted poolaggregate, which ollama.com itself does not publish). - No request rewriting: models must exist on ollama.com.
- A dead key stays dead until restart (
/_keysshows why). Restarting is cheap: it's stateless. - Localhost trust model: anything that can reach the port can use your quota. Don't expose it beyond loopback.
Ollama's terms currently describe one account per person; this tool exists to pool keys you legitimately hold (e.g. personal + work seats). Don't use it to evade per-user limits you're not entitled to.
cargo test # unit + integration (hermetic: local upstream)
cargo test --features net # also hits real ollama.com lightly
cargo clippy --all-targets -- -D warnings
cargo fmt --checkdevelopis the integration branch,masterholds release tags. Both are protected: changes land via PR only, gated on thecheckandpackaging-lintCI checks (enforced for admins too).- Feature branches (
feat/...,fix/...,release/vX.Y.Z) PR intodevelop. - Releases PR
developintomasterand tagvX.Y.Zonmaster; the tag push triggers the release pipeline (packaging/README.md has the full checklist).
- Invalid/concurrency counts in the keys file are startup errors, not
silently defaulted:
KEY:99999999999refuses to start, duplicate key lines are rejected (they would double-count slots), and a keys file with only comments refuses to start. - Non-ASCII key lines are fine (suffixes use character boundaries).
- Upstream redirects are not followed (the proxy classifies or relays them verbatim); query strings are forwarded to upstream untouched.
- Relayed upstream error bodies are byte-exact (no truncation).
- SIGINT shutdown is signal-mask based: the first ctrl-c starts the drain, the second force-quits (exit code 130).