Free-first LLM routing for Python: local Ollama ($0) -> OpenRouter free models ($0) -> paid Anthropic API, with automatic fallback and a savings ledger. Zero required dependencies.
Most teams route every prompt to the same frontier model regardless of how hard the task actually is, and the API bill grows with usage instead of with value delivered. llm-cascade flips that: it tries the cheapest tier that can plausibly do the job first, and only escalates when that tier is unreachable or genuinely too weak for the task. In production, that means classification, extraction, and summarization run for $0 on a local model or a free cloud model, while code review, planning, and long-form writing still get a frontier model -- and if the paid API is ever down or out of credits, your scheduled jobs degrade to a free tier instead of crashing.
pip install llm-cascadeThe base package has zero required third-party dependencies. To use the paid Anthropic tier, install the extra:
pip install "llm-cascade[anthropic]"Without it, the local (Ollama) and free-cloud (OpenRouter) tiers still work end to end -- you'll only see an error if a task actually routes to the paid tier and the SDK isn't installed.
from llm_cascade import route
result = route("summarize", "Paste a long document here...")
print(result.text) # the completion
print(result.tier) # which tier served it: trivial/local/cheap/light/heavy
print(result.model) # e.g. "local:ollama", "openrouter:qwen/...", "claude-sonnet-5"
print(result.cost_usd) # 0.0 for local/free-cloud tiersOr from the command line:
llm-cascade "Summarize this text: ..." --task summarize
llm-cascade --routes # show the tier ladder and task -> tier table
llm-cascade --stats # show cumulative cost and free-vs-paid call countsEach call resolves a task name (or a tier name directly) to a tier, then runs that tier's provider chain in order until one succeeds:
| Tier | Chain | Use for |
|---|---|---|
trivial |
local Ollama only ($0) | high-volume micro-tasks, never worth the network |
local |
local Ollama ($0) -> OpenRouter free ($0) -> paid | mechanical NLP: classify, extract, summarize, translate |
cheap |
OpenRouter free ($0) -> paid | manual opt-in tier, invoke directly with task="cheap" |
light |
paid (Sonnet); free-cloud only if LLM_CASCADE_LIGHT_FREE=1 |
customer-facing copy that needs reliable polish |
heavy |
paid (Opus), adaptive thinking, high effort | real reasoning: code, review, planning, long-form |
If the paid API call fails for any reason -- bad or revoked key, no credits, a provider outage, a network blip -- llm-cascade degrades to the free tiers instead of raising, so scheduled or automated callers don't break outright. It only raises once every tier has been tried and failed.
The default task -> tier table:
classify, extract, summarize, translate, chat -> local
rewrite -> light
code, review, plan, creative -> heavy
Anything not in that table falls back to default_task_tier (light by
default). See config/routes.example.toml for the full, commented schema.
examples/agent_workflow.py composes three roles
using the existing route() API:
goal -> planner -> worker -> reviewer -> accepted or rejected design
plan code review (no execution or publication)
From a source checkout:
pip install -e .
python examples/agent_workflow.py --demo
# Explicitly opt in to real configured providers, which may incur API costs:
python examples/agent_workflow.py --live --goal "Outline an API health check"The demo uses fixed fixtures and needs no keys or network. It demonstrates handoffs and failure handling, not model quality. Live mode uses the configured task tiers and existing provider fallback. The workflow makes at most three router calls, limits output tokens per call, validates each role's JSON, and records the actual model, tier and reported cost. Provider fallback may make more than one provider request per router call; this is not a hard spending cap. A rejected review stays rejected, and malformed output stops the workflow.
This is a small integration example, not a durable agent runtime or a guarantee
against prompt injection. It has no tools, code execution, automatic repair
loop, publishing or authority to change external systems. Treat live designs
as material for human review. Avoid sending secrets or private source data to
cloud providers. For monitoring, write an accepted_at ledger entry only after
the required acceptance checks pass; keep attempt traces in a separate file.
See the companion
accepted-result monitoring example.
Configuration is resolved with this precedence (lowest to highest):
- Built-in defaults.
- An optional TOML file:
~/.llm_cascade/routes.toml, or the path given inLLM_CASCADE_CONFIG. - Environment variables, for the handful of settings that make sense as env vars.
Copy config/routes.example.toml to
~/.llm_cascade/routes.toml to add your own task names, retune which tier
handles a task, or override model ids and pricing.
Environment variables:
| Variable | Purpose |
|---|---|
ANTHROPIC_API_KEY |
Enables the paid tier (light/heavy, and the final safety net). |
OPENROUTER_API_KEY |
Enables the free-cloud tier (OpenRouter :free models). |
LLM_CASCADE_CONFIG |
Path to a TOML config file, instead of ~/.llm_cascade/routes.toml. |
LLM_CASCADE_LOG_PATH |
Path to the savings ledger, instead of ~/.llm_cascade/log.jsonl. |
LLM_CASCADE_LIGHT_FREE |
Set to 1 to try free-cloud first on the light tier too. |
LLM_CASCADE_NO_FREE_CLOUD |
Set to any value to disable the free-cloud tier globally. |
LLM_CASCADE_HTTP_REFERER / LLM_CASCADE_APP_TITLE |
Sent as HTTP-Referer / X-Title on OpenRouter requests (optional; omitted if unset). |
LLM_CASCADE_OPENROUTER_MODEL / LLM_CASCADE_OPENROUTER_MODELS |
Override the free model (singular) or ordered list (comma-separated) tried on OpenRouter. |
LLM_CASCADE_OLLAMA_BASE_URL |
Ollama server URL (default http://localhost:11434). |
See .env.example for a copy-pasteable starting point.
Every call -- free or paid -- appends one line to a JSONL ledger
(~/.llm_cascade/log.jsonl by default). stats() / llm-cascade --stats
summarize it:
$ llm-cascade --stats
{
"paid_calls": 3,
"free_calls": 41,
"total_cost_usd": 0.0842,
"by_model": {
"local:ollama": 30,
"openrouter:qwen/qwen3-next-80b-a3b-instruct:free": 11,
"claude-sonnet-5": 3
}
}Unlike heavier multi-provider routers, llm-cascade ships with zero required dependencies and a single opinionated cascade -- it's a routing policy you can read in one file, not a platform.
Issues and pull requests are welcome at github.com/luandv92/llm-cascade. Before opening a PR:
pip install -e ".[dev,anthropic]"
ruff check .
pytest tests/ -vTests never touch real networks -- Ollama, OpenRouter, and the Anthropic SDK
are all mocked (see tests/conftest.py).
MIT -- see LICENSE.