Skip to content

About

Free-first LLM routing for Python: local Ollama -> OpenRouter free tier -> paid Anthropic API, with automatic fallback and a savings ledger. Zero required dependencies.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

llm-cascade

CI PyPI License: MIT Python versions

Free-first LLM routing for Python: local Ollama ($0) -> OpenRouter free models ($0) -> paid Anthropic API, with automatic fallback and a savings ledger. Zero required dependencies.

Why

Most teams route every prompt to the same frontier model regardless of how hard the task actually is, and the API bill grows with usage instead of with value delivered. llm-cascade flips that: it tries the cheapest tier that can plausibly do the job first, and only escalates when that tier is unreachable or genuinely too weak for the task. In production, that means classification, extraction, and summarization run for $0 on a local model or a free cloud model, while code review, planning, and long-form writing still get a frontier model -- and if the paid API is ever down or out of credits, your scheduled jobs degrade to a free tier instead of crashing.

Install

pip install llm-cascade

The base package has zero required third-party dependencies. To use the paid Anthropic tier, install the extra:

pip install "llm-cascade[anthropic]"

Without it, the local (Ollama) and free-cloud (OpenRouter) tiers still work end to end -- you'll only see an error if a task actually routes to the paid tier and the SDK isn't installed.

Quickstart

from llm_cascade import route

result = route("summarize", "Paste a long document here...")
print(result.text)        # the completion
print(result.tier)        # which tier served it: trivial/local/cheap/light/heavy
print(result.model)       # e.g. "local:ollama", "openrouter:qwen/...", "claude-sonnet-5"
print(result.cost_usd)    # 0.0 for local/free-cloud tiers

Or from the command line:

llm-cascade "Summarize this text: ..." --task summarize
llm-cascade --routes   # show the tier ladder and task -> tier table
llm-cascade --stats    # show cumulative cost and free-vs-paid call counts

How routing works

Each call resolves a task name (or a tier name directly) to a tier, then runs that tier's provider chain in order until one succeeds:

Tier Chain Use for
trivial local Ollama only ($0) high-volume micro-tasks, never worth the network
local local Ollama ($0) -> OpenRouter free ($0) -> paid mechanical NLP: classify, extract, summarize, translate
cheap OpenRouter free ($0) -> paid manual opt-in tier, invoke directly with task="cheap"
light paid (Sonnet); free-cloud only if LLM_CASCADE_LIGHT_FREE=1 customer-facing copy that needs reliable polish
heavy paid (Opus), adaptive thinking, high effort real reasoning: code, review, planning, long-form

If the paid API call fails for any reason -- bad or revoked key, no credits, a provider outage, a network blip -- llm-cascade degrades to the free tiers instead of raising, so scheduled or automated callers don't break outright. It only raises once every tier has been tried and failed.

The default task -> tier table:

classify, extract, summarize, translate, chat  -> local
rewrite                                        -> light
code, review, plan, creative                   -> heavy

Anything not in that table falls back to default_task_tier (light by default). See config/routes.example.toml for the full, commented schema.

Bounded agent workflow example

examples/agent_workflow.py composes three roles using the existing route() API:

goal -> planner -> worker -> reviewer -> accepted or rejected design
          plan      code      review       (no execution or publication)

From a source checkout:

pip install -e .
python examples/agent_workflow.py --demo
# Explicitly opt in to real configured providers, which may incur API costs:
python examples/agent_workflow.py --live --goal "Outline an API health check"

The demo uses fixed fixtures and needs no keys or network. It demonstrates handoffs and failure handling, not model quality. Live mode uses the configured task tiers and existing provider fallback. The workflow makes at most three router calls, limits output tokens per call, validates each role's JSON, and records the actual model, tier and reported cost. Provider fallback may make more than one provider request per router call; this is not a hard spending cap. A rejected review stays rejected, and malformed output stops the workflow.

This is a small integration example, not a durable agent runtime or a guarantee against prompt injection. It has no tools, code execution, automatic repair loop, publishing or authority to change external systems. Treat live designs as material for human review. Avoid sending secrets or private source data to cloud providers. For monitoring, write an accepted_at ledger entry only after the required acceptance checks pass; keep attempt traces in a separate file. See the companion accepted-result monitoring example.

Configuration

Configuration is resolved with this precedence (lowest to highest):

  1. Built-in defaults.
  2. An optional TOML file: ~/.llm_cascade/routes.toml, or the path given in LLM_CASCADE_CONFIG.
  3. Environment variables, for the handful of settings that make sense as env vars.

Copy config/routes.example.toml to ~/.llm_cascade/routes.toml to add your own task names, retune which tier handles a task, or override model ids and pricing.

Environment variables:

Variable Purpose
ANTHROPIC_API_KEY Enables the paid tier (light/heavy, and the final safety net).
OPENROUTER_API_KEY Enables the free-cloud tier (OpenRouter :free models).
LLM_CASCADE_CONFIG Path to a TOML config file, instead of ~/.llm_cascade/routes.toml.
LLM_CASCADE_LOG_PATH Path to the savings ledger, instead of ~/.llm_cascade/log.jsonl.
LLM_CASCADE_LIGHT_FREE Set to 1 to try free-cloud first on the light tier too.
LLM_CASCADE_NO_FREE_CLOUD Set to any value to disable the free-cloud tier globally.
LLM_CASCADE_HTTP_REFERER / LLM_CASCADE_APP_TITLE Sent as HTTP-Referer / X-Title on OpenRouter requests (optional; omitted if unset).
LLM_CASCADE_OPENROUTER_MODEL / LLM_CASCADE_OPENROUTER_MODELS Override the free model (singular) or ordered list (comma-separated) tried on OpenRouter.
LLM_CASCADE_OLLAMA_BASE_URL Ollama server URL (default http://localhost:11434).

See .env.example for a copy-pasteable starting point.

Savings ledger

Every call -- free or paid -- appends one line to a JSONL ledger (~/.llm_cascade/log.jsonl by default). stats() / llm-cascade --stats summarize it:

$ llm-cascade --stats
{
  "paid_calls": 3,
  "free_calls": 41,
  "total_cost_usd": 0.0842,
  "by_model": {
    "local:ollama": 30,
    "openrouter:qwen/qwen3-next-80b-a3b-instruct:free": 11,
    "claude-sonnet-5": 3
  }
}

Comparison

Unlike heavier multi-provider routers, llm-cascade ships with zero required dependencies and a single opinionated cascade -- it's a routing policy you can read in one file, not a platform.

Contributing

Issues and pull requests are welcome at github.com/luandv92/llm-cascade. Before opening a PR:

pip install -e ".[dev,anthropic]"
ruff check .
pytest tests/ -v

Tests never touch real networks -- Ollama, OpenRouter, and the Anthropic SDK are all mocked (see tests/conftest.py).

License

MIT -- see LICENSE.

About

Free-first LLM routing for Python: local Ollama -> OpenRouter free tier -> paid Anthropic API, with automatic fallback and a savings ledger. Zero required dependencies.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages