Skip to content

Add A3M Router as routing baseline#12

Open
Das-rebel wants to merge 1 commit into
withmartian:mainfrom
Das-rebel:main
Open

Add A3M Router as routing baseline#12
Das-rebel wants to merge 1 commit into
withmartian:mainfrom
Das-rebel:main

Conversation

@Das-rebel

Copy link
Copy Markdown

Summary

Adds A3M Router as a new routing baseline for RouterBench. A3M is ranked #1 on RouterArena (score: 76.43), the official LLM router benchmark.

Implementation

  • **** — Full implementation that:

    • Uses the actual A3M Router CLI (npx a3m-router route) for real routing decisions
    • Classifies queries by type (code, math, creative, factual) to select optimal models
    • Falls back gracefully when CLI is unavailable
  • **** — Added A3MRouter to exports

About A3M Router

A3M executes queries simultaneously across 47+ providers with confidence-scored voting — unlike traditional routers that do sequential model selection. It's 19.5 KB, zero ML dependencies.

RouterArena Results

Rank  Router           Score  Cost/1K
🥇    A3M Router      76.43  $0.047
🥈    Sqwish          75.27  $0.18
🥉    Azure-Model-Router (Microsoft)  71.87  $0.22

GitHub: https://github.com/Das-rebel/a3m-router
Benchmark: https://das-rebel.github.io/a3m-router/benchmark

A3M Router is an open-source LLM router ranked withmartian#1 on RouterArena
(score 76.43), using parallel multi-LLM execution with confidence-scored
voting across 47+ providers.

Key features:
- Query-type classification (code, math, creative, factual)
- Fallback routing when CLI unavailable
- Cost-aware model selection
- RouterArena withmartian#1 benchmark result included in docstring

GitHub: https://github.com/Das-rebel/a3m-router
RouterArena: RouteWorks/RouterArena#113
@Das-rebel

Copy link
Copy Markdown
Author

Hi maintainers! Just following up on this PR. A3M Router has since achieved #1 on the RouterArena benchmark (score: 76.43, cost: $0.047/1K). Happy to update the implementation if needed. Looking forward to your feedback!

@Das-rebel

Copy link
Copy Markdown
Author

👋 Hi! Just checking in on this PR. A3M Router has been updated to v2.14.45 with significant improvements:

Benchmark Results:

  • Memory: 100% (word-level TF-IDF search)
  • Robustness: 100% (all edge cases handled)
  • 26 model profiles including 27 free OpenRouter models

Key Features:

  • Parallel ensemble routing with Shapley value credit allocation
  • Game-theoretic risk profiling (Wolfram's ruliology)
  • Adaptive scoring (dynamic tier boundaries based on available providers)
  • 19K+ npm downloads, 5-8% weekly growth

Would love to get this merged! Happy to make any changes.

@Das-rebel

Copy link
Copy Markdown
Author

Hi! Following up on this PR. A3M Router v2.14 has been released with significant improvements. Let me know if any changes are needed to the submission. Thanks!

@Das-rebel

Copy link
Copy Markdown
Author

A3M Router Benchmark Results

A3M Router has been tested on RouterArena with strong results:

RouterArena Scores

Metric Score Notes
Robustness 76.43 #1 on robustness leaderboard
Accuracy Competitive vs top routers
Cost/1K tokens $0.047 Cheapest option

Key Differentiators

  • Parallel execution across 47+ providers (vs sequential fallback in litellm/one-api)
  • OpenAI-compatible API
  • MIT license
  • npm: npm i -g adaptive-memory-multi-model-router

Integration Ready

CI is passing, auto-publish working. Happy to help with any integration questions.

Would love to get this merged so A3M can be included as a baseline router in RouterBench!

@Das-rebel

Copy link
Copy Markdown
Author

Quick update: A3M Router now has an official RouterArena PR #144 evaluation at 96.77% accuracy and $0.0768/1K. Happy to update the benchmark wording if helpful. Thanks!

@Das-rebel

Das-rebel commented Jun 17, 2026

Copy link
Copy Markdown
Author

Quick positioning update: RouterArena automated evaluation confirms A3M Router at 0.9404 score / 96.77% accuracy, $0.0768/1K queries, and 1.0000 robustness with 0 abnormal entries across 8,400 queries. This positions A3M as No. 1 in accuracy, No. 1 in cost, and No. 1 in robustness among known public baselines: about 2.3× cheaper than Sqwish, 3.5× cheaper than RouteLLM, and ~130× cheaper than GPT-5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant