Feature request: middleware for dynamic model routing

[gabrielmoreira]

I'd like to propose a middleware-style hook for request routing.

Today, LLM gateway routing is usually static: fixed aliases, fixed fallbacks, or simple balancing. That works for basic cases, but it becomes limiting when users combine multiple providers with very different rolling limits, monthly quotas, reserve strategies, and budget versus premium lanes.

A real-world example is combining subscriptions like OpenAI and Anthropic that uses short quota limits, GitHub Copilot with a monthly premium requests quota, and cheaper providers (like opencode go subscriptions, e.g. glm, minimax, kimi, mimo, ...) for lower-value work. In that setup, the best route depends on rules like:

• use premium providers for important coding or planning
• shift traffic during favorable quota windows (discount windows)
• preserve some providers as reserve
• fail over on quota or rate-limit errors
• capture usage, retries, latency, and local quota state

This is hard to express with static routing alone.

What would help is a middleware or scripting hook that receives the request and a next(...) function, so user logic can choose a route, execute it, retry or fail over if needed, and inspect the response.

Conceptually, something like:

const middleware = async (request, next) => {
  const candidateModels = [...selectBestCandidateModels(request)]
  let lastError
  while (candidateModels.length > 0) {
    const selectedModel = candidateModels.shift()
    const customRequest= createRequestForModel(request, selectedModel)
    try {
      const response = await next(customRequest)
      updateModelProviderStatistics(selectedModel, request, response)
      return response
    } catch (error) {
      lastError = error
      updateModelProviderStatistics(selectedModel, request, error)

      if (!shouldContinueToNextModel(error, selectedModel)) {
        throw error
      }
    }
  }
  throw lastError
}

This would also make virtual routing models much more useful. For example, clients could send model names like coding-budget, coding-premium, planning-budget, planning-premium, simple-task, cheap, free, vision, fastest, and the proxy could resolve them dynamically based on user-defined rules instead of fixed aliases. Or via custom header when possible like X-MODEL-PRESET: coding-budget.

In short, I am not asking for a full built-in policy engine, only for a supported middleware-style extension point for dynamic routing and execution control.