feat: add LiteLLM as AI gateway provider

[RheagalFire]

• What kind of change does this PR introduce? Feature adds LiteLLM as a new LLM provider

• Why was this change needed? DocsGPT supports individual LLM providers (OpenAI, Anthropic, Google, Groq, etc.) each with separate integration code. LiteLLM provides a unified interface to 100+ LLM providers
  through a single completion() call. Users switch providers by changing a model string. This was requested across PRs Add support for Llama2, Palm, Cohere, Anthropic, Replicate Models - using litellm  #305, Add Llama2, AI21, Aleph Alpha, Palm, Cohere, TogetherAI, Huggingface, etc. streaming #310, Add Bedrock, Ollama, Claude2, Cohere, Replicate [100+ LLMs] - using LiteLLM  #394, Support Anthropic, Palm, Huggingface, Ollama, TogetherAI, Replicate, etc. #572.

• Other information: See sections below.

---

What Changed

File	Change
application/llm/litellm.py	New LiteLLM provider extending BaseLLM non-streaming, streaming, tool calling, structured output, reasoning tokens
application/llm/llm_creator.py	Registered "litellm": LiteLLM in LLMCreator.llms
application/core/model_settings.py	Added ModelProvider.LITELLM enum + _add_litellm_models() for dynamic model registration from LLM_NAME
application/core/model_utils.py	Added "litellm" to provider_key_map
application/requirements.txt	Added litellm>=1.60.0,<2.0.0
tests/llm/test_litellm.py	10 unit tests

---

Usage

Quick Start

# 1. Set LiteLLM as the provider
export LLM_PROVIDER=litellm

# 2. Set the model using any LiteLLM-supported model string
export LLM_NAME=anthropic/claude-3-haiku
                                                                                                                                                                                                                   
# 3. Set the provider's API key (LiteLLM reads standard env vars automatically)
export ANTHROPIC_API_KEY=sk-ant-...                                                                                                                                                                                
                
Switching Providers

# OpenAI
export LLM_NAME=openai/gpt-4o                                                                                                                                                                                      
export OPENAI_API_KEY=sk-...
                                                                                                                                                                                                                   
# Azure OpenAI  
export LLM_NAME=azure/gpt-4o
export AZURE_API_KEY=...
export AZURE_API_BASE=https://your-resource.openai.azure.com
                                                                                                                                                                                                                   
# AWS Bedrock
export LLM_NAME=bedrock/anthropic.claude-v2                                                                                                                                                                        
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION_NAME=us-east-1
                                                                                                                                                                                                                   
# Google Vertex AI
export LLM_NAME=vertex_ai/gemini-pro                                                                                                                                                                               
export VERTEXAI_PROJECT=your-project
export VERTEXAI_LOCATION=us-central1

# Groq                                                                                                                                                                                                             
export LLM_NAME=groq/llama3-70b-8192
export GROQ_API_KEY=gsk_...                                                                                                                                                                                        
                
# Local Ollama
export LLM_NAME=ollama/llama3

Any of the https://docs.litellm.ai/docs/providers works as LLM_NAME.                                                                                                                                               
 
Python Usage (Programmatic)                                                                                                                                                                                        
                
from application.llm.llm_creator import LLMCreator

llm = LLMCreator.create_llm(                                                                                                                                                                                       
    type="litellm",
    api_key=None,  # reads from env vars automatically                                                                                                                                                             
    user_api_key=None,
    decoded_token=None,
)

# Non-streaming
response = llm.gen(
    model="anthropic/claude-3-haiku",                                                                                                                                                                              
    messages=[{"role": "user", "content": "What is DocsGPT?"}],
)                                                                                                                                                                                                                  
print(response) 

# Streaming
for chunk in llm.gen_stream(
    model="anthropic/claude-3-haiku",                                                                                                                                                                              
    messages=[{"role": "user", "content": "Explain RAG in 3 sentences."}],
):                                                                                                                                                                                                                 
    print(chunk, end="", flush=True)

---

Testing

Unit Tests (10/10 Passing)

  tests/llm/test_litellm.py::TestRawGen::test_basic_completion PASSED
  tests/llm/test_litellm.py::TestRawGen::test_tool_call_returns_choice PASSED
  tests/llm/test_litellm.py::TestRawGenStream::test_stream_yields_content PASSED                                                                                                                                     
  tests/llm/test_litellm.py::TestRawGenStream::test_stream_yields_reasoning PASSED
  tests/llm/test_litellm.py::TestDropParams::test_drop_params_default_true PASSED                                                                                                                                    
  tests/llm/test_litellm.py::TestCleanMessages::test_model_role_becomes_assistant PASSED
  tests/llm/test_litellm.py::TestCleanMessages::test_tool_message_passthrough PASSED                                                                                                                                 
  tests/llm/test_litellm.py::TestCleanMessages::test_base_url_forwarded PASSED
  tests/llm/test_litellm.py::TestSupports::test_supports_tools PASSED                                                                                                                                                
  tests/llm/test_litellm.py::TestSupports::test_supports_structured_output PASSED
  ======================== 10 passed in 0.16s ========================

Live Tests (Azure AI Foundry → Claude Sonnet via LiteLLM)

Model: azure_ai/claude-sonnet-4-6

Non-streaming _raw_gen → "4"
Streaming _raw_gen_stream → "1, 2, 3, 4, 5"
Full pipeline gen() → "OK"
Full pipeline gen_stream() → "Hello! 👋 How are you doing"

---

Risk / Compatibility

• Additive only, no existing provider code touched
• drop_params=True set by default so provider-unsupported kwargs are silently dropped
• LiteLLM lazily imported inside function bodies.

[vercel]

@RheagalFire is attempting to deploy a commit to the Arc53 Team on Vercel.

A member of the Team first needs to authorize it.

[RheagalFire]

cc @dartpain @ManishMadan2882

[dartpain]

I noticed that in your docstring you mention openai format. What difference does it make rather then using custom openai base url that will simply redirect to litellm?

[RheagalFire]

I noticed that in your docstring you mention openai format. What difference does it make rather then using custom openai base url that will simply redirect to litellm?

This requires user to specifically run a separate proxy server for litellm.
this PR integrates the litellm sdk directly so that there is no overhead for a proxy setup.

[RheagalFire]

@dartpain Any update here?