English | 中文
java-litellm deliberately aligns with LiteLLM at the config file and management API layers, so most existing setups port over smoothly. This document lists what aligns and what doesn't.
The core structure of model_list / litellm_params / general_settings / litellm_settings is identical, and the os.environ/VAR reference syntax is preserved:
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
api_base: https://... # optional; OpenAI-compatible endpoints use this override
api_version: 2024-10-21 # Azure
aws_region_name: us-east-1 # Bedrock
weight: 2 # load-balancing weight
tpm: 100000 # deployment-level TPM/RPM (used by usage-based routing)
rpm: 600
litellm_settings:
cache: true
cache_params:
ttl: 300
general_settings:
master_key: os.environ/LITELLM_MASTER_KEYOnly the subset of fields above is supported today. Less-common
litellm_paramsfields (some guardrail and router tuning knobs in LiteLLM) are not implemented yet and will be ignored silently instead of erroring on migration — cross-check against this list to confirm your key fields are covered.
| Endpoint | Status |
|---|---|
POST /v1/chat/completions (incl. SSE) |
✅ |
POST /v1/embeddings |
✅ |
GET /v1/models |
✅ (filtered by key permissions) |
POST /key/generate|info|update|delete |
✅ |
POST /team/new|info|update|delete |
✅ |
POST /user/new|info |
✅ |
POST /model/new|delete, GET /model/info |
✅ (hot-reload) |
GET /spend/logs|keys |
✅ |
GET /key/list, /team/list, /user/list |
✅ (portal support; paginated, page/size params, response {"data": [...], "total": n}) |
GET /spend/daily?days&group_by=model|key |
✅ (portal support; daily aggregates, days clamped to 90) |
GET /router/state |
✅ (portal support; per-deployment cooldown / minute counters / avg latency) |
GET /health, /actuator/health, /actuator/prometheus |
✅ |
POST /v1/completions, /v1/images/*, /v1/audio/*, rerank, pass-through |
⏳ v1.x |
Virtual-key behavior matches: sk- prefix, SHA-256-only storage, model whitelist, max_budget, duration (30s/30m/30h/30d), TPM/RPM limits, expiry and blocking.
Model route strings follow the same convention as LiteLLM: openai/..., anthropic/..., azure/..., gemini/..., bedrock/..., mistral/.... A route without a prefix defaults to OpenAI. OpenAI-compatible providers (DeepSeek, Groq, Together, Ollama, vLLM, ...) work through provider-openai + an api_base override — same pattern as LiteLLM.
- Provider coverage: v1.0 focuses on the 6 Tier-1 providers (see CAPABILITIES.md), not LiteLLM's 100+. The rest are SPI extension points for the community.
- Enterprise features are out of scope: SSO/SAML, audit logs, JWT team auth, enterprise Guardrails are not implemented (these aren't pure open-source in LiteLLM either).
- Distributed backends: in-process by default; setting
litellm_settings.redis_url(orgeneral_settings.redis_url) inconfig.yamlswitches cache, rate limiter and router state (cooldown + latency windows) to Redis, giving accurate cross-replica counters (consistency bounded by Redis itself, not per-process drift). - Token counting: OpenAI-family models use jtokkit for exact counts; other providers prefer the response
usagefield and fall back to approximate estimation when usage is unavailable — same behavior as LiteLLM'stoken_counter. - Price table: reuses LiteLLM's
model_prices_and_context_window.json; a snapshot is bundled in the jar. Unknown models returnnullcost (not 0) so you can distinguish "free" from "unknown". - Callback ecosystem: v1.0 ships logging / Prometheus / OTel built in; Langfuse, custom Webhook etc. are in v1.x.
- Error body: uniformly mapped to the OpenAI shape
{"error":{"message","type","code"}}.typeis the internal exception class name (e.g.RateLimitException).
- All
model_listfields used inconfig.yamlare covered by §1 - Required providers are in the 6 Tier-1 list, or are OpenAI-compatible endpoints
- No dependency on enterprise features (SSO / audit logs / enterprise Guardrails)
- Multi-replica deployments: either single-replica counter precision is acceptable, or you set
redis_urlfor the Redis backend - Clients only need to point
base_urlat the new gateway; auth still usesAuthorization: Bearer sk-...
Missing something on the alignment list? Open an issue.