English | 中文
A Java rewrite of the open-source parts of LiteLLM (the MIT-licensed SDK + Router + Proxy gateway). Version: v0.1 (design draft) Date: 2026-06-12
LiteLLM is a Python-implemented "LLM unification layer". Its open-source portion has three parts:
| Component | Responsibility |
|---|---|
| SDK (litellm core library) | Calls 100+ model providers in one unified OpenAI-shaped API: completion() / embedding() / image_generation() / transcription() / rerank(), etc.; unified exception mapping, streaming, cost calculation, token counting |
| Router | Load balancing, retries, fallback, cooldown across deployments; concurrency- and budget-aware routing |
| Proxy (LLM gateway) | Exposes an OpenAI-compatible REST API; virtual key management, budgets + rate limits (TPM/RPM), team/user management, caching, callback logging (Langfuse/Prometheus/OTel, etc.), Guardrails, pass-through endpoints |
Note: parts of LiteLLM's Proxy feature set (SSO, audit logs, some enterprise Guardrails) are enterprise-only and are out of scope for this rewrite.
- SDK: a Java library with no heavyweight dependencies, calling mainstream LLM providers in one OpenAI-shaped API; sync, async and streaming.
- Router: a routing layer decoupled from the SDK, supporting multiple load-balancing strategies, retries, fallback and cooldown.
- Proxy: a Spring Boot OpenAI-compatible gateway with virtual keys, budgets, rate limiting, caching and observability.
- Engineering quality: comprehensive contract tests (aligned with the OpenAI API spec), mocked integration tests, clean SPI extension points.
- Enterprise features: SSO/SAML, audit logs, JWT team auth, enterprise Guardrails.
- Admin UI — v1.0 ships only the management REST API; UI is in v1.x.
- Aligning with all of LiteLLM's 100+ providers — v1.0 focuses on Tier-1 providers (see §4.2); the long tail is exposed via SPI for community extension.
- Fine-tuning, Batch API, Assistants API and other low-traffic endpoints.
| Dimension | Choice | Rationale |
|---|---|---|
| Language / JDK | Java 21 LTS | Virtual threads (Loom) suit the high-concurrency IO-bound gateway workload natively; record/sealed model API types cleanly |
| Build | Maven multi-module | Mature ecosystem, good IDE support, standard publishing path to Maven Central |
| HTTP client (SDK) | JDK java.net.http.HttpClient |
SDK core stays free of third-party HTTP deps, reducing version conflicts on the user side; native HTTP/2 and async support |
| JSON | Jackson | De-facto standard; OpenAI format has many polymorphic fields (e.g. content can be a string or an array), Jackson's polymorphic deserialization is the best fit |
| Logging | SLF4J (SDK only depends on the API) | No implementation lock-in |
| Proxy framework | Spring Boot 3.3+ (with virtual threads enabled) | Most mature ecosystem / hiring pool / ops know-how; SSE, security and configuration come out of the box |
| Persistence | PostgreSQL + Flyway + Spring Data JDBC | Matches LiteLLM's Postgres choice; Spring Data JDBC is more controllable than JPA |
| Cache / rate limiting | Redis (Lettuce) + in-process Caffeine | Two-tier cache; Redis also carries the distributed rate-limit counters |
| Observability | Micrometer + OpenTelemetry | One set of instrumentation, two outputs (Prometheus metrics + OTel traces) |
| Testing | JUnit 5 + WireMock + Testcontainers | WireMock mocks provider APIs; Testcontainers runs Postgres/Redis integration tests |
Streaming API style decision: the SDK's streaming surface does not bring in Reactor/RxJava — those are contagious dependencies. Three forms are exposed:
- Synchronous:
Stream<ChatChunk>(virtual thread + SSE parser underneath) - Callback:
StreamHandler(onChunk/onComplete/onError) - Async:
CompletableFuture<ChatResponse>+Flow.Publisher<ChatChunk>(the JDK's built-in Reactive Streams API, bridgeable to Reactor at zero cost)
java-litellm/
├── litellm-core # canonical types, exception hierarchy, token counting, cost calc (zero HTTP deps)
├── litellm-client # entry-point facade LiteLlm / LiteLlmClient, SSE parsing, retry plumbing
├── litellm-providers/ # provider adapters (one submodule each, opt in as needed)
│ ├── provider-openai # also covers OpenAI-compatible providers (DeepSeek, Groq, Together, Ollama, ...)
│ ├── provider-anthropic
│ ├── provider-azure-openai
│ ├── provider-gemini # Google AI Studio + Vertex AI
│ ├── provider-bedrock
│ └── provider-mistral
├── litellm-router # load balancing, fallback, cooldown
├── litellm-cache # cache abstraction + Caffeine/Redis impls
├── litellm-callbacks # callback / logging abstraction + Langfuse/OTel/Prometheus adapters
├── litellm-proxy # Spring Boot gateway application
└── litellm-bom # dependency version alignment
Dependency direction (strictly one-way):
proxy ──► router ──► client ──► providers/* ──► core
│ │ ▲
└──► cache/callbacks ──────────────────────────┘
Everything is modeled against the OpenAI Chat Completions format as the canonical form, using record + sealed:
public record ChatRequest(
String model, // "anthropic/claude-sonnet-4-6" — provider/model shape
List<Message> messages,
Double temperature, Integer maxTokens, Boolean stream,
List<Tool> tools, ToolChoice toolChoice,
ResponseFormat responseFormat,
Map<String, Object> extraParams // pass-through for provider-specific params (mirrors litellm's drop_params semantics)
) {}
public sealed interface Content permits TextContent, ImageContent, AudioContent {}
public record ChatResponse(
String id, String model, List<Choice> choices,
Usage usage, // prompt/completion/total tokens + breakdown (cached, reasoning)
BigDecimal costUsd // SDK computes cost up front, aligned with litellm's completion_cost
) {}Model route strings follow LiteLLM's convention: "openai/gpt-4o", "anthropic/claude-sonnet-4-6", "bedrock/us.anthropic.claude...". No prefix means OpenAI.
public interface LlmProvider {
String name(); // "anthropic"
Set<Capability> capabilities(); // CHAT, STREAMING, EMBEDDING, VISION, TOOLS...
ChatResponse chat(ChatRequest req, ProviderConfig cfg);
void chatStream(ChatRequest req, ProviderConfig cfg, StreamHandler handler);
EmbeddingResponse embedding(EmbeddingRequest req, ProviderConfig cfg);
// image / audio etc. are default methods that throw UnsupportedCapabilityException
}- Auto-discovered via
ServiceLoader; whichever provider modules are on the classpath are the providers that work. - Each provider's internal shape is a three-step pure function:
transformRequest(OpenAI format → provider format) → HTTP call →transformResponse(provider format → OpenAI format). Transformers are separated from HTTP for unit-testability. - v1.0 Tier-1 providers: OpenAI (incl. compatible endpoints), Anthropic, Azure OpenAI, Gemini (AI Studio + Vertex), AWS Bedrock, Mistral. OpenAI-compatible providers (DeepSeek / Groq / Together / Ollama / vLLM, etc.) work through
provider-openai'sapiBaseoverride — one module covers the long tail.
Semantics aligned with LiteLLM's exception mapping (every provider error is mapped to an OpenAI-shaped exception):
LiteLlmException (carries provider, model, statusCode, retryable)
├── AuthenticationException (401)
├── PermissionDeniedException (403)
├── NotFoundException (404)
├── BadRequestException (400) ├── ContextWindowExceededException
├── RateLimitException (429) // retryable
├── InternalServerException (500) // retryable
├── ServiceUnavailableException(503) // retryable
└── ApiTimeoutException // retryableThe retryable flag drives client retries and the router's cooldown / fallback decisions.
- Model prices reuse LiteLLM's community-maintained
model_prices_and_context_window.json(a continually updated community asset). A snapshot is bundled in the jar; runtime refresh from a remote URL plus local overrides are supported. - Token counting:
jtokkit(the Java port of tiktoken) gives exact counts for OpenAI-family models; for other providers, the responseusagefield is preferred, with approximate counting as a fallback whenusageisn't reported — same behavior as LiteLLM.
LiteLlm client = LiteLlm.builder()
.apiKey("anthropic", System.getenv("ANTHROPIC_API_KEY"))
.retryPolicy(RetryPolicy.exponential(3))
.build();
ChatResponse resp = client.chat(ChatRequest.builder()
.model("anthropic/claude-sonnet-4-6")
.message(Message.user("Hello"))
.build());
client.chatStream(req).forEach(chunk -> System.out.print(chunk.delta()));Built-in cross-cutting concerns: timeouts, exponential-backoff retries (retryable exceptions only), callback hooks (litellm-callbacks pre/post/failure), redacted-sensitive-data logging.
Core concepts mirror LiteLLM's: one model group (e.g. "gpt-4o") hosts multiple deployments (concrete instances behind different API keys / regions / providers).
public record Deployment(
String id, String modelGroup,
ChatRequestDefaults litellmParams, // actual model, apiBase, apiKey
Integer tpm, Integer rpm, BigDecimal inputCostOverride
) {}Routing strategies (pluggable Strategy interface):
simple-shuffle(default): weighted randomleast-busy: picks the deployment with the fewest in-flight requestslatency-based: picks the lowest mean latency in the sliding windowusage-based: picks by TPM/RPM headroom (counters in Redis or in-process)
Reliability machinery:
- Retries: retry the same deployment N times on retryable exceptions.
- Cooldown: when a single deployment exceeds a failure threshold inside a window → eject from the candidate pool for X seconds (default 5s; 429 honors
Retry-After). - Fallback: model-group chain, e.g.
gpt-4o → [azure-gpt-4o, claude-sonnet]. Configurable per exception type —ContextWindowExceededcan have its own fallback to a long-context model. - Timeout budget: the whole fallback chain shares one total deadline.
Router state (in-flight counts, latency window, cooldown list, TPM/RPM counters) is abstracted as RouterStateStore with both in-process and Redis impls — single-machine SDK uses in-process; multi-replica Proxy uses Redis.
| Endpoint | Notes |
|---|---|
POST /v1/chat/completions |
Core; supports SSE streaming |
POST /v1/completions / /v1/embeddings |
Text completion / embeddings |
POST /v1/images/generations, /v1/audio/* |
v1.x |
GET /v1/models |
Models filtered by key permissions |
GET /health, /health/liveness, /health/readiness |
Health probes |
/anthropic/* etc. pass-through |
v1.x |
/key/generate|update|delete|info, /user/*, /team/*, /model/*, /spend/* — semantics aligned with LiteLLM Proxy's management API for easy migration.
AuthFilter (virtual-key check) → BudgetCheck → RateLimitCheck (TPM/RPM) → CacheLookup
→ Router.route() → Provider call → Response
→ Async post-processing (cost accounting / spend logs / callbacks / cache write-back)
- Virtual keys: random
sk--prefixed strings; the DB only stores their SHA-256 hash. A key carries a budget (max_budget/durationrolling reset), a model whitelist, TPM/RPM limits and an expiry time. - Rate limiting: Redis sliding window (atomic Lua scripts), layered Key → User → Team → global, checked layer by layer.
- Cost accounting runs on an async queue (in-process queue + batched flush to
spend_logs), off the request hot path. - Caching: exact cache key =
hash(model + messages + relevant params); semantic caching is v1.x.
Compatible with LiteLLM's config.yaml mental model (model_list + litellm_settings + general_settings), mapped via Spring's @ConfigurationProperties. Supports os.environ/KEY references for environment variables — lowers migration friction.
virtual_keys, users, teams, budgets, spend_logs (monthly-partitioned), model_deployments (dynamic model configuration with API hot-reload).
public interface LlmCallback {
void onRequest(CallContext ctx);
void onSuccess(CallContext ctx, ChatResponse resp);
void onFailure(CallContext ctx, LiteLlmException e);
void onStreamComplete(CallContext ctx, List<ChatChunk> chunks); // aggregated callback
}- v1.0 built-in implementations: structured-JSON logging; Micrometer/Prometheus metrics (QPS, latency quantiles, token usage, cost, error rate — tagged by model/provider/key); OpenTelemetry traces.
- v1.x: Langfuse, custom Webhook.
- All callbacks run asynchronously on virtual threads; callback exceptions do not affect the request path.
- Transformer unit tests: each provider's request/response transformer is a pure function — tested with golden cases built from real fixture JSON taken from LiteLLM's repo.
- WireMock integration tests: mocked provider APIs (incl. SSE streams, 429/500 error injection) cover retry / fallback / cooldown paths.
- Contract tests: pointing the official OpenAI Java SDK at our Proxy as a client guarantees "OpenAI compatible" is true.
- Testcontainers: Postgres + Redis integration tests for keys / budgets / rate limiting.
- Real-provider smoke (optional CI job): runs a minimal scenario when real keys are supplied as environment variables.
| Risk | Mitigation |
|---|---|
| Provider APIs keep evolving — high adapter maintenance cost | Pure-function transformers + fixture-driven tests, so upgrades touch only the transformer layer; the price table is externalized and hot-reloadable |
| Modeling OpenAI's polymorphic fields cleanly in a strongly-typed language is hard | Sealed interface + Jackson polymorphism; extraParams is the safety net for fields not modeled yet |
| Streaming + virtual threads under high concurrency: backpressure | The SSE writer uses a bounded queue; load testing is an M5 acceptance criterion |
| Behavior-detail mismatch with LiteLLM creates migration friction | Management API + config-file semantics aligned; a "differences from LiteLLM" doc is maintained continuously |
| Scope creep (100+ providers temptation) | SPI is open to the community; the core team maintains only the Tier-1 set |