Problem
The chat LLM path (dedup judge, enrichment, contradiction detection, entity extraction) cannot use any endpoint except each provider's hard coded hosted URL, and it sends a response_format that only hosted OpenAI accepts. Both gaps have the same effect: the call fails, call_with_fallback walks to the fake provider, and every LLM feature silently runs on heuristics. Nothing in the API or the health output shows that this happened; common.llm.retry logs a warning and the write succeeds.
Two things cause it.
1. The base URLs are constants. OPENAI_CHAT_BASE_URL, ANTHROPIC_CHAT_BASE_URL, and OPENROUTER_CHAT_BASE_URL are module constants in common/llm/constants.py, and _credentials.py passes them to OpenAILLMProvider explicitly, which also defeats the SDK's own OPENAI_BASE_URL variable. The comment above them says the URL is "swapped via tenant-config override", but no tenant-config field carries a URL; only the provider name is swapped, and each name has its own hard-wired URL. The embedding path already has OPENAI_EMBEDDING_BASE_URL; the chat path has no equivalent. Further, a service needing a direct line out in my current production environment (and I assume others in my industry) is hard NO due to security and compliance concerns.
2. The response_format shapes are hosted-OpenAI only. OpenAILLMProvider.complete_json sends {"type": "json_object"} when the caller passes no schema, and json_schema with strict: false when it does. Observed on 2.30.0 (I'm running LM Studio, but I suspect Ollama and similar would have the same issue):
- LM Studio (0.4.21) rejects
json_object: 'response_format.type' must be 'json_schema' or 'text'.
- Anthropic's OpenAI-compatible endpoint (which
ENTITY_EXTRACTION_PROVIDER=anthropic uses) rejects both. No-schema callers get response_format.type: Input should be 'json_schema'; entity extraction gets response_format.json_schema.strict: Input should be True. With a funded key, all 44 chat calls in one write sweep returned HTTP 400 and fell back to the fake provider. So the shipped Anthropic setting cannot complete a single JSON call.
Direct probes of the Anthropic endpoint show what it does accept: json_schema with strict: true and a closed schema (additionalProperties: false on every object). A permissive strict schema is rejected, type: text is rejected, and omitting response_format works but the model wraps the JSON in a Markdown code fence, which the bare json.loads in complete_json does not survive.
Proposed solution
Still going through things, but from what i've gleaned so far, should be straight forward and consistent with what the code already does.
- Read the three chat base URLs from the environment, with the current values as defaults, the same way
ANTHROPIC_DEFAULT_MODEL is read two lines below. Keep the literal hosted URL in a separate constant so the provider can tell whether it is talking to api.openai.com after an override. Fix the stale comment. Document the variables in .env.example next to OPENAI_EMBEDDING_BASE_URL.
- In
complete_json, send what the endpoint accepts. Hosted OpenAI keeps today's shapes. Any other base URL gets no response_format without a schema, and strict: true with a closed schema when one is given (a small helper closes each object, lists every property under required, and drops default and title). Strip a Markdown code fence before json.loads.
An operator then sets OPENAI_CHAT_BASE_URL=http://host.docker.internal:1234/v1 and the judge runs on a local model, or sets ENTITY_EXTRACTION_PROVIDER=anthropic and it runs on Anthropic. Hosted OpenAI behaviour does not change.
Alternatives considered
- The tenant provider name. It only swaps between the three hosted URLs.
- The SDK's
OPENAI_BASE_URL. Defeated, because the constructor passes base_url explicitly.
- A
sitecustomize patch of the constants. Works, but it is not a supported configuration and does not fix the response_format half.
- A native Anthropic provider on the Messages API. The right long-term shape for that provider, and a larger change. This proposal makes the existing compatible path work first.
Affected surface
Server configuration / env var
Are you willing to contribute?
Yes, I can open a PR
Additional context
Reproduced against Caura 2.30.0 with LM Studio 0.3.x serving qwen2.5-coder-7b-instruct, and against https://api.anthropic.com/v1 with claude-haiku-4-5-20251001. A branch with the change, tests, and a CHANGELOG entry is ready; the PR follows this issue. With the patched image, the dedup judge on Haiku 4.5 rejects a paraphrase in the 0.85 to 0.97 band and admits a refinement at confidence 0.90, which the shipped build cannot do from any provider setting except a funded hosted OpenAI key.
Pre-flight
Problem
The chat LLM path (dedup judge, enrichment, contradiction detection, entity extraction) cannot use any endpoint except each provider's hard coded hosted URL, and it sends a
response_formatthat only hosted OpenAI accepts. Both gaps have the same effect: the call fails,call_with_fallbackwalks to the fake provider, and every LLM feature silently runs on heuristics. Nothing in the API or the health output shows that this happened;common.llm.retrylogs a warning and the write succeeds.Two things cause it.
1. The base URLs are constants.
OPENAI_CHAT_BASE_URL,ANTHROPIC_CHAT_BASE_URL, andOPENROUTER_CHAT_BASE_URLare module constants incommon/llm/constants.py, and_credentials.pypasses them toOpenAILLMProviderexplicitly, which also defeats the SDK's ownOPENAI_BASE_URLvariable. The comment above them says the URL is "swapped via tenant-config override", but no tenant-config field carries a URL; only the provider name is swapped, and each name has its own hard-wired URL. The embedding path already hasOPENAI_EMBEDDING_BASE_URL; the chat path has no equivalent. Further, a service needing a direct line out in my current production environment (and I assume others in my industry) is hard NO due to security and compliance concerns.2. The
response_formatshapes are hosted-OpenAI only.OpenAILLMProvider.complete_jsonsends{"type": "json_object"}when the caller passes no schema, andjson_schemawithstrict: falsewhen it does. Observed on 2.30.0 (I'm running LM Studio, but I suspect Ollama and similar would have the same issue):json_object:'response_format.type' must be 'json_schema' or 'text'.ENTITY_EXTRACTION_PROVIDER=anthropicuses) rejects both. No-schema callers getresponse_format.type: Input should be 'json_schema'; entity extraction getsresponse_format.json_schema.strict: Input should be True. With a funded key, all 44 chat calls in one write sweep returned HTTP 400 and fell back to the fake provider. So the shipped Anthropic setting cannot complete a single JSON call.Direct probes of the Anthropic endpoint show what it does accept:
json_schemawithstrict: trueand a closed schema (additionalProperties: falseon every object). A permissive strict schema is rejected,type: textis rejected, and omittingresponse_formatworks but the model wraps the JSON in a Markdown code fence, which the barejson.loadsincomplete_jsondoes not survive.Proposed solution
Still going through things, but from what i've gleaned so far, should be straight forward and consistent with what the code already does.
ANTHROPIC_DEFAULT_MODELis read two lines below. Keep the literal hosted URL in a separate constant so the provider can tell whether it is talking toapi.openai.comafter an override. Fix the stale comment. Document the variables in.env.examplenext toOPENAI_EMBEDDING_BASE_URL.complete_json, send what the endpoint accepts. Hosted OpenAI keeps today's shapes. Any other base URL gets noresponse_formatwithout a schema, andstrict: truewith a closed schema when one is given (a small helper closes each object, lists every property underrequired, and dropsdefaultandtitle). Strip a Markdown code fence beforejson.loads.An operator then sets
OPENAI_CHAT_BASE_URL=http://host.docker.internal:1234/v1and the judge runs on a local model, or setsENTITY_EXTRACTION_PROVIDER=anthropicand it runs on Anthropic. Hosted OpenAI behaviour does not change.Alternatives considered
OPENAI_BASE_URL. Defeated, because the constructor passesbase_urlexplicitly.sitecustomizepatch of the constants. Works, but it is not a supported configuration and does not fix theresponse_formathalf.Affected surface
Server configuration / env var
Are you willing to contribute?
Yes, I can open a PR
Additional context
Reproduced against Caura 2.30.0 with LM Studio 0.3.x serving
qwen2.5-coder-7b-instruct, and againsthttps://api.anthropic.com/v1withclaude-haiku-4-5-20251001. A branch with the change, tests, and a CHANGELOG entry is ready; the PR follows this issue. With the patched image, the dedup judge on Haiku 4.5 rejects a paraphrase in the 0.85 to 0.97 band and admits a refinement at confidence 0.90, which the shipped build cannot do from any provider setting except a funded hosted OpenAI key.Pre-flight