feat(common): allow a self-hosted base URL for the chat LLM path

[zznate]

Summary

Let the chat LLM path (dedup judge, enrichment, contradiction detection, entity extraction) use any OpenAI-compatible endpoint. The three chat base URLs now come from the environment, and complete_json sends a response_format the endpoint accepts. Hosted OpenAI behaviour does not change.

Related Issue

Closes #992

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that changes existing behavior)
• Documentation update
• Refactor / internal cleanup
• Other:

How Has This Been Tested?

New tests in tests/test_llm_openai_compatible_json_mode.py build the real provider and swap only its transport, the way test_llm_provider_sdk_retries.py does. They pin:

• hosted OpenAI still gets json_object without a schema and a non-strict json_schema with one;
• any other base URL gets no response_format without a schema, and strict: true with a closed schema when one is given, including nested $defs;
• the schema helper does not modify its input and drops default and title;
• a reply wrapped in a Markdown code fence parses;
• each *_CHAT_BASE_URL variable wins over the default, checked in a fresh interpreter because the constants are read at import time.

core-api/.venv/bin/ruff check common/llm tests/test_llm_openai_compatible_json_mode.py   # All checks passed
core-api/.venv/bin/ruff format --check common/llm tests/test_llm_openai_compatible_json_mode.py
core-api/.venv/bin/mypy common/llm/providers/openai.py common/llm/constants.py             # no errors in these files
core-api/.venv/bin/pytest tests/                                                           # 5315 passed, rebased on 71e9981

Verified live against 2.30.0 plus this change, built with core-api/Dockerfile:

• ENTITY_EXTRACTION_PROVIDER=anthropic with claude-haiku-4-5-20251001: all chat calls succeed. The dedup judge rejects a paraphrase in the judge band, admits a refinement and a negation at confidence 0.90, and the entity extraction calls complete through the strict schema with no fallback warning in the log. On the shipped build every one of these calls returned HTTP 400 and fell back to the fake provider.
• OpenAILLMProvider.complete_json with base_url=http://localhost:1234/v1 against LM Studio serving qwen2.5-coder-7b-instruct: the no-schema path returns a parsed verdict, where the shipped build was rejected with 'response_format.type' must be 'json_schema' or 'text'.

Checklist

• I have read CONTRIBUTING.md
• I have added tests that cover my changes (or explained why none are needed)
• ruff check and ruff format --check pass
• mypy passes
• pytest passes locally
• I have updated relevant documentation (README, docs, etc.)
• I have updated CHANGELOG.md under the Unreleased section (if user-facing)

Additional Notes

• OPENAI_HOSTED_CHAT_BASE_URL keeps the literal hosted URL so the provider can tell, after an override, whether it is talking to api.openai.com. That is the one endpoint that accepts json_object, so it keeps the old shapes and nothing changes for existing deployments.
• The strict-schema helper exists because Anthropic's compatible endpoint requires strict: true and rejects any object without additionalProperties: false. Pydantic-generated schemas leave both open for fields with defaults. The helper closes them on a copy; the models are untouched, and the client-side Pydantic parse stays the real guard, as the docstring already says.
• CHANGELOG.md had no Unreleased section; this adds one above 2.30.0.
• A native Anthropic provider on the Messages API would be the deeper fix for that provider. This change makes the existing compatible path work and is the smaller step.

[erni-a]

Thanks for this — it's obvious you ran it against a real self-hosted stack rather than writing to the docs, and the host.docker.internal note in .env.example is the kind of detail that saves the next person an hour. I've approved the workflows so CI runs.

I validated it against current main locally as well. Recording what came back clean, so none of it gets re-litigated in review:

• Applies onto today's main; full-suite delta is zero (same 16 pre-existing local failures with and without it).
• The module-level os.environ.get pattern matches what constants.py already does for GEMINI_DEFAULT_MODEL, ANTHROPIC_DEFAULT_MODEL and friends, so it's consistent with the file rather than a new idiom.
• The override changes only the constructor default — registry.py still passes base_url= explicitly, so per-tenant configuration keeps precedence. That's the right layering.
• test_complete_json_preserves_json_object_when_no_schema (the deliberate back-compat pin in tests/test_a5b_prompt_and_schema.py) still passes, because hosted OpenAI keeps its old shape. Good — that's the test I expected this to trip.
• _strict_schema not mutating its input is pinned by its own test. Appreciated.

Four things before this can land, roughly in order of how much they matter:

1. The non-hosted response_format change needs evidence. The default Anthropic and OpenRouter base URLs aren't api.openai.com, so after this they stop receiving response_format={"type": "json_object"} and rely on the prompt plus fence-stripping. Your reasoning is that those endpoints reject json_object — if so, those paths were already broken and this fixes them, which is a stronger claim than the PR currently makes. Could you paste the actual error response from Anthropic's compatible endpoint (and LM Studio, if that's where you hit it)? It's the load-bearing premise, and I'd like it in the PR record rather than in reviewers' heads.

2. _strict_schema marks every property required. That's what strict mode demands, but it's a behavioural change for the model, not just the wire: fields that were optional must now be emitted, so the model may invent a value where it previously omitted one. Our enrichment schemas have optional fields (PII types among them) where "absent" and "empty" aren't the same thing. A test that runs a real enrichment schema through _strict_schema and shows what happens to its optional fields would settle it.

3. A plaintext override silently ships the API key. OPENAI_CHAT_BASE_URL=http://… to anything non-loopback sends the provider key in the clear, with nothing in the logs. This is operator configuration rather than caller input, so it isn't a privilege issue — but a one-line warning at startup when the override is non-https and non-loopback would be cheap and kind.

4. _is_hosted_openai is a string comparison, so http://api.openai.com/v1, or a regional/proxy alias that is genuinely hosted OpenAI, silently takes the non-hosted path and a different request shape. Worth either normalising the scheme or saying in the docstring that exact-match is intentional.

Two mechanical items, same as the other PR: please rebase (this is currently conflicting), and drop the CHANGELOG.md hunk — that file is generated by release-please from Conventional Commit subjects (changelog-path in release-please-config.json), so a hand-written entry fights the generated one.

None of the four are objections to the idea. Pointing an OpenAI-compatible provider at LM Studio or vLLM is exactly what a self-hoster should be able to do, and the incompatibility handling is the part most people would have skipped.

[zznate]

Rebased onto main at d82eebd and dropped the CHANGELOG.md hunk. The review points are in a second commit, a7f321b, so the delta is readable on its own.

1. Evidence. Verbatim error bodies, 2026-08-25/26, Caura 2.30.0 as shipped, ENTITY_EXTRACTION_PROVIDER=anthropic with a funded key, claude-haiku-4-5-20251001:

• every no-schema caller (dedup judge, enrichment, contradiction):
  Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': "response_format.type: Input should be 'json_schema'", 'type': 'invalid_request_error', 'param': None}}
• entity extraction, which passes a schema with strict: False:
  Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': 'response_format.json_schema.strict: Input should be True', 'type': 'invalid_request_error', 'param': None}}

All 44 chat calls in one write sweep returned one of those two, then call_with_fallback walked to OpenAI (401, no key) and to the fake provider, with only common.llm.retry warnings in the log. Direct probes of https://api.anthropic.com/v1/chat/completions with the same key: json_schema with strict: true and every object closed works; strict: true with a permissive {"type": "object"} is rejected (For 'object' type, 'additionalProperties' must be explicitly set to false); {"type": "text"} is rejected (Input should be 'json_schema'); no response_format works but the reply comes fenced in ```json.

LM Studio 0.3.x serving qwen2.5-coder-7b-instruct, same no-schema call: 'response_format.type' must be 'json_schema' or 'text'.

So the Anthropic and OpenRouter defaults were not receiving a working json_object before: on Anthropic every JSON call failed. I have not tested OpenRouter; it advertises OpenAI compatibility and takes the strict shapes, and it also takes the no-response_format form, which is what it now gets without a schema.

2. Optional fields. You are right that "required" changed model behaviour. Now: a property the source schema left optional is made nullable on the wire (anyOf: [<schema>, {"type": "null"}]), and after parsing _drop_optional_nulls removes a null for such a key, so callers see the same absent-or-present shape as before. A null for a field the source schema itself made nullable (Mention.cluster_id, EnrichmentResult.ts_valid_start) is kept as sent. Tests run ExtractedGraph.model_json_schema() and EnrichmentResult.model_json_schema() through both helpers and validate the cleaned reply with the models; hosted OpenAI is untouched and returns replies as sent.

3. Plaintext override. One warning at construction when the base URL is http to a non-loopback host, naming the provider and the URL: the key travels in the clear. Loopback stays silent.

4. _is_hosted_openai. Decides by host now, case-insensitive, any scheme, trailing slash or not. The docstring says a proxy or alias on another host is non-hosted on purpose: the shapes are chosen for what the server accepts, and hosted OpenAI takes the strict shapes as well, so a proxy loses nothing.

Local run at a7f321b: the three provider test files 44 passed; full suite 5389 passed, 16 failed, the same 16 as on main.

[zznate]

Rebased onto main at 38649e1 (head 002c618, the same two commits). One conflict, in complete_json: #1010's raise_if_truncated check and this branch's fence-stripped parse touched the same lines; the resolution keeps both in that order, the truncation check first, then json.loads(_strip_code_fence(content)).

Local run at 002c618 from a clean worktree: the three provider test files 44 passed; full suite 5470 passed, 16 failed, and the same 16 fail on main at 38649e1 in a clean worktree (test_integration_search, test_p3_2_relation_weights, test_a7_classifier_recall, pipeline/test_search_pipeline, test_ph6_entity_linking_storage), none in the LLM provider suites. ruff, ruff format, and mypy clean on the touched files.

[Eldad-Caura]

Maintainer update pushed at 0d6e95c. I rebased the two original signed contributor commits onto current main unchanged, then added a separate signed one-line fix replacing the Ruff C420 comprehension with dict.fromkeys. Verification: Ruff check passed for tests/, Ruff format --check passed for tests/ (395 files), the 30 focused compatibility tests passed, and both naming gates passed. Waiting for exact-head CI before retriggering review.

[Eldad-Caura]

@claude

[github-actions]

🤖 Review by Claude Code

Summary

This PR adds self-hosted/OpenAI-compatible chat LLM support by making the three chat base URLs environment-overridable and by branching complete_json's response_format shape on whether the configured base_url resolves to api.openai.com. The core JSON-schema strictening/null-stripping logic is well tested for the "hosted OpenAI" vs. "self-hosted/localhost" cases, but the classification silently changes behavior for the existing production OpenRouter path, and one docstring paragraph was left stale after the behavior it describes changed.

Medium/Low Issues

OpenRouter is reclassified into the "non-hosted" branch without verification, changing existing production behavior

Severity: Medium
File: common/llm/providers/openai.py:103-121 (new _is_hosted_openai/complete_json branching)
Problem: _is_hosted_openai only returns True for api.openai.com, so calls routed through the existing, already-in-production OPENROUTER_CHAT_BASE_URL (openrouter.ai) now silently drop response_format={"type": "json_object"} for schema-less calls and switch to strict: true with a closed (_strict_schema) schema for schema-based calls — the same treatment given to LM Studio and Anthropic, whose rejection of json_object/strict:false was the stated justification for the change. The docstring and tests only cite LM Studio and Anthropic as verified; OpenRouter's forwarded backends (which vary per model) were never confirmed to reject json_object or to support OpenAI's strict structured-output mode, so existing dedup/enrichment/entity-extraction calls that fall back to OpenRouter could start failing (parse errors from unconstrained JSON, or 400s from unsupported strict schemas) where they previously worked reliably.

🤖 Claude Code Prompt

In common/llm/providers/openai.py, `_is_hosted_openai` (around line 103) is used in `complete_json` (around lines 360-430) to decide whether to send `{"type": "json_object"}` / non-strict json_schema (the "hosted" shape) or no response_format / strict json_schema (the "non-hosted" shape). Because it only matches `api.openai.com`, the existing production OpenRouter path (base_url = OPENROUTER_CHAT_BASE_URL, host "openrouter.ai") now falls into the non-hosted branch alongside newly-added self-hosted/LM-Studio/Anthropic support, even though there is no test or comment confirming OpenRouter actually rejects `json_object` or supports `strict: true` json_schema for the models it proxies. Before merging, either (a) add an explicit third classification (e.g. a set of "known-strict-incompatible" hosts limited to Anthropic/self-hosted, keeping OpenRouter on the previous hosted-style shape it already worked with), or (b) add integration/contract test evidence that OpenRouter's compatible endpoint accepts the non-hosted shapes for the models actually used via OPENROUTER_DEFAULT_MODEL, and document that verification the same way LM Studio/Anthropic are documented in the complete_json docstring and in tests/test_llm_openai_compatible_json_mode.py.

---

Stale docstring still claims strict=False for all schema-based calls

Severity: Low
File: common/llm/providers/openai.py:209-215 (unchanged response_schema docstring paragraph in complete_json)
Problem: The response_schema (A5b #3) docstring paragraph still says "strict=False — Pydantic-generated schemas don't always satisfy OpenAI's strict-mode requirements...", but the new code now sends strict: True (with _strict_schema-closed schemas) for every non-hosted endpoint, making the docstring inaccurate for the new default self-hosted/OpenRouter/Anthropic paths.

🤖 Claude Code Prompt

In common/llm/providers/openai.py, update the `response_schema` (A5b #3) paragraph in the `complete_json` docstring (around line 209-215) to reflect that `strict` is no longer unconditionally `False`: it is `False` (schema sent as-is) only for the hosted OpenAI endpoint, and `True` with a closed schema produced by `_strict_schema` for every other base URL. Cross-reference the new explanatory paragraph already added earlier in the same docstring so the two don't contradict each other.

---

Reviewed by claude-sonnet-5 · cost $0.6781055

[Eldad-Caura]

@claude

[github-actions]

🤖 Review by Claude Code

Summary

The PR adds environment-var overrides for chat-provider base URLs and teaches OpenAILLMProvider to speak strict, non-json_object JSON mode to self-hosted/Anthropic-compatible endpoints (schema closing, null-wrapping optional fields, code-fence stripping). The core logic is well thought out and covered by tests, but the security/observability warning added to the constructor will fire on every call in the exact self-hosted deployment shape the PR documents, and two of the new generic schema-transform helpers have latent correctness gaps that aren't exercised by today's schemas but will silently misbehave if hit.

Medium Issues

Plaintext-off-host warning logs on every LLM call, not once

Severity: Medium
File: common/llm/providers/openai.py (new code in OpenAILLMProvider.__init__, ~lines 264-276)
Problem: get_llm_provider() (common/llm/registry.py) constructs a fresh OpenAILLMProvider per call with no caching, so the new logger.warning(...) for a plain-http non-loopback base_url fires on every single enrichment/dedup/contradiction/entity-extraction request — exactly the host.docker.internal:1234 LM Studio setup the PR's own .env.example recommends.

🤖 Claude Code Prompt

In common/llm/providers/openai.py, the OpenAILLMProvider.__init__ warning added
under `if _is_plaintext_off_host(base_url): logger.warning(...)` runs on every
construction. common/llm/registry.py's get_llm_provider() builds a new
OpenAILLMProvider on every call (no caching/singleton), so any self-hosted
deployment using the documented `OPENAI_CHAT_BASE_URL=http://host.docker.internal:1234/v1`
setup (see .env.example) will log this WARNING on every single LLM call —
enrichment, dedup, contradiction detection, entity extraction — which will
flood logs and trip WARNING-based alerting in exactly the use case this PR
is meant to support.

Fix by deduplicating the warning, e.g. keep a module-level
`_warned_plaintext_urls: set[str] = set()` and only log the first time a given
base_url is seen, or log once via `logging.getLogger(__name__).warning(...,
stacklevel=...)`-independent caching such as `functools.lru_cache` on a small
helper that performs the log side-effect keyed by base_url. Confirm the fix
still warns at least once per distinct offending base_url so the operator is
not left with silent plaintext key transmission.

---

_drop_optional_nulls picks the wrong anyOf/oneOf branch for object-shaped unions

Severity: Medium
File: common/llm/providers/openai.py, _drop_optional_nulls (new function, ~lines 202-230)
Problem: When a schema property is a union of multiple object/array shapes, the loop for option in schema.get("anyOf", []) + schema.get("oneOf", []) returns the first option that merely has properties/items/$ref, without checking whether it structurally matches the actual parsed value, so nulls can be dropped/kept using the wrong sibling schema's required list.

🤖 Claude Code Prompt

In common/llm/providers/openai.py, `_drop_optional_nulls` (the function that
walks a parsed reply next to its *original* Pydantic-generated schema and
strips nulls for fields the source schema left optional) has this fallback for
non-object, non-list values:

    for option in schema.get("anyOf", []) + schema.get("oneOf", []):
        if isinstance(option, dict) and (
            option.get("properties") or option.get("items") or "$ref" in option
        ):
            return _drop_optional_nulls(value, option, root)

This picks the first structurally-eligible option in the union without
checking that `value`'s actual shape (its keys, or whether it's a dict vs
list) matches that particular option. For a schema field typed as a union of
two different object shapes (e.g. a discriminated union), this can apply the
wrong option's `required` set when deciding whether to drop a `None` value,
silently corrupting the returned dict for any future schema that has such a
field (today's ExtractedGraph/EnrichmentResult schemas don't, so this is
currently latent).

Fix by matching the option to the value's shape before recursing — e.g. for a
dict value, only pick an option whose `properties` keys are a superset of (or
overlap with) `value`'s keys, or by resolving `$ref` first and comparing
`type`/`properties` compatibility; if no option matches, return `value`
unchanged rather than guessing the first eligible one.

Low Issues

_strict_schema unconditionally closes additionalProperties, silently breaking open dict schemas

Severity: Low
File: common/llm/providers/openai.py, _strict_schema (new function, ~lines 133-192)
Problem: out["additionalProperties"] = False is set for every object-typed schema, including ones whose original additionalProperties was true or a schema (e.g. a future dict[str, X] field), narrowing them to no extra keys with no error or log, so the model would be unable to emit such a field's contents once routed through the strict path.

🤖 Claude Code Prompt

In common/llm/providers/openai.py, `_strict_schema` always sets
`out["additionalProperties"] = False` and `out["required"] = list(props.keys())`
for any schema with `type == "object"` or a `properties` key. A JSON-Schema
object representing a free-form map (Pydantic's `dict[str, X]`, which renders
as `{"type": "object", "additionalProperties": {...}}` with no `properties`
key) would get `additionalProperties` forced to `False` and `required` forced
to `[]`, silently disallowing the model from returning any of the map's
entries — with no warning that this happened. None of the current schemas
(ExtractedGraph, EnrichmentResult) hit this, but the function is generic
infrastructure reused for any future response_schema.

Either (a) detect this case (object schema with an `additionalProperties`
dict and no `properties`) and leave it unmodified / raise a clear error so a
future author sees the limitation immediately instead of silent data loss, or
add a code comment documenting that dict-typed fields are unsupported under
this strict-mode path and will be silently narrowed to empty objects.

---

New chat-LLM .env.example block interrupts the embedding-instruction comment

Severity: Low
File: .env.example:105-117
Problem: The new "Self-hosted chat LLM" block is inserted directly above the pre-existing # Optional: only set when running an instruction-aware model... comment (which documents EMBEDDING_QUERY_INSTRUCTION, an unrelated embedding setting), with no blank-line/heading separation, making it read as though it belongs to the new chat-LLM section.

🤖 Claude Code Prompt

In .env.example, the new "-- Self-hosted chat LLM (opt-in) --" block (added
around lines 105-117) is inserted immediately before the existing comment
"# Optional: only set when running an instruction-aware model (Qwen3-Embedding,
e5-instruct). bge-m3 is symmetric — leave empty." which documents
EMBEDDING_QUERY_INSTRUCTION, not the chat LLM settings. Move the new chat-LLM
block so it doesn't sit directly above an unrelated embedding comment (e.g.
place it after the EMBEDDING_QUERY_INSTRUCTION line, or add a blank line plus
a section header before the embedding comment) so a reader doesn't associate
the two.

---

Reviewed by claude-sonnet-5 · cost $1.3421975000000002