Add support for OpenAI Chat Completions API compatible third party providers & local models

[nathan-t4]

Add support for local models on CADAM (e.g. hosted by LMStudio, llama.cpp, etc.), as mentioned in #120.

To add a new local model, the user adds local-models.json to the base repo. Detailed instructions are in the README.md

Features

1. Does not break compatibility (without the new local-models.json, everything works as normal)
2. Not all model providers have multimodal tool results for VLMs (https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling). In fact, all OpenAI chat completions compatible third-party model providers (e.g. LMStudio, llama.cpp, vllm, ollama, etc...) do not support multimodal tool results. Thus, the tool result message format is modified from the original hydratedMessages:

return {
  type: 'content' as const,
  value: [
    { type: 'text' as const, text },
    {
      type: 'image-data' as const,
      data: downloaded.base64,
      mediaType: downloaded.mediaType,
    },
  ],
};

to a text-only tool output APPENDED WITH a text and image user messages that includes the multi-view rendering.

result.push({
  role: 'user',
  parts: [
    {
      type: 'text',
      text: 'Multi-view inspection render from the tool result above.',
    },
    {
      type: 'file',
      mediaType: downloaded.mediaType,
      url: `data:${downloaded.mediaType};base64,${downloaded.base64}`,
    },
  ],
});

For more details, see changes in src/server/aiChat.ts.

3. Conversation suggestions and titles can be generated with third-party models that have "useForAux": true, instead of just the default claude haiku model.

Tested with DeepSeek v4 Flash (hosted by DeepSeek) -- a pure text LLM! :

---

Summary by cubic

Adds first‑class support for OpenAI Chat Completions–compatible local/third‑party models (LM Studio, ollama, llama.cpp, vllm) via local-models.json, surfaced in the picker through /api/local-models. Also fixes strict local VLM tool‑image handling by appending the inspection render as a follow‑up user message, routes titles/suggestions to a local aux model when available, caps model picker height, and makes local/third‑party runs free and skip token gates.

• New Features

  • Catalog (local-models.json): entries support id, name, description, baseUrl, optional apiKey (env var name), supportsTools, supportsThinking, supportsVision, supportsForcedToolChoice, useForAux; only entries with baseUrl are active; file is gitignored.
  • API + UI: new /api/local-models exposes picker‑safe configs (no secrets); client hook useParametricModels merges local with cloud; useIsLocalModel removes UI token gating for local runs; UI vision checks use the dynamic list; model pickers are scrollable with a capped height.
  • Routing: anthropic/*, google/* stay direct; catalog ids with a baseUrl route to a per‑target @openrouter/ai-sdk-provider client with compatibility: "compatible" and normalized /v1; others go via OpenRouter; per‑model API keys are resolved server‑side from .env.local.
  • Tool results: cloud models with vision receive inline inspection images; local/strict servers send text‑only tool output and the latest inspection image is appended as a user message (storage‑checked), gated by supportsVision.
  • Behavior: forced tool_choice is disabled for Claude 5 and catalog models with supportsForcedToolChoice: false (fallback: disable when supportsThinking: true); local/third‑party models are billed at 0, skip preflight token checks, and do not consume tokens; selecting a catalog id without a baseUrl returns a clear error.
  • Aux tasks: conversation titles/suggestions use a catalog model marked useForAux when available; else fall back to Anthropic when configured.
  • Tests: added coverage for catalog parsing/routing, supportsForcedToolChoice, and latest‑build detection.

• Migration

  • Optional: copy local-models.json.example to local-models.json, set each model’s id, baseUrl, and flags.
  • If a provider needs auth, set apiKey to the env var name (e.g., DEEPSEEK_API_KEY) and add that variable to .env.local.
  • Restart the dev server; models with a baseUrl appear in the picker. Cloud routes remain unchanged.

^{Written for commit 723e535. Summary will update on new commits.}

[vercel]

@nathan-t4 is attempting to deploy a commit to the Adam Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

Adds first-class support for OpenAI Chat Completions–compatible local and third-party models via an optional local-models.json catalog, surfaced through /api/local-models. Cloud routing is fully preserved when the file is absent.

• Routing: catalog entries with a baseUrl are routed via @openrouter/ai-sdk-provider in compatible mode; anthropic/* and google/* stay on their direct providers; everything else goes through OpenRouter.
• Tool images for local VLMs: instead of inlining inspection renders in tool-result messages (rejected by strict OAI-compatible servers), buildHydratedMessages appends a follow-up role: user message carrying the render — creating consecutive user messages that are accepted by modern OAI servers but untested with an actual VLM in this PR.
• Aux tasks: title and suggestion generation now prefer a catalog entry marked useForAux: true over the hard-coded claude-haiku-4-5 fallback; credentials are resolved server-side from .env.local, never exposed in the client bundle.

Confidence Score: 5/5

Safe to merge. All three findings are non-blocking style/type nits; cloud routing is unchanged, secrets stay server-side, and auth/billing gates are properly applied.

The cloud path is structurally untouched. The new local routing, server-side key resolution, and picker-config API are implemented cleanly. Issues found are a dead variable initialisation, a missing metadata field on a synthetic message that is never read at runtime, and a consecutive-user-message pattern in the VLM injection path that was not smoke-tested but is accepted by all targeted servers.

src/server/aiChat.ts — the toModelOutput else-branch and buildHydratedMessages injection logic warrant a VLM integration test before the feature is considered fully proven.

Important Files Changed

Filename	Overview
src/server/aiChat.ts	Contains the core local-model routing, tool-output branching, and buildHydratedMessages injection. Three issues: dead imageAttached initialisation, missing metadata on the injected user message, and consecutive-user-message pattern for VLMs (untested path).
shared/localModels.ts	Core routing/parsing module for the local catalog. Schema validation, duplicate-id dedup, and provider routing logic are clean and well-tested.
shared/localModels.test.ts	Good coverage of all exported functions including the newly-added getLocalModels and supportsForcedToolChoice cases. Tests verify invariants, not just happy paths.
src/server/localChatConfig.ts	Clean disk-I/O wrapper with first-read caching. API-key resolution via env-var indirection correctly keeps secrets server-side.
src/routes/api/local-models.ts	Server-side endpoint that strips sensitive fields (apiKey, baseUrl) before sending picker configs to the client. Correct server/client separation.
src/hooks/useLocalModels.ts	Merges cloud and local models for the picker; validates response with zod schema; provides useIsLocalModel hook to gate token checks. Correct use of React Query with staleTime.
src/server/messageUtils.ts	New findLatestBuildToolCallId helper used by buildHydratedMessages. Walks branch in reverse, handles multi-tool assistant messages. Well-tested.
src/server/messageUtils.test.ts	Covers all edge cases for findLatestBuildToolCallId including empty branch, multi-message, multi-part, and non-output states.
local-models.json.example	Valid JSON (no trailing comma). Demonstrates both a local VLM and a third-party text model with per-field documentation via the README.
shared/imageRefs.ts	Adds inspectionPreviewStoragePath; follows the existing imageStoragePath pattern exactly.

_{Reviews (11): Last reviewed commit: "Add logging for loadCatalogFromDisk" | Re-trigger Greptile}

[cubic-dev-ai]

5 issues found across 9 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[nathan-t4]

Hey @zachdive @SamStenner, can you help review this PR?
I made some new changes today to merge to the current master branch.