LLM chat integration

[ContextVM-org]

Chat interface on the ContextVM website that lets users connect to upstream models and interact with ContextVM servers from the browser. The goal is to make the ecosystem easier to try and demonstrate through a simple web experience.

[1amKhush]

@ContextVM-org

Proposed Approach - LLM Chat Integration

Taking the approach of the standalone demo (sv5-chat-demo) - functional streaming, configurable provider/model, markdown rendering, cancel support.

Architecture:

• New /chat route - standalone page with full-height chat layout, integrated into the site navigation
• BYOT model - users configure their own endpoint (OpenAI, Groq, Ollama, etc.), API key, and model name. Uses the openai npm SDK (dangerouslyAllowBrowser: true) since the OpenAI Chat Completions API is the de-facto standard and all major providers expose compatible endpoints
• No backend proxy - everything runs in the browser. Keys are stored in localStorage only
• All chat components namespaced in src/lib/components/chat/ to keep the existing codebase clean
• Reuses existing site primitives (shadcn button, textarea, sheet, sonner, etc.)

Phase 1 - Working BYOT Chat (this PR):

1. Service layer (llm.ts) - configurable base URL/key/model, streaming with AbortController, typed error handling
2. Chat components - markdown-rendered bubbles (marked + DOMPurify, both already in deps), auto-resize input, stop/send toggle
3. Config panel - settings sheet for endpoint/key/model with localStorage persistence and test connection
4. /chat route + header nav link

Phase 2 - ContextVM MCP Integration (thought this might be something which we might work on in the future so keeping this as the phase 2):

• Expose connected ContextVM server tools to the LLM as function calls (tool calling flow via mcpClientService)
• Conversation management (IndexedDB persistence, sidebar)
• Server picker for selecting which MCP server the LLM should interact with

I'll keep the scope deliberately small for Phase 1. The chat should be fully functional with any OpenAI-compatible provider out of the box.

Happy to discuss any concerns or alternative approaches before I start.

[ContextVM-org]

Phase 1 sounds great, I would just enhance it a bit with the conversation management so we have a good ux from the get go and users can create new conversations without removing the previous ones. Aside from that we are going to also solve the bootstrap problem by using a default open router api key that we can provide which will use just free models. We can still allow users to set the model, but we can also have an 'auto' mode that basically get the models from the models endpoint filter by the free ones and automatically rotates if they are rate limited. We can have a disclaimer about that and mention that for better stability byot is the way to go, but in this way we would have a 0 friction demo. Also another ux improvements would be to display all the list of models that we can get form the models enpoint and use a combobox component to pick from it (https://www.shadcn-svelte.com/docs/components/combobox). We can also use a similar approach for the provider open ai api endpoint, so it is easy to configure different common providers like open ai, open router, ollama, and so on. Ideally after the phase 1 we have a great chat experience with an auto mode. We can also let the ui in god state for phase 2.
I think we can use a collapsible sidebar so the user can navigate between panels, for example default left panel are conversations, but when start the phase 2 iteration we could add a new item in the sidebar to switch the panel to server selection so the user can see the catalog of servers, to what servers it is connected and so on.
In order to manage the servers we can just reuse the ones that are available in the mcpClient.svelte.ts service so we dont need a separate service layer for connected chat servers and also makes the usage more straightforward since for example users can connect to a server from the servers page in the page and that would already be present in the chat.

Let me know your thoughts

[1amKhush]

@ContextVM-org Awesome feedback, thanks! I've incorporated all of that into the updated plan. Here is the revised approach for Phase 1:

1. Zero-Friction "Auto Mode" (Default)

• We'll ship with the provided OpenRouter key hardcoded as the default.
• By default, the chat will fetch from /v1/models, filter for :free, and automatically rotate through available free models.
• If a rate limit or 429 error occurs mid-stream, it will transparently retry with the next free model in the rotation.
• We'll show a small disclaimer banner noting that they are using free models and recommending BYOT for better stability.

2. Conversation Management (Moved to Phase 1)

• I'll implement IndexedDB persistence from the start so users don't lose context.
• We'll use the shadcn-svelte Sidebar component (collapsible to an icon rail on desktop, offcanvas on mobile).
• The default left panel will be the Conversation List (create, rename, delete).

3. UX Improvements for Config

• Settings will be moved into a sleek sheet/dialog.
• Provider Combobox: Preset list for OpenRouter, OpenAI, Ollama, LM Studio, etc., making base URL switching a one-click action.
• Model Combobox: Dynamic fetch from the active provider's /models endpoint. For OpenRouter, "Auto (Free Models)" will be pinned to the top.

4. Phase 2 Readiness

• Servers Panel: The sidebar will use a panel-switcher pattern. For Phase 2, we just add a "Servers" tab.
• mcpClientService Reuse: As suggested, we won't need a new service. Servers connected from the /servers page are already in the mcpClientService singleton, so the chat will just read from that state.

[ContextVM-org]

Sounds great! No objections