Allow for self-hosted inference engines.

[voidstar-x]

Right now the front-end only allows for Google or Anthropic models, and requires API keys by default. Hope you can include on the roadmap more model provider / inference engine options. I am running my own vLLM inference engine.

[jpbreda]

I've created a branch that does just this. It connects to my local vLLM but should support any OpenAI-compatible endpoint.
https://github.com/jpbreda/mike/tree/feature/localllm-provider-support and created PR #20

[danielesalpietro]

This is exactly what I was looking for! I'm currently testing Mike on an RTX 5080 using NVIDIA NIM (Llama 3.1). I'm building a 'one-click' Docker Compose setup to make this process seamless for others. The PR #20 is a game changer for privacy-focused legal environments!