Feature
I've installed the openai-endpoint plugin (https://github.com/Mesh-LLM/openai-endpoint) via mesh-llm plugins install openai-endpoint. I've configured it in ~/.mesh-llm/config.toml:
[[plugin]]
name = "openai-endpoint"
url = "http://127.0.0.1:8080/v1"
but since I don't have a working runtime for my machine's GPU (#1203) it gives the following error when I run mesh-llm serve:
root@ollama:~# mesh-llm serve
no compatible native runtime found for Skippy ABI 0.1.35 on linux/x86_64: no compatible MeshLLM native runtime is installed or installable for MeshLLM 0.75.0 / Skippy ABI 0.1.35; run `mesh-llm runtime install` or inspect available runtimes with `mesh-llm runtime list --available`
It seems like there should be a way to tell mesh-llm to ignore the lack of a runtime and depend on the openai-endpoint to provide a model. Not sure if this is a documentation issue, or a missing feature.
Feature
I've installed the openai-endpoint plugin (https://github.com/Mesh-LLM/openai-endpoint) via
mesh-llm plugins install openai-endpoint. I've configured it in~/.mesh-llm/config.toml:but since I don't have a working runtime for my machine's GPU (#1203) it gives the following error when I run
mesh-llm serve:It seems like there should be a way to tell mesh-llm to ignore the lack of a runtime and depend on the openai-endpoint to provide a model. Not sure if this is a documentation issue, or a missing feature.