Skip to content

Using openai-endpoint plugin without a runtime #1204

Description

@coryalder

Feature

I've installed the openai-endpoint plugin (https://github.com/Mesh-LLM/openai-endpoint) via mesh-llm plugins install openai-endpoint. I've configured it in ~/.mesh-llm/config.toml:

[[plugin]]
    name = "openai-endpoint"
    url = "http://127.0.0.1:8080/v1"

but since I don't have a working runtime for my machine's GPU (#1203) it gives the following error when I run mesh-llm serve:

root@ollama:~# mesh-llm serve
no compatible native runtime found for Skippy ABI 0.1.35 on linux/x86_64: no compatible MeshLLM native runtime is installed or installable for MeshLLM 0.75.0 / Skippy ABI 0.1.35; run `mesh-llm runtime install` or inspect available runtimes with `mesh-llm runtime list --available`

It seems like there should be a way to tell mesh-llm to ignore the lack of a runtime and depend on the openai-endpoint to provide a model. Not sure if this is a documentation issue, or a missing feature.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions