Problem statement:
Currently QMD downloads and runs models automatically in the same environment (host, sandbox, container) it is installed in. This can be redundant and wasteful of precious local resources in a local LLM environment unless QMD's opinion that all models should be run in a monolithic environment aligns with the end user's environment.
models.embed / models.rerank / models.generate settings and QMD_EMBED_MODEL environmental variable honoring exist, but only accepts Huggingface GGUF URIs within the same local download and run framework.
This formulation blocks the usage of other model runtimes, particularly Docker Model Runner provided models.
Proposal:
Expand models.embed / models.rerank / models.generate, refactoring into separate model, endpoint, and source URI fields, with optional authentication (not necessary for DMR, but useful for the general pattern) and matching environmental variables, as example:
models:
embed:
name: "embeddinggemma-300M-Q8_0.gguf",
# If URI exists, assume local. Require either URI or endpoint/api
URI: "hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf",
endpoint: "http://host.docker.internal:12434/engines/llama.cpp/v1/",
api: "openai-completions",
apiKey: "null"
rerank:
name: "qwen3-reranker-0.6b-q8_0.gguf"
...
Desired End-state
The user would be able to specify a custom model endpoint and run QMD against DMR models, saving resources across isolated projects and leveraging existing model endpoints.
Problem statement:
Currently QMD downloads and runs models automatically in the same environment (host, sandbox, container) it is installed in. This can be redundant and wasteful of precious local resources in a local LLM environment unless QMD's opinion that all models should be run in a monolithic environment aligns with the end user's environment.
models.embed / models.rerank / models.generate settings and QMD_EMBED_MODEL environmental variable honoring exist, but only accepts Huggingface GGUF URIs within the same local download and run framework.
This formulation blocks the usage of other model runtimes, particularly Docker Model Runner provided models.
Proposal:
Expand models.embed / models.rerank / models.generate, refactoring into separate model, endpoint, and source URI fields, with optional authentication (not necessary for DMR, but useful for the general pattern) and matching environmental variables, as example:
Desired End-state
The user would be able to specify a custom model endpoint and run QMD against DMR models, saving resources across isolated projects and leveraging existing model endpoints.