Skip to content

Support OpenAI-compatible backends for generation, embeddings, and reranking #620

Description

@loopyd

Problem

QMD is currently centered around local llama.cpp model access.

That makes it harder to use QMD with a local or self-hosted OpenAI-compatible server when a user wants to route:

  • query expansion / generation
  • embeddings
  • reranking

through the same remote service.

There are already requests for remote embeddings, but the broader problem is allowing QMD to work with a compatible server across the whole retrieval flow.

Related issues:

Solution

Add support for an OpenAI-compatible backend that can be configured in QMD and used for:

  • generation
  • embeddings
  • reranking

This should allow users to point QMD at a compatible local or self-hosted server, configure model aliases, and run the normal QMD flow without requiring direct local model access.

Reference

PR #619 implements this feature:
#619

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions