Configurable URL for local models + remove Aitta-specific code#205
Open
maciejjan wants to merge 1 commit intoCliDyn:mainfrom
Open
Configurable URL for local models + remove Aitta-specific code#205maciejjan wants to merge 1 commit intoCliDyn:mainfrom
maciejjan wants to merge 1 commit intoCliDyn:mainfrom
Conversation
This introduces new config options for configuring the url of the local models: - `llm_local_endpoint_url` (for LLMs), - `rag_settings.local_endpoint_url` (for the RAG embedding model). All references to Aitta are removed from the code. Instead, Aitta can be used by just setting the above URLs. The script for creating RAG DB has been adjusted: - removed redundant embedding calculation, - batching documents for better performance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This introduces new config options for configuring the URL of the local models:
llm_local_endpoint_url(for LLMs),rag_settings.local_endpoint_url(for the RAG embedding model).All references to Aitta are removed from the code. Instead, Aitta can be used by just setting the above URLs and the API token. (We can get in touch on Mattermost to provide a token.) By default, the URLs point to localhost where e.g. an own vLLM can be running.
The script for creating RAG DB has been adjusted (see #182):
Rationale: Aitta is now a drop-in replacement for the OpenAI API, so it can be used by setting the base URL and access token in classes like
langchain_openai.ChatOpenAI. I have thus repurposed the "local" endpoint option by making the URL configurable (instead oflocalhost:8000) so that it can point to a custom OpenAI-compatible LLM provider, either local or remote - for example Aitta.Early versions of Aitta (a year ago) needed some custom initialization code and the
aitta-clientdependency, which was added in #174 and other earlier PRs. This is no longer needed. Thus, I have cleaned up all Aitta-specific code and settings.