Split Ollama into its own Deployment, with optional GPU - #6
Merged
Conversation
Follows the same split made in forail-assistant: the all-in-one image could never load a model, and Ollama now runs as a separate service. - forail-assistant-ollama Deployment + Service + its own PVC. Model blobs are far larger than the vector index and outlive a rebuild of the API, so keeping the claims apart means the index PVC is sized by the corpus, not the model. - assistant.ollama.gpu.enabled requests nvidia.com/gpu on that pod alone. Only the model server has to land on a GPU node; the API stays schedulable anywhere. Off by default -- it pins the pod to a node advertising the device, so it must stay off until the cluster has one and the device plugin is installed. - images.assistantOllama pinned to ollama/ollama:0.30.10, for the same reason the application image stopped tracking `latest`. - assistant.storage drops 20Gi -> 5Gi now that models live elsewhere, and the API's resource budget no longer has to cover inference. PVCs cannot shrink, so an existing install needs a fresh claim; noted in values.yaml.
…t shrink The chart README never mentioned the assistant at all, and the split added values nobody could discover from it (images.assistantOllama, assistant.ollama.gpu.*). The storage drop also needs saying out loud: it is the one part of this change that breaks an existing install rather than a fresh one.
…be recreated The storage change is the only part of this that can fail an upgrade, so it is called out under Breaking with both ways out rather than left as a values.yaml comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Renders the assistant's model server as its own Deployment, matching the split
made in
forail-assistant.What changed
forail-assistant-ollamaDeployment + Service + its own PVC. Model blobs arefar larger than the vector index and outlive a rebuild of the API, so keeping
the claims apart means the index PVC is sized by the corpus, not the model.
assistant.ollama.gpu.enabledrequestsnvidia.com/gpuon that pod alone —only the model server has to land on a GPU node, the API stays schedulable
anywhere. Off by default: the request pins the pod to a node advertising the
device, so without one it stays
Pending.images.assistantOllamapinned toollama/ollama:0.30.10, for the same reasonthe application image stopped tracking
latest.assistant at all, so the new values were undiscoverable.
Breaking — read before upgrading
assistant.storage.sizedrops 20Gi → 5Gi now that models live on their ownclaim. PVCs cannot shrink, so an existing install with
assistant.enabled=truefails the upgrade on the immutable field. Two ways out:
forail-assistant-dataclaim — the vector index rebuilds itselffrom
docs_to_index/, nothing irreplaceable is stored there; or--set assistant.storage.size=20Gi.Fresh installs need no action.
Verified
helm lintclean.assistant.enabled=true --set assistant.ollama.gpu.enabled=trueand puts
nvidia.com/gpu: 1on the ollama pod only.Pairs with the matching change in
forail-assistant.