Skip to content

Split Ollama into its own Deployment, with optional GPU - #6

Merged
krlex merged 3 commits into
developfrom
feat/split-ollama-service
Aug 19, 2026
Merged

Split Ollama into its own Deployment, with optional GPU#6
krlex merged 3 commits into
developfrom
feat/split-ollama-service

Conversation

@krlex

@krlex krlex commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Renders the assistant's model server as its own Deployment, matching the split
made in forail-assistant.

What changed

  • forail-assistant-ollama Deployment + Service + its own PVC. Model blobs are
    far larger than the vector index and outlive a rebuild of the API, so keeping
    the claims apart means the index PVC is sized by the corpus, not the model.
  • assistant.ollama.gpu.enabled requests nvidia.com/gpu on that pod alone
    only the model server has to land on a GPU node, the API stays schedulable
    anywhere. Off by default: the request pins the pod to a node advertising the
    device, so without one it stays Pending.
  • images.assistantOllama pinned to ollama/ollama:0.30.10, for the same reason
    the application image stopped tracking latest.
  • The Service is ClusterIP; Ollama has no authentication, so only the API reaches it.
  • Chart README gained an assistant section — it previously did not mention the
    assistant at all, so the new values were undiscoverable.

Breaking — read before upgrading

assistant.storage.size drops 20Gi → 5Gi now that models live on their own
claim. PVCs cannot shrink, so an existing install with assistant.enabled=true
fails the upgrade on the immutable field. Two ways out:

  • delete the forail-assistant-data claim — the vector index rebuilds itself
    from docs_to_index/, nothing irreplaceable is stored there; or
  • keep the old size: --set assistant.storage.size=20Gi.

Fresh installs need no action.

Verified

  • helm lint clean.
  • Renders with assistant.enabled=true --set assistant.ollama.gpu.enabled=true
    and puts nvidia.com/gpu: 1 on the ollama pod only.
  • With the assistant disabled, no ollama resources are rendered at all.

Pairs with the matching change in forail-assistant.

krlex added 3 commits August 1, 2026 17:48
Follows the same split made in forail-assistant: the all-in-one image could
never load a model, and Ollama now runs as a separate service.

- forail-assistant-ollama Deployment + Service + its own PVC. Model blobs are
  far larger than the vector index and outlive a rebuild of the API, so keeping
  the claims apart means the index PVC is sized by the corpus, not the model.
- assistant.ollama.gpu.enabled requests nvidia.com/gpu on that pod alone. Only
  the model server has to land on a GPU node; the API stays schedulable
  anywhere. Off by default -- it pins the pod to a node advertising the device,
  so it must stay off until the cluster has one and the device plugin is
  installed.
- images.assistantOllama pinned to ollama/ollama:0.30.10, for the same reason
  the application image stopped tracking `latest`.
- assistant.storage drops 20Gi -> 5Gi now that models live elsewhere, and the
  API's resource budget no longer has to cover inference. PVCs cannot shrink,
  so an existing install needs a fresh claim; noted in values.yaml.
…t shrink

The chart README never mentioned the assistant at all, and the split added
values nobody could discover from it (images.assistantOllama,
assistant.ollama.gpu.*). The storage drop also needs saying out loud: it is the
one part of this change that breaks an existing install rather than a fresh
one.
…be recreated

The storage change is the only part of this that can fail an upgrade, so it is
called out under Breaking with both ways out rather than left as a values.yaml
comment.
@krlex
krlex merged commit 682fee7 into develop Aug 19, 2026
1 check passed
@krlex
krlex deleted the feat/split-ollama-service branch August 19, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant