Run Ollama as its own pinned service, not a binary lifted from its image

CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to the Forail Assistant will be documented in this file.
 
 ## [Unreleased]
 
+### Fixed
+- **The assistant could not run a model at all.** The image copied `/bin/ollama`
+  out of `ollama/ollama:latest` and nothing else, but Ollama keeps its inference
+  engine in `/usr/lib/ollama` (`llama-server`, `libggml`, the CUDA backends).
+  The server started and answered `/api/tags`, so health checks looked fine,
+  while every generation failed with
+  `error starting llama-server: llama-server binary not found` — HTTP 500. This
+  affects the published `2026.06.0` image, which ships Ollama 0.30.8 without
+  that directory.
+
+### Changed
+- **Ollama now runs as its own service** instead of being bundled into the API
+  image, and its version is **pinned** (`ollama/ollama:0.30.10`) rather than
+  tracking `latest`. Tracking `latest` is what let an upstream layout change
+  break inference without a line of our code changing. Splitting it also means
+  only the model server needs a GPU: in Kubernetes just that pod requests
+  `nvidia.com/gpu` and lands on a GPU node, while the API stays schedulable
+  anywhere.
+- The API waits for Ollama on startup and pulls models over its HTTP API (the
+  `ollama` CLI is no longer present in the image). The wait is bounded at 300s
+  and exits with a clear message instead of hanging.
+
+### Added
+- `docker-compose.gpu.yml` overlay that attaches an NVIDIA GPU to the model
+  server. Kept separate so a host without a GPU fails loudly instead of quietly
+  running on CPU. Measured on an RTX 3080 with `gemma3:1b`: ~5–6× generation
+  throughput over 24-thread CPU inference.
+
 ### Security
 - **CORS** no longer combines a wildcard origin with credentials (a wildcard now
   disables `allow_credentials`).

Dockerfile

@@ -1,7 +1,16 @@
-### Forail Assistant — All-in-one image
-### Ollama (LLM) + ChromaDB (embedded) + FastAPI in a single container
-
-FROM ollama/ollama:latest AS ollama
+### Forail Assistant — API image
+### FastAPI (RAG pipeline) + ChromaDB (embedded).
+###
+### Ollama is NOT bundled here. It runs as its own service so that only the
+### model server needs a GPU (and, in Kubernetes, only that pod needs to land
+### on a GPU node). See docker-compose.yml, or the forail-assistant-ollama
+### Deployment in the Helm chart.
+###
+### The previous all-in-one layout copied /bin/ollama out of the official
+### image on its own. That silently stopped working: modern Ollama keeps the
+### inference engine in /usr/lib/ollama (llama-server, libggml, the CUDA
+### backends), so the copied binary could start a server but never load a
+### model — every request came back 500.
 
 FROM python:3.12-slim
 
@@ -13,9 +22,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 
 WORKDIR /app
 
-# Copy Ollama binary from official image
-COPY --from=ollama /bin/ollama /usr/local/bin/ollama
-
 # Install Python dependencies
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
@@ -28,11 +34,14 @@ COPY docs_to_index/ ./docs_to_index/
 COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /entrypoint.sh
 
-# Directories for data persistence
-RUN mkdir -p /data/ollama /data/chroma
+# Directory for the ChromaDB index. Model blobs live in the Ollama service's
+# own volume, not here.
+RUN mkdir -p /data/chroma
 
-ENV OLLAMA_MODELS=/data/ollama
-ENV FORAIL_ASSISTANT_OLLAMA_BASE_URL=http://localhost:11434
+# Default points at the Ollama service by its compose/Service name. Both the
+# compose file and the Helm chart set this explicitly; the default only keeps
+# a bare `docker run` on the same network working.
+ENV FORAIL_ASSISTANT_OLLAMA_BASE_URL=http://ollama:11434
 ENV FORAIL_ASSISTANT_OLLAMA_MODEL=gemma3:1b
 ENV FORAIL_ASSISTANT_CHROMA_HOST=localhost
 ENV FORAIL_ASSISTANT_CHROMA_PORT=8000

README.md

@@ -10,21 +10,25 @@ AI-powered assistant for the Forail infrastructure automation platform. Uses a l
 
 ## Overview
 
-Forail Assistant is an **optional, standalone service** that can be plugged into or removed from any Forail deployment. It runs as a **single all-in-one container** with Ollama (LLM) and ChromaDB (embedded) bundled inside.
+Forail Assistant is an **optional, standalone service** that can be plugged into or removed from any Forail deployment. It runs as **two containers**: the API (FastAPI + embedded ChromaDB) and Ollama, the model server.
 
 ```
-┌──────────────────┐     ┌──────────────────────────────────────┐
-│  Forail Frontend  │────▶│         Forail Assistant               │
-│  (React chat)    │ SSE │  ┌──────────┐  ┌──────────────────┐  │
-└──────────────────┘     │  │  Ollama   │  │    FastAPI        │  │
-                         │  │ gemma3:1b │  │  (RAG pipeline)   │  │
-                         │  └──────────┘  └────────┬──────────┘  │
-                         │                ┌────────▼──────────┐  │
-                         │                │  ChromaDB (embed)  │  │
-                         │                └───────────────────┘  │
-                         └──────────────────────────────────────┘
+┌──────────────────┐     ┌───────────────────────────────┐     ┌──────────────┐
+│  Forail Frontend  │────▶│      Forail Assistant API      │────▶│    Ollama     │
+│  (React chat)    │ SSE │  ┌──────────────────────────┐ │HTTP │  gemma3:1b    │
+└──────────────────┘     │  │  FastAPI (RAG pipeline)   │ │     │  (GPU here)   │
+                         │  └────────────┬─────────────┘ │     └──────────────┘
+                         │  ┌────────────▼─────────────┐ │
+                         │  │    ChromaDB (embedded)    │ │
+                         │  └──────────────────────────┘ │
+                         └───────────────────────────────┘
 ```
 
+They are separate on purpose: only the model server benefits from a GPU, so
+only it carries the GPU requirement. In Kubernetes that means just one pod has
+to land on a GPU node while the API stays schedulable anywhere. Ollama has no
+authentication, so it is never given a published port — only the API talks to it.
+
 ## Features
 
 - **Contextual help** — knows which page the user is on
@@ -36,9 +40,12 @@ Forail Assistant is an **optional, standalone service** that can be plugged into
 ## Quick Start
 
 ```bash
-# Start the assistant (all-in-one: Ollama + ChromaDB + FastAPI)
+# Start the assistant (API + Ollama)
 docker compose up -d
 
+# ...or with GPU acceleration for the model server (see Hardware below)
+docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
+
 # Wait ~2 minutes for Ollama to load the model on first start,
 # then index documentation
 curl -X POST http://localhost:8100/api/v1/index
@@ -49,7 +56,7 @@ curl -X POST http://localhost:8100/api/v1/chat \
   -d '{"message": "How do I create a job template?"}'
 ```
 
-> **Note:** On first start, the entrypoint automatically pulls the LLM model (`gemma3:1b`) and embedding model (`nomic-embed-text`). The healthcheck `start_period` is 120 seconds to allow time for this.
+> **Note:** On first start, the API waits for Ollama and then pulls the LLM model (`gemma3:1b`) and embedding model (`nomic-embed-text`) over Ollama's API. The healthcheck `start_period` is 120 seconds to allow time for this.
 
 ## Integration with Forail
 
@@ -68,21 +75,41 @@ All settings via environment variables with `FORAIL_ASSISTANT_` prefix:
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `FORAIL_ASSISTANT_OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API URL (localhost — runs inside the same container) |
+| `FORAIL_ASSISTANT_OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API URL. The code defaults to localhost for local development; the image and the Helm chart both override it to point at the Ollama service |
 | `FORAIL_ASSISTANT_OLLAMA_MODEL` | `gemma3:1b` | LLM model |
 | `FORAIL_ASSISTANT_OLLAMA_EMBED_MODEL` | `nomic-embed-text` | Embedding model |
-| `FORAIL_ASSISTANT_CHROMA_HOST` | `localhost` | ChromaDB host (localhost — embedded in the same container) |
+| `FORAIL_ASSISTANT_CHROMA_HOST` | `localhost` | ChromaDB host (localhost — embedded in the API container) |
 | `FORAIL_ASSISTANT_CHROMA_PORT` | `8000` | ChromaDB port |
-| `FORAIL_ASSISTANT_RAG_TOP_K` | `5` | Number of docs to retrieve |
+| `FORAIL_ASSISTANT_RAG_TOP_K` | `3` | Number of docs to retrieve |
 | `FORAIL_ASSISTANT_LOG_LEVEL` | `INFO` | Logging level |
 
 ## Hardware Requirements
 
-| Setup | RAM | GPU | Response Time |
-|-------|-----|-----|---------------|
-| CPU-only (phi3:mini) | 8 GB | None | 10-20s |
-| GPU (mistral:7b) | 16 GB | 8 GB VRAM | 2-5s |
-| GPU (llama3.1:8b) | 32 GB | 12 GB VRAM | 1-3s |
+The GPU overlay needs the NVIDIA driver plus `nvidia-container-toolkit`
+registered with Docker (`nvidia-ctk runtime configure --runtime=docker`).
+Without it the reservation fails and the stack refuses to start — deliberately,
+so that a missing GPU is loud rather than a silent fall back to CPU.
+
+Ollama picks CPU silently when it cannot see a device. Always confirm:
+
+```bash
+docker compose logs ollama | grep "inference compute"
+# GPU:  library=CUDA ... description="NVIDIA GeForce RTX 3080" total="11.6 GiB"
+# CPU:  library=cpu ... name=cpu
+```
+
+Measured on a Ryzen 9 5900X / RTX 3080 12GB, `gemma3:1b`, warm (model already
+resident), same two questions against the same index:
+
+| Setup | Time to first token | Generation throughput |
+|-------|--------------------|----------------------|
+| CPU (24 threads) | ~0.5s | ~680 B/s |
+| GPU (RTX 3080) | ~0.5s | ~3900 B/s |
+
+Time to first token is dominated by RAG retrieval, so it barely moves; the GPU
+buys roughly **5–6× generation throughput**. That matters most as a headroom
+budget: it is what makes a larger, more accurate model affordable at all, since
+an 8B-class model on CPU is slower again by a wide margin.
 
 ## Development
 

docker-compose.gpu.yml

@@ -0,0 +1,30 @@
+# GPU overlay — hands an NVIDIA GPU to the Ollama service.
+#
+# Kept as an overlay rather than folded into docker-compose.yml because a
+# `devices` reservation is a hard requirement: on a host without a GPU the
+# stack refuses to start instead of quietly running on CPU.
+#
+# Requires on the host:
+#   - NVIDIA driver
+#   - nvidia-container-toolkit, registered with Docker:
+#       sudo nvidia-ctk runtime configure --runtime=docker
+#       sudo systemctl restart docker
+#
+# Verify the host is ready before using this file:
+#   docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
+#
+# Usage:
+#   docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
+#
+# Confirm Ollama actually picked the GPU up (it falls back to CPU silently):
+#   docker compose logs ollama | grep "inference compute"
+# The line must report a CUDA library and non-zero VRAM, not `library=cpu`.
+services:
+  ollama:
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]

docker-compose.integration.yml

@@ -1,5 +1,9 @@
 # Integration overlay for forail-deploy
 # Usage: docker compose -f docker-compose.yml -f docker-compose.integration.yml up -d
+#
+# Only the API joins the Forail network. The ollama service stays on this
+# stack's default network alone — it has no authentication, so nothing outside
+# the assistant should be able to reach it.
 services:
   forail-assistant:
     networks:

docker-compose.yml

@@ -1,22 +1,35 @@
 services:
+  # Model server. Pinned deliberately: this used to be `ollama/ollama:latest`
+  # scavenged for its binary, and an upstream layout change broke inference
+  # without changing a line of our code. Bump this tag on purpose, not by drift.
+  ollama:
+    image: ollama/ollama:0.30.10
+    volumes:
+      - ollama_models:/root/.ollama
+    # No published port on purpose — Ollama has no authentication. It is
+    # reachable only by services on this compose network.
+    healthcheck:
+      test: ["CMD-SHELL", "ollama list > /dev/null 2>&1 || exit 1"]
+      interval: 10s
+      timeout: 5s
+      start_period: 30s
+      retries: 5
+    restart: unless-stopped
+
   forail-assistant:
     build: .
     image: ghcr.io/forail-platform/forail-assistant:latest
+    depends_on:
+      ollama:
+        condition: service_healthy
     ports:
       - "8100:8100"
     volumes:
       - assistant_data:/data
     environment:
+      - FORAIL_ASSISTANT_OLLAMA_BASE_URL=http://ollama:11434
       - FORAIL_ASSISTANT_OLLAMA_MODEL=gemma3:1b
       - FORAIL_ASSISTANT_LOG_LEVEL=INFO
-    # Uncomment for GPU support:
-    # deploy:
-    #   resources:
-    #     reservations:
-    #       devices:
-    #         - driver: nvidia
-    #           count: 1
-    #           capabilities: [gpu]
     healthcheck:
       test: ["CMD", "curl", "-sf", "http://localhost:8100/api/v1/health"]
       interval: 30s
@@ -27,3 +40,4 @@ services:
 
 volumes:
   assistant_data:
+  ollama_models:

docs/architecture.md

@@ -12,20 +12,34 @@ Forail Assistant is a standalone microservice that provides AI-powered help for
                                     │ SSE (Server-Sent Events)
                                     ▼
                     ┌─────────────────────────────────┐
-                    │   Forail Assistant (all-in-one)   │
+                    │   Forail Assistant (API)          │
                     │                                  │
                     │  FastAPI  /api/v1/chat  → SSE    │
                     │           /api/v1/health→ JSON   │
                     │           /api/v1/index → reindex│
                     │                                  │
-                    │  Ollama   gemma3:1b (LLM)        │
-                    │           nomic-embed-text (emb)  │
-                    │                                  │
                     │  ChromaDB (embedded vector store) │
+                    └───────────────┬──────────────────┘
+                                    │ HTTP (internal network only)
+                                    ▼
+                    ┌─────────────────────────────────┐
+                    │   Ollama (model server)           │
+                    │                                  │
+                    │  gemma3:1b        (generation)   │
+                    │  nomic-embed-text (embeddings)   │
+                    │                                  │
+                    │  optional: NVIDIA GPU             │
                     └─────────────────────────────────┘
 ```
 
-> **Note:** All three components (FastAPI, Ollama, ChromaDB) run inside a single Docker container. The entrypoint script starts Ollama and ChromaDB as background processes before launching the FastAPI server.
+> **Note:** These are **two services**, not one container. The API image carries
+> FastAPI and an embedded ChromaDB; Ollama runs beside it from its own pinned
+> upstream image. In Compose that is the `ollama` service, in Kubernetes the
+> `forail-assistant-ollama` Deployment. Only the model server needs a GPU or a
+> large volume, so only it has to land on a GPU node.
+>
+> Ollama has no authentication. It is reachable on the internal network alone —
+> no published port in Compose, and a ClusterIP Service in Kubernetes.
 
 ## Components
 
@@ -44,13 +58,28 @@ The core service, responsible for:
 - `app/indexer.py` — Document loading, chunking, indexing
 - `app/config.py` — Pydantic settings from environment
 
-### Ollama (LLM Server)
+### Ollama (Model Server)
 
-Runs the language model locally inside the same container. Two models are used:
+Runs the language models locally, in its own container. Two models are used:
 - **gemma3:1b** (default, or configured model) — for chat generation
 - **nomic-embed-text** — for generating document/query embeddings
 
-The Ollama binary is copied from the official `ollama/ollama` image at build time. GPU acceleration is optional but recommended for larger models.
+It runs from the official `ollama/ollama` image, **pinned** to an exact tag. It
+used to be a single binary copied out of that image at build time, and that
+broke silently: modern Ollama keeps its inference engine in `/usr/lib/ollama`
+(`llama-server`, `libggml`, the CUDA backends), so the copied binary could start
+a server and answer `/api/tags` — passing the health check — while every
+generation returned HTTP 500. Running the upstream image whole, at a tag we
+choose deliberately, is what closes that class of failure.
+
+The API never shells out to the `ollama` CLI; it is not in the API image at all.
+Models are pulled over Ollama's HTTP API on first start, and the API waits for
+the model server before it accepts traffic (bounded at 300s, then it exits with
+an actionable message rather than hanging).
+
+GPU acceleration is optional and applies to this service alone — the
+`docker-compose.gpu.yml` overlay in Compose, `assistant.ollama.gpu.enabled` in
+the Helm chart. See the README for measured numbers.
 
 ### ChromaDB (Vector Store)
 
@@ -90,7 +119,7 @@ Stores document chunks as vectors for similarity search. When a user asks a ques
 
 - **No Django dependency** — FastAPI is lighter, async-native, no ORM needed
 - **Optional** — can be added/removed without touching core Forail
-- **Independent scaling** — can run on a separate GPU server
+- **Independent scaling** — the model server is a separate service and can sit on its own GPU node
 - **Independent release cycle** — update models without redeploying Forail
 
 ### Why Ollama (Not OpenAI/Claude API)?