Release: Ollama as its own pinned service, GPU support, and bounded chat requests

CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to the Forail Assistant will be documented in this file.
 
 ## [Unreleased]
 
+### Fixed
+- **The assistant could not run a model at all.** The image copied `/bin/ollama`
+  out of `ollama/ollama:latest` and nothing else, but Ollama keeps its inference
+  engine in `/usr/lib/ollama` (`llama-server`, `libggml`, the CUDA backends).
+  The server started and answered `/api/tags`, so health checks looked fine,
+  while every generation failed with
+  `error starting llama-server: llama-server binary not found` — HTTP 500. This
+  affects the published `2026.06.0` image, which ships Ollama 0.30.8 without
+  that directory.
+
+### Changed
+- **Ollama now runs as its own service** instead of being bundled into the API
+  image, and its version is **pinned** (`ollama/ollama:0.30.10`) rather than
+  tracking `latest`. Tracking `latest` is what let an upstream layout change
+  break inference without a line of our code changing. Splitting it also means
+  only the model server needs a GPU: in Kubernetes just that pod requests
+  `nvidia.com/gpu` and lands on a GPU node, while the API stays schedulable
+  anywhere.
+- The API waits for Ollama on startup and pulls models over its HTTP API (the
+  `ollama` CLI is no longer present in the image). The wait is bounded at 300s
+  and exits with a clear message instead of hanging.
+
+### Added
+- `docker-compose.gpu.yml` overlay that attaches an NVIDIA GPU to the model
+  server. Kept separate so a host without a GPU fails loudly instead of quietly
+  running on CPU. Measured on an RTX 3080 with `gemma3:1b`: ~5–6× generation
+  throughput over 24-thread CPU inference.
+
 ### Security
 - **CORS** no longer combines a wildcard origin with credentials (a wildcard now
   disables `allow_credentials`).

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 Thanks for your interest in contributing!
 
-The full contributing guide — git workflow, commit conventions, coding standards, PR process — lives in the [forail-deploy repository](https://github.com/forail-platform/forail-devops/blob/main/docs/10-contributing-guide.md). Please read it before submitting a pull request.
+The full contributing guide — git workflow, commit conventions, coding standards, PR process — lives in the [Forail developer docs](https://forail-platform.github.io/dev/contributing.html). Please read it before submitting a pull request.
 
 ## What lives here
 

Dockerfile

@@ -1,7 +1,16 @@
-### Forail Assistant — All-in-one image
-### Ollama (LLM) + ChromaDB (embedded) + FastAPI in a single container
-
-FROM ollama/ollama:latest AS ollama
+### Forail Assistant — API image
+### FastAPI (RAG pipeline) + ChromaDB (embedded).
+###
+### Ollama is NOT bundled here. It runs as its own service so that only the
+### model server needs a GPU (and, in Kubernetes, only that pod needs to land
+### on a GPU node). See docker-compose.yml, or the forail-assistant-ollama
+### Deployment in the Helm chart.
+###
+### The previous all-in-one layout copied /bin/ollama out of the official
+### image on its own. That silently stopped working: modern Ollama keeps the
+### inference engine in /usr/lib/ollama (llama-server, libggml, the CUDA
+### backends), so the copied binary could start a server but never load a
+### model — every request came back 500.
 
 FROM python:3.12-slim
 
@@ -13,9 +22,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 
 WORKDIR /app
 
-# Copy Ollama binary from official image
-COPY --from=ollama /bin/ollama /usr/local/bin/ollama
-
 # Install Python dependencies
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
@@ -28,11 +34,14 @@ COPY docs_to_index/ ./docs_to_index/
 COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /entrypoint.sh
 
-# Directories for data persistence
-RUN mkdir -p /data/ollama /data/chroma
+# Directory for the ChromaDB index. Model blobs live in the Ollama service's
+# own volume, not here.
+RUN mkdir -p /data/chroma
 
-ENV OLLAMA_MODELS=/data/ollama
-ENV FORAIL_ASSISTANT_OLLAMA_BASE_URL=http://localhost:11434
+# Default points at the Ollama service by its compose/Service name. Both the
+# compose file and the Helm chart set this explicitly; the default only keeps
+# a bare `docker run` on the same network working.
+ENV FORAIL_ASSISTANT_OLLAMA_BASE_URL=http://ollama:11434
 ENV FORAIL_ASSISTANT_OLLAMA_MODEL=gemma3:1b
 ENV FORAIL_ASSISTANT_CHROMA_HOST=localhost
 ENV FORAIL_ASSISTANT_CHROMA_PORT=8000

README.md

@@ -10,21 +10,25 @@ AI-powered assistant for the Forail infrastructure automation platform. Uses a l
 
 ## Overview
 
-Forail Assistant is an **optional, standalone service** that can be plugged into or removed from any Forail deployment. It runs as a **single all-in-one container** with Ollama (LLM) and ChromaDB (embedded) bundled inside.
+Forail Assistant is an **optional, standalone service** that can be plugged into or removed from any Forail deployment. It runs as **two containers**: the API (FastAPI + embedded ChromaDB) and Ollama, the model server.
 
 ```
-┌──────────────────┐     ┌──────────────────────────────────────┐
-│  Forail Frontend  │────▶│         Forail Assistant               │
-│  (React chat)    │ SSE │  ┌──────────┐  ┌──────────────────┐  │
-└──────────────────┘     │  │  Ollama   │  │    FastAPI        │  │
-                         │  │ gemma3:1b │  │  (RAG pipeline)   │  │
-                         │  └──────────┘  └────────┬──────────┘  │
-                         │                ┌────────▼──────────┐  │
-                         │                │  ChromaDB (embed)  │  │
-                         │                └───────────────────┘  │
-                         └──────────────────────────────────────┘
+┌──────────────────┐     ┌───────────────────────────────┐     ┌──────────────┐
+│  Forail Frontend  │────▶│      Forail Assistant API      │────▶│    Ollama     │
+│  (React chat)    │ SSE │  ┌──────────────────────────┐ │HTTP │  gemma3:1b    │
+└──────────────────┘     │  │  FastAPI (RAG pipeline)   │ │     │  (GPU here)   │
+                         │  └────────────┬─────────────┘ │     └──────────────┘
+                         │  ┌────────────▼─────────────┐ │
+                         │  │    ChromaDB (embedded)    │ │
+                         │  └──────────────────────────┘ │
+                         └───────────────────────────────┘
 ```
 
+They are separate on purpose: only the model server benefits from a GPU, so
+only it carries the GPU requirement. In Kubernetes that means just one pod has
+to land on a GPU node while the API stays schedulable anywhere. Ollama has no
+authentication, so it is never given a published port — only the API talks to it.
+
 ## Features
 
 - **Contextual help** — knows which page the user is on
@@ -36,9 +40,12 @@ Forail Assistant is an **optional, standalone service** that can be plugged into
 ## Quick Start
 
 ```bash
-# Start the assistant (all-in-one: Ollama + ChromaDB + FastAPI)
+# Start the assistant (API + Ollama)
 docker compose up -d
 
+# ...or with GPU acceleration for the model server (see Hardware below)
+docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
+
 # Wait ~2 minutes for Ollama to load the model on first start,
 # then index documentation
 curl -X POST http://localhost:8100/api/v1/index
@@ -49,7 +56,7 @@ curl -X POST http://localhost:8100/api/v1/chat \
   -d '{"message": "How do I create a job template?"}'
 ```
 
-> **Note:** On first start, the entrypoint automatically pulls the LLM model (`gemma3:1b`) and embedding model (`nomic-embed-text`). The healthcheck `start_period` is 120 seconds to allow time for this.
+> **Note:** On first start, the API waits for Ollama and then pulls the LLM model (`gemma3:1b`) and embedding model (`nomic-embed-text`) over Ollama's API. The healthcheck `start_period` is 120 seconds to allow time for this.
 
 ## Integration with Forail
 
@@ -68,21 +75,41 @@ All settings via environment variables with `FORAIL_ASSISTANT_` prefix:
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `FORAIL_ASSISTANT_OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API URL (localhost — runs inside the same container) |
+| `FORAIL_ASSISTANT_OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API URL. The code defaults to localhost for local development; the image and the Helm chart both override it to point at the Ollama service |
 | `FORAIL_ASSISTANT_OLLAMA_MODEL` | `gemma3:1b` | LLM model |
 | `FORAIL_ASSISTANT_OLLAMA_EMBED_MODEL` | `nomic-embed-text` | Embedding model |
-| `FORAIL_ASSISTANT_CHROMA_HOST` | `localhost` | ChromaDB host (localhost — embedded in the same container) |
+| `FORAIL_ASSISTANT_CHROMA_HOST` | `localhost` | ChromaDB host (localhost — embedded in the API container) |
 | `FORAIL_ASSISTANT_CHROMA_PORT` | `8000` | ChromaDB port |
-| `FORAIL_ASSISTANT_RAG_TOP_K` | `5` | Number of docs to retrieve |
+| `FORAIL_ASSISTANT_RAG_TOP_K` | `3` | Number of docs to retrieve |
 | `FORAIL_ASSISTANT_LOG_LEVEL` | `INFO` | Logging level |
 
 ## Hardware Requirements
 
-| Setup | RAM | GPU | Response Time |
-|-------|-----|-----|---------------|
-| CPU-only (phi3:mini) | 8 GB | None | 10-20s |
-| GPU (mistral:7b) | 16 GB | 8 GB VRAM | 2-5s |
-| GPU (llama3.1:8b) | 32 GB | 12 GB VRAM | 1-3s |
+The GPU overlay needs the NVIDIA driver plus `nvidia-container-toolkit`
+registered with Docker (`nvidia-ctk runtime configure --runtime=docker`).
+Without it the reservation fails and the stack refuses to start — deliberately,
+so that a missing GPU is loud rather than a silent fall back to CPU.
+
+Ollama picks CPU silently when it cannot see a device. Always confirm:
+
+```bash
+docker compose logs ollama | grep "inference compute"
+# GPU:  library=CUDA ... description="NVIDIA GeForce RTX 3080" total="11.6 GiB"
+# CPU:  library=cpu ... name=cpu
+```
+
+Measured on a Ryzen 9 5900X / RTX 3080 12GB, `gemma3:1b`, warm (model already
+resident), same two questions against the same index:
+
+| Setup | Time to first token | Generation throughput |
+|-------|--------------------|----------------------|
+| CPU (24 threads) | ~0.5s | ~680 B/s |
+| GPU (RTX 3080) | ~0.5s | ~3900 B/s |
+
+Time to first token is dominated by RAG retrieval, so it barely moves; the GPU
+buys roughly **5–6× generation throughput**. That matters most as a headroom
+budget: it is what makes a larger, more accurate model affordable at all, since
+an 8B-class model on CPU is slower again by a wide margin.
 
 ## Development
 

app/config.py

@@ -41,6 +41,21 @@ class Settings(BaseSettings):
     # exhausts GPU/CPU. Excess requests get 429.
     chat_max_concurrency: int = 4
 
+    # Bounds on a single request (Codex M3). The concurrency cap limits how many
+    # generations run at once, but says nothing about how large or how long any
+    # one of them is -- four callers could hold every slot for the full Ollama
+    # timeout with a prompt the size of a book.
+    #
+    # A question is a question: 4000 characters is longer than anyone types.
+    chat_max_message_chars: int = 4000
+    # Turns of prior conversation kept. Each one is re-sent to the model, so an
+    # unbounded history is an unbounded prompt, paid for on every request.
+    chat_max_history_turns: int = 20
+    chat_max_history_chars: int = 16000
+    # Hard ceiling on one streamed response, independent of the model's own
+    # timeout. A generation that will not stop still ends.
+    chat_deadline_seconds: int = 180
+
     model_config = {"env_prefix": "FORAIL_ASSISTANT_"}
 
 

app/main.py

@@ -4,6 +4,7 @@
 import hmac
 import json
 import logging
+import time
 
 from fastapi import FastAPI, Header, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
@@ -72,6 +73,46 @@ class ChatRequest(BaseModel):
     history: list[dict] | None = None
 
 
+def _bounded_request(req: "ChatRequest") -> tuple[str, list[dict]]:
+    """
+    The message and history this request is allowed to spend, or 413.
+
+    The concurrency cap limits how many generations run at once and says nothing
+    about how large any one of them is: four callers could hold every slot for
+    the full Ollama timeout with a prompt the size of a book. History matters
+    more than the message, because every turn is re-sent to the model and paid
+    for again on the next request.
+    """
+    message = (req.message or "").strip()
+    if not message:
+        raise HTTPException(status_code=400, detail="message must not be empty")
+    if len(message) > settings.chat_max_message_chars:
+        raise HTTPException(
+            status_code=413,
+            detail=f"message must be at most {settings.chat_max_message_chars} characters",
+        )
+
+    history = req.history or []
+    if not isinstance(history, list):
+        raise HTTPException(status_code=400, detail="history must be a list")
+
+    # Trimmed rather than rejected: dropping the oldest turns degrades the answer
+    # a little, while a 413 in the middle of a conversation ends it.
+    history = history[-settings.chat_max_history_turns:]
+    budget = settings.chat_max_history_chars
+    kept: list[dict] = []
+    for turn in reversed(history):
+        if not isinstance(turn, dict):
+            continue
+        cost = len(str(turn.get("content", "")))
+        if cost > budget:
+            break
+        budget -= cost
+        kept.append(turn)
+    kept.reverse()
+    return message, kept
+
+
 class HealthResponse(BaseModel):
     status: str
     version: str
@@ -129,18 +170,30 @@ async def chat(req: ChatRequest, authorization: str | None = Header(default=None
     if _chat_semaphore.locked():
         raise HTTPException(status_code=429, detail="Assistant busy, retry shortly")
 
+    message, history = _bounded_request(req)
+
     page_context = ""
     if req.context and req.context.get("page"):
-        page_context = req.context["page"]
+        page_context = str(req.context["page"])[:200]
 
     async def event_generator():
         async with _chat_semaphore:
+            deadline = time.monotonic() + settings.chat_deadline_seconds
             try:
                 async for token in stream_chat(
-                    message=req.message,
+                    message=message,
                     page_context=page_context,
-                    history=req.history,
+                    history=history,
                 ):
+                    # A generation that will not stop still has to end: the
+                    # slot it holds is one of only chat_max_concurrency.
+                    if time.monotonic() > deadline:
+                        logger.warning(
+                            "Chat generation exceeded %ss deadline; cutting the stream",
+                            settings.chat_deadline_seconds,
+                        )
+                        yield {"data": json.dumps({"error": "response timed out", "done": True})}
+                        return
                     yield {"data": json.dumps({"token": token})}
                 yield {"data": json.dumps({"done": True})}
             except Exception:

docker-compose.gpu.yml

@@ -0,0 +1,30 @@
+# GPU overlay — hands an NVIDIA GPU to the Ollama service.
+#
+# Kept as an overlay rather than folded into docker-compose.yml because a
+# `devices` reservation is a hard requirement: on a host without a GPU the
+# stack refuses to start instead of quietly running on CPU.
+#
+# Requires on the host:
+#   - NVIDIA driver
+#   - nvidia-container-toolkit, registered with Docker:
+#       sudo nvidia-ctk runtime configure --runtime=docker
+#       sudo systemctl restart docker
+#
+# Verify the host is ready before using this file:
+#   docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
+#
+# Usage:
+#   docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
+#
+# Confirm Ollama actually picked the GPU up (it falls back to CPU silently):
+#   docker compose logs ollama | grep "inference compute"
+# The line must report a CUDA library and non-zero VRAM, not `library=cpu`.
+services:
+  ollama:
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]

docker-compose.integration.yml

@@ -1,5 +1,9 @@
 # Integration overlay for forail-deploy
 # Usage: docker compose -f docker-compose.yml -f docker-compose.integration.yml up -d
+#
+# Only the API joins the Forail network. The ollama service stays on this
+# stack's default network alone — it has no authentication, so nothing outside
+# the assistant should be able to reach it.
 services:
   forail-assistant:
     networks: