diff --git a/README.md b/README.md index 8534ed98d..be19cb498 100644 --- a/README.md +++ b/README.md @@ -135,6 +135,50 @@ Kimchi tags every LLM request with a `phase:{name}` label for usage analytics an Subagents inherit the current phase from the orchestrator but cannot change it. +## Inference + +By default, kimchi runs inference against kimchi's hosted LLM infrastructure using the API key configured during `kimchi setup`. Kimchi also ships with native **local inference** support via [Ollama](https://ollama.com/) — no separate package or plugin required. + +### Local inference with Ollama + +On every startup, kimchi silently probes for a locally-running Ollama server, discovers its models, and merges them into the model registry under the `ollama` provider. The probe is non-blocking: if Ollama is absent, unreachable, or returns nothing, startup proceeds normally and the `ollama` provider is omitted from `models.json` (any stale block from a previous run is removed). + +The Ollama endpoint is resolved with this precedence: + +1. `$OLLAMA_HOST` +2. `$KIMCHI_OLLAMA_HOST` +3. `http://localhost:11434` + +```bash +# Point kimchi at a non-default Ollama host +export OLLAMA_HOST=http://192.168.1.50:11434 +``` + +Discovered models talk to Ollama through its OpenAI-compatible surface (`${host}/v1`), so they work with the existing Pi model registry without a real API key. Cost is zero across the board for Ollama models — input, output, cache read, and cache write all report `0`. + +### Role integration + +Discovered Ollama models are automatically appended to the **builder**, **reviewer**, and **explorer** role pools. The orchestrator, planner, judge, and researcher roles are intentionally left untouched, so local models never displace the main agent loop or the research role (research models tend to be proprietary web-search wrappers, not something a local Ollama instance should be augmenting). + +Each model's tier is derived from its parameter size and feeds the orchestrator's complexity-based routing: + +| Parameter size | Tier | +|---|---| +| `< 8B` | `light` | +| `8B` – `< 30B` | `standard` | +| `≥ 30B` | `heavy` | +| unknown | `standard` (override in `models.json` if needed) | + +Ollama models appear in `/model` and `ctrl+p` like any other model, and can be toggled per role via `/multi-model`. Tier and description overrides can be set through the same `/multi-model` → "Edit model metadata" menu used for external cloud models. + +### Observability + +Ollama failures are silent by default — the probe must never block startup. To see what the probe is doing (or why a previously-discovered model vanished), enable the debug namespace: + +```bash +NODE_DEBUG=kimi:ollama kimchi +``` + ## Tags Kimchi supports tagging LLM requests for usage tracking and cost attribution. Tags are included with every request and displayed in the footer, grouped by key with color coding.