Skip to content

docs: add Inference section to README #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
igor-susic1 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: add Inference section to README #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,50 @@ Kimchi tags every LLM request with a `phase:{name}` label for usage analytics an

Subagents inherit the current phase from the orchestrator but cannot change it.

## Inference

By default, kimchi runs inference against kimchi's hosted LLM infrastructure using the API key configured during `kimchi setup`. Kimchi also ships with native **local inference** support via [Ollama](https://ollama.com/) — no separate package or plugin required.

### Local inference with Ollama

On every startup, kimchi silently probes for a locally-running Ollama server, discovers its models, and merges them into the model registry under the `ollama` provider. The probe is non-blocking: if Ollama is absent, unreachable, or returns nothing, startup proceeds normally and the `ollama` provider is omitted from `models.json` (any stale block from a previous run is removed).

The Ollama endpoint is resolved with this precedence:

1. `$OLLAMA_HOST`
2. `$KIMCHI_OLLAMA_HOST`
3. `http://localhost:11434`

```bash
# Point kimchi at a non-default Ollama host
export OLLAMA_HOST=http://192.168.1.50:11434
```

Discovered models talk to Ollama through its OpenAI-compatible surface (`${host}/v1`), so they work with the existing Pi model registry without a real API key. Cost is zero across the board for Ollama models — input, output, cache read, and cache write all report `0`.

### Role integration

Discovered Ollama models are automatically appended to the **builder**, **reviewer**, and **explorer** role pools. The orchestrator, planner, judge, and researcher roles are intentionally left untouched, so local models never displace the main agent loop or the research role (research models tend to be proprietary web-search wrappers, not something a local Ollama instance should be augmenting).

Each model's tier is derived from its parameter size and feeds the orchestrator's complexity-based routing:

| Parameter size | Tier |
|---|---|
| `< 8B` | `light` |
| `8B` – `< 30B` | `standard` |
| `≥ 30B` | `heavy` |
| unknown | `standard` (override in `models.json` if needed) |

Ollama models appear in `/model` and `ctrl+p` like any other model, and can be toggled per role via `/multi-model`. Tier and description overrides can be set through the same `/multi-model` → "Edit model metadata" menu used for external cloud models.

### Observability

Ollama failures are silent by default — the probe must never block startup. To see what the probe is doing (or why a previously-discovered model vanished), enable the debug namespace:

```bash
NODE_DEBUG=kimi:ollama kimchi
```

## Tags

Kimchi supports tagging LLM requests for usage tracking and cost attribution. Tags are included with every request and displayed in the footer, grouped by key with color coding.
Expand Down