Skip to content
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **Robust Ollama integration** - the Ollama adapter now supports thinking (`reasoning_effort`/`thinking_budget_tokens` map to the native `think` parameter, with `ThinkingBlock` content and `thinking` stream events), structured outputs via `chat(..., response_format=...)` translated to Ollama's `format`, sampling options (`max_tokens` → `num_predict`, plus a `default_options` passthrough for `seed`, `top_p`, `num_ctx`, and friends), `keep_alive`, and ollama.com cloud authentication through `OLLAMA_API_KEY` or `api_key=`.

### Fixed

- **Ollama image input and error reporting** - image blocks are now sent as the native base64 `images` array (previously an unsupported `image_url` shape the server ignored), mid-stream NDJSON error objects surface as terminal `error` stream events instead of a silently truncated answer, `done_reason: "length"` maps to `stop_reason="max_tokens"`, and a `404` names the missing model with the exact `ollama pull` command.

## [0.8.2] - 2026-07-19

### Changed
Expand Down
23 changes: 22 additions & 1 deletion site_docs/guides/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ AnyCode is provider-agnostic: every agent names a `provider` and a `model`, and
| `anthropic` | `claude-haiku-4-5`, `claude-sonnet-5` | `ANTHROPIC_API_KEY` | `anycode-py[anthropic]` |
| `openai` | `gpt-4o-mini`, `gpt-5` | `OPENAI_API_KEY` | `anycode-py[openai]` |
| `google` | `gemini-2.5-flash` | `GOOGLE_API_KEY` or `GEMINI_API_KEY` | `anycode-py[google]` |
| `ollama` | any local model (e.g. `llama3.1`) | none (local server) | `anycode-py[ollama]` |
| `ollama` | any local model (e.g. `llama3.1`), or cloud models on ollama.com | none locally; `OLLAMA_API_KEY` for ollama.com | `anycode-py[ollama]` |
| `bedrock` | AWS-hosted model IDs | AWS creds + `AWS_DEFAULT_REGION` | `anycode-py[bedrock]` |
| `azure` | your deployment name | `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY` | `anycode-py[azure]` |

Expand Down Expand Up @@ -110,6 +110,27 @@ print(result.content[0].text, result.usage.input_tokens)
| `endpoint`, `api_version` | azure | Azure resource endpoint and API version |
| `region`, `profile` | bedrock | AWS region and named profile |

### Ollama: local and cloud

The Ollama adapter speaks the native `/api/chat` endpoint and covers streaming, tool calling, thinking, structured outputs, and multimodal image input. A few knobs are adapter-specific, so construct `OllamaAdapter` directly when you need them:

```python title="ollama_tuned.py"
from anycode.providers.ollama import OllamaAdapter

adapter = OllamaAdapter(
model="qwen3",
keep_alive="10m", # keep the model loaded between calls
think="high", # default thinking level (bool or low/medium/high/max)
default_options={"num_ctx": 8192, "seed": 7}, # any native Ollama option
)
```

- `LLMChatOptions.max_tokens` maps to Ollama's `num_predict`; `temperature` passes through. Per-call options always override `default_options`.
- `reasoning_effort` maps to the `think` level and `message.thinking` comes back as `ThinkingBlock` content and `thinking` stream events — see [Use reasoning models](reasoning-models.md).
- `chat(..., response_format=...)` translates to Ollama's `format` parameter (`"json"` or a JSON schema).
- Image blocks are sent as native base64 `images` arrays for multimodal models such as `llava`.
- For [Ollama cloud](https://docs.ollama.com/cloud), set `base_url="https://ollama.com"` and export `OLLAMA_API_KEY` (or pass `api_key=`); the adapter sends the `Authorization: Bearer` header. A `404` from any server raises an error that names the missing model and the exact `ollama pull` command, and mid-stream server failures surface as terminal `error` stream events instead of a silently truncated answer.

## Resilience is on by default

Every adapter from `create_adapter` (and every agent run) is wrapped in a `ResilientAdapter`: a concurrency bulkhead, optional request pacing, automatic retries with exponential backoff and full jitter, a circuit breaker, and a per-call deadline. You get the default concurrency bound without configuring anything.
Expand Down
10 changes: 7 additions & 3 deletions site_docs/guides/reasoning-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ options = RunnerOptions(
| `openai` | `reasoning_effort` only | `reasoning_effort` + `max_completion_tokens` on reasoning models | Non-reasoning models drop the effort and use `max_tokens` + `temperature` |
| `azure` | `reasoning_effort` only | Same as `openai` | Same as `openai` |
| `bedrock` | — | Ignored (thinking blocks in responses are still parsed) | Silently ignored |
| `ollama` | — | Ignored | Silently ignored |
| `ollama` | both knobs | `reasoning_effort` → `think` level (`minimal`→`low`); a token budget enables `think: true` | Non-thinking models return no `thinking` field |
| `google` | — | Ignored | Silently ignored |

### Anthropic: budget resolution
Expand Down Expand Up @@ -82,8 +82,12 @@ async for event in adapter.stream(messages, options):
... # answer text delta
```

!!! note "Streaming thinking is Anthropic-only today"
Bedrock parses thinking blocks in non-streaming responses but does not emit `thinking` stream events, and the OpenAI-family adapters emit none at all. Don't build UI that requires live thinking deltas on those providers.
!!! note "Streaming thinking: Anthropic and Ollama"
Anthropic and Ollama emit live `thinking` stream events. Bedrock parses thinking blocks in non-streaming responses but does not emit `thinking` stream events, and the OpenAI-family adapters emit none at all. Don't build UI that requires live thinking deltas on those providers.

### Ollama: think levels

The Ollama adapter maps `reasoning_effort` to the native `think` parameter (`minimal` and `low` → `"low"`, `medium` → `"medium"`, `high` → `"high"`); a `thinking_budget_tokens` request enables `think: true` since Ollama has no budget concept. You can also set a default on the adapter itself with `OllamaAdapter(think=...)`, including the `"max"` level that has no portable tier. Thinking text returns as a `ThinkingBlock` (no signature) and as `thinking` stream events on models such as `qwen3`, `gpt-oss`, and `deepseek-r1`.

See [`examples/32_reasoning_models.py`](https://github.com/Quantlix/anycode/blob/main/examples/32_reasoning_models.py) for a runnable walkthrough, including model classification and parameter shaping.

Expand Down
2 changes: 1 addition & 1 deletion src/anycode/providers/adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ async def _create_raw_adapter(
elif provider == PROVIDER_OLLAMA:
from anycode.providers.ollama import OllamaAdapter

return OllamaAdapter(base_url=base_url, model=model)
return OllamaAdapter(base_url=base_url, model=model, api_key=api_key)

elif provider == PROVIDER_BEDROCK:
from anycode.providers.bedrock import BedrockAdapter
Expand Down
188 changes: 153 additions & 35 deletions src/anycode/providers/ollama.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,18 @@
from __future__ import annotations

import json
import os
import uuid
from collections.abc import AsyncIterator
from typing import Any

from anycode.constants import OLLAMA_DEFAULT_BASE_URL, OLLAMA_REQUEST_TIMEOUT_S, STOP_REASON_END_TURN, STOP_REASON_TOOL_USE
from anycode.constants import (
OLLAMA_DEFAULT_BASE_URL,
OLLAMA_REQUEST_TIMEOUT_S,
STOP_REASON_END_TURN,
STOP_REASON_MAX_TOKENS,
STOP_REASON_TOOL_USE,
)
from anycode.providers._openai_compat import map_messages, map_tool_def, parse_json_safe
from anycode.security.redaction import safe_exception_message
from anycode.types import (
Expand All @@ -18,6 +25,7 @@
LLMStreamOptions,
StreamEvent,
TextBlock,
ThinkingBlock,
TokenUsage,
ToolUseBlock,
)
Expand All @@ -29,51 +37,156 @@

OLLAMA_REQUEST_TIMEOUT = OLLAMA_REQUEST_TIMEOUT_S

# Ollama's `think` levels; `minimal` has no native equivalent and maps to "low".
_EFFORT_TO_THINK = {"minimal": "low", "low": "low", "medium": "medium", "high": "high"}

_DONE_REASON_MAP = {"stop": STOP_REASON_END_TURN, "length": STOP_REASON_MAX_TOKENS}


def _map_done_reason(done_reason: str | None, *, has_tool_calls: bool) -> str:
if has_tool_calls:
return STOP_REASON_TOOL_USE
return _DONE_REASON_MAP.get(done_reason or "stop", STOP_REASON_END_TURN)


def _ensure_httpx() -> None:
if httpx is None:
raise ImportError('httpx is required for the Ollama provider. Install it with: pip install "anycode-py[ollama]"')


def _to_native_messages(oai_msgs: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Flatten OpenAI-style content-part lists into Ollama's native message shape.

The native ``/api/chat`` endpoint takes ``content`` as a plain string plus an
optional per-message ``images`` array of base64 payloads; it does not accept
``image_url`` parts.
"""
native: list[dict[str, Any]] = []
for msg in oai_msgs:
content = msg.get("content")
if not isinstance(content, list):
native.append(msg)
continue
texts: list[str] = []
images: list[str] = []
for part in content:
if part.get("type") == "text":
texts.append(part.get("text", ""))
elif part.get("type") == "image_url":
url = part.get("image_url", {}).get("url", "")
images.append(url.split(",", 1)[1] if "," in url else url)
flattened = {**msg, "content": "".join(texts)}
if images:
flattened["images"] = images
native.append(flattened)
return native


class OllamaAdapter:
"""HTTP-based adapter for Ollama — uses the OpenAI-compatible chat endpoint."""
"""HTTP-based adapter for Ollama's native ``/api/chat`` endpoint."""

def __init__(self, base_url: str | None = None, model: str | None = None) -> None:
def __init__(
self,
base_url: str | None = None,
model: str | None = None,
*,
api_key: str | None = None,
keep_alive: str | float | None = None,
think: bool | str | None = None,
default_options: dict[str, Any] | None = None,
) -> None:
_ensure_httpx()
self._base_url = (base_url or OLLAMA_DEFAULT_BASE_URL).rstrip("/")
self._default_model = model
self._api_key = api_key or os.environ.get("OLLAMA_API_KEY")
self._keep_alive = keep_alive
self._think = think
self._default_options = dict(default_options) if default_options else {}

def _headers(self) -> dict[str, str]:
# ollama.com cloud requires a Bearer key; local servers ignore the header.
if self._api_key:
return {"Authorization": f"Bearer {self._api_key}"}
return {}

def _raise_for_status(self, resp: Any, model: str | None) -> None:
try:
resp.raise_for_status()
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
raise httpx.HTTPStatusError(
f"Model '{model}' was not found on the Ollama server at {self._base_url}. Pull it first: ollama pull {model}",
request=e.request,
response=e.response,
) from e
raise

@property
def name(self) -> str:
return "ollama"

async def chat(
self,
messages: list[LLMMessage],
options: LLMChatOptions,
) -> LLMResponse:
model = self._default_model or options.model
oai_msgs = map_messages(messages, options.system_prompt)

def _build_payload(self, messages: list[LLMMessage], options: LLMChatOptions, *, stream: bool) -> dict[str, Any]:
payload: dict[str, Any] = {
"model": model,
"messages": oai_msgs,
"stream": False,
"model": self._default_model or options.model,
"messages": _to_native_messages(map_messages(messages, options.system_prompt)),
"stream": stream,
}

if options.tools:
payload["tools"] = [map_tool_def(t) for t in options.tools]
model_options: dict[str, Any] = dict(self._default_options)
if options.max_tokens is not None:
model_options["num_predict"] = options.max_tokens
if options.temperature is not None:
payload.setdefault("options", {})["temperature"] = options.temperature
model_options["temperature"] = options.temperature
if model_options:
payload["options"] = model_options
if self._keep_alive is not None:
payload["keep_alive"] = self._keep_alive
think = self._resolve_think(options)
if think is not None:
payload["think"] = think
return payload

def _resolve_think(self, options: LLMChatOptions) -> bool | str | None:
if options.reasoning_effort is not None:
return _EFFORT_TO_THINK[options.reasoning_effort]
if options.thinking_budget_tokens is not None:
# Ollama has no budget concept; any budget request enables thinking.
return True
return self._think

@staticmethod
def _to_native_format(response_format: dict[str, Any]) -> str | dict[str, Any]:
"""Translate an OpenAI-style response_format into Ollama's ``format`` value."""
kind = response_format.get("type")
if kind == "json_object":
return "json"
if kind == "json_schema":
return response_format.get("json_schema", {}).get("schema", {})
return response_format

async def chat(
self,
messages: list[LLMMessage],
options: LLMChatOptions,
*,
response_format: dict[str, Any] | None = None,
) -> LLMResponse:
payload = self._build_payload(messages, options, stream=False)
if response_format:
payload["format"] = self._to_native_format(response_format)
model = payload["model"]

async with httpx.AsyncClient(timeout=OLLAMA_REQUEST_TIMEOUT) as client:
async with httpx.AsyncClient(timeout=OLLAMA_REQUEST_TIMEOUT, headers=self._headers()) as client:
resp = await client.post(f"{self._base_url}/api/chat", json=payload)
resp.raise_for_status()
self._raise_for_status(resp, model)
data = resp.json()

content: list[ContentBlock] = []
msg_data = data.get("message", {})

if msg_data.get("thinking"):
content.append(ThinkingBlock(thinking=msg_data["thinking"]))
if msg_data.get("content"):
content.append(TextBlock(text=msg_data["content"]))

Expand All @@ -85,7 +198,7 @@ async def chat(
tool_id = f"call_{uuid.uuid4().hex[:24]}"
content.append(ToolUseBlock(id=tool_id, name=func.get("name", ""), input=args))

stop_reason = STOP_REASON_TOOL_USE if msg_data.get("tool_calls") else STOP_REASON_END_TURN
stop_reason = _map_done_reason(data.get("done_reason"), has_tool_calls=bool(msg_data.get("tool_calls")))

input_tokens = data.get("prompt_eval_count", 0)
output_tokens = data.get("eval_count", 0)
Expand All @@ -99,29 +212,20 @@ async def chat(
)

async def stream(self, messages: list[LLMMessage], options: LLMStreamOptions) -> AsyncIterator[StreamEvent]:
model = self._default_model or options.model
oai_msgs = map_messages(messages, options.system_prompt)

payload: dict[str, Any] = {
"model": model,
"messages": oai_msgs,
"stream": True,
}

if options.tools:
payload["tools"] = [map_tool_def(t) for t in options.tools]
if options.temperature is not None:
payload.setdefault("options", {})["temperature"] = options.temperature
payload = self._build_payload(messages, options, stream=True)
model = payload["model"]

full_text = ""
full_thinking = ""
tool_blocks: list[ToolUseBlock] = []
input_tokens = 0
output_tokens = 0
done_reason: str | None = None

try:
async with httpx.AsyncClient(timeout=OLLAMA_REQUEST_TIMEOUT) as client:
async with httpx.AsyncClient(timeout=OLLAMA_REQUEST_TIMEOUT, headers=self._headers()) as client:
async with client.stream("POST", f"{self._base_url}/api/chat", json=payload) as resp:
resp.raise_for_status()
self._raise_for_status(resp, model)
async for line in resp.aiter_lines():
if not line.strip():
continue
Expand All @@ -130,12 +234,24 @@ async def stream(self, messages: list[LLMMessage], options: LLMStreamOptions) ->
except json.JSONDecodeError:
continue

# Ollama reports mid-stream failures as NDJSON error objects
# on an already-established 200 response.
if chunk.get("error"):
yield StreamEvent(type="error", data=str(chunk["error"]))
return

if chunk.get("done_reason"):
done_reason = chunk["done_reason"]
if chunk.get("prompt_eval_count"):
input_tokens = chunk["prompt_eval_count"]
if chunk.get("eval_count"):
output_tokens = chunk["eval_count"]

msg_data = chunk.get("message", {})
if msg_data.get("thinking"):
thinking_text = msg_data["thinking"]
full_thinking += thinking_text
yield StreamEvent(type="thinking", data=thinking_text)
if msg_data.get("content"):
text = msg_data["content"]
full_text += text
Expand All @@ -152,11 +268,13 @@ async def stream(self, messages: list[LLMMessage], options: LLMStreamOptions) ->
yield StreamEvent(type="tool_use", data=block)

done_content: list[ContentBlock] = []
if full_thinking:
done_content.append(ThinkingBlock(thinking=full_thinking))
if full_text:
done_content.append(TextBlock(text=full_text))
done_content.extend(tool_blocks)

stop = STOP_REASON_TOOL_USE if tool_blocks else STOP_REASON_END_TURN
stop = _map_done_reason(done_reason, has_tool_calls=bool(tool_blocks))

yield StreamEvent(
type="done",
Expand Down
Loading