Document openai/anthropic/openrouter/ollama explicitly and default to local ollama

[johncarpenter]

First, a correction to the framing

There is no provider list in the code to expand. kgmd/llm.py passes llm.model straight to
litellm.completion, and litellm routes on the model-id prefix. So all four providers already work
today with no code change:

llm:
  model: openai/gpt-4o-mini          # works now
  model: anthropic/claude-sonnet-4-5 # works now
  model: openrouter/openai/gpt-4o    # works now — the current default
  model: ollama/llama3.1:8b          # works now, if Ollama is on localhost:11434

docs/reference/configuration.md:81 already documents the accepted value as "any litellm-routable
model id". Adding a provider registry, an enum, or a provider: config key would duplicate litellm's
routing table and immediately rot.

So this issue is not integration work. It is three separate things that actually block the four
providers named above from being usable and documented — and one product decision.

1. Ollama cannot be pointed anywhere but the default socket

call_structured builds its kwargs without api_base (kgmd/llm.py:42-48), and neither does
induce.py's direct call (:108-116). litellm needs api_base (or OLLAMA_API_BASE in the
environment) to reach an Ollama on another host or port. A user with Ollama on a workstation, a
homelab box, or a non-default port has no configuration path at all.

This is the one genuine code gap. It wants a new llm.api_base key, defaulting to unset, threaded
through both call sites. Unset must mean "don't pass it", so hosted providers are unaffected.

2. response_format is sent unconditionally, and the fallback is dead code

# kgmd/llm.py:50-54
# Attempt response_format; fall back gracefully
try:
    kwargs["response_format"] = {"type": "json_object"}
except Exception:
    pass

Assigning a dict literal to a dict key cannot raise, so the except is unreachable and the comment
is false — there is no graceful fallback. Every provider gets response_format whether it accepts it
or not, and a provider that rejects it fails the call rather than degrading.

This matters most for Ollama, where JSON-mode support varies by model. The fix is a real fallback:
catch litellm's BadRequestError on the first attempt and retry once without response_format,
relying on the existing _strip_code_fences + retry-with-corrective-message path that already exists
for exactly this class of failure.

3. Schema induction is the stage most likely to break on a local model

kgmd/induce.py bypasses call_structured and parses YAML rather than JSON — the recorded
constitutional deviation. It therefore gets no JSON mode, no schema validation, and no parse-failure
retry. Freeform YAML is precisely what small local models are worst at.

If the default becomes a 7B-class local model, induction is where the first bug report comes from.
Converging induce.py onto call_structured (or giving it its own retry and validation) should be in
scope here, because this issue is what will expose it.

4. The default is hardcoded in four places

kgmd/config.py:19    "model": "openrouter/anthropic/claude-sonnet-4-5",
kgmd/extract.py:85   llm_cfg.get("model", "openrouter/anthropic/claude-sonnet-4-5")
kgmd/resolve.py:32   llm_cfg.get("model", "openrouter/anthropic/claude-sonnet-4-5")
kgmd/induce.py:86    llm_cfg.get("model", "openrouter/anthropic/claude-sonnet-4-5")

Changing DEFAULT_CONFIG alone leaves three stale copies that still name OpenRouter. The
constitution already prohibits restating a default at the call site, and the llm.max_tokens split
(16384 in config.py, 4096 in extract.py and llm.py) is recorded debt on the same rule. This
issue touches all four modules, so the "fix it or re-justify it when next touched" obligation applies:
the call sites should read the default from DEFAULT_CONFIG rather than repeating a literal.

Proposal

• Add llm.api_base, default unset, consumed by both kgmd/llm.py and kgmd/induce.py. Unset means
  the key is not passed.
• Give response_format a real fallback, and delete the dead try/except.
• Change the llm.model default to an Ollama model, and remove the three duplicated literals.
• Document all four providers explicitly with a worked example each, including the environment
  variable each expects (OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY, none for
  Ollama).
• Keep litellm as the only routing layer. No provider enum, no registry, no provider: key.

Why local-by-default is coherent with the project

The constitution already says embeddings default to local fastembed "so the tool remains usable
with no embedding credentials". An Ollama default extends that to the whole pipeline: pip install kgmd && kgmd init && kgmd build would produce a graph with no account, no key, and no spend. That is
a materially better first run, and it removes the awkwardness of a tool that refuses to demonstrate
itself until you have a credit card.

It also makes the existing spend documentation honest by default rather than by exception.

Risk, and the decision to confirm

Extraction quality will drop, and CI cannot tell you by how much. The test suite mocks
litellm.completion and never makes a provider call, so every test passes identically regardless of
which model is the default. A quality regression here is invisible to the gate — this is the one
change in the project so far whose main risk is untestable by the existing suite.

Relation extraction and cluster verification are the sensitive stages; a 7B model typically produces
fewer relations, looser typing, and more parse retries than claude-sonnet-4-5. The tests/fixtures
alias variants ("Sarah Chen" / "Dr. Chen" / "S. Chen") exist because resolution quality is
load-bearing.

So the decision to confirm before implementing: is the goal "works with no credential out of the
box" (Ollama default, accept lower quality, document it) or "good output out of the box" (keep a
hosted default, ship first-class Ollama support and a documented local recipe)? The proposal above
assumes the former. Both are defensible; the second is the conservative option and can be reversed
into the first later, whereas a default that produces disappointing graphs is a first-impression
problem.

If Ollama becomes the default, the default model id must be one that (a) exists as a plain
ollama pull target and (b) is reliable at JSON — a candidate needs picking and testing, not
guessing. ollama/llama3.1:8b and ollama/qwen2.5:7b are the obvious ones to compare.

Note also that existing corpora are unaffected either way: kgmd init writes DEFAULT_CONFIG
into .kgmd/config.yaml, so the model is pinned per corpus at creation. Only newly initialized
corpora would pick up a new default.

Touchpoints

• kgmd/config.py — DEFAULT_CONFIG: new llm.api_base, changed llm.model.
• kgmd/llm.py — thread api_base; real response_format fallback; delete the dead try/except.
• kgmd/induce.py — thread api_base; ideally converge onto call_structured.
• kgmd/extract.py, kgmd/resolve.py — stop restating the model default.
• tests/ — llm.api_base present and absent (assert the key is not passed when unset); the
  response_format fallback path via a mocked BadRequestError. Still no network, still mocked at
  the litellm.completion seam.

Acceptance criteria

• llm.api_base is in DEFAULT_CONFIG, consumed by every litellm call site, documented, and
  omitted from the request when unset — asserted by a test, since silently sending
  api_base=None to a hosted provider is the failure mode.
• A provider that rejects response_format still completes via fallback, proven with a mocked
  BadRequestError. The dead try/except is gone.
• The model default appears exactly once in the codebase.
• All four providers have a documented worked example naming the credential each needs.
• kgmd init && kgmd build on a corpus of two notes succeeds against a local Ollama with no
  credential in the environment — verified by hand, since CI cannot.
• A recorded quality comparison on tests/fixtures/ between the current default and the proposed
  one: entity count, relation count, and parse-retry count for each. Not a pass/fail gate, but the
  number goes in the PR so the tradeoff is explicit rather than assumed.
• llm.max_tokens divergence reconciled or re-justified, since config.py, extract.py, and
  llm.py are all touched.
• No new runtime dependency. litellm already handles all four providers.

Documentation is part of this change

tests/test_docs.py compares the configuration page against DEFAULT_CONFIG in both directions, so
a new key fails the suite until documented. The key count is also written out in prose — "Nineteen
keys" and "all nineteen keys" in docs/reference/configuration.md — and becomes twenty.

Roughly fifteen places hardcode the current default or OPENROUTER_API_KEY and need revisiting:

• docs/install.md — requirements list ("An LLM provider credential"), the credential section, and
  the "network access" requirement, which stops being true for a local default.
• docs/quickstart.md — the prerequisite export, and the "costs money" warning, which needs a
  local-default path that does not.
• docs/reference/configuration.md — the llm.model row, the precedence example, and the full
  example block.
• docs/guides/troubleshooting.md — the litellm.AuthenticationError entry assumes a hosted
  provider; a local default needs a "cannot reach Ollama" entry instead. **Symptom**: lines must
  keep exactly one code span that appears verbatim in kgmd/**/*.py.
• docs/guides/maintenance.md — the spend table and the two build.log samples that show
  model=openrouter/anthropic/claude-sonnet-4-5.
• docs/reference/cli.md — the "Needs a provider credential" grouping.
• README.md — requirements and the Quickstart export.
• docs/examples/personal-notes.md, docs/examples/mcp-assistant.md — prerequisites.

Notes

Line references are against main after #3. Related: #4 covers stale state in the destructive paths
and is independent of this.