PRD: Local model manager — browse, fit-filter, and pull open-weight models via Ollama

[briandilley]

Problem Statement

Setting up a local LLM in Gilbert today is expert-only. The ollama Backend exists, but to use
it you must already know Ollama, already know which model tags to ollama pull from the command
line, type a free-text tag into settings, and guess whether a model will even run on your machine.
There is no way, from inside Gilbert, to discover open-weight models, see which ones will
actually fit your hardware (RAM / VRAM / GPU), or install one without dropping to a terminal. And
because the ollama backend's generation settings (temperature, max_tokens, enabled models) are
global, you cannot tune them per model — a model that wants temperature 0.2 and one that wants
0.8 are forced to share.

Solution

A Local model manager: an in-app page that lists open-weight models from the Hugging Face Hub
(the Model catalog), sortable by Hugging Face's own signals (downloads / likes / trending /
recency), with a "Compatible" filter that narrows to models whose Hardware fit says they will
run on this host, and a Recommended overlay badge for vetted, tool-capable picks. Pulling a model
is one click; afterward it appears in the ollama backend's model list and is selectable in chat
(binding it to a Tier stays the existing, separate step). The whole thing is built on Ollama
as the runtime/server/quantizer — no bundled inference runtime — and Hugging Face stays the catalog
because Ollama pulls GGUF straight from HF. Generation settings become Per-model config, layered
so a model can carry sensible defaults while a profile or call can still override them.

See ADRs: core 0018 (enablement dependencies), 0019 (per-model config), 0020 (host resources);
plugins 0007 (build on Ollama; HF catalog), 0008 (enablement-aware runtime_dependencies).

User Stories

1. As an admin, I want to browse open-weight models from Hugging Face inside Gilbert, so that I don't have to research model names on the command line.
2. As an admin, I want the list sorted by Hugging Face popularity/recency by default, so that the models other people actually use surface first.
3. As an admin, I want to re-sort by downloads, likes, trending, recency, or size, so that I can scan the catalog the way that suits me.
4. As an admin, I want to type a search query, so that I can find a specific model family (e.g. "qwen", "llama").
5. As an admin, I want to filter the list to "Compatible" models, so that I only see models that will actually run on my host.
6. As an admin, I want each model to show a Hardware-fit verdict (fits-VRAM / fits-RAM / won't-fit / unknown), so that I understand why it is or isn't compatible and how fast it will be.
7. As an admin, I want each quantization of a model to be selectable with its own size and fit verdict, so that I can pick the largest quant that still fits.
8. As an admin, I want models vetted by Gilbert (tool-capable, known-good templates) badged as "Recommended", so that I can avoid models that pull fine but behave badly in the agent loop.
9. As an admin, I want to optionally filter to only Recommended models, so that I can stick to safe choices.
10. As an admin, I want to pull a model with one click, so that I don't have to run ollama pull myself.
11. As an admin, I want to see pull progress and completion, so that I know when the model is ready.
12. As an admin, I want a pulled model to immediately become selectable in chat, so that I can try it without extra configuration.
13. As an admin, I want to see which models I have already pulled (installed), so that I don't pull the same thing twice.
14. As an admin, I want to delete an installed model, so that I can reclaim disk space.
15. As an admin, I want to set per-model temperature, max_tokens, and enabled flags, so that each model runs with settings that suit it.
16. As an admin, I want a pulled model's per-model defaults seeded from Hugging Face / GGUF metadata (e.g. context window), so that I start from sensible values.
17. As an admin, I want a profile or an individual call to override a model's default temperature/max_tokens, so that the same model can be used for both precise and creative tasks.
18. As an admin, I want per-model config to apply to any AI backend (Anthropic, Groq, …), not just Ollama, so that the behavior is consistent everywhere.
19. As an admin, I want to disable a specific model without deleting it, so that it stops appearing in the chat model picker but stays installed.
20. As an admin, I want the manager to refuse to start (with a clear "disabled — requires Ollama backend" badge) when the Ollama backend isn't enabled, so that I'm not confused by a non-functional page.
21. As an admin, I want a toast when I try to enable the manager while Ollama is disabled, so that I immediately understand the prerequisite.
22. As an admin, I want Gilbert to not silently auto-enable the Ollama backend on my behalf, so that nothing starts daemons or polling I didn't choose.
23. As an admin running doctor, I want to be told how to install the Ollama daemon when the Ollama backend is enabled but the daemon is unreachable, so that I can fix it without searching.
24. As an admin who uses only Anthropic, I want doctor to not nag me about installing Ollama, so that the report stays trustworthy.
25. As an admin, I want the "Compatible" filter to reflect the host where Ollama runs, so that the verdict is accurate for my setup.
26. As an admin running Ollama on a remote box, I want fit to read "unknown" rather than a wrong answer, so that I'm not misled.
27. As an admin, I want the Hardware-fit estimate to account for context/KV-cache overhead, not just raw file size, so that a model marked "fits" actually loads.
28. As a developer of another local-compute backend (whisper, kokoro), I want a reusable host-resources capability, so that I can detect GPU/VRAM instead of asking the user to pick a device.
29. As an admin, I want the model picker in chat to reflect exactly the models I have installed and enabled, so that I never select a tag that isn't really there.
30. As an admin, I want the manager to drive pull/list/delete without me re-entering the Ollama URL, so that there's a single source of truth for where Ollama lives.

Implementation Decisions

The work is four deliverables in dependency order; the first three are reusable core changes.

1. HostResourcesProvider capability (core). A @runtime_checkable protocol in interfaces/
   plus a vendor-free integrations/ probe (using psutil for memory + best-effort GPU/VRAM
   detection via system tools; no heavy deps). Returns raw data (total/available RAM, GPU presence,
   per-GPU VRAM); localhost-only and best-effort, returning "unknown" when it can't tell. The
   runnability verdict is NOT in core (ADR-0020).

2. Enablement-dependency mechanism (core). A service/plugin can declare it needs a named
   Backend or Service enabled. When unmet, the dependent does not start and is surfaced
   as disabled, with the reason (a Settings badge + a toast on the toggle), never auto-enabled
   (ADR-0018). Exposes an "is backend/service X enabled?" query. This same query makes
   Plugin.runtime_dependencies() enablement-aware: doctor passes the resolved config it
   already loads into the hook, so a plugin returns a RuntimeDependency only when its backend is
   enabled (ADR-0008). doctor builds plugins without booting Gilbert, so config is passed
   explicitly; existing runtime_dependencies() overrides are updated to accept (and may ignore) it.

3. Per-model config (core). AIService owns per-(backend, model) settings — an enabled flag
   plus generation defaults (max_tokens, temperature, context window), seeded from HF/GGUF
   metadata at pull time — and exposes them via a capability so plugins read/write without touching
   each other's storage. Generation params resolve in layers: backend default ← per-model ←
   profile ← call. To carry resolved values, AIRequest (and optionally AIContextProfile) gain
   optional generation fields; a backend applies what it's handed and falls back to its own
   default when a field is unset (so existing backends keep working). The ollama backend's current
   global temperature/max_tokens become the backend-default layer, and its enabled_models
   array is subsumed by the per-model enabled flag (ADR-0019).

4. The manager plugin (separate std-plugin; depends on 1–3). A UIRoute SPA page under the
   plugin's frontend/ rendering the Model catalog: Hugging Face Hub API for breadth + relevance
   signals + per-quant GGUF sizes, enriched with the Gilbert Recommended overlay (successor to
   the ollama backend's static curated list). Shows all models; HF-native sort; "Compatible" is
   a filter derived from Hardware fit (the manager's policy: per-quant size × overhead factor
   vs HostResourcesProvider data → fits-VRAM / fits-RAM / won't-fit / unknown), not a ranking
   weight. Pull/list/delete (and the resolved base_url) go through a LocalModelRuntimeProvider
   capability that the ollama plugin implements — so the manager never reads the backend's config
   and a future runtime could replace Ollama unchanged. The manager declares an enablement
   dependency on the ollama backend and a runtime_dependencies() entry that exercises the
   Ollama daemon (GET /api/tags, not a path probe; auto_install_cmd empty → manual install hint,
   per ADR-0003).

   Accompanying ollama plugin/backend changes: make available_models() dynamic (reflect
   actually-installed tags via /api/tags, joined with the recommended overlay for names), implement
   LocalModelRuntimeProvider, declare the enablement-aware daemon RuntimeDependency, and consume
   per-model config instead of its globals.

Testing Decisions

Tests assert external behavior at the highest existing seam, never internal wiring. Confirmed
seams and prior art:

• Per-model config + layering → at the AIService seam. The existing StubAIBackend
  (test_ai_service.py) records every AIRequest; assert the resolved generation params (backend
  ← per-model ← profile ← call) arrive on the request, and that available_models() reflects
  per-model enabled.
• ollama backend (dynamic available_models(), LocalModelRuntimeProvider, per-model
  consumption) → at the backend seam with mocked httpx, the exact pattern in
  test_ollama_ai.py.
• HostResourcesProvider → unit/integration test the probe with mocked psutil/subprocess,
  asserting both the structured shape and the best-effort "unknown" path (prior art:
  test_local_whisper.py, tests/integration/test_local_*).
• Enablement-dependency mechanism → at the service_manager seam (unmet dep → service does
  not start, reason surfaced) and the runtime_dependencies(config) method seam (returns the
  Ollama daemon dep only when the backend is enabled; empty when disabled).
• Manager service (browse / pull / fit) → at the service-method seam with
  LocalModelRuntimeProvider + HostResourcesProvider faked; the Hardware-fit verdict is pure
  logic → a direct unit test across the fit tiers.

A good test here checks the request a backend receives, the verdict the fit policy returns, or
whether a service starts — not which function called which.

Out of Scope

• Bundling an inference runtime (llama.cpp / transformers / vLLM) and running models in-process
  (ADR-0007). Safetensors-only repos and exotic/high-throughput serving — those users are served
  by the existing openai-compatible plugin pointed at their own vLLM/LM Studio endpoint.
• Auto-enable cascade (enabling the manager enabling Ollama for you) — explicitly rejected
  (ADR-0018).
• Automated frontend tests — Gilbert has no frontend test harness; the UIRoute is verified
  manually, and all testable logic lives behind the Python service methods.
• Per-user model config — manager actions and per-model config are admin-global, consistent with
  multi-user isolation (core ADR-0009).
• Remote-host hardware probing — fit for a remote Ollama is "unknown", not estimated.
• Auto-installing the Ollama daemon — the install hint is manual (needs sudo/curl|sh),
  per ADR-0003.

Further Notes

• Building on Ollama does not forfeit Hugging Face: ollama pull hf.co/<repo>:<quant> makes HF
  the catalog while Ollama remains the runtime.
• HostResourcesProvider is independently useful: it lets whisper/kokoro's device=auto and
  future local-compute backends detect GPU/VRAM instead of asking the operator.
• During implementation, refresh the docs that will drift: the root + std-plugins CLAUDE.md
runtime_dependencies() example signature, and std-plugins/README.md (new manager plugin
  row/section + the ollama config changes), per the documentation-freshness rules.