Skip to content

PR-J: inference-server catalog — which local server runs this model (llama.cpp/Ollama/vLLM…) + links + wizard filter - #29

Merged
andyholst merged 1 commit into
mainfrom
feat/inference-servers
Aug 10, 2026
Merged

andyholst merged 1 commit into
mainfrom
feat/inference-servers

Conversation

@andyholst

Copy link
Copy Markdown
Owner

What

Issue #27 — inference-server support: depending on the model you choose, the site now tells you WHICH local inference server can run it on YOUR hardware — with links — and the wizard can filter by server.

  • servers.py: curated catalog — llama.cpp, Ollama, vLLM, LM Studio, koboldcpp, TGI — each with URL, model format (GGUF / safetensors) and a per-hardware backend matrix (CUDA / ROCm / Metal / SYCL+Vulkan / CPU; verified against llama.cpp README, docs.ollama.com/gpu, docs.vllm.ai)
  • Contract v4: servers[] per model, computed by the crawler/samples from quant names (GGUF → llama.cpp/Ollama/LM Studio/koboldcpp; safetensors → vLLM/TGI); carried into the index via SUMMARY_FIELDS so the wizard can filter
  • Details pane: "Run locally with ↗" chips, each linking to the server's site
  • Wizard: new "inference server" select — picks are restricted to models the server supports and whose backend covers the chosen hardware (e.g. vLLM on a Mac → honest "No models fit")
  • Tests both sides: pytest 121 (catalog completeness, backend matrix sanity — vLLM has no Vulkan/NPU path — GGUF vs safetensors computation) · jsdom 19/19 (chips + links; Ollama keeps GGUF picks; vLLM no-match guidance) · Playwright 7/7 · 7× openspec valid

The rebuilt gate caught a real bug before shipping: the index summary was missing servers, which would have made the wizard filter return nothing — fixed in SUMMARY_FIELDS.

Closes #27

)

- servers.py: curated catalog (llama.cpp/Ollama/vLLM/LM Studio/koboldcpp/TGI) with
  URLs, formats, per-hardware backend matrix (verified: llama.cpp README,
  docs.ollama.com/gpu, docs.vllm.ai)
- servers_for_model(): GGUF-style quant names -> GGUF servers; safetensors -> vLLM/TGI
- schema v4: servers[] required; samples + crawler emit it; SUMMARY_FIELDS carries
  it into the index (wizard filter needs it)
- frontend: 'Run locally with' chips with links; wizard server select with
  backend-aware filtering (vLLM -> none of the GGUF samples -> honest no-match)
- tests both sides: pytest 121 (catalog/matrix/computation), jsdom 19/19
  (chips+links, Ollama keeps GGUF picks, vLLM no-match), Playwright 7/7
Closes #27
@andyholst
andyholst merged commit 841cadc into main Aug 10, 2026
1 check passed
@andyholst
andyholst deleted the feat/inference-servers branch August 10, 2026 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants