Skip to content

PR-K: recommend the inference server per model × hardware (MLX/vLLM/TRT-LLM + reasons + badges) - #31

Merged
andyholst merged 1 commit into
mainfrom
feat/recommend-server
Aug 10, 2026
Merged

andyholst merged 1 commit into
mainfrom
feat/recommend-server

Conversation

@andyholst

Copy link
Copy Markdown
Owner

What

Issue #30 — "I should be SUGGESTED which inference server I can use, based on the LLM model I choose AND my hardware." The flat 4-server list is replaced by a real recommendation engine.

Verified research (2 workers, 2026-08-10, URL-cited):

  • vLLM + GGUF = "highly experimental", out-of-tree plugin, ~8× slower than Marlin (docs.vllm.ai)
  • vLLM officially supports AMD Radeon RDNA3/RDNA4 (ROCm first-class since Jan 2026); TGI + SGLang = Instinct-only, no consumer AMD
  • MLX = Apple-native, safetensors/MLX format only (cannot load GGUF directly); TensorRT-LLM = NVIDIA-only; gpt4all ~2k context (hobby)

Shipped:

  • servers.py: catalog expanded 6 → 14 (llama-server, Jan, oobabooga, llamafile, gpt4all, MLX, SGLang, TensorRT-LLM) with tier (prod/desktop/hobby) + per-hardware backend matrix + reasons; recommend_servers(quants, category) → ranked {name, tier, backend, reason} — format × backend × rank policy (MLX > Ollama > LM Studio > rest; vLLM listed as experimental alternative for GGUF on CUDA/ROCm)
  • Details pane: "★ Recommended for NVIDIA GPUs — CUDA: Ollama (easiest install)…" + backend badges on every chip + a category picker (switch to Mac → Metal/MLX view)
  • Wizard rows: each pick shows → Server (backend) (reason) suggestion
  • Tests both sides: pytest 125 (9 recommendation tests: NVIDIA/GGUF → Ollama, Mac/safetensors → MLX, AMD excludes TGI/SGLang, Arc/Snapdragon), jsdom 21/21 (recommendation line + CUDA/Metal badges + category switch + wizard suggestions), Playwright 7/7, 8× openspec valid

Closes #30

Issue #30: 'I should be suggested which inference server I can use, based on
the LLM model I choose AND my hardware.'

Verified research (2 workers, URL-cited): vLLM+GGUF = experimental out-of-tree
plugin (~8x slower than Marlin); vLLM officially supports AMD RDNA3/4 (ROCm
first-class Jan 2026); TGI+SGLang = Instinct-only (no consumer AMD); MLX =
Apple-native, safetensors-only; TensorRT-LLM = NVIDIA-only; gpt4all ~2k ctx.

- servers.py: catalog expanded to 14 (llama-server, Jan, oobabooga, llamafile,
  gpt4all, MLX, SGLang, TensorRT-LLM) with tier (prod/desktop/hobby) + backend
  matrix + reason map; recommend_servers(quants, category) -> ranked
  {name, tier, backend, reason}: format x backend x rank policy
  (MLX > Ollama > LM Studio > rest; vLLM = experimental alt for GGUF on CUDA/ROCm)
- frontend: details pane shows '★ Recommended for <category>: X (reason)' with
  backend badges + a category picker; chips carry backend labels; wizard rows
  show '-> Server (backend) (reason)' suggestions
- tests both sides: pytest 125 (9 recommendation tests: NVIDIA/GGUF, Mac/MLX,
  AMD exclusions, Arc/Snapdragon), jsdom 21/21 (recommendation + badges +
  category switch + wizard suggestions), Playwright 7/7, 8x openspec valid
Closes #30
@andyholst
andyholst merged commit 624dd4c into main Aug 10, 2026
1 check passed
@andyholst
andyholst deleted the feat/recommend-server branch August 10, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Suggest the inference server per model × hardware (recommendation, not just a list) — MLX/vLLM/TRT-LLM/SGLang/long-tail + reasons + backend badges

2 participants