sysllm

System LLM compatibility profiler. Detects your CPU, RAM, and GPUs, then tells you which LLMs — out of the top ~1000 text-generation models on HuggingFace — will run at a given quantization and context length.

---

Install

Requires Python 3.9+. Clone the repo and install in a venv:

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Optional extras:

pip install -e ".[nvidia]"   # for NVIDIA GPU detection via pynvml
pip install -e ".[dev]"      # pytest + ruff for development

---

Quick start

sysllm                        # profile the system and list runnable models
sysllm --only-runnable        # hide the "won't fit" rows
sysllm profile                # just dump hardware info
sysllm update                 # refresh the model DB from HuggingFace

Default filters: context 32,768 tokens, quant Q4_K_M (4-bit).

---

Usage

Profile only

sysllm profile

CPU
┌───────┬──────────────────────────┐
│ arch  │ aarch64                  │
│ brand │ Apple M5 Max             │
│ cores │ 18 physical / 18 logical │
│ simd  │ asimd, neon              │
└───────┴──────────────────────────┘
Memory
┌───────────┬──────────┐
│ total     │ 128.0 GB │
│ available │ 88.8 GB  │
│ unified   │ yes      │
└───────────┴──────────┘
GPUs
┏━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ vendor ┃ name         ┃    VRAM ┃ compute ┃ unified ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ apple  │ Apple M5 Max │ 96.0 GB │ -       │ yes     │
└────────┴──────────────┴─────────┴─────────┴─────────┘

Add --json for machine-readable output.

Match models against your system

sysllm

Shows every known model with a ✓/✗ for whether it fits at your chosen context and quant.

Higher quality quantization

Use --quant to raise the quality (which reduces how many models fit). Valid values from highest to lowest: FP16, Q8_0, Q5_K_M, Q4_K_M (default), Q3_K_S.

sysllm --quant FP16       # require full-precision weights
sysllm --quant Q8_0       # require 8-bit

Longer context

Use --context in tokens.

sysllm --context 65536    # 64k
sysllm --context 131072   # 128k — drops models whose context_max is smaller

Runtime filter

Models differ in which runtimes can load them. Filter to a specific runtime:

sysllm --runtime llama.cpp     # permissive; GGUF-convertible models
sysllm --runtime vllm          # CUDA-only, restricted to vLLM-supported families
sysllm --runtime transformers  # HuggingFace transformers

Multi-GPU

Only valid under vLLM (tensor parallelism). Sums VRAM across detected GPUs:

sysllm --runtime vllm --multi-gpu --quant FP16

Partial offload (llama.cpp)

Splits model layers between GPU VRAM and CPU RAM. Useful when the model doesn't fit on a single GPU:

sysllm --partial-offload --quant Q4_K_M

The Device column then shows e.g. nvidia:RTX 4090 + cpu and a layers_on_gpu figure in the JSON output.

Output formats

sysllm --json                 # full JSON (profile + query + results[])
sysllm --markdown             # GitHub-flavored table, for pasting into docs
sysllm                        # default: rich ANSI table

Refreshing the model database

sysllm update                 # fetch top 1000 by download count
sysllm update --limit 500     # fewer models
HF_TOKEN=hf_xxx sysllm update # higher rate limit, fewer 429s

Writes to ~/.cache/sysllm/models.json. Subsequent sysllm runs prefer the cache over the bundled seed list. Delete the file to revert to the bundled seed.

---

How it decides

For each model, sysllm computes:

weights_gb   = params_b * quant.bytes_per_param
kv_cache_gb  = 2 * n_layers * n_kv_heads * head_dim * ctx * 2 bytes / 1024³
                 (falls back to 0.125 GB per 1k ctx if layer metadata is missing)
total_gb     = weights_gb + kv_cache_gb + runtime_overhead_gb

Then, in order:

1. Arch check — if the model's whitelist excludes your CPU arch, skip.
2. Runtime check — if --runtime X is set and the model doesn't list X, skip.
3. Context check — if the requested context exceeds the model's context_max, skip.
4. Quant check — if the requested quant name isn't in the model's quant list, skip.
5. Memory check, in this order:
   • Unified memory (Apple Silicon): total_gb ≤ available RAM → apple-gpu.
   • Any discrete GPU with vram_gb ≥ total_gb → full offload.
   • --multi-gpu + --runtime vllm: sum of VRAM ≥ total_gb → multi-gpu.
   • --partial-offload: split layers GPU/CPU → gpu+cpu.
   • CPU fallback: available RAM ≥ total_gb + 2 GB OS headroom → cpu.

The reason for each ✓ or ✗ is shown in the rightmost column.

---

Platforms

Platform	CPU	RAM	GPU detection
Linux x86_64	✓	✓	NVIDIA (pynvml), AMD (rocm-smi), Intel (xpu-smi → lspci)
Linux aarch64	✓	✓	NVIDIA (pynvml), AMD (rocm-smi)
macOS arm64 (Apple Silicon)	✓	✓ (unified)	Apple GPU via system_profiler
macOS x86_64 (Intel)	✓	✓	Intel/AMD via system_profiler; no NVIDIA

All GPU probes fail silently — an unknown vendor just means no GPUs are reported for that vendor.

---

Development

pip install -e ".[dev]"
pytest -v                     # 54 tests
ruff check sysllm tests

The test suite uses fixture profiles for non-local hardware (8GB laptop, RTX 4090, dual A100, Graviton, Raspberry Pi 5, M3 Pro) so multi-GPU and partial-offload paths are covered even on machines without discrete GPUs.

CI (see .github/workflows/ci.yml) runs on Linux x86_64 and aarch64 plus macOS arm64, on Python 3.10, 3.11, and 3.12.

---

Known limitations

• KV-cache math assumes FP16 K and V. Runtimes that quantize the KV cache (llama.cpp -ctk q8_0) will consume less, so sysllm is conservative — if it says a model fits, it will; if it says no, the actual runtime might still succeed.
• The Apple Silicon unified-memory GPU budget is approximated at 75% of system RAM (the default iogpu.wired_limit_mb). Users who raise the sysctl aren't auto-detected.
• Model param counts for HF entries are inferred from the model ID (e.g. "Llama-3.1-8B" → 8B) or from a SwiGLU+GQA-aware config heuristic. Both are within ~5% for mainstream architectures; MoE and unusual layouts may be off.
• Quant availability is assumed to be the standard GGUF set (FP16, Q8_0, Q5_K_M, Q4_K_M, Q3_K_S) for every model — true in practice because llama.cpp can quantize any supported arch on the fly.