Skip to content

models.md

Latest commit

 

History

History
40 lines (33 loc) · 2.91 KB

models.md

File metadata and controls

40 lines (33 loc) · 2.91 KB
Raw

Supported models

FreeToken loads HF safetensors checkpoints directly (plus native GGUF for Gemma-4). The checkpoints below are known-good — the prebuilt kernels are tuned for them; other checkpoints of the same architectures work too.

Model HF checkpoints
DeepSeek-V4 deepseek-ai/DeepSeek-V4-Flash-0731
GLM-5.2 nvidia/GLM-5.2-NVFP4
GLM-4.7 nvidia/GLM-4.7-NVFP4
Qwen3.6 / Qwen3.5 MoE Qwen/Qwen3.6-35B-A3B (-FP8), nvidia/Qwen3.6-35B-A3B-NVFP4, Qwen/Qwen3.5-35B-A3B (-FP8)
Qwen3.6 dense Qwen/Qwen3.6-27B (-FP8), nvidia/Qwen3.6-27B-NVFP4
Qwen3-MoE Qwen/Qwen3-30B-A3B
gpt-oss openai/gpt-oss-120b, openai/gpt-oss-20b
Gemma-4 google/gemma-4-26B-A4B-it, nvidia/Gemma-4-26B-A4B-NVFP4, google/gemma-4-12B-it, nvidia/Gemma-4-31B-IT-NVFP4 ..
MiniMax-M2.5 nvidia/MiniMax-M2.5-NVFP4
Muse-Glimmer meta-models/Muse-Glimmer-30B, RedHatAI/Muse-Glimmer-30B-NVFP4

MoE backends

ft serve --moe-backend {auto,fused,offload,cpu,hybrid}:

  • fused — experts resident on GPU (needs the VRAM); never auto-selected.
  • offload — experts live in host RAM, an LRU cache of expert slots on GPU; misses stream over PCIe.
  • cpu — misses are computed on the CPU instead of fetched.
  • hybrid — per step, fetches some misses over PCIe and computes the rest on CPU, overlapped. Run ft bench bw once per machine to calibrate the split.
  • auto — dense models always resolve to fused; MoE models resolve to offload, upgraded to hybrid when a cached ft bench bw profile recommends it.

Notes

  • ft checkpoint conversion is optional — it pre-converts a checkpoint into FreeToken's fast-load format, and ft serve --model auto-detects the result.
  • DeepSeek-V4 checkpoints must keep the inference/config.json subdir — the authoritative model args are read from there.
  • Multimodal checkpoints are served text-only.