Skip to content

models.md

Latest commit

 

History

History
78 lines (62 loc) · 4.31 KB

models.md

File metadata and controls

78 lines (62 loc) · 4.31 KB
Raw

Supported Models

Higgs detects local model support from config.json model_type. The tables below are representative rather than exhaustive.

Supported Architectures

Architecture model_type Examples
LLaMA llama Llama 3 and CodeLlama
Mistral mistral Mistral 7B
Qwen2 qwen2 Qwen2 and Qwen2.5
Qwen3 qwen3 Qwen3
Qwen3.5+ (dense) qwen3_5, qwen3_5_text Qwen3.5 dense checkpoints; Qwen3.8-27B
Qwen3.5+ (MoE) qwen3_5_moe, qwen3_5_text_moe Qwen3.5-35B-A3B, Qwen3.6-35B-A3B
Qwen3-Next qwen3_next Qwen3-Coder hybrid checkpoints
Qwen3-MoE qwen3_moe Qwen3-30B-A3B
Gemma 2 gemma2 Gemma 2 2B, 9B, and 27B
Gemma 3 gemma3, gemma3_text Gemma 3 1B, 4B, 12B, and 27B
Gemma 4 gemma4, gemma4_text, gemma4_unified Gemma 4 E2B, E4B (edge); 12B, 31B; 26B-A4B (MoE)
Phi-3 phi3 Phi-3 Mini, Small, and Medium
Starcoder2 starcoder2 Starcoder2 3B, 7B, and 15B
DeepSeek-V2 deepseek_v2 DeepSeek-V2-Lite
LLaVA-Qwen2 llava-qwen2 nanoLLaVA-1.5

Gemma 3 / Gemma 4 notes

  • These are text-language-model implementations. Multimodal checkpoints (gemma3, gemma4) load fine — their vision/audio tower weights are skipped and only the text model runs. The text weights may be nested under language_model. in such checkpoints; Higgs strips that prefix automatically.
  • Gemma 4 E2B/E4B (per-layer-input embeddings + cross-layer KV sharing) and dense text variants are supported. The MoE variant (gemma4 with enable_moe_block, e.g. 26B-A4B) is supported only with unquantized expert weights; a checkpoint with quantized experts is rejected at load with a clear error rather than producing incorrect output.

Continuous Batching Support

batch=true enables true batched decode only for these model_type values:

  • llama
  • mistral
  • qwen2
  • qwen3

Other supported architectures still serve normally in simple mode, but Higgs now rejects batch=true during config load, doctor, and server startup.

Representative Working MLX Model IDs

Family Example model IDs
LLaMA mlx-community/Llama-3.2-1B-Instruct-4bit
Qwen2.5 mlx-community/Qwen2.5-3B-Instruct-4bit
Qwen3 mlx-community/Qwen3-1.7B-4bit
Qwen3-Next mlx-community/Qwen3-Coder-Next-4bit
Qwen3.5 dense mlx-community/Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-4bit
Qwen3.5 MoE NexVeridian/Qwen3.5-35B-A3B-3bit
Qwen3.6 MoE mlx-community/Qwen3.6-35B-A3B-4bit
Qwen3.8 dense mlx-community/Qwen3.8-27B-4bit
DeepSeek-V2 mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx

Qwen 3.5+ Adapter and Version Notes

  • Qwen 3.5, 3.6, and 3.8 dense and MoE checkpoints use the Qwen 3.5 adapters. This includes *ForConditionalGeneration wrapper configs: Higgs detects the top-level wrapper and consumes the nested text_config used by the text loader.
  • _text aliases such as qwen3_5_text and qwen3_5_text_moe resolve to the corresponding dense or MoE adapter.
  • Unknown newer versions within a supported family can use the nearest adapter only after the resolved config passes structural validation. Higgs logs an untested-version warning when it takes this tolerant path. A missing or invalid required field is rejected by name; an unknown family is rejected with the supported family/version list.
  • mlx-community/Qwen3.8-27B-4bit (27B dense, 4-bit) is verified working through its top-level qwen3_5 wrapper and nested qwen3_5_text config.
  • The cached-model smoke matrix covered mlx-community/Qwen3.6-35B-A3B-4bit plus mlx-community/Llama-3.2-1B-Instruct-4bit, mlx-community/Qwen2.5-3B-Instruct-4bit, mlx-community/Qwen3-1.7B-4bit, and mlx-community/Qwen3-Coder-Next-4bit.
  • OpenAI-style chat requests use non-thinking mode by default for Qwen3.6 unless the request explicitly opts into reasoning.

Model Input Requirements

  • Local models can be referenced by Hugging Face model ID or local path.
  • The model must be in MLX safetensors format.
  • The checkpoint must use a supported config.json model_type.
  • macOS local serving requires mlx.metallib next to the executable. Release artifacts bundle it, and source builds restore it from Cargo build output when possible.

For configuration details, see configuration.md.