Gemma 4 support requires transformers >= 5.5.0 (currently pinned <= 4.57.6)

[2imi9]

Summary

Gemma 4 (model_type: gemma4) was introduced in transformers >= 5.5.0, but llm-compressor currently pins transformers>=4.56.1,<=4.57.6 in setup.py. This means Gemma 4 models cannot be loaded or quantized without manually overriding the transformers version.

Steps to reproduce

from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("google/gemma-4-E4B-it", dtype="auto")

With the pinned transformers version, this fails because gemma4 is not a recognized model type.

Current workaround

Install llm-compressor with --no-deps and force-install transformers from git main. A Dockerfile demonstrating this workaround is included in PR #2561.

Suggested fix

Bump the transformers upper bound in setup.py to include >= 5.5.0 once compatibility is verified across the codebase.

Related

• PR [Examples] Add Gemma 4 E4B NVFP4A16 quantization example #2561 — Gemma 4 NVFP4A16 quantization example (includes Dockerfile workaround)
• Model: google/gemma-4-E4B-it

[2imi9]

Additional note: NVIDIA's official quantized model (nvidia/Gemma-4-31B-IT-NVFP4) shows 21B params after quantization from 31B, indicating they quantize the full model including vision/audio components.

With llm-compressor, we have to skip vision_tower, audio_tower, embed_vision, and embed_audio because Gemma 4 uses custom layer types like Gemma4ClippableLinear in the audio tower, which don't inherit from standard nn.Linear. The QuantizationModifier with targets="Linear" doesn't match these layers.

Supporting these Gemma 4-specific layer types would allow full model quantization and better size reduction, matching what NVIDIA achieves with modelopt.

[brian-dellabetta]

Hi @2imi9 , your workaround in the example to just upgrade transformers>=5.5 after installing llm-compressor is a good quick fix. You can track the progress to remove the transformers<5 constraint from llm-compressor in this issue:

• [upstream] Expecting future huggingface/transformer incompatibility #2036

---

Additional note: NVIDIA's official quantized model (nvidia/Gemma-4-31B-IT-NVFP4) shows 21B params after quantization from 31B, indicating they quantize the full model including vision/audio components.

Regarding your note above, the nvidia checkpoint does not quantize vision and audio components. See safetensors metadata here, language_model modules are quantized with qparams like weight_scale , but the vision tower remains untouched. This is a general practice, as vision/audio towers are more sensitive to quantization notwithstanding the fact that they are using custom clippable linear ops instead. If you quantize audio/vision and try to run in vllm, it will fail.

[2imi9]

Thanks for the clarification! I misunderstood the param count difference. It's good to know the vision/audio towers are kept unquantized in NVIDIA's checkpoint too.