Qwen3.5-35B-A3B GPTQ INT4 ONNX export ERROR using tensorrt edge llm 0.9.0

[johnny-zjh]

Describe the bug

When exporting the Hugging Face checkpoint of Qwen3.5-35B-A3B GPTQ INT4 with TensorRT Edge-LLM 0.9.0, the generated ONNX graph appears to combine the shared-expert output and routed-expert output twice inside each MoE layer.

The expected MoE output should conceptually be computed once as:

moe_output = routed_expert_output + shared_expert_output

However, inspection of the exported ONNX graph shows two equivalent accumulation paths involving the shared-expert and routed-expert outputs. In other words, the operation that adds the shared-expert branch to the routed-expert branch appears to be inserted twice. The onnx graph is showed below:

No explicit Python exception is raised during export. The issue was identified by inspecting the generated ONNX graph.

Steps/Code to reproduce bug

1. Install TensorRT Edge-LLM 0.9.0.
2. Download the open-source Hugging Face checkpoint for Qwen3.5-35B-A3B GPTQ INT4.
3. Export the checkpoint to ONNX using tensorrt-edgellm-export-llm.
4. Open the exported ONNX model with Netron or another ONNX graph viewer.
5. Inspect the output aggregation section of an MoE decoder layer.

Installation method:

pip install .

Export command used:

tensorrt-edgellm-export-llm \
    --model_dir <DIR_OF_HUGGINGFACE_MODEL> \
    --output_dir <ONNX_OUTPUT_DIRECTORY> \

Expected behavior

Fix the onnx export bug~

The exported ONNX model should preserve the numerical behavior of the original Hugging Face model and should be usable for subsequent TensorRT engine building and inference.

System information (x86 Host with GPU)

• Container used (if applicable): No applicable?

• OS (e.g., Ubuntu 22.04, CentOS 7): Ubuntu 22.04 LTS

• CPU architecture: x86_64

• GPU name (e.g. H100, A100, RTX 4090): A100

• GPU memory size: 80G

• Number of GPUs: 8

• Library versions:

  • Python: 3.10.6
  • TensorRT Edge-LLM version or commit hash: 0.9.0
  • CUDA: 13.0
  • PyTorch: 2.12.0+cu130
  • Transformers: 5.9.0
  • ModelOpt: 0.44.0
  • ONNX: 1.19.0

• Any other details that may help: No~

  • Model: Qwen3.5-35B-A3B GPTQ INT4
  • Model source: Hugging Face open-source checkpoint
  • Export itself does not necessarily report an explicit exception.
  • The incorrect behavior was identified by inspecting the exported ONNX graph.
  • The generated ONNX model fails or behaves incorrectly in the subsequent build/inference pipeline.
  • ONNX graph viewer used: <NETRON_OR_OTHER_TOOL_VERSION>

[nvluxiaoz]

Thanks. We are actively working on this. Can you try if release/0.9.1 already fixes this? If not I have tested in our next release to fix it.

[johnny-zjh]

Thanks. We are actively working on this. Can you try if release/0.9.1 already fixes this? If not I have tested in our next release to fix it.

I followed the same process on version 0.9.1 and experienced the same problem. Can you reproduce this issue on your side？

[johnny-zjh]

After fixing the duplicate addition issue of shared_expert_output in the ONNX graph in 0.9.1, I exported the ONNX model and generated the TensorRT engine. However, when I use llm_inference for single-modal LLM inference, the generated output is abnormal and completely unrelated to the prompt.
Could you try to reproduce this issue on your end to see if you get the same results? This would help us determine if there's any error in my workflow.

Thanks. We are actively working on this. Can you try if release/0.9.1 already fixes this? If not I have tested in our next release to fix it.