[TASK] Add INT8 Quantization Support in the Frontend

[zhanghb97]

Deliverables

• A pull request (PR) enabling end-to-end DeepSeek inference with INT8 quantization

Task Description

• Run the DeepSeek R1 inference pipeline to become familiar with the workflow.
• Apply PyTorch INT8 quantization to DeepSeek R1 and evaluate inference performance on an AVX512-VNNI machine.
• Extend the buddy-mlir frontend to support PyTorch INT8-quantized models.
• After generating MLIR for the full model, build a complete end-to-end inference example based on the existing example.

Timeline

Phase	Time
Coding Phase	Oct 30, 2025 – Nov 6, 2025
Code Review	Begins Nov 7, 2025

If finished ahead of schedule, the review process may begin earlier.

[zhanghb97]

Assignee: @NiFFu

[NiFFu]

Progress Tracking:

1. Compiled DeepSeek-R1-Distill-Qwen-1.5B according to the workflow in the examples.
2. Applied INT8 quantization using PyTorch's built-in static quantization tool.
3. Adjusting frontend.py and import-deepseek-r1.py to meet the requirements for compiling INT8 models.

[zhanghb97]

@NiFFu Nice progress!
Can you now generate the quantized version of the MLIR file? Have you encountered any issues?

[NiFFu]

@zhanghb97
I successfully imported the quantized model into buddy-mlir using the default workflow and generated an executable file.
However, the inference time of the executable file generated from the quantized model was almost the same as that of the original model. In contrast, when I ran the models using a simple Python script, the quantized model was four to five times faster than the original model.
I think the model was reverted to higher precision (e.g. FP32) during the default workflow. So I'm trying to modify the processes.

[NiFFu]

@zhanghb97
While reviewing and preparing for the presentation, I discovered a serious problem with my previous work. So, I was unable to submit the submission on time.
I used PyTorch's eager mode to perform static quantization on the model. Although this accelerated the inference time, it didn't actually reduce the data precision correctly, and the dtype of output model was still f32. I had only focused on inference performance and hadn't checked the data type. So, I only discovered the problem by tracing back after finding that the generated .mlir data had f32 precision.
I think the next step will be experiment of converting the model into an FX graph before performing static quantization. If this method is also unreliable, we may need to consider buddy-mlir's support for dynamic quantization or onnx model.

[zhanghb97]

I used PyTorch's eager mode to perform static quantization on the model. Although this accelerated the inference time, it didn't actually reduce the data precision correctly, and the dtype of output model was still f32.

@NiFFu It looks like a problem with our frontend; it seems like it doesn't support quantized data types?

@R-Tars, please help and check it when you have time.

[R-Tars]

@NiFFu Could you please provide the code you ran so that I can try to reproduce it on my side?

[NiFFu]

@R-Tars
Thanks for your assistance! I currently think the issue is with the precision of my quantized model. I plan to modify my quantization process, and if there's any further front-end work needed, I'll reach out to you again.

[zhanghb97]

Assignee: @stomfaig

[stomfaig]

Status update:

1. State of Torch native quantisation solutions

It seems that the goal of torch AO currently is to eliminate any torch-native quantisation. This, in the new framework that is currently being phased in will mean that the implementation of any quantisation will be handled by a "backend". Current backends include the torch inductor among other ones. So the process looks approximately like:

torch.nn.Module
   ↓
FX Graph (Float)
   ↓
Quantized FX Graph (with quantize/dequantize wrappers)
   ↓
Backend-specific lowering (e.g., Inductor, XNNPACK)

I briefly describe what the stages do here:

Quantized FX. Here, essentially every torch native op is wrapped between a quantisation(Q) and a dequantisation(DQ) layer: torch.linear(x) → quantize(torch.linear(dequantize(x))). This makes the network "functionally quantised", which can be used to determine quantisation constants by passing some sample inputs through the network. But importantly, this is not lowerable: there is no native quantised implementation, only backend specific ones.

Backend. This is responsible for execution, etc., but according to my research, this is not in FX anymore, and is also hard to access. It is also target architecture dependent, so we would need to choose which backend will be "supported" by buddy-mlir, which is dangerous, since these change rapidly, and will need maintenance.

So neither of these is a good tapping point.

A few more remarks:

• The old quantisation interface (with native torch quantised operations) is being phased out in new torch versions, so relying on them is not a solution.
• Current backend IRs are not stable for consumption. They are not intended/designed/stable for tapping like torch FX. But since the bulk of the quantisation logic happens there, this makes it inaccessible for buddy.

2. Proposed solution

With @zhanghb97, we made the decision, to add support for quantisation within Buddy's graph passes. The aim here is to:

1. Import torch FX graph as before,
2. Modify frontend and add new passes that allow quantising the model,
3. Lower to already existing Buddy MLIR ops.