[Feature] [RFC] Add Qwen3.5 Support for Speculative Decoding Training (EAGLE3 / DFlash)

[EanWang211123]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Feature: Add Qwen3.5 Support for Speculative Decoding Training (EAGLE3 / DFlash)

Background and Goal

This feature aims to extend the current speculative decoding training framework to support the Qwen3.5 series, enabling training and integration of speculative decoding draft models such as EAGLE3 and DFlash based on Qwen3.5.

Target models include (but are not limited to):

• Qwen/Qwen3.5-27B
  https://huggingface.co/Qwen/Qwen3.5-27B
• Qwen/Qwen3.5-35B-A3B
  https://huggingface.co/Qwen/Qwen3.5-35B-A3B
• Other Qwen3.5 variants (to be extended gradually)

---

Current Progress

Based on the current DFlash training code, I have already completed a Qwen3.5 DFlash training adaptation and validation in my personal verification repository, including the following work:

• Adapted to newer dependency interfaces, with compatibility updates for interface changes in newer versions of:

  • SGLang (newer versions)
  • transformers
  • Hugging Face target-model backend interfaces

  This includes handling differences in target model loading, forward outputs, and config parsing to ensure stable training execution.

• Added Qwen3.5-specific training configs and logic, including:

  • Target model configuration
  • Draft model configuration
  • Qwen3.5-compatible training flow / parameter handling
  • DFlash adaptation logic (e.g., training pipeline integration)

• I would like to further complete and integrate Qwen3.5 speculative decoding training support for EAGLE3 / DFlash, with a unified training entry and configuration workflow (and potentially extend to more speculative decoding training methods later).

。

Related resources

No response

[jiapingW]

Great work!

[EanWang211123]

@jiapingW

I have submitted a PR that implements the requested feature. You can review it here: PR link.

Current Implementation:

• DFlash speculative decoding is now supported for Qwen3.5-27B and Qwen3.5-35B-A3B models.
• The PR includes VLM support, allowing for integration of pixel-based inputs (e.g., images, videos) alongside text data.
• HF backend online training is supported, but SGLang backend and TP > 1 are not yet implemented.

Future Support:

• SGLang backend support for Qwen3.5 models has already been implemented in my personal validation branch, but it requires adaptation to a higher version of SGLang.
• TP backend support will be explored in a follow-up iteration.

Performance Results:

• Speedup: DFlash accelerated the training as follows:

  • 1x concurrency: 2.000x speedup
  • 4x concurrency: 1.894x speedup
  • 32x concurrency: 1.137x speedup

• Acceptance Length: No significant impact on acceptance length across different concurrencies.

[gerayking]

great work! I will support sglang backend and TP > 1