[Feature] What is the upper limit to the performance improvement of Dflash training as the step increases?

[5SSjw]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Hi maintainers,
I’m trying to reproduce the official DFlash training pipeline, following the instructions from issue #465. During reproduction, I’m observing that DFlash decoding efficiency keeps improving as training steps increase, but the marginal gains become quite small in the later stage. I’d like to understand what the expected “ceiling” (if any) is when scaling training steps/epochs.

Observation:
In my runs, improvements are clearly diminishing, but even after ~12 epochs the metrics are still moving upward.

Measured results:
official baseline: speedup: 4.72x, τ: 5.97
Step 60,000: speedup: 2.58×, τ: 3.18
Step 80,000: speedup: 2.64×, τ: 3.26
Step 100,000: speedup: 2.71×, τ: 3.34
Step 142,000: speedup: 2.86×, τ: 3.55

(Training curves attached: train/accuracy and train/loss. They look stable overall, and the improvement trend flattens later.)

Additionally, is this task strongly correlated with the dataset? Is multi-dataset training necessary, or are there other training techniques? Welcome everyone to participate in the discussion.

Related resources

Here's my training shell template
`set -e

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
ROOT_DIR=$(dirname $SCRIPT_DIR)
export TORCHINDUCTOR_CACHE_DIR=$ROOT_DIR/cache/compiled_kernels
export SPECFORGE_DATA_NUM_PROC=32
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NUM_GPUS=${1:-8}

ATTENTION_BACKEND=${2:-flex_attention}

torchrun
--standalone
--nproc_per_node $NUM_GPUS
$ROOT_DIR/scripts/train_dflash.py
--target-model-path /models/Qwen3-8B
--draft-config-path $ROOT_DIR/configs/qwen3-8b-dflash.json
--train-data-path $ROOT_DIR/cache/dataset/ultrachat_train_regen_nonthinking.jsonl
--output-dir $ROOT_DIR/outputs/qwen3-8b-dflash-perfectblend-v2
--num-epochs 20
--batch-size 2
--accumulation-steps 2
--learning-rate 6e-4
--warmup-ratio 0.04
--max-grad-norm 1.0
--max-length 3072
--chat-template qwen
--attention-backend $ATTENTION_BACKEND
--random-anchor
--num-anchors 512
--loss-decay-gamma 7.0
--log-interval 50
--save-interval 1000
--cache-dir $ROOT_DIR/cache
--report-to tensorboard`

[xiaomin-D]

Thank you so much for your experiment and sharing! This is consistent with what I observed～
I understand that the current training performance has limited convergence, mainly due to the small dataset. As mentioned in the #465 (comment) , the current dataset only has 15% of the perfectblend. I think you can increase the dataset and continue to try！

[5SSjw]

I used the regenerated Ultrachat 200k dataset, and now I'm considering regenerating a complete PerfectBlend 1.42M dataset. I have a question: dflash evaluation disables thinking mode; should I also disable it during the regenerated dataset and training? I see you mentioned training in thinking mode; will this cause a misalignment with the official evaluation process?

Thank you so much for your experiment and sharing! This is consistent with what I observed～ I understand that the current training performance has limited convergence, mainly due to the small dataset. As mentioned in the #465 (comment) , the current dataset only has 15% of the perfectblend. I think you can increase the dataset and continue to try！

[xiaomin-D]

I used the regenerated Ultrachat 200k dataset, and now I'm considering regenerating a complete PerfectBlend 1.42M dataset. I have a question: dflash evaluation disables thinking mode; should I also disable it during the regenerated dataset and training? I see you mentioned training in thinking mode; will this cause a misalignment with the official evaluation process?

Thank you so much for your experiment and sharing! This is consistent with what I observed～ I understand that the current training performance has limited convergence, mainly due to the small dataset. As mentioned in the #465 (comment) , the current dataset only has 15% of the perfectblend. I think you can increase the dataset and continue to try！

I think that regenerate, training and inference alignment is fine～
I would be very grateful if you could share any progress you make in your experiments!

[xiaomin-D]

I used the regenerated Ultrachat 200k dataset, and now I'm considering regenerating a complete PerfectBlend 1.42M dataset. I have a question: dflash evaluation disables thinking mode; should I also disable it during the regenerated dataset and training? I see you mentioned training in thinking mode; will this cause a misalignment with the official evaluation process?

Thank you so much for your experiment and sharing! This is consistent with what I observed～ I understand that the current training performance has limited convergence, mainly due to the small dataset. As mentioned in the #465 (comment) , the current dataset only has 15% of the perfectblend. I think you can increase the dataset and continue to try！

Alignment can be achieved simply by modifying the evaluation process script.

[xiaomin-D]

hello @5SSjw , I think you can try this version~
Given that previous code versions experienced some data leaks, I think the new code will perform better.

#472

[5SSjw]

hello @5SSjw , I think you can try this version~ Given that previous code versions experienced some data leaks, I think the new code will perform better.

#472

Thank you for reminding me. I will update the code. My 1.4M data regeneration process is also halfway complete. I expect to start a new round of training in 12 hours.

[xiaomin-D]

hello @5SSjw , I think you can try this version~ Given that previous code versions experienced some data leaks, I think the new code will perform better.
#472

Thank you for reminding me. I will update the code. My 1.4M data regeneration process is also halfway complete. I expect to start a new round of training in 12 hours.

good luck～

[5SSjw]

hello @5SSjw , I think you can try this version~ Given that previous code versions experienced some data leaks, I think the new code will perform better.

#472

I'm unsure whether the task's performance is still dependent on the dataset size after the latest bug update( #473 ). I found that training with the full 1.4M PerfectBlend dataset for 6 epochs would take over 80 hours. Is 15% of the dataset sufficient for this task? Looking forward to your reply.

[xiaomin-D]

hello @5SSjw , I think you can try this version~ Given that previous code versions experienced some data leaks, I think the new code will perform better.
#472

I'm unsure whether the task's performance is still dependent on the dataset size after the latest bug update( #473 ). I found that training with the full 1.4M PerfectBlend dataset for 6 epochs would take over 80 hours. Is 15% of the dataset sufficient for this task? Looking forward to your reply.

I haven't done much data ablation experiment here, so I can't give a definite reply, but considering the support for resume during training, my suggestion is to run first and test the performance more. Just use specforge main branch.

[ggg-s]

hi@5SSjw It's quite strange — when I train with the latest code, the loss only decreases to around 1.5, and the accuracy stays at about 0.6, even after training on 400k samples.

[xinanjiao]

hi @ggg-s ,I had the same problem, but the difference was that I modified the training code to train qwen3_vl, which wasn't difficult, but the final training loss was still around 1, with an accuracy of around 60%. Have you solved this problem?

[5SSjw]

hi@5SSjw It's quite strange — when I train with the latest code, the loss only decreases to around 1.5, and the accuracy stays at about 0.6, even after training on 400k samples.

Actually when I train with 1.4M data for around 2 epochs, the acc and loss are quite similar to yours, but when I test the new model, the results including speed up and tau are very close to the paper, so I think don’t worry about the loss.

[Monstertail]

hi@5SSjw It's quite strange — when I train with the latest code, the loss only decreases to around 1.5, and the accuracy stays at about 0.6, even after training on 400k samples.

Actually when I train with 1.4M data for around 2 epochs, the acc and loss are quite similar to yours, but when I test the new model, the results including speed up and tau are very close to the paper, so I think don’t worry about the loss.

Hi @5SSjw Is it necessary to regen the perfectblend dataset? Or dflash can work well with the default perfectblend dataset?

BTW, do you evaluate with the script in specforge or the original dflash repo to get above 6 token per forward?