Can not reproduce the results, showing the model downgrade performance.

[linkedlist771]

Hi, I really appreciate your work on the R1 reproduce experiments, I have tried to reproduce you work on a 8x3090 server with the train_1.5b.sh, the workflow runs, but the model's accuracy downgrades. here is my config:

1. Config

/workspace/simpleR1/src/run_grpo.py \
  --do_train True \
  --config configs/grpo_config.yaml \
  --output_dir outputs/models/Qwen2.5-1.5B-Instruct_data-hendrycks-MATH-benchmark_date-2025-07-12 \
  --check_gpu_idle True \
  --model_name_or_path /home/llm_models/Qwen2.5-1.5B-Instruct \
  --train_dataset_name nlile/hendrycks-MATH-benchmark \
  --eval_dataset_name HuggingFaceH4/MATH-500 \
  --num_train_epochs 1 \
  --num_generations 24 \
  --num_eval_generations 1 \
  --per_device_train_batch_size 3 \
  --per_device_eval_batch_size 2 \
  --dynamic_sampling True \
  --max_resample_attempts 3 \
  --gradient_accumulation_steps 1 \
  --num_iterations 3 \
  --torch_empty_cache_steps 1 \
  --num_train_samples_per_dataset 2000 \
  --num_test_samples_per_dataset -1 \
  --max_completion_length 2048 \
  --max_eval_completion_length 4096 \
  --use_vllm True \
  --vllm_mode colocate \
  --vllm_gpu_memory_utilization 0.3 \
  --reward_funcs accuracy format tag \
  --reward_weights 8 1 1 \
  --loss_type bnpo \
  --scale_rewards False \
  --mask_truncated_completions True \
  --epsilon 0.2 \
  --epsilon_high 0.3 \
  --temperature 1.0 \
  --top_p 0.95 \
  --eval_temperature 0.7 \
  --eval_top_p 0.95 \
  --repetition_penalty 1.0 \
  --beta 1e-6 \
  --compute_kl True \
  --lr_scheduler_type constant \
  --learning_rate 5e-6 \
  --save_strategy steps \
  --save_steps 100 \
  --eval_strategy steps \
  --eval_steps 10 \
  --eval_on_start True \
  --log_level info \
  --wandb_project simpleR1-test \
  --run_name Qwen2.5-1.5B-Instruct_data-hendrycks-MATH-benchmark_date-2025-07-12

Here is the results:

2. Results:

The accuracy related reward keeps on fluctuating and in eval stage the model's performance keeps downgrade. For the eval results below, it seems that the model learns nothing.

[yflyzhang]

Thank you for your interest in simpleR1 and for trying to reproduce the experiments!

It seems the issue might be related to the --vllm_mode colocate setting. For multi-GPU training, I recommend using --vllm_mode server instead, as the colocate mode may not be compatible with your hardware setup.

For example, you can use 1-2 GPUs to serve the model (bash scripts/run_vllm_serve_*.sh) and the remaining GPUs for training (bash scripts/train_grpo_*.sh). Make sure vllm_server_port is set consistent between server and the training client.

For the Qwen2.5 1.5B models, the MATH-500 accuracy should be around 0.5. Note that the instruct model is generally more challenging to train than the base model. For reference, here are some previous training results:

Qwen2.5-1.5B (base model):

Qwen2.5-1.5B-Instruct:

The Qwen3 model tends to perform better than Qwen2.5, but the instruct model is much more harder to train than the base model. Here are the results on Qwen3 base models for reference:

Please use the latest code, which includes improvements for resampling and dynamic sampling control. Feel free to adjust the logic as needed for your setup.

I’ll retrain the 1.5B model when I have spare devices and time and share the results with you.

[linkedlist771]

I'll try the server mode later. I think the colocate model should work as well. Once I make any progress, I'll keep you updated.

[linkedlist771]

On my evaluation, the instruct model's initial accuracy is about 0.55, that means this framework can not facilitate the model to improve? Maybe I should use the Qwen2.5 1.5B model to train then?

[yflyzhang]

I just trained Qwen/Qwen2.5-1.5B, please see the example scripts:

simpleR1/scripts/run_vllm_serve_1.5b.sh

Lines 7 to 10 in 6bcc8b3

	python src/vllm_serve.py \
	--model Qwen/Qwen2.5-1.5B \
	--seed 42 \
	--gpu_memory_utilization 0.9

simpleR1/scripts/train_grpo_1.5b.sh

Lines 68 to 121 in 6bcc8b3

	accelerate launch \
	--main_process_port $MASTER_PORT \
	--config_file configs/accelerate_configs/ddp.yaml \
	--num_processes=2 \
	src/run_grpo.py \
	--do_train True \
	--config configs/grpo_config.yaml \
	--output_dir $OUTPUT_DIR \
	--check_gpu_idle True \
	--model_name_or_path $model_name_or_path \
	--train_dataset_name $train_dataset \
	--eval_dataset_name $eval_dataset \
	--num_train_epochs 1 \
	--num_generations 10 \
	--num_eval_generations 1 \
	--per_device_train_batch_size 10 \
	--per_device_eval_batch_size 128 \
	--dynamic_sampling True \
	--max_resample_attempts 3 \
	--gradient_accumulation_steps 1 \
	--num_iterations 3 \
	--torch_empty_cache_steps 1 \
	--num_train_samples_per_dataset 2000 \
	--num_test_samples_per_dataset -1 \
	--max_completion_length 2048 \
	--max_eval_completion_length 4096 \
	--use_vllm True \
	--vllm_mode server \
	--vllm_server_host 0.0.0.0 \
	--vllm_server_port 8000 \
	--reward_funcs accuracy format tag \
	--reward_weights 8 1 1 \
	--loss_type bnpo \
	--scale_rewards False \
	--mask_truncated_completions True \
	--epsilon 0.2 \
	--epsilon_high 0.3 \
	--temperature 1.0 \
	--top_p 0.95 \
	--eval_temperature 0.7 \
	--eval_top_p 0.95 \
	--repetition_penalty 1.02 \
	--beta 1e-5 \
	--lr_scheduler_type constant \
	--learning_rate 5e-6 \
	--save_strategy steps \
	--save_steps 100 \
	--eval_strategy steps \
	--eval_steps 10 \
	--eval_on_start True \
	--log_level info \
	--wandb_project simpleR1-train \
	--run_name $run_name \
	2>&1 | tee $LOG_FILE

Please note that this is not the optimal parameter setting, just an illustrative example. Other parameters may yield better results.
You may need to enforce repetition_penalty, for example --repetition_penalty 1.02, as Qwen/Qwen2.5-1.5B is likely to generate repetitive text during the training.

Below please find the wandb log on the eval dataset (MATH-500):

For instruct models, I found them hard to train compared with base models. Most of the time, the performance of the instrcut model gets better quickly, and will also degrade then

For instruct models, I’ve found them more challenging to train compared to base models. Typically, the performance of instruct models improves quickly but can degrade afterward. Finer parameter tuning is required, such as adjusting the system prompt, number of completion tokens, as these can significantly impact evaluation results. Unfortunately, I don’t yet have a definitive guideline for selecting parameters to share with you. In the given examples, max_completion_length (for train) is set to 2048, max_eval_completion_length (for eval) is set to 4096, and increasing them will generally yield better eval results but takes much more time and computing resources.

[linkedlist771]

I have literally just followed your guide now, however, the results still turns out to be downgrade:

I have lower the batchsize for the resource constrain, other configs should just align with yours, I use 7 GPUS for training one for inference.

Here is the training config, not much different from yours:

/workspace/simpleR1/src/run_grpo.py \
  --do_train True \
  --config configs/grpo_config.yaml \
  --output_dir outputs/models/Qwen2.5-1.5B_date-2025-07-17 \
  --check_gpu_idle True \
  --model_name_or_path /home/llm_models/Qwen2.5-1.5B \
  --train_dataset_name nlile/hendrycks-MATH-benchmark \
  --eval_dataset_name HuggingFaceH4/MATH-500 \
  --num_train_epochs 1 \
  --num_generations 14 \
  --num_eval_generations 1 \
  --per_device_train_batch_size 4 \
  --per_device_eval_batch_size 10 \
  --dynamic_sampling True \
  --max_resample_attempts 3 \
  --gradient_accumulation_steps 1 \
  --num_iterations 3 \
  --torch_empty_cache_steps 1 \
  --num_train_samples_per_dataset 2000 \
  --num_test_samples_per_dataset -1 \
  --max_completion_length 2048 \
  --max_eval_completion_length 4096 \
  --use_vllm True \
  --vllm_mode server \
  --vllm_server_host 0.0.0.0 \
  --vllm_server_port 8000 \
  --reward_funcs accuracy format tag \
  --reward_weights 8 1 1 \
  --loss_type bnpo \
  --scale_rewards False \
  --mask_truncated_completions True \
  --epsilon 0.2 \
  --epsilon_high 0.3 \
  --temperature 1.0 \
  --top_p 0.95 \
  --eval_temperature 0.7 \
  --eval_top_p 0.95 \
  --repetition_penalty 1.02 \
  --beta 1e-5 \
  --lr_scheduler_type constant \
  --learning_rate 5e-6 \
  --save_strategy steps \
  --save_steps 100 \
  --eval_strategy steps \
  --eval_steps 10 \
  --eval_on_start True \
  --log_level info \
  --wandb_project simpleR1-train \
  --run_name Qwen2.5-1.5B_date-2025-07-17

It is more likely to generate duplicate chars for eval when setting it as 4096 while in training it is 2048:

[yflyzhang]

Thank you for providing more details about your setup and the issue you're facing. I'm sorry to hear that you're still experiencing performance degradation with the model.

Based on the information you’ve shared:

1. Ensure You're Using the Latest Code: From the headers in your wandb logs, it seems you might not be using the latest version of the simpler1 codebase. Please pull the most recent code from the GitHub repository (https://github.com/yflyzhang/simpleR1/tree/main) to ensure you have the latest resampling and dynamic sampling logic, which could impact training stability and performance. Double-check the commit hash to confirm you're on the latest version.

2. Verify vLLM and Dependency Versions: Your issue might also be related to incompatible or outdated dependencies. The prarameters you listed seem mostly aligned, but I recommend ensuring the exact versions below for consistency with my setup, especially since vLLM 0.7.* has known issues.
   Please refer to: https://github.com/yflyzhang/simpleR1/tree/main?tab=readme-ov-file#dependencies
   Especially for the following:

    # CUDA 12.4
    pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
   
    pip install transformers==4.52.4 accelerate==1.7.0 trl==0.18.2 deepspeed==0.16.9
    pip install flash-attn==2.7.4.post1
    pip install vllm==0.8.5.post1
    pip install math-verify==0.8.0 latex2sympy2_extended==1.10.2
    pip install wandb==0.20.1

3. Training Configuration and Metrics: The mean completion length being too high and causing excessive truncation is concerning. This could indicate a mismatch in training dynamics. If you have sufficient compute resources, you can also increase the completion length to allow more flexibility in model outputs, which might reduce truncation issues. But I feel this is mainly due to the repetitive generation issue.
   As found in the snapshots you shared, the MATH-500 accuracy increases over 0.45, but drops very soon. This means that the training framework works, but as no further enough possitve signals are provied or bad signals are provided, the model didn't learn well afterwards.
   For the instruct model, try disabling format reward and tag count reward in the training script sometimes helps, as they may distract from the instruction-following focus. You can refer to scripts/train_grpo_1.5b-instruct.sh in the repository for guidance on this.
   But I'm not sure if retaining accuracy reward only works for the base model.

Here are the results I just ran for the 1.5B instruct model (along with the 1.5B base model I ran few days ago) for your reference:

Their setups are uploaded in the given scripts, or you can confirm it by the following:

Qwen2.5-1.5B(Base)

/src/run_grpo.py --do_train True --config configs/grpo_config.yaml --output_dir outputs/models/Qwen2.5-1.5B_date-2025-07-15 --check_gpu_idle True --model_name_or_path Qwen/Qwen2.5-1.5B --train_dataset_name nlile/hendrycks-MATH-benchmark --eval_dataset_name HuggingFaceH4/MATH-500 --num_train_epochs 1 --num_generations 10 --num_eval_generations 1 --per_device_train_batch_size 10 --per_device_eval_batch_size 128 --dynamic_sampling True --max_resample_attempts 3 --gradient_accumulation_steps 1 --num_iterations 3 --torch_empty_cache_steps 1 --num_train_samples_per_dataset 2000 --num_test_samples_per_dataset -1 --max_completion_length 2048 --max_eval_completion_length 4096 --use_vllm True --vllm_mode server --vllm_server_host 0.0.0.0 --vllm_server_port 8000 --reward_funcs accuracy format tag --reward_weights 8 1 1 --loss_type bnpo --scale_rewards False --mask_truncated_completions True --epsilon 0.2 --epsilon_high 0.3 --temperature 1.0 --top_p 0.95 --eval_temperature 0.7 --eval_top_p 0.95 --repetition_penalty 1.02 --beta 1e-5 --lr_scheduler_type constant --learning_rate 5e-6 --save_strategy steps --save_steps 100 --eval_strategy steps --eval_steps 10 --eval_on_start True --log_level info --wandb_project simpleR1-train --run_name Qwen2.5-1.5B_date-2025-07-15

Qwen2.5-1.5B-Instruct

/src/run_grpo.py --do_train True --config configs/grpo_config.yaml --output_dir outputs/models/Qwen2.5-1.5B-Instruct_date-2025-07-16 --check_gpu_idle True --model_name_or_path Qwen/Qwen2.5-1.5B-Instruct --train_dataset_name nlile/hendrycks-MATH-benchmark --eval_dataset_name HuggingFaceH4/MATH-500 --num_train_epochs 1 --num_generations 20 --num_eval_generations 1 --per_device_train_batch_size 10 --per_device_eval_batch_size 128 --dynamic_sampling True --max_resample_attempts 3 --gradient_accumulation_steps 1 --num_iterations 3 --torch_empty_cache_steps 1 --num_train_samples_per_dataset 2000 --num_test_samples_per_dataset -1 --max_completion_length 2048 --max_eval_completion_length 4096 --use_vllm True --vllm_mode server --vllm_server_host 0.0.0.0 --vllm_server_port 8000 --reward_funcs accuracy --reward_weights 8 --loss_type bnpo --scale_rewards False --mask_truncated_completions True --epsilon 0.2 --epsilon_high 0.3 --temperature 1.0 --top_p 1.0 --eval_temperature 0.7 --eval_top_p 0.95 --repetition_penalty 1.0 --beta 5e-5 --lr_scheduler_type constant --learning_rate 3e-6 --save_strategy steps --save_steps 100 --eval_strategy steps --eval_steps 10 --eval_on_start True --log_level info --wandb_project simpleR1-train --run_name Qwen2.5-1.5B-Instruct_date-2025-07-16

[linkedlist771]

Thank you again for your kindly rely, I will give it another shot now. This time, I will reinstall all the dependencies and use just the code you provide and let's see wether it works.

[linkedlist771]

I have now pulled your latest code and reinstalled the training requirements. The accuracy reward mechanism appears to be performing well, while format reward stays 0 all the way. Once the training is complete, I will keep you informed and then proceed to test the instruction model. Thank you again for your kind reply.

[linkedlist771]

Update: still downgrade model performance .....

[yflyzhang]

Sorry to hear this. Can you share the specific training parameters you're using? I'll rerun the training on my end with your parameters as close as possible and compare the results.

[linkedlist771]

The only difference between me and yours may be the DDP.yaml and the batch size, I am training it on a 8x3090 server, so the batch size is limited, here is full config as you go:

training command

python /workspace/simpleR1/src/run_grpo.py \
  --do_train True \
  --config configs/grpo_config.yaml \
  --output_dir outputs/models/Qwen2.5-1.5B_date-2025-07-18 \
  --check_gpu_idle True \
  --model_name_or_path /home/llm_models/Qwen2.5-1.5B \
  --train_dataset_name nlile/hendrycks-MATH-benchmark \
  --eval_dataset_name HuggingFaceH4/MATH-500 \
  --num_train_epochs 1 \
  --num_generations 14 \
  --num_eval_generations 1 \
  --per_device_train_batch_size 2 \
  --per_device_eval_batch_size 4 \
  --dynamic_sampling True \
  --max_resample_attempts 3 \
  --gradient_accumulation_steps 1 \
  --num_iterations 3 \
  --torch_empty_cache_steps 1 \
  --num_train_samples_per_dataset 2000 \
  --num_test_samples_per_dataset -1 \
  --max_completion_length 2048 \
  --max_eval_completion_length 4096 \
  --use_vllm True \
  --vllm_mode server \
  --vllm_server_host 0.0.0.0 \
  --vllm_server_port 8000 \
  --reward_funcs accuracy format tag \
  --reward_weights 8 1 1 \
  --loss_type bnpo \
  --scale_rewards False \
  --mask_truncated_completions True \
  --epsilon 0.2 \
  --epsilon_high 0.3 \
  --temperature 1.0 \
  --top_p 0.95 \
  --eval_temperature 0.7 \
  --eval_top_p 0.95 \
  --repetition_penalty 1.1 \
  --beta 1e-5 \
  --lr_scheduler_type constant \
  --learning_rate 5e-6 \
  --save_strategy steps \
  --save_steps 100 \
  --eval_strategy steps \
  --eval_steps 10 \
  --eval_on_start True \
  --log_level info \
  --wandb_project simpleR1-train \
  --run_name Qwen2.5-1.5B_date-2025-07-18

Here is the full config:

{
  "beta": {
    "value": 0.00001
  },
  "bf16": {
    "value": true
  },
  "fp16": {
    "value": false
  },
  "fsdp": {
    "value": []
  },
  "seed": {
    "value": 42
  },
  "tf32": {
    "value": null
  },
  "debug": {
    "value": []
  },
  "min_p": {
    "value": null
  },
  "optim": {
    "value": "adamw_torch"
  },
  "top_k": {
    "value": 50
  },
  "top_p": {
    "value": 0.95
  },
  "_wandb": {
    "value": {
      "e": {
        "bhh1neswavmrqbmwikrye7vvuyh1j8f1": {
          "os": "Linux-4.15.0-194-generic-x86_64-with-glibc2.35",
          "git": {
            "commit": "ebe7ac5a136e6f74157b777bc858bd491c45e6ea",
            "remote": "https://github.com/yflyzhang/simpleR1.git"
          },
          "gpu": "NVIDIA GeForce RTX 3090",
          "args": [
            "--do_train",
            "True",
            "--config",
            "configs/grpo_config.yaml",
            "--output_dir",
            "outputs/models/Qwen2.5-1.5B_date-2025-07-18",
            "--check_gpu_idle",
            "True",
            "--model_name_or_path",
            "/home/llm_models/Qwen2.5-1.5B",
            "--train_dataset_name",
            "nlile/hendrycks-MATH-benchmark",
            "--eval_dataset_name",
            "HuggingFaceH4/MATH-500",
            "--num_train_epochs",
            "1",
            "--num_generations",
            "14",
            "--num_eval_generations",
            "1",
            "--per_device_train_batch_size",
            "2",
            "--per_device_eval_batch_size",
            "4",
            "--dynamic_sampling",
            "True",
            "--max_resample_attempts",
            "3",
            "--gradient_accumulation_steps",
            "1",
            "--num_iterations",
            "3",
            "--torch_empty_cache_steps",
            "1",
            "--num_train_samples_per_dataset",
            "2000",
            "--num_test_samples_per_dataset",
            "-1",
            "--max_completion_length",
            "2048",
            "--max_eval_completion_length",
            "4096",
            "--use_vllm",
            "True",
            "--vllm_mode",
            "server",
            "--vllm_server_host",
            "0.0.0.0",
            "--vllm_server_port",
            "8000",
            "--reward_funcs",
            "accuracy",
            "format",
            "tag",
            "--reward_weights",
            "8",
            "1",
            "1",
            "--loss_type",
            "bnpo",
            "--scale_rewards",
            "False",
            "--mask_truncated_completions",
            "True",
            "--epsilon",
            "0.2",
            "--epsilon_high",
            "0.3",
            "--temperature",
            "1.0",
            "--top_p",
            "0.95",
            "--eval_temperature",
            "0.7",
            "--eval_top_p",
            "0.95",
            "--repetition_penalty",
            "1.1",
            "--beta",
            "1e-5",
            "--lr_scheduler_type",
            "constant",
            "--learning_rate",
            "5e-6",
            "--save_strategy",
            "steps",
            "--save_steps",
            "100",
            "--eval_strategy",
            "steps",
            "--eval_steps",
            "10",
            "--eval_on_start",
            "True",
            "--log_level",
            "info",
            "--wandb_project",
            "simpleR1-train",
            "--run_name",
            "Qwen2.5-1.5B_date-2025-07-18"
          ],
          "disk": {
            "/": {
              "used": "3185775239168",
              "total": "3779302981632"
            }
          },
          "host": "sse-lg-113-99",
          "root": "/workspace/simpleR1",
          "email": "213193509seu@gmail.com",
          "memory": {
            "total": "540689219584"
          },
          "python": "CPython 3.10.12",
          "program": "/workspace/simpleR1/src/run_grpo.py",
          "codePath": "src/run_grpo.py",
          "writerId": "bhh1neswavmrqbmwikrye7vvuyh1j8f1",
          "cpu_count": 56,
          "gpu_count": 8,
          "startedAt": "2025-07-18T07:50:55.701849Z",
          "executable": "/workspace/simpleR1/simple_R1/bin/python3",
          "gpu_nvidia": [
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-45f56ef5-a01d-2238-944b-e75948286fe0",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-4b041332-d2f3-6dd4-17c5-8ebd74f5cc4e",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-d8c83e82-f894-f3ad-16bd-c3b8ca4fa178",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-49ce4879-0051-efbf-7de0-43171bf2748b",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-6de6b760-9772-0714-0d7e-b61b1259ad7f",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-0ffd3c15-c0c4-f2e7-2f84-5d34501e106b",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-c664be58-48f7-46cd-8562-5fb29e406a5f",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            },
            {
              "name": "NVIDIA GeForce RTX 3090",
              "uuid": "GPU-fa4c69d4-7a40-f4f4-3fbf-27728eb222c8",
              "cudaCores": 10496,
              "memoryTotal": "25769803776",
              "architecture": "Ampere"
            }
          ],
          "cudaVersion": "12.8",
          "codePathLocal": "src/run_grpo.py",
          "cpu_count_logical": 112
        }
      },
      "m": [
        {
          "1": "train/global_step",
          "6": [
            3
          ],
          "7": []
        },
        {
          "2": "*",
          "5": 1,
          "6": [
            1
          ],
          "7": []
        }
      ],
      "t": {
        "1": [
          1,
          11,
          30,
          41,
          49,
          51,
          71,
          84,
          95,
          98,
          105
        ],
        "2": [
          1,
          11,
          30,
          41,
          49,
          51,
          71,
          84,
          95,
          98,
          105
        ],
        "3": [
          7,
          13,
          19,
          62,
          66
        ],
        "4": "3.10.12",
        "5": "0.21.0",
        "6": "4.52.4",
        "9": {
          "1": "transformers_trainer"
        },
        "12": "0.21.0",
        "13": "linux-x86_64"
      },
      "cli_version": "0.21.0",
      "python_version": "3.10.12"
    }
  },
  "prefix": {
    "value": null
  },
  "do_eval": {
    "value": true
  },
  "epsilon": {
    "value": 0.2
  },
  "no_cuda": {
    "value": false
  },
  "use_cpu": {
    "value": false
  },
  "do_train": {
    "value": true
  },
  "id2label": {
    "value": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    }
  },
  "label2id": {
    "value": {
      "LABEL_0": 0,
      "LABEL_1": 1
    }
  },
  "run_name": {
    "value": "Qwen2.5-1.5B_date-2025-07-18"
  },
  "use_ipex": {
    "value": false
  },
  "use_vllm": {
    "value": true
  },
  "adafactor": {
    "value": false
  },
  "callbacks": {
    "value": []
  },
  "data_seed": {
    "value": null
  },
  "deepspeed": {
    "value": null
  },
  "do_sample": {
    "value": false
  },
  "hub_token": {
    "value": "<HUB_TOKEN>"
  },
  "log_level": {
    "value": "info"
  },
  "loss_type": {
    "value": "bnpo"
  },
  "max_steps": {
    "value": -1
  },
  "num_beams": {
    "value": 1
  },
  "ray_scope": {
    "value": "last"
  },
  "report_to": {
    "value": [
      "wandb"
    ]
  },
  "typical_p": {
    "value": 1
  },
  "use_cache": {
    "value": false
  },
  "use_mrope": {
    "value": false
  },
  "vllm_mode": {
    "value": "server"
  },
  "adam_beta1": {
    "value": 0.9
  },
  "adam_beta2": {
    "value": 0.999
  },
  "benchmarks": {
    "value": []
  },
  "compute_kl": {
    "value": false
  },
  "do_predict": {
    "value": false
  },
  "eval_delay": {
    "value": 0
  },
  "eval_min_p": {
    "value": null
  },
  "eval_steps": {
    "value": 10
  },
  "eval_top_k": {
    "value": null
  },
  "eval_top_p": {
    "value": 0.95
  },
  "hidden_act": {
    "value": "silu"
  },
  "is_decoder": {
    "value": false
  },
  "local_rank": {
    "value": 0
  },
  "max_length": {
    "value": 20
  },
  "min_length": {
    "value": 0
  },
  "model_type": {
    "value": "qwen2"
  },
  "optim_args": {
    "value": null
  },
  "output_dir": {
    "value": "outputs/models/Qwen2.5-1.5B_date-2025-07-18"
  },
  "past_index": {
    "value": -1
  },
  "rope_theta": {
    "value": 1000000
  },
  "save_steps": {
    "value": 100
  },
  "vllm_dtype": {
    "value": null
  },
  "vocab_size": {
    "value": 151936
  },
  "ddp_backend": {
    "value": null
  },
  "ddp_timeout": {
    "value": 1800
  },
  "fsdp_config": {
    "value": {
      "xla": false,
      "xla_fsdp_v2": false,
      "min_num_params": 0,
      "xla_fsdp_grad_ckpt": false
    }
  },
  "hidden_size": {
    "value": 1536
  },
  "label_names": {
    "value": null
  },
  "logging_dir": {
    "value": "outputs/models/Qwen2.5-1.5B_date-2025-07-18/runs/Jul18_07-50-07_sse-lg-113-99"
  },
  "push_to_hub": {
    "value": false
  },
  "return_dict": {
    "value": true
  },
  "temperature": {
    "value": 1
  },
  "torch_dtype": {
    "value": "bfloat16"
  },
  "torchdynamo": {
    "value": null
  },
  "torchscript": {
    "value": false
  },
  "vllm_device": {
    "value": "auto"
  },
  "adam_epsilon": {
    "value": 1e-8
  },
  "bos_token_id": {
    "value": 151643
  },
  "disable_tqdm": {
    "value": false
  },
  "eos_token_id": {
    "value": 151643
  },
  "epsilon_high": {
    "value": 0.3
  },
  "fp16_backend": {
    "value": "auto"
  },
  "hub_model_id": {
    "value": null
  },
  "hub_strategy": {
    "value": "every_save"
  },
  "pad_token_id": {
    "value": null
  },
  "problem_type": {
    "value": null
  },
  "rms_norm_eps": {
    "value": 0.000001
  },
  "rope_scaling": {
    "value": null
  },
  "sep_token_id": {
    "value": null
  },
  "use_bfloat16": {
    "value": false
  },
  "wandb_entity": {
    "value": null
  },
  "warmup_ratio": {
    "value": 0.1
  },
  "warmup_steps": {
    "value": 0
  },
  "weight_decay": {
    "value": 0
  },
  "_name_or_path": {
    "value": "/home/llm_models/Qwen2.5-1.5B"
  },
  "architectures": {
    "value": [
      "Qwen2ForCausalLM"
    ]
  },
  "bad_words_ids": {
    "value": null
  },
  "chat_template": {
    "value": null
  },
  "eval_on_start": {
    "value": true
  },
  "eval_strategy": {
    "value": "steps"
  },
  "jit_mode_eval": {
    "value": false
  },
  "learning_rate": {
    "value": 0.000005
  },
  "logging_steps": {
    "value": 1
  },
  "max_grad_norm": {
    "value": 1
  },
  "mp_parameters": {
    "value": ""
  },
  "output_scores": {
    "value": false
  },
  "save_strategy": {
    "value": "steps"
  },
  "scale_rewards": {
    "value": false
  },
  "system_prompt": {
    "value": "You are a helpful AI Assistant that provides well-reasoned and detailed responses.\nFor a given question, you should first think the reasoning process in the mind step by step and then provide the user with the answer.\nThe reasoning process and answer must be enclosed within <think> </think> and <answer> </answer> tags, respectively.\nYou must use the following format:\n<think>\n<put your think process here>\n</think>\n<answer>\n<put your final answer only within \\boxed{} in LaTeX format>\n</answer>"
  },
  "torch_compile": {
    "value": false
  },
  "tpu_num_cores": {
    "value": null
  },
  "wandb_project": {
    "value": "simpleR1-train"
  },
  "bf16_full_eval": {
    "value": false
  },
  "early_stopping": {
    "value": false
  },
  "fp16_full_eval": {
    "value": false
  },
  "fp16_opt_level": {
    "value": "O1"
  },
  "length_penalty": {
    "value": 1
  },
  "num_iterations": {
    "value": 3
  },
  "reward_weights": {
    "value": [
      8,
      1,
      1
    ]
  },
  "sliding_window": {
    "value": 131072
  },
  "sync_ref_model": {
    "value": false
  },
  "tf_legacy_loss": {
    "value": false
  },
  "use_mps_device": {
    "value": false
  },
  "disable_dropout": {
    "value": false
  },
  "finetuning_task": {
    "value": null
  },
  "group_by_length": {
    "value": false
  },
  "hub_always_push": {
    "value": false
  },
  "log_completions": {
    "value": true
  },
  "num_beam_groups": {
    "value": 1
  },
  "num_generations": {
    "value": 14
  },
  "save_only_model": {
    "value": true
  },
  "shuffle_dataset": {
    "value": true
  },
  "suppress_tokens": {
    "value": null
  },
  "tokenizer_class": {
    "value": null
  },
  "dynamic_sampling": {
    "value": true
  },
  "eval_temperature": {
    "value": 0.7
  },
  "full_determinism": {
    "value": false
  },
  "hub_private_repo": {
    "value": null
  },
  "ignore_data_skip": {
    "value": false
  },
  "log_on_each_node": {
    "value": true
  },
  "logging_strategy": {
    "value": "steps"
  },
  "num_train_epochs": {
    "value": 1
  },
  "save_safetensors": {
    "value": true
  },
  "save_total_limit": {
    "value": null
  },
  "use_liger_kernel": {
    "value": false
  },
  "vllm_server_host": {
    "value": "0.0.0.0"
  },
  "vllm_server_port": {
    "value": 8000
  },
  "attention_dropout": {
    "value": 0
  },
  "ddp_bucket_cap_mb": {
    "value": null
  },
  "diversity_penalty": {
    "value": 0
  },
  "greater_is_better": {
    "value": null
  },
  "initializer_range": {
    "value": 0.02
  },
  "intermediate_size": {
    "value": 8960
  },
  "log_level_replica": {
    "value": "warning"
  },
  "lr_scheduler_type": {
    "value": "constant"
  },
  "max_prompt_length": {
    "value": 512
  },
  "max_window_layers": {
    "value": 28
  },
  "model_init_kwargs": {
    "value": {
      "revision": "main",
      "use_cache": false,
      "torch_dtype": "bfloat16",
      "trust_remote_code": false,
      "attn_implementation": "flash_attention_2"
    }
  },
  "num_hidden_layers": {
    "value": 28
  },
  "output_attentions": {
    "value": false
  },
  "push_to_hub_token": {
    "value": "<PUSH_TO_HUB_TOKEN>"
  },
  "save_on_each_node": {
    "value": false
  },
  "tpu_metrics_debug": {
    "value": false
  },
  "accelerator_config": {
    "value": {
      "even_batches": true,
      "non_blocking": false,
      "split_batches": false,
      "dispatch_batches": null,
      "use_seedable_sampler": true,
      "gradient_accumulation_kwargs": null
    }
  },
  "batch_eval_metrics": {
    "value": false
  },
  "hub_model_revision": {
    "value": "main"
  },
  "is_encoder_decoder": {
    "value": false
  },
  "length_column_name": {
    "value": "length"
  },
  "logging_first_step": {
    "value": true
  },
  "repetition_penalty": {
    "value": 1.1
  },
  "torch_compile_mode": {
    "value": null
  },
  "use_sliding_window": {
    "value": false
  },
  "vllm_max_model_len": {
    "value": null
  },
  "add_cross_attention": {
    "value": false
  },
  "forced_bos_token_id": {
    "value": null
  },
  "forced_eos_token_id": {
    "value": null
  },
  "fsdp_min_num_params": {
    "value": 0
  },
  "include_for_metrics": {
    "value": []
  },
  "lr_scheduler_kwargs": {
    "value": {
      "min_lr_rate": 0.1
    }
  },
  "neftune_noise_alpha": {
    "value": null
  },
  "num_attention_heads": {
    "value": 12
  },
  "num_key_value_heads": {
    "value": 2
  },
  "skip_memory_metrics": {
    "value": true
  },
  "tie_encoder_decoder": {
    "value": false
  },
  "tie_word_embeddings": {
    "value": true
  },
  "vllm_server_timeout": {
    "value": 240
  },
  "auto_find_batch_size": {
    "value": false
  },
  "cache_implementation": {
    "value": null
  },
  "dataloader_drop_last": {
    "value": false
  },
  "model/num_parameters": {
    "value": 0
  },
  "no_repeat_ngram_size": {
    "value": 0
  },
  "num_eval_generations": {
    "value": 1
  },
  "num_return_sequences": {
    "value": 1
  },
  "optim_target_modules": {
    "value": null
  },
  "output_hidden_states": {
    "value": false
  },
  "overwrite_output_dir": {
    "value": true
  },
  "prediction_loss_only": {
    "value": false
  },
  "push_to_hub_model_id": {
    "value": null
  },
  "push_to_hub_revision": {
    "value": false
  },
  "ref_model_sync_steps": {
    "value": 512
  },
  "task_specific_params": {
    "value": null
  },
  "transformers_version": {
    "value": "4.52.4"
  },
  "vllm_server_base_url": {
    "value": null
  },
  "begin_suppress_tokens": {
    "value": null
  },
  "dataloader_pin_memory": {
    "value": true
  },
  "ddp_broadcast_buffers": {
    "value": null
  },
  "max_completion_length": {
    "value": 2048
  },
  "max_resample_attempts": {
    "value": 3
  },
  "metric_for_best_model": {
    "value": null
  },
  "ref_model_mixup_alpha": {
    "value": 0.6
  },
  "remove_invalid_values": {
    "value": false
  },
  "remove_unused_columns": {
    "value": false
  },
  "torch_compile_backend": {
    "value": null
  },
  "dataloader_num_workers": {
    "value": 0
  },
  "decoder_start_token_id": {
    "value": null
  },
  "eval_do_concat_batches": {
    "value": true
  },
  "eval_use_gather_object": {
    "value": false
  },
  "gradient_checkpointing": {
    "value": true
  },
  "half_precision_backend": {
    "value": "auto"
  },
  "label_smoothing_factor": {
    "value": 0
  },
  "load_best_model_at_end": {
    "value": false
  },
  "logging_nan_inf_filter": {
    "value": true
  },
  "overwrite_hub_revision": {
    "value": false
  },
  "resume_from_checkpoint": {
    "value": null
  },
  "chunk_size_feed_forward": {
    "value": 0
  },
  "eval_accumulation_steps": {
    "value": null
  },
  "max_position_embeddings": {
    "value": 131072
  },
  "per_gpu_eval_batch_size": {
    "value": null
  },
  "return_dict_in_generate": {
    "value": false
  },
  "torch_empty_cache_steps": {
    "value": 1
  },
  "per_gpu_train_batch_size": {
    "value": null
  },
  "push_to_hub_organization": {
    "value": null
  },
  "ds3_gather_for_generation": {
    "value": true
  },
  "include_tokens_per_second": {
    "value": false
  },
  "vllm_tensor_parallel_size": {
    "value": 1
  },
  "dataloader_prefetch_factor": {
    "value": null
  },
  "ddp_find_unused_parameters": {
    "value": null
  },
  "include_inputs_for_metrics": {
    "value": false
  },
  "mask_truncated_completions": {
    "value": true
  },
  "max_eval_completion_length": {
    "value": 4096
  },
  "per_device_eval_batch_size": {
    "value": 4
  },
  "use_legacy_prediction_loop": {
    "value": false
  },
  "vllm_enable_prefix_caching": {
    "value": null
  },
  "vllm_guided_decoding_regex": {
    "value": null
  },
  "cross_attention_hidden_size": {
    "value": null
  },
  "gradient_accumulation_steps": {
    "value": 1
  },
  "per_device_train_batch_size": {
    "value": 2
  },
  "vllm_gpu_memory_utilization": {
    "value": 0.2
  },
  "encoder_no_repeat_ngram_size": {
    "value": 0
  },
  "average_tokens_across_devices": {
    "value": false
  },
  "dataloader_persistent_workers": {
    "value": false
  },
  "gradient_checkpointing_kwargs": {
    "value": {
      "use_reentrant": false
    }
  },
  "include_num_input_tokens_seen": {
    "value": false
  },
  "exponential_decay_length_penalty": {
    "value": null
  },
  "fsdp_transformer_layer_cls_to_wrap": {
    "value": null
  },
  "restore_callback_states_from_checkpoint": {
    "value": false
  }
}

• serving command:

model_name_or_path="/home/llm_models/Qwen2.5-1.5B" # -Instruct"

export CUDA_VISIBLE_DEVICES=0

python src/vllm_serve.py \
    --model $model_name_or_path \
    --seed 42 \
    --gpu_memory_utilization 0.9

• training command:

# Train via command line

model_name_or_path="/home/llm_models/Qwen2.5-1.5B" # -Instruct"
#Qwen/Qwen2.5-1.5B
# model_name_or_path=Qwen/Qwen2.5-1.5B-Instruct
# model_name_or_path=Qwen/Qwen2.5-Math-1.5B
# model_name_or_path=Qwen/Qwen2.5-Math-1.5B-Instruct
# model_name_or_path=Qwen/Qwen3-1.7B
export HF_ENDPOINT=https://hf-mirror.com
# model_name_or_path=meta-llama/Llama-3.2-1B
# model_name_or_path=meta-llama/Llama-3.2-1B-Instruct
export WANDB_MODE=online
# model_name_or_path=Qwen/Qwen2.5-1.5B-Instruct
# model_name_or_path=Qwen/Qwen2.5-Math-1.5B
# model_name_or_path=Qwen/Qwen2.5-Math-1.5B-Instruct
# model_name_or_path=Qwen/Qwen3-1.7B

# model_name_or_path=meta-llama/Llama-3.2-1B
# model_name_or_path=meta-llama/Llama-3.2-1B-Instruct

# model_name_or_path=Qwen/Qwen2.5-3B
# model_name_or_path=Qwen/Qwen2.5-3B-Instruct

# model_name_or_path=Qwen/Qwen3-4B

# train_dataset=openai/gsm8k
train_dataset=nlile/hendrycks-MATH-benchmark
# train_dataset=meta-math/MetaMathQA
# train_dataset=SynthLabsAI/Big-Math-RL-Verified
# train_dataset=hiyouga/math12k
# train_dataset=gneubig/aime-1983-2024
# train_dataset=open-r1/OpenR1-Math-220k
# train_dataset=agentica-org/DeepScaleR-Preview-Dataset
# train_dataset=RUC-AIBOX/STILL-3-Preview-RL-Data
# train_dataset=Maxwell-Jia/AIME_2024


eval_dataset=HuggingFaceH4/MATH-500
# eval_dataset=opencompass/AIME2025


model_name=$(basename $model_name_or_path)
# run_name=$model_name-$(date +%Y-%m-%d)
# run_name=${model_name}_data-$(basename $train_dataset)_date-$(date +%Y-%m-%d)
run_name=${model_name}_date-$(date +%Y-%m-%d)


OUTPUT_DIR=outputs/models/$run_name
LOG_FILE="$OUTPUT_DIR/train_log_$(date +%Y-%m-%d_%H:%M:%S.log)"

mkdir -p $OUTPUT_DIR

echo "current file is: $0"
cp "$0" "$OUTPUT_DIR"/run.sh


echo
echo "==================== Training: ===================="
echo "[INFO] run name: $run_name"
echo "[INFO] logs are saved to: $LOG_FILE"
echo

# sleep 7h


MASTER_PORT=$(shuf -n 1 -i 10000-65535)

export CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7
# export CUDA_VISIBLE_DEVICES=2
export TOKENIZERS_PARALLELISM=false
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
# export HF_HOME=/mnt/sgnfsdata/tolo-02-95/yafei/.cache/huggingface
# export HF_HOME=/xxx/.cache/huggingface
# Set `HF_HOME` when necessary, by default HF_HOME=~/.cache/huggingface
# where `~` is user's home directory.

accelerate launch \
    --main_process_port $MASTER_PORT \
    --config_file configs/accelerate_configs/zero3.yaml \
    --num_processes=7 \
src/run_grpo.py \
    --do_train True \
    --config configs/grpo_config.yaml \
    --output_dir $OUTPUT_DIR \
    --check_gpu_idle True \
    --model_name_or_path $model_name_or_path \
    --train_dataset_name $train_dataset \
    --eval_dataset_name $eval_dataset \
    --num_train_epochs 1 \
    --num_generations 14 \
    --num_eval_generations 1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 4 \
    --dynamic_sampling True \
    --max_resample_attempts 3 \
    --gradient_accumulation_steps 1 \
    --num_iterations 3 \
    --torch_empty_cache_steps 1 \
    --num_train_samples_per_dataset 2000 \
    --num_test_samples_per_dataset -1 \
    --max_completion_length 2048 \
    --max_eval_completion_length 4096 \
    --use_vllm True \
    --vllm_mode server \
    --vllm_server_host 0.0.0.0 \
    --vllm_server_port 8000 \
    --reward_funcs accuracy format tag \
    --reward_weights 8 1 1 \
    --loss_type bnpo \
    --scale_rewards False \
    --mask_truncated_completions True \
    --epsilon 0.2 \
    --epsilon_high 0.3 \
    --temperature 1.0 \
    --top_p 0.95 \
    --eval_temperature 0.7 \
    --eval_top_p 0.95 \
    --repetition_penalty 1.1 \
    --beta 1e-5 \
    --lr_scheduler_type constant \
    --learning_rate 5e-6 \
    --save_strategy steps \
    --save_steps 100 \
    --eval_strategy steps \
    --eval_steps 10 \
    --eval_on_start False \
    --log_level info \
    --wandb_project simpleR1-train \
    --run_name $run_name \
    2>&1 | tee $LOG_FILE

Here is the plot, the metrics did increase at the beginning:

Here is the training curve:

The trained answers seem to repeat:

[yflyzhang]

Thanks very much for sharing your practice.

For a 1.5B model, there's no need to enable zero3—it doesn't save much memory and significantly slows down the training process. Please consider ddp first, and could try zero2 instead if memory is limited, though other parameters would need appropriate adjustments accordingly.

I'm sorry, but I can't reproduce the issue you're encountering.
In your training, the metric initially increases, indicating the model learns the correct direction at the beginning, but later fails to improve.
The setting --repetition_penalty 1.1 might be too high. Moreover, even with repetition_penalty=1.1, there's still excessive repetition, and the completions/truncated_ratio remains unusually high, which is unreasonable—something must be wrong with the training.

After applying different training strategies, key parameters may require retuning—although it's currently unclear why this adjustment is necessary.
I tried to use zero2 and only reduced the learning rate to 1e-6. The results are shown in the figure below. I think the performance can be further improved with proper parameters.
Moreover, training reasoning models are indeed highly sensitive to hyperparameters , making them significantly more challenging to tune compared to standard SFT. Small changes in learning rate, kl coefficient, number of completions, batch size, or even repetition penalty can lead to drastically different behaviors in reasoning performance and output quality.