[Feature] Add DAPO preset and Soft Overlong support

[CalvinXKY]

Summary

• Expose --advantage-estimator dapo as a preset over the existing GRPO implementation.
• Enable Clip-Higher, token-level loss, the default dynamic-sampling filter, and Soft Overlong defaults.
• Add --soft-overlong-cache and apply its length penalty before existing reward normalization.
• Document the preset in English and Chinese.

Design

DAPO does not add a second advantage, loss, or normalization path. Argument validation expands it to the existing GRPO primitives; the only new rollout behavior is Soft Overlong reward shaping.

Test plan

• Pre-commit on all changed files
• pytest tests/test_megatron_argument_validation.py (18 passed)
• Local DAPO preset smoke validation
• Author GPU/NPU smoke evidence in PR comments

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #34276655 | 📁 Comparing 0851942 against latest (c0ed6d8)

  🔍 Preview build  

46 files changed · + 1 added · ± 43 modified · - 2 deleted

+ Added

• advanced/on-policy-distillation.html

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 33 more...

- Deleted

• examples/gemma4.html
• platform_support/ascend_tutorial.html

[gemini-code-assist]

Code Review

This pull request introduces support for the DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) algorithm, adding its documentation, configuration arguments, default recipe validation, and the implementation of Soft Overlong Punishment in reward post-processing. The review feedback points out potential TypeError crashes during argument validation and rollout if rollout_max_response_len is not specified while DAPO or Soft Overlong is enabled, and suggests adding explicit checks and clearer error messages to prevent these issues.

[gemini-code-assist]

If --rollout-max-response-len is not specified (i.e., it is None), calling int(args.rollout_max_response_len) will raise a TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'. Since DAPO defaults to enabling Soft Overlong, we should check if args.rollout_max_response_len is None and raise a clear ValueError to guide the user.

        if args.soft_overlong_cache is None:
            if args.rollout_max_response_len is None:
                raise ValueError("DAPO requires --rollout-max-response-len to be set to enable Soft Overlong.")
            args.soft_overlong_cache = max(1, int(args.rollout_max_response_len) // 4)

[gemini-code-assist]

If the user explicitly sets --soft-overlong-cache to a positive value but does not specify --rollout-max-response-len, a TypeError will occur later during rollout when calculating L_max. We should validate this condition here and raise a clear ValueError.

    if args.soft_overlong_cache is not None:
        if args.soft_overlong_cache < 0:
            raise ValueError(f"--soft-overlong-cache must be >= 0, got {args.soft_overlong_cache}.")
        if args.soft_overlong_cache > 0 and args.rollout_max_response_len is None:
            raise ValueError("--rollout-max-response-len must be set when --soft-overlong-cache is enabled.")

[CalvinXKY]

NPU tests:

[CalvinXKY]

GPU tests:

• Qwen3-4B
• vime (commit 5cabf1f)
• vllm/vime:latest

[aoshen02]

I think most of the ability of dapo is already support in vime, and for the soft overlong cache, I would say we recommend just use the --custom-reward-post-process-path instead of merging it, I'm still ok with merging this amount of code if it is needed.

[CalvinXKY]

OK. Let's leave this as the solution for now.