Skip to content

feat: integrate VeRL with native Hyper training - #1609

Open
uhuuh wants to merge 1 commit into
mindspore-ai:masterfrom
uhuuh:debug/20261118-rebase-master-verl
Open

uhuuh wants to merge 1 commit into
mindspore-ai:masterfrom
uhuuh:debug/20261118-rebase-master-verl

Conversation

@uhuuh

@uhuuh uhuuh commented Nov 18, 2026 •

Copy link
Copy Markdown

Paired: GitHub #1609 ↔ GitCode !1983

What type of PR is this?

/kind feature

What does this PR do / why do we need it:

新增 VeRL 的 Hyper 后端,使 VeRL 能使用 Hyper 的模型构建、并行执行和优化器能力完成训练。

  • Engine 负责 VeRL 批次适配、log-prob/entropy 输出转换和 loss;Trainer 复用 Hyper 执行组件,并在 train_batch() 中统一管理训练步生命周期,仅训练阶段触发训练回调。
  • 通过 HyperEngineConfig 加载原生 recipe,支持使用 trainer_config 覆盖配置;在 patch 包中注册后端及 VeRL 兼容修复,无需修改 VeRL 源码。
  • 复用 Hyper checkpoint 和权重转换接口,提供 VeRL 检查点适配及逐参数 rollout 权重导出。
  • 完善 Qwen3-MoE 的 attention norm 替换与 RoPE buffer 物化,补充启动示例、中英文文档及测试。

实验信息

项目 配置
NPU 16 张 Ascend NPU
模型 Qwen3-30B-A3B-Base,完整 48 层,MoE 架构
训练数据集 DAPO-Math-17k
验证数据集 AIME-2024,30 条样本
MindSpeed 并行 TP=2、CP=4、EP=8、ETP=1
Hyper 并行 TP=2、CP=4、EP=8
Batch size 16 个 prompt/步
采样数 n 每个 prompt 生成 16 条 response,即每步 256 条训练序列
最大输入长度 2,048 token
最大输出长度 8,192 token
训练步数 10 步

对比汇总

全 10 步中,training/rollout_actor_probs_pearson_corr 的范围为 MindSpeed 0.999066234–0.999530435、Hyper 0.998790264–0.999674618,波动幅度分别为 0.000464201 和 0.000884354。actor/grad_norm 的范围为 MindSpeed 0.000000–0.239585、Hyper 0.000000–0.515615,波动幅度分别为 0.239585 和 0.515615。波动幅度按最大值减最小值计算。各后端的 Pearson 相关系数比较其 actor 与自身 rollout 概率。

training/rollout_actor_probs_pearson_corr

step MindSpeed Hyper Hyper 相对变化
1 0.999312639 0.999674618 +0.0362%
2 0.999262452 0.999226153 -0.0036%
3 0.999303222 0.999322534 +0.0019%
4 0.999393404 0.999428868 +0.0035%
5 0.999441326 0.999516368 +0.0075%
6 0.999162316 0.999419987 +0.0258%
7 0.999530435 0.999190032 -0.0341%
8 0.999443293 0.999506414 +0.0063%
9 0.999066234 0.998790264 -0.0276%
10 0.999497712 0.999280155 -0.0218%

actor/grad_norm

step MindSpeed Hyper Hyper 相对变化
1 0.000000 0.000000 —
2 0.000000 0.000000 —
3 0.239585 0.089249 -62.75%
4 0.236099 0.172334 -27.01%
5 0.000000 0.156113 —
6 0.000000 0.000000 —
7 0.000000 0.000000 —
8 0.000000 0.000000 —
9 0.000000 0.515615 —
10 0.054139 0.000000 —

grad norm 任一侧为 0 时,相对变化记为「—」。

首步与后续各步差异较大,下表为第 2 步至第 10 步的汇总数据对比。

指标 统计 MindSpeed Hyper Hyper 相对变化
actor/perf/max_memory_allocated_gb max 47.991 41.369 -13.80%
actor/perf/max_memory_reserved_gb max 49.004 42.824 -12.61%
actor/perf/cpu_memory_used_gb mean 986.503 797.242 -19.19%
actor/perf/cpu_memory_used_gb max 991.369 801.606 -19.14%
timing_s/old_log_prob mean 25.860 27.092 +4.76%
timing_s/update_actor mean 67.209 58.324 -13.22%
timing_s/gen mean 117.465 108.423 -7.70%
timing_s/update_weights mean 14.728 23.164 +57.28%
timing_s/step mean 225.827 217.574 -3.65%
perf/total_num_tokens mean 248556.1 254082.3 +2.22%
timing_per_token_ms/update_actor mean 0.2757 0.2314 -16.08%

相对变化以 MindSpeed 为基准。

逐步明细

以下列出第 2–10 步。M = MindSpeed,H = Hyper。

输入与优势

step perf/total_num_tokens M / H critic/advantages/max M / H
2 250052.000000 / 236069.000000 0.000000 / 0.000000
3 243202.000000 / 229414.000000 0.652791 / 2.015562
4 191622.000000 / 255110.000000 3.749992 / 1.097887
5 198480.000000 / 234541.000000 0.000000 / 0.465130
6 189874.000000 / 250574.000000 0.000000 / 0.000000
7 342113.000000 / 225259.000000 0.000000 / 0.000000
8 347968.000000 / 393074.000000 0.000000 / 0.000000
9 246189.000000 / 195869.000000 0.000000 / 2.015562
10 227505.000000 / 266831.000000 1.436139 / 0.000000

资源占用

前两列为 Device 峰值显存,最后一列为 Host 整机已用内存,单位均为 GiB。

step actor/perf/max_memory_allocated_gb M / H actor/perf/max_memory_reserved_gb M / H actor/perf/cpu_memory_used_gb M / H
2 47.931396 / 41.320365 48.917969 / 42.667969 987.284691 / 791.737873
3 47.931396 / 41.369367 48.917969 / 42.667969 987.349358 / 798.308437
4 47.931396 / 41.369367 48.917969 / 42.667969 984.387791 / 797.504910
5 47.931396 / 41.369367 48.917969 / 42.667969 983.187889 / 794.614159
6 47.931396 / 41.369367 48.917969 / 42.746094 984.088215 / 799.136150
7 47.991244 / 41.369367 49.003906 / 42.746094 986.019829 / 795.146057
8 47.991244 / 41.369367 49.003906 / 42.746094 985.739105 / 795.664471
9 47.991244 / 41.369367 49.003906 / 42.824219 991.369446 / 801.606068
10 47.991244 / 41.369367 49.003906 / 42.824219 989.101147 / 801.457024

阶段性能

step timing_s/old_log_prob M / H timing_s/update_actor M / H timing_s/gen M / H timing_s/update_weights M / H timing_s/step M / H timing_per_token_ms/update_actor M / H
2 25.177786 / 32.138132 65.476065 / 56.166692 121.474679 / 81.300179 15.798291 / 23.243434 228.477301 / 193.392977 0.261850 / 0.237925
3 25.420530 / 24.403085 65.344339 / 54.138557 107.279899 / 103.148864 14.340320 / 23.081515 212.954523 / 205.343607 0.268683 / 0.235986
4 22.842102 / 26.332608 56.321012 / 58.353568 65.330091 / 79.030850 14.585898 / 21.504134 159.619417 / 185.769339 0.293917 / 0.228739
5 23.969463 / 24.306245 64.670307 / 53.037816 158.200290 / 125.259061 14.352130 / 27.842150 261.789101 / 231.036268 0.325828 / 0.226135
6 22.861568 / 28.049850 57.460046 / 57.834322 65.435845 / 156.936033 14.637740 / 22.609800 160.934849 / 266.033984 0.302622 / 0.230807
7 31.384507 / 23.565130 82.001061 / 55.342288 165.659365 / 63.006272 14.409019 / 23.353274 294.057073 / 165.807904 0.239690 / 0.245683
8 31.951510 / 36.427055 83.924327 / 83.306443 173.383440 / 167.123652 14.602994 / 22.299617 304.448964 / 309.760870 0.241184 / 0.211936
9 25.139998 / 22.539706 67.394346 / 48.012667 127.332763 / 95.195822 15.176850 / 21.156555 235.599154 / 187.471611 0.273750 / 0.245126
10 23.996364 / 26.064629 62.293555 / 58.723932 73.091344 / 104.810145 14.646908 / 23.386558 174.565599 / 213.544999 0.273812 / 0.220079

Which issue(s) this PR fixes:

无。

Code review checklist [illustration]:

  • whether to verify the function's return value
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid
  • Whether the API change is involved:新增 VeRL Hyper Engine 及配置入口。
  • Whether official document modification is involved:新增中英文集成说明。

@CLAassistant

CLAassistant commented Nov 18, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@uhuuh
uhuuh force-pushed the debug/20261118-rebase-master-verl branch from d1c5593 to b08f81b Compare November 18, 2026 09:51
@uhuuh uhuuh changed the title feat: integrate VeRL with native Hyper training [wip]feat: integrate VeRL with native Hyper training Nov 18, 2026
@uhuuh
uhuuh force-pushed the debug/20261118-rebase-master-verl branch 3 times, most recently from 46a5d08 to ce98541 Compare November 19, 2026 08:41
@uhuuh uhuuh changed the title [wip]feat: integrate VeRL with native Hyper training feat: integrate VeRL with native Hyper training Nov 19, 2026
@uhuuh
uhuuh force-pushed the debug/20261118-rebase-master-verl branch from ce98541 to 23280cc Compare November 19, 2026 09:19
@uhuuh

uhuuh commented Nov 19, 2026

Copy link
Copy Markdown
Author

/check-pr

@uhuuh

uhuuh commented Nov 19, 2026

Copy link
Copy Markdown
Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1820 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1820
Check_Pylint FAILURE 1820

@mindspore-ai-bot

Copy link
Copy Markdown

@uhuuh, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

@uhuuh

uhuuh commented Nov 19, 2026

Copy link
Copy Markdown
Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1824 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1824
Check_Pylint FAILURE 1824

@mindspore-ai-bot

Copy link
Copy Markdown

@uhuuh, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

c3f5e7809eaa485549a45626184f50888f694459 feat: integrate VeRL with native Hyper training
137cadb3a17a34333d090cb41dc596ee3d70bd33 fix: align VeRL Hyper configuration and document integration usage
3ce7798fb17b7f8afa460b5766873af1af481816 test: cover Hyper engine validation and optional entropy chunking
54607e7517d2671a31257bb4b1c34f081f44d85b fix: quote VeRL checkpoint paths for Hydra overrides
e60f573effc8e2093be3a4a2f1ef95297871ccb3 Revert "fix: quote VeRL checkpoint paths for Hydra overrides"
67a18cbfd0a1a5226900d6eb8096be7b77b1e4ce docs: update the VeRL integration guides
1d940b814a91c0a0b0272c2cf0cabe69de17e764 fix: align VeRL actor progress with optimizer steps
@uhuuh
uhuuh force-pushed the debug/20261118-rebase-master-verl branch from c7c522e to bc708a2 Compare November 19, 2026 16:34
@uhuuh

uhuuh commented Nov 19, 2026

Copy link
Copy Markdown
Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1838 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1838
Check_Pylint FAILURE 1824

@mindspore-ai-bot

Copy link
Copy Markdown

@uhuuh, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants