check_mtp_only_grad fails on GLM-4.7-Flash
Symptom
tests/test_glm4.7_30B_A3B_npu.py (CI build) fails at Step 0 on the isolation assertion
assert len(non_mtp_nonzero_grads) == 0 in vime-ascend/vime/backends/megatron_utils/ci_utils.py:
AssertionError: Expected all non-MTP parameters to have zero gradients, but found 1302 with non-zero gradients.
Step 0 gradient stats:
[CI MTP Grad Check] Step 0: MTP params with non-zero grad: 32,
non-MTP params with non-zero grad: 1302
[CI MTP Grad Check] Non-MTP param with non-zero grad:
module.module.embedding.word_embeddings.weight, max_grad=1.02e-03 ← largest
module.module.decoder.layers.0.input_layernorm.weight, max_grad=1.00e-04
module.module.decoder.layers.0.self_attention.linear_proj.weight, max_grad=9.5e-06
module.module.decoder.layers.0.self_attention.linear_q_down_proj.weight, max_grad=9.6e-06
module.module.decoder.layers.0.self_attention.linear_q_up_proj.layer_norm_weight, max_grad=1.87e-05
Root Cause
check_mtp_only_grad runs unconditionally every step, but its assertion len(non_mtp_nonzero_grads) == 0 only holds under the docstring's "all outputs truncated" precondition. When rewards differ across samples in a GRPO group, advantages are non-zero, the policy loss is non-zero, and gradients flow to non-MTP params (embedding, decoder layers, ...), so the assertion fails.
Reproduction
Test entry: tests/test_glm4.7_30B_A3B_npu.py (16×NPU, ray job submit).
Key args:
--ci-test
--enable-mtp-training
--rollout-max-response-len 2048
Impact / Suggestions
- Affected
The assertion assert len(non_mtp_nonzero_grads) == 0 in ci_utils.py::check_mtp_only_grad produces a false positive whenever rewards differ within a GRPO group.
- Candidate directions
Reduce the max output length (e.g. lower --rollout-max-response-len) so all rollout responses truncate, satisfying the assertion's all-truncated precondition.
check_mtp_only_grad fails on GLM-4.7-Flash
Symptom
tests/test_glm4.7_30B_A3B_npu.py(CI build) fails at Step 0 on the isolation assertionassert len(non_mtp_nonzero_grads) == 0invime-ascend/vime/backends/megatron_utils/ci_utils.py:Step 0 gradient stats:
Root Cause
check_mtp_only_gradruns unconditionally every step, but its assertionlen(non_mtp_nonzero_grads) == 0only holds under the docstring's "all outputs truncated" precondition. When rewards differ across samples in a GRPO group, advantages are non-zero, the policy loss is non-zero, and gradients flow to non-MTP params (embedding, decoder layers, ...), so the assertion fails.Reproduction
Test entry:
tests/test_glm4.7_30B_A3B_npu.py(16×NPU, ray job submit).Key args:
Impact / Suggestions
The assertion
assert len(non_mtp_nonzero_grads) == 0inci_utils.py::check_mtp_only_gradproduces a false positive whenever rewards differ within a GRPO group.Reduce the max output length (e.g. lower
--rollout-max-response-len) so all rollout responses truncate, satisfying the assertion's all-truncated precondition.