Skip to content

[NPU][Bug] check_mtp_only_grad fails on GLM-4.7-Flash #421

Description

@Fulin-Gao

check_mtp_only_grad fails on GLM-4.7-Flash

Symptom

tests/test_glm4.7_30B_A3B_npu.py (CI build) fails at Step 0 on the isolation assertion
assert len(non_mtp_nonzero_grads) == 0 in vime-ascend/vime/backends/megatron_utils/ci_utils.py:

AssertionError: Expected all non-MTP parameters to have zero gradients, but found 1302 with non-zero gradients.

Step 0 gradient stats:

[CI MTP Grad Check] Step 0: MTP params with non-zero grad: 32,
non-MTP params with non-zero grad: 1302
[CI MTP Grad Check] Non-MTP param with non-zero grad:
  module.module.embedding.word_embeddings.weight,        max_grad=1.02e-03   ← largest
  module.module.decoder.layers.0.input_layernorm.weight, max_grad=1.00e-04
  module.module.decoder.layers.0.self_attention.linear_proj.weight,                   max_grad=9.5e-06
  module.module.decoder.layers.0.self_attention.linear_q_down_proj.weight,             max_grad=9.6e-06
  module.module.decoder.layers.0.self_attention.linear_q_up_proj.layer_norm_weight,   max_grad=1.87e-05

Root Cause

check_mtp_only_grad runs unconditionally every step, but its assertion len(non_mtp_nonzero_grads) == 0 only holds under the docstring's "all outputs truncated" precondition. When rewards differ across samples in a GRPO group, advantages are non-zero, the policy loss is non-zero, and gradients flow to non-MTP params (embedding, decoder layers, ...), so the assertion fails.

Reproduction

Test entry: tests/test_glm4.7_30B_A3B_npu.py (16×NPU, ray job submit).
Key args:

--ci-test 
--enable-mtp-training 
--rollout-max-response-len 2048

Impact / Suggestions

  • Affected
    The assertion assert len(non_mtp_nonzero_grads) == 0 in ci_utils.py::check_mtp_only_grad produces a false positive whenever rewards differ within a GRPO group.
  • Candidate directions
    Reduce the max output length (e.g. lower --rollout-max-response-len) so all rollout responses truncate, satisfying the assertion's all-truncated precondition.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions