Skip to content

[ci] chore: npu ci use cann9.0.0#6520

Open
daikang6 wants to merge 34 commits into
verl-project:mainfrom
daikang6:main
Open

[ci] chore: npu ci use cann9.0.0#6520
daikang6 wants to merge 34 commits into
verl-project:mainfrom
daikang6:main

Conversation

@daikang6
Copy link
Copy Markdown
Contributor

@daikang6 daikang6 commented May 28, 2026

What does this PR do?

update npu's cann to 9.0.0 version

Checklist Before Starting

  • Search for similar PRs. Paste at least one query link here: ...
  • Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
    • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
    • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
    • {type} is in feat, fix, refactor, chore, test
    • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
    • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

@daikang6 daikang6 requested a review from wucong25 as a code owner May 28, 2026 11:57
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Ascend CI documentation to use a newer vLLM image version, configures the GRPO Qwen3-8B test script to calculate log probabilities, and simplifies the output log path for the PPO Qwen3-8B test script. A review comment points out that the log directory for the PPO script might not exist, which could cause the tee command to fail and result in lost CI logs. It is recommended to ensure the directory is created prior to writing the log.

trainer.max_actor_ckpt_to_keep=1 \
trainer.max_critic_ckpt_to_keep=1 \
trainer.total_training_steps=15 2>&1 | tee /root/.cache/nightly_log/qwen3-8b-ppo/ppo_qwen3-8b_fsdp_npu-$(date +%Y%m%d_%H%M).log No newline at end of file
trainer.total_training_steps=15 2>&1 | tee /root/.cache/nightly_log/ppo_qwen3_8b/ppo_qwen3-8b_fsdp_npu.log No newline at end of file
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the directory /root/.cache/nightly_log/ppo_qwen3_8b does not exist, the tee command will fail to write the log file, resulting in lost CI logs. To prevent this, ensure the directory is created before writing the log. Additionally, redirect stderr to stdout using 2>&1 before piping to tee to ensure that error messages are captured in the log file.

Suggested change
trainer.total_training_steps=15 2>&1 | tee /root/.cache/nightly_log/ppo_qwen3_8b/ppo_qwen3-8b_fsdp_npu.log
trainer.total_training_steps=15 2>&1 | { mkdir -p /root/.cache/nightly_log/ppo_qwen3_8b && tee /root/.cache/nightly_log/ppo_qwen3_8b/ppo_qwen3-8b_fsdp_npu.log; }
References
  1. Ensure that the log directory exists before writing to it in shell scripts.
  2. In shell scripts, redirect stderr to stdout using 2>&1 before piping to tee to ensure that error messages are captured in the log file.

permissions:
contents: read

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

npu unit test加上
import verl.utils.device as device_module
monkeypatch.setattr(device_module, "is_npu_available", False)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants