fix(hcu): avoid full-vocabulary top-k during sampler profiling - #41
Draft
alexanderbin123 wants to merge 3 commits into
Draft
alexanderbin123 wants to merge 3 commits into
alexanderbin123 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
该 MR 从 #38 拆出,仅讨论 HCU dummy sampler 的大词表启动 profile 策略,不包含 SlimQuant W4A8、AITER、DeepEP 或 DeepGEMM 修改。
背景
官方及 HYGON
v0.25.1在_dummy_sampler_run()中使用top_k=vocab_size-1和top_p=0.9。在 DeepSeek-V4 大词表、max_num_seqs=8、HCU 自定义 Top-K/Top-P sampler 场景中,该 profile 调用曾出现长时间卡顿/高开销。当前草案:
top_k=0,避开首次 full-vocabulary Top-K。top_k=vocab_size,进入 native random sampler;vocab_size是 vLLM native sampler 表示禁用 Top-K 的内部值。raw_logprobs、processed_logits、processed_logprobs三种模式。Draft 原因 / 已知风险
该实现尚不具备合入条件:
dummy_metadata改成 all-greedy 后,DSpark rejection sampler 会复用它,可能只预热 greedy specialization,而没有覆盖原来的 mixed/random specialization。top_p=0.9,native 路径仍可能进行全词表排序;本修改只规避首个 HCU custom Top-K 的异常开销,不等于消除所有全词表 sampler 开销。合入前需要重新设计并验证:分别覆盖 LightOp custom、native seeded random、DSpark mixed 与 all-greedy,同时给出 DeepSeek-V4 实机启动、峰值显存及首请求证据。
Verification
7 passed, 14 warnings in 23.12s。857 passed, 1 failed, 14 warnings in 219.87s。vllm_hcu/csrc/hcu_cache_kernel.hip时触发FileNotFoundError,与本 MR 的 sampler diff 无关;该文件被仓库*.hipignore 规则排除。zhangzbb <1414695739@qq.com>。Scope
修改文件仅有:
vllm_hcu/v1/hcu_model_runner.pytests/runtime_patch/test_hcu_sampler.py