Skip to content

perf(qwen): project only the final token during trainside generation - #442

Merged
CjhHa1 merged 2 commits into
Tencent-Hunyuan:mainfrom
Yuanjia-Lee:perf/qwen3-keep
Sep 13, 2026
Merged

CjhHa1 merged 2 commits into
Tencent-Hunyuan:mainfrom
Yuanjia-Lee:perf/qwen3-keep

Conversation

@Yuanjia-Lee

@Yuanjia-Lee Yuanjia-Lee commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Qwen trainside autoregressive loops only sample logits[:, -1, :], but prefill projected every prompt position through lm_head. Pass the models' native logits_to_keep=1 contract so Qwen3, Qwen2.5-VL, and Qwen3.5 materialize only [B, 1, vocab] logits.

Qwen3.5 applies the same selection on its cacheless VeOmni path, where the full growing sequence is forwarded each step. Replay remains unchanged because it needs every response-token log-probability.

Related Issue

N/A

Test Plan

  • ruff check / ruff format --check / python lint/check_docstring_lines.py on the three changed AR modules — pass
  • Qwen3-0.6B, Transformers 5.6: last-token logits and KV match the full forward within tolerance; prefill peak 2.50 -> 1.39 GiB (B=4, L=512, bf16)
  • Transformers 5.6 tiny-model checks:
    • Qwen2.5-VL: final-token logits match the full projection within tolerance; KV length unchanged
    • Qwen3.5 cacheless path: final-token logits match the full projection within tolerance
  • Live PE train: not run; sampling and cache-update paths are unchanged, and this matches the native generate() behavior.

Compatibility / Risk

No config, checkpoint, state-dict, or public API changes. The repository pins Transformers 5.6, and the HF plus VeOmni Qwen forwards all support logits_to_keep. Prompts are left-padded, so the final row is each sample's last real token.

Reviewer Notes

Open #430 edits qwen3/ar.py on the replay exact-actor path, not these generation-loop arguments.

Checklist

  • I reviewed the changed code and removed unrelated/generated artifacts.
  • I updated tests, docs, and configs where needed, or explained why not.

HF generate() already sets logits_to_keep=1; the custom trainside loop
only reads [:, -1, :] but still projected the full prefill [B, L, vocab].
@github-actions github-actions Bot added the need review Ready and waiting for review label Sep 12, 2026
@CjhHa1
CjhHa1 self-requested a review as a code owner September 13, 2026 04:21
@CjhHa1 CjhHa1 changed the title perf(qwen3): project only the final token during trainside generation perf(qwen): project only the final token during trainside generation Sep 13, 2026

@CjhHa1 CjhHa1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added approved Approved by reviewer and removed need review Ready and waiting for review labels Sep 13, 2026
@CjhHa1
CjhHa1 merged commit 674ebc6 into Tencent-Hunyuan:main Sep 13, 2026
7 of 8 checks passed
@github-actions github-actions Bot removed the approved Approved by reviewer label Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants