Skip to content

PR4: on-demand CUDA graph capture extension - #4

Open
Astrangemaninhere wants to merge 1 commit into
masterfrom
pr4/on-demand-graphs
Open

PR4: on-demand CUDA graph capture extension#4
Astrangemaninhere wants to merge 1 commit into
masterfrom
pr4/on-demand-graphs

Conversation

@Astrangemaninhere

@Astrangemaninhere Astrangemaninhere commented Aug 31, 2026

Copy link
Copy Markdown
Owner

PR4: on-demand CUDA graph capture extension

Background

The ordinary decode CUDA-graph ladder was captured for the full window range up front, which is slow and memory-hungry at very long contexts.

Changes

  • On-demand capture: extends the ordinary decode ladder only as far as needed instead of capturing the full range at startup.
  • Reduces capture time and graph memory; the ladder grows as the active window grows.

Files

  • src/runtime/engine/shared_capture_planner.h (capture target planning)
  • src/targets/qwen3_6/impl/runtime/graph_impl.h (decode ladder)

Dependency

Depends on PR1-3. Merge after PR3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant