Torch compile deepseekv2fw - #1
Open
mattteochen wants to merge 49 commits into
Open
Conversation
…ntal compile options for DeepSeek layers. Introduce new methods for handling key buffer storage and improve native compilation state preparation in the attention layers.
…ental prefill options. Enhance native compilation state preparation in the Indexer class and improve tensor management in DeepSeek attention methods.
Added model level compilation
…l compile options in DeepSeek. Added CLI argument for saving prefill logits, introduced validation for batch size/input length combinations, and integrated random seed handling for reproducibility. Updated environment variables for experimental compile features.
… and caching. Introduce context manager for managing experimental prefill compile mode in Deepseek, ensuring proper restoration of compile state. Enhance unit tests for compile mode transitions and callable reuse.
…ing and registration. Replace deprecated functions with new implementations and streamline the handling of `tune_max_num_tokens`. Update imports and remove unused code for better clarity and performance.
github-actions
Bot
added
documentation
…ds in NativeSparseAttnBackend; enhance tests for metadata handling
…duce fusion flag; add native compile readiness flag in NSAMetadata and related tests for metadata handling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ci