fix(deps): use PyTorch 2.13 to fix SM89 hangs - #184
Open
endenis wants to merge 1 commit into
Open
Conversation
endenis
marked this pull request as ready for review
August 25, 2026 18:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #182
Possibly related to #72 and #123
Summary
PyTorch 2.11 can hang indefinitely during prefill on SM89 GPUs.
I reproduced this with
Qwen3.6-35B-A3B-NVFP4on an RTX 4070: the request stops making progress while GPU utilization remains near 100%.This PR updates FreeToken to PyTorch 2.13 and along with these dependencies:
Root cause
I isolated the non-completing GPU submission to the fused QKV projection’s row-wise FP8
torch._scaled_mmcall:CUDA_LAUNCH_BLOCKING=1avoided the hang but only by changing the timing (not a real fix).The behavior matches an upstream PyTorch SM89 bug fixed by pytorch/pytorch@252bb4a
The fix is present in PyTorch 2.12 and 2.13.
I made two reproduction scripts:
You can find them in the description of the #182 issue.
Changes
3.7.1(required by PyTorch 2.13).sglang-kernelto0.4.6.post1which is built against the PyTorch 2.13.freetoken-kernel-cache.There are no kernel or synchronization workarounds in this change.
Testing
Tested on:
Standalone
_scaled_mmreproducer (attached in the #182 issue):Notes
I only have access to an RTX 4070. It would be useful to test on other cards as well.
Don't hesitate to reach out if you need more details