Repository navigation
HIP: release cancelled prompt chunks at layer boundaries - #1028
Open
ThinkOffApp wants to merge 3 commits into
Open
ThinkOffApp wants to merge 3 commits into
ThinkOffApp wants to merge 3 commits into
Conversation
Pass the lane cancellation hook to both prompt-step paths and enable layer polling for the 512/1024-row chunks used by the server. Keep tensor-parallel groups and short passes on their existing between-chunk cancellation path. Add CPU wiring/policy regressions and a served 27B gate which checks disconnect-to-release time, short-request latency and full native token hashes against an idle baseline.
Keep the fresh unmodified-versus-fixed 27B comparison, complete native token hashes, checkpoint pins and exact build/server commands. Separate disconnect-to-idle timing from request lifetime and mark the two unexercised drafting comparisons. Publish a sanitized copy with checksums while retaining original evidence privately.
Use short source comments and docstrings; retain the full methodology in the receipt. Pin the script hash to its measured commit. Runtime and gate execution logic are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cancelling a long single-GPU HIP Qwen prompt left the current 512/1,024-row chunk running through every layer: the lane's cancellation hook was not passed to the engine, and the engine's old 2,048-row polling threshold exceeded those chunks. The next short request therefore waited for the cancelled work.
Pass the stream cancellation hook to both prompt-step paths and poll at layer boundaries for single-rank passes of at least 512 rows. Shorter passes retain between-chunk cancellation. Tensor-parallel groups keep their existing behavior so ranks cannot diverge at a collective.
Fresh gfx1151/Qwen3.8-27B A/B results, three trials per build:
76aafcf25429554eAll fourteen short replies have the same complete native token SHA-256. An identical uncancelled 14,975-token prompt also has identical native tokens; its prefill takes 79.021 s before and 78.867 s after (one timing control, not a broad performance claim). All twelve cancelled prompts fit the 16,384-token window and end as client-left with zero emitted tokens. Request lifetime is recorded separately from disconnect-to-idle time.
Validation:
serve_cancel_release.pygate checks admission, release latency, the following short request, and complete native token hashes; it retains the explicit historical 27B comparison behind--baseline-27b.70ae7fac63274ff2eac54152031433374cb80f2f. Build/server commands, binary hashes, raw exchanges, original failing control and sanitization checksums are in the receipt directory.Follow-up to the unresolved cancellation case in #994. This is scoped single-GPU 27B evidence, not a complete release qualification or a change to drafter support.
Current CI: source comments/docstrings were shortened to satisfy the lean rule (local comparison: zero new violations). Windows CI reports socket/dynamic-loader compile errors in unchanged platform paths, including
speech.zig,cuda_rank_wire.zigandcuda_load_gpu.zig; the same errors are visible in an earlier unrelated run that predates this PR. Windows qualification is not claimed.