Skip to content

HIP: release cancelled prompt chunks at layer boundaries - #1028

Open
ThinkOffApp wants to merge 3 commits into
ashhart:mainfrom
ThinkOffApp:amd-cancel-release
Open

ThinkOffApp wants to merge 3 commits into
ashhart:mainfrom
ThinkOffApp:amd-cancel-release

Conversation

@ThinkOffApp

@ThinkOffApp ThinkOffApp commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Cancelling a long single-GPU HIP Qwen prompt left the current 512/1,024-row chunk running through every layer: the lane's cancellation hook was not passed to the engine, and the engine's old 2,048-row polling threshold exceeded those chunks. The next short request therefore waited for the cancelled work.

Pass the stream cancellation hook to both prompt-step paths and poll at layer boundaries for single-rank passes of at least 512 rows. Shorter passes retain between-chunk cancellation. Tensor-parallel groups keep their existing behavior so ranks cannot diverge at a collective.

Fresh gfx1151/Qwen3.8-27B A/B results, three trials per build:

Measurement Public main 76aafcf2 Fixed runtime 5429554e
Idle short request 1.899 s 1.913 s
Disconnect to engine idle metric 4.720–4.744 s 0.032–0.042 s
Next short request 6.532–6.548 s 1.902–1.907 s
Release/short-latency gate 3/3 fail 3/3 pass

All fourteen short replies have the same complete native token SHA-256. An identical uncancelled 14,975-token prompt also has identical native tokens; its prefill takes 79.021 s before and 78.867 s after (one timing control, not a broad performance claim). All twelve cancelled prompts fit the 16,384-token window and end as client-left with zero emitted tokens. Request lifetime is recorded separately from disconnect-to-idle time.

Validation:

  • Both new CPU regressions fail with the old null hook/threshold and pass with the fix. The host test build passes 292/292 steps with cached results reused; its fresh results report 126 passes and two skips.
  • The new serve_cancel_release.py gate checks admission, release latency, the following short request, and complete native token hashes; it retains the explicit historical 27B comparison behind --baseline-27b.
  • The fixed server passes 13 exercised upstream served comparisons. Two drafting comparisons are unexercised because this checkpoint has no draft head.
  • All 17 checkpoint files match revision 70ae7fac63274ff2eac54152031433374cb80f2f. Build/server commands, binary hashes, raw exchanges, original failing control and sanitization checksums are in the receipt directory.

Follow-up to the unresolved cancellation case in #994. This is scoped single-GPU 27B evidence, not a complete release qualification or a change to drafter support.

Current CI: source comments/docstrings were shortened to satisfy the lean rule (local comparison: zero new violations). Windows CI reports socket/dynamic-loader compile errors in unchanged platform paths, including speech.zig, cuda_rank_wire.zig and cuda_load_gpu.zig; the same errors are visible in an earlier unrelated run that predates this PR. Windows qualification is not claimed.

Pass the lane cancellation hook to both prompt-step paths and enable layer polling for the 512/1024-row chunks used by the server. Keep tensor-parallel groups and short passes on their existing between-chunk cancellation path. Add CPU wiring/policy regressions and a served 27B gate which checks disconnect-to-release time, short-request latency and full native token hashes against an idle baseline.
Keep the fresh unmodified-versus-fixed 27B comparison, complete native token hashes, checkpoint pins and exact build/server commands. Separate disconnect-to-idle timing from request lifetime and mark the two unexercised drafting comparisons. Publish a sanitized copy with checksums while retaining original evidence privately.
Use short source comments and docstrings; retain the full methodology in the receipt. Pin the script hash to its measured commit. Runtime and gate execution logic are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant