Skip to content

fix(qwen35): trim legacy device pools between requests - #668

Merged
davide221 merged 3 commits into
Luce-Org:mainfrom
boxwrench:fix/qwen35-trim-device-pools
Aug 27, 2026
Merged

fix(qwen35): trim legacy device pools between requests#668
davide221 merged 3 commits into
Luce-Org:mainfrom
boxwrench:fix/qwen35-trim-device-pools

Conversation

@boxwrench

@boxwrench boxwrench commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an explicit trim hook for legacy CUDA/HIP device pools
  • release per-evaluation Q8 memo allocations before trimming cached blocks
  • trim target and distinct draft backends when Qwen3.5 request scratch is released
  • extend the existing CUDA pool-shutdown regression test through explicit trimming

Root cause

On non-VMM ROCm devices, long prefills returned operator temporaries to the legacy ggml pool, but the pool retained those blocks indefinitely. A later request with a different allocation shape could require another large contiguous allocation and fatally OOM even though the previous request had released its scratch tensors.

Validation

  • HIP gfx1201 release build completed for dflash_server and test_server_unit
  • two independent back-to-back requests completed on an AMD Radeon AI PRO R9700: 40,013 and 39,234 prompt tokens
  • request cleanup returned 2,405.3 MiB and 2,243.6 MiB from device pools respectively
  • unit suite: 421/423 passed in the restricted workspace; the two socket-permission failures passed when rerun with normal socket permissions
  • git diff --check clean

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
Comment thread server/test/test_cuda_pool_shutdown.cpp
@davide221

Copy link
Copy Markdown
Contributor

@boxwrench thanks for the contribution! We have fixed cubic issues.

@davide221
davide221 merged commit ea54481 into Luce-Org:main Aug 27, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants