Is it possible to apply hipMM to vLLM sleep mode?

[YangWang92]

Hi all,
Thanks for sharing the project! In vLLM, we need a special sleep mode to offload model weights and kv cache to cpu memory.
Our current implementation is based on chunk-wised memory allocation to avoid some bugs in rocm.
vllm-project/vllm#12695

Is it possible to apply hipMM to vLLM sleep mode to avoid these bugs?

Thanks!

[domcharrier]

Hi @YangWang92 ,

Thanks for your question.

I tried reading through the PR that you linked but as I have limited experience with vLLM's design I had some trouble following
the discussion.

Can you summarize the state of the PR?

• Does it only work with MI300X?

• Which ROCm versions are required? 6.3.4 / 6.4.0 / ... ?

• What HIP APIs are you using and which cause trouble (presumably)?

  I am assuming the ones below :

   cuda_to_hip_mapping = {
        "cudaSetDevice": "hipSetDevice",
        "cudaDeviceSynchronize": "hipDeviceSynchronize",
        "cudaDeviceReset": "hipDeviceReset",
        "cudaGetErrorString": "hipGetErrorString",
        "cudaMalloc": "hipMalloc",
        "cudaFree": "hipFree",
        "cudaMemset": "hipMemset",
        "cudaMemcpy": "hipMemcpy",
        "cudaIpcGetMemHandle": "hipIpcGetMemHandle",
        "cudaIpcOpenMemHandle": "hipIpcOpenMemHandle",
  }

  • Does it only work for particular chunk sizes? Why are you using chunks? Which HIP API are affected by bugs if you don't use chunks?
  • What kind of instance are you using (bare metal/docker/VM)? What OS?

• What do you expect from using hipMM as allocator instead of HIP?
  (I am assuming reduced fragementation, less frequent usage of HIP APIs.)

• Is it fine to pull in additional dependencies (like hipMM) into vLLM?