Installing aphrodite-engine on Ubuntu 24.04 server with Cuda 12.8 and RTX 5080

[marc-ramsey]

I've been trying for several days now to get what should (?) have been a simple installation to work. However, I need to use at least Cuda 12.8 (earliest with 5080 support), which complicates matters. Using the default NVidia driver installation instructions results in cuda r12.9, which doesn't appear to be supported by PyTorch yet Forcing use of cuda r12.8 for a headless installation apparently requires the following gyrations:

sudo apt purge dkms *nvidia*
sudo apt install linux-headers-$(uname -r)
sudo apt update
sudo apt install nvidia-headless-570-server-open
sudo apt install nvidia-utils-570-server
sudo apt install nvidia-cuda-dev
sudo apt install cuda-toolkit-12-8

That all seems to work, nvidia-smi and clinfo detect the GPU correctly, stand alone installation of torch 2.7 works fine. So:

git clone https://github.com/PygmalionAI/aphrodite-engine.git
cd aphrodite-engine/
python3 -m venv venv
source venv/bin/activate
pip install -r requirements-cuda.txt
pip install wheel ninja==1.10
export PATH=/usr/local/cuda-12.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH
export CUDA_HOME=/usr/local/cuda-12.8
MAX_JOBS=4 pip install -e .

Clean build, no problem, how about aphrodite run Qwen/Qwen2.5-Coder-7B-Instruct:

/home/pfunk/project/aphrodite-engine/venv/lib/python3.12/site-packages/torch/cuda/__init__.py:235: UserWarning: 
NVIDIA GeForce RTX 5080 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.

As far as I can figure out, the main issue here is that requirements-common.txt requires PyTorch 2.6 which is built with Cuda 12.4 libraries, no sm_120. The PyTorch.org nightly build is 2.7 built with Cuda 12.8 which does have sm_120 (and it works), but is not compatible with building the current aphrodite-engine source. How do I get around this? Thanks, Marc