From 6c8766cb5fdb8a3d224283ebd5453b504ff7f054 Mon Sep 17 00:00:00 2001 From: Brian Gebel Date: Wed, 27 May 2026 08:09:39 -0700 Subject: [PATCH] fix(complete): add cu130 extra-index-url to extras pip install MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prevents pip from replacing the base image's cu130-built torchaudio wheel with a stock cu128 wheel from PyPI when transitive deps in extra-requirements.txt trigger re-resolution. ## Symptom this fixes After #45 added transformers/peft/sentencepiece/sam2 etc., a production K8s pod failed to start with: RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions. PyTorch has CUDA version 13.0 whereas TorchAudio has CUDA version 12.8. Please install the TorchAudio version that matches your PyTorch version. Pod went into CrashLoopBackOff; downstream harmony repo had to revert the image SHA bump (ductiletoaster/harmony#384) to recover. ## Root cause The complete stage's `pip install -r extra-requirements.txt` had no `--extra-index-url`. The base image's PyTorch is built against CUDA 13.0 (torch==2.12.0+cu130) — pulled from https://download.pytorch.org/whl/cu130 by ComfyUI's own requirements.txt during the core build. cu130 wheels are NOT on PyPI; PyPI's torchaudio is cu128-only. When a dep in extras (most likely transformers, sam2, or one of their transitive trees) requested torchaudio with a version specifier, pip didn't see the cu130 wheel because the cu130 index wasn't configured for this layer, so it pulled the cu128 wheel from PyPI and replaced the cu130 install — leaving a runtime mismatch. ## Fix Add `--extra-index-url https://download.pytorch.org/whl/cu130` to the complete stage's pip install. Pip now sees cu130 wheels for torch-family packages during extras resolution and keeps the matching version. (The core stage uses cu128 explicitly; we don't touch that — the cu130 torch install in the running image comes from ComfyUI's requirements.txt pins, not the core's extra-index-url. Worth a separate cleanup to align both stages, but not blocking.) Co-Authored-By: Claude Opus 4.7 (1M context) --- .../comfy/complete/dockerfile.comfy.cuda.complete | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/services/comfy/complete/dockerfile.comfy.cuda.complete b/services/comfy/complete/dockerfile.comfy.cuda.complete index 86145d4..ffa5323 100644 --- a/services/comfy/complete/dockerfile.comfy.cuda.complete +++ b/services/comfy/complete/dockerfile.comfy.cuda.complete @@ -8,9 +8,20 @@ COPY --chown=comfy:comfy ./extra-requirements.txt ./ # Install additional python requirements # Using the same cache path as the core image +# +# --extra-index-url cu130 is REQUIRED. The base image's PyTorch is built +# against CUDA 13.0 (torch==2.12.0+cu130). Without the cu130 index, pip +# only sees the stock cu128 torchaudio wheel on PyPI and will replace +# the cu130 one whenever a transitive dep triggers re-resolution, +# producing at runtime: +# RuntimeError: Detected that PyTorch and TorchAudio were compiled +# with different CUDA versions. PyTorch has CUDA version 13.0 whereas +# TorchAudio has CUDA version 12.8. RUN --mount=type=cache,mode=0755,uid=1000,gid=1000,target=/app/.cache/pip \ source $VENV_PATH/bin/activate && \ - pip install -r extra-requirements.txt + pip install \ + --extra-index-url https://download.pytorch.org/whl/cu130 \ + -r extra-requirements.txt # Re-apply world-writable permissions after installing extra packages. # The core stage's chmod doesn't cover files added in this layer — newly