fix(core): use --index-url for torch family + parameterise via TORCH_INDEX arg - #47
Conversation
…INDEX arg ComfyUI's requirements.txt has bare `torch / torchaudio / torchvision` lines with no version pins or wheel URLs. The core Dockerfile used \`--extra-index-url https://download.pytorch.org/whl/cu128\` when installing those, but \`--extra-index-url\` is **additive**: pip picks the highest version found across PyPI + the extra index. When PyPI ships a newer torch than the cu128 index (which happens regularly — e.g. torch 2.12.0 landed on PyPI built for cu130 while the cu128 index still topped out at 2.11.0+cu128), pip picks the PyPI build for torch and the cu128 build for torchaudio (no newer torchaudio on PyPI). Result: torch compiled against cu130 + torchaudio compiled against cu128 → runtime crash on import: RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions. PyTorch has CUDA version 13.0 whereas TorchAudio has CUDA version 12.8. The current production image (built before torch 2.12.0 hit PyPI) works by luck; the next clean rebuild — with or without recent extra-requirements changes — would have failed regardless. ## Change Two-stage install in the core layer: 1. \`pip install --index-url https://download.pytorch.org/whl/${TORCH_INDEX} torch torchaudio torchvision\` — replaces PyPI for this command so all three are pulled from the same wheel index with a consistent CUDA ABI. 2. \`pip install -r /app/ComfyUI/requirements.txt\` then \`manager_requirements.txt\` — runs normally; torch family is already satisfied and pip skips re-resolution. Also adds \`ARG TORCH_INDEX=cu128\` parameterised via docker-bake.hcl: - \`core-cuda\` → \`TORCH_INDEX=cu128\` - \`core-cpu\` → \`TORCH_INDEX=cpu\` Adding a future cu130 variant becomes a single bake-target addition with \`args = { TORCH_INDEX = "cu130" }\`, no Dockerfile changes needed. ## Related Supersedes the closed #46 (which targeted the wrong layer with --extra-index-url cu130 in \`complete\` — would have masked the symptom while leaving the index-priority bug unfixed and also unnecessarily locking the image to cu130). Verified by local \`docker buildx bake complete-cuda\` rebuild: produces matching cu128 wheels for all torch-family packages; ComfyUI imports torch/torchaudio without the CUDA-mismatch error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
✅ Local verification complete. Force-rebuilt `core-cuda` and `complete-cuda` from scratch with this branch: Build log — core layer's torch install: All three pulled from the cu128 index with matching local-version IDs — no more PyPI generic torch sneaking in. The follow-up `pip install -r /app/ComfyUI/requirements.txt` ran cleanly without re-resolving torch family. Runtime check in the built image (`docker run --entrypoint /app/.venv/bin/python ... -c "import torch, torchaudio, torchvision; ..."`): No `RuntimeError`, no CUDA mismatch. Ready for review. |
Summary
ComfyUI's
requirements.txthas baretorch / torchaudio / torchvisionlines (no version pins, no wheel URLs). The core Dockerfile installed those with--extra-index-url https://download.pytorch.org/whl/cu128— but that flag is additive. Pip picks the highest version found across PyPI + the extra index.When PyPI ships a newer torch than the cu128 index (which happens regularly), pip splits the resolution:
torch2.12.0(cu130 ABI)2.11.0+cu1282.12.0(PyPI) — newer winstorchaudio2.11.0+cu1282.11.0+cu128torchvision0.27.0(cu130 ABI)0.27.0(PyPI)Result: torch / torchvision compiled against cu130 + torchaudio compiled against cu128 → runtime crash on first
import torchaudio:The current published
complete:cuda-latestimage happens to work because it was built when PyPI didn't yet havetorch 2.12.0— pure timing luck. The next clean Weekly Fresh Rebuild would have failed regardless of any extras changes.How I confirmed
Built
complete-cudalocally frommain(no PRs in flight). Resulting image errors onpython -c "import torchaudio"with the exact crash above. Inspecting the build log shows the core layer downloadingtorch-2.12.0from PyPI (no+cu128suffix) andtorchaudio-2.11.0+cu128from the cu128 index.Fix
Two-stage install in core:
pip install --index-url https://download.pytorch.org/whl/${TORCH_INDEX} torch torchaudio torchvision—--index-urlreplaces PyPI for this command, so all three are pulled from the same wheel index with a consistent CUDA ABI.pip install -r /app/ComfyUI/requirements.txtruns normally afterwards — torch family already satisfied, pip skips re-resolution.Wired through
ARG TORCH_INDEX=cu128and parameterised indocker-bake.hcl:core-cuda→TORCH_INDEX=cu128core-cpu→TORCH_INDEX=cpuAdding a cu130 variant becomes a single bake-target addition (
args = { TORCH_INDEX = "cu130" }), no Dockerfile changes.Local verification
Building locally with the fix now to confirm consistent CUDA versions across the torch family. Will comment with the result before requesting merge.
Supersedes
#46 (closed). That one tried to fix it in
completewith--extra-index-url cu130, which (a) targeted the wrong layer, (b) wouldn't fix the index-priority bug, and (c) would have unnecessarily moved the image to cu130.Test plan
docker buildx bake core-cuda complete-cudaproduces consistent cu128 wheels for torch / torchaudio / torchvision (in-flight)docker run --rm <built-image> /app/.venv/bin/python -c "import torchaudio; print(torchaudio.__version__)"succeeds without CUDA-mismatch errorcore-cpubuild still works (TORCH_INDEX=cpu — separate validation since prod uses cuda)🤖 Generated with Claude Code