Skip to content

fix(core): use --index-url for torch family + parameterise via TORCH_INDEX arg - #47

Merged
ductiletoaster merged 1 commit into
mainfrom
fix/core-torch-index-pin
May 27, 2026
Merged

fix(core): use --index-url for torch family + parameterise via TORCH_INDEX arg#47
ductiletoaster merged 1 commit into
mainfrom
fix/core-torch-index-pin

Conversation

@ductiletoaster

Copy link
Copy Markdown
Member

Summary

ComfyUI's requirements.txt has bare torch / torchaudio / torchvision lines (no version pins, no wheel URLs). The core Dockerfile installed those with --extra-index-url https://download.pytorch.org/whl/cu128 — but that flag is additive. Pip picks the highest version found across PyPI + the extra index.

When PyPI ships a newer torch than the cu128 index (which happens regularly), pip splits the resolution:

Package PyPI has cu128 index has Pip picks
torch 2.12.0 (cu130 ABI) 2.11.0+cu128 2.12.0 (PyPI) — newer wins
torchaudio (none matching) 2.11.0+cu128 2.11.0+cu128
torchvision 0.27.0 (cu130 ABI) older 0.27.0 (PyPI)

Result: torch / torchvision compiled against cu130 + torchaudio compiled against cu128 → runtime crash on first import torchaudio:

RuntimeError: Detected that PyTorch and TorchAudio were compiled with
different CUDA versions. PyTorch has CUDA version 13.0 whereas
TorchAudio has CUDA version 12.8.

The current published complete:cuda-latest image happens to work because it was built when PyPI didn't yet have torch 2.12.0 — pure timing luck. The next clean Weekly Fresh Rebuild would have failed regardless of any extras changes.

How I confirmed

Built complete-cuda locally from main (no PRs in flight). Resulting image errors on python -c "import torchaudio" with the exact crash above. Inspecting the build log shows the core layer downloading torch-2.12.0 from PyPI (no +cu128 suffix) and torchaudio-2.11.0+cu128 from the cu128 index.

Fix

Two-stage install in core:

  1. pip install --index-url https://download.pytorch.org/whl/${TORCH_INDEX} torch torchaudio torchvision--index-url replaces PyPI for this command, so all three are pulled from the same wheel index with a consistent CUDA ABI.
  2. pip install -r /app/ComfyUI/requirements.txt runs normally afterwards — torch family already satisfied, pip skips re-resolution.

Wired through ARG TORCH_INDEX=cu128 and parameterised in docker-bake.hcl:

  • core-cudaTORCH_INDEX=cu128
  • core-cpuTORCH_INDEX=cpu

Adding a cu130 variant becomes a single bake-target addition (args = { TORCH_INDEX = "cu130" }), no Dockerfile changes.

Local verification

Building locally with the fix now to confirm consistent CUDA versions across the torch family. Will comment with the result before requesting merge.

Supersedes

#46 (closed). That one tried to fix it in complete with --extra-index-url cu130, which (a) targeted the wrong layer, (b) wouldn't fix the index-priority bug, and (c) would have unnecessarily moved the image to cu130.

Test plan

  • Local docker buildx bake core-cuda complete-cuda produces consistent cu128 wheels for torch / torchaudio / torchvision (in-flight)
  • docker run --rm <built-image> /app/.venv/bin/python -c "import torchaudio; print(torchaudio.__version__)" succeeds without CUDA-mismatch error
  • CI rebuild passes
  • In-cluster pod boots cleanly when harmony bumps to the new image SHA
  • core-cpu build still works (TORCH_INDEX=cpu — separate validation since prod uses cuda)

🤖 Generated with Claude Code

…INDEX arg

ComfyUI's requirements.txt has bare `torch / torchaudio / torchvision`
lines with no version pins or wheel URLs. The core Dockerfile used
\`--extra-index-url https://download.pytorch.org/whl/cu128\` when installing
those, but \`--extra-index-url\` is **additive**: pip picks the highest
version found across PyPI + the extra index.

When PyPI ships a newer torch than the cu128 index (which happens
regularly — e.g. torch 2.12.0 landed on PyPI built for cu130 while the
cu128 index still topped out at 2.11.0+cu128), pip picks the PyPI build
for torch and the cu128 build for torchaudio (no newer torchaudio on
PyPI). Result: torch compiled against cu130 + torchaudio compiled
against cu128 → runtime crash on import:

    RuntimeError: Detected that PyTorch and TorchAudio were compiled
    with different CUDA versions. PyTorch has CUDA version 13.0 whereas
    TorchAudio has CUDA version 12.8.

The current production image (built before torch 2.12.0 hit PyPI) works
by luck; the next clean rebuild — with or without recent
extra-requirements changes — would have failed regardless.

## Change

Two-stage install in the core layer:

1. \`pip install --index-url https://download.pytorch.org/whl/${TORCH_INDEX} torch torchaudio torchvision\` — replaces PyPI for this command so all three are pulled from the same wheel index with a consistent CUDA ABI.
2. \`pip install -r /app/ComfyUI/requirements.txt\` then \`manager_requirements.txt\` — runs normally; torch family is already satisfied and pip skips re-resolution.

Also adds \`ARG TORCH_INDEX=cu128\` parameterised via docker-bake.hcl:
- \`core-cuda\` → \`TORCH_INDEX=cu128\`
- \`core-cpu\` → \`TORCH_INDEX=cpu\`

Adding a future cu130 variant becomes a single bake-target addition with
\`args = { TORCH_INDEX = "cu130" }\`, no Dockerfile changes needed.

## Related

Supersedes the closed #46 (which targeted the wrong layer with
--extra-index-url cu130 in \`complete\` — would have masked the symptom
while leaving the index-priority bug unfixed and also unnecessarily
locking the image to cu130).

Verified by local \`docker buildx bake complete-cuda\` rebuild: produces
matching cu128 wheels for all torch-family packages; ComfyUI imports
torch/torchaudio without the CUDA-mismatch error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ductiletoaster

Copy link
Copy Markdown
Member Author

Local verification complete.

Force-rebuilt `core-cuda` and `complete-cuda` from scratch with this branch:

Build log — core layer's torch install:
```
Downloading torch-2.11.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl
Downloading torchaudio-2.11.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl
Downloading torchvision-0.26.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl
Successfully installed ... torch-2.11.0+cu128 torchaudio-2.11.0+cu128 torchvision-0.26.0+cu128 ...
```

All three pulled from the cu128 index with matching local-version IDs — no more PyPI generic torch sneaking in. The follow-up `pip install -r /app/ComfyUI/requirements.txt` ran cleanly without re-resolving torch family.

Runtime check in the built image (`docker run --entrypoint /app/.venv/bin/python ... -c "import torch, torchaudio, torchvision; ..."`):
```
torch = 2.11.0+cu128
torchaudio = 2.11.0+cu128
torchvision = 0.26.0+cu128
```

No `RuntimeError`, no CUDA mismatch. Ready for review.

@ductiletoaster
ductiletoaster merged commit 9270afc into main May 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant