Skip to content

fix(complete): add cu130 extra-index-url to extras pip install - #46

Closed
ductiletoaster wants to merge 1 commit into
mainfrom
fix/torchaudio-cu130-index
Closed

fix(complete): add cu130 extra-index-url to extras pip install#46
ductiletoaster wants to merge 1 commit into
mainfrom
fix/torchaudio-cu130-index

Conversation

@ductiletoaster

Copy link
Copy Markdown
Member

Summary

Prevents pip from replacing the base image's cu130-built torchaudio wheel with a stock cu128 wheel from PyPI when transitive deps in extra-requirements.txt trigger re-resolution.

Symptom this fixes

After #45 added transformers / peft / sentencepiece / sam2 and friends, a production K8s pod failed to start with:

```
RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions.
PyTorch has CUDA version 13.0 whereas TorchAudio has CUDA version 12.8.
Please install the TorchAudio version that matches your PyTorch version.
```

Pod went into CrashLoopBackOff; downstream harmony repo had to revert the image SHA bump (ductiletoaster/harmony#384) to recover service.

Root cause

The complete stage's pip install -r extra-requirements.txt had no --extra-index-url. The base image's PyTorch is built against CUDA 13.0 (torch==2.12.0+cu130) — pulled from https://download.pytorch.org/whl/cu130 by ComfyUI's own requirements.txt during the core build. cu130 wheels are not published on PyPI; PyPI's torchaudio is cu128-only.

When a dep added in #45 (transformers, sam2, or one of their transitive trees) requested torchaudio with a version specifier, pip didn't see the cu130 wheel because the cu130 index wasn't configured for this layer. So it pulled the cu128 wheel from PyPI and replaced the cu130 install — leaving torch and torchaudio compiled against different CUDA versions.

Fix

```diff
-RUN --mount=type=cache,...,target=/app/.cache/pip
+RUN --mount=type=cache,...,target=/app/.cache/pip
source $VENV_PATH/bin/activate && \

  • pip install -r extra-requirements.txt
  • pip install \
  •  --extra-index-url https://download.pytorch.org/whl/cu130 \
    
  •  -r extra-requirements.txt
    

```

Pip now sees cu130 wheels for torch-family packages during extras resolution and keeps the matching CUDA version installed.

Note on the cu128 index in core

The core Dockerfile uses --extra-index-url https://download.pytorch.org/whl/cu128 (line 27 of dockerfile.comfy.core), but the resulting image runs torch==2.12.0+cu130. The cu130 install comes from ComfyUI's own requirements.txt pinning specific wheel URLs that win the resolution. Aligning both stages explicitly to cu130 would be a worthwhile cleanup but is out of scope for this fix.

Test plan

🤖 Generated with Claude Code

Prevents pip from replacing the base image's cu130-built torchaudio
wheel with a stock cu128 wheel from PyPI when transitive deps in
extra-requirements.txt trigger re-resolution.

## Symptom this fixes

After #45 added transformers/peft/sentencepiece/sam2 etc., a
production K8s pod failed to start with:

    RuntimeError: Detected that PyTorch and TorchAudio were compiled
    with different CUDA versions. PyTorch has CUDA version 13.0
    whereas TorchAudio has CUDA version 12.8. Please install the
    TorchAudio version that matches your PyTorch version.

Pod went into CrashLoopBackOff; downstream harmony repo had to
revert the image SHA bump (ductiletoaster/harmony#384) to recover.

## Root cause

The complete stage's `pip install -r extra-requirements.txt` had no
`--extra-index-url`. The base image's PyTorch is built against
CUDA 13.0 (torch==2.12.0+cu130) — pulled from
https://download.pytorch.org/whl/cu130 by ComfyUI's own
requirements.txt during the core build. cu130 wheels are NOT on
PyPI; PyPI's torchaudio is cu128-only.

When a dep in extras (most likely transformers, sam2, or one of
their transitive trees) requested torchaudio with a version
specifier, pip didn't see the cu130 wheel because the cu130 index
wasn't configured for this layer, so it pulled the cu128 wheel from
PyPI and replaced the cu130 install — leaving a runtime mismatch.

## Fix

Add `--extra-index-url https://download.pytorch.org/whl/cu130` to
the complete stage's pip install. Pip now sees cu130 wheels for
torch-family packages during extras resolution and keeps the
matching version.

(The core stage uses cu128 explicitly; we don't touch that — the
cu130 torch install in the running image comes from ComfyUI's
requirements.txt pins, not the core's extra-index-url. Worth a
separate cleanup to align both stages, but not blocking.)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ductiletoaster

Copy link
Copy Markdown
Member Author

Closing — diagnosis was wrong direction.

Local rebuild of main (with no PRs in flight) reproduced the same crash and traced it back to core, not complete. The core layer uses --extra-index-url cu128, but that flag is additive — pip picks the highest version across PyPI + cu128 index. PyPI now ships torch==2.12.0 (compiled for cu130) which outranks torch==2.11.0+cu128, so the build ends up with cu130 torch + cu128 torchaudio.

The fix belongs in core with --index-url cu128 (replacing PyPI, not extending it) for just the torch family install. A follow-up PR with that approach (and an ARG TORCH_INDEX so cuda/cpu/future variants are parameterised cleanly) is incoming.

ductiletoaster added a commit that referenced this pull request Jun 12, 2026
…h loop (#48)

* fix(core): use --index-url for torch family + parameterise via TORCH_INDEX arg

ComfyUI's requirements.txt has bare `torch / torchaudio / torchvision`
lines with no version pins or wheel URLs. The core Dockerfile used
\`--extra-index-url https://download.pytorch.org/whl/cu128\` when installing
those, but \`--extra-index-url\` is **additive**: pip picks the highest
version found across PyPI + the extra index.

When PyPI ships a newer torch than the cu128 index (which happens
regularly — e.g. torch 2.12.0 landed on PyPI built for cu130 while the
cu128 index still topped out at 2.11.0+cu128), pip picks the PyPI build
for torch and the cu128 build for torchaudio (no newer torchaudio on
PyPI). Result: torch compiled against cu130 + torchaudio compiled
against cu128 → runtime crash on import:

    RuntimeError: Detected that PyTorch and TorchAudio were compiled
    with different CUDA versions. PyTorch has CUDA version 13.0 whereas
    TorchAudio has CUDA version 12.8.

The current production image (built before torch 2.12.0 hit PyPI) works
by luck; the next clean rebuild — with or without recent
extra-requirements changes — would have failed regardless.

## Change

Two-stage install in the core layer:

1. \`pip install --index-url https://download.pytorch.org/whl/${TORCH_INDEX} torch torchaudio torchvision\` — replaces PyPI for this command so all three are pulled from the same wheel index with a consistent CUDA ABI.
2. \`pip install -r /app/ComfyUI/requirements.txt\` then \`manager_requirements.txt\` — runs normally; torch family is already satisfied and pip skips re-resolution.

Also adds \`ARG TORCH_INDEX=cu128\` parameterised via docker-bake.hcl:
- \`core-cuda\` → \`TORCH_INDEX=cu128\`
- \`core-cpu\` → \`TORCH_INDEX=cpu\`

Adding a future cu130 variant becomes a single bake-target addition with
\`args = { TORCH_INDEX = "cu130" }\`, no Dockerfile changes needed.

## Related

Supersedes the closed #46 (which targeted the wrong layer with
--extra-index-url cu130 in \`complete\` — would have masked the symptom
while leaving the index-priority bug unfixed and also unnecessarily
locking the image to cu130).

Verified by local \`docker buildx bake complete-cuda\` rebuild: produces
matching cu128 wheels for all torch-family packages; ComfyUI imports
torch/torchaudio without the CUDA-mismatch error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(complete): bake ComfyUI-Lora-Manager deps to prevent startup crash loop

lora-manager is runtime-installed onto the custom_nodes volume; after a
self-update it began importing natsort, which isn't in the image venv.
Every fresh container boot then dies on ModuleNotFoundError before the
server binds — observed as CrashLoopBackOff on harmony talos-2101.

Bake all of lora-manager's requirements that aren't already provided by
core/extra deps, same pattern as the Impact/Subpack/Inspire fix (#44).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant