fix(complete): add cu130 extra-index-url to extras pip install - #46
Closed
ductiletoaster wants to merge 1 commit into
Closed
fix(complete): add cu130 extra-index-url to extras pip install#46ductiletoaster wants to merge 1 commit into
ductiletoaster wants to merge 1 commit into
Conversation
Prevents pip from replacing the base image's cu130-built torchaudio wheel with a stock cu128 wheel from PyPI when transitive deps in extra-requirements.txt trigger re-resolution. ## Symptom this fixes After #45 added transformers/peft/sentencepiece/sam2 etc., a production K8s pod failed to start with: RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions. PyTorch has CUDA version 13.0 whereas TorchAudio has CUDA version 12.8. Please install the TorchAudio version that matches your PyTorch version. Pod went into CrashLoopBackOff; downstream harmony repo had to revert the image SHA bump (ductiletoaster/harmony#384) to recover. ## Root cause The complete stage's `pip install -r extra-requirements.txt` had no `--extra-index-url`. The base image's PyTorch is built against CUDA 13.0 (torch==2.12.0+cu130) — pulled from https://download.pytorch.org/whl/cu130 by ComfyUI's own requirements.txt during the core build. cu130 wheels are NOT on PyPI; PyPI's torchaudio is cu128-only. When a dep in extras (most likely transformers, sam2, or one of their transitive trees) requested torchaudio with a version specifier, pip didn't see the cu130 wheel because the cu130 index wasn't configured for this layer, so it pulled the cu128 wheel from PyPI and replaced the cu130 install — leaving a runtime mismatch. ## Fix Add `--extra-index-url https://download.pytorch.org/whl/cu130` to the complete stage's pip install. Pip now sees cu130 wheels for torch-family packages during extras resolution and keeps the matching version. (The core stage uses cu128 explicitly; we don't touch that — the cu130 torch install in the running image comes from ComfyUI's requirements.txt pins, not the core's extra-index-url. Worth a separate cleanup to align both stages, but not blocking.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
|
Closing — diagnosis was wrong direction. Local rebuild of The fix belongs in |
5 tasks
ductiletoaster
added a commit
that referenced
this pull request
Jun 12, 2026
…h loop (#48) * fix(core): use --index-url for torch family + parameterise via TORCH_INDEX arg ComfyUI's requirements.txt has bare `torch / torchaudio / torchvision` lines with no version pins or wheel URLs. The core Dockerfile used \`--extra-index-url https://download.pytorch.org/whl/cu128\` when installing those, but \`--extra-index-url\` is **additive**: pip picks the highest version found across PyPI + the extra index. When PyPI ships a newer torch than the cu128 index (which happens regularly — e.g. torch 2.12.0 landed on PyPI built for cu130 while the cu128 index still topped out at 2.11.0+cu128), pip picks the PyPI build for torch and the cu128 build for torchaudio (no newer torchaudio on PyPI). Result: torch compiled against cu130 + torchaudio compiled against cu128 → runtime crash on import: RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions. PyTorch has CUDA version 13.0 whereas TorchAudio has CUDA version 12.8. The current production image (built before torch 2.12.0 hit PyPI) works by luck; the next clean rebuild — with or without recent extra-requirements changes — would have failed regardless. ## Change Two-stage install in the core layer: 1. \`pip install --index-url https://download.pytorch.org/whl/${TORCH_INDEX} torch torchaudio torchvision\` — replaces PyPI for this command so all three are pulled from the same wheel index with a consistent CUDA ABI. 2. \`pip install -r /app/ComfyUI/requirements.txt\` then \`manager_requirements.txt\` — runs normally; torch family is already satisfied and pip skips re-resolution. Also adds \`ARG TORCH_INDEX=cu128\` parameterised via docker-bake.hcl: - \`core-cuda\` → \`TORCH_INDEX=cu128\` - \`core-cpu\` → \`TORCH_INDEX=cpu\` Adding a future cu130 variant becomes a single bake-target addition with \`args = { TORCH_INDEX = "cu130" }\`, no Dockerfile changes needed. ## Related Supersedes the closed #46 (which targeted the wrong layer with --extra-index-url cu130 in \`complete\` — would have masked the symptom while leaving the index-priority bug unfixed and also unnecessarily locking the image to cu130). Verified by local \`docker buildx bake complete-cuda\` rebuild: produces matching cu128 wheels for all torch-family packages; ComfyUI imports torch/torchaudio without the CUDA-mismatch error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(complete): bake ComfyUI-Lora-Manager deps to prevent startup crash loop lora-manager is runtime-installed onto the custom_nodes volume; after a self-update it began importing natsort, which isn't in the image venv. Every fresh container boot then dies on ModuleNotFoundError before the server binds — observed as CrashLoopBackOff on harmony talos-2101. Bake all of lora-manager's requirements that aren't already provided by core/extra deps, same pattern as the Impact/Subpack/Inspire fix (#44). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prevents pip from replacing the base image's cu130-built
torchaudiowheel with a stock cu128 wheel from PyPI when transitive deps inextra-requirements.txttrigger re-resolution.Symptom this fixes
After #45 added
transformers/peft/sentencepiece/sam2and friends, a production K8s pod failed to start with:```
RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions.
PyTorch has CUDA version 13.0 whereas TorchAudio has CUDA version 12.8.
Please install the TorchAudio version that matches your PyTorch version.
```
Pod went into
CrashLoopBackOff; downstream harmony repo had to revert the image SHA bump (ductiletoaster/harmony#384) to recover service.Root cause
The
completestage'spip install -r extra-requirements.txthad no--extra-index-url. The base image's PyTorch is built against CUDA 13.0 (torch==2.12.0+cu130) — pulled fromhttps://download.pytorch.org/whl/cu130by ComfyUI's ownrequirements.txtduring the core build. cu130 wheels are not published on PyPI; PyPI's torchaudio is cu128-only.When a dep added in #45 (transformers, sam2, or one of their transitive trees) requested torchaudio with a version specifier, pip didn't see the cu130 wheel because the cu130 index wasn't configured for this layer. So it pulled the cu128 wheel from PyPI and replaced the cu130 install — leaving torch and torchaudio compiled against different CUDA versions.
Fix
```diff
-RUN --mount=type=cache,...,target=/app/.cache/pip
+RUN --mount=type=cache,...,target=/app/.cache/pip
source $VENV_PATH/bin/activate && \
```
Pip now sees cu130 wheels for torch-family packages during extras resolution and keeps the matching CUDA version installed.
Note on the cu128 index in core
The
coreDockerfile uses--extra-index-url https://download.pytorch.org/whl/cu128(line 27 ofdockerfile.comfy.core), but the resulting image runstorch==2.12.0+cu130. The cu130 install comes from ComfyUI's ownrequirements.txtpinning specific wheel URLs that win the resolution. Aligning both stages explicitly to cu130 would be a worthwhile cleanup but is out of scope for this fix.Test plan
complete-cudacleanly (cu130 index reachable from build runners)comfyuipod boots successfully (noRuntimeError: ... PyTorch and TorchAudio were compiled with different CUDA versions)python -c "import torchaudio; print(torchaudio.__version__)"in the new pod shows a+cu130buildModuleNotFoundError)🤖 Generated with Claude Code