Skip to content

Snapshot restore fails when FORCE_COLOR is set: uv pip freeze output parsed with ANSI escapes (nightly Lifecycle CI has never passed) #1514

Description

@Kosinkadink

Summary

pipFreeze() parses the raw stdout of uv pip freeze. When the launcher process inherits FORCE_COLOR or CLICOLOR_FORCE from its environment, uv emits ANSI bold escapes around every package name even though stdout is a pipe. Every pipPackages key in a snapshot then becomes \x1b[1mtorch\x1b[0m instead of torch. Restoring such a snapshot fails: the protected-package check does not recognize torch/torchvision, the restore tries to uv pip install ESCAPED_NAME==VERSION from PyPI, fails, reverts, and never relaunches ComfyUI.

This is the reason the nightly Lifecycle workflow (.github/workflows/lifecycle.yml) has never had a green run since it was added in #1247. Playwright sets FORCE_COLOR=1 in every test worker, the e2e harness passes the environment through to the app, and the app passes it on to uv.

Impact

  • Product: any user who launches the app with FORCE_COLOR or CLICOLOR_FORCE set (common in developer shells, terminal apps, and CI) gets snapshots that cannot be restored. Snapshot capture succeeds silently, so the problem only shows up when a restore is attempted, with the message Snapshot package restore failed. Failed to install ESCAPED_NAME==VERSION ....
  • CI: 0 of 60 Lifecycle runs have passed (47 of 47 nightly runs since 2026-07-24 failed). Since 2026-08-14 the deterministic failure is e2e/lifecycle.test.ts:1457 "snapshot restore re-applies the switched pytorch stack" (added in Add trusted PyTorch stack switching and reliable v2 snapshot torch restore #1248). It times out after the 15-minute relaunch poll in waitForInPlaceOpRelaunch, and because the file runs in serial mode the 20 tests after it never execute. Latest example: https://github.com/Comfy-Org/Comfy-Desktop/actions/runs/34209263719

Root cause chain

  1. Playwright's worker host sets FORCE_COLOR: "1" unconditionally (node_modules/playwright/lib/runner/workerHost.js).
  2. e2e/support/electronHarness.ts buildIsolatedEnv spreads process.env into the app environment, so the Electron main process inherits FORCE_COLOR=1.
  3. src/main/lib/pip.ts:206 pipFreeze() runs uv pip freeze --python PYTHON_EXE through execFile with the inherited environment.
  4. uv resolves its color mode from FORCE_COLOR / CLICOLOR_FORCE before checking whether stdout is a TTY (crates/uv/src/settings.rs, resolve_color; behavior since uv 0.2.7, NO_COLOR takes precedence). Output becomes \x1b[1mtorch\x1b[0m==2.13.0+cpu.
  5. pipFreeze() splits each line on == and stores the escaped name as the key. Verified in a local run: all 100 keys in INSTALL_DIR\.launcher\snapshots\*.json were bold-wrapped.
  6. src/main/lib/snapshots/restore.ts:102 isProtectedPackage() compares against plain names, so torch/torchvision are no longer treated as protected.
  7. restorePipPackages() (restore.ts:413) builds ESCAPED_NAME==VERSION specs from the snapshot keys and runs uv pip install against the PyPI index. The install fails, the restore reverts, no relaunch happens, and the e2e poll runs out.

pipFreeze() is the only place uv stdout is parsed. pipFreezeDirect in desktopDetect.ts uses python -m pip freeze and is unaffected; other uv calls only stream output to the UI.

Reproduction

Product-level, on an existing CPU install:

Remove-Item Env:\NO_COLOR -ErrorAction SilentlyContinue
$env:FORCE_COLOR = '1'
& "INSTALL_DIR\standalone-env\uv.exe" pip freeze --python "INSTALL_DIR\standalone-env\python.exe" | Select-Object -First 3
# names are wrapped in ESC[1m ... ESC[0m
& "INSTALL_DIR\standalone-env\uv.exe" --color never pip freeze --python "INSTALL_DIR\standalone-env\python.exe" | Select-Object -First 3
# names are plain

Verified with uv 0.11.8 in the standalone env (the bootstrap uv is 0.11.18, same behavior). --color always reproduces the exact bytes seen in the CI artifact.

Test-level, from the repo root:

Remove-Item Env:\NO_COLOR -ErrorAction SilentlyContinue   # NO_COLOR masks the bug
$env:LIFECYCLE_VARIANT = 'cpu'
pnpm run build
pnpm run test:e2e:lifecycle

On main (b9b50cd): 59 passed, 1 failed (lifecycle.test.ts:1457, 15.0m poll), 20 did not run, 22.8 min total. Failure text matches CI.

Fix

Pass uv's global --color never flag in pipFreeze():

['--color', 'never', 'pip', 'freeze', '--python', pythonPath]

The flag must precede the pip subcommand. This keeps the parsed names plain regardless of the parent environment and does not change any other uv invocation.

Test change alongside it: waitForInPlaceOpRelaunch in e2e/lifecycle.test.ts accepts a failure selector, and the snapshot-restore helper passes the snapshots op card with role="alert". A failed restore then fails the test immediately with the card text instead of waiting out the relaunch budget.

Local result with the fix: 80 passed in 16.5 min, no retries. Unit tests, integration tests, lint, format, and typecheck all pass. Branch: fix/pip-freeze-force-color.

Acceptance criteria

  • pipFreeze() produces plain package names with FORCE_COLOR=1 and CLICOLOR_FORCE=1 set in the parent environment.
  • Unit test asserts --color never precedes pip in the uv argv and fails on the unfixed code.
  • pnpm run test:e2e:lifecycle (CPU variant) passes end to end with NO_COLOR unset, locally and in the run-lifecycle labelled PR run.
  • The nightly Lifecycle workflow has at least one green scheduled run after merge.
  • A failed snapshot restore in the lifecycle e2e reports the op card error text instead of a relaunch timeout.

Out of scope / follow-ups

  • Existing snapshots already written with escaped keys are not migrated by this fix; they will keep failing to restore until re-captured. Decide separately whether to normalize keys on read.
  • Tests that pass on retry in the nightly runs, unrelated to this bug: lifecycle-delete-untrack.test.ts:135 (3 nights, "deleted tile never disappeared from chooser"), lifecycle-periodic-update-check.test.ts:103 (2), lifecycle-copy-update-fail.test.ts:97 (2, ENOTEMPTY rmdir .git), first-use-skip:48, first-use-migrate:71, lifecycle.test.ts:913 (1 each). These will become visible as the only remaining failures once the deterministic one is gone.
  • pnpm run bridge-types:check fails on Windows because the single-quoted pathspec in the script is passed literally by cmd.exe; harmless in CI (Linux) but worth fixing for local runs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions