Skip to content

fix(desktop): NVIDIA build silently falling back to CPU (#247) - #267

Merged
thcp merged 4 commits into
mainfrom
fix/nvidia-cpu-fallback-247
Jul 15, 2026
Merged

fix(desktop): NVIDIA build silently falling back to CPU (#247)#267
thcp merged 4 commits into
mainfrom
fix/nvidia-cpu-fallback-247

Conversation

@thcp

@thcp thcp commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Closes #247.

Verdict: still valid

Nothing has touched GPU selection since the issue was filed (only FFmpeg/Unraid commits since 2026-07-04). Code inspection confirmed three concrete, independent defects -- stronger than the issue's own hypotheses.

Root causes

1. Cross-install poisoning (primary). is_cpu_only_package trusted a cpu-only marker in the shared per-user data dir, not just the app root. The CPU build wrote that marker there on every ensure_workspace, and the legacy migration copied it too. Result: run the CPU build once, then install the NVIDIA build -> GPU detection never runs, permanent silent CPU. Explains "works on some NVIDIA PCs but not others."

2. One bad first run pins CPU forever. A CPU result from a transient failure (no GPU detected, CUDA verify failed) was persisted identically to a real CPU-only package, and the setup gate treated any truthy torchDevice as "done" -- never re-probing.

3. Fragile nvidia-smi discovery. Only checked System32 and PATH; some DCH driver installs place it only under DriverStore\FileRepository\nv*\. 10s timeout also tight for Optimus laptops waking a sleeping dGPU.

Fix

  • is_cpu_only_package: app root only. Stale data-dir marker is auto-deleted and logged (self-heals a poisoned install).
  • Persist torchDeviceReason alongside the device; the setup gate (setup.js) only treats cuda/mps, or cpu on a genuine cpu-only package, as settled. A failure-born CPU or a legacy reason-less install re-probes the GPU on the next launch -- existing affected installs self-heal just by relaunching, no user action needed.
  • nvidia_smi_exe: scan DriverStore\FileRepository\nv*\ (newest package wins) before falling back to PATH; first probe timeout 10s -> 30s.
  • Every GPU-detection decision now logged to setup.log (today a silent CPU leaves zero trace).
  • Windows CPU-only portable package no longer stages data\cpu-only (the source of the poisoned marker).

Deliberately out of scope

Verify

  • cargo test (desktop/src-tauri): 16/16 pass, including 5 new tests (marker precedence, self-heal + setup.log line, CPU build doesn't churn its own marker, DriverStore newest-wins scan, missing-dir handling).
  • cargo fmt --check: clean.
  • node --check desktop/ui/setup.js: clean.
  • Manual matrix (Windows NVIDIA machine, not yet run by me):
    • Fresh NVIDIA build -> CUDA enabled, setup.log shows detection lines.
    • Poisoned install (place cpu-only in %LOCALAPPDATA%\StemDeck) -> launch NVIDIA build -> marker auto-removed (logged), GPU probed, CUDA enabled.
    • Pinned install ("torchDevice": "cpu", no reason, in config.json) -> relaunch -> GPU step re-runs and heals to cuda.
    • CPU build -> root marker settles it, no repeated probing.
    • GPU-less machine, NVIDIA build -> "No NVIDIA GPU" summary each launch, ~1s added, no error dialogs.

Follow-up

After merge + a release, comment on #247 explaining the three causes and the self-heal (relaunch fixes pinned installs), and ask the reporter to confirm before closing further.

Added in this PR: compute device selector for the self-hosted server

Companion to the desktop fix above, covering the server/Docker/Unraid path (which is a completely separate code path and was NOT touched by the desktop fix):

  • New Settings -> Advanced -> Compute device dropdown: Auto (default, hardware probe) / CPU / CUDA (NVIDIA) / MPS (Apple Silicon). Shows the currently resolved device.
  • Read fresh per job (app/pipeline/separate.py) -- changes apply to the next separation, no restart. Previously the device was a frozen constant readable only via the STEMDECK_DEMUCS_DEVICE env var + restart.
  • Verify-before-persist: forcing CUDA/MPS checks availability first and rejects with a clear 422 ("cuda is not available on this machine") instead of persisting a device that would silently fall back later -- the same principle as the desktop fix, applied to the server.
  • STEMDECK_DEMUCS_DEVICE still seeds the default, so existing env-based deployments are unaffected.
  • /api/settings gains demucs_device (user choice) + demucs_device_resolved (what jobs run on); /api/health's demucs_device now reflects the live value.

Tests: 5 new (auto-resolution, env seeding, verify-before-persist rejection, unknown choice, API round trip + 422s) -- 148 total pass. ruff clean, node --check clean.

@thcp
fix(desktop): NVIDIA build silently falling back to CPU (#247)
2177bb6 
Three independent defects each land the NVIDIA build on CPU with no visible
error and no recovery path:

1. The cpu-only marker was trusted in the shared per-user data dir, not just
   the app root. The CPU build wrote/migrated that marker there, so anyone who
   ever ran the CPU build got the NVIDIA build permanently pinned to CPU --
   GPU detection never even ran. is_cpu_only_package now checks the app root
   only; a stale data-dir marker is auto-deleted and logged.

2. A CPU result from a transient failure (no GPU detected, CUDA verify
   failed) was persisted the same as a real CPU-only package, and the setup
   gate treated any truthy torchDevice as "done" -- one bad first run pinned
   CPU forever. Device selection now persists a reason (torchDeviceReason),
   and the setup gate only treats cuda/mps or a genuine cpu-only package as
   settled; a failure-born CPU or a legacy install with no reason re-probes
   the GPU on the next launch. Existing affected installs self-heal on
   relaunch, no user action needed.

3. nvidia-smi discovery only checked System32 and PATH; some DCH driver
   installs place it only under DriverStore\FileRepository\nv*\. Added that
   scan (newest package wins) and raised the first probe's timeout to 30s for
   Optimus laptops waking a sleeping dGPU. Every detection decision is now
   logged to setup.log.

Also drops the Windows CPU-only portable package's data\cpu-only staging
(scripts/windows/make-portable.ps1), which was the source of the poisoned
marker.

5 new Rust unit tests cover marker precedence, the self-heal + log line, CPU
builds not churning their own marker, and the DriverStore newest-wins scan.
@thcp
thcp marked this pull request as ready for review July 14, 2026 14:21
thcp added 3 commits July 14, 2026 16:26
@thcp
feat(settings): compute device selector for the self-hosted server
4f0b07c 
Companion to the desktop #247 fix, for the server/Docker/Unraid path: device
selection was a frozen constant (DEMUCS_DEVICE, computed once at import), so
the only override was the STEMDECK_DEMUCS_DEVICE env var plus a restart --
invisible to Docker/Unraid users without container access.

- app/core/settings.py: demucs_device setting (auto | cuda | mps | cpu,
  default auto = hardware probe). Forcing cuda/mps verifies availability
  BEFORE persisting and rejects with a clear error otherwise -- never persist
  a device that would silently fall back later (the #247 lesson applied
  here). STEMDECK_DEMUCS_DEVICE seeds the default so existing env-based
  deployments keep their forced device.
- app/core/config.py: _detect_device -> detect_torch_device (pure hardware
  probe; env handling moved to the settings seed); DEMUCS_DEVICE constant
  removed.
- app/pipeline/separate.py: reads the device fresh per job -- a Settings
  change applies to the next separation, no restart.
- app/main.py: /api/settings gains demucs_device (choice) and
  demucs_device_resolved (what jobs will run on); POST validates via the
  setter (422 with the reason). Startup log and /api/health read live.
- static/js/catalog.js: "Compute device" select in Settings -> Advanced,
  showing the resolved device; a rejected force surfaces the server's reason
  via showError and reverts the select. Also aligns the port-input fallback
  with the 8000 default from the earlier port unification.
- .docs/improvements/self-hosted-compute-device-setting.md: design doc.

5 new tests: auto-resolution, env seeding, verify-before-persist rejection,
unknown-choice rejection, and the API round trip incl. 422 paths.
@thcp
feat(settings): gray out compute devices this machine can't use
22206b3 
The Compute device dropdown now disables options that aren't available or
detected (Auto and CPU are always selectable; CUDA/MPS depend on the
hardware + torch build), labeling them "— not available" so it's clear why.

- config.py: available_torch_devices() returns the usable devices best-first;
  detect_torch_device() is now its first element (no duplicated torch probe).
- settings.py: set_demucs_device verifies against membership in
  available_torch_devices() rather than only the top pick.
- /api/settings: new demucs_devices_available list for the UI.
- catalog.js: disable + relabel unavailable <option>s on load and after each
  change.
@thcp
fix(ui): settings scrollbar no longer overlaps right-aligned controls
72f5516 
The Advanced settings pane scrolls, and its scrollbar drew directly over the
right-aligned Port / Compute device controls. Reserve a scrollbar gutter
(padding-right + equal negative margin so it sits in the card's existing 12px
padding), keeping content aligned with the fixed header/footer. Surfaced once
the new Compute device row made the pane tall enough to scroll.
@thcp
thcp merged commit c19d67e into main Jul 15, 2026
8 checks passed
@thcp
thcp deleted the fix/nvidia-cpu-fallback-247 branch July 15, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: NVIDIA build still runs on CPU on machines with an NVIDIA GPU

1 participant

@thcp