feat(pipeline): retry separation on CPU when a GPU attempt fails

[thcp]

Closes #276. Phase 2 of #273 — plan: #273 (comment) (PR-2A, the flagship).

What

A single MPS/CUDA failure (OOM, unsupported op, driver hiccup) killed the whole job with the generic "Audio processing failed" — the Mac Mini field report verbatim, where the user resorted to a LaunchAgent env-var hack to force CPU. Jobs now retry once on CPU and complete: slower, but alive and self-explanatory.

Design decisions (from the plan)

• Loud, never silent (the [Bug]: NVIDIA build still runs on CPU on machines with an NVIDIA GPU #247 lesson applied to the runtime path): stage line shows "GPU failed — retrying on CPU (slower)…" during the retry; the WARNING log carries the classified cause (via errors.classify_failure from feat(pipeline): failed-job quarantine + error classification + stage timings #296) + full stderr tail; gpu_fallback and compute_device ("cpu (fallback from cuda)") persist to job state, metadata.json, and the registry.
• Fires for every failure class and even user-forced devices — the unknown GPU failure is exactly what field reports hit; a dead job with no diagnostics is strictly worse for a user who forced cuda than a slow job that explains itself.
• Partial GPU output is cleared before the CPU run (collect() must never see half-written stems); progress resets to 0 (CPU restarts from scratch); the stall watchdog applies per attempt.
• Cancel during the GPU attempt → JobCancelled, no pointless CPU retry.
• CPU-also-fails → SeparationError carrying both attempts' stderr tails, so the Error path destroys all evidence #277 quarantine's error.txt records the full story.

Mechanics

separate() is now the policy layer over the extracted _run_demucs() (one attempt: spawn, progress streaming, watchdog, cancel translation — moved verbatim), with a module-level _demucs_cmd() seam for tests.

Testing

New tests/test_separate_fallback.py — stubs _demucs_cmd with Python one-liners so the real process machinery runs end-to-end without demucs/GPU:

• GPU fail → CPU success: stems produced, gpu_fallback=True, warning names device+cause+stderr
• GPU success: single attempt, no fallback
• CPU-first failure: exactly one attempt, no self-retry
• Both fail: one SeparationError with both tails and device="mps, then cpu"
• Cancel mid-GPU-attempt: JobCancelled, no CPU retry
• Partial GPU output cleared before retry

Full suite: 186 passed; ruff check + ruff format --check clean.

Manual verification path

On the NVIDIA machine: force demucs_device=cuda with CPU-only torch (the .venv-win dev env is exactly this) → job must complete with the fallback stage visible and "gpu_fallback": true in /api/jobs/{id}.

🤖 Generated with Claude Code