feat(pipeline): failed-job quarantine + error classification + stage timings - #296
Merged
Conversation
…; stage timings The error path destroyed all evidence: rmtree on failure threw away the demucs stderr, the stage, and the device, leaving "Audio processing failed" as the only artifact -- undebuggable after the fact. - Failed jobs now move to jobs/failed/<id> with an error.txt recording stage, device, model, classified cause, stage timings, and the demucs stderr tail. Heavy payloads (source, stems, video) are stripped first so quarantines stay KB-scale. Expired after 7 days by a new sweep that runs even on persistent-library deployments (failure evidence is diagnostics, not library content). The TTL sweep skips failed/. - New app/pipeline/errors.py: SeparationError carries the stderr tail + device out of separate(); classify_failure() maps failure text to out-of-memory / unsupported-device / disk-full / bad-input / unknown. The classified cause surfaces as Job.error_detail, shown in the studio as a muted secondary line under the generic error message. - Per-stage wall-clock timings (download/prepare, analyze, separate, post) recorded on the job, written to metadata.json, included in error.txt, and emitted as a one-line completion summary with the compute device -- performance regressions and the CPU-vs-GPU question are now answerable from logs. Closes #277 Closes #294 Closes #293
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #277, closes #294, closes #293. Phase 1 of #272 — plan: #272 (comment) (PR-1C + PR-1E, merged into one PR since they rewrite the same runner paths and the timings feed the quarantine's error.txt).
What
Failure quarantine (#277)
On pipeline failure the job dir no longer vanishes. Instead:
error.txtis written: timestamp, job/title/source, stage, device, model, classified cause, stage timings, exception, and the demucs stderr tail (carried out ofseparate()by the newSeparationError).jobs/failed/<id>, expired after 7 days (STEMDECK_FAILED_TTL_SECONDS) by a newsweep_failed_jobsthat runs unconditionally — persistent-library deployments (desktop, run.sh) skip the library TTL sweep but still expire failure evidence.Guards verified with tests: the TTL sweep skips
failed/(it would otherwise delete the quarantine root as an "orphan dir" after 24 h), andregistry.restoreignores it.Error classification (#294)
New
app/pipeline/errors.py:classify_failure()maps stderr/exception text →out-of-memory/unsupported-device/disk-full/bad-input/unknown. Surfaced asJob.error_detail(cause + last stderr line), rendered in the studio as a muted mono secondary line under the generic message. Phase 2's GPU→CPU fallback will reuse this classifier.Stage timings (#293)
stage_timings({download|prepare, analyze, separate, post}) recorded per job, persisted tometadata.json, included inerror.txt, and emitted as a one-line completion summary:Job.compute_devicerecords what the separation actually ran on.Testing
tests/test_errors.py(classifier table, SeparationError contract).failed/root immune to the TTL sweep, quarantine TTL expiry, no-op without quarantine, restore ignores quarantine.ruff check+ruff format --checkclean;node --checkon the JS.