dragon+orbit backends: idempotent cancels; dragon failure tracebacks - #91
Open
andre-merzky wants to merge 4 commits into
Open
dragon+orbit backends: idempotent cancels; dragon failure tracebacks#91andre-merzky wants to merge 4 commits into
andre-merzky wants to merge 4 commits into
Conversation
A cancel can race a task that already finished or was never dispatched -- ROSE's ParallelActiveLearner cancels its losing candidate branch exactly this way. Raising broke the caller's teardown and turned a clean branch-cancel into a DependencyFailureError on the surviving pipeline (first observed on the remote DT demo: rf branch trained, mlp branch's cancel raised, twin failed). The concurrent backend treats this as idempotent; dragon now does too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
The worker captures it, _deliver_batch drops it into task_desc and nothing ever prints it -- a failing function task surfaces to the consumer as a bare exception message. Log it endpoint-side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
The full dragon test module skips without the runtime, so CI never ran these paths; the new file drives them on a bare instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
Same defect class as the dragon fix in this PR, other backend: the
uid passes the early registry check, the remote cancel awaits, the
completion notification prunes the task -- and the post-await lookup
raised KeyError ("Task exception was never retrieved" noise on every
twin teardown under load; observed converting the xGFabric twin to
service mode). The lookup is now a get; a completed task answers
False, its terminal callback already fired via the notification.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
andre-merzky
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes from the first remote DT demo (dragon endpoint on Perlmutter, ROSE ParallelActiveLearner on top):
cancel_taskof an untracked uid is a no-op, not an error. A cancel can race a task that already finished or was never dispatched — ROSE cancels its losing candidate branch exactly this way. Raising broke the caller's teardown and turned a clean branch-cancel into aDependencyFailureErroron the surviving pipeline. The concurrent backend treats this as idempotent; dragon now does too._deliver_batchlogs the remote traceback on failure. The worker captures it, it lands intask_desc, and nothing ever printed it — a failing function task surfaced to the consumer as a bare exception message. Debugging the demo required exactly this log line to find aTypeErrorthree libraries deep.Related: the joblib/sklearn
n_jobs != 1failure inside dragon workers is tracked separately in #92.Cherry-picked from the demo-pinned branch
fix/dragon-cancel-idempotent(e491cd2-based). Unit suite: 191 passed, 23 skipped.🤖 Generated with Claude Code
https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU