V3 dragon backend: silent failure when a function-task rank exits non-zero

[andre-merzky]

Summary

When a function task's worker rank exits with a non-zero status under the V3 dragon backend (DragonExecutionBackendV3), the rank's own Python traceback is dropped: only a dragon-side DragonUserCodeError reaches the awaiter, with no information about why the rank failed.

This makes function-task failures effectively undebuggable from the client side — the user sees RuntimeError: Error(s) in user-provided code resulted in exit codes: [1] / Dragon Error Code: DRAGON_USER_CODE_ERROR and nothing else.

Reproduction

import asyncio
from rhapsody.api import ComputeTask, Session
from rhapsody.backends import DragonExecutionBackendV3

def bad_function():
    return {"a": 1}["nonexistent_key"]   # KeyError

async def main():
    backend = await DragonExecutionBackendV3()
    async with Session(backends=[backend]) as session:
        futures = await session.submit_tasks([
            ComputeTask(
                function=bad_function,
                task_backend_specific_kwargs={"process_templates": [(2, {})]},
            ),
        ])
        try:
            await futures[0]
        except BaseException as e:
            print(type(e).__name__, e)

asyncio.run(main())

Run with dragon repro.py.

Expected: the awaiter sees the KeyError: 'nonexistent_key' traceback (or at least a reference to bad_function's line).

Actual: the awaiter sees only RuntimeError: Error(s) in user-provided code resulted in exit codes: [1] / Dragon Error Code: DRAGON_USER_CODE_ERROR plus a dragon-internal stack inside batch.py / process_group.py. No reference to the user's function.

Root cause

Dragon batch's ProcessGroup.join() raises DragonUserCodeError when any rank exits non-zero, and that exception is what gets stored in self.batch.results_ddict[tuid]. The per-rank Python traceback is printed to the rank's own stderr stream by dragon's _dragon_native_python_process_main but is not captured into the result tuple — so by the time V3's _monitor_loop reads result, tb, raised, stdout, stderr = self.batch.results_ddict[tuid], the stderr field is empty and tb only contains dragon's internal trace.

Scope

• Affects function tasks only (Priorities 1–4 in V3's build_task).
• Executable tasks already have a working stdout/stderr redirect path via the bash-script wrapper (capture_stdio).

Proposed fix

Wrap function targets in a module-level helper that redirects sys.stdout/sys.stderr to per-rank files under self._work_dir and writes the Python traceback to the stderr file before re-raising. _deliver_batch then globs the per-rank files on failure and folds their contents into the augmented exception's message, preserving the original exception type.

PR: #51 — will edit once the PR number is known.

[AymenFJA]

@andre-merzky is this still happening with 0.4.0 (current release?)