joblib/sklearn n_jobs != 1 breaks inside dragon workers (mpbridge ThreadPool)

[andre-merzky]

Any function task that lets joblib parallelize — e.g. RandomForestRegressor(n_jobs=-1).fit(...) — dies inside a dragon worker:

File ".../joblib/_parallel_backends.py", line 507, in _get_pool
    self._pool = ThreadPool(self._n_jobs)
File ".../multiprocessing/pool.py", line 930, in __init__
    Pool.__init__(self, processes, initializer, initargs)
File ".../dragon/mpbridge/pool.py", line 273, in __init__
    super().__init__(*args, **kwargs)
TypeError: super(type, obj): obj must be an instance or subtype of type

Dragon's mpbridge monkeypatches multiprocessing.pool, so the stdlib ThreadPool.__init__ chains into DragonPool.__init__, whose zero-arg super() cell belongs to a different class than self. Upstream this is a dragon mpbridge limitation, but rhapsody's dragon backend is where users hit it: a task that runs fine on the concurrent backend fails opaquely on dragon.

Observed live on Perlmutter (dragon 0.14, DT demo 2026-08-31); workaround was n_jobs=1.

Options for rhapsody: document the constraint on the dragon backend; detect-and-warn (e.g. set JOBLIB_START_METHOD/force joblib's sequential backend in the worker env); or carry the report upstream to dragon.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU