Dask backend fails with TypeError

[AymenFJA]

File "x/lib/python3.13/site-packages/distributed/protocol/serialize.py", line 283, in serialize
    return serialize(
        x.data,
    ...<3 lines>...
        iterate_collection=True,
    )
  File "x/lib/python3.13/site-packages/distributed/protocol/serialize.py", line 391, in serialize
    raise TypeError(msg, str_x) from exc
TypeError: ('Could not serialize object of type LLGExpr', "LLGExpr(dsk={'infer_batch_task-2b70c0a106e49445644d3a253240b447': <Task 'infer_batch_task-2b70c0a106e49445644d3a253240b447' infer_batch_task()>})")

[mtitov]

further investigation showed - a cascade of serialization errors occurred:

• Because the script runs as __main__ and the task functions are wrapped by AsyncFlow, standard pickle fails by reference (it's not the same object as __main__.infer_batch_task).
• Dask falls back to cloudpickle.dumps() to serialize the task function and its referenced global scope by value.
• Because _MODEL_LOCAL = threading.local() was instantiated at the module level, cloudpickle encountered the live _thread._local object while traversing module globals (infer_batch_task -> get_cached_model), which cannot be pickled (TypeError: cannot pickle '_thread._local' object).

Solution: Use lazy initialization so that the module-level variable starts as None (which pickles cleanly) and is only instantiated as threading.local() inside get_cached_model() when executed by worker threads.

-> application-level implementation issue, and not RHAPSODY, sorry for the noise