In some environments test_pre_import may fail as the module that is picked arbitrarily may not be importable by the dask cuda worker process. This behavior has been observed in CI when testing wheels, but testing conda doesn't present any problems at this time, but given the non-deterministic nature of the test it may start failing at some point. The traceback of the error observed is as follows:
Traceback (most recent call last):
File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/core.py", line 528, in start
await wait_for(self.start_unsafe(), timeout=timeout)
File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/utils.py", line 1915, in wait_for
return await asyncio.wait_for(fut, timeout)
File "/pyenv/versions/3.10.16/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
return await fut
File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/worker.py", line 1493, in start_unsafe
raise plugins_exceptions[0]
File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/utils.py", line 805, in wrapper
return await func(*args, **kwargs)
File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/worker.py", line 1890, in plugin_add
result = plugin.setup(worker=self)
File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/dask_cuda/plugins.py", line 213, in setup
importlib.import_module(l)
File "/pyenv/versions/3.10.16/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'benchmarks'Ideally, we would pick a module that we are absolutely certain is not gonna be loaded when pre-import runs, but we don't control the entire software stack to be always sure this is going to be the case for any module we choose. To test LocalCUDACluster there would be a simple solution to that, we could register a module at runtime as below:
import types
import sys
mock = types.ModuleType('mock')
# Add it to sys.modules
sys.modules['mock'] = mock
import mockWe could then simply try to import the module as above. However, with dask cuda worker that's not possible, given the process we run with popen will not inherit anything we do at runtime. One alternative I can think is perhaps creating a submodule like dask_cuda/utils_test_mock.py we can control not to be loaded, it feels a bit dirty but may do the trick.
In some environments
test_pre_importmay fail as the module that is picked arbitrarily may not be importable by thedask cuda workerprocess. This behavior has been observed in CI when testing wheels, but testing conda doesn't present any problems at this time, but given the non-deterministic nature of the test it may start failing at some point. The traceback of the error observed is as follows:Ideally, we would pick a module that we are absolutely certain is not gonna be loaded when pre-import runs, but we don't control the entire software stack to be always sure this is going to be the case for any module we choose. To test
LocalCUDAClusterthere would be a simple solution to that, we could register a module at runtime as below:We could then simply try to import the module as above. However, with
dask cuda workerthat's not possible, given the process we run withpopenwill not inherit anything we do at runtime. One alternative I can think is perhaps creating a submodule likedask_cuda/utils_test_mock.pywe can control not to be loaded, it feels a bit dirty but may do the trick.