distributed.utils_test.popen does not use binaries from PATH

[jacobtomlinson]

While debugging some dask-mpi issue I noticed that tests fail if openmpi is installed with apt rather than conda. xref dask/dask-mpi#71 (comment)

This seems to be because distributed.utils_test.popen assumes all binaries will be within the sys.prefix which in this case is not true.

distributed/distributed/utils_test.py

Line 1053 in 942f235

os.environ.get("DESTDIR", "") + sys.prefix, "bin", args[0]

I would assume that popen(["foo", ...]... would call the same binary that which foo displays.

It seems this was done in #1954 but there's isn't much context as to why this was done. @mrocklin could you shed any light on this?

[jameslamb]

I just ran into this in another place and found this old issue.

Just using pip install --user dask[test] distributed is enough to hit this, so I'd expect others may have experienced it and just chosen to avoid dask.distributed.utils_test.popen().

$ docker run --rm -it python:3.13 bash

$ python -m pip install --user 'dask[test]==2025.12.0' 'distributed==2025.12.0'
WARNING: The script dask is installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

$ export PATH="/root/.local/bin:${PATH}"
$ which dask
/root/.local/bin/dask

$ python -c 'from dask.distributed.utils_test import popen; popen(["dask", "scheduler", "--no-dashboard"]).__enter__()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from dask.distributed.utils_test import popen; popen(["dask", "scheduler", "--no-dashboard"]).__enter__()
                                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/root/.local/lib/python3.13/site-packages/distributed/utils_test.py", line 1201, in popen
    with subprocess.Popen(args, **kwargs) as proc:
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/subprocess.py", line 1039, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        pass_fds, cwd, env,
                        ^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
                        gid, gids, uid, umask,
                        ^^^^^^^^^^^^^^^^^^^^^^
                        start_new_session, process_group)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/subprocess.py", line 1876, in _execute_child
    self._posix_spawn(args, executable, env, restore_signals, close_fds,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      p2cread, p2cwrite,
                      ^^^^^^^^^^^^^^^^^^
                      c2pread, c2pwrite,
                      ^^^^^^^^^^^^^^^^^^
                      errread, errwrite)
                      ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/subprocess.py", line 1820, in _posix_spawn
    self.pid = os.posix_spawn(executable, args, env, **kwargs)
               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/dask'

That's referencing /usr/local/bin because /usr/local is sys.prefix in that image:

$ python -c "import sys; print(sys.prefix)"
/usr/local

[jameslamb]

It seems this was done in #1954 but there's isn't much context as to why this was done. @mrocklin could you shed any light on this?

Looking at the relevant code:

distributed/distributed/utils_test.py

Lines 1197 to 1200 in 06f59c2

	if sys.platform.startswith("win"):
	args[0] = os.path.join(sys.prefix, "Scripts", args[0])
	else:
	args[0] = os.path.join(sys.prefix, "bin", args[0])

I'm guessing that the goal was to ensure that the dask executable found matches the Python interpreter that popen() was run from and therefore the installation of dask / distributed relevant to that Python process. For systems with multiple installations of dask laying aorund.

I think it might be better to use sysconfig.get_path("scripts") for this purpose, which should also take care of it being bin/ on macOS/Linux and Scripts/ on Windows.

$ python -c 'import sysconfig; print(sysconfig.get_path("scripts"))'
/usr/local/bin

And that it should be possible for users to override this behavior by providing an absolute path.

# avoid searching for executables... only accept absolute paths or look in this Python interpreter's default location for scripts
executable_path = args[0]
if not os.path.isabs(executable_path):
    executable_path = os.path.join(sysconfig.get_path("scripts"), executable_path)

if not os.path.isfile(executable_path):
    raise FileNotFoundError(f"Could not find '{executable_path}'. To avoid this warning, provide an absolute path to an existing installation to popen()".

Alternatively, this function could just allow PATH to be searched and leave it to users to control their installations.

Would maintainers be open to a patch with one of those approaches?

[jacobtomlinson]

I think the sysconfig approach sounds fine to me. Do you have time to open a PR to fix this @jameslamb?

[jameslamb]

Yep happy to! I'll put something up shortly.

[jameslamb]

Put up a proposal: #9174