Skip to content

scripts: force 'fork' multiprocessing when creating sharedir#84

Open
morehouse wants to merge 2 commits into
masterfrom
python3.14_fix
Open

scripts: force 'fork' multiprocessing when creating sharedir#84
morehouse wants to merge 2 commits into
masterfrom
python3.14_fix

Conversation

@morehouse
Copy link
Copy Markdown
Owner

Python 3.14 switched the default to 'forkserver' multiprocessing, which causes an error when running scripts/setup-nyx.sh:

python 3.14 setup-nyx.sh log
Creating sharedir at: /tmp/smite-nyx
Exporting Docker container to container.tar...
Copying packer binaries...
Generating Nyx config...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import sys; from multiprocessing.forkserver import main; main(5, 7, ['__main__'], sys_argv=sys.argv[1:], **{'sys_path': ['AFLplusplus/nyx_mode/packer/packer', '/usr/lib64/python314.zip', '/usr/lib64/python3.14', '/usr/lib64/python3.14/lib-dynload', '.local/lib/python3.14/site-packages', '/usr/lib64/python3.14/site-packages', '/usr/lib/python3.14/site-packages'], 'main_path': 'AFLplusplus/nyx_mode/packer/packer/nyx_config_gen.py', 'authkey_r': 9})
                                                             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/forkserver.py", line 230, in main
    spawn.import_main_path(main_path)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/spawn.py", line 307, in import_main_path
    _fixup_main_from_path(main_path)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                                  run_name="__mp_main__")
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "AFLplusplus/nyx_mode/packer/packer/nyx_config_gen.py", line 7, in <module>
    from common.util import ask_for_permission
  File "AFLplusplus/nyx_mode/packer/packer/common/util.py", line 33, in <module>
    from common.debug import logger
  File "AFLplusplus/nyx_mode/packer/packer/common/debug.py", line 47, in <module>
    manager = Manager()
  File "/usr/lib64/python3.14/multiprocessing/context.py", line 57, in Manager
    m.start()
    ~~~~~~~^^
  File "/usr/lib64/python3.14/multiprocessing/managers.py", line 566, in start
    self._process.start()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.14/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/context.py", line 306, in _Popen
    return Popen(process_obj)
  File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib64/python3.14/multiprocessing/spawn.py", line 164, in get_preparation_data
    _check_not_importing_main()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.14/multiprocessing/spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
    ...<16 lines>...
    ''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

Traceback (most recent call last):
  File "AFLplusplus/nyx_mode/packer/packer/nyx_config_gen.py", line 7, in <module>
    from common.util import ask_for_permission
  File "AFLplusplus/nyx_mode/packer/packer/common/util.py", line 33, in <module>
    from common.debug import logger
  File "AFLplusplus/nyx_mode/packer/packer/common/debug.py", line 47, in <module>
    manager = Manager()
  File "/usr/lib64/python3.14/multiprocessing/context.py", line 57, in Manager
    m.start()
    ~~~~~~~^^
  File "/usr/lib64/python3.14/multiprocessing/managers.py", line 566, in start
    self._process.start()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.14/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/context.py", line 306, in _Popen
    return Popen(process_obj)
  File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 51, in _launch
    self.sentinel, w = forkserver.connect_to_new_process(self._fds)
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/forkserver.py", line 106, in connect_to_new_process
    connection.answer_challenge(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
            wrapped_client, self._forkserver_authkey)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/multiprocessing/connection.py", line 992, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/usr/lib64/python3.14/multiprocessing/connection.py", line 223, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib64/python3.14/multiprocessing/connection.py", line 448, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python3.14/multiprocessing/connection.py", line 413, in _recv
    chunk = read(handle, to_read)
ConnectionResetError: [Errno 104] Connection reset by peer

This PR forces the older 'fork' multiprocessing mode which fixes the issue on python 3.14. It should also work with older pythons, but someone with an older python should verify that before we merge this.

Python 3.14 switched the default to 'forkserver' multiprocessing, which
is incompatible with the packer's python scripts.
Copy link
Copy Markdown
Contributor

@ekzyis ekzyis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, tested it with python 3.13.12. Encountered no issues when fuzzing.

Since due to my NixOS setup, I can't run python3 nyx_config_gen.py but need to run ./nyx_config_gen.py, I patched it like this:

diff --git a/packer/nyx_config_gen.py b/packer/nyx_config_gen.py
index 1ad5b28..23d505b 100755
--- a/packer/nyx_config_gen.py
+++ b/packer/nyx_config_gen.py
@@ -8,6 +8,9 @@ from common.util import ask_for_permission
 from common.info import show_banner
 import glob, shutil

+import multiprocessing
+multiprocessing.set_start_method('fork', force=True)
+
 template_kernel_default = \
   """
   #![enable(implicit_some)]

I don't think this changed the behavior under test.

Rather than executing a wrapper script that sets the start method to
'fork', we can load a sitecustomize.py file to do it instead.  This may
work better on NixOS where script shebangs get rewritten.
@morehouse
Copy link
Copy Markdown
Owner Author

@ekzyis I changed the approach to set the start method in a sitecustomize.py file instead. Can you retest and see if it works for you without patching nyx_config_gen.py?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants