RuntimeError: New Process Starting Before Old Process Finishes

[s-muscarnero]

Hello,

When I run redrock in the directory that contains the relevant FITS file (rrboss --spplate spPlate*.fits --zbest zbest.fits --output rrdetails.h5), I receive a RuntimeError message (while the code continues to run, despite never finishing). This is after I already fixed some other errors, including an error relating to int32 being used instead of int64.

Here is the output (the full error message is towards the bottom):

Running with 7 processes
WARNING: using multiprocessing, but the OMP_NUM_THREADS
WARNING: environment variable is not set- your system may
WARNING: be oversubscribed.
Loading targets...
DEBUG: read spPlate-7339-56768.fits
Read of 1000 targets: 0.9 seconds
Distribution of 1000 targets: 0.0 seconds
DEBUG: Reading templates from /opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates
Reading templates from ['/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-GALAXY-None-v2.6.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-QSO-LOZ-v1.0.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-QSO-HIZ-v1.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-A-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-B-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-CV-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-F-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-G-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-K-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-M-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-WD-v0.1.fits']
INFO: rrtemplate-GALAXY-None-v2.6.fits GALAXY PCA IGM=Calura12 z=-0.005-1.6997
INFO: rrtemplate-QSO-LOZ-v1.0.fits QSO LOZ PCA IGM=Calura12 z=0.05-1.5983
INFO: rrtemplate-QSO-HIZ-v1.1.fits QSO HIZ PCA IGM=Kamble20 z=1.4-6.993
INFO: rrtemplate-STAR-A-v0.1.fits STAR A PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-B-v0.1.fits STAR B PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-CV-v0.1.fits STAR CV PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-F-v0.1.fits STAR F PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-G-v0.1.fits STAR G PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-K-v0.1.fits STAR K PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-M-v0.1.fits STAR M PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-WD-v0.1.fits STAR WD PCA IGM=None z=-0.002-0.002
Read and broadcast of 11 templates: 0.0 seconds
Creating GPU context: 0.0 seconds
Rebinning templates: 1.4 seconds
Computing redshifts
Scanning redshifts for template GALAXY
Running with 7 processes
WARNING: using multiprocessing, but the OMP_NUM_THREADS
WARNING: environment variable is not set- your system may
WARNING: be oversubscribed.
Loading targets...
DEBUG: read spPlate-7339-56768.fits
Read of 1000 targets: 0.9 seconds
Distribution of 1000 targets: 0.0 seconds
DEBUG: Reading templates from /opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates
Reading templates from ['/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-GALAXY-None-v2.6.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-QSO-LOZ-v1.0.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-QSO-HIZ-v1.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-A-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-B-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-CV-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-F-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-G-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-K-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-M-v0.1.fits', '/opt/miniconda3/lib/python3.13/site-packages/redrock/data/templates/rrtemplate-STAR-WD-v0.1.fits']
INFO: rrtemplate-GALAXY-None-v2.6.fits GALAXY PCA IGM=Calura12 z=-0.005-1.6997
INFO: rrtemplate-QSO-LOZ-v1.0.fits QSO LOZ PCA IGM=Calura12 z=0.05-1.5983
INFO: rrtemplate-QSO-HIZ-v1.1.fits QSO HIZ PCA IGM=Kamble20 z=1.4-6.993
INFO: rrtemplate-STAR-A-v0.1.fits STAR A PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-B-v0.1.fits STAR B PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-CV-v0.1.fits STAR CV PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-F-v0.1.fits STAR F PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-G-v0.1.fits STAR G PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-K-v0.1.fits STAR K PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-M-v0.1.fits STAR M PCA IGM=None z=-0.002-0.002
INFO: rrtemplate-STAR-WD-v0.1.fits STAR WD PCA IGM=None z=-0.002-0.002
Read and broadcast of 11 templates: 0.0 seconds
Creating GPU context: 0.0 seconds
Rebinning templates: 1.4 seconds
Computing redshifts
Scanning redshifts for template GALAXY
--- Process 0 raised an exception ---
Proc 0: Traceback (most recent call last):
Proc 0: File "/opt/miniconda3/lib/python3.13/site-packages/redrock/external/boss.py", line 489, in rrboss
scandata, zfit = zfind(dtargets, dtemplates, mpprocs,
~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nminima=args.nminima, archetypes=args.archetypes,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
priors=args.priors, chi2_scan=args.chi2_scan)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Proc 0: File "/opt/miniconda3/lib/python3.13/site-packages/redrock/zfind.py", line 289, in zfind
results = calc_zchi2_targets(targets, templates, mp_procs=mp_procs, use_gpu=use_gpu)
Proc 0: File "/opt/miniconda3/lib/python3.13/site-packages/redrock/zscan.py", line 1057, in calc_zchi2_targets
p.start()
~~~~~~~^^
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
~~~~~~~~~~~^^^^^^
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/context.py", line 289, in _Popen
return Popen(process_obj)
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
~~~~~~~~~~~~^^^^^^^^^^^^^
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 164, in get_preparation_data
_check_not_importing_main()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
Proc 0: File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 140, in _check_not_importing_main
raise RuntimeError('''
...<16 lines>...
''')
Proc 0: RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

    To fix this issue, refer to the "Safe importing of main module"
    section in https://docs.python.org/3/library/multiprocessing.html

Traceback (most recent call last):
File "", line 1, in
from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=209, pipe_handle=218)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
~~~~~~~^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
run_name="mp_main")
File "", line 287, in run_path
File "", line 98, in _run_module_code
File "", line 88, in _run_code
File "/opt/miniconda3/bin/rrboss", line 9, in
boss.rrboss(comm=None)
~~~~~~~~~~~^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/site-packages/redrock/external/boss.py", line 533, in rrboss
raise err
File "/opt/miniconda3/lib/python3.13/site-packages/redrock/external/boss.py", line 489, in rrboss
scandata, zfit = zfind(dtargets, dtemplates, mpprocs,
~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nminima=args.nminima, archetypes=args.archetypes,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
priors=args.priors, chi2_scan=args.chi2_scan)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/site-packages/redrock/zfind.py", line 289, in zfind
results = calc_zchi2_targets(targets, templates, mp_procs=mp_procs, use_gpu=use_gpu)
File "/opt/miniconda3/lib/python3.13/site-packages/redrock/zscan.py", line 1057, in calc_zchi2_targets
p.start()
~~~~~~~^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
~~~~~~~~~~~^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/context.py", line 289, in _Popen
return Popen(process_obj)
File "/opt/miniconda3/lib/python3.13/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 164, in get_preparation_data
_check_not_importing_main()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/miniconda3/lib/python3.13/multiprocessing/spawn.py", line 140, in _check_not_importing_main
raise RuntimeError('''
...<16 lines>...
''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

    To fix this issue, refer to the "Safe importing of main module"
    section in https://docs.python.org/3/library/multiprocessing.html

Can you advise me on how to fix this problem?

Thanks!

Sincerely,
Sebastian

[sbailey]

We have not run rrboss in many years. I think this crash comes from a new restriction in how multiprocessing works from scripts (possibly introduced in python 3, perhaps after that, but I haven't run rrboss since python 2-era).

Try changing redrock/bin/rrboss to

from redrock.external import boss

if __name__ == '__main__':
    boss.rrboss(comm=None)

If that works, please let us know and we can update the repo. If that doesn't work, please post whatever messages you get.

Also, please export OMP_NUM_THREADS=1 (bash) or setenv OMP_NUM_THREADS 1 (tcsh) prior to running. That isn't the cause of the crash, but it can dramatically help with performance.

[s-muscarnero]

Hello,

Thank you for your feedback! I was able to successfully run redrock with the edit you suggested.

[sbailey]

Great, thanks for the confirmation. I'll leave this ticket open until we get it fixed in the main branch (simple, but competing with other priorities too).