Skip to content

prevent too many open files error in multiprocessing mode#351

Merged
sbailey merged 1 commit into
mainfrom
too_many_files_fix
Mar 13, 2026
Merged

prevent too many open files error in multiprocessing mode#351
sbailey merged 1 commit into
mainfrom
too_many_files_fix

Conversation

@segasai
Copy link
Copy Markdown
Contributor

@segasai segasai commented Mar 5, 2026

With current redrock, when running rrdesi in multiprocessing mode (on my machine with python 3.12) I get this error:

$ env CUDA_VISIBLE_DEVICES='' RR_TEMPLATE_DIR=$HOME//tmp/rr_template_dir/ PYTHONPATH=py rrdesi  --outfile /tmp/aa.fits -i /home/skoposov/Downloads/coadd-main-dark-130.fits 
Running with 18 multiprocesing processes
WARNING:  using multiprocessing, but the OMP_NUM_THREADS
WARNING:  environment variable is not set- your system may
WARNING:  be oversubscribed.
Loading targets...
no archetypes or --archetypes-no-legendre; will turn off all the Legendre related arguments
Read and distribution of 1238 targets: 5.8 seconds
DEBUG: Reading templates from /home/skoposov//tmp/rr_template_dir/
Reading templates from ['/home/skoposov//tmp/rr_template_dir//rrtemplate-GALAXY-None-v2.6.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-QSO-LOZ-v1.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-QSO-HIZ-v1.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-A-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-B-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-CV-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-F-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-G-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-K-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-M-v0.1.fits', '/home/skoposov//tmp/rr_template_dir//rrtemplate-STAR-WD-v0.1.fits']
INFO: rrtemplate-GALAXY-None-v2.6.fits GALAXY  PCA IGM=Calura12 z=-0.005-1.6997
...
INFO: rrtemplate-STAR-WD-v0.1.fits STAR WD PCA IGM=None z=-0.002-0.002
Read and broadcast of 11 templates: 0.0 seconds
Creating GPU context: 0.0 seconds
Rebinning templates: 6.3 seconds
Computing redshifts
  Scanning redshifts for template GALAXY
--- Process 0 raised an exception ---
Proc 0: Traceback (most recent call last):
Proc 0:   File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/external/desi.py", line 1087, in rrdesi
Proc 0:   File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/zfind.py", line 290, in zfind
Proc 0:   File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/zscan.py", line 1043, in calc_zchi2_targets
Proc 0:   File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/targets.py", line 379, in sharedmem_pack
Proc 0:   File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/targets.py", line 103, in sharedmem_pack
Proc 0:   File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/utils.py", line 180, in mp_array
Proc 0:   File "/usr/lib/python3.12/multiprocessing/context.py", line 130, in RawArray
Proc 0:   File "/usr/lib/python3.12/multiprocessing/sharedctypes.py", line 66, in RawArray
Proc 0:   File "/usr/lib/python3.12/multiprocessing/sharedctypes.py", line 41, in _new_value
Proc 0:   File "/usr/lib/python3.12/multiprocessing/heap.py", line 331, in __init__
Proc 0:   File "/usr/lib/python3.12/multiprocessing/heap.py", line 309, in malloc
Proc 0:   File "/usr/lib/python3.12/multiprocessing/heap.py", line 192, in _malloc
Proc 0:   File "/usr/lib/python3.12/multiprocessing/heap.py", line 166, in _new_arena
Proc 0:   File "/usr/lib/python3.12/multiprocessing/heap.py", line 89, in __init__
Proc 0: OSError: [Errno 24] Too many open files

Traceback (most recent call last):
  File "/home/skoposov/pyenv_dir/pyenv312/bin/rrdesi", line 10, in <module>
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/external/desi.py", line 1229, in rrdesi
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/external/desi.py", line 1087, in rrdesi
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/zfind.py", line 290, in zfind
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/zscan.py", line 1043, in calc_zchi2_targets
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/targets.py", line 379, in sharedmem_pack
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/targets.py", line 103, in sharedmem_pack
  File "/home/skoposov/pyenv_dir/pyenv312/lib/python3.12/site-packages/redrock/utils.py", line 180, in mp_array
  File "/usr/lib/python3.12/multiprocessing/context.py", line 130, in RawArray
  File "/usr/lib/python3.12/multiprocessing/sharedctypes.py", line 66, in RawArray
  File "/usr/lib/python3.12/multiprocessing/sharedctypes.py", line 41, in _new_value
  File "/usr/lib/python3.12/multiprocessing/heap.py", line 331, in __init__
  File "/usr/lib/python3.12/multiprocessing/heap.py", line 309, in malloc
  File "/usr/lib/python3.12/multiprocessing/heap.py", line 192, in _malloc
  File "/usr/lib/python3.12/multiprocessing/heap.py", line 166, in _new_arena
  File "/usr/lib/python3.12/multiprocessing/heap.py", line 89, in __init__
OSError: [Errno 24] Too many open files

The patch fixes it by explicitly join()'in the processes and closing one file handle.
( i haven't investigated, but i wouldn't be surprised if this issue occurs now due to recent python's switching to spawn from for fork)

@sbailey sbailey merged commit b19c305 into main Mar 13, 2026
16 checks passed
@sbailey sbailey deleted the too_many_files_fix branch March 13, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants