Too many files

[cafletezbrant]

Hi @zouter @OlgaPushkarev @VincentGardeux @Victor-fe @dav-pav ,

Thanks again for this tool. I am getting this error:

  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/models/pred/model/better.py", line 1269, in train_models
    model.train_model(
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/models/pred/model/better.py", line 825, in train_model
    loaders_validation = LoaderPool(
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/loaders/pool.py", line 76, in __init__
    self.loaders = [loader_cls(**loader_kwargs) for i in range(n_workers)]
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/loaders/pool.py", line 76, in <listcomp>
    self.loaders = [loader_cls(**loader_kwargs) for i in range(n_workers)]
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/loaders/transcriptome_fragments.py", line 50, in __init__
    self.transcriptome = TranscriptomeGene(transcriptome, gene_oi=region_oi, layer=layer)
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/loaders/transcriptome.py", line 64, in __init__
    self.X = X.oindex[:, gene_ix]  # open a tensorstore reader with orthogonal indexing
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/flow/tensorstore.py", line 281, in oindex
    return OIndex(self.open_reader())
  File "/cv/home/fletezbk/ChromatinHD/src/chromatinhd/flow/tensorstore.py", line 156, in open_reader
    fp = np.memmap(path, dtype=metadata["dtype"], mode="r", shape=tuple(metadata["shape"]))
  File "/cv/home/fletezbk/miniforge3/envs/chd2/lib/python3.9/site-packages/numpy/_core/memmap.py", line 279, in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
OSError: [Errno 24] Too many open files

While I know this is an OS limit, if I want to train models genome-wide (20k genes = 20k files if I am reading the comment in transcriptome.py line 64 correctly), what is a simple way to avoid this? I've tried setting transcriptome.layers['X'] based on loaders/transcriptome.py (

ChromatinHD/src/chromatinhd/loaders/transcriptome.py

Line 60 in b353564

elif torch.is_tensor(X):

) but it seems to be a bit resistant.

Thank you,
Kipper Fletez-Brant

[cafletezbrant]

Just thought I'd add that I get this with 5000 genes as well.