FACTS writes no experiment output on NCCS

[ehultee]

Continuing troubleshooting from issue #364 . My installation on NCCS Discover seems complete; R module loaded correctly and emulandice installed. The dummy experiment shows no errors. However, global experiments still fail and/or write no output.

Opening this issue as requested by @AymenFJA . Will attach session logs below.

[ehultee]

client-re.session.borgi138.eultee.020318.0000.zip

Session log from today's attempt

[AymenFJA]

Hey @ehultee thanks for opening the ticket. From the logs I see the following:

WARNING: Retrying (...) after connection broken by 'NewConnectionError(... Failed to establish a new connection: [Errno -2] Name or service not known')'
ERROR: Could not find a version that satisfies the requirement numpy
ERROR: No matching distribution found for numpy

Which tells me that:
1- Your compute nodes have no internet access.
2- Your setup is trying to create the Python env on the fly.

Can you please provide me with the session folder located in:

//localhost/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi138.eultee.020318.0000/

You can zip it and upload it here as well. Once I confirm what is happening, you can switch to re-using your existing Python env instead of creating new one.

[ehultee]

That is interesting, because once on the node I have activated the env and the modules already.

Session folder is 705 MB compressed. It fails to upload here. I can put it on Google Drive for you, or share it in another way - let me know.

[AymenFJA]

Yes please, Google Drive should be fine.

[ehultee]

Drive link here.

Thank you @AymenFJA !

[AymenFJA]

Drive link here.

Thank you @AymenFJA !

Thank you, can you give me access please?

[AymenFJA]

Hey @ehultee, and thanks for the upload. I can confirm that you are re-using the same env, which is good.

So the issue is that for each task, they do some pre_execution, and what I mean by that is doing some package installation, which is technically wrong as we are re-using the same env (if we, for example, submit the same task 10 times, the installation will be repeated 10 times). For the fair module, you can see here:

facts/modules/fair/temperature/pipeline.global.yml

Line 6 in f736f97

python_dependencies: "numpy scipy netCDF4 pyyaml matplotlib pandas xarray dask fair==1.6.4"

and here:

facts/modules/fair/temperature/pipeline.yml

Line 6 in f736f97

python_dependencies: "numpy scipy netCDF4 pyyaml matplotlib pandas xarray dask fair==1.6.4"

, a list of packages is provided to be installed before the task starts to execute. I would suggest doing the following:

1- In your FACTs repo, remove that line or leave it empty
2- Install these packages from a point where you have an internet connection (login node, maybe?)
numpy scipy netCDF4 pyyaml matplotlib pandas xarray dask fair==1.6.4 (maybe you already installed them in your /gpfsm/dnb33/eultee/Software/facts/ve4)

And run again. Note that I do not know which file you are using (pipeline.global.yml or pipeline.yml) to instruct your experiment, but to be on the safe side, apply the instructions above to both.

LMK if this suggestion did not fix the issue.

[ehultee]

Hi Aymen,
Thanks! That's right, the compute nodes have no internet access but the login nodes do.

I followed your instructions and I checked the environment with a pip list before running. Unfortunately, commenting out the install did not fix the problem. The output folder exists but is empty. Several (maybe all?) of the tasks said they had failed, a RuntimeError with exit code: 1.

Client logs attached here. Agent logs on GDrive

This is interesting, because I had tested the environment prior to setting up emulandice and didn't run into this problem then. I wonder what new problem I've introduced in the process 😬

[AymenFJA]

@ehultee, I do not think it is a problem you introduced; I think it is more of a configuration corner case that was not considered.

So let's try a bit more aggressive approach, can you do the following: in here:

facts/FACTS.py

Line 185 in f736f97

if "python_dependencies" in tcfg.keys():

can you:

1. Comment out these 2 lines, please:

   # if their are python dependencies, add pip call to pre_exec
    if "python_dependencies" in tcfg.keys():
        t.pre_exec.append('pip install --upgrade pip; pip install ' + tcfg['python_dependencies'])

2. Clean up any __pycache__ folders in your FACTs repo.
3. Run again.

In this way, we can guarantee that no packages pre_exec will be added to the task.

[ehultee]

Aha, you mean the lack of internet access on compute nodes was not expected? That's interesting -- and I'm guessing it could take a while to find all the pieces of the problem if FACTS has previously been deployed on internet-connected systems.

Commenting those lines and clearing the __pycache__ folders unfortunately still produced a run with all tasks failed, no output. Logs here and here.

[AymenFJA]

Hey @ehultee, sorry to hear that your issue still exists. But, I can confirm that the packages issues disappeared, which means commenting out the lines as I suggested fixed that issue. But you have other issues here which is related to the science part:

aymenalsaadi@Aymens-Mac-mini:~/Downloads/agent-re.session.borgi190.eultee.020322.0000/pilot.0000$ cat task.0000*/*.err
cpu-bind=MASK - borgi190, task  0  0 [39699]: mask 0x1 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000000/fair_temperature_preprocess.py", line 101, in <module>
    fair_preprocess_temperature(scenario=args.scenario, rcmip_file=args.rcmip_file, pipeline_id=args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000000/fair_temperature_preprocess.py", line 74, in fair_preprocess_temperature
    emis = prep_rcmip_emissions_conc(scenario, rcmip_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000000/fair_temperature_preprocess.py", line 19, in prep_rcmip_emissions_conc
    emis_all = pd.read_csv(rcmip_file)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
                   ^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
             ^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './rcmip/rcmip-emissions-annual-means-v5-1-0.csv'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [39788]: mask 0x1 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000001/parameters/fair_ar6_climate_params_v4.0.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'aa0685af-6f18-4821-bb57-df64e65b64d6']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000001/fair_temperature_fit.py", line 36, in <module>
    fair_fit_temperature(param_file=args.param_file, pipeline_id=args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000001/fair_temperature_fit.py", line 11, in fair_fit_temperature
    pars = xr.load_dataset("./parameters/fair_ar6_climate_params_v4.0.nc")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 343, in load_dataset
    with open_dataset(filename_or_obj, **kwargs) as ds:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: '/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000001/parameters/fair_ar6_climate_params_v4.0.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [39873]: mask 0x1 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000002/fair_temperature_project.py", line 252, in <module>
    fair_project_temperature(nsamps=args.nsamps, seed=args.seed, cyear_start=args.cyear_start, cyear_end=args.cyear_end, smooth_win=args.smooth_win, pipeline_id=args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000002/fair_temperature_project.py", line 118, in fair_project_temperature
    with open(preprocess_file, 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'global.coupling.ssp245.temperature.fair.temperature_preprocess.pkl'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [39964]: mask 0x1 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000003/Import2lmData.py:62: SyntaxWarning: invalid escape sequence '\d'
  scenario_test = re.search("^tlim(\d*\.?\d+)win(\d*\.?\d+)$", scenario)
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000003/FittedISMIP_GrIS_preprocess.py", line 8, in <module>
    from Import2lmData import *
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000003/Import2lmData.py", line 4, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40475]: mask 0x200 set
cpu-bind=MASK - borgi190, task  0  0 [41390]: mask 0x2 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000005/FittedISMIP_GrIS_project.py", line 269, in <module>
    FittedISMIP_project_icesheet(args.nsamps, args.pyear_start, args.pyear_end, args.pyear_step, args.crateyear_start, args.crateyear_end, args.baseyear, args.pipeline_id, args.seed)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000005/FittedISMIP_GrIS_project.py", line 33, in FittedISMIP_project_icesheet
    with open(datafile, 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000005/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_data.pkl'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40013]: mask 0x2 set
cpu-bind=MASK - borgi190, task  0  0 [40480]: mask 0x400 set
cpu-bind=MASK - borgi190, task  0  0 [41178]: mask 0x20 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000008/deconto21_AIS_project.py", line 6, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40225]: mask 0x1 set
cpu-bind=MASK - borgi190, task  0  0 [42276]: mask 0x1 set
cpu-bind=MASK - borgi190, task  0  0 [42451]: mask 0x1 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000011/bamber19_icesheets_project.py", line 7, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40219]: mask 0x2 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000012/emulandice_preprocess.py", line 110, in <module>
    emulandice_preprocess(args.input_data_file, args.baseyear, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000012/emulandice_preprocess.py", line 72, in emulandice_preprocess
    samps, scenario = GetSamples(infile, years, baseyear)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000012/emulandice_preprocess.py", line 16, in GetSamples
    with Dataset(ncfile, "r") as nc:
         ^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000//pilot.0000//climate/global.coupling.ssp245.temperature.fair.temperature_gsat.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40922]: mask 0x2 set
cpu-bind=MASK - borgi190, task  0  0 [41687]: mask 0x4 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000014/emulandice_AIS_project.py", line 211, in <module>
    emulandice_project_AIS(args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000014/emulandice_AIS_project.py", line 90, in emulandice_project_AIS
    with open(preprocess_file, 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'global.coupling.ssp245.emuAIS.emulandice.AIS_preprocess.pkl'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40209]: mask 0x4 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000015/emulandice_preprocess.py", line 110, in <module>
    emulandice_preprocess(args.input_data_file, args.baseyear, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000015/emulandice_preprocess.py", line 72, in emulandice_preprocess
    samps, scenario = GetSamples(infile, years, baseyear)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000015/emulandice_preprocess.py", line 16, in GetSamples
    with Dataset(ncfile, "r") as nc:
         ^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000//pilot.0000//climate/global.coupling.ssp245.temperature.fair.temperature_gsat.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41018]: mask 0x4 set
cpu-bind=MASK - borgi190, task  0  0 [41832]: mask 0x10 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000017/emulandice_GrIS_project.py", line 188, in <module>
    emulandice_project_GrIS(args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000017/emulandice_GrIS_project.py", line 76, in emulandice_project_GrIS
    with open(preprocess_file, 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'global.coupling.ssp245.emuGrIS.emulandice.GrIS_preprocess.pkl'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40211]: mask 0x8 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000018/emulandice_preprocess.py", line 110, in <module>
    emulandice_preprocess(args.input_data_file, args.baseyear, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000018/emulandice_preprocess.py", line 72, in emulandice_preprocess
    samps, scenario = GetSamples(infile, years, baseyear)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000018/emulandice_preprocess.py", line 16, in GetSamples
    with Dataset(ncfile, "r") as nc:
         ^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000//pilot.0000//climate/global.coupling.ssp245.temperature.fair.temperature_gsat.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40996]: mask 0x8 set
cpu-bind=MASK - borgi190, task  0  0 [41685]: mask 0x8 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000020/emulandice_glaciers_project.py:56: SyntaxWarning: invalid escape sequence '\d'
  region_regex = re.compile("^region_(\d+)$")
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000020/emulandice_glaciers_project.py", line 224, in <module>
    emulandice_project_glaciers(args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000020/emulandice_glaciers_project.py", line 90, in emulandice_project_glaciers
    with open(preprocess_file, 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'global.coupling.ssp245.emuglaciers.emulandice.glaciers_preprocess.pkl'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40213]: mask 0x10 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000021/larmip_icesheet_preprocess.py", line 3, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41175]: mask 0x10 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000022/larmip_icesheet_fit.py", line 142, in <module>
    larmip_fit_smb(pipeline_id=args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000022/larmip_icesheet_fit.py", line 70, in larmip_fit_smb
    my_config = pickle.load(f)
                            ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41834]: mask 0x20 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000023/larmip_icesheet_project.py", line 4, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40482]: mask 0x20 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000024/Import2lmData.py:62: SyntaxWarning: invalid escape sequence '\d'
  scenario_test = re.search("^tlim(\d*\.?\d+)win(\d*\.?\d+)$", scenario)
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000024/ipccar5_glaciers_preprocess.py", line 10, in <module>
    from Import2lmData import *
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000024/Import2lmData.py", line 4, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41372]: mask 0x4 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000025/ipccar5_glaciers_fit.py", line 87, in <module>
    ar5_fit_glaciers(args.gmip, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000025/ipccar5_glaciers_fit.py", line 20, in ar5_fit_glaciers
    my_config = pickle.load(f)
                            ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41842]: mask 0x100 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000026/ipccar5_glaciers_project.py:216: SyntaxWarning: invalid escape sequence '\s'
  header_items = re.split(",\s*", fp.readline())
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000026/ipccar5_glaciers_project.py:224: SyntaxWarning: invalid escape sequence '\s'
  line_parts = re.split(",\s*", line)
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000026/ipccar5_glaciers_project.py", line 281, in <module>
    ar5_project_glaciers(args.seed, args.pyear_start, args.pyear_end, args.pyear_step, args.nmsamps, args.ntsamps, args.nsamps, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000026/ipccar5_glaciers_project.py", line 40, in ar5_project_glaciers
    my_data = pickle.load(f)
                          ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40464]: mask 0x40 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000027/Import2lmData.py:62: SyntaxWarning: invalid escape sequence '\d'
  scenario_test = re.search("^tlim(\d*\.?\d+)win(\d*\.?\d+)$", scenario)
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000027/ipccar5_icesheets_preprocess.py", line 10, in <module>
    from Import2lmData import *
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000027/Import2lmData.py", line 4, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41374]: mask 0x8 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000028/ipccar5_icesheets_fit.py", line 92, in <module>
    ar5_fit_icesheets(args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000028/ipccar5_icesheets_fit.py", line 20, in ar5_fit_icesheets
    my_config = pickle.load(f)
                            ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41844]: mask 0x200 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000029/ipccar5_icesheets_project.py:306: SyntaxWarning: invalid escape sequence '\s'
  header_items = re.split(",\s*", fp.readline())
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000029/ipccar5_icesheets_project.py:314: SyntaxWarning: invalid escape sequence '\s'
  line_parts = re.split(",\s*", line)
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000029/ipccar5_icesheets_project.py", line 417, in <module>
    ar5_project_icesheets(args.seed, args.pyear_start, args.pyear_end, args.pyear_step, args.crateyear_start,
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000029/ipccar5_icesheets_project.py", line 190, in ar5_project_icesheets
    my_data = pickle.load(f)
                          ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40466]: mask 0x80 set
/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000030/Import2lmData.py:62: SyntaxWarning: invalid escape sequence '\d'
  scenario_test = re.search("^tlim(\d*\.?\d+)win(\d*\.?\d+)$", scenario)
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000030/tlm_sterodynamics_preprocess_thermalexpansion.py", line 10, in <module>
    from Import2lmData import *
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000030/Import2lmData.py", line 4, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41182]: mask 0x40 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000031/tlm_sterodynamics_fit_thermalexpansion.py", line 80, in <module>
    tlm_fit_thermalexpansion(args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000031/tlm_sterodynamics_fit_thermalexpansion.py", line 18, in tlm_fit_thermalexpansion
    my_data = pickle.load(f)
                          ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41837]: mask 0x40 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000032/tlm_sterodynamics_project.py", line 184, in <module>
    tlm_project_thermalexpansion(args.seed, args.nsamps, args.pipeline_id, args.scenario, args.pyear_start, args.pyear_end, args.pyear_step, args.baseyear)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000032/tlm_sterodynamics_project.py", line 61, in tlm_project_thermalexpansion
    my_data = pickle.load(f)
                          ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [40468]: mask 0x100 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000033/ssp_landwaterstorage_preprocess.py", line 201, in <module>
    ssp_preprocess_landwaterstorage(args.scenario, args.dotriangular, args.includepokherl, args.baseyear, args.pyear_start, args.pyear_end, args.pyear_step, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000033/ssp_landwaterstorage_preprocess.py", line 50, in ssp_preprocess_landwaterstorage
    with open(pophistfile,'r', newline='') as csvfile:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000033/UNWPP2012 population historical.csv'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [41376]: mask 0x80 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000034/ssp_landwaterstorage_fit.py", line 222, in <module>
    ssp_fit_landwaterstorage(args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000034/ssp_landwaterstorage_fit.py", line 92, in ssp_fit_landwaterstorage
    my_data = pickle.load(f)
                          ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42199]: mask 0x2 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000035/ssp_landwaterstorage_project.py", line 279, in <module>
    ssp_project_landwaterstorage(args.nsamps, args.seed, args.dcyear_start, args.dcyear_end, args.dcrate_lo, args.dcrate_hi, args.pipeline_id)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000035/ssp_landwaterstorage_project.py", line 42, in ssp_project_landwaterstorage
    my_fit = pickle.load(f)
                         ^
UnboundLocalError: cannot access local variable 'f' where it is not associated with a value
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42662]: mask 0x1 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_GIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '65191b71-1a37-4664-a692-c29eb3e43d8a']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000036/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000036/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000036/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000036/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_GIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42668]: mask 0x2 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_GIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '07eeaebe-cb59-4a82-9e71-20356a8950a6']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000037/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000037/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000037/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000037/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_GIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42674]: mask 0x4 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_GIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '6a841831-2473-4748-a15b-ea608e7a5087']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000038/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000038/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000038/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000038/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.GrIS1f.FittedISMIP.GrIS_GIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42678]: mask 0x8 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.deconto21.deconto21.AIS_AIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '6bbb9d20-3333-4e3a-81e7-1213539ff61f']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000039/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000039/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000039/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000039/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.deconto21.deconto21.AIS_AIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42658]: mask 0x10 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.bamber19.bamber19.icesheets_GIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'fb819b9b-8090-4e8c-bce2-11cf0b35882b']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000040/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000040/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000040/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000040/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.bamber19.bamber19.icesheets_GIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42660]: mask 0x20 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.emuAIS.emulandice.AIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '80c96e16-a3ba-4b52-836c-f50c793061e7']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000041/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000041/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000041/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000041/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.emuAIS.emulandice.AIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1
cpu-bind=MASK - borgi190, task  0  0 [42681]: mask 0x40 set
Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.emuGrIS.emulandice.GrIS_globalsl.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '2bf7dd48-97c1-4054-a055-03616a32a5ec']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000042/total_workflow.py", line 165, in <module>
    TotalSamplesInWorkflows(args.directory,  args.pyear_start, args.pyear_end, args.pyear_step, args.chunksize, wfcfg_file= args.workflows, workflow=[args.workflow], scale=[args.scale], experiment_name = args.experiment_name)
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000042/total_workflow.py", line 90, in TotalSamplesInWorkflows
    r.append(TotalSampleInWorkflow(wfcfg, directory, targyears, this_workflow, this_scale, chunksize, experiment_name=experiment_name))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000042/total_workflow.py", line 55, in TotalSampleInWorkflow
    rout=TotalSamples(infiles, outfile, targyears, chunksize)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/task.000042/total_workflow.py", line 107, in TotalSamples
    ds = xr.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 1778, in open_mfdataset
    ds = open_(p, **open_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/api.py", line 750, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 671, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 457, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 466, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/gpfsm/dulocal15/sles15/other/GEOSpyD/24.4.0-0_py3.12/2024-05-31/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfsm/dnb33/eultee/Software/facts/ve4/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/discover/nobackup/eultee/eultee/radical.pilot.sandbox/re.session.borgi190.eultee.020322.0000/pilot.0000/to_total/global/global.coupling.ssp245.emuGrIS.emulandice.GrIS_globalsl.nc'
srun: error: borgi190: task 0: Exited with exit code 1

Which tells us that the first task failed due to a path error:

FileNotFoundError: [Errno 2] No such file or directory: './rcmip/rcmip-emissions-annual-means-v5-1-0.csv'

which led to another task failure as well (as it depends on the 1st task).

To fix this, you can update the fair_temperature_preprocess.py in the FACTs repo with:

import os

default_path = os.path.join(os.getcwd(), "rcmip", "rcmip-emissions-annual-means-v5-1-0.csv")

parser.add_argument(
    '--rcmip_file',
    help="Full path to RCMIP emissions file",
    default=default_path
)

[ehultee]

Alright, well, we'll keep finding the problems as they come I suppose. Thanks for reviewing.

In your suggested update to fair_temperature_preprocess.py, do you recommend adding the parser.add_argument() call in the file header (directly after updating the path) or in lines 93-95, where there are other parser.add_argument() calls? Does this happen in other files I should edit at the same time?

[AymenFJA]

Just update line 94 default value: https://github.com/radical-collaboration/facts/blob/f736f974a546b8969928d2c06299ff8f5bd51f62/modules/fair/temperature/fair_temperature_preprocess.py#L94C90-L94C136

Yes, very much so:

1. facts/modules/fair/temperature/fair_temperature_fit.py

   Line 30 in f736f97

   parser.add_argument('--param_file', help="Full path to FAIR parameter file", default="./parameters/fair_ar6_climate_params_v4.0.nc")

2. facts/modules/fair/temperature/fair_temperature_preprocess.py

   Line 94 in f736f97

   parser.add_argument('--rcmip_file', help="Full path to RCMIP emissions file", default="./rcmip/rcmip-emissions-annual-means-v5-1-0.csv")

[ehultee]

Oh yes okay, that update in line 94 makes sense. Thanks. This is one of several things I'm managing at the moment and I wasn't really thinking about what was happening there.

To your point (1), I did not update that line but simply pasted the new parser.add_argument() lines to direct to the rcmip file after it. Line 30 of fair_temperature_fit was already as you showed, directing to the params file. I made a similar update in fair_temperature_postprocess.py.

Perhaps unsurprisingly...I ran after these updates and it failed. 😞 Do you think we're making progress here, or would it be better to start from scratch?

[AymenFJA]

Hey @ehultee, I believe so, and I believe that we are almost there.

What we can do is we can have a 1-1 meeting to speed up the test and fix FACTs on your machine. We can do the test interactively, with a much smaller range of samples, which would be much faster.

LMK if this is ok with you.