Support for handling spatially-separated history files (MOM6 output tiles)

[phillips-ad]

I am filing this issue as a placeholder. At present CESM3 MOM6 output includes monthly files that do not end with ".nc" but are indeed netcdf files. These files are currently ignored by GenTS. The files are metrics from staits/passages/channels, examples of which are here:

b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Mozambique_Channel.0009-12.nc.0219
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Pacific_undercurrent.0009-12.nc.0254
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Pacific_undercurrent.0009-12.nc.0278
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Robeson_Channel.0009-12.nc.0480
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Robeson_Channel.0009-12.nc.0504
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Taiwan_Luzon.0009-12.nc.0343
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Windward_Passage.0009-12.nc.0355
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Yucatan_Channel.0009-12.nc.0354

By convention these files should end in ".nc". I will raise this issue with the CESM oceanographers. One option would be to rename the files from (say):
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Mozambique_Channel.0009-12.nc.0219
to:
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Mozambique_Channel_0219.0009-12.nc

I am unsure though how much control CESM has over MOM6 output, so there is a chance the output does not change, and GenTS will need to read this data.

[phillips-ad]

The oceanographers have limited control of the output of MOM6, so GenTS will have to run with these files as they are. Example files are here:
casper: /glade/derecho/scratch/asphilli/archive/GenTS_testdata/CESM3_atmlndiceocn_monday/ocn/hist/

Concatenation over time will be a bit more complex with these files.

Example file names for the Drake Passage for 000910-000912:
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-10.nc.0057
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-10.nc.0058
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-10.nc.0083
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-10.nc.0084
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-10.nc.0109
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-10.nc.0110
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-11.nc.0057
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-11.nc.0058
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-11.nc.0083
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-11.nc.0084
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-11.nc.0109
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-11.nc.0110
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-12.nc.0057
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-12.nc.0058
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-12.nc.0083
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-12.nc.0084
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-12.nc.0109
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage.0009-12.nc.0110

Of note: The same variable is present in multiple files for the same time step but over differing domains. For example, thetao is present in all file names that end in 0058, 0084 and 0110. But the dimensions of the yh_sub01 arrays differ amongst the 3 files. Therefore, each variable should be concatenated across all file names that end in the same number.

For example, in the above file name list, thetao should be concatenated across all the *.0058 files and placed in a file named:
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage_0058.thetao.000910-000912.nc
thetao should be concatenated across all the *.0084 files and placed in a file named:
b.e30_alpha07c_cesm.B1850C_LTso.ne30_t232_wgx3.226.mom6.h.Drake_Passage_0084.thetao.000910-000912.nc
and so on.

@gustavo-marques can you please confirm the above as to how GenTS should proceed with creating CESM timeseries files from these MOM6 output file types?

[AgentOxygen]

I changed the glob pattern from *.nc to *.nc* to include these files, and made it customizable when instantiating an HFCollection. My only concern would be any sort of temporary files that match this pattern could cause an IO error. GenTS checks for validity before concatenation, so it should happen there. I want to eventually add some IO error handling in a separate branch, though I am still thinking about how to write unit tests for that

[AgentOxygen]

#28

[gustavo-marques]

Thanks for working on this!
In addition to concatenating in time, the tiles need to be combined as well (e.g., Drake_Passage.0009-12.nc.0109, Drake_Passage.0009-12.nc.0110, Drake_Passage.0009-12.nc.0084, etc). Otherwise, the time series will represent only a particular tile (or region).

Here is an example of how to combine the tiles:

https://github.com/NCAR/mom6-tools/blob/main/docs/source/examples/combining-tiles.ipynb

[AgentOxygen]

Ah, my ADHD brain totally misread the original issue and I didn't realize these were spatially different history files. This seems like a larger issue since stitching tiles together requires interpreting the grid configuration. I am unsure whether GenTS should handle this because I want to be able to generalize it to other models, so maybe spatial concatenation of time series files should be handled by another component after GenTS time concatenation? I could make it a separate module within GenTs potentially