[RES] dask, xarray (, numba) for gedipy

[slumnitz]

Hello @armstonj

as discussed before I took some time investigating the use of dask/ xarray for data processing and integration into gedipy.

You can find insight I gathered about the integration into the current library in this notebook and general information about data processing with h5py, xarray and dask here.

Overall I would say xarray and dask

• could be useful for new functionality i.e. taking the output of read_shots() or with an addition for reading and writing data.
• still also have limitations for reshaping arrays i.e. 1D waveform to 2D (or require very specific expert knowledge)
  (As a side note I have not used dask too much before and there might be better way's of going about using it. I found it very interesting to get to know it more and I think it's overall benefit shows on distributed clusters not necessarily on small machines, which I used. More in the notebooks)

Let me know if you would find it interesting to merge my experiments and I'd be happy to set up a PR. There were also a couple general additions in my dask-integration branch I could extract if they are of interest. I hope this helps the library a little.

[armstonj]

Yes I would be interested in merging your experiments, and enhancements such as use of numpy.broadcast_to

Implementing new methods such as .to_xarray() or .to_dask() and rename .read_shots to to_nparray() sounds like a good place to start, since wrappers will be needed to handle GEDI's non-standard format and subgroups. I'm keen to learn more about it, so will run though your notebooks when I get a chance.

Something to keep in mind is that for version 2 L1 and L2 files (will be available this year, sometime after September), GEDI will be:

• moving to sub-orbit granules (4 per orbit) so there will be smaller files to process, but 4x as many
• optimizing chunksize for non-waveform datasets