Proposal: Adopt `grib2io` AWS streaming capabilities for data pipelines

[bbakernoaa]

Hi everyone,

Upstream grib2io recently gained the ability to stream data directly from AWS (see PR: NOAA-MDL/grib2io#232). I wanted to open this issue to discuss how we can leverage this for our workflows.

Currently, downloading and temporarily storing GRIB2 files can be a bottleneck. By utilizing this new streaming capability, we could achieve:

• Faster Processing: Read data directly into memory without waiting for full file downloads.
• Reduced Storage footprint: Eliminate the need for temporary local storage volumes.
• Cloud-Native Efficiency: Easier integration with serverless or containerized environments.

Are there specific pipelines or future applications where we could pilot this? If there's interest, I'd be happy to put together a quick proof-of-concept to benchmark the performance difference.

[LinlinCui-NOAA]

Hi @bbakernoaa,

Thanks for the proposal. I know xarray has this feature for the cfgrib engine, so it would be great to adopt the grib2io AWS streaming capability.

Would you mind opening this issue on the MLGlobal repo instead? That repository hosts the workflows for running AIGFS and AIGEFS, whereas the graphcast repo is currently used only for training purposes.

A quick proof-of-concept would be very helpful.

[bbakernoaa]

grib2io also has an xarray engine. Sure I can open a new issue on that repo Barry Baker National Oceanic and Atmospheric Administration National Weather Service Environmental Modeling Division Physical Research Scientist Air Composition Modeling Lead NCWCP 5830 University Research Court College Park, Maryland 20740 Phone: ‪(301) 683-1395‬

…

On Wed, Feb 25, 2026 at 9:07 PM Linlin Cui ***@***.***> wrote: *LinlinCui-NOAA* left a comment (NOAA-EMC/graphcast#73) <#73 (comment)> Hi @bbakernoaa <https://github.com/bbakernoaa>, Thanks for the proposal. I know xarray has this feature for the cfgrib engine, so it would be great to adopt the grib2io AWS streaming capability. Would you mind opening this issue on the MLGlobal repo <https://github.com/NOAA-EMC/MLGlobal> instead? That repository hosts the workflows for running AIGFS and AIGEFS, whereas the graphcast repo is currently used only for training purposes. A quick proof-of-concept would be very helpful. — Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIUVN7WPPIISTINYDEYD6T4NZIMPAVCNFSM6AAAAACV7MVT4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSNRTGQ4DMNJQGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>