Add xarray-sql as a Python REPL tool to make it effortless to join raster and tabular data

[alxmrs]

Hey! I was excited to see this project today on LinkedIn. I'm also a fan of the EarthMover platform and their ERA5 dataset. Great work, awesome repo!

This is project is very similar to a demo that I've been fixing to make for some time, but never got around to: alxmrs/xarray-sql#45. I've started reading some of your sources, and I like your approach.

My repo, https://github.com/alxmrs/xarray-sql, is still in its early phase of development, but it I believe it fills a gap in the current standard set of PyData tools for working with weather data, namely that is is difficult to join tabular and raster data at scale. To illustrate the kind of operation that I am talking about, check out this demo notebook (the join in near the bottom).

(TL;DR: This joins monthly aggregates of ERA5 data with yearly energy production data in the Texas energy grid by treating the gridded raster data as a table).

I would love to see the Python REPL in the tools portion of the repo be equipped with an XQL context, e.g.:

import xarray_sql as xql
ctx = xql.XarrayContext()
ctx.from_dataset("era5", ds)  # the dataset needs to be chunked somehow)

From their, your AI agent would be able to make queries and aggregations of ERA5 (or any Xarray dataset) via SQL.

Fair warning: my project is still in its early days and has sharp edges. But I believe that SQL is an interface that LLMs are well equipped to leverage to wield data well.

[dmpantiu]

Thanks a lot for looking at our project. For now we are trying to use only well-established tools with a lot of examples in the LLM training dataset, so that the results are robust (even that is challenging). Happy to consider it when the project is more mature.

[alxmrs]

Thanks for the response. You're right to be hesitant about xarray-sql, as it definitely has many ways it can improve. However, all the library does is make Xarray datasets readable via SQL. I suspect this would be exciting for this project because there are probably more examples of SQL in the LLM's training data than all the other libraries combined.

I'll be sure to study this weather agent in the future, and maybe consider making experiments in a fork. Thank you for putting together a great demo.