Call for data

[Karinon]

This is an open call for funky datasets! I don't care if it is Zarr V2, V3 or Icechunk. I am also open for anything behind some kind of login wall (as long as I am able to get to the data eventually) as I want to support token-authentication at some point.

If you think that gridlook should support your dataset, please answer to this request.

If your data is already supported by gridlook and shows some kind of strange quirks (I am open to anything: Invalid date, strange fragments, suprising error messages despite it is working in general) please respond aswell.

[eeholmes]

Here is my catalog of examples. Lots of funky grids. Note many of these are grouped Zarr v2 and v3 (like multilevel) which I don't think GridLook handles yet

https://github.com/eeholmes/gridlook/blob/main/public/static/catalog.json

Example:
https://eeholmes.github.io/gridlook/#https://atlantis-vis-o.s3-ext.jc.rl.ac.uk/noc-npd-era5-demo/npd-eorca1-era5v1/gn/U1y/uo2::catalog=static/catalog.json::projectionCenterLat=0::projectionCenterLon=0::varname=0/uo::dimIndices_depthu=0::dimIndices_time=0

[Karinon]

Found this one here: Doesn't work with current gridlook. I could make it work locally, but the amount of created code was not worth it..

https://whiffle-wins50-data.s3.amazonaws.com/data/whiffle_wins50_2020_scenario_turbines.zarr

[eeholmes]

New animation feature is great!

Here is an icechunk store with big 4D chunks 10 (z_l) x 100 (time) x 200 x 200. Note, this dataset loads fine. This just illustrates the animation feature for 4D chunks where the 4th D is chunked so 10x100x200x200 instead of 1x100x200x200.

https://gridlook.pages.dev/#https://data.source.coop/eeholmes/cefi/nepacific-icechunk::catalog=static/catalog.json::projectionCenterLat=0::projectionCenterLon=0::varname=monthly/main/chl_Di::dimIndices_time=67::camerastate=Jcw5DsJAEAXRuzgej_r33lwFERAQOLFZTIS4O2ORlkrvM92317Iv2zqdzjPQSSLMNYOVg6zN1sXKSUQLXKBqoJE4SwFLEw67tOnxvu635_qHjiFgVeqUom2mziyOIQ4YIiVH0wSrI0dhj0SjHp5u48kyVvK4fH8::dimIndices_z_l=0

Illustrates:

• slow load (big chunks)
• animation plays ok but slower than I might expect since I though it would cache a whole chunk

Here is a 3D var from the same dataset 100 (time) x 200 x 200
https://gridlook.pages.dev/#https://data.source.coop/eeholmes/cefi/nepacific-icechunk::catalog=static/catalog.json::projectionCenterLat=0::projectionCenterLon=0::varname=monthly/main/MIB::camerastate=Jcw5DsIwEEDRu7i2R7MvXAWloKBIk7CECnF3jCj_L9673fbneqz71k7nkUCBUcgeKhnUhwGqkkilejoTdWJIdxQkFStGraW3--tyXB_bn0HgMMrQiUlUFVkfCGRiNNOMy0JUflOdxTWc1RDLoyMkZ2KSziXi4cvnCw::dimIndices_time=5::dimIndices_z_l=0

• fast animation

[Karinon]

Thanks for the dataset and your findings. Right now I can't tell you why it is so much slower, but it is definitely not the network-traffic. At least not after the first load. I had the network-tab open during a time increment and no additional data has been fetched. So it is related to the rendering or some calculations happening in the browser. But it is still worth doing some more investigations on this one.

[eeholmes]

BTW I detached https://github.com/eeholmes/gridlook-xl from the fork network so you won't automatically get updates on what I am doing there. Plan is to fork your current gridlook and slowly start migrating my needed features into this new fork. I think all the datasets that eeholmes opens now open in gridlook, except the pcodecs one.

These are the bits I will add in my new fork

• grouped Zarr support for specifying url to subgroups and not just the root Url to group in grouped Zarr does not load #177
• Support for log10 transforms as like 10 of the variables that I come across could use this Add Log10 display transform mode and WebGL-safe NaN handling #161
• More sort buttons and tags for the catalog, but that is just to ease my debugging to help me find datasets with particular features. Temporary

Bits to still solve: What is needed to interpret xy coords successfully? I did lots of hacky work to get various xy grids to be recognized. Don't like that approach at all. Feels like 2 pieces. 1) logic to interpret xy grids. 2) getting info out to Zarr creators on what info needs to be in Zarr metadata. Too much guessing was involved.

[Karinon]

Thanks for the notification. Your first two issues will be implemented into our official version eventually. Not so sure about the catalog-features. I absolutely like to have them in your catalog for debugging reasons, as you said, but I don't think they are necessary for the average user. Maybe I am going to have some discussions with other people about it.

Bits to still solve: What is needed to interpret xy coords successfully? I did lots of hacky work to get various xy grids to be recognized. Don't like that approach at all. Feels like 2 pieces. 1) logic to interpret xy grids. 2) getting info out to Zarr creators on what info needs to be in Zarr metadata. Too much guessing was involved.

I had some brief discussions with @wachsylon about it (not xy coords in particular, but all the things happening in gridTypeDetector). In the end we have to support a few conventions (dggs, geozarr, cf) in a more standardized way (easier said than done). Any decisions beyond that is up to discussions and I don't think we have to support everything (I am also not sure whether we really need to support all the different xy-flavours which we already do. I am not very familiar with the common standards, but some of the datasets felt already quite strange). One of my personal goals with Gridlook is also to guilt-trip users into providing better metadata ;). There is lots of data out there which simply does not work because the authors did a very bad job with their metadata and I think they have to feel the consequences. Either way, the grid detection will probably remain a moving target..