Support UGRID conventions?

[rsignell]

I have some triangular mesh FVCOM model output in a public icechunk that supports the UGRID conventions for unstructured grid models.

The icechunk dataset renders in gridlook (hurray!), but as colored dots, which I'm assuming means that gridlook couldn't figure out how to use the connectivity array, right?

Here the gridlook URL

And a snapshot of what this looks like:

The xugrid python package handles the UGRID conventions -- maybe an agent could just convert the UGRID logic to javascript? Or maybe the conventions have clear enough documentation that the agent could just figure it out from that!

[Karinon]

Thanks for the dataset.

First of all, this specific grid does not seem to be something too crazy, at least with some vibecoding I could get a result like this quite quickly:

That being said, there is something extremely funky going on with this dataset as the results are not reproducible at all for all of the variables. Sometimes I get a result like this one here (main branch):

On other occasions (or maybe the same ones) values simply change on a reload or when I switch between two variables back and forth with the arrow buttons. I would tend to say that this is something in gridlook, but this is the only dataset where I have ever experienced something like this. Could also be something with the icechunk implementation as I haven't made many tests with it, yet

[eeholmes]

When I tried to load Rich's link (hi Rich!), I get an error, but I can load 'temp'.
https://gridlook.pages.dev/#https://umassd-fvcom.s3.amazonaws.com/gom3/hindcast/icechunk/gom3-whole-nochunk.icechunk::varname=temp::camerastate=Jcw5DsJAEAXRuzgeRt39e-UqyAEBgRObxUSIuzMWaan0PtN9ey37sq3T-YIOCoIWRxRrcjsx9RBWTwUhYJUtO8LNi9302Gxu0-N93W_P9c9QVwVnZHESw1MadXKPGklhoZDhUGdAQGbF5KJ6pEyUiQWQUmObvz8::dimIndices_time=0::dimIndices_siglay=0

Try switching back and forth between irregular and irregular_delauney

Result is wild.

Note, I changed something in my implementation when fixing curivilinear since eeholmes.github.io/gridlook doesn't do this.

https://eeholmes.github.io/gridlook/#https://umassd-fvcom.s3.amazonaws.com/gom3/hindcast/icechunk/gom3-whole-nochunk.icechunk::varname=temp::catalog=static/catalog.json::projectionCenterLat=0::projectionCenterLon=0::dimIndices_time=0::dimIndices_siglay=0::camerastate=JYw7DsJAEMXuknpZzX_ecBWUgoIiTcInVIi7syKlLcuf6b69ln3Z1ul84W5eleygQDop2km7lYk4ZWSmGqJJL6Ew4hRCwH1u0-N93W_P9fhQNxXPFJgWWJ2tUR9taoqaclng7zhjoKBCmOAxFLKKCCMsRLnO3x8

regular

regular_delaunay

[Karinon]

Your changes with the curvilinear grid shouldn't affect the irregular-implementation, but on the other hand I don't know which parts you touched. It really seems, that the temp-variable doesn't do any funky things on your side, but the other vars still do. From what I saw they all are supposed to work, not sure why your changes only working correctly on temp.

I tried to extract the actual issue and it seems to have something to do with the caching. It works correctly, when caching is disabled (comment out the line (s) => zarr.withByteCaching(s, { cache: cache }) in ZarrDataManager.ts). With caching enabled I get wrong results every second get. I created a minimal example. Getting the first 10 values for temp 4 times in a row looks like this when caching is enabled:

round 1
temp {
  dtype: 'float32',
  shape: [ 342348, 45, 48451 ],
  chunks: [ 1, 45, 48451 ],
  dimensions: undefined,
  first10: [
    1.7179484367370605,
    1.7143038511276245,
    1.8022050857543945,
    1.9406880140304565,
    1.7471094131469727,
    1.7225805521011353,
    1.7378705739974976,
    1.7851808071136475,
    1.8042963743209839,
    1.812827467918396
  ]
}
byte cache { hits: 0, misses: 4, sets: 4, size: 4 }

round 2
temp {
  dtype: 'float32',
  shape: [ 342348, 45, 48451 ],
  chunks: [ 1, 45, 48451 ],
  dimensions: undefined,
  first10: [
    -0.028058646246790886,
    4007345920,
    -1.9417733954497744e-14,
    4.725187109906344e+33,
    331513.96875,
    -1.1936455648914331e-35,
    -4.6774704031896193e-32,
    -1081220992,
    4.4365042195515514e-10,
    -0.002119555836543441
  ]
}
byte cache { hits: 2, misses: 4, sets: 4, size: 4 }

round 3
temp {
  dtype: 'float32',
  shape: [ 342348, 45, 48451 ],
  chunks: [ 1, 45, 48451 ],
  dimensions: undefined,
  first10: [
    1.7179484367370605,
    1.7143038511276245,
    1.8022050857543945,
    1.9406880140304565,
    1.7471094131469727,
    1.7225805521011353,
    1.7378705739974976,
    1.7851808071136475,
    1.8042963743209839,
    1.812827467918396
  ]
}
byte cache { hits: 4, misses: 4, sets: 4, size: 4 }

round 4
temp {
  dtype: 'float32',
  shape: [ 342348, 45, 48451 ],
  chunks: [ 1, 45, 48451 ],
  dimensions: undefined,
  first10: [
    -0.028058646246790886,
    4007345920,
    -1.9417733954497744e-14,
    4.725187109906344e+33,
    331513.96875,
    -1.1936455648914331e-35,
    -4.6774704031896193e-32,
    -1081220992,
    4.4365042195515514e-10,
    -0.002119555836543441
  ]
}

Still figuring out which part is buggy here. Could be a zarrita or an icechunk issue

[Karinon]

I am quite certain this is a zarrita-issue. This dataset is big-endian. When running on a little-endian machine it swaps the bytes on every readout, which means only every second readout is correct... I will create an issue there. As soon as we have a solution for this, I will take a closer look on the dataset.

[Karinon]

manzt/zarrita.js#431

[rsignell]

Nice work @Karinon! I totally forgot that this virtual Icechunk points at byte ranges in NetCDF3-64bit offset files, which is quite unusual these days.

[rsignell]

Also a note on FVCOM datasets like this: most triangular mesh models have their connectivity array dimensioned as (nodes,3), but FVCOM has the array dimensioned as (3, nodes). This is captured by the UGRID face_dimension attribute.

It would be great to support UGRID. Every tool that supports UGRID makes model providers more likely to include them when they create their Zarr or Icechunk datasets!