Issues using same versions as pangeo notebook docker image for our pangeo image

[grallewellyn]

We would like to use the same package versions from pangeo notebook v2024.11.11 found here: https://github.com/pangeo-data/pangeo-docker-images/blob/2024.11.11/pangeo-notebook/packages.txt

However, if I pin all those package versions and leave our packages that aren't on that list unpinned, then the builds fail because of the xcape, geogif, and basemap packages. Doing a flexible install removing the bug fix version pin for the versions from pangeo notebook doesn't solve this problem
Failed build: https://repo.dit.maap-project.org/root/maap-workspaces/-/jobs/16163
From this commit: MAAP-Project/maap-workspaces@0078739 that includes xcape, geogif, and basemap

We agreed that ideally we don't remove the basemap package. basemap hasn't had a release in 9 months

Related to #1102

cc @wildintellect

[wildintellect]

@grallewellyn I don't see any of those in the pangeo packages list, are they dependencies of something or MAAP only add ons?

[grallewellyn]

MAAP only add ons likely requested by users at some point: https://docs.google.com/spreadsheets/d/1krnOZ1SFW-GA_jOiL-nWzhNAA3JKTNqFrG0IpObHsBg/edit?usp=sharing
I don't see basemap on that sheet though, only geogif and xcape

[wildintellect]

I haven't tried it yet but I think this tool will help us trace where basemap comes from https://pixi.sh/latest/reference/cli/#tree

[wildintellect]

Researching xcape it was in Pangeo until a few months ago when they moved to Python 3.12 and is now commented out. We should probably match and drop. I don't think it was user requested.

[wildintellect]

geogif was removed from pangeo back in early 2023 due to a bad dependency pangeo-data/pangeo-docker-images#431 (comment)

We should check that it's actually up to date https://anaconda.org/conda-forge/geogif , last release was 6 months ago.

[grallewellyn]

Yes, we can drop xcape

That comment about the geogif dask strict dependency was on Feb 3 2023 and Feb 9 2023 geogif relaxed the upper bound: gjoseph92/geogif@57342a1. v0.1.4 is the earliest version that this fix is available.
If I do a flexible install for all other packages, comment out xcape and basemap and pin the geogif version to be >=0.1.4, I still get dependency conflicts building the image for geogif: https://repo.dit.maap-project.org/root/maap-workspaces/-/jobs/16310

I then did

pixi init --import pangeo.xml
pixi tree

And this generated:
pixiTreeOutput.txt
geogif, xcape, and basemap do not seem to be dependencies of other packages

[wildintellect]

@weiji14 do you have any opinions about these 3 packages, possibly alternatives.

@grallewellyn I say we consider dropping for now, but we should inform the UWG.

[wildintellect]

Basemap is because of a weird pinning
- package basemap-1.4.0-np126py312h745efaa_2 requires geos >=3.12.1,<3.12.2.0a0, but none of the providers can be installed
Tested on VEDAHub with 2024-08 Pangeo image that has 3.12.2 geos. Maybe there's a conda forge feedstock fix. (Just needs to be uploaded again?)
https://github.com/conda-forge/basemap-feedstock/blob/main/recipe/meta.yaml

[weiji14]

xcape

If someone has some time to look into xgcm/xcape#55, then that would be great.

geogif

Wouldn't mind adding this back, if getting a new version works. Related alternative is https://github.com/ahuang11/streamjoy perhaps.

basemap

I thought this was deprecated/unmaintained already 😅 But apparently it is still going matplotlib/basemap#527? Still wouldn't recommend adding this back, unless someone really needs this. I think cartopy is what most old basemap users have moved to.

[wildintellect]

xcape

If someone has some time to look into xgcm/xcape#55, then that would be great.

Not really our role, and afaik none of our users actually use this right now.

geogif

Wouldn't mind adding this back, if getting a new version works. Related alternative is https://github.com/ahuang11/streamjoy perhaps.

Let's drop for now, but look into options in the future.

basemap

I thought this was deprecated/unmaintained already 😅 But apparently it is still going matplotlib/basemap#527? Still wouldn't recommend adding this back, unless someone really needs this. I think cartopy is what most old basemap users have moved to.

Thanks for confirming, I was wondering if cartopy solved all this since that's already in the Pangeo image.

So @grallewellyn drop all 3 for now, and we'll just put it in the changelog/announce when the next version rolls out.

[grallewellyn]

Sounds good. I generated an updated image with everything from pangeo-notebook 11.11.2024 pinned and then got the versions of the other packages from that created image.

The process was much easier than our old strategy and I would like to continue updating this way in the future

If you are curious:

This is the new base image environment.yml: https://github.com/MAAP-Project/maap-workspaces/blob/feature/update-packages/base_images/pangeo/environment.yml
And the new jupyterlab environment.yml: https://github.com/MAAP-Project/maap-workspaces/blob/feature/update-packages/jupyterlab/pangeo/environment.yml

Successful build: https://repo.dit.maap-project.org/root/maap-workspaces/-/jobs/16352

And this is the image I created for testing: 'mas.dit.maap-project.org/root/maap-workspaces/jupyterlab/pangeo:update-packages'