Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/ci_notebook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:

jobs:
notebook-test:
name: Lint & Tests check
name: Notebook check
runs-on: ubuntu-latest

steps:
Expand Down Expand Up @@ -37,7 +37,6 @@ jobs:

papermill pixcdust/notebooks/convert/download_pixc_to_gpkg.ipynb /tmp/notebooks/download_pixc_to_gpkg
papermill pixcdust/notebooks/convert/download_pixc_to_zarr.ipynb /tmp/notebooks/download_pixc_to_zarr
papermill pixcdust/notebooks/convert/convert_to_zcollection.ipynb /tmp/notebooks/convert_to_zcollection
papermill pixcdust/notebooks/convert/dggs_tuto.ipynb /tmp/notebooks/dggs_tuto
papermill pixcdust/notebooks/read_and_use/read_netcdf.ipynb /tmp/notebooks/read_netcdf

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Then, carefully store your API-Key :

## 🪄 *Converters*
The converter classes allow you to create more easy-to-use databases than the original netcdf4 format. The various databases are designed for **local** studies, not for huge country-scale databases (though it should work, they will not be efficient).
Zarr (with zcollection), geopackage and shapefile are currently supported.
Zarr, geopackage and shapefile are currently supported.
The converters allow you to limit the databases to areas of interest (provided by polygons) and variables of interest (limitated to the pixel_cloud group mono-dimensional variables).
Users are encouraged to limit the number of variables to what is useful, especially for geopackage format, but also for the planet ;)

Expand All @@ -53,9 +53,9 @@ python tests/init_tests.py -I INPUT_FOLDER -H HYDROWEB_AUTH

You then can run the converters tests:
```
pytest
pytest pixcdust/tests --realdata
```
You can also run all the tests including the downloaders tests with:
```
pytest --dl
pytest pixcdust/tests --realdata --dl
```
20 changes: 11 additions & 9 deletions doc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ pixcdust documentation
======================

🎩 🪄 A Pinch of PixC Dust 🐇
----------------------------
-----------------------------

This python project centralizes librairies to facilitate local studies based on SWOT-HR Level-2 Pixel Cloud products.

Expand All @@ -34,14 +34,14 @@ Start here to understand what you can do: :doc:`tutorials`.
----------------
The downloader classes allow you to directly download SWOT Pixel Cloud files from hydroweb.next (or other sources such as PO.DAAC to be implemented).
For hydroweb.next, it requires you to create an account and an API Key (token) from the platform: https://hydroweb.next.theia-land.fr.
Then, carefully store your API-Key either :
* in your eodag configuration file (usually `~/.config/eodag/eodag.yml`, automatically generated the first time you use eodag) in ``auth/credentials/apikey="PLEASE_CHANGE_ME"``
* in an environment variable ``export EODAG__HYDROWEB_NEXT__AUTH__CREDENTIALS__APIKEY="PLEASE_CHANGE_ME"``
Then, carefully store your API-Key either
* in your eodag configuration file: (usually `~/.config/eodag/eodag.yml`, automatically generated the first time you use eodag) in ``auth/credentials/apikey="PLEASE_CHANGE_ME"``
* in an environment variable: ``export EODAG__HYDROWEB_NEXT__AUTH__CREDENTIALS__APIKEY="PLEASE_CHANGE_ME"``

🪄 *Converters*
---------------
The converter classes allow you to create more easy-to-use databases than the original netcdf4 format. The various databases are designed for **local** studies, not for huge country-scale databases (though it should work, they will not be efficient).
Zarr (with zcollection), geopackage and shapefile are currently supported.
Zarr, geopackage and shapefile are currently supported.
The converters allow you to limit the databases to areas of interest (provided by polygons) and variables of interest (limitated to the pixel_cloud group mono-dimensional variables).
Users are encouraged to limit the number of variables to what is useful, especially for geopackage format, but also for the planet ;)

Expand All @@ -56,23 +56,25 @@ Here are some python script implementing the classes.
🔶 *Discrete Global Grid System (experimental)*
-----------------------------------------------

I enjoy DDGS a lot. It is pretty great if you want to perform on-the-fly "rasterization", partitionning, comparing pixels over time or space, etc.
I enjoy DGGS a lot. It is pretty great if you want to perform on-the-fly "rasterization", partitionning, comparing pixels over time or space, etc.
Currently H3 and HEALPix are implemented.

✅ *Tests*
---------
----------
You first need to configure the tests and download the tests data with init_tests.py. We recommend setting the following options:
* INPUT_FOLDER is where the tests data will be downloaded (or are already available).
* HYDROWEB_AUTH is your hydroweb.next API key. It is required to automatically download the tests data or run the downloaders tests.

``python tests/init_tests.py -I INPUT_FOLDER -H HYDROWEB_AUTH``

You then can run the converters tests:
``pytest``

``pytest pixcdust/tests --realdata``


You can also run all the tests including the downloaders tests with:
``pytest --dl``

``pytest pixcdust/tests --realdata --dl``


.. toctree::
Expand Down
58 changes: 25 additions & 33 deletions pixcdust/converters/zarr.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,18 @@
from pathlib import Path

import dask
import dask.distributed
import dask.utils
import fsspec
import geopandas as gpd
import zcollection
import zcollection.indexing
import pandas

from pixcdust.converters.core import Converter
from pixcdust.readers.netcdf import NcSimpleConstants, NcSimpleReader

TIME_VARNAME = "time"
# TODO is there a better solution than fix the chunk size here ?
CHUNK_SIZE = dask.utils.parse_bytes("2MiB")


class Nc2ZarrConverter(Converter):
Expand Down Expand Up @@ -74,7 +76,6 @@ def __init__(
area_of_interest=area_of_interest,
conditions=conditions,
)
self.collection: zcollection.collection.Collection = None
self.__time_varname: str = TIME_VARNAME
self.__fs = fsspec.filesystem("file")
self.__chunk_size = dask.utils.parse_bytes("2MiB")
Expand All @@ -85,6 +86,9 @@ def database_from_nc(self, path_out: str | Path, mode: str = "w") -> None:
if mode in ["o", "overwrite"] and os.path.exists(path_out):
shutil.rmtree(path_out)

path_out = Path(path_out)
append = path_out.exists()

with (
dask.distributed.LocalCluster(processes=True) as cluster,
dask.distributed.Client(cluster) as _,
Expand All @@ -106,41 +110,29 @@ def database_from_nc(self, path_out: str | Path, mode: str = "w") -> None:
)

time = xr_ds_xarray["time"].values
import pandas
import xarray

time2 = pandas.DatetimeIndex(time).tz_localize(None).to_numpy()
xr_ds_xarray["time"] = xarray.DataArray(time2, dims=("points",))
zc_ds = zcollection.Dataset.from_xarray(xr_ds_xarray)

# zc_ds = zcollection.Dataset.from_xarray(
# xr_ds.to_xarray().drop_vars(self.__cst.default_added_points_name),
# )
zc_ds.block_size_limit = self.__chunk_size
zc_ds.chunks = {next(iter(zc_ds.dimensions.keys())): self.__chunk_size}

init = True
if not os.path.exists(path_out) and init:
partition_handler = zcollection.partitioning.Date(
(xr_ds.cst.default_added_time_name,),
"s",
)
xr_ds_xarray = xr_ds_xarray.assign_coords(time=("points", time2))

self.collection = zcollection.create_collection(
axis=self.__time_varname,
ds=zc_ds,
partition_handler=partition_handler,
partition_base_dir=path_out,
filesystem=self.__fs,
)
init = False
# Approximate the original 2 MiB chunk-size target.
npoints = xr_ds_xarray.sizes.get("points", 0)
if npoints:
bytes_per_point = max(1, xr_ds_xarray.nbytes // npoints)
chunk_points = max(1, CHUNK_SIZE // bytes_per_point)
xr_ds_xarray = xr_ds_xarray.chunk({"points": int(chunk_points)})

if append:
xr_ds_xarray.to_zarr(
path_out,
mode="a",
append_dim="points",
consolidated=True,
zarr_version=2,
)
else:
self.collection = zcollection.open_collection(
xr_ds_xarray.to_zarr(
path_out,
filesystem=self.__fs,
mode="w",
consolidated=True,
zarr_version=2,
)
self.collection.insert(
zc_ds, merge_callable=zcollection.collection.merging.merge_time_series
)
Loading
Loading