Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
71085bd
wip: draft new dataset module
xiqiangliu Jan 2, 2025
7aef6fa
fix: update cdsapi dependency version
xiqiangliu Jan 4, 2025
e13ca1f
wip: MERRA2 dataset
xiqiangliu Jan 6, 2025
78e875e
feat: add some functionalities to base datasets class
xiqiangliu Jan 7, 2025
2cf4580
wip: MERRA2 dataset
xiqiangliu Jan 7, 2025
f1ef222
feat: draft impl of HRRR hourly wind dataset for CONUS
xiqiangliu Jan 16, 2025
32c69d1
feat: draft impl of wind_solar hourly dataset for HRRR
xiqiangliu Jan 16, 2025
e05be6e
feat: enhance HRRR dataset processing with postprocessing methods and…
xiqiangliu Feb 2, 2025
bf5187c
feat: Use AtomicDataset to represent individual xarray files rather t…
xiqiangliu Feb 4, 2025
f4d96cc
feat: update HRRR dataset processing to include WRF data and improve …
xiqiangliu Feb 6, 2025
c4a29b2
misc: let uv sync keep editable self install
xiqiangliu Feb 17, 2025
122eda5
feat: incorporate hybrid levels into 3D dataset
xiqiangliu Feb 18, 2025
8e51eb9
misc: add devcontainer config
xiqiangliu Feb 19, 2025
b465928
wip: make cutout interoperable with new dataset
xiqiangliu Feb 19, 2025
c6ad195
feat: add wind_solar variables for HRRR
xiqiangliu Feb 21, 2025
baafcbb
feat: finalize HRRR 3D wind dataset
xiqiangliu Feb 21, 2025
69177c5
fix: make all datasets directories modules
xiqiangliu Feb 28, 2025
bbcdbf4
misc: remove progressbar as a dependency of the package
xiqiangliu Mar 3, 2025
31e22c0
feat: cutout's preparation is now working with the new dataset
xiqiangliu Mar 3, 2025
659566b
fix: proper preparation check and refactorization of conversion methods
xiqiangliu Mar 4, 2025
0a338c0
fix: limit wildcard import from plot module
xiqiangliu Mar 4, 2025
d212392
draft: ERA5 Wind 3D Dataset
xiqiangliu Mar 4, 2025
f23847f
feat: transition ERA5 3D wind to daily storage due to CDS OOM issue
xiqiangliu Mar 29, 2025
3e2ff8a
fix: skip atomic files during download if they are already present
xiqiangliu Mar 30, 2025
8681039
feat: MERRA2 support for multi-file datasets with slv_flux_hourly
xiqiangliu Apr 9, 2025
6a5711d
feat: add MERRA2 daily and hourly datasets with surface flux, radiati…
xiqiangliu Apr 9, 2025
fa53f56
feat: add monthly datasets for MERRA2 including surface flux and radi…
xiqiangliu Apr 9, 2025
6ac68a3
faet: support automatic retry for ERA datasets
xiqiangliu Apr 10, 2025
a86bdfa
feat: register HRRR dataset and update weather configuration for hour…
xiqiangliu Apr 10, 2025
e29fa95
feat: update HRRR dataset processing to reindex time and remove valid…
xiqiangliu Apr 11, 2025
aa38c44
fix: update model to conform with new dataset
xiqiangliu Apr 16, 2025
7fac81c
fix: conform model interface to new dataset interface
xiqiangliu Apr 17, 2025
bbac2c4
fix: save file integrity check state
xiqiangliu Apr 17, 2025
3c19d2c
feat: support partial saving of parameters during interruption
xiqiangliu Apr 17, 2025
2c64f4f
refactor: new model preparation logic
xiqiangliu Apr 25, 2025
0f0e791
refactor: update proper type annotation for `BaseDataset` class
xiqiangliu May 5, 2025
40a8d46
feat: implement model result classes for daily and monthly computations
xiqiangliu May 11, 2025
ccf4fae
feat: include h5netcdf for better performance
xiqiangliu May 12, 2025
c58d5e6
feat: incorporate parallel registering of model result
xiqiangliu May 12, 2025
2d53e9a
fix: few improvements
xiqiangliu May 16, 2025
63d9ebf
feat: support CF computation in wind modeling
xiqiangliu May 23, 2025
d3e552a
feat: pvlib support
xiqiangliu May 27, 2025
1765a92
docs: update
xiqiangliu Jun 2, 2025
29b9455
docs: add draft docs for dataset module
xiqiangliu Jun 3, 2025
2cdfa3d
fix: update herbie version to avoid crashing on cluster
xiqiangliu Jun 6, 2025
631aac5
fix: update model and dataset
xiqiangliu Jun 6, 2025
6f60064
fix: herbie now works on cluster
xiqiangliu Jun 6, 2025
cd9a521
fix: HRRR edge case with last day of the month
xiqiangliu Jun 8, 2025
f2b0b05
docs: update dataset overview and remove outdated pages
xiqiangliu Jun 9, 2025
90c503d
fix: update CI to conform with new dataset interface
xiqiangliu Jun 9, 2025
de43769
fix: typo
xiqiangliu Jun 9, 2025
6dc9802
fix: CI update
xiqiangliu Jun 10, 2025
edf8601
docs: add more extensive docs on wind estimation
xiqiangliu Jun 12, 2025
2db43e3
ERA5: fix 2m dewpoint key (2m_dewpoint_temperature) in wind_solar.py
michaelcjanelle Aug 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM mcr.microsoft.com/devcontainers/base:bookworm

USER root
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
&& apt-get -y install --no-install-recommends libgdal-dev \
&& rm -rf /var/lib/apt/lists/*

USER vscode
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
30 changes: 30 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/debian
{
"name": "Debian",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"build": {
// Path is relative to the devcontainer.json file.
"dockerfile": "Dockerfile"
},
"postCreateCommand": "uv sync",
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"charliermarsh.ruff"
]
}
},
"mounts": [
"source=${localEnv:HOME}${localEnv:USERPROFILE}/.local/geodata,target=/home/vscode/.local/geodata,type=bind"
]
// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Configure tool-specific properties.
// "customizations": {},
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root"
}
2 changes: 1 addition & 1 deletion .github/workflows/dev_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
max-parallel: 10
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
os: [ubuntu-latest, macos-latest]
python-version: [3.10.11, 3.12.6] # These versions are compatibale with all three OS on GitHub Actions
steps:
- uses: actions/checkout@v3
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
max-parallel: 10
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
os: [ubuntu-latest, macos-latest]
python-version: [3.10.11, 3.12.6] # These versions are compatibale with all three OS on GitHub Actions
steps:
- uses: actions/checkout@v3
Expand Down
657 changes: 0 additions & 657 deletions docs/source/application/wind_estimation.ipynb

This file was deleted.

2 changes: 1 addition & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from geodata import __version__

project = "Geodata"
copyright = "2023, Geodata Contributors"
copyright = "2025, Geodata Contributors"
author = "Geodata Contributors"
release = __version__

Expand Down
81 changes: 81 additions & 0 deletions docs/source/datasets/era5.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
ERA5 Specific Instructions
==========================

This page explains how you can set up access to ERA5 data from the `Copernicus Data Store <https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview>`_.

Creating a CDS account
----------------------

To download ERA5 data from the CDS, you'll need to create a free `CDS account here <https://cds.climate.copernicus.eu/>`_.

Download data through CDS API
-----------------------------

Once your account has been created, set up access to the API by following these steps:

1. Log into your CDS account and visit your `profile page <https://cds.climate.copernicus.eu/profile>`_.
2. Install the API key. There will be a section called **Personal Access Token**.
Copy these two lines into a file called ``.cdsapirc`` in your user root folder.

- **macOS/Linux**: Open a terminal and run:

.. code-block:: bash

touch ~/.cdsapirc

Then add the lines using:

.. code-block:: bash

echo [line 1 of the code] >> ~/.cdsapirc
echo [line 2 of the code] >> ~/.cdsapirc


- **Windows**: The process is slightly more complicated. Please refer to the in-depth guide at the Copernicus Knowledge Base `here <https://confluence.ecmwf.int/display/CKB/How+to+install+and+use+CDS+API+on+Windows>`_.

3. Install the CDS API client by opening a terminal/shell and running

.. code-block:: bash

pip install ".[download]"

(Assuming you are in Geodata's *root directory*.)

1. Once you've installed the API key and the API client, confirm access by running an
example in a Python script or a Jupyter notebook:

.. code-block:: python

import cdsapi

c = cdsapi.Client()

c.retrieve(
"reanalysis-era5-single-levels",
{
"product_type": "reanalysis",
"format": "netcdf",
"variable": [
"2m_dewpoint_temperature",
"2m_temperature",
],
"year": "2011",
"month": [
"01",
],
"day": ["01", "02", "03"],
"time": [
"00:00",
"12:00",
],
},
"download.nc",
)

The above example downloads 2m temperature and 2m dewpoint temperature with data points
at 00:00 and 12:00 for each day, from January 1-3, 2011, in NetCDF format.

If this works, you have successfully set up access to the ERA5 data through the CDS API.
Please subsequently refer to the `general documentation on datasets <../overview.rst>`_
for more information on how to download ERA5-based datasets using the ``geodata``
package.
Loading
Loading