diff --git a/docs/source/development/offline-era5-fixture-datasets.md b/docs/source/development/offline-era5-fixture-datasets.md new file mode 100644 index 00000000..74018c37 --- /dev/null +++ b/docs/source/development/offline-era5-fixture-datasets.md @@ -0,0 +1,82 @@ +# Offline ERA5 fixture datasets (`*_test` weather configs) + +This document records the **design and implementation plan** for small, committed NetCDF fixtures used in automated tests—without calling the CDS API or relying on `DATASET_ROOT_PATH` downloads. + +## Goals + +- Ship **minimal** ERA5-shaped files in the repository for CI and local testing. +- Expose them via **`load_dataset("…_test")`** so code paths mirror production (`wind_3d_hourly`, `wind_solar_hourly`) while staying **offline**. +- Avoid coupling tests to arbitrary year/month ranges: fixture datasets should use a **fixed catalog** (typically a single file) even if `BaseDataset.__init__` still requires `years` / `months` arguments (those values can be **ignored** for catalog construction in test configs). + +## Non-goals + +- The legacy **`geodata.dataset.Dataset`** (`module=` + `weather_data_config=` dict) is **not** in scope; the plan targets **`load_dataset` + `BaseDataset` subclasses** used by models and current tests. + +## Current fixture layout (repository) + +Fixtures live under **`tests/fixtures/`** so they stay close to pytest and do not inflate the installable package unless explicitly packaged later. + +| Test weather config (planned) | Mirrors production config | On-disk layout under `tests/fixtures/era5/` | +|-------------------------------|---------------------------|---------------------------------------------| +| `wind_3d_hourly_test` | `wind_3d_hourly` (`frequency="daily"`) | `wind_3d_hourly_test/2016/01/01.nc` | +| `wind_solar_hourly_test` | `wind_solar_hourly` (default `frequency="monthly"`) | `wind_solar_hourly_test/2016/01.nc` | + +Production datasets store files under: + +`DATASET_ROOT_PATH / / / …` + +with: + +- **Daily** (3D wind): `…///.nc` +- **Monthly** (wind/solar hourly): `…//.nc` + +The fixture tree **matches those relative paths** so `AtomicDataset.path` resolution stays aligned with the real datasets. + +## Registry and naming + +- Each test variant is a **`BaseDataset` subclass** with `weather_config = "wind_3d_hourly_test"` or `"wind_solar_hourly_test"`. +- Subclasses are registered automatically via `BaseDataset.__init_subclass__` into `geodata.datasets.registry`. +- Callers use **`load_dataset("wind_3d_hourly_test")`** (same pattern as production). + +## Behavioral contract + +### Storage root + +Fixture classes should set **`storage_root`** to the directory that contains the fixture tree for that config—for example, the absolute path to `tests/fixtures/era5/wind_3d_hourly_test` resolved at runtime (repo-relative or via `importlib.resources` if fixtures are ever packaged). + +### Catalog + +Override **`catalog`** so it returns **only** the `AtomicDataset` entries that refer to committed files (commonly **one** file): + +- 3D wind: one daily file, e.g. `(year=2016, month=1, day=1)` → `…/2016/01/01.nc` +- Wind/solar: one monthly file, e.g. `(year=2016, month=1)` → `…/2016/01.nc` + +Constructor arguments **`years` / `months`** may remain required by `BaseDataset.__init__` but **need not drive** the fixture catalog. + +### Download + +- **`download()`** must **not** call CDS: implement as a no-op or raise a clear error if invoked. +- **`_download_file`** should not perform network I/O. + +### Prepared state + +`downloaded` should become **`True`** when fixture files exist (the default `_check_downloaded()` loop over `catalog` is sufficient if paths resolve correctly). + +## Models and `SUPPORTED_WEATHER_DATA_CONFIGS` + +`BaseModel` validates both **`weather_config`** and **`source.downloaded`**. Any model that should run on fixtures must **allow** the `*_test` config names—e.g. extend `SUPPORTED_WEATHER_DATA_CONFIGS` on `WindInterpolationModel`, pvlib-related models, and any other entry points used in tests—to include `wind_3d_hourly_test` / `wind_solar_hourly_test` (or document a single shared alias strategy). + +## Implementation checklist + +1. Add **`ERA5Wind3DHourlyTestDataset`** / **`ERA5WindSolarHourlyTestDataset`** (names may vary) beside the existing ERA5 hourly classes, or in a small `fixture.py` module imported from `era5` packages so subclasses register on import. +2. Wire **`storage_root`** to `tests/fixtures/era5//` (resolve path robustly from the repo root or test layout). +3. Override **`catalog`** to the fixed fixture file(s); ignore user `years`/`months` for catalog purposes (documented). +4. Override **`download`** / **`_download_file`** to prevent CDS usage. +5. Update **`SUPPORTED_WEATHER_DATA_CONFIGS`** on affected models. +6. Add or adjust tests: `load_dataset("…_test")`, assert `downloaded`, **no** `download()`, then run the intended model or pipeline assertion. + +## References (code) + +- Registry: `geodata.datasets._base.BaseDataset.__init_subclass__` +- Paths: `AtomicDataset.path` in `geodata.datasets._base` +- Legacy downloader: `geodata.dataset.Dataset` (separate from this plan) diff --git a/docs/source/index.rst b/docs/source/index.rst index 86836036..9c19e82d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -65,6 +65,13 @@ Welcome to Geodata's documentation! .. application/* +.. toctree:: + :maxdepth: 1 + :caption: Development + :hidden: + + development/offline-era5-fixture-datasets + .. toctree:: :maxdepth: 1 :caption: API Reference diff --git a/src/geodata/datasets/era5/__init__.py b/src/geodata/datasets/era5/__init__.py index 1a1d476e..a1afc8b4 100644 --- a/src/geodata/datasets/era5/__init__.py +++ b/src/geodata/datasets/era5/__init__.py @@ -13,6 +13,6 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -from . import wind_3d, wind_solar +from . import fixture, wind_3d, wind_solar -__all__ = ["wind_3d", "wind_solar"] +__all__ = ["fixture", "wind_3d", "wind_solar"] diff --git a/src/geodata/datasets/era5/fixture.py b/src/geodata/datasets/era5/fixture.py new file mode 100644 index 00000000..28f58d07 --- /dev/null +++ b/src/geodata/datasets/era5/fixture.py @@ -0,0 +1,145 @@ +# Copyright 2025 Michael Davidson (UCSD), Xiqiang Liu (UCSD), Keyu Long (UCSD) + +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 3 of the +# License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +"""Offline ERA5 datasets backed by committed NetCDF files under ``tests/fixtures/``. + +On construction, small template files are **copied** into +``DATASET_ROOT_PATH / era5 / / …`` so paths stay compatible with +model code that uses :meth:`~geodata.model.results.BaseModelResult.ref_path`. + +Importing this module registers ``wind_3d_hourly_test`` and ``wind_solar_hourly_test`` +in :data:`geodata.datasets.registry`. +""" + +from __future__ import annotations + +import logging +import shutil +from pathlib import Path + +from geodata.config import DATASET_ROOT_PATH + +from .._base import AtomicDataset +from .wind_3d.hourly import ERA5Wind3DHourlyDataset +from .wind_solar.hourly import ERA5WindSolarHourlyDataset + +logger = logging.getLogger(__name__) + +# Paths must match tests/fixtures/era5//... +_FIXTURE_YEAR = 2016 +_FIXTURE_MONTH = 1 +_FIXTURE_DAY = 1 + + +def _resolve_fixture_root(config_dirname: str) -> Path: + """Return ``tests/fixtures/era5/`` by walking parents of this file. + + Works for editable installs where the repo contains ``tests/fixtures``. Wheel-only + installs without that tree raise ``FileNotFoundError``. + """ + here = Path(__file__).resolve() + for root in [here.parent, *here.parents]: + candidate = root / "tests" / "fixtures" / "era5" / config_dirname + if candidate.is_dir(): + return candidate + raise FileNotFoundError( + f"Could not find tests/fixtures/era5/{config_dirname} starting from {here}. " + "Offline fixture datasets need the repository tests/fixtures tree (e.g. editable install)." + ) + + +def _copy_fixture_into_storage(template_root: Path, storage_root: Path, relative: Path) -> None: + src = template_root / relative + if not src.is_file(): + raise FileNotFoundError(f"Expected fixture NetCDF at {src}") + dest = storage_root / relative + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dest) + + +class ERA5Wind3DHourlyTestDataset(ERA5Wind3DHourlyDataset): + """Same schema as :class:`ERA5Wind3DHourlyDataset`, but points at a single local file. + + ``years`` / ``months`` passed to :meth:`__init__` do not expand the catalog; the + catalog is always the fixture for ``{_FIXTURE_YEAR}/{_FIXTURE_MONTH:02d}/{_FIXTURE_DAY:02d}.nc``. + """ + + weather_config = "wind_3d_hourly_test" + + def _extra_setup(self, **kwargs): + template_root = _resolve_fixture_root("wind_3d_hourly_test") + self.storage_root = DATASET_ROOT_PATH / self.module / self.weather_config + rel = ( + Path(str(_FIXTURE_YEAR)) + / f"{_FIXTURE_MONTH:02d}" + / f"{_FIXTURE_DAY:02d}.nc" + ) + _copy_fixture_into_storage(template_root, self.storage_root, rel) + + @property + def catalog(self) -> list[AtomicDataset]: + return [AtomicDataset(self, _FIXTURE_YEAR, _FIXTURE_MONTH, _FIXTURE_DAY)] + + def get_monthly_catalog(self, year: int, month: int) -> list[AtomicDataset]: + """Only the committed fixture day exists under ``ref_path``; do not list full month.""" + if not isinstance(year, int): + raise ValueError("year must be an integer") + if not isinstance(month, int): + raise ValueError("month must be an integer") + if not 1 <= month <= 12: + raise ValueError("month must be between 1 and 12") + if not self.years.start <= year <= self.years.stop: + raise ValueError( + f"year must be between {self.years.start} and {self.years.stop}" + ) + if not self.months.start <= month <= self.months.stop: + raise ValueError( + f"month must be between {self.months.start} and {self.months.stop}" + ) + if year == _FIXTURE_YEAR and month == _FIXTURE_MONTH: + return [AtomicDataset(self, year, month, _FIXTURE_DAY)] + return [] + + def _download_file(self, file: AtomicDataset): + raise RuntimeError( + f"{self.weather_config} uses committed fixtures under tests/fixtures; download is disabled." + ) + + +class ERA5WindSolarHourlyTestDataset(ERA5WindSolarHourlyDataset): + """Same schema as :class:`ERA5WindSolarHourlyDataset`, but points at one monthly fixture file.""" + + weather_config = "wind_solar_hourly_test" + + def _extra_setup(self, **kwargs): + template_root = _resolve_fixture_root("wind_solar_hourly_test") + self.storage_root = DATASET_ROOT_PATH / self.module / self.weather_config + rel = Path(str(_FIXTURE_YEAR)) / f"{_FIXTURE_MONTH:02d}.nc" + _copy_fixture_into_storage(template_root, self.storage_root, rel) + + @property + def catalog(self) -> list[AtomicDataset]: + return [AtomicDataset(self, _FIXTURE_YEAR, _FIXTURE_MONTH)] + + def _download_file(self, file: AtomicDataset): + raise RuntimeError( + f"{self.weather_config} uses committed fixtures under tests/fixtures; download is disabled." + ) + + +__all__ = [ + "ERA5Wind3DHourlyTestDataset", + "ERA5WindSolarHourlyTestDataset", +] diff --git a/src/geodata/model/_base.py b/src/geodata/model/_base.py index 7589c8c8..4c301f69 100644 --- a/src/geodata/model/_base.py +++ b/src/geodata/model/_base.py @@ -19,7 +19,8 @@ import os import platform import shutil -from typing import Optional +from collections.abc import Collection +from typing import ClassVar, Optional import xarray as xr from tqdm.auto import tqdm @@ -158,7 +159,7 @@ class BaseModel(abc.ABC): **kwargs: Additional keyword arguments to pass to the model. """ - SUPPORTED_WEATHER_DATA_CONFIGS: tuple[str] + SUPPORTED_WEATHER_DATA_CONFIGS: ClassVar[Collection[str]] def __init__(self, source: BaseDataset, **kwargs): if source.weather_config not in self.SUPPORTED_WEATHER_DATA_CONFIGS: diff --git a/src/geodata/model/pvlib/_base.py b/src/geodata/model/pvlib/_base.py index 6cbc898f..13bb6808 100644 --- a/src/geodata/model/pvlib/_base.py +++ b/src/geodata/model/pvlib/_base.py @@ -434,8 +434,24 @@ def _process_single_coordinate(args): 'eta': eta } progress_dict['should_log'] = True - - return (y, x), subset + + # Re-pack results into a MultiIndex so that + # xr.Dataset.from_dataframe() reconstructs x and y as dimensions. + # + # `subset` is currently indexed only by `time` (x/y were reset into columns), + # which would otherwise cause the output to have only `time` as a coordinate. + subset_out = subset[['ac', 'pv']].copy() + subset_out = subset_out.assign(y=y, x=x) + subset_out = subset_out.reset_index() + + # After reset_index(), the time column name can vary (e.g. 'time' vs 'index'). + if subset.index.name is None: + subset_out = subset_out.rename(columns={'index': 'time'}) + elif subset.index.name != 'time': + subset_out = subset_out.rename(columns={subset.index.name: 'time'}) + + subset_out = subset_out.set_index(['time', 'x', 'y']) + return (y, x), subset_out except Exception as e: logger.error(f"Error processing coordinate ({y}, {x}): {str(e)}") @@ -444,10 +460,11 @@ def _process_single_coordinate(args): class Pvlib(BaseModel): """The pvlib model""" + @property + def type(self) -> str: + return "pvlib" - type: str = "pvlib" - - SUPPORTED_WEATHER_DATA_CONFIGS = ("wind_solar_hourly",) + SUPPORTED_WEATHER_DATA_CONFIGS = ("wind_solar_hourly", "wind_solar_hourly_test") @property def prepared(self) -> bool: @@ -739,6 +756,11 @@ def estimate(self, if 'time' in combined_result.coords: combined_result = combined_result.sortby('time') + # Standardize output dimension order across models: + # `("time", "x", "y")`. + desired_order = ("time", "x", "y") + if all(d in combined_result.dims for d in desired_order): + combined_result = combined_result.transpose(*desired_order) return combined_result def _prepare_pvlib_ds(self, ds: xr.Dataset, *varnames: str) -> xr.Dataset: @@ -1069,9 +1091,13 @@ def progress_monitor(): f"({elapsed_total/total_coords:.2f}s per coordinate on average)" ) - weather_data_final = pd.concat(coord_subsets) + weather_data_final = pd.concat(coord_subsets).sort_index() - return xr.Dataset.from_dataframe(weather_data_final) + out = xr.Dataset.from_dataframe(weather_data_final) + desired_order = ("time", "x", "y") + if all(d in out.dims for d in desired_order): + out = out.transpose(*desired_order) + return out def _prepare_dataset(self, source: xr.Dataset) -> xr.Dataset: """This will never be called, but must be implemented (abstract method).""" diff --git a/src/geodata/model/wind/interpolate.py b/src/geodata/model/wind/interpolate.py index 76cb2c54..28f96fdd 100644 --- a/src/geodata/model/wind/interpolate.py +++ b/src/geodata/model/wind/interpolate.py @@ -15,6 +15,7 @@ import logging from typing import Hashable +from typing import cast import numpy as np import scipy.interpolate as sinterp @@ -196,8 +197,8 @@ def _splev_ker(c: np.ndarray, t: np.ndarray, k: int, height: np.ndarray) -> np.n return np.atleast_1d(sinterp.splev(height, (t, c, k))) -def _splev(da: xr.DataArray, height: float) -> xr.DataArray: - height = np.atleast_1d(height) +def _splev(da: xr.Dataset, height: float) -> xr.DataArray: + height_arr = np.atleast_1d(height) return xr.apply_ufunc( _splev_ker, da["c"], @@ -206,7 +207,7 @@ def _splev(da: xr.DataArray, height: float) -> xr.DataArray: vectorize=True, dask="parallelized", output_dtypes=[da["c"].dtype], - kwargs={"t": da.attrs["t"], "k": da.attrs["k"], "height": height}, + kwargs={"t": da.attrs["t"], "k": da.attrs["k"], "height": height_arr}, ) @@ -226,11 +227,11 @@ class WindInterpolationModel(WindBaseModel): >>> model.estimate(height=12, xs=slice(1, 2), ys=slice(1, 2), years=slice(2010, 2010), months=slice(1, 2)) """ - SUPPORTED_WEATHER_DATA_CONFIGS = {"wind_3d_hourly"} + SUPPORTED_WEATHER_DATA_CONFIGS = ("wind_3d_hourly", "wind_3d_hourly_test") def _prepare_dataset( self, - ds: xr.Dataset, + source: xr.Dataset, half_precision: bool = True, ) -> xr.Dataset: """Compute wind speed using the ERA5 3D dataset. @@ -244,20 +245,20 @@ def _prepare_dataset( """ assert ( - "model_level" in ds.coords + "model_level" in source.coords ), "Dataset does not contain model levels. Please double-check the dataset." - ds.coords["model_level"] = np.array( - [LEVEL_TO_HEIGHT[int(level)] for level in ds["model_level"].values] + source.coords["model_level"] = np.array( + [LEVEL_TO_HEIGHT[int(level)] for level in source["model_level"].values] ) - ds = ( - ds.rename({"model_level": "height"}) + source = ( + source.rename({"model_level": "height"}) .transpose("height", ...) .sortby("height") ) - logger.debug("Shape of heights: %s", ds["height"].shape) - speeds = (ds["u"] ** 2 + ds["v"] ** 2) ** 0.5 + logger.debug("Shape of heights: %s", source["height"].shape) + speeds = (source["u"] ** 2 + source["v"] ** 2) ** 0.5 logger.debug(f"[_prepare_dataset] Computed speeds, shape: {speeds.shape}, dims: {speeds.dims}, " f"is dask: {isinstance(speeds.data, array_type('dask'))}") @@ -269,7 +270,23 @@ def _prepare_dataset( return params - def _estimate_dataset(self, params: xr.Dataset, height: float) -> xr.Dataset: + def _estimate_dataset(self, params: xr.Dataset, **kwargs) -> xr.DataArray: + height = float(kwargs["height"]) params = params.transpose("height", ...) - params = rechunk_dataset(params, force_full_chunk_dims=["height"]) - return _splev(params, height) + params = cast( + xr.Dataset, + rechunk_dataset(params, force_full_chunk_dims=["height"]), + ) + result = _splev(params, height) + + # Some upstream ERA5 pipelines historically use `valid_time` for the time-like + # coordinate. Normalize to `time` so we can enforce consistent dims. + if "valid_time" in result.dims or "valid_time" in result.coords: + result = result.rename({"valid_time": "time"}) + + # Standardize output dimension order across wind models: + # `("time", "x", "y")` (wind interpolation should match pvlib). + desired_order = ("time", "x", "y") + if all(d in result.dims for d in desired_order): + result = result.transpose(*desired_order) + return result diff --git a/tests/fixtures/era5/wind_3d_hourly_test/2016/01/01.nc b/tests/fixtures/era5/wind_3d_hourly_test/2016/01/01.nc new file mode 100644 index 00000000..1fa4d084 Binary files /dev/null and b/tests/fixtures/era5/wind_3d_hourly_test/2016/01/01.nc differ diff --git a/tests/fixtures/era5/wind_solar_hourly_test/2016/01.nc b/tests/fixtures/era5/wind_solar_hourly_test/2016/01.nc new file mode 100644 index 00000000..901d16ed Binary files /dev/null and b/tests/fixtures/era5/wind_solar_hourly_test/2016/01.nc differ diff --git a/tests/pr/test_dataset_comprehensive.py b/tests/pr/test_dataset_comprehensive.py index 63c7bf3e..9f0ff398 100644 --- a/tests/pr/test_dataset_comprehensive.py +++ b/tests/pr/test_dataset_comprehensive.py @@ -70,6 +70,7 @@ """ import logging +import os from typing import Optional import xarray as xr @@ -78,6 +79,10 @@ logging.basicConfig(level=logging.INFO) +# PRs should run with zero CDS calls. Enable integration download tests only via: +# GEODATA_RUN_CDS_TESTS=1 +RUN_CDS_TESTS = os.getenv("GEODATA_RUN_CDS_TESTS") == "1" + # ============================================================================ # TEST CONFIGURATION HELPERS @@ -85,22 +90,25 @@ def get_data_configs() -> list[str]: """Get list of dataset configurations to test.""" - return ["wind_3d_hourly"] + # Default to offline fixtures for PR safety. + return ["wind_3d_hourly"] if RUN_CDS_TESTS else ["wind_3d_hourly_test"] def get_bounds() -> list[list[float]]: """Get list of bounding boxes to test (lon_min, lat_min, lon_max, lat_max).""" - return [[50, 0, 48, 3]] # Small test region + # Bounds are only meaningful for real downloads. Fixture files are not + # regenerated per-bounds and therefore shouldn't be validated against bounds. + return [[50, 0, 48, 3]] if RUN_CDS_TESTS else [None] # type: ignore[list-item] def get_years() -> list[slice]: """Get list of year ranges to test.""" - return [slice(2005, 2005)] + return [slice(2005, 2005)] if RUN_CDS_TESTS else [slice(2016, 2016)] def get_months() -> list[slice]: """Get list of month ranges to test.""" - return [slice(1, 2)] + return [slice(1, 2)] if RUN_CDS_TESTS else [slice(1, 1)] def get_dataset( @@ -116,7 +124,13 @@ def get_dataset( years=year, months=month, bounds=bound, testing=testing ) if not dataset.downloaded: - dataset.download() + if RUN_CDS_TESTS: + dataset.download() + else: + raise AssertionError( + f"Dataset {data_config} is not downloaded, but CDS tests are disabled. " + "Use fixture configs or set GEODATA_RUN_CDS_TESTS=1." + ) return dataset @@ -133,6 +147,10 @@ def test_download(): they work before running longer tests. This is the foundation for all other data-dependent tests. """ + if not RUN_CDS_TESTS: + # This test verifies CDS download pipeline. Keep it opt-in. + return + configs = get_data_configs() years = get_years() months = get_months() @@ -155,11 +173,15 @@ def test_catalog_generation(): catalog generation means missing data or unnecessary downloads. Testing this ensures we know exactly what will be downloaded before we download it. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) # Test monthly catalog (if applicable) - dataset = dataset_cls(years=slice(2005, 2005), months=slice(1, 1), testing=True) + dataset = dataset_cls( + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), + months=slice(1, 1), + testing=True, + ) catalog = dataset.catalog assert len(catalog) > 0, "Catalog should contain at least one file" @@ -169,7 +191,7 @@ def test_catalog_generation(): assert hasattr(file, "year"), "Catalog entry should have year" assert hasattr(file, "month"), "Catalog entry should have month" assert hasattr(file, "path"), "Catalog entry should have path" - assert file.year == 2005, "Year should match" + assert file.year == (2005 if RUN_CDS_TESTS else 2016), "Year should match" assert file.month == 1, "Month should match" @@ -180,6 +202,10 @@ def test_catalog_testing_mode(): WHY: Testing mode should limit downloads to a few days/months to speed up tests. If this doesn't work correctly, tests become slow and expensive. """ + if not RUN_CDS_TESTS: + # Fixture datasets have fixed catalogs; testing mode isn't meaningful here. + return + config = "wind_3d_hourly" dataset_cls = load_dataset(config) @@ -211,9 +237,13 @@ def test_catalog_paths(): WHY: File paths determine where data is stored. Incorrect paths lead to data being saved in wrong locations or files overwriting each other. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) - dataset = dataset_cls(years=slice(2005, 2005), months=slice(1, 1), testing=True) + dataset = dataset_cls( + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), + months=slice(1, 1), + testing=True, + ) catalog = dataset.catalog paths = {file.path for file in catalog} @@ -294,6 +324,9 @@ def test_bounds_validation(): downloading unnecessary data or missing required data. Also validates that invalid bounds are rejected early. """ + if not RUN_CDS_TESTS: + return + config = "wind_3d_hourly" dataset_cls = load_dataset(config) @@ -358,9 +391,13 @@ def test_dataset_properties(): WHY: Dataset properties (projection, lat_direction, frequency) are used throughout the codebase for processing. Incorrect properties break analysis. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) - dataset = dataset_cls(years=slice(2005, 2005), months=slice(1, 1), testing=True) + dataset = dataset_cls( + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), + months=slice(1, 1), + testing=True, + ) # Test required properties exist assert hasattr(dataset, "projection"), "Dataset should have projection property" @@ -383,15 +420,19 @@ def test_dataset_repr(): WHY: The __repr__ method is used for debugging and logging. It should provide useful information about the dataset state. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) - dataset = dataset_cls(years=slice(2005, 2005), months=slice(1, 1), testing=True) + dataset = dataset_cls( + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), + months=slice(1, 1), + testing=True, + ) repr_str = repr(dataset) # Should contain key information - assert "wind_3d_hourly" in repr_str, "repr should contain weather_config" - assert "2005" in repr_str, "repr should contain years" + assert dataset.weather_config in repr_str, "repr should contain weather_config" + assert ("2005" if RUN_CDS_TESTS else "2016") in repr_str, "repr should contain years" assert "1" in repr_str, "repr should contain months" @@ -442,10 +483,16 @@ def test_data_dimensions(): a 3D wind dataset should have a level/height dimension. Missing dimensions indicate incorrect data structure. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) - dataset = dataset_cls(years=slice(2005, 2005), months=slice(1, 1), bounds=get_bounds()[0], testing=True) - dataset.download() + dataset = dataset_cls( + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), + months=slice(1, 1), + bounds=get_bounds()[0], + testing=True, + ) + if RUN_CDS_TESTS and not dataset.downloaded: + dataset.download() # Check first downloaded file for file in dataset.catalog: @@ -473,15 +520,16 @@ def test_data_value_ranges(): WHY: Data values should be within physically plausible ranges. Out-of-range values indicate data corruption or processing errors. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) dataset = dataset_cls( - years=slice(2005, 2005), + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), months=slice(1, 1), bounds=get_bounds()[0], testing=True ) - dataset.download() + if RUN_CDS_TESTS and not dataset.downloaded: + dataset.download() # Check first downloaded file for file in dataset.catalog: @@ -496,7 +544,8 @@ def test_data_value_ranges(): # Wind components should be within reasonable range # (typical wind speeds are -100 to 100 m/s) - if "u" in var.lower() or "v" in var.lower(): + var_str = str(var) + if "u" in var_str.lower() or "v" in var_str.lower(): if data.notnull().any(): data_min = float(data.min()) data_max = float(data.max()) @@ -522,15 +571,16 @@ def test_postprocessing_applied(): applied consistently. If postprocessing fails silently, downstream code expecting transformed data will fail. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) dataset = dataset_cls( - years=slice(2005, 2005), + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), months=slice(1, 1), bounds=get_bounds()[0], testing=True ) - dataset.download() + if RUN_CDS_TESTS and not dataset.downloaded: + dataset.download() # Check that postprocessed files have correct structure for file in dataset.catalog: @@ -584,6 +634,10 @@ def test_testing_mode(): WHY: Testing mode is crucial for fast CI/CD pipelines. If it doesn't work correctly, tests become too slow or download too much data. """ + if not RUN_CDS_TESTS: + # Fixture datasets ignore testing-mode catalog limiting; keep this check opt-in. + return + config = "wind_3d_hourly" dataset_cls = load_dataset(config) @@ -611,15 +665,19 @@ def test_storage_path(): WHY: Files must be saved to the correct location for proper organization and retrieval. Wrong paths make it impossible to find downloaded data. """ - config = "wind_3d_hourly" + config = "wind_3d_hourly" if RUN_CDS_TESTS else "wind_3d_hourly_test" dataset_cls = load_dataset(config) - dataset = dataset_cls(years=slice(2005, 2005), months=slice(1, 1), testing=True) + dataset = dataset_cls( + years=slice(2005, 2005) if RUN_CDS_TESTS else slice(2016, 2016), + months=slice(1, 1), + testing=True, + ) # Storage root should follow expected pattern assert dataset.storage_root is not None, "Storage root should be set" assert "era5" in str(dataset.storage_root), \ "Storage root should contain module name" - assert "wind_3d_hourly" in str(dataset.storage_root), \ + assert dataset.weather_config in str(dataset.storage_root), \ "Storage root should contain weather_config" @@ -630,6 +688,9 @@ def test_bounds_applied(): WHY: When bounds are specified, data should be filtered to those bounds. Downloading global data when only a region is needed wastes resources. """ + if not RUN_CDS_TESTS: + return + config = "wind_3d_hourly" dataset_cls = load_dataset(config) diff --git a/tests/pr/test_era5_lengthy.py b/tests/pr/test_era5_lengthy.py index 9979ace3..4b06b522 100644 --- a/tests/pr/test_era5_lengthy.py +++ b/tests/pr/test_era5_lengthy.py @@ -16,11 +16,14 @@ """Tests in this file are lengthy due to the nature of the dataset being tested.""" import logging +import os -from geodata.datasets import DatasetType, load_dataset +from geodata.datasets import load_dataset logging.basicConfig(level=logging.INFO) +RUN_CDS_TESTS = os.getenv("GEODATA_RUN_CDS_TESTS") == "1" + # TODO: Test other functionalities with the 3D dataset def get_data_configs() -> list[str]: @@ -41,7 +44,7 @@ def get_months() -> list[slice]: def get_era5(data_config: str, bound: list[int], year: slice, month: slice): dataset_cls = load_dataset(data_config) - dataset: DatasetType = dataset_cls( + dataset = dataset_cls( years=year, months=month, bounds=bound, testing=True ) if not dataset.downloaded: @@ -50,6 +53,11 @@ def get_era5(data_config: str, bound: list[int], year: slice, month: slice): def test_download(): + if not RUN_CDS_TESTS: + # PRs should not require CDS keys / network. Run this test only in + # an opt-in integration job: GEODATA_RUN_CDS_TESTS=1. + return + configs = get_data_configs() years = get_years() months = get_months() diff --git a/tests/pr/test_era5_wind3d.py b/tests/pr/test_era5_wind3d.py index 05adef1e..f27ff9f3 100644 --- a/tests/pr/test_era5_wind3d.py +++ b/tests/pr/test_era5_wind3d.py @@ -14,76 +14,71 @@ # along with this program. If not, see . import logging -from dask.distributed import Client import xarray as xr +from dask.distributed import Client from geodata.datasets import load_dataset +from geodata.datasets._base import BaseDataset from geodata.logging import logger from geodata.model.wind import WindInterpolationModel -# Set logger to DEBUG level to see all debug messages logger.setLevel(logging.DEBUG) +def _fixture_xy_slices(dataset: BaseDataset) -> tuple[slice, slice]: + """Build ``xs``, ``ys`` slices on the fixture grid (``x``/``y`` or ERA5 ``longitude``/``latitude``).""" + path = dataset.catalog[0].path + with xr.open_dataset(path, engine="h5netcdf") as opened: + if "x" in opened.coords: + xv = opened["x"].values + yv = opened["y"].values + else: + xv = opened["longitude"].values + yv = opened["latitude"].values + xs = slice(float(xv[0]), float(xv[-1])) + ys = slice(float(yv[0]), float(yv[-1])) + return xs, ys + + def test_wind_interpolation_workflow(): - """Test that the wind interpolation workflow completes without errors. - - This test verifies: - - Dataset can be loaded and downloaded + """Wind interpolation workflow using offline ``wind_3d_hourly_test`` fixtures (no CDS). + + Verifies: + - Fixture dataset is registered and on disk - Model can be created and prepared - - Capacity factor estimation works (globally and with bounds) - - Wind speed estimation works at a specific height - - Results can be computed and have valid values + - Capacity factor and wind-speed estimates run on the fixture extent """ - client = Client(processes=True, threads_per_worker=1) - years = slice(2016, 2016) months = slice(1, 1) - ds_cls = load_dataset("wind_3d_hourly") - ds = ds_cls(years=years, months=months, testing=True) - - ds.download() - assert ds.downloaded, "Dataset should be downloaded successfully" - - # Create model with the dataset - model = WindInterpolationModel(ds) - assert model is not None, "Model should be created successfully" - - # Force re-preparation to see debug logs (comment out if you want to skip preparation) - model.prepare(force=True) - - turbine_name = "Enercon_E126_7500kW" - china_bbox = (73.5, 18.2, 135.1, 53.6) # China bounding box - xs = slice(china_bbox[0], china_bbox[2]) - ys = slice(china_bbox[3], china_bbox[1]) - - # Test capacity factor estimation globally - cf_global = model.estimate(turbine=turbine_name) - assert cf_global is not None, "Capacity factor estimation should return a result" - assert isinstance(cf_global, (xr.DataArray, xr.Dataset)), \ - "Capacity factor should be an xarray DataArray or Dataset" - - # Test capacity factor estimation for China only - cf_china = model.estimate(turbine=turbine_name, xs=xs, ys=ys) - assert cf_china is not None, "Capacity factor estimation with bounds should return a result" - assert isinstance(cf_china, (xr.DataArray, xr.Dataset)), \ - "Capacity factor with bounds should be an xarray DataArray or Dataset" - - # Test wind speed estimation at specific height - speed = model.estimate(height=100.0, xs=xs, ys=ys) - assert speed is not None, "Wind speed estimation should return a result" - assert isinstance(speed, xr.DataArray), \ - "Wind speed should be an xarray DataArray" - - # Test that results can be computed - cf_computed = cf_china.compute() - assert cf_computed is not None, "Computed capacity factor should not be None" - - # Test that max value can be calculated (verifies data is valid and operations work) - max_cf = cf_computed.max() - assert max_cf is not None, "Max capacity factor should be calculable" - - client.close() \ No newline at end of file + with Client(processes=True, threads_per_worker=1): + ds_cls = load_dataset("wind_3d_hourly_test") + ds = ds_cls(years=years, months=months) + assert ds.downloaded, "Fixture NetCDF should be present" + + xs, ys = _fixture_xy_slices(ds) + + model = WindInterpolationModel(ds) + assert model is not None + model.prepare(force=True) + + turbine_name = "Enercon_E126_7500kW" + + cf_global = model.estimate(turbine=turbine_name) + assert cf_global is not None + assert isinstance(cf_global, (xr.DataArray, xr.Dataset)) + + cf_region = model.estimate(turbine=turbine_name, xs=xs, ys=ys) + assert cf_region is not None + assert isinstance(cf_region, (xr.DataArray, xr.Dataset)) + + speed = model.estimate(height=100.0, xs=xs, ys=ys) + assert speed is not None + assert isinstance(speed, xr.DataArray) + + cf_computed = cf_region.compute() + assert cf_computed is not None + max_cf = cf_computed.max() + assert max_cf is not None diff --git a/tests/pr/test_era5_windsolar.py b/tests/pr/test_era5_windsolar.py index d8d6edcf..6c3c00b8 100644 --- a/tests/pr/test_era5_windsolar.py +++ b/tests/pr/test_era5_windsolar.py @@ -14,100 +14,105 @@ # along with this program. If not, see . import logging -from dask.distributed import Client import xarray as xr +from dask.distributed import Client from geodata.datasets import load_dataset +from geodata.datasets._base import BaseDataset from geodata.logging import logger from geodata.model.pvlib import Pvlib +from geodata.model.wind import WindInterpolationModel -# Set logger to DEBUG level to see all debug messages logger.setLevel(logging.DEBUG) -def test_wind_solar_workflow(): - """Test that the wind interpolation workflow completes without errors. - - This test verifies: - - Dataset can be loaded and downloaded - - Model can be created and prepared - - Capacity factor estimation works (globally and with bounds) - - Wind speed estimation works at a specific height - - Results can be computed and have valid values - """ - - client = Client(processes=True, threads_per_worker=1) - - years = slice(2016, 2016) - months = slice(1, 1) - ds_cls = load_dataset("wind_solar_hourly") - ds = ds_cls(years=years, months=months, testing=True) +def _fixture_xy_slices(dataset: BaseDataset) -> tuple[slice, slice]: + """Build ``xs``, ``ys`` slices on the fixture grid (``x``/``y`` or ERA5 ``longitude``/``latitude``).""" + path = dataset.catalog[0].path + with xr.open_dataset(path, engine="h5netcdf") as opened: + if "x" in opened.coords: + xv = opened["x"].values + yv = opened["y"].values + else: + xv = opened["longitude"].values + yv = opened["latitude"].values + xs = slice(float(xv[0]), float(xv[-1])) + ys = slice(float(yv[0]), float(yv[-1])) + return xs, ys - ds.download() - assert ds.downloaded, "Dataset should be downloaded successfully" - # Create model with the dataset - model = Pvlib(ds) - assert model is not None, "Model should be created successfully" - - # TODO: use a smaller region for testing - # china_bbox = (73.5, 18.2, 135.1, 53.6) # China bounding box - # xs = slice(china_bbox[0], china_bbox[2]) - # ys = slice(china_bbox[3], china_bbox[1]) - - # Central Europe (Germany/Switzerland border - definitely on land) - xs = slice(8, 10) # 2 degrees longitude (8°E to 10°E) - ys = slice(48, 46) # 2 degrees latitude (48°N to 46°N, north to south) +def test_wind_solar_workflow(): + """Pvlib + wind interpolation using offline ``*_test`` fixtures (no CDS).""" years = slice(2016, 2016) months = slice(1, 1) - # TODO: add a test here to test that - # the model must not estimate without pv_system and model_config - - # create the pv_system - n_mods = 50 - n_strings = 1 - cec_modules = model.retrieve_sam('CECMod') - module = cec_modules['Kaneka_U_SA105'] - inv = model.retrieve_sam("CECInverter")['Fronius_USA__CL_33_3_Delta__208V_'] - model.init_pv_system( - arrays = None, - surface_tilt=35, - surface_azimuth=180, - racking_model = 'open_rack', - module_parameters=module, - modules_per_string = n_mods, - module_type = 'glass_polymer', - module = 'Kaneka_U_SA105', - strings_per_inverter = n_strings, - inverter_parameters=inv - ) - - assert model.pv_system is not None, "pv_system should be seccesfully created" - # TODO: assert it to the correct type - - # create the model_config - model.init_model_config( - clearsky_model= 'haurwitz', - transposition_model='perez', - solar_position_method= 'nrel_numpy', - airmass_model= 'kastenyoung1989', - dc_model='cec', - ac_model='sandia', - aoi_model="physical", - spectral_model='first_solar', - dc_ohmic_model='no_loss' - ) - assert model.config is not None, "model config should be successfully created" - - # Test capacity factor estimation globally - ac_power_and_pv_capacity_global = model.estimate(years=years, months=months, xs=xs, ys=ys) - assert ac_power_and_pv_capacity_global is not None, "Capacity factor estimation should return a result" - assert isinstance(ac_power_and_pv_capacity_global, (xr.DataArray, xr.Dataset)), \ - "Capacity factor should be an xarray DataArray or Dataset" - - # TODO: design correct output test specific regard to the pvlib output - - client.close() \ No newline at end of file + with Client(processes=True, threads_per_worker=1): + ds_cls = load_dataset("wind_solar_hourly_test") + ds = ds_cls(years=years, months=months) + assert ds.downloaded, "Fixture NetCDF should be present" + + xs, ys = _fixture_xy_slices(ds) + + model = Pvlib(ds) + assert model is not None + + n_mods = 50 + n_strings = 1 + cec_modules = model.retrieve_sam("CECMod") + module = cec_modules["Kaneka_U_SA105"] + inv = model.retrieve_sam("CECInverter")["Fronius_USA__CL_33_3_Delta__208V_"] + model.init_pv_system( + arrays=None, + surface_tilt=35, + surface_azimuth=180, + racking_model="open_rack", + module_parameters=module, + modules_per_string=n_mods, + module_type="glass_polymer", + module="Kaneka_U_SA105", + strings_per_inverter=n_strings, + inverter_parameters=inv, + ) + assert model.pv_system is not None + + model.init_model_config( + clearsky_model="haurwitz", + transposition_model="perez", + solar_position_method="nrel_numpy", + airmass_model="kastenyoung1989", + dc_model="cec", + ac_model="sandia", + aoi_model="physical", + spectral_model="first_solar", + dc_ohmic_model="no_loss", + ) + assert model.config is not None + + ac_power_and_pv_capacity_global = model.estimate( + years=years, months=months, xs=xs, ys=ys + ) + assert ac_power_and_pv_capacity_global is not None + assert isinstance( + ac_power_and_pv_capacity_global, (xr.DataArray, xr.Dataset) + ) + assert list(ac_power_and_pv_capacity_global.dims) == ["time", "x", "y"] + + wind_ds_cls = load_dataset("wind_3d_hourly_test") + wind_ds = wind_ds_cls(years=years, months=months) + assert wind_ds.downloaded, "Wind fixture NetCDF should be present" + + wxs, wys = _fixture_xy_slices(wind_ds) + + wind_model = WindInterpolationModel(wind_ds) + wind_model.prepare() + + wind_speed = wind_model.estimate( + years=years, months=months, xs=wxs, ys=wys, height=12 + ) + assert list(wind_speed.dims) == ["time", "x", "y"] + assert ( + "valid_time" not in wind_speed.dims + and "valid_time" not in wind_speed.coords + )