Integrate pvlib new

docs/source/development/offline-era5-fixture-datasets.md

@@ -0,0 +1,82 @@
+# Offline ERA5 fixture datasets (`*_test` weather configs)
+
+This document records the **design and implementation plan** for small, committed NetCDF fixtures used in automated tests—without calling the CDS API or relying on `DATASET_ROOT_PATH` downloads.
+
+## Goals
+
+- Ship **minimal** ERA5-shaped files in the repository for CI and local testing.
+- Expose them via **`load_dataset("…_test")`** so code paths mirror production (`wind_3d_hourly`, `wind_solar_hourly`) while staying **offline**.
+- Avoid coupling tests to arbitrary year/month ranges: fixture datasets should use a **fixed catalog** (typically a single file) even if `BaseDataset.__init__` still requires `years` / `months` arguments (those values can be **ignored** for catalog construction in test configs).
+
+## Non-goals
+
+- The legacy **`geodata.dataset.Dataset`** (`module=` + `weather_data_config=` dict) is **not** in scope; the plan targets **`load_dataset` + `BaseDataset` subclasses** used by models and current tests.
+
+## Current fixture layout (repository)
+
+Fixtures live under **`tests/fixtures/`** so they stay close to pytest and do not inflate the installable package unless explicitly packaged later.
+
+| Test weather config (planned) | Mirrors production config | On-disk layout under `tests/fixtures/era5/` |
+|-------------------------------|---------------------------|---------------------------------------------|
+| `wind_3d_hourly_test` | `wind_3d_hourly` (`frequency="daily"`) | `wind_3d_hourly_test/2016/01/01.nc` |
+| `wind_solar_hourly_test` | `wind_solar_hourly` (default `frequency="monthly"`) | `wind_solar_hourly_test/2016/01.nc` |
+
+Production datasets store files under:
+
+`DATASET_ROOT_PATH / <module> / <weather_config> / …`
+
+with:
+
+- **Daily** (3D wind): `…/<year>/<month>/<day>.nc`
+- **Monthly** (wind/solar hourly): `…/<year>/<month>.nc`
+
+The fixture tree **matches those relative paths** so `AtomicDataset.path` resolution stays aligned with the real datasets.
+
+## Registry and naming
+
+- Each test variant is a **`BaseDataset` subclass** with `weather_config = "wind_3d_hourly_test"` or `"wind_solar_hourly_test"`.
+- Subclasses are registered automatically via `BaseDataset.__init_subclass__` into `geodata.datasets.registry`.
+- Callers use **`load_dataset("wind_3d_hourly_test")`** (same pattern as production).
+
+## Behavioral contract
+
+### Storage root
+
+Fixture classes should set **`storage_root`** to the directory that contains the fixture tree for that config—for example, the absolute path to `tests/fixtures/era5/wind_3d_hourly_test` resolved at runtime (repo-relative or via `importlib.resources` if fixtures are ever packaged).
+
+### Catalog
+
+Override **`catalog`** so it returns **only** the `AtomicDataset` entries that refer to committed files (commonly **one** file):
+
+- 3D wind: one daily file, e.g. `(year=2016, month=1, day=1)` → `…/2016/01/01.nc`
+- Wind/solar: one monthly file, e.g. `(year=2016, month=1)` → `…/2016/01.nc`
+
+Constructor arguments **`years` / `months`** may remain required by `BaseDataset.__init__` but **need not drive** the fixture catalog.
+
+### Download
+
+- **`download()`** must **not** call CDS: implement as a no-op or raise a clear error if invoked.
+- **`_download_file`** should not perform network I/O.
+
+### Prepared state
+
+`downloaded` should become **`True`** when fixture files exist (the default `_check_downloaded()` loop over `catalog` is sufficient if paths resolve correctly).
+
+## Models and `SUPPORTED_WEATHER_DATA_CONFIGS`
+
+`BaseModel` validates both **`weather_config`** and **`source.downloaded`**. Any model that should run on fixtures must **allow** the `*_test` config names—e.g. extend `SUPPORTED_WEATHER_DATA_CONFIGS` on `WindInterpolationModel`, pvlib-related models, and any other entry points used in tests—to include `wind_3d_hourly_test` / `wind_solar_hourly_test` (or document a single shared alias strategy).
+
+## Implementation checklist
+
+1. Add **`ERA5Wind3DHourlyTestDataset`** / **`ERA5WindSolarHourlyTestDataset`** (names may vary) beside the existing ERA5 hourly classes, or in a small `fixture.py` module imported from `era5` packages so subclasses register on import.
+2. Wire **`storage_root`** to `tests/fixtures/era5/<config>/` (resolve path robustly from the repo root or test layout).
+3. Override **`catalog`** to the fixed fixture file(s); ignore user `years`/`months` for catalog purposes (documented).
+4. Override **`download`** / **`_download_file`** to prevent CDS usage.
+5. Update **`SUPPORTED_WEATHER_DATA_CONFIGS`** on affected models.
+6. Add or adjust tests: `load_dataset("…_test")`, assert `downloaded`, **no** `download()`, then run the intended model or pipeline assertion.
+
+## References (code)
+
+- Registry: `geodata.datasets._base.BaseDataset.__init_subclass__`
+- Paths: `AtomicDataset.path` in `geodata.datasets._base`
+- Legacy downloader: `geodata.dataset.Dataset` (separate from this plan)

docs/source/index.rst

@@ -65,6 +65,13 @@ Welcome to Geodata's documentation!
 
 ..    application/*
 
+.. toctree::
+   :maxdepth: 1
+   :caption: Development
+   :hidden:
+
+   development/offline-era5-fixture-datasets
+
 .. toctree::
    :maxdepth: 1
    :caption: API Reference

src/geodata/datasets/era5/__init__.py

@@ -13,6 +13,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from . import wind_3d, wind_solar
+from . import fixture, wind_3d, wind_solar
 
-__all__ = ["wind_3d", "wind_solar"]
+__all__ = ["fixture", "wind_3d", "wind_solar"]

src/geodata/datasets/era5/fixture.py

@@ -0,0 +1,145 @@
+# Copyright 2025 Michael Davidson (UCSD), Xiqiang Liu (UCSD), Keyu Long (UCSD)
+
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 3 of the
+# License, or (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+"""Offline ERA5 datasets backed by committed NetCDF files under ``tests/fixtures/``.
+
+On construction, small template files are **copied** into
+``DATASET_ROOT_PATH / era5 / <weather_config> / …`` so paths stay compatible with
+model code that uses :meth:`~geodata.model.results.BaseModelResult.ref_path`.
+
+Importing this module registers ``wind_3d_hourly_test`` and ``wind_solar_hourly_test``
+in :data:`geodata.datasets.registry`.
+"""
+
+from __future__ import annotations
+
+import logging
+import shutil
+from pathlib import Path
+
+from geodata.config import DATASET_ROOT_PATH
+
+from .._base import AtomicDataset
+from .wind_3d.hourly import ERA5Wind3DHourlyDataset
+from .wind_solar.hourly import ERA5WindSolarHourlyDataset
+
+logger = logging.getLogger(__name__)
+
+# Paths must match tests/fixtures/era5/<config>/...
+_FIXTURE_YEAR = 2016
+_FIXTURE_MONTH = 1
+_FIXTURE_DAY = 1
+
+
+def _resolve_fixture_root(config_dirname: str) -> Path:
+    """Return ``tests/fixtures/era5/<config_dirname>`` by walking parents of this file.
+
+    Works for editable installs where the repo contains ``tests/fixtures``. Wheel-only
+    installs without that tree raise ``FileNotFoundError``.
+    """
+    here = Path(__file__).resolve()
+    for root in [here.parent, *here.parents]:
+        candidate = root / "tests" / "fixtures" / "era5" / config_dirname
+        if candidate.is_dir():
+            return candidate
+    raise FileNotFoundError(
+        f"Could not find tests/fixtures/era5/{config_dirname} starting from {here}. "
+        "Offline fixture datasets need the repository tests/fixtures tree (e.g. editable install)."
+    )
+
+
+def _copy_fixture_into_storage(template_root: Path, storage_root: Path, relative: Path) -> None:
+    src = template_root / relative
+    if not src.is_file():
+        raise FileNotFoundError(f"Expected fixture NetCDF at {src}")
+    dest = storage_root / relative
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    shutil.copy2(src, dest)
+
+
+class ERA5Wind3DHourlyTestDataset(ERA5Wind3DHourlyDataset):
+    """Same schema as :class:`ERA5Wind3DHourlyDataset`, but points at a single local file.
+
+    ``years`` / ``months`` passed to :meth:`__init__` do not expand the catalog; the
+    catalog is always the fixture for ``{_FIXTURE_YEAR}/{_FIXTURE_MONTH:02d}/{_FIXTURE_DAY:02d}.nc``.
+    """
+
+    weather_config = "wind_3d_hourly_test"
+
+    def _extra_setup(self, **kwargs):
+        template_root = _resolve_fixture_root("wind_3d_hourly_test")
+        self.storage_root = DATASET_ROOT_PATH / self.module / self.weather_config
+        rel = (
+            Path(str(_FIXTURE_YEAR))
+            / f"{_FIXTURE_MONTH:02d}"
+            / f"{_FIXTURE_DAY:02d}.nc"
+        )
+        _copy_fixture_into_storage(template_root, self.storage_root, rel)
+
+    @property
+    def catalog(self) -> list[AtomicDataset]:
+        return [AtomicDataset(self, _FIXTURE_YEAR, _FIXTURE_MONTH, _FIXTURE_DAY)]
+
+    def get_monthly_catalog(self, year: int, month: int) -> list[AtomicDataset]:
+        """Only the committed fixture day exists under ``ref_path``; do not list full month."""
+        if not isinstance(year, int):
+            raise ValueError("year must be an integer")
+        if not isinstance(month, int):
+            raise ValueError("month must be an integer")
+        if not 1 <= month <= 12:
+            raise ValueError("month must be between 1 and 12")
+        if not self.years.start <= year <= self.years.stop:
+            raise ValueError(
+                f"year must be between {self.years.start} and {self.years.stop}"
+            )
+        if not self.months.start <= month <= self.months.stop:
+            raise ValueError(
+                f"month must be between {self.months.start} and {self.months.stop}"
+            )
+        if year == _FIXTURE_YEAR and month == _FIXTURE_MONTH:
+            return [AtomicDataset(self, year, month, _FIXTURE_DAY)]
+        return []
+
+    def _download_file(self, file: AtomicDataset):
+        raise RuntimeError(
+            f"{self.weather_config} uses committed fixtures under tests/fixtures; download is disabled."
+        )
+
+
+class ERA5WindSolarHourlyTestDataset(ERA5WindSolarHourlyDataset):
+    """Same schema as :class:`ERA5WindSolarHourlyDataset`, but points at one monthly fixture file."""
+
+    weather_config = "wind_solar_hourly_test"
+
+    def _extra_setup(self, **kwargs):
+        template_root = _resolve_fixture_root("wind_solar_hourly_test")
+        self.storage_root = DATASET_ROOT_PATH / self.module / self.weather_config
+        rel = Path(str(_FIXTURE_YEAR)) / f"{_FIXTURE_MONTH:02d}.nc"
+        _copy_fixture_into_storage(template_root, self.storage_root, rel)
+
+    @property
+    def catalog(self) -> list[AtomicDataset]:
+        return [AtomicDataset(self, _FIXTURE_YEAR, _FIXTURE_MONTH)]
+
+    def _download_file(self, file: AtomicDataset):
+        raise RuntimeError(
+            f"{self.weather_config} uses committed fixtures under tests/fixtures; download is disabled."
+        )
+
+
+__all__ = [
+    "ERA5Wind3DHourlyTestDataset",
+    "ERA5WindSolarHourlyTestDataset",
+]

src/geodata/model/_base.py

@@ -19,7 +19,8 @@
 import os
 import platform
 import shutil
-from typing import Optional
+from collections.abc import Collection
+from typing import ClassVar, Optional
 
 import xarray as xr
 from tqdm.auto import tqdm
@@ -158,7 +159,7 @@ class BaseModel(abc.ABC):
         **kwargs: Additional keyword arguments to pass to the model.
     """
 
-    SUPPORTED_WEATHER_DATA_CONFIGS: tuple[str]
+    SUPPORTED_WEATHER_DATA_CONFIGS: ClassVar[Collection[str]]
 
     def __init__(self, source: BaseDataset, **kwargs):
         if source.weather_config not in self.SUPPORTED_WEATHER_DATA_CONFIGS:

src/geodata/model/pvlib/_base.py

@@ -434,8 +434,24 @@ def _process_single_coordinate(args):
                         'eta': eta
                     }
                     progress_dict['should_log'] = True
-
-        return (y, x), subset
+
+        # Re-pack results into a MultiIndex so that
+        # xr.Dataset.from_dataframe() reconstructs x and y as dimensions.
+        #
+        # `subset` is currently indexed only by `time` (x/y were reset into columns),
+        # which would otherwise cause the output to have only `time` as a coordinate.
+        subset_out = subset[['ac', 'pv']].copy()
+        subset_out = subset_out.assign(y=y, x=x)
+        subset_out = subset_out.reset_index()
+
+        # After reset_index(), the time column name can vary (e.g. 'time' vs 'index').
+        if subset.index.name is None:
+            subset_out = subset_out.rename(columns={'index': 'time'})
+        elif subset.index.name != 'time':
+            subset_out = subset_out.rename(columns={subset.index.name: 'time'})
+
+        subset_out = subset_out.set_index(['time', 'x', 'y'])
+        return (y, x), subset_out
 
     except Exception as e:
         logger.error(f"Error processing coordinate ({y}, {x}): {str(e)}")
@@ -444,10 +460,11 @@ def _process_single_coordinate(args):
 
 class Pvlib(BaseModel):
     """The pvlib model"""
+    @property
+    def type(self) -> str:
+        return "pvlib"
 
-    type: str = "pvlib"
-
-    SUPPORTED_WEATHER_DATA_CONFIGS = ("wind_solar_hourly",)
+    SUPPORTED_WEATHER_DATA_CONFIGS = ("wind_solar_hourly", "wind_solar_hourly_test")
 
     @property
     def prepared(self) -> bool:
@@ -739,6 +756,11 @@ def estimate(self,
             if 'time' in combined_result.coords:
                 combined_result = combined_result.sortby('time')
 
+            # Standardize output dimension order across models:
+            # `("time", "x", "y")`.
+            desired_order = ("time", "x", "y")
+            if all(d in combined_result.dims for d in desired_order):
+                combined_result = combined_result.transpose(*desired_order)
             return combined_result
 
     def _prepare_pvlib_ds(self, ds: xr.Dataset, *varnames: str) -> xr.Dataset:
@@ -1069,9 +1091,13 @@ def progress_monitor():
             f"({elapsed_total/total_coords:.2f}s per coordinate on average)"
         )
 
-        weather_data_final = pd.concat(coord_subsets)
+        weather_data_final = pd.concat(coord_subsets).sort_index()
 
-        return xr.Dataset.from_dataframe(weather_data_final)
+        out = xr.Dataset.from_dataframe(weather_data_final)
+        desired_order = ("time", "x", "y")
+        if all(d in out.dims for d in desired_order):
+            out = out.transpose(*desired_order)
+        return out
 
     def _prepare_dataset(self, source: xr.Dataset) -> xr.Dataset:
         """This will never be called, but must be implemented (abstract method)."""