feat(config): build a Catchment from a YAML run configuration

.github/workflows/lint.yml

@@ -62,16 +62,16 @@ jobs:
         # fix to apply.
         run: pre-commit run --all-files --show-diff-on-failure --color=always
         env:
-          SKIP: no-commit-to-branch,mypy,pytest-check,notebook-check,pixi-lock-check
+          SKIP: no-commit-to-branch,mypy,pytest-check,notebook-check,pixi-lock-check,doctest-executable-modules
 
   # Type check. mypy needs the full `dev` pixi env (typed deps and an importable
   # package) but is neither platform- nor Python-version-sensitive, so one runner
   # covers it. It runs nowhere else in CI today.
   #
-  # No doctest step yet: the `>>>` examples in the HBV modules are stale and the
-  # matching pre-commit hook is disabled for the same reason. Add
-  # `pixi run -e dev pytest --doctest-modules src -p no:cacheprovider` here once they
-  # are repaired, and re-enable the hook alongside it.
+  # The doctest step below covers only the modules whose examples run; the `>>>` examples
+  # in the HBV modules are stale, which is why the whole-`src` pre-commit hook is still
+  # disabled. Widen the `doctests` task in pyproject.toml as those are repaired, and
+  # re-enable that hook once it covers everything.
   static:
     runs-on: ubuntu-latest
     timeout-minutes: 20
@@ -89,3 +89,12 @@ jobs:
 
       - name: Type check with mypy
         run: pixi run -e dev mypy
+
+      # These examples assert on concrete values read from real data, so running them is
+      # what keeps them from drifting into prose. Needs the repo root as the working
+      # directory, which is where the paths in them are written from.
+      - name: Run the executable docstring examples
+        run: pixi run -e dev doctests
+        env:
+          HAPI_DATA_DIR: src/hapi/parameters
+          MPLBACKEND: Agg

.gitignore

@@ -200,3 +200,9 @@ tests/mo/*
 /tests/rrm/data/test_results/prepare_inputs/7_precip_2009.01.08.tif
 /tests/rrm/data/test_results/prepare_inputs/8_precip_2009.01.09.tif
 /tests/rrm/data/test_results/prepare_inputs/9_precip_2009.01.10.tif
+
+# Output of the four YAML-driven Coello examples. They write into the same data tree they
+# read from, so their results sit beside tracked inputs; the inputs stay tracked and the
+# per-run artefacts do not.
+/examples/hydrological-model/data/distributed_model/results/Result_*.tif
+/examples/hydrological-model/data/lumped_model/*Results-Lumped-Model_*.txt

.pre-commit-config.yaml

@@ -140,6 +140,18 @@ repos:
         pass_filenames: false
         always_run: true
 
+  # The narrow half of the hook below: the modules whose examples do run. They assert on
+  # concrete values read from real data, so leaving them unexecuted means they drift silently.
+  # Add a module to the `doctests` task in pyproject.toml once its examples pass.
+  - repo: local
+    hooks:
+      - id: doctest-executable-modules
+        name: "[py   - doctest] executable examples"
+        entry: pixi run --frozen -e dev doctests
+        language: system
+        pass_filenames: false
+        always_run: true
+
 #   - repo: local
 #     hooks:
 #       # Disabled until the docstring examples are repaired: 19 doctests in the

docs/api/catchment.md

@@ -1,4 +1,26 @@
 # Catchment
 
+## Routing methods
+
+`Catchment` and `Calibration` accept exactly three routing methods, matched case-insensitively
+and stored in the one spelling the internals compare against:
+
+| Written as | Stored as | Routes |
+|---|---|---|
+| `muskingum` | `Muskingum` | Cell to cell along the flow-direction network. |
+| `maxbas` | `MAXBAS` | Every cell straight to the outlet through a triangular function. |
+| `kinematic` | `Kinematic` | The flood model's own path (`Run.RunFloodModel`). |
+
+Anything else raises a `ValueError` naming the three. Up to and including version 1.7.0 the
+constructor stored whatever string it was handed, so a run configured as `"Max_bas"` — or as a
+descriptive label such
+as `"Muskingum-Cunge"` — was accepted and then silently routed with Muskingum, because
+`distrrm.SpatialRouting` compares against `"Muskingum"` exactly. Rejecting the spelling is what
+makes that comparison trustworthy; a script passing a spelling outside the table has to be updated
+to one of the three.
+
+A YAML run configuration reaches only the first two: `kinematic` selects the flood model, which
+[`hapi.config`](config.md) does not describe.
+
 ## Catchment
 ::: hapi.catchment.Catchment

docs/api/config.md

@@ -0,0 +1,32 @@
+# Config
+
+The schema of a YAML run configuration. Each block below is one top-level key of the file; the
+rules tying them together — which blocks a given `spatial_resolution` requires, and which it
+refuses — live on `RunConfig`.
+
+Build a model from a file with
+[`Catchment.from_yaml`](catchment.md#hapi.catchment.Catchment.from_yaml).
+
+## RunConfig
+::: hapi.config.RunConfig
+
+## CatchmentConfig
+::: hapi.config.CatchmentConfig
+
+## MeteoConfig
+::: hapi.config.MeteoConfig
+
+## FlowNetworkConfig
+::: hapi.config.FlowNetworkConfig
+
+## ParametersConfig
+::: hapi.config.ParametersConfig
+
+## ConceptualModelConfig
+::: hapi.config.ConceptualModelConfig
+
+## GaugesConfig
+::: hapi.config.GaugesConfig
+
+## OutputsConfig
+::: hapi.config.OutputsConfig

docs/examples/distributed-model-run.md

@@ -1,4 +1,11 @@
 # Distributed Hydrological Model
+
+!!! tip "Or drive it from a YAML file"
+
+    Everything this page assembles in Python can live in a run configuration instead --
+    one file holding the paths, dates and settings, read by `Catchment.from_yaml`. See
+    [Run configuration](run-configuration.md).
+
 After preparing all the meteorological, GIS inputs required for the model, and Extracting the parameters for the catchment
 
 ```python

docs/examples/lumped-model-run.md

@@ -1,4 +1,11 @@
 # Lumped Model Run
+
+!!! tip "Or drive it from a YAML file"
+
+    Everything this page assembles in Python can live in a run configuration instead --
+    one file holding the paths, dates and settings, read by `Catchment.from_yaml`. See
+    [Run configuration](run-configuration.md).
+
 To run the HBV lumped model inside Hapi you need to prepare the meteorological inputs (rainfall, temperature and potential evapotranspiration), HBV parameters, and the HBV model (you can load Bergström, 1992 version of HBV from Hapi )
 
 - First load the prepared lumped version of the HBV module inside Hapi, the triangular routing function and the wrapper function that runs the lumped model `RUN`.

docs/examples/run-configuration.md

@@ -0,0 +1,167 @@
+# Run Configuration (YAML)
+
+The other example pages assemble a model in Python: construct a `Catchment`, then call
+`read_lumped_inputs`, `read_parameters`, `read_lumped_model` and the rest in the right order.
+That works, but it puts every path, date and area inside the script, so a script is only ever
+about one catchment — and the call order is something you have to know.
+
+A run configuration moves all of it into a YAML file that sits beside the data it names.
+`Catchment.from_yaml` reads the file, validates it, and makes the same `read_*` calls in the
+same order:
+
+```python
+from hapi.catchment import Catchment
+from hapi.run import Run
+
+Coello = Catchment.from_yaml("coello-lumped-model-run.yaml")
+Run.runLumped(Coello, Routing.triangular_routing_1)
+```
+
+The four shipped examples under `examples/hydrological-model/coello/run/` are each a pair — a
+`.py` that runs the model and a `.yaml` beside it holding everything the run needs.
+
+## A complete lumped configuration
+
+```yaml
+# Paths are relative to this file, so the run works from any working directory.
+catchment:
+  name: Coello
+  start: "2009-01-01"
+  end: "2011-12-31"
+  spatial_resolution: lumped
+  temporal_resolution: daily
+
+# Lumped mode reads one CSV of catchment-average drivers, not a grid: columns are
+# [date, precipitation, ET, temperature], optionally followed by the long-term average.
+meteo:
+  path: ../../data/lumped_model/meteo_data-MSWEP.csv
+
+parameters:
+  path: ../../data/lumped_model/Coello_Lumped2021-03-08_muskingum.txt
+  snow: false
+  maxbas: false
+
+conceptual_model:
+  model_class: HBVBergestrom92
+  catchment_area: 1530
+  initial_condition: [0, 10, 10, 10, 0]
+
+# One discharge file, and no gauge table: locating gauges on a grid is a distributed concern.
+gauges:
+  discharge: ../../data/lumped_model/Qout_c.csv
+  fmt: "%Y-%m-%d"
+
+outputs:
+  results_dir: ../../data/lumped_model
+```
+
+`catchment`, `meteo` and `conceptual_model` are required. `parameters`, `gauges` and `outputs`
+are optional: omit `parameters` for a calibration, which derives them from the bounds given to
+`read_parameters_bound`, and omit `gauges` for a run that is not scored against observations.
+
+## What changes for a distributed run
+
+`spatial_resolution: distributed` changes the shape of two blocks and requires a third. `meteo`
+becomes a grid, described by its `source`:
+
+```yaml
+catchment:
+  name: Coello
+  start: "2009-01-01"
+  end: "2009-04-10"
+  spatial_resolution: distributed
+  routing_method: maxbas
+
+meteo:
+  source: rasters
+  precipitation: ../../data/distributed_model/prec
+  temperature: ../../data/distributed_model/temp
+  evapotranspiration: ../../data/distributed_model/evap
+  file_name_data_fmt: "%Y.%m.%d"
+
+# MAXBAS sends every cell straight to the outlet, so no flow-direction raster is read.
+flow_network:
+  flow_accumulation: ../../data/distributed_model/GIS/acc4000.tif
+
+gauges:
+  table: ../../data/distributed_model/stations/gauges.csv
+  discharge: ../../data/distributed_model/stations/
+```
+
+`meteo.source` picks which `MeteoInputs` loader builds the grid, and what the three driver
+fields mean:
+
+| `source` | The three driver fields name | Reads |
+|---|---|---|
+| `rasters` (default) | A folder of dated GeoTIFFs each | `MeteoInputs.from_rasters` |
+| `netcdf_files` | One NetCDF each | `MeteoInputs.from_netcdf_files` |
+| `netcdf` | A variable inside `meteo.path` | `MeteoInputs.from_netcdf` |
+
+The last is the fastest: one file, opened once, with the calendar inside it. See
+[Meteorological inputs](meteo-inputs.md) for how to pack a folder of rasters into one.
+
+## Paths are relative to the file
+
+A relative path in a configuration is resolved against the configuration's own directory, not
+against whatever directory you happen to run from. That is what makes a configuration portable:
+it travels with the data it names, and the run works from anywhere. Absolute paths are used as
+written.
+
+The example scripts rely on this — each loads the YAML sitting next to it:
+
+```python
+Coello = Catchment.from_yaml(__file__.removesuffix(".py") + ".yaml")
+```
+
+## What the file is checked for
+
+The file is validated in full before anything is opened, so a mistake is reported as a mistake
+in the file rather than as a failure deep inside a reader:
+
+- **Unknown keys are refused.** A misspelled `precipitaton` fails at parse time instead of being
+  dropped and reappearing as a missing input.
+- **So are keys that do not apply.** A `flow_network` block on a lumped run, `glob` under
+  `source: netcdf`, `gauges.table` on a lumped run — each is a line that would do nothing, and
+  each is named in the error. Only keys you actually wrote count; defaults are never held
+  against you.
+- **Required blocks are checked per shape.** A distributed run needs `flow_network` and all
+  three drivers; Muskingum additionally needs `flow_network.flow_direction`, which MAXBAS never
+  reads. A lumped run needs `meteo.path`.
+- **`routing_method` must agree with `parameters.maxbas`.** The two parameter counts differ by
+  one and `maxbas` selects which is expected, so a disagreeing pair still counts correctly and
+  then reads the wrong parameter as the routing one. Leave `routing_method` out and it is
+  derived from the parameter set.
+- **Every date is parsed against its own `fmt`**, and the period must run forwards — including
+  the meteorological window, whose bounds fall back to the catchment's when unstated.
+- **Every path is checked for existence** before the first reader runs, and all the missing ones
+  are reported together.
+
+Dates may be quoted or not: `start: 2009-01-01` is a date to YAML, and it is written back out in
+the block's `fmt`.
+
+## Reading the configuration back
+
+The parsed configuration stays on the model as `model.config`, so the blocks the build does not
+itself consume remain reachable — `outputs` above all:
+
+```python
+outputs = Coello.config.outputs
+save_to = (outputs.results_dir if outputs is not None else None) or ""
+Coello.save_results(
+    flow_acc_path=Coello.config.flow_network.flow_accumulation,
+    result=1,
+    path=save_to,
+)
+```
+
+## Out of scope
+
+The schema describes a `Catchment` run. It carries no field for a lake record, a river geometry,
+or a flow-path-length raster, so lake-aware runs (`Run.RunHapiwithLake`), the flood model
+(`Run.RunFloodModel`) and `DistMaxbas2` are still assembled in Python.
+
+`Calibration.from_yaml` works — it takes the same constructor arguments — and gives back a
+`Calibration` to call the calibration methods on. `Run.from_yaml` does not: `Run` holds entry
+points called on a model built elsewhere, so it refuses and says so.
+
+The full field-by-field reference is on the [Config API page](../api/config.md).