From 8b53740f774c5036309daede13f86454f4efb0fb Mon Sep 17 00:00:00 2001 From: mansurjisan Date: Mon, 19 Oct 2026 23:18:12 -0400 Subject: [PATCH 1/3] docs: refresh README and Read the Docs pages (current systems, all modules) --- README.md | 58 +++++++++++------------ docs/api/coords.rst | 7 +++ docs/api/forcing.rst | 110 ++++++++++++++++++++++++++++++++++++++----- docs/api/index.rst | 2 + docs/api/interp.rst | 9 ++-- docs/api/io.rst | 3 +- docs/api/post.rst | 61 ++++++++++++++++++++++++ docs/conf.py | 4 +- docs/index.rst | 66 +++++++++++++------------- docs/usage.rst | 77 +++++++++++++++++------------- 10 files changed, 285 insertions(+), 112 deletions(-) create mode 100644 docs/api/coords.rst create mode 100644 docs/api/post.rst diff --git a/README.md b/README.md index 991c700..4072009 100644 --- a/README.md +++ b/README.md @@ -2,50 +2,50 @@ [![Docs](https://readthedocs.org/projects/nos-utils/badge/?version=latest)](https://nos-utils.readthedocs.io/en/latest/) -Python forcing generators for NOAA National Ocean Service's operational ocean forecast systems. +nos-utils is a Python package that generates the forcing and run-configuration files for NOAA National Ocean Service coastal forecast systems, and some of their post-processing products. It replaces parts of the COMF and STOFS Fortran/shell prep chains with testable Python modules. [nos-workflow](https://github.com/mansurjisan/nos-workflow) uses it as a submodule (`ush/python/nos-utils`) to run the prep and post stages. -**Documentation**: [nos-utils.readthedocs.io](https://nos-utils.readthedocs.io/en/latest/) +## Systems -## Install +| System | Model | What nos-utils provides | +|---|---|---| +| SECOFS | SCHISM, DATM+SCHISM, DATM+SCHISM+WW3 (UFS-Coastal) | sflux or DATM atmospheric forcing, RTOFS open boundary, river, tides, nudging, param.nml, UFS-Coastal configuration | +| STOFS-3D-ATL | SCHISM standalone, DATM+SCHISM; ported to match operational v3.1 | the same set plus ADT and tide-gauge SSH adjustment, NWM and St. Lawrence rivers, and the post products | +| STOFS-2D-Global | ADCIRC | GFS meteorology in ADCIRC OWI NetCDF form (NWS=14), tidal factors | + +## Install, test, docs ```bash -pip install -e ".[full]" # numpy, netCDF4, scipy +pip install -e ".[full]" # adds netCDF4, scipy, cfgrib, pyyaml +pytest # unit tests; synthetic fixtures, no data downloads ``` -## Usage +Documentation: [nos-utils.readthedocs.io](https://nos-utils.readthedocs.io/en/latest/). Example: ```python from nos_utils.config import ForcingConfig from nos_utils.forcing import GFSProcessor config = ForcingConfig.for_secofs(pdy="20260324", cyc=12) -gfs = GFSProcessor(config, input_path="/data/gfs/v16.3", output_path="/data/sflux") -result = gfs.process() +result = GFSProcessor(config, input_path="/data/gfs", output_path="/work/sflux").process() ``` -CLI: -```bash -nos-utils prep --ofs secofs --pdy 20260324 --cyc 12 --gfs /data/gfs --output /work/ -``` +A full SCHISM prep cycle runs through `PrepOrchestrator` or `nos-utils prep --ofs secofs --pdy 20260324 --cyc 12 --output /work/prep` (see `nos-utils --help` and `nos-utils list`). -## Processors +## Main modules -| Processor | Input | Output | -|-----------|-------|--------| -| GFSProcessor | GFS 0.25° GRIB2 | sflux or DATM | -| HRRRProcessor | HRRR 3km GRIB2 | sflux (secondary) | -| GEFSProcessor | GEFS ensemble GRIB2 | sflux per member | -| RTOFSProcessor | RTOFS NetCDF | elev2D, TEM/SAL_3D, uv3D | -| NWMProcessor | NWM channel_rt | vsource.th, msource.th | -| TidalProcessor | TPXO9 template | bctides.in | -| ParamNmlProcessor | param.nml template | param.nml | -| HotstartProcessor | restart archive | hotstart.nc | -| PartitionProcessor | hgrid.gr3 | partition.prop | -| ESMFMeshProcessor | forcing grid | esmf_mesh.nc | -| BlenderProcessor | GFS+HRRR sflux | datm_forcing.nc | +| Purpose | Modules in `nos_utils.forcing` | +|---|---| +| Atmospheric | `gfs`, `hrrr`, `gefs`, `blender` (HRRR+GFS), `sflux_writer`, `datm_writer`, `forcing_writer` | +| Ocean boundary | `rtofs`, `adt`, `dynamic_adjust`, `wl_bias`, `obc_ops_interp`, `nudging`, `nudge_ops` | +| River | `nwm`, `river_clim`, `st_lawrence` | +| Tides | `tidal` (bctides.in), `adcirc_tides` (ADCIRC nodal factors) | +| Restart, partition, UFS | `hotstart`, `partition`, `param_nml`, `esmf_mesh`, `ufs_config` | +| Waves | `wave_boundary` (GFS-Wave spectra to WW3 `nest.ww3`) | +| ADCIRC | `adcirc_met` (GFS to OWI NetCDF, fort.22), `adcirc_tides` | +| Post-processing | `nos_utils.post`: `adcirc`, `geopkg`, `maxele`, `profiles`, `slab2d`, `stations` | -## Test +Other packages: `nos_utils.interp` (structured and precomputed-weight interpolation), `nos_utils.io` (GRIB2, NetCDF and SCHISM grid readers), `nos_utils.config` (`ForcingConfig`), `nos_utils.orchestrator`, `nos_utils.nco_bridge` (NCO environment variables), `nos_utils.cli`. -```bash -pytest -v # 206 unit tests, no data needed -``` +## Disclaimer + +This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an "as is" basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government. diff --git a/docs/api/coords.rst b/docs/api/coords.rst new file mode 100644 index 0000000..e099424 --- /dev/null +++ b/docs/api/coords.rst @@ -0,0 +1,7 @@ +Longitude conventions +===================== + +.. automodule:: nos_utils.coords + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/forcing.rst b/docs/api/forcing.rst index c32a54a..6643e31 100644 --- a/docs/api/forcing.rst +++ b/docs/api/forcing.rst @@ -2,8 +2,7 @@ Forcing Processors ================== .. automodule:: nos_utils.forcing - :members: - :undoc-members: + :no-members: Base classes ------------ @@ -40,6 +39,14 @@ GEFS :undoc-members: :show-inheritance: +HRRR and GFS blender +~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.blender + :members: + :undoc-members: + :show-inheritance: + Ocean boundary -------------- @@ -59,6 +66,46 @@ ADT satellite SSH blending :undoc-members: :show-inheritance: +Dynamic SSH adjustment +~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.dynamic_adjust + :members: + :undoc-members: + :show-inheritance: + +Water-level bias correction +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.wl_bias + :members: + :undoc-members: + :show-inheritance: + +Operational RTOFS interpolation (3-D open boundary) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.obc_ops_interp + :members: + :undoc-members: + :show-inheritance: + +T/S interior nudging +~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.nudging + :members: + :undoc-members: + :show-inheritance: + +Operational nudging records +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.nudge_ops + :members: + :undoc-members: + :show-inheritance: + River ----- @@ -78,18 +125,29 @@ River climatology :undoc-members: :show-inheritance: +St. Lawrence River +~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.st_lawrence + :members: + :undoc-members: + :show-inheritance: + Tidal ----- +SCHISM bctides.in +~~~~~~~~~~~~~~~~~ + .. automodule:: nos_utils.forcing.tidal :members: :undoc-members: :show-inheritance: -Nudging -------- +ADCIRC tidal parameters +~~~~~~~~~~~~~~~~~~~~~~~ -.. automodule:: nos_utils.forcing.nudging +.. automodule:: nos_utils.forcing.adcirc_tides :members: :undoc-members: :show-inheritance: @@ -122,7 +180,7 @@ Partition :show-inheritance: UFS-Coastal ------------- +----------- ESMF mesh ~~~~~~~~~ @@ -132,10 +190,32 @@ ESMF mesh :undoc-members: :show-inheritance: -Blender -~~~~~~~ +UFS-Coastal configuration files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. automodule:: nos_utils.forcing.blender +.. automodule:: nos_utils.forcing.ufs_config + :members: + :undoc-members: + :show-inheritance: + +Waves +----- + +GFS-Wave boundary spectra +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.wave_boundary + :members: + :undoc-members: + :show-inheritance: + +ADCIRC (STOFS-2D-Global) +------------------------ + +GFS to ADCIRC OWI NetCDF +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.adcirc_met :members: :undoc-members: :show-inheritance: @@ -144,7 +224,7 @@ Writers ------- Sflux writer -~~~~~~~~~~~~~ +~~~~~~~~~~~~ .. automodule:: nos_utils.forcing.sflux_writer :members: @@ -152,9 +232,17 @@ Sflux writer :show-inheritance: DATM writer -~~~~~~~~~~~~ +~~~~~~~~~~~ .. automodule:: nos_utils.forcing.datm_writer :members: :undoc-members: :show-inheritance: + +GFS and HRRR forcing writer +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. automodule:: nos_utils.forcing.forcing_writer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/api/index.rst b/docs/api/index.rst index b0ca055..7dd94b8 100644 --- a/docs/api/index.rst +++ b/docs/api/index.rst @@ -5,7 +5,9 @@ API Reference :maxdepth: 2 config + coords forcing + post interp io orchestrator diff --git a/docs/api/interp.rst b/docs/api/interp.rst index 8f8d50a..2215381 100644 --- a/docs/api/interp.rst +++ b/docs/api/interp.rst @@ -2,8 +2,7 @@ Interpolation ============= .. automodule:: nos_utils.interp - :members: - :undoc-members: + :no-members: Overview -------- @@ -30,7 +29,7 @@ There are two steps: 1. **One-time build** (offline, after any grid change): run the Fortran prep once with ``NOS_EXPORT_WEIGHTS=YES`` to dump ``*_remesh_export.txt`` - text files, then convert them to NPZ using :func:`build_ssh_npz`, + text files, then convert them to NPZ using :func:`build_npz`, :func:`build_nudge_npz`, or :func:`build_3d_npz`. The NPZ is committed to the FIX directory and keyed to the RTOFS grid via an MD5 checksum. @@ -48,12 +47,12 @@ Example import numpy as np from pathlib import Path from nos_utils.interp.precomputed_weights import ( - build_ssh_npz, + build_npz, apply_precomputed_ssh, ) # --- One-time build (offline) --- - build_ssh_npz( + build_npz( export_txt=Path("obc_ssh_remesh_export.txt"), rtofs_lon_2d=rtofs_lon, # (ny, nx) from rtofs_glo_2ds rtofs_lat_2d=rtofs_lat, diff --git a/docs/api/io.rst b/docs/api/io.rst index 4241fae..0231579 100644 --- a/docs/api/io.rst +++ b/docs/api/io.rst @@ -2,8 +2,7 @@ I/O Utilities ============= .. automodule:: nos_utils.io - :members: - :undoc-members: + :no-members: GRIB2 extraction ----------------- diff --git a/docs/api/post.rst b/docs/api/post.rst new file mode 100644 index 0000000..5972ab5 --- /dev/null +++ b/docs/api/post.rst @@ -0,0 +1,61 @@ +Post-processing +=============== + +.. automodule:: nos_utils.post + :no-members: + +ADCIRC-format water level +------------------------- + +.. automodule:: nos_utils.post.adcirc + :members: + :undoc-members: + :show-inheritance: + +Disturbance GeoPackage +---------------------- + +.. automodule:: nos_utils.post.geopkg + :members: + :undoc-members: + :show-inheritance: + +Maximum water level +------------------- + +.. automodule:: nos_utils.post.maxele + :members: + :undoc-members: + :show-inheritance: + +Station vertical profiles +------------------------- + +.. automodule:: nos_utils.post.profiles + :members: + :undoc-members: + :show-inheritance: + +2-D slab fields +--------------- + +.. automodule:: nos_utils.post.slab2d + :members: + :undoc-members: + :show-inheritance: + +Station time series +------------------- + +.. automodule:: nos_utils.post.stations + :members: + :undoc-members: + :show-inheritance: + +Mesh helpers +------------ + +.. automodule:: nos_utils.post.mesh + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/conf.py b/docs/conf.py index abbc230..77a87cb 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -33,7 +33,9 @@ "show-inheritance": True, } autodoc_member_order = "bysource" -autodoc_mock_imports = ["numpy", "netCDF4", "scipy", "cfgrib", "xarray", "nco", "cdo"] +# numpy is a core dependency and is installed on Read the Docs; mocking it breaks +# module-level numpy arithmetic in forcing.nudge_ops. MJ (10/19/26) +autodoc_mock_imports = ["netCDF4", "scipy", "cfgrib", "xarray", "nco", "cdo"] # Suppress duplicate object warnings from dataclass fields re-exported via __init__ suppress_warnings = ["duplicate.object.description"] diff --git a/docs/index.rst b/docs/index.rst index 31ae04e..fa7f54a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,52 +3,54 @@ nos-utils **Python forcing generators for NOAA NOS-OFS ocean forecast systems.** -``nos-utils`` is a standalone Python package that replaces the legacy Fortran -and shell preprocessing toolchain (COMF/STOFS) with a modular, testable -pipeline for generating atmospheric, ocean-boundary, river, and tidal forcing -files for `SCHISM `_-based Operational -Forecast Systems (OFS). +``nos-utils`` is a standalone Python package that generates forcing and +run-configuration files for NOAA NOS coastal forecast systems, and some of +their post-processing products. It replaces parts of the legacy Fortran and +shell preprocessing toolchain (COMF/STOFS) with modular, testable Python. +`nos-workflow `_ uses it as a +submodule to run the prep and post stages. Supported systems ----------------- -* **SECOFS** -- SE Coastal Ocean Forecast System. Production-ready on WCOSS2: - validated against the operational COMF Fortran pipeline - (SSH RMSE = 0.5 mm, R = 0.999998; prep time ~23 min vs 40 min sequential). -* **STOFS-3D-ATL** -- Storm Surge 3-D Atlantic. Work in progress: atmospheric, - NWM medium-range river, ADT satellite SSH blending, and interior T/S nudging - are wired up; operational validation ongoing. -* **UFS-Coastal variants** -- DATM-coupled (``nws=4``) configurations of both - SECOFS and STOFS-3D-ATL (ESMF mesh + HRRR/GFS Delaunay blender). +* **SECOFS** -- SE Coastal Ocean Forecast System. SCHISM (``nws=2`` sflux), + DATM+SCHISM (``nws=4``) and DATM+SCHISM+WW3 under UFS-Coastal. The SCHISM + prep was validated against the operational COMF Fortran pipeline (SSH RMSE + 0.5 mm, measured on SECOFS RTOFS open-boundary output). +* **STOFS-3D-ATL** -- Storm Surge 3-D Atlantic. Standalone SCHISM and + DATM+SCHISM, ported to match operational v3.1: GFS/HRRR atmospheric forcing, + RTOFS open boundary with ADT and tide-gauge SSH adjustment, NWM and + St. Lawrence rivers, interior T/S nudging, and the post products. +* **STOFS-2D-Global** -- ADCIRC. GFS meteorology in ADCIRC OWI NetCDF form + (``NWS=14``) and tidal nodal factors. * **Ensemble** -- GEFS-driven perturbation members via :meth:`ForcingConfig.for_ensemble`. Key features ------------ -* **Byte-identical COMF output structure** -- ``obc.tar`` (6 files), - ``sflux_*.nc``, ``bctides.in``, ``param.nml``, ``vsource.th``, and - ``source_sink.in`` match the layout expected by the operational model. +* **COMF output structure** -- ``obc.tar``, ``sflux_*.nc``, ``bctides.in``, + ``param.nml``, ``vsource.th`` and ``source_sink.in`` follow the layout + expected by the operational SCHISM model. * **Precomputed INTERP_REMESH weights** -- The Fortran triangulation is - exported once and replayed every cycle as pure NumPy gather operations, - giving Fortran-equivalent accuracy (SSH RMSE 0.5 mm) at a fraction of the - runtime. -* **ROI subsetting for RTOFS** -- 3D temperature and salinity reads are - restricted to the SECOFS subdomain, yielding ~92% I/O reduction. -* **Parallel Phase 1 prep** -- GFS, HRRR, NWM, and tidal processors run - concurrently via ``ThreadPoolExecutor``; heavy NetCDF+Delaunay work for - RTOFS and nudging runs sequentially in Phase 2. -* **Full-cycle speedup** -- Total SECOFS prep: ~23 min with nos-utils vs - ~40 min for the sequential COMF Fortran pipeline (1.7x faster). -* **Interior T/S nudging** -- Python generates ``TEM_nu.nc`` and ``SAL_nu.nc`` - for all 32,613 SECOFS interior nodes using precomputed nudge weights. -* **Unified API across OFS** -- the same ``PrepOrchestrator`` runs SECOFS or - STOFS just by swapping the :class:`ForcingConfig` factory. + exported once and replayed every cycle as NumPy gather operations. +* **Parallel Phase 1 prep** -- GFS, HRRR, NWM and tidal processors run + concurrently via ``ThreadPoolExecutor``; the heavy NetCDF work for RTOFS + and nudging runs sequentially in Phase 2. +* **Interior T/S nudging** -- ``TEM_nu.nc`` and ``SAL_nu.nc`` from precomputed + nudge weights. +* **UFS-Coastal support** -- ESMF mesh, HRRR/GFS blender, DATM writer, and + generation of the UFS configuration files, including the WW3 boundary + spectra (``nest.ww3``) from GFS-Wave. +* **ADCIRC support** -- GFS to OWI NetCDF meteorology and ADCIRC tidal + parameters for STOFS-2D-Global. +* **Post-processing** -- the ``nos_utils.post`` subpackage writes the + ADCIRC-format water-level field, hourly disturbance GeoPackage, maximum + water level, station time series and profiles, and 2-D slab products. * **NCO environment bridge** -- ``nco_bridge.run_prep()`` reads standard ``PDY``/``cyc``/``COMINgfs``/``FIXofs`` variables and drives the orchestrator from inside a J-job. -* **206 unit tests passing** -- all inputs are synthetic fixtures; no data - downloads required. +* **Synthetic test fixtures** -- the unit tests need no data downloads. Quick start ----------- diff --git a/docs/usage.rst b/docs/usage.rst index 1848dc2..acdd6d5 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -50,8 +50,8 @@ in the config: The STOFS factory pre-configures GFS 0.25 deg resolution, a separate HRRR domain, RTOFS ROI indices for both OBC and nudging, ADT satellite SSH -blending, interior T/S nudging, and the NWM ``medium_range_mem1`` product -with 121 target files. +blending, dynamic SSH adjustment, St. Lawrence River forcing, interior T/S +nudging, and the NWM ``medium_range_mem1`` product with 121 target files. Configuration ------------- @@ -70,12 +70,15 @@ create preconfigured instances: # SECOFS with UFS-Coastal DATM coupling (nws=4) config = ForcingConfig.for_secofs_ufs(pdy="20260406", cyc=0) - # STOFS-3D-ATL (24h nowcast / 108h forecast, ADT blending, nudging) + # STOFS-3D-ATL (24h nowcast / 96h forecast, ADT blending, nudging) config = ForcingConfig.for_stofs_3d_atl(pdy="20260406", cyc=0) # STOFS-3D-ATL + UFS-Coastal DATM config = ForcingConfig.for_stofs_3d_atl_ufs(pdy="20260406", cyc=0) + # STOFS-2D-Global (ADCIRC; GFS 0.25 deg, nws=14) + config = ForcingConfig.for_stofs_2d_glo(pdy="20260406", cyc=0) + # Ensemble member (GEFS-driven; member 0 = control, met_num=2) config = ForcingConfig.for_ensemble(pdy="20260406", cyc=0, member=3) @@ -109,33 +112,42 @@ Each processor follows a consistent pattern: Available processors (imported from :mod:`nos_utils.forcing`): -====================== ========================================================== -Processor Responsibility -====================== ========================================================== -``GFSProcessor`` GFS GRIB2 -> ``sflux_air_1/rad_1/prc_1.*.nc`` -``HRRRProcessor`` HRRR GRIB2 -> ``sflux_air_2/rad_2/prc_2.*.nc`` -``GEFSProcessor`` GEFS ensemble GRIB2 -> sflux per member -``RTOFSProcessor`` RTOFS NetCDF -> ``elev2D``, ``TEM_3D``, ``SAL_3D``, ``uv3D`` + ``obc.tar`` -``NWMProcessor`` NWM channel_rt -> ``vsource.th``, ``msource.th``, ``source_sink.in`` -``RiverClimProcessor`` USGS climatology fallback for rivers -``TidalProcessor`` bctides.in generation (calls Fortran ``tide_fac`` exe when available) -``NudgingProcessor`` ``TEM_nu.nc``, ``SAL_nu.nc`` interior nudging -``ParamNmlProcessor`` ``param.nml`` from template with runtime substitution -``HotstartProcessor`` Locate and validate restart file; extract ``time_hotstart`` -``PartitionProcessor`` ``partition.prop`` for MPI domain decomposition -``ESMFMeshProcessor`` ESMF mesh for UFS-Coastal DATM coupling -``BlenderProcessor`` HRRR+GFS Delaunay blending -> ``datm_forcing.nc`` -``SfluxWriter`` SCHISM sflux NetCDF writer (``nws=2``) -``DATMWriter`` UFS-Coastal DATM NetCDF writer (``nws=4``) -====================== ========================================================== +============================ ========================================================== +Processor Responsibility +============================ ========================================================== +``GFSProcessor`` GFS GRIB2 -> ``sflux_air_1/rad_1/prc_1.*.nc`` +``HRRRProcessor`` HRRR GRIB2 -> ``sflux_air_2/rad_2/prc_2.*.nc`` +``GEFSProcessor`` GEFS ensemble GRIB2 -> sflux per member +``RTOFSProcessor`` RTOFS NetCDF -> ``elev2D``, ``TEM_3D``, ``SAL_3D``, ``uv3D`` + ``obc.tar`` +``NWMProcessor`` NWM channel_rt -> ``vsource.th``, ``msource.th``, ``source_sink.in`` +``RiverClimProcessor`` USGS climatology fallback for rivers +``StLawrenceProcessor`` St. Lawrence River ``flux.th`` (STOFS-3D-ATL) +``DynamicAdjustProcessor`` SSH boundary correction from tide-gauge observations +``TidalProcessor`` bctides.in generation (calls Fortran ``tide_fac`` exe when available) +``NudgingProcessor`` ``TEM_nu.nc``, ``SAL_nu.nc`` interior nudging +``ParamNmlProcessor`` ``param.nml`` from template with runtime substitution +``HotstartProcessor`` Locate and validate restart file; extract ``time_hotstart`` +``PartitionProcessor`` ``partition.prop`` for MPI domain decomposition +``ESMFMeshProcessor`` ESMF mesh for UFS-Coastal DATM coupling +``BlenderProcessor`` HRRR+GFS Delaunay blending -> ``datm_forcing.nc`` +``UFSConfigProcessor`` UFS-Coastal ``model_configure``, ``datm_in``, ``ufs.configure``, ... +``SfluxWriter`` SCHISM sflux NetCDF writer (``nws=2``) +``DATMWriter`` UFS-Coastal DATM NetCDF writer (``nws=4``) +============================ ========================================================== + +``WaveBoundaryProcessor`` (GFS-Wave spectra to the WW3 ``nest.ww3`` boundary) is +imported from :mod:`nos_utils.forcing.wave_boundary`. The ADCIRC modules +(:mod:`nos_utils.forcing.adcirc_met`, :mod:`nos_utils.forcing.adcirc_tides`) +are function-based and are listed under the API reference. RTOFS with precomputed weights and ADT blending ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:class:`~nos_utils.forcing.RTOFSProcessor` will automatically pick up -precomputed INTERP_REMESH weights from the FIX directory if they are -present, falling back to on-the-fly Delaunay otherwise. The weights give -Fortran-equivalent results (SSH RMSE = 0.5 mm vs the operational Fortran). +When run through :class:`~nos_utils.orchestrator.PrepOrchestrator`, the SSH +open boundary uses precomputed INTERP_REMESH weights from the FIX directory +if they are present (``*.obc_ssh_weights.npz``) and falls back to on-the-fly +Delaunay interpolation otherwise. For SECOFS the weights reproduce the +operational Fortran SSH boundary to a 0.5 mm RMSE. .. code-block:: python @@ -145,11 +157,13 @@ Fortran-equivalent results (SSH RMSE = 0.5 mm vs the operational Fortran). config=config, input_path="/data/rtofs", output_path="/work/secofs/obc", - fix_path="/path/to/fix/secofs", + grid_file="/path/to/fix/secofs/secofs.hgrid.ll", + obc_ctl_file="/path/to/fix/secofs/secofs.obc.ctl", + vgrid_file="/path/to/fix/secofs/secofs.vgrid.in", ) result = rtofs.process() - # ADT blending runs automatically when config.adt_enabled is True - # and the ADT data + fix weights are available. + # ADT blending runs when config.adt_enabled is True and the ADT data + # and weight file are available. NWM river forcing ~~~~~~~~~~~~~~~~~ @@ -182,12 +196,11 @@ Interior nudging config=config, input_path="/data/rtofs", output_path="/work/secofs", - fix_path="/path/to/fix/secofs", + nudge_weight_file="/path/to/fix/secofs/secofs.nudge.gr3", ) result = nudge.process() -The nudging processor reads the ~32K interior-node weight file -(``secofs.obc_nudge_weights.npz``) and writes ``TEM_nu.nc`` and +The nudging processor reads the interior-node nudge weights and writes ``TEM_nu.nc`` and ``SAL_nu.nc`` with a configurable relaxation timescale (``config.nudging_timescale_seconds``, default 86400 s). From 6aa85ff5e8d40626efe3e401ede24aed2fd0b2da Mon Sep 17 00:00:00 2001 From: mansurjisan Date: Mon, 19 Oct 2026 23:28:02 -0400 Subject: [PATCH 2/3] docs: review fixes (prep CLI scope, weights lookup, cross-refs, river, WW3) --- README.md | 10 +++++----- docs/api/interp.rst | 2 ++ docs/contributing.rst | 6 +++--- docs/index.rst | 22 ++++++++++++++-------- docs/installation.rst | 4 ++-- docs/usage.rst | 28 +++++++++++++++++++--------- 6 files changed, 45 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 4072009..a06c90f 100644 --- a/README.md +++ b/README.md @@ -8,15 +8,15 @@ nos-utils is a Python package that generates the forcing and run-configuration f | System | Model | What nos-utils provides | |---|---|---| -| SECOFS | SCHISM, DATM+SCHISM, DATM+SCHISM+WW3 (UFS-Coastal) | sflux or DATM atmospheric forcing, RTOFS open boundary, river, tides, nudging, param.nml, UFS-Coastal configuration | +| SECOFS | SCHISM, DATM+SCHISM, DATM+SCHISM+WW3 (UFS-Coastal) | sflux or DATM atmospheric forcing, RTOFS open boundary, river, tides, nudging, param.nml, UFS-Coastal configuration; for WW3, the GFS-Wave `nest.ww3` boundary and WAV PET bounds / coupling interval | | STOFS-3D-ATL | SCHISM standalone, DATM+SCHISM; ported to match operational v3.1 | the same set plus ADT and tide-gauge SSH adjustment, NWM and St. Lawrence rivers, and the post products | -| STOFS-2D-Global | ADCIRC | GFS meteorology in ADCIRC OWI NetCDF form (NWS=14), tidal factors | +| STOFS-2D-Global | ADCIRC | GFS meteorological forcing in ADCIRC OWI NetCDF form (NWS=14) | ## Install, test, docs ```bash -pip install -e ".[full]" # adds netCDF4, scipy, cfgrib, pyyaml -pytest # unit tests; synthetic fixtures, no data downloads +pip install -e ".[full,dev]" # full: netCDF4, scipy, cfgrib, pyyaml; dev: pytest +pytest # unit tests; synthetic fixtures, no data downloads ``` Documentation: [nos-utils.readthedocs.io](https://nos-utils.readthedocs.io/en/latest/). Example: @@ -29,7 +29,7 @@ config = ForcingConfig.for_secofs(pdy="20260324", cyc=12) result = GFSProcessor(config, input_path="/data/gfs", output_path="/work/sflux").process() ``` -A full SCHISM prep cycle runs through `PrepOrchestrator` or `nos-utils prep --ofs secofs --pdy 20260324 --cyc 12 --output /work/prep` (see `nos-utils --help` and `nos-utils list`). +A prep cycle runs through `PrepOrchestrator(config, paths, skip_legacy=False)`, which nos-workflow reaches via `nco_bridge.run_prep(skip_legacy=False)`. The `nos-utils prep` CLI (see `nos-utils --help`) keeps the default `skip_legacy=True`, which skips the SECOFS RTOFS, NWM and nudging steps. ## Main modules diff --git a/docs/api/interp.rst b/docs/api/interp.rst index 2215381..8df9739 100644 --- a/docs/api/interp.rst +++ b/docs/api/interp.rst @@ -27,6 +27,8 @@ Precomputed weight workflow There are two steps: +.. currentmodule:: nos_utils.interp.precomputed_weights + 1. **One-time build** (offline, after any grid change): run the Fortran prep once with ``NOS_EXPORT_WEIGHTS=YES`` to dump ``*_remesh_export.txt`` text files, then convert them to NPZ using :func:`build_npz`, diff --git a/docs/contributing.rst b/docs/contributing.rst index 7855a5a..0ec1b9f 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -19,7 +19,7 @@ whole suite runs in a few seconds. .. code-block:: bash - pytest tests/ -q # full suite (206 tests) + pytest tests/ -q # full suite pytest tests/ -q -k gfs # only GFS tests pytest tests/ -q -k stofs # only STOFS-3D-ATL tests pytest --cov=nos_utils --cov-report=html @@ -42,9 +42,9 @@ Adding a new processor ---------------------- 1. Create ``nos_utils/forcing/my_processor.py`` subclassing - :class:`~nos_utils.forcing.ForcingProcessor`. + :class:`~nos_utils.forcing.base.ForcingProcessor`. 2. Implement ``process()`` and return a - :class:`~nos_utils.forcing.ForcingResult` -- populate ``success``, + :class:`~nos_utils.forcing.base.ForcingResult` -- populate ``success``, ``output_files``, ``metadata``, and ``errors``. 3. Export it from ``nos_utils/forcing/__init__.py`` and add it to ``__all__``. diff --git a/docs/index.rst b/docs/index.rst index fa7f54a..1f221dd 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,17 +14,17 @@ Supported systems ----------------- * **SECOFS** -- SE Coastal Ocean Forecast System. SCHISM (``nws=2`` sflux), - DATM+SCHISM (``nws=4``) and DATM+SCHISM+WW3 under UFS-Coastal. The SCHISM + DATM+SCHISM (``nws=4``) and DATM+SCHISM+WW3 under UFS-Coastal. For the + WW3 case nos-utils builds the ``nest.ww3`` boundary from GFS-Wave and sets + the WAV PET bounds and coupling interval in the UFS configuration files. The SCHISM prep was validated against the operational COMF Fortran pipeline (SSH RMSE 0.5 mm, measured on SECOFS RTOFS open-boundary output). * **STOFS-3D-ATL** -- Storm Surge 3-D Atlantic. Standalone SCHISM and DATM+SCHISM, ported to match operational v3.1: GFS/HRRR atmospheric forcing, RTOFS open boundary with ADT and tide-gauge SSH adjustment, NWM and St. Lawrence rivers, interior T/S nudging, and the post products. -* **STOFS-2D-Global** -- ADCIRC. GFS meteorology in ADCIRC OWI NetCDF form - (``NWS=14``) and tidal nodal factors. -* **Ensemble** -- GEFS-driven perturbation members via - :meth:`ForcingConfig.for_ensemble`. +* **STOFS-2D-Global** -- ADCIRC. GFS meteorological forcing in ADCIRC OWI + NetCDF form (``NWS=14``). Key features ------------ @@ -42,8 +42,10 @@ Key features * **UFS-Coastal support** -- ESMF mesh, HRRR/GFS blender, DATM writer, and generation of the UFS configuration files, including the WW3 boundary spectra (``nest.ww3``) from GFS-Wave. -* **ADCIRC support** -- GFS to OWI NetCDF meteorology and ADCIRC tidal - parameters for STOFS-2D-Global. +* **ADCIRC support** -- GFS to OWI NetCDF meteorology for STOFS-2D-Global + (the ``adcirc_tides`` module also provides nodal-factor routines). +* **Ensemble** -- GEFS-driven perturbation members via + :meth:`~nos_utils.config.ForcingConfig.for_ensemble`. * **Post-processing** -- the ``nos_utils.post`` subpackage writes the ADCIRC-format water-level field, hourly disturbance GeoPackage, maximum water level, station time series and profiles, and 2-D slab products. @@ -73,7 +75,7 @@ Quick start "fix": "/path/to/fix/secofs", "output": "/tmp/secofs_prep", } - result = PrepOrchestrator(config, paths).run(phase="nowcast") + result = PrepOrchestrator(config, paths, skip_legacy=False).run(phase="nowcast") print(result.summary()) CLI: @@ -86,6 +88,10 @@ CLI: nos-utils list # show available OFS factories and processors +The ``skip_legacy`` default (``True``) skips the RTOFS, NWM and nudging steps +for SECOFS, and the CLI does not change it. Use ``skip_legacy=False`` in +Python (nos-workflow calls ``nco_bridge.run_prep(skip_legacy=False)``) to run them. + Documentation contents ---------------------- diff --git a/docs/installation.rst b/docs/installation.rst index 67591ca..ec4aa45 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -76,11 +76,11 @@ and then reused every cycle. Expected files (SECOFS, in ``$FIXofs``): * ``secofs.obc_ssh_weights.npz`` -- 1488 boundary nodes, global RTOFS 2D - (3298x4500). Used by :class:`~nos_utils.forcing.RTOFSProcessor` for SSH. + (3298x4500). Used by :class:`~nos_utils.forcing.rtofs.RTOFSProcessor` for SSH. * ``secofs.obc_3d_weights.npz`` -- 1488 boundary nodes, regional RTOFS 3D (e.g. US_east 1710x742). Used for T, S, and UV boundary interpolation. * ``secofs.obc_nudge_weights.npz`` -- ~32K interior nodes. Used by - :class:`~nos_utils.forcing.NudgingProcessor` for ``TEM_nu.nc`` and + :class:`~nos_utils.forcing.nudging.NudgingProcessor` for ``TEM_nu.nc`` and ``SAL_nu.nc``. Generating the weight files diff --git a/docs/usage.rst b/docs/usage.rst index acdd6d5..ad6e93f 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -4,7 +4,7 @@ Usage Quick start: PrepOrchestrator ----------------------------- -The simplest way to run a full SCHISM prep cycle is via +The simplest way to run a SCHISM prep cycle is via :class:`~nos_utils.orchestrator.PrepOrchestrator`. It chains every processor in the correct order and runs the lightweight ones (GFS, HRRR, NWM, tidal) concurrently via a ``ThreadPoolExecutor``; the heavy RTOFS and nudging steps @@ -28,12 +28,19 @@ concurrent thread access on large reads. "comout": "/tmp/secofs_comout", } - orch = PrepOrchestrator(config, paths, run_name="secofs") + orch = PrepOrchestrator(config, paths, run_name="secofs", skip_legacy=False) result = orch.run(phase="nowcast") print(result.summary()) +With the default ``skip_legacy=True`` the RTOFS, NWM and nudging steps are +skipped for SECOFS (they were handled by the legacy shell scripts). Pass +``skip_legacy=False`` to run them in Python; the same +flag is available as :func:`nos_utils.nco_bridge.run_prep`\ ``(skip_legacy=False)``, +which nos-workflow calls. The ``nos-utils prep`` CLI +does not set it, so for SECOFS it skips those steps. + ``PrepResult`` exposes ``success`` (all critical steps passed), -``results`` (one :class:`~nos_utils.forcing.ForcingResult` per step), +``results`` (one :class:`~nos_utils.forcing.base.ForcingResult` per step), ``all_output_files``, ``all_errors``, and ``elapsed_seconds``. Switching OFS @@ -143,10 +150,11 @@ are function-based and are listed under the API reference. RTOFS with precomputed weights and ADT blending ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When run through :class:`~nos_utils.orchestrator.PrepOrchestrator`, the SSH -open boundary uses precomputed INTERP_REMESH weights from the FIX directory -if they are present (``*.obc_ssh_weights.npz``) and falls back to on-the-fly -Delaunay interpolation otherwise. For SECOFS the weights reproduce the +For the SSH open boundary, :class:`~nos_utils.forcing.rtofs.RTOFSProcessor` +looks for precomputed INTERP_REMESH weights in ``$FIXofs`` or ``$FIXstofs3d`` +(environment variables), then in its input directory, under the exact names +``secofs.obc_ssh_weights.npz`` or ``obc_ssh_weights.npz``. If neither is +found it falls back to on-the-fly Delaunay interpolation. For SECOFS the weights reproduce the operational Fortran SSH boundary to a 0.5 mm RMSE. .. code-block:: python @@ -182,8 +190,10 @@ NWM river forcing For the STOFS domain this assembles 121 ``medium_range_mem1`` files into ``vsource.th`` / ``msource.th`` / ``source_sink.in``. SECOFS has no matching -NWM reaches and uses :class:`~nos_utils.forcing.RiverClimProcessor` (USGS -daily climatology) instead. +NWM reaches; it runs ``NWMProcessor`` with the ``analysis_assim`` product. +:class:`~nos_utils.forcing.river_clim.RiverClimProcessor` (USGS daily +climatology) is available as a separate processor but the orchestrator does +not call it. Interior nudging ~~~~~~~~~~~~~~~~ From 83652b2c10c9d974d1037ab3a54d4d526bc2ded6 Mon Sep 17 00:00:00 2001 From: mansurjisan Date: Mon, 19 Oct 2026 23:30:16 -0400 Subject: [PATCH 3/3] docs: skip_legacy skips RTOFS and NWM only; drop tidal-factor note and river contradiction --- README.md | 2 +- docs/index.rst | 5 ++--- docs/usage.rst | 8 ++++---- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index a06c90f..063f0ce 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ config = ForcingConfig.for_secofs(pdy="20260324", cyc=12) result = GFSProcessor(config, input_path="/data/gfs", output_path="/work/sflux").process() ``` -A prep cycle runs through `PrepOrchestrator(config, paths, skip_legacy=False)`, which nos-workflow reaches via `nco_bridge.run_prep(skip_legacy=False)`. The `nos-utils prep` CLI (see `nos-utils --help`) keeps the default `skip_legacy=True`, which skips the SECOFS RTOFS, NWM and nudging steps. +A prep cycle runs through `PrepOrchestrator(config, paths, skip_legacy=False)`, which nos-workflow reaches via `nco_bridge.run_prep(skip_legacy=False)`. The `nos-utils prep` CLI (see `nos-utils --help`) keeps the default `skip_legacy=True`, which skips the SECOFS RTOFS and NWM steps. ## Main modules diff --git a/docs/index.rst b/docs/index.rst index 1f221dd..fa297c3 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -42,8 +42,7 @@ Key features * **UFS-Coastal support** -- ESMF mesh, HRRR/GFS blender, DATM writer, and generation of the UFS configuration files, including the WW3 boundary spectra (``nest.ww3``) from GFS-Wave. -* **ADCIRC support** -- GFS to OWI NetCDF meteorology for STOFS-2D-Global - (the ``adcirc_tides`` module also provides nodal-factor routines). +* **ADCIRC support** -- GFS to OWI NetCDF meteorology for STOFS-2D-Global. * **Ensemble** -- GEFS-driven perturbation members via :meth:`~nos_utils.config.ForcingConfig.for_ensemble`. * **Post-processing** -- the ``nos_utils.post`` subpackage writes the @@ -88,7 +87,7 @@ CLI: nos-utils list # show available OFS factories and processors -The ``skip_legacy`` default (``True``) skips the RTOFS, NWM and nudging steps +The ``skip_legacy`` default (``True``) skips the RTOFS and NWM steps for SECOFS, and the CLI does not change it. Use ``skip_legacy=False`` in Python (nos-workflow calls ``nco_bridge.run_prep(skip_legacy=False)``) to run them. diff --git a/docs/usage.rst b/docs/usage.rst index ad6e93f..7f4af3c 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -32,10 +32,10 @@ concurrent thread access on large reads. result = orch.run(phase="nowcast") print(result.summary()) -With the default ``skip_legacy=True`` the RTOFS, NWM and nudging steps are +With the default ``skip_legacy=True`` the RTOFS and NWM steps are skipped for SECOFS (they were handled by the legacy shell scripts). Pass ``skip_legacy=False`` to run them in Python; the same -flag is available as :func:`nos_utils.nco_bridge.run_prep`\ ``(skip_legacy=False)``, +flag is available as ``run_prep(skip_legacy=False)`` (:func:`nos_utils.nco_bridge.run_prep`), which nos-workflow calls. The ``nos-utils prep`` CLI does not set it, so for SECOFS it skips those steps. @@ -189,8 +189,8 @@ NWM river forcing result = nwm.process() For the STOFS domain this assembles 121 ``medium_range_mem1`` files into -``vsource.th`` / ``msource.th`` / ``source_sink.in``. SECOFS has no matching -NWM reaches; it runs ``NWMProcessor`` with the ``analysis_assim`` product. +``vsource.th`` / ``msource.th`` / ``source_sink.in``. SECOFS runs +``NWMProcessor`` with the ``analysis_assim`` product. :class:`~nos_utils.forcing.river_clim.RiverClimProcessor` (USGS daily climatology) is available as a separate processor but the orchestrator does not call it.