Skip to content

New NetCDF interface + support for parallel NetCDF#272

Open
CasparJungbacker wants to merge 64 commits into
devfrom
parallel-netcdf
Open

New NetCDF interface + support for parallel NetCDF#272
CasparJungbacker wants to merge 64 commits into
devfrom
parallel-netcdf

Conversation

@CasparJungbacker

@CasparJungbacker CasparJungbacker commented Mar 16, 2026

Copy link
Copy Markdown
Collaborator
  • Added netcdf_file_t which is an abstract derived type containing all relevent NetCDF file info. Essentially, this wraps the boiler plate code that was needed for every statistical module (ncname, vars, calls to ncinfo, define_nc etc.)
  • Added implementations of netcdf_file_t for time series, profile, cross section and field dump files. These contain a buffer with the correct dimensions that replace the old vars arrays. The buffer can be written to by setting up a pointer using the get_pointer method of the derived types.
  • Added modstat_nc_files.f90 (name due to change, don't like this one), which keeps track of all output files and does the timekeeping.
  • Updates to modstat_nc.f90 to support opening NetCDF files in parallel.
  • Added modcloudstat.f90, which is a re-implementation of some statistical quantities of modcape.f90. This module makes use of the new NetCDF interface.
  • Added CMake script to automatically deduce if the selected NetCDF library has parallel capabilities. If not, opening files in parallel is disabled in DALES.

TODO:

  • Re-opening an existing file in parallel doesn't work yet. NetCDF crashes with an HDF error on the first write.
  • Handle parallel output of vertical cross sections. Right now, if parallel output is enabled, cross section files are opened using the 3D communicator while only a subset of the MPI ranks will actually have data to write. This shouldn't be a problem if independent access is enabled for the variable to write (the default), but one needs to make sure that the correct subset of MPI ranks makes the call to writestat_nc.
  • Add André's improvements from Add coordinate origin to output, and add coordinate to crosssection slice. plus free netcdf timedep #265.
  • Update all statistical modules to make use of the new interface. (doesn't have to be done immediately, but I'd say let's do at least timestat + genstat + cross + fielddump)
    • genstat
    • timestat
    • fielddump
    • cross sections
    • agscross
    • depcross
    • lsmstat
    • samptend
    • microphysics stats
    • cape
    • radstat
    • radfield
    • msebudg
    • budgetstat
    • varbudget
    • leibniztend?
    • quadrant
  • Add OpenACC directives
  • Add clean-up routine for modstat_nc_files
  • Benchmarking
  • Re-add loutdir functionality

@CasparJungbacker

Copy link
Copy Markdown
Collaborator Author

Setting deflate=0 in NAMNETCDFSTATS fixes the HDF crash. Now, it's crashing with the following error:

FINISH PE:     0 modstat:: NetCDF: Attempt to extend dataset during NC_INDEPENDENT I/O operation

which I already suspected from looking at the PALM NetCDF code. I.e., we cannot use independent variable access mode on variables with unlimited dimensions. I see two options here:

  1. Fix the length of the time axis when defining the files. For restarting, we'd have to find some way to extend the time axis.
  2. Switch to collective access for all variables. This would be fine for xy cross sections and field dumps, but requires some MPI voodoo for vertical cross sections, since those are only written by a subset of the ranks.

@CasparJungbacker CasparJungbacker marked this pull request as ready for review March 27, 2026 09:56
CasparJungbacker and others added 16 commits March 27, 2026 11:25
)

* add albedo variable to copyin

* Set minimum threshold for friction velocity in modslurb

* modslurb: fix incorrect wall/win/road accesses

* Modslurb: fix wrong varname bug

* modtimestat: prevent reading slurb tile and possibly segfault

need to think of some way to allow timestat to interact with modslurb in future

* modprecursor: make swap public

* modslurb: fix incorrect skin temperature/moisture calculation

* modslurb: calculate thl_rad_urb instead of t_rad_urb

* Modradiation: allow spatially varying emissivity

* thermodynamics: zero ql when a layer is below saturation (#277)

* modthermodynamics bugfix: set ql to 0 in layers known to be below saturation

* modthermodynamics: remove redundant assignment

* Bugfix: tskin * exnr substituted for tskin in DEPAC call

* modslurb: fix no index used

* Modslurb: ensure a maximum resistance for urban canyon

* Modslurb: fix bug where emissivity was rest to 1

* Modnudge: fix wrong read for w, only allocate to kmax, we didnt use k1

* Modslurb: set reference atmosphere level to first half-level

* Modprecursor: allow both real32 and real64 swap

* modslurb: workaround, avoid initialization with signalling nan on Fujitsu compiler

...because those seem to crash the compiler.

* Modsurface: copyin emissivity on GPU

* Remove modstringutils, use fortran-support functions instead

* Checksim: format

* netcdf: only use nchandle_error from modstat_nc

* modstat_nc: netcdf debug options

* Modnetcdf_files: close output files at exit

* Fix fortran-support logging calls being used instead of modlogging

* Modstat_nc_files: make close_output_files actually public...

* Modstat_nc_files: increase file limit, give good error if too many files

* Modstat_nc: more helpful netcdf errors.

* fix close_output modules called after exitmpi

* Modcrosssection: various fixes

* modslurbcrosssection: slurb now has it's own output module with switches

* Modcrosssection: output variables thl0,u0,... instead of thlm,um....

* Modcrosssection: update dep,ags,lsm crosssection to new netcdf output

* modnetcdf_file_t: fix comment typo

* Some formatting and clarifications

* modslurb: fix SP compilation

---------

Co-authored-by: Fredrik Jansson <fjansson@abo.fi>
Co-authored-by: Fredrik Jansson <f.r.jansson@tudelft.nl>
Co-authored-by: Leon Geers <leon.geers@tno.nl>
Co-authored-by: Caspar Jungbacker <caspar.jungbacker@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants