Update Delft3D-FLOW submit script to slurm (AL8) by hrajagers · Pull Request #1007 · Deltares/Delft3D

hrajagers · 2026-06-25T12:32:51Z

What was done

Update Delft3D-FLOW submit script to slurm consistent with our Alma 8 cluster.
Replaced the content of submit_dflow2d3d.sh and rd2d3d.sh scripts by that of the _AL8 versions that some internal DIMRsets included. Replaced instead of including additional files since the previous H6 system is no longer available anyway.

Evidence of the work done

Video/figures
<add video/figures if applicable>
Clear from the issue description
Not applicable

Tests

Tests updated
<add testcase numbers if applicable, Issue number>
Not applicable

Documentation

Documentation updated
<add description of changes if applicable, Issue number>
Not applicable

Issue link

Closes DELFT3D-37935

…fication

willem4

Thanks for the update of the scripts. I have not tested the scripts, as I trust you have. I have some minor suggestions.

willem4 · 2026-06-26T09:17:45Z

+#SBATCH --job-name=delft3d4     # Specify a name for the job allocation.
+#SBATCH --time=00:15:00         # Set a limit on the total run time of the job allocation.
+#SBATCH --partition=4vcpu       # Request a specific partition for the resource allocation.
+                                # See: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes.


Do we want to include this in the run script? Or is it accessible for externals?

shouldnt this (and the other files) receive the _AL8 as described in the text? Then it is clear that it is tailored for Alma8/Slurm

Due to the number of different running script, can you add a comment line about when to use this sh-file file?
One line such as: "This is the master calling script to run ....."

Removed the reference to the publicwiki page. It's not accessible outside Deltares. The AL8 slurm cloud isn't accessible either. Both scripts should work fine on other slurm systems ... except when using C-SUMO within MATLAB as that still includes a hardcoded path for the MATLAB installation.

I don't want to keep the _AL8 postfix. There is nothing in the script doesn't contain anything Alma Linux 8 specific. The primary dependencies are (1) module intelmpi/2021.11.0, (2) MATLAB in case of C-SUMO, obviously (3) Delft3D installation, and (4) slurm resource management.

A purpose statement added to the start of both scripts.

willem4 · 2026-06-26T09:33:10Z

 debuglevel=-1
 runscript_extraopts=
-NNODES=1
 wavefile=runwithoutwaveonlinebydefault


Suggested change

wavefile=runwithoutwaveonlinebydefault

# wavefile is set to "runwithoutwaveonlinebydefault". When adding the option -w, --wavefile <wname> in the run script a different file can be specified and this activates wave online.

wavefile=runwithoutwaveonlinebydefault

willem4 · 2026-06-26T09:34:52Z

    fi
-
+    
    if [ $NSLOTS -eq 1 ]; then


Is the condition correct here. A single node, multi core also uses mpi right?

willem4 · 2026-06-26T09:35:36Z

-numnode=1
-queue=normal-e3-c7
 runscript_extraopts=
 wavefile=runwithoutwaveonlinebydefault


consider above suggestion

willem4 · 2026-06-26T09:36:34Z

+    echo "-p, --PARTITION <PARTITION>"
+    echo "       PARTITION, default $PARTITION"
+    echo "       see also: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes"
+    echo "-t, --TIME_LIMIT <TIME_LIMIT>"


Is it useful to add the formatting of the time string?

willem4 · 2026-06-26T09:36:56Z

+    echo "       number of nodes, default $NODES"
+    echo "-p, --PARTITION <PARTITION>"
+    echo "       PARTITION, default $PARTITION"
+    echo "       see also: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes"


Should this be included in external scripts?

willem4 · 2026-06-26T09:37:39Z

+# Usage:
    #
-    # This script runs Delft3D-FLOW in parallel on Linux
+    # This script runs Delft3D-FLOW in parallel on Linux H7


Is H7 necessary?

marcio-albernaz

Hi Bert,
Nice that you worked on it! ;)
I havent tested the scripts myself. But look good. Mainly comments to improve clarity

marcio-albernaz · 2026-06-26T10:48:34Z

+#SBATCH --job-name=delft3d4     # Specify a name for the job allocation.
+#SBATCH --time=00:15:00         # Set a limit on the total run time of the job allocation.
+#SBATCH --partition=4vcpu       # Request a specific partition for the resource allocation.
+                                # See: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes.


shouldnt this (and the other files) receive the _AL8 as described in the text? Then it is clear that it is tailored for Alma8/Slurm

marcio-albernaz · 2026-06-26T10:49:14Z

-    echo "Usage: ${0##*/} [OPTION]..."
-    echo "This script is called by submit_dflow2d3d.sh."
+    echo "Usage: sbatch [SLURM OPTIONS]... ${0##*/} [OPTION]..."
+    echo "Run delft3d4 on H7."


H7 or Hal8?

marcio-albernaz · 2026-06-26T10:51:09Z

 corespernodedefault=1
 corespernode=$corespernodedefault
-D3D_HOME=
+NSLOTS=$SLURM_TASKS_PER_NODE


where is SLURM_TASKS_PER_NODE defined? Like Willem, I havent test it myself. But I don't see this variable being defined here

SLURM_TASKS_PER_NODE is set by the slurm process. It should be equal to the TASKS_PER_NODE value in the submit script and passed to slurm by means of the -c (--corespernode) option.

Your comment made me realize that NSLOTS was set equal to SLURM_TASKS_PER_NODE. I have adjusted this to SLURM_NTASKS (or 1 if not set).

marcio-albernaz · 2026-06-26T10:53:16Z

+export I_MPI_FABRICS=ofi
+export FI_PROVIDER=tcp
+export I_MPI_OFI_PROVIDER=tcp
+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK


I had a recent discussion with Adri about OMP_NUM_THREADS and OMP_NUM_THREADS_SWAN. Nowadays, for FM, Swan is controlled by the latter. Which one is correct for D3D-4? Good to double check it and perhaps make it consistent with OMP_NUM_THREADS_SWAN

Indeed, OMP_NUM_THREADS for SWAN is set to OMP_NUM_THREADS_SWAN inside the swan.sh file (which is used to start the swan executable by both Delft3D-WAVE and D-Waves). I've now also set OMP_NUM_THREADS_SWAN ... the OMP_NUM_THREADS variable could be used by Delft3D-FLOW, but isn't used now (afaik).

marcio-albernaz · 2026-06-26T10:55:38Z

+#SBATCH --job-name=delft3d4     # Specify a name for the job allocation.
+#SBATCH --time=00:15:00         # Set a limit on the total run time of the job allocation.
+#SBATCH --partition=4vcpu       # Request a specific partition for the resource allocation.
+                                # See: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes.


Due to the number of different running script, can you add a comment line about when to use this sh-file file?
One line such as: "This is the master calling script to run ....."

hrajagers · 2026-06-26T11:52:13Z

Thank you @marcio-albernaz and @willem4 for your comments. This pull request started as quick action for PK to get the submit_dflow2d3d_AL8.sh script for running on "AL8" systematically included in the DIMRset folder (it's now only included in 2.29.14 and 2.30.12). It made sense to remove the old submit_dflow2d3d.sh script in the process since the H6 is no longer available. However, you're asking all kinds of questions that I don't have an immediate answer for. So, let's try to find some time next week to discuss the best way forward. These scripts are primarily intended for internal use (e.g. the name change from H7 to Alma8 suggests that the script can be used on all Alma8 systems, but I expect that it depends more on the slurm scheduler than on the OS version itself) and I know that there is a discussion whether they should be here or somewhere else.

marcio-albernaz · 2026-06-26T14:15:55Z

Hi @hrajagers , I am fine to skip the phylosophical and cosmetic reviews. But I had at least one about the definition of a variable that may actually cause problems. I am fine if you restrict to check the technical points only. And maybe clarify if you actually tested it, which I am almost sure you did :)

hrajagers · 2026-06-26T15:03:32Z

@marcio-albernaz Sorry to disappoint you. I didn't test it ... I just copied the files that have been unchanged for the past two years. So, either nobody is using them, or other users have tested them. Let's discuss next week in person (PK is running some tests now ... using a manual copy of the files).

hrajagers · 2026-06-26T15:08:22Z

If these files are renamed, then we also need to adjust
src\engines_gpl\flow2d3d\packages\flow2d3d\CMakeLists.txt
which refers to these files by full name for copying as part of the "install" step.

… the submit script doesn't have to know anything about the C-SUMO keywords ... updated the bash conditions updated the C-SUMO checking inside rd2d3d.sh ... the previous [ -n $csumoscript ] check only convers the compiled C-SUMO case

…//github.com/Deltares/Delft3D into d3d4/task/DELFT3D-37935_update_submit_script

… to copy the dedicated submit scripts anymore. Skip processing of RTC and WAVE flags in the submit script ... they are processed by the next rd2d3d script.

marcio-albernaz

Hi @bert, before approving it - can you say if these scripts have been tested?
The previous comments from Willem and I are also still open. You can close if they are not relevant anymore.

hrajagers

Comments processed. I did some very basic testing a while back, but I run a few more simulations using the latest scripts. Furthermore, I want to have a green test bench before approval. Since these scripts are not used in the testbench, all failures should be unrelated to these changes ... but a green test bench also doesn't give any guarantee that these scripts are correct.

hrajagers · 2026-07-03T09:26:00Z

+#SBATCH --job-name=delft3d4     # Specify a name for the job allocation.
+#SBATCH --time=00:15:00         # Set a limit on the total run time of the job allocation.
+#SBATCH --partition=4vcpu       # Request a specific partition for the resource allocation.
+                                # See: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes.


Removed the reference to the publicwiki page. It's not accessible outside Deltares. The AL8 slurm cloud isn't accessible either. Both scripts should work fine on other slurm systems ... except when using C-SUMO within MATLAB as that still includes a hardcoded path for the MATLAB installation.

I don't want to keep the _AL8 postfix. There is nothing in the script doesn't contain anything Alma Linux 8 specific. The primary dependencies are (1) module intelmpi/2021.11.0, (2) MATLAB in case of C-SUMO, obviously (3) Delft3D installation, and (4) slurm resource management.

A purpose statement added to the start of both scripts.

hrajagers · 2026-07-03T09:36:39Z

 corespernodedefault=1
 corespernode=$corespernodedefault
-D3D_HOME=
+NSLOTS=$SLURM_TASKS_PER_NODE


SLURM_TASKS_PER_NODE is set by the slurm process. It should be equal to the TASKS_PER_NODE value in the submit script and passed to slurm by means of the -c (--corespernode) option.

Your comment made me realize that NSLOTS was set equal to SLURM_TASKS_PER_NODE. I have adjusted this to SLURM_NTASKS (or 1 if not set).

hrajagers · 2026-07-03T09:49:34Z

+export I_MPI_FABRICS=ofi
+export FI_PROVIDER=tcp
+export I_MPI_OFI_PROVIDER=tcp
+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK


Indeed, OMP_NUM_THREADS for SWAN is set to OMP_NUM_THREADS_SWAN inside the swan.sh file (which is used to start the swan executable by both Delft3D-WAVE and D-Waves). I've now also set OMP_NUM_THREADS_SWAN ... the OMP_NUM_THREADS variable could be used by Delft3D-FLOW, but isn't used now (afaik).

hrajagers · 2026-07-03T09:50:21Z

+    echo "-p, --PARTITION <PARTITION>"
+    echo "       PARTITION, default $PARTITION"
+    echo "       see also: https://publicwiki.deltares.nl/display/Deltareken/Compute+nodes"
+    echo "-t, --TIME_LIMIT <TIME_LIMIT>"


marcio-albernaz

Indeed, here testing the scripts is important because it won't be tested in the tesbench (as far as I know).
Happy to approve it and see what happens :)

Update Delft3D-FLOW submit script to AL8/H7

35f5761

hrajagers self-assigned this Jun 25, 2026

hrajagers requested review from marcio-albernaz and willem4 June 25, 2026 15:25

hrajagers and others added 2 commits June 26, 2026 09:36

Rename Delft3D 4 C-SUMO submit scripts as well ... drop the AL8 speci…

ff088f9

…fication

Merge branch 'main' into d3d4/task/DELFT3D-37935_update_submit_script

67aedbd

willem4 reviewed Jun 26, 2026

View reviewed changes

marcio-albernaz reviewed Jun 26, 2026

View reviewed changes

hrajagers commented Jun 26, 2026

View reviewed changes

hrajagers added 5 commits July 2, 2026 15:19

Support for spaces in folder paths

6e90400

merge rd2d3d_csumo.sh into rd2d3d.sh

9abe7cb

Merge branch 'd3d4/task/DELFT3D-37935_update_submit_script' of https:…

5945615

…//github.com/Deltares/Delft3D into d3d4/task/DELFT3D-37935_update_submit_script

C-SUMO logic merged into the default slurm submit scripts, so no need…

3dfeecd

… to copy the dedicated submit scripts anymore. Skip processing of RTC and WAVE flags in the submit script ... they are processed by the next rd2d3d script.

marcio-albernaz reviewed Jul 3, 2026

View reviewed changes

Process review comments.

cbd33d4

hrajagers commented Jul 3, 2026

View reviewed changes

hrajagers changed the title ~~Update Delft3D-FLOW submit script to AL8/H7~~ Update Delft3D-FLOW submit script to slurm (AL8) Jul 3, 2026

marcio-albernaz approved these changes Jul 3, 2026

View reviewed changes

	wavefile=runwithoutwaveonlinebydefault
	# wavefile is set to "runwithoutwaveonlinebydefault". When adding the option -w, --wavefile <wname> in the run script a different file can be specified and this activates wave online.
	wavefile=runwithoutwaveonlinebydefault

Uh oh!

Conversation

hrajagers commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was done

Evidence of the work done

Tests

Documentation

Issue link

Uh oh!

willem4 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

willem4 Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcio-albernaz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hrajagers commented Jun 26, 2026

Uh oh!

marcio-albernaz commented Jun 26, 2026

Uh oh!

hrajagers commented Jun 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcio-albernaz left a comment

Choose a reason for hiding this comment

Uh oh!

hrajagers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcio-albernaz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

hrajagers commented Jun 25, 2026 •

edited

Loading

willem4 Jun 26, 2026 •

edited

Loading