Added routines for Preprocessing forcing on WCOSS2 t forecast peri… by kestonsmith-noaa · Pull Request #64 · NOAA-EMC/RWPS

kestonsmith-noaa · 2026-07-30T13:54:46Z

Added routines for pre-processing forcing [wind water level currents ice], on WCOSS2 (for recent forecast periods). These routines
(A) retrieve the necessary forecast files
(B) interpolate fields to an RWPS mesh
(C) interpolate disparate forecasts to a common time frame
(D) Combine disparate forecasts in a bayesian manner.
Note: I will mark this as draft while we discuss code organization

…ods), interpolating fields to RWPS mesh, interpolating to common time frame, and blending forcing in bayesian manner

…n WW3.

…eight and distance files to cl inputs of ComputeGriddedToRWPSInterpWeights.py

…RWPS into AddPreProcessRoutines

…uctured fields

…XXX.sh and GetXXX.sh. Moved working program to ush/PreProcess

JessicaMeixner-NOAA

Not sure where you are in the process of things but wanted to make comments on some things I saw and noticed.

JessicaMeixner-NOAA · 2026-08-11T13:03:14Z

@@ -0,0 +1,102 @@
+#!/bin/bash


jobs names should follow the JRWPS_JOBNAME where JOBNAME is something specific.

Also, would we create interpolation weights each forecast cycle? If not, this should move to dev/jobs/ and then the name is not as important.

@kestonsmith-noaa - This will need to be addressed. Also I believe this should be moved to dev/

JessicaMeixner-NOAA · 2026-08-11T13:03:38Z

 set -x 

-ww3switch=model/bin/switch_NCEP_rwps
+ww3switch=model/bin/switch_NWS_rwps


I want to say this change was made in another PR, perhaps your branch needs to have the develop merged into it?

@kestonsmith-noaa - Your branch needs the develop branch merged into it still.

JessicaMeixner-NOAA · 2026-08-11T13:04:11Z

@@ -0,0 +1,97 @@
+#!/bin/sh


Job cards do not belong in ush.

JessicaMeixner-NOAA · 2026-08-11T13:04:22Z

@@ -0,0 +1,80 @@
+#!/bin/sh


job cards do not belong in ush

JessicaMeixner-NOAA · 2026-08-11T13:05:22Z

@@ -0,0 +1,11 @@
+date=20260803


Make sure this file has an extension (.sh, .py, etc).

JessicaMeixner-NOAA · 2026-08-11T13:06:42Z

+date=20260803
+cycl=00
+rootdir=$(pwd)
+mesh="$rootdir/../../fix/rwps.oc_1500m_30km.msh"


the fix file directory would be ${FIXrwps}

So mesh would be:
mesh="${FIXrwps}/rwps.oc_1500m_30km.msh"

Also, you can use the path ${FIXrwps} within other scripts.

Also I think with the reorg of the fix, there's another folder inbetween fix and the mesh file?

JessicaMeixner-NOAA · 2026-08-11T13:12:44Z

+cycl=00
+rootdir=$(pwd)
+mesh="$rootdir/../../fix/rwps.oc_1500m_30km.msh"
+fix="$rootdir/../../fix"


use FIXrwps instead of fix see: https://nws-hpc-standards.readthedocs.io/en/stable/standards.html#a-standard-environment-variables

JessicaMeixner-NOAA · 2026-08-11T13:14:00Z

+rootdir=$(pwd)
+mesh="$rootdir/../../fix/rwps.oc_1500m_30km.msh"
+fix="$rootdir/../../fix"
+prep="$rootdir/../../PrepInputs"


Curious about this folder because it seems like it would be ${HOMErwps}/PrepInputs with HOMErwps being your clone and the same level as fix, and I'm not sure that we should have a "PrepInputs" there.

JessicaMeixner-NOAA · 2026-08-11T13:15:28Z

@@ -0,0 +1,11 @@
+date=20260803
+cycl=00
+rootdir=$(pwd)


Is this supposed to be ${HOMErwps}? Ie where your git clone is?

JessicaMeixner-NOAA · 2026-08-11T13:16:17Z

+import math
+flin=sys.argv[1] 
+flout=sys.argv[2] 
+#fld="wdir10m.nc"


remove commented out code.

kestonsmith-noaa · 2026-08-11T13:20:23Z

will do

…e arg issues

…). fixed paths for some variables and simplified envvar names

JessicaMeixner-NOAA · 2026-08-13T19:59:36Z

@@ -0,0 +1,102 @@
+#!/bin/bash


@kestonsmith-noaa - Will interpolation weights be calculated for every cycle or just once and stored as fix files?

If for every cycle, then let's move this to a jrwps_*.ecf file.
If not, then let's move this to dev/ecf.

JessicaMeixner-NOAA · 2026-08-13T20:31:03Z

+#cycl=00
+#mesh=../../fix/rwps.oc_1500m_30km.msh
+
+source /lfs/h2/emc/couple/noscrub/keston.smith/TestRWPS/RWPS/ush/PreProcess/rwpsenv


The stuff below this line, particularly the sh .sh files should be all contained in jobs/JRWPS_prep_currents.job

JessicaMeixner-NOAA · 2026-08-18T19:29:35Z

FYI - All scripts and ush files need to be all lowercase names: https://nws-hpc-standards.readthedocs.io/en/stable/standards.html#c-interpreted-code-bash-ksh-perl-or-python-scripts

JessicaMeixner-NOAA · 2026-08-18T19:30:18Z

+
+rm $prep/FetchCurrents.out $prep/ProcCurrents.out 
+
+sh GetCurrents.sh $prep > $tmp/FetchCurrents.out


ecf scripts should call a JJOB

JessicaMeixner-NOAA · 2026-08-18T19:30:53Z

 set -x 

-ww3switch=model/bin/switch_NCEP_rwps
+ww3switch=model/bin/switch_NWS_rwps


@kestonsmith-noaa - Your branch needs the develop branch merged into it still.

…tines

…sing

…preprocessing on wcoss2. Added scripts to dev for computing interpolation weights

…les for preparing forcing and generating interpolation weights.

kestonsmith-noaa · 2026-08-25T18:26:46Z

Pre-processing routines have been changed to more closely follow global workflow. Currently only set to work on wcoss2. Weights are confirmed in a mirror fix location under my noscrub directory for mesh oc_1500m_30km. Others should be added shortly and then will be copied to the group RWPS fix location when these are all prepared and tested.

…p some files

…outside of rtofs coverage. occurs only for coarse test meshes. cleaned up weight generation ecf file for clarity

JessicaMeixner-NOAA · 2026-08-31T14:01:11Z

+
+
+# copy mesh to local fix directory
+cp -p $RWPSfix/fix/$meshname/20260722/rwps.$meshname.msh ../fix/


meshname or RWPSfix are not defined.

I believe this is actually done in link_workflow.sh -

I have not removed get_externals.sh but we should move things to link_workflow.sh if possible.

…s now done in dev/compute_interpolation_weights

JessicaMeixner-NOAA · 2026-09-03T19:11:21Z

+
+    time_var=ncout.createVariable('time', 'f8', ('time',))
+    time_var.units         = 'seconds since 1970-01-01 00:00:00.0 0:00'
+    time_var.long_name     = 'verification time generated by wgrib2 function verftime()'


Is it a verification time or valid time?

This is unnecessarily specific. I am going to copy time attributes from the input file rather than explicitly re-specifying. Will update shortly

JessicaMeixner-NOAA · 2026-09-03T19:19:02Z

+
+rm $tmp/FetchCurrent.out $tmp/ProcCurrent.out
+
+sh get_current.sh > $tmp/FetchCurrent.out


Each ecf script should be calling a JJOB, so for example, at this point of the script, I would expect:

${HOMErwps}/jobs/JRWPS_PREP_CURRENT

Then the
${HOMErwps}/jobs/JRWPS_PREP_CURRENT
would call a script:

${SCRrwps}/exrwps_prep_current.sh , which is either what process_current.sh should be renamed to, or then it would be calling the ush/process_current script.

JessicaMeixner-NOAA · 2026-09-03T19:20:00Z

+
+module list 
+
+rm $tmp/FetchIce.out $tmp/ProcIce.out


All the comments from the above ecf script are valid here.

JessicaMeixner-NOAA · 2026-09-03T19:21:05Z

+
+
+# Consolidate u,v surface currents from global RTOFS into a single NetCDF file.
+


Can you add a comment about what the two sys argument inputs are?

Added description of command line arguments

Co-authored-by: Jessica Meixner <jessica.meixner@noaa.gov>

remove unnecessary specific piped output files Co-authored-by: Jessica Meixner <jessica.meixner@noaa.gov>

…bm.py. seperated prep tasks from ecf to ecf(load modules) and jobs/JRWPS_PREP_ to execute task

JessicaMeixner-NOAA · 2026-09-03T20:40:11Z

@@ -0,0 +1,64 @@
+#!/bin/bash


Is this file supposed to be here?

removed unused file

JessicaMeixner-NOAA · 2026-09-03T20:40:47Z

@@ -0,0 +1,29 @@
+#!/bin/bash


This is not an ush script, should this move somewhere else?

removed unused file

JessicaMeixner-NOAA · 2026-09-03T20:42:58Z

+!/bin/bash
+# --------------------------------------------------------------------------- #
+#                                                                             #
+# Copy external fix files that are too large to store in repository           #


I think this description needs updated//removed along with the "last changed" date.

changed- I will go through the rest to make sure there aren't other bad headers

…process to dev/compute_interpolation_weights/ush. split dev/compute_interpolation_weights/ecf/compute_interpolation_weights.ecf into module loading portion and dev/compute_interpolation_weights/jobs/COMPUTE_INTERPOLATION_WEIGHTS

fixed typo's in README

JessicaMeixner-NOAA · 2026-09-04T15:03:01Z

+
+cd RWPS/dev/compute_interpolation_weights/sorc
+
+dev/compute_interpolation_weights/sorc/compute_interpolation_weights.sh


Where does the ecf script play in this?

If I wanted to run this would I try to run

qsub compute_interpolation_weights.ecf

are there things I should change or update when I do?

clarified in README

$ cd RWPS/dev/compute_interpolation_weights/sorc
$ sh compute_interpolation_weights.sh oc_1500m_30km

...

…rcing.sh

…ights.sh to dev/compute_interpolation_weights/ush/

…. Quieted mkdir statements where missing in preprocessing routines

JessicaMeixner-NOAA · 2026-09-09T18:57:03Z

+
+#machine dependend path to rtofs, nbm, rrfs, and stofs forecast files
+export COMINrtofs="/lfs/h1/ops/prod/com/rtofs/v2.5/rtofs.$PDY/"
+export COMINnbm="/lfs/h3/mdl/ptmp/mdl.nbm/blend/v5.2/blend.$PDY/$cyc/grib2"


Remove variables that are not used or needed.

Added routines for retrieving forcing on WCOSS2 (recent forecast peri…

07d10ee

…ods), interpolating fields to RWPS mesh, interpolating to common time frame, and blending forcing in bayesian manner

kestonsmith-noaa marked this pull request as draft July 30, 2026 13:55

kestonsmith-noaa added 5 commits July 30, 2026 10:06

updated switch filename in sorc/build_ww3.sh to reflect name change i…

1fbf8e8

…n WW3.

Added scripts for launching interpolation weights calulation. Added w…

d6b7eaf

…eight and distance files to cl inputs of ComputeGriddedToRWPSInterpWeights.py

Merge branch 'AddPreProcessRoutines' of github.com:/kestonsmith-noaa/…

5a5ed4a

…RWPS into AddPreProcessRoutines

Added scripts to support generating weights for STOFS and other unstr…

d82dc06

…uctured fields

changed path specification and removed command line args from Process…

38580ec

…XXX.sh and GetXXX.sh. Moved working program to ush/PreProcess

JessicaMeixner-NOAA reviewed Aug 11, 2026

View reviewed changes

kestonsmith-noaa added 3 commits August 11, 2026 14:52

cleaned up files in ush/PreProcess and removed some other command lin…

5d69933

…e arg issues

seperated preprocessing jobs by field and moved to ecf(wcoss2 version…

4ba7b2d

…). fixed paths for some variables and simplified envvar names

fixed other path errors

531f59e

JessicaMeixner-NOAA reviewed Aug 13, 2026

View reviewed changes

JessicaMeixner-NOAA reviewed Aug 18, 2026

View reviewed changes

kestonsmith-noaa added 12 commits August 19, 2026 14:10

move script file names to all lower case

97af00d

Merge remote-tracking branch 'upstream/develop' into AddPreProcessRou…

a94b153

…tines

brought up to date with develop

a805b29

fixed filename convention for get_rtofs.sh

4aedb5a

fixed numerous path problems and added script for launching preproces…

5988ba9

…sing

restored missing .tmpl files and other fix files

a493c00

removed redundant weight generation file for stofs v3

b9fd93f

fixed corrupted code in ush/preprocess/nbm/make_nbm_wind.sh

c7b20dc

Removed module loading and added preprocessing script to version for …

005a568

…preprocessing on wcoss2. Added scripts to dev for computing interpolation weights

removed esmpy log files in dev that were unintentionally incorporated

6c1c479

fixed filename in sorc/prepare_forcing.sh and added minimal readme fi…

afbbd56

…les for preparing forcing and generating interpolation weights.

removed redundant preprocessing library from dev

5f0e7b6

kestonsmith-noaa marked this pull request as ready for review August 25, 2026 18:22

resource allocation set to work for oc_500m_10km on wcoss2. Cleaned u…

976fa89

…p some files

fixed a failure in dist2bnd calculation when no rwps mesh nodes fall …

8f21c8f

…outside of rtofs coverage. occurs only for coarse test meshes. cleaned up weight generation ecf file for clarity

JessicaMeixner-NOAA reviewed Aug 31, 2026

View reviewed changes

removed unused file jobs/GenerateInterpolationWeights.jobcard. This i…

4848eae

…s now done in dev/compute_interpolation_weights

JessicaMeixner-NOAA reviewed Sep 3, 2026

View reviewed changes

kestonsmith-noaa and others added 4 commits September 3, 2026 16:18

Update spd_dir_to_uv_nbm.py

10e8b0c

Added description of command line arguments

Update ecf/jrwps_prep_current.ecf

88effe3

Co-authored-by: Jessica Meixner <jessica.meixner@noaa.gov>

Update ecf/jrwps_prep_current.ecf

76cfdd4

remove unnecessary specific piped output files Co-authored-by: Jessica Meixner <jessica.meixner@noaa.gov>

switched to copying time attributes in ush/preprocess/spd_dir_to_uv_n…

e24200e

…bm.py. seperated prep tasks from ecf to ecf(load modules) and jobs/JRWPS_PREP_ to execute task

JessicaMeixner-NOAA reviewed Sep 3, 2026

View reviewed changes

Comment thread dev/compute_interpolation_weights/ush/compute_gridded_to_rwps_interp_weights.py

kestonsmith-noaa added 2 commits September 4, 2026 14:37

removed unused file compute_unstr_to_rwps_interp_weights.sh

56a8ab9

JessicaMeixner-NOAA reviewed Sep 4, 2026

View reviewed changes

Comment thread dev/compute_interpolation_weights/ush/compute_interpolation_weights.sh

Update README.txt

4aed3c2

fixed typo's in README

JessicaMeixner-NOAA reviewed Sep 4, 2026

View reviewed changes

kestonsmith-noaa and others added 6 commits September 4, 2026 15:15

moved prepare_forcing.sh to ush/. Corrected header for ush/prepare_fo…

8b9f696

…rcing.sh

Update compute_interpolation_weights.sh

5d95849

Update README.txt

9363b14

moved dev/compute_interpolation_weights/sorc/compute_interpolation_we…

e63f791

…ights.sh to dev/compute_interpolation_weights/ush/

made new scripts, jobs/JRWPS_PREP_* executeable

7842f94

added missing import statement in ush/preprocess/spd_dir_to_uv_nbm.py…

95a511f

…. Quieted mkdir statements where missing in preprocessing routines

JessicaMeixner-NOAA reviewed Sep 9, 2026

View reviewed changes


		rm $prep/FetchCurrents.out $prep/ProcCurrents.out

		sh GetCurrents.sh $prep > $tmp/FetchCurrents.out



		# copy mesh to local fix directory
		cp -p $RWPSfix/fix/$meshname/20260722/rwps.$meshname.msh ../fix/


		rm $tmp/FetchCurrent.out $tmp/ProcCurrent.out

		sh get_current.sh > $tmp/FetchCurrent.out



		# Consolidate u,v surface currents from global RTOFS into a single NetCDF file.


		cd RWPS/dev/compute_interpolation_weights/sorc

		dev/compute_interpolation_weights/sorc/compute_interpolation_weights.sh

Uh oh!

Conversation

kestonsmith-noaa commented Jul 30, 2026

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kestonsmith-noaa commented Aug 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JessicaMeixner-NOAA commented Aug 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kestonsmith-noaa commented Aug 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!