[MansurJisan-NCO]

Description

STOFS-3D-ATL prep segfaulted in the STOFS container on AWS ParallelWorks. dmesg showed: python3 segfault ... in libhdf5-0b47eb58.so.310.2.0.

Cause. PrepOrchestrator runs its Phase 1 steps (GFS, HRRR, NWM, TIDAL) in a ThreadPoolExecutor. Since netCDF4-python 1.6.0, the netCDF-C calls run without the GIL. netCDF-C and the HDF5 bundled in the wheels are not thread-safe, so concurrent netCDF I/O from two threads can crash; here GFS was writing its forcing file while NWM read 139 netCDF files. The same run passed on a faster machine only because the steps did not overlap.

Fix. A new _phase1_workers(n_tasks) helper picks the executor size:

• NOS_PREP_WORKERS (an integer >= 1) overrides, capped at the number of tasks. An invalid value logs a warning and is ignored.
• Otherwise, netCDF4 >= 1.6, or a version that does not parse, gives 1 worker (sequential).
• Older netCDF4, or no netCDF4, keeps all tasks in parallel, so stacks with netCDF4 < 1.6 behave as before.

The executor, result collection and error handling are unchanged. The log line now states the worker count and the reason. The docs index feature bullet is updated.

No other thread pool in nos_utils does netCDF I/O; post/geopkg.py uses processes.

Type of change

• Bug fix (non-breaking change that fixes an issue)

Change characteristics

• Changes outputs (values, COMOUT files or names, archives)? NO. Only the order in which Phase 1 runs changes; each step writes the same files.
• Breaking change to existing functionality? NO
• Requires a nos-utils submodule bump (nos-workflow) or a paired nos-workflow change (nos-utils)? YES: a nos-workflow submodule bump follows, together with the JNOS_PREP exit-code fix.
• Requires new or regenerated fix files on the machine? NO
• Requires a re-prep or a new hotstart before the next cycle? NO
• Machines affected: any stack with netCDF4 >= 1.6 (containers, Hercules). WCOSS2 behaviour is unchanged if its netCDF4 is older.

How has this been tested?

• tests/test_phase1_workers.py covers the env override, the cap, invalid values, versions 1.5.8, 1.6.0rc1, 1.7.2 and 1.10.0, an unparseable version, and netCDF4 not importable.
• Full suite: 1309 passed, 9 skipped.
• The crash was seen in a container on ParallelWorks. A rerun with the fixed image is pending the nos-workflow bump.

Checklist

• Dependent PRs merged and published (nos-utils before any submodule bump)
• pytest suite passes
• Existing operational behavior preserved, or the change is called out above
• No edits to a shared FIXofs while it is cycling
• Self-review done; yaml comments and docstrings updated where behavior changed
• No markdown files added to the repo (this template is the only exception)