Reduce the memory usage that is important for ne1024 simulation

[sjsprecious]

Description of changes

This PR aims at reducing memory usage of a simulation at ne1024 resolution. All the changes are done by Claude under my supervisory.

---

The goal is to cut CTSM initialization memory (and some init time) at high resolution (like ne1024), where per-rank data replication and duplicate ESMF mesh construction dominate startup cost.

The detailed edits:

1. New per-node shared-memory helper: clm_shmem_mod.F90

A MPI-3 shared-memory module and specialized for CTSM's decomposition setup. The idea is that arrays that are otherwise allocated identically on every MPI rank instead get one physical copy per shared-memory node, mapped into every rank on that node — freeing ranks_per_node − 1 copies per node.

• clm_shmem_alloc_i4_1d(ptr, win, n) — allocate a node-shared default-integer rank-1 array (only the node leader requests storage via MPI_Win_allocate_shared; peers map the leader's segment via MPI_Win_shared_query).

• clm_shmem_leader_allreduce_sum_i4(ptr, win, n) — fence → node leaders sum partials across nodes over a leader-only communicator → fence to publish. Builds a globally-summed array in the shared buffer without every rank holding a global-sized copy.

• clm_shmem_free / clm_shmem_fence / clm_shmem_is_leader / clm_shmem_leader_comm / clm_shmem_npes_per_node — lifecycle and query helpers; lazily build node-local and node-leader communicators via mpi_comm_split_type(MPI_COMM_TYPE_SHARED).

2. lnd_set_decomp_and_domain.F90 — apply the shmem helper to the global land mask

The global land mask lndmask_glob(gsize) was previously allocated on every rank and built with an all-rank ESMF_VMAllReduce into a second global-sized temporary (itemp_glob). Now, in both code paths (lnd_set_lndmask_from_maskmesh and lnd_set_lndmask_from_lndmesh):

• lndmask_glob is allocated once per node via clm_shmem_alloc_i4_1d, with a new lndmask_win window handle threaded through both subroutine signatures.

• Leader zeroes it, fence, each rank fills its disjoint local indices, then clm_shmem_leader_allreduce_sum_i4 replaces the ESMF_VMAllReduce + itemp_glob temporary (the temporary is deleted entirely).

• Cleanup is now branch-aware: the cmeps driver paths free via clm_shmem_free(lndmask_glob, lndmask_win); the lilac path still uses plain deallocate (it uses a plain allocate).

This removes two global-sized integer arrays per rank (the mask copy + the all-reduce temp), replaced by one node-shared copy.

3. NetCDF file-handle close fixes

Closing pio file handles that were opened but closed late or never — frees buffers earlier in init:

• clm_instMod.F90: moves ncd_pio_closefile(params_ncid) earlier — to right after its last use (bgc_vegetation_inst%Init) instead of at the end of init_accflds.
• initVerticalMod.F90: moves ncd_pio_closefile(ncid) to right after the last read (STD_ELEV) instead of the end of initVertical.
• UrbanParamsType.F90: adds a missing ncd_pio_closefile(ncid) on the early-return path (nlevurb == 0) that previously leaked the handle.
• organicFileMod.F90: adds ncd_pio_closefile(ncid) after reading ORGANIC.
  surfrdMod.F90: adds two ncd_pio_closefile(ncid) calls after dimension reads complete (after the pft/cft dims, and after nlevurb).

4. reuse already-built model mesh for redist streams

• PrigentRoughnessStreamType.F90 / UrbanTimeVarType.F90: the changes are now done in CDEPS whenever these streams use stream_mapalgo='redist'

---

Specific notes

Contributors other than yourself, if any:

• (Replace this text and add more list items as needed)

CTSM issues resolved or otherwise addressed, if any:

Resolves #4103

Any user interface changes (namelist or namelist defaults changes)?

No.

Testing planned or performed, if any:

aux_clm

Requirements before merge:

• I have followed the CTSM contribution guidelines.

• The code in this PR branch builds with no errors.
• The code in this PR branch runs with no errors. Briefly describe tested configuration(s): aux_clm
• This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. (a) does not change answers

• I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe:
• This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?:

[sjsprecious]

SE meeting:

• This is blocked by the required CDEPS update.
• Since this has submodule updates, we will merge this to master just to be safe—even though it should be bit-for-bit.
• @ekluzek will handle this.

Thanks @samsrabin . My CDEPS PR was merged and I just updated the CDEPS tag here. Let me know if you or @ekluzek have any questions or comments before merging it.

[samsrabin]

@briandobbins I might have misremembered. Is there also a CMEPS PR this depends on?

[sjsprecious]

My CMEPS PR is independent of my CTSM PR here.

[briandobbins]

How critical is your CMEPS PR to the memory reduction you need to run, Jian? (On a conversation about it right now - there's some fundamental complexities in CMEPS that this is highlighting, so I would just like your assessment.)

[sjsprecious]

How critical is your CMEPS PR to the memory reduction you need to run, Jian? (On a conversation about it right now - there's some fundamental complexities in CMEPS that this is highlighting, so I would just like your assessment.)

Thanks @briandobbins . We have a very tight memory headroom on Derecho so I need all the changes in my CDEPS, CMEPS, CTSM, and future CAM PRs to get the ne1024 simulation to work with more ranks per node. Bill mentioned some concerns about CMEPS but I think that is unrelated to my changes directly.

My CMEPS PR seems to be BFB so if Bill or you can merge it, I can update the tag here as well.

[briandobbins]

Unfortunately it's not that simple -- Bill, Bob and Mariana are speaking this afternoon about the CMEPS issues, and I'll know more after that, but I can't commit to the CMEPS PR yet.

[sjsprecious]

Unfortunately it's not that simple -- Bill, Bob and Mariana are speaking this afternoon about the CMEPS issues, and I'll know more after that, but I can't commit to the CMEPS PR yet.

I see. Just want to clarify again: my CTSM PR is independent of my CMEPS PR, so it should not affect the review or merge process here.

[briandobbins]

Right, we're not worried about the CTSM one, just the CMEPS one -- and I'll know more about that later. - Brian

…

On Thu, Jul 2, 2026 at 11:41 AM Jian Sun ***@***.***> wrote: *sjsprecious* left a comment (ESCOMP/CTSM#4102) <#4102 (comment)> Unfortunately it's not that simple -- Bill, Bob and Mariana are speaking this afternoon about the CMEPS issues, and I'll know more after that, but I can't commit to the CMEPS PR yet. I see. Just want to clarify again: my CTSM PR is independent of my CMEPS PR, so it should not affect the review or merge process here. — Reply to this email directly, view it on GitHub <#4102?email_source=notifications&email_token=ACL2HPLLE7J67YB647NFLYT5C2NC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBWHA4DENZTHE22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4868827395>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACL2HPNOGHNN6AFG5TDXP235C2NC7AVCNFSNUABFKJSXA33TNF2G64TZHMYTCNBUHA3TEMJQHNEXG43VMU5TINZUG42TANZQGYY2C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ACL2HPLELMIAPWQZZWZBMAD5C2NC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBWHA4DENZTHE22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ACL2HPLN4XZ5YDW7TZNWTYT5C2NC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBWHA4DENZTHE22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[sjsprecious]

Hi @samsrabin and @ekluzek , just wants to give you a heads up: Both my CDEPS and CMEPS PRs are now merged. The CDEPS PR is BFB but the CMEPS PR is answer-changing at roundoff level.

Since my CTSM PR does not depend on the CMEPS PR, I prefer not to include the CMEPS changes in my PR here so that my CTSM changes remain BFB as shown in the previous test. Let me know if you would like to proceed with a different option.

[ekluzek]

Hi @samsrabin and @ekluzek , just wants to give you a heads up: Both my CDEPS and CMEPS PRs are now merged. The CDEPS PR is BFB but the CMEPS PR is answer-changing at roundoff level.

Since my CTSM PR does not depend on the CMEPS PR, I prefer not to include the CMEPS changes in my PR here so that my CTSM changes remain BFB as shown in the previous test. Let me know if you would like to proceed with a different option.

Thanks so much @sjsprecious this is great. Our preference is to seperate out any answer changes from things that are expected to be bit-for-bit. And actually it might be good to remove the CDEPS submodule update here to a seperate thing.

Before we bring this in we need to see how it merges with the mpi_scan branch that was merged onto b4b-dev. So it might be best to remove the CDEPS update here, and rebase this to the b4b-dev branch. See the instructions under "1" here: https://github.com/ESCOMP/CTSM/wiki/b4b%E2%80%90dev-branch-procedures#merging-to-b4b-dev

Since you've already done work here, and I'm asking for more -- I'll create a PR on master to bring the bit-for-bit CDEPS update in, as well as a different one to bring in the answer changing CMEPS one. How does that sound?

[sjsprecious]

Thanks so much @sjsprecious this is great. Our preference is to seperate out any answer changes from things that are expected to be bit-for-bit. And actually it might be good to remove the CDEPS submodule update here to a seperate thing.

Thanks @ekluzek . It sounds good to me.

Before we bring this in we need to see how it merges with the mpi_scan branch that was merged onto b4b-dev. So it might be best to remove the CDEPS update here, and rebase this to the b4b-dev branch. See the instructions under "1" here: https://github.com/ESCOMP/CTSM/wiki/b4b%E2%80%90dev-branch-procedures#merging-to-b4b-dev

I will do it as suggested and let you know if I run into any problem later.

Since you've already done work here, and I'm asking for more -- I'll create a PR on master to bring the bit-for-bit CDEPS update in, as well as a different one to bring in the answer changing CMEPS one. How does that sound?

That sounds good to me. Thanks for your help.

[sjsprecious]

Hi @ekluzek , I have removed the CDEPS update in my PR and rebased my branch to the latest b4b-dev branch. Let me know if you have any questions about it.