I/O and memory diagnostic improvements in DistributedClosestPoint driver by kennyweiss · Pull Request #1952 · llnl/axom

kennyweiss · 2026-08-17T00:35:50Z

Summary

This PR improves the I/O capabilities for our DistributedClosestPoint (DCP) driver example
The driver now takes a distributed blueprint mesh for the object mesh and for the query mesh
- It previously used an internally generated mesh of a circle in 2D and a sphere in 3D
- Re retained the ability to analytically verify the results for input data with the appropriate verification tag
- The two inputs can be multi-domain or single domain
- We added test meshes to the axom_data repo, and updated the unit tests to use these
Also adds some memory diagnostics to the DCP driver to see how much memory is being used and the extent to which malloc_trim helps clear memory
- This led to a fix to the underlying DCP implementation to release memory after copying to MPI buffers
- This bugfix might resolve some user-reported memory issues w/ the DCP algorithm.
This PR also improves the convert_sidre_protocol.py script to convert from conduit blueprint formats, instead of just sidre formats
It also adds a new python script to generate distributed blueprint point meshes w/ following several point distributions (circle/sphere, annulus/torus, Gaussian, uniform)

Misc:

I updated our build skill to be able to build/test w/ MPI on an allocated node, and to work around issues with shroud and symlinks in sandboxed environments

kennyweiss · 2026-08-17T03:17:49Z

        auto& req = isendRequests.back();
        relay::mpi::isend_using_schema(*xferNodes[m_rank], firstRecipForMyQuery, tag, m_mpiComm, &req);
+        // isend_using_schema() deep-copies into the request buffer.
+        xferNodes.erase(m_rank);


These three xferNodes.erase() seem to save a lot of memory during computeClosestPoint in hip runs

input branch query time RSS peak RSS post-query RSS after trim

100k develop 2.16 s 3.76 GiB 3.76 GiB 3.57 GiB

100k branch 2.14 s 3.64 GiB 3.64 GiB 3.55 GiB

1M develop 22.91 s 5.97 GiB 5.38 GiB 4.33 GiB

1M branch 23.23 s 4.77 GiB 4.27 GiB 4.13 GiB

10M develop 286.76 s 27.81 GiB 9.04 GiB 9.00 GiB

10M branch 285.99 s 15.89 GiB 9.04 GiB 9.00 GiB

Notes:

In the 10M case, the peak memory went down from ~28 GB to ~16GB with this change.

malloc_trim only helped marginally in develop and even less after the changes in this file

This was in a Debug config, so don't read too much into the query times.

Nice memory reduction for larger problem sizes.

Thanks. I think it might essentially be the same fix as #1889, which hasn't merged yet.

Tag: @MrBurmark

Indeed reducing the lifespan of the transfer nodes is the essence of #1889.

@kennyweiss are you planning to collect data on the post #1948 develop?

@MrBurmark -- I rebased my branch on axom@develop after merging both of your PRs and removed my node.erase calls.

I reran the tests w/ and w/o dynamic filtering to confirm that your changes captured the memory improvements (w/ amdclang++ / rocm@7.2.1):

input dynamic filtering query time RSS peak RSS post-query RSS after trim

1M enabled 2.318 s 4.67 GiB 4.16 GiB 4.03 GiB

1M disabled 2.300 s 4.66 GiB 4.16 GiB 4.00 GiB

10M enabled 25.588 s 15.78 GiB 8.93 GiB 8.89 GiB

10M disabled 27.196 s 15.78 GiB 8.93 GiB 8.89 GiB

Note: the previous timings were in a Debug (-O0) config and these are in a DebWithRelInfo (-O2), so don't read too much into the timings changes.

Looks good, thanks @kennyweiss.

Thanks for rerunning. AFAIK, codes don't put debug builds out for users 😆

(I used a Debug config for the initial run since I was focusing on the memory rather than the timings.)

kennyweiss · 2026-08-17T16:18:53Z

Note: The failing github actions CI job (gcc@13.3.1, shared, 32 bit) is related using a 64-bit type in a 32-bit build. Should be easy to fix.

rhornung67

@kennyweiss thank you for this.

When provided, we use that instead of an analytically generated mesh.

Allow for topologies other than "mesh" and for unstructured topologies.

And for input meshes without a "fields" group.

... and remove analytical generation from DCP example.

Also adds troubleshooting sections about shroud and about symlinked paths.

The test meshes match the previous analytically generated circle/sphere and are validated.

…print protocols

* In convert_sidre_protocol, run w/ correct protocol * In new point mesh generator script, ensure MPI/conduit are available when we expect them * Fix check for multi-domain runs

kennyweiss assigned kennyweiss and publixsubfan Aug 17, 2026

kennyweiss added bug Something isn't working enhancement New feature or request Quest Issues related to Axom's 'quest' component User Request Issues related to user requests mpi Related to MPI communication Python labels Aug 17, 2026

kennyweiss commented Aug 17, 2026

View reviewed changes

kennyweiss changed the title ~~Feature/kweiss/improve dcp example input~~ I/O and memory diagnostic improvements in DistributedClosestPoint driver Aug 17, 2026

rhornung67 approved these changes Aug 17, 2026

View reviewed changes

kennyweiss force-pushed the feature/kweiss/improve-dcp-example-input branch 2 times, most recently from 964752c to 7404b58 Compare August 19, 2026 04:07

kennyweiss mentioned this pull request Aug 19, 2026

Adds some generated datasets for distributed closest point driver llnl/axom_data#41

Merged

kennyweiss added 15 commits August 24, 2026 19:57

Quest: Adds support to DCP example to read in blueprint object mesh

d6a1842

When provided, we use that instead of an analytically generated mesh.

Quest: Increase flexibilty for blueprint mesh in DCP example

bdd5436

Allow for topologies other than "mesh" and for unstructured topologies.

Quest: Adds some memory diagnostics to DCP example

1c8f7a3

Quest: Adds support to DCP example for single-domain bp meshes

2013504

And for input meshes without a "fields" group.

Adds Python script to generate blueprint Point meshes for DCP query

0d49848

Quest: Improves python script to generate DCP inputs

8bc5d40

... and remove analytical generation from DCP example.

Adds back verification for analytic test cases in DCP example

aa0908c

Updates the build skill about running MPI commands

ad37fcf

Also adds troubleshooting sections about shroud and about symlinked paths.

Quest: DCP example runs on object meshes in test repository

81c338e

The test meshes match the previous analytically generated circle/sphere and are validated.

Extents convert_sidre_protocol script to support reading conduit blue…

575ab5f

…print protocols

Adds support for truncating data using --strip

47174c5

Quest: Refactors and improves memory diagnostics

26b92bb

Quest: Adds allReduce helper to dcp driver

8ad04aa

Minor fixups

2b03546

* In convert_sidre_protocol, run w/ correct protocol * In new point mesh generator script, ensure MPI/conduit are available when we expect them * Fix check for multi-domain runs

Bugfix for configs that do not have a 64-bit int type

764538b

Updates data submodule to include dcp point meshes

f97c476

kennyweiss force-pushed the feature/kweiss/improve-dcp-example-input branch from 14519a2 to f97c476 Compare August 25, 2026 03:36

Navigation Menu

Uh oh!

I/O and memory diagnostic improvements in DistributedClosestPoint driver - #1952

I/O and memory diagnostic improvements in DistributedClosestPoint driver#1952
kennyweiss wants to merge 16 commits into
developfrom
feature/kweiss/improve-dcp-example-input

kennyweiss commented Aug 17, 2026 •

edited

Loading

Uh oh!

kennyweiss Aug 17, 2026 •

edited

Loading

Uh oh!

rhornung67 Aug 17, 2026

Uh oh!

kennyweiss Aug 17, 2026

Uh oh!

MrBurmark Aug 17, 2026

Uh oh!

MrBurmark Aug 18, 2026

Uh oh!

kennyweiss Aug 18, 2026 •

edited

Loading

Uh oh!

MrBurmark Aug 19, 2026

Uh oh!

rhornung67 Aug 19, 2026

Uh oh!

kennyweiss Aug 19, 2026

Uh oh!

kennyweiss commented Aug 17, 2026

Uh oh!

rhornung67 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

input	branch	query time	RSS peak	RSS post-query	RSS after trim
`100k`	develop	2.16 s	3.76 GiB	3.76 GiB	3.57 GiB
`100k`	branch	2.14 s	3.64 GiB	3.64 GiB	3.55 GiB
`1M`	develop	22.91 s	5.97 GiB	5.38 GiB	4.33 GiB
`1M`	branch	23.23 s	4.77 GiB	4.27 GiB	4.13 GiB
`10M`	develop	286.76 s	27.81 GiB	9.04 GiB	9.00 GiB
`10M`	branch	285.99 s	15.89 GiB	9.04 GiB	9.00 GiB

input	dynamic filtering	query time	RSS peak	RSS post-query	RSS after trim
`1M`	enabled	2.318 s	4.67 GiB	4.16 GiB	4.03 GiB
`1M`	disabled	2.300 s	4.66 GiB	4.16 GiB	4.00 GiB
`10M`	enabled	25.588 s	15.78 GiB	8.93 GiB	8.89 GiB
`10M`	disabled	27.196 s	15.78 GiB	8.93 GiB	8.89 GiB

Uh oh!

Conversation

kennyweiss commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Misc:

Uh oh!

kennyweiss Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhornung67 Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

kennyweiss Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

MrBurmark Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

MrBurmark Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

kennyweiss Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MrBurmark Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

rhornung67 Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

kennyweiss Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

kennyweiss commented Aug 17, 2026

Uh oh!

rhornung67 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kennyweiss commented Aug 17, 2026 •

edited

Loading

kennyweiss Aug 17, 2026 •

edited

Loading

kennyweiss Aug 18, 2026 •

edited

Loading