Avoid overflow for volume fraction calculation - #1895
Conversation
…ractions on larger meshes.
…o bugfix/whitlock/batched_volfrac_creation_mfem
…o bugfix/whitlock/batched_volfrac_creation_mfem
There was a problem hiding this comment.
Thanks @BradWhitlock
It's a bit difficult to tell from the github review page, but it looks like the shaping_helpers_mfem no longer runs on device, where previously several parts (but not all of it) ran on device, when using a hip or cuda policy.
Can you please run this in a GPU config/policy and confirm that we're not losing functionality? The speedups were significant in some of my runs, and getting the memory transfers correct/working was challenging.
Update: I don't think any of our hip/cuda configs are currently using mfem::Device functionality. Outside of this PR, we should enable/exercise mfem on-device for our shaping problems in our hip/cuda configs.
I created this issue to track this: #1966
| * \param volfracOrder The order of the volume fraction function to create. | ||
| * \param sampleResolution The number of samples in each mesh dimension. | ||
| * \param quadratureType The quadrature type that determines the sample point locations. | ||
| */ |
There was a problem hiding this comment.
Minor: Please update the doxygen for this function
| axom::ArrayView<int> sampleResolution, | ||
| axom::numerics::QuadratureType quadratureType) | ||
| { | ||
| constexpr std::int64_t MAX_CACHED_MASS_BYTES = 1LL << 30; |
There was a problem hiding this comment.
Please add a comment to this line.
Presumably, it is sized to avoid overflow w/ int32.
Should we upstream a concern/fix to mfem about overflow w.r.t. int32 vs int64 indices?
There was a problem hiding this comment.
Perhaps it would be sufficient (or at least an improvement) for mfem to check for and give a warning about overflow?
| SLIC_WARNING_ROOT( | ||
| axom::fmt::format("MFEM volume fraction processing for '{}' uses host data and currently " | ||
| "falls back to sequential execution for device runtime policies.", | ||
| vfName)); |
There was a problem hiding this comment.
Should we warn the user about this?
Our implementation (in Axom/MFEM) is only on the CPU, so there's nothing they can do about it.
If we do want to post a warning, perhaps it would be sufficient to only warn once in a given run?
E.g. using SLIC_WARNING_ROOT_ONCE
axom/src/axom/slic/interface/slic_macros.hpp
Lines 237 to 255 in 12695a3
| applyFCTProjectionImpl<omp_exec>(massMat, rhs, dofs, numElements, vfData, scratchBuffer); | ||
| break; | ||
| #endif | ||
| default: |
There was a problem hiding this comment.
I'm reasonably sure that the FCT ran on device.
Update: Nevermind -- this never ran on device
axom/src/axom/quest/detail/shaping/shaping_helpers_mfem.cpp
Lines 694 to 699 in 12695a3
This is a follow-on for #1875 (merged into develop now).
This PR resolves #1871 by:
shaping::computeVolumeFractionsForMaterialso it works on groups of elements rather than all elements at once when it looks like the number of values will overflow int32shaping::computeVolumeFractionsForMaterialso we can parallelize for OpenMP some of the more expensive algorithm phasesShaping driver varying mesh size for spheres.yaml input with order 4 output
