Add support for Kokkos v5, with multi-backend device support.

[mpokorny]

Kokkos_Array.hpp is included in inst_layout.h to get the correct template parameters in the Kokkos::Array declaration. It's not quite as simple as declaring the two-parameter version because the correct choice depends on whether or not v4 deprecated code is enabled in Kokkos.
I tested the build with both Kokkos v4.7 and v5.0, but there are not unit tests in Realm to verify the result. I did, however, build Legion with this branch against both Kokkos versions, and verified the results using kokkos_saxpy.

[rbberger]

@lightsighter the main issues I'm having building with Legion/Realm/Kokkos 5 all are now about CUDA 12.9/13.3, GCC 14+, LLVM 20/21/22+ and C++20. I have yet to find the magic combo that works.

[rbberger]

@lightsighter @mpokorny I've looked at this closer. The issues I'm seeing come from the inclusion of Kokkos_Array.hpp, which messes up the Host compilations. I'll push a branch with changes that should be more robust.

[rbberger]

@lightsighter See #462 for my additional patch. It requires CMake changes to avoid dealing with the Kokkos headers directly.

[mpokorny]

It just so happens that I've been working on this this week, too. I will try to update this branch today.

[mpokorny]

I've updated this branch with the latest patch that I've been developing and using. I've included the multi-backend device capability because it's useful, I'm getting tired of maintaining it separately, and I'm feeling lazy. However, if people object, I will separate out those changes.

[rbberger]

@mpokorny in case you looked at my commit, look again please. My late night commit was missing the important part 😁

[rbberger]

I'll test out this updated branch. I'll prefer this simpler solution if we can avoid the Kokkos::Array parts entirely.

[rbberger]

I can confirm that this patch set works for FleCSI and our LLVM-based builds.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 30.41%. Comparing base (7d11051) to head (506abba).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/realm/openmp/openmp_module.cc	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #418      +/-   ##
==========================================
- Coverage   30.41%   30.41%   -0.01%     
==========================================
  Files         199      199              
  Lines       41281    41282       +1     
  Branches    14942    14758     -184     
==========================================
  Hits        12554    12554              
+ Misses      28272    27279     -993     
- Partials      455     1449     +994     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[lightsighter]

I've included the multi-backend device capability because it's useful, I'm getting tired of maintaining it separately, and I'm feeling lazy. However, if people object, I will separate out those changes.

I think these changes look straight-forward enough that we can evaluate them all in the same pull request. They also related since you couldn't do some of these things until Kokkos 5 anyway right?

[mpokorny]

That's correct. I'm still actively testing this at the moment. I'll report my findings and/or update the PR as needed, and provide an indication when I think all is OK.

[mpokorny]

This branch is passing all my tests for the radio camera imaging pipeline application (plus its unit tests) with multiple processes and >=1 GPU per process.

[rbberger]

@lightsighter this still needs to get merged in for us.

[lightsighter]

@mpokorny Are you ready to merge this? at a minimum you need to run the formatting command from the Realm root directory which should resolve the formatting issue in the CI:

git diff -U0 --no-color main | clang-format-diff -p1 -i

If you can't get it to work then I can do it for you.

[mpokorny]

I applied the cmake-format-diff command. The branch should be ready to merge.

[elliottslaughter]

It looks like the format is still unhappy. Maybe you're on a different clang-format version? Looks like we use LLVM 15.

Alternatively you could use the diff printed in the CI job: https://github.com/StanfordLegion/realm/actions/runs/32170123110/job/95819562500?pr=418

[elliottslaughter]

@mpokorny Is there a Legion-side change that needs to go along with this when we update Realm there?

[mpokorny]

Yes, examples/kokkos_saxpy/kokkos_saxpy.cc needs a few changes. Should I create an MR on the Legion repo for it?

[elliottslaughter]

That or send a patch. I'll probably want to incorporate it into the Realm update MR so that it lands together.

[mpokorny]

Does this work?
lg-kmp5.patch

[elliottslaughter]

MR here: https://gitlab.com/StanfordLegion/legion/-/merge_requests/2089

[elliottslaughter]

This has now been merged into Legion along with the example patch. Thanks @mpokorny!