GPRT integration

[Waqar-ukaea]

This will be a large PR which will likely consist of many smaller PRs in the hopes of bringing the General Purpose Raytracing Toolkit (GPRT) into XDG to enable GPU the capability to work with GPUs. The idea will be that GPRT could be specified as an optional to build the XDG library with at compile time. This PR will aim to close out a number of existing issues and will probably inform a bunch of new ones too.

TODO

• GPU Ray Tracing Interface #65
• Array-based ray query interface #72
• GPU buffer design #74
• Create a GPRT based particle-sim tool #105
• Write some tests for GPRT integration
• Figure out a way to better handle exporting GPRT target installs for XDG and downstream applications https://github.com/pshriwise/xdg/pull/94#issuecomment-2796476896
• Double precision meshes with double precision ray tracing
• Handling non-triangular elements (quads)
• Higher order elements?
• Single precision meshes
• Think about how the mesh will actually interact with the GPRT context
• Setup the CI to bring in GPRT as a dependency when building XDG
• Make GPRT an optional dependency

I'm sure there are more tasks which I haven't captured yet and will have to have a think about which ones map to their own PRs/issues.

14/05/2025

• Started working on the XDG-GPRT interface in a new class GPRTRayTracer which inherits our generic ray tracer interface base class.
• Testing this in a new tool gprt-particle-sim which is essentially the particle-sim tool but making use of GPRT instead of Embree for ray tracing operations. If this becomes a fully featured tool it can probably be merged into the other particle-sim tool with the options to choose ray tracing back-end properly exposed.

30/04/2025

• Improved gprt-test tool to ray trace against a h5m mesh in single precision.

07/04/2025

• CI building XDG and GPRT and is passing.
• GPRT is now an optional dependency in CMake.

31/03/2025

• First 5 commits have got GPRT working as a dependency via the inclusion of a git submodule.
• I have made a simple test program as an XDG tool called gprt-test. This program creates an XDG instance from a h5m mesh passed as a command line argument at runtime and then goes on to create a GPRT context where the S_2_0_TriangleHits sample program from GPRT is loaded.
• This is not anything new and is literally the sample program copied into this XDG tool but it does show that the correct CMake commands are in place to bring in GPRT as a dependency.

[Waqar-ukaea]

@pshriwise Thanks for the help getting started with the CI! I have added an extra job to the CI workflow to explicitly set some vulkan-sdk environment variables to fix the issue where the workflow was failing when attempting to compile spirv code. I also switched back to ubuntu-latest for the container image for the runner since it was having issues with mismatched glibc references between MOAB/libmesh and XDG.

The CI workflow is successfully building XDG with GPRT and passing now but it looks like the OpenMC-Testing workflow is still failing.

[pshriwise]

The CI workflow is successfully building XDG with GPRT and passing now but it looks like the OpenMC-Testing workflow is still failing.

Ah yeah I think I was making similar changes for the OpenMC workflow to add the VULKAN_SDK and GPRT, but at this point I don't think that makes sense. I've reverted those changes. Once we're testing GPRT with OpenMC transport we can add those changes back in. At that point it probably makes sense to combine the OpenMC testing into the CI workflow as they're doing pretty much the same thing at that point.

[pshriwise]

Ah, as you mentioned though GPRT is a requirement at the moment. Can you make it optional with an ENABLE_GPRT option in the CMake files so we don't have to maintain two temperamental workflows in the PR checks?

[Waqar-ukaea]

Can you make it optional with an ENABLE_GPRT option in the CMake files so we don't have to maintain two temperamental workflows in the PR checks?

GPRT should now be disabled by default.

I can see a scenario in which we may have multiple GPRT based tools in which case it makes sense for them to potentially be separate? The latest commit restructures the xdg/tools directory to do so and adds a new CMakeLists.txt specifically for generating gprt tools:

tools/
├── CMakeLists.txt # CMakeLists.txt for the rest of the tools and optionally adds subdirectory(gprt)
├── find_volume.cpp
├── gprt
│   ├── CMakeLists.txt # CMakeLists.txt to create gprt tools and embed device code for each
│   ├── deviceCode.slang
│   ├── gprt_test.cpp
│   └── sharedCode.h
├── overlap_check.cpp
├── particle_sim.cpp
├── particle_sim.h
├── point_in_volume.cpp
└── ray_fire.cpp

If in the future we did have more tools though we would probably want another directory layer to separate out the shader code for each tool though. Something like this:

tools/
├── CMakeLists.txt
├── gprt
│   ├── CMakeLists.txt
│   ├── tool-1
│   │   ├── deviceCode.slang
│   │   ├── gprt_test.cpp
│   │   └── sharedCode.h
│   └── tool-2
│       ├── deviceCode.slang
│       ├── gprt_test.cpp
│       └── sharedCode.h

[Waqar-ukaea]

When attempting to link gprt to xdg in the CMakeLists.txt, I was encountering a CMake error to do with GPRT not exporting install targets:

CMake Error: install(EXPORT "xdg-targets" ...) includes target "xdg" which requires target "gprt" that is not in any export set.
-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

This error would be thrown but CMake would still generate the Makefile and XDG could be compiled as normal with tests passing and installation still seemingly working.

As a work around for this CMake error I have changed the following line in the CMakeLists.txt:
target_link_libraries(xdg gprt)
to:
target_link_libraries(xdg $<BUILD_INTERFACE:gprt>)
This ensures linking only happens at build time and not installation time. This is more of a workaround than a real fix and I am unsure if we will need GPRT to be linked at installation time as well as build time.

[Waqar-ukaea]

Made a PR #100 to expose methods in the mesh manager to return a flattened array of a surface's vertices and triangle/element connectivity - this should help in the efforts to triangle data to device in GPRT.

[Waqar-ukaea]

Merged a bunch of changes I had on a separate branch in my fork. This is mostly me just experimenting with trying to get single precision h5m meshes read into GPRT. At this point looks like its working.

Changes with this merge:

• New methods in MeshManager to return surface vertices and connectivity.
• Updated gprt-test tool to make use of these methods and use the mesh to populate some AABBs to ray trace against.
• Single precision only at the moment.

More details of the progress made has been documented in this issue Waqar-ukaea#3

[Waqar-ukaea]

Description copied below from a PR into this branch from my fork Waqar-ukaea#5:

Quite a lot of new stuff in here and all of it is still super experimental. I've been playing around with what the internal XDG-GPRT interface will look like with the GPRTRayTracer class and have been using that to set up a new XDG tool that builds on the particle sim tool to use this GPRTRayTracer instead of Embree.

There is still a lot of placeholder at the moment and random comments here and I need to verify that the device is actually doing what I think it is supposed to be doing within the raygen, closesthit and miss shaders because right now ray_fire() is just returning {-1, -1} on the host. But I am able to call XDG->GPRTRayTracer->ray_fire() within a new gprt-particle-sim tool and it doesn't seg fault so I see that as a win :)

Edit 2025/05/15 - The raygen shader is working as it should with origins/directions correctly passed to it. The problem is that every ray is missing when there should be hits - so only the miss shader is being run (hence why ray_fire() returns the default values for a miss back on the host.

All of this work essentially builds on from what I learnt producing my first simple gprt-test tool with XDG. That tool has all of the GPRT calls contained within its main() function whereas this new gprt-particle-sim tool intends to abstract all of those GPRT related calls into the GPRTRayTracer class.

Some extra notes:

• All of this GPRT abstraction is with the intention of running in single precision ONLY. Once I am happy with a single precision implementation then I can think about how I might extend this to support both double and single precision ray tracing.
• Right now I am creating a BLAS for each surface and a TLAS for each volume containing all of those BLAS structures. There is a function to create a "world" TLAS but it is not currently used for anything.
• There are currently two .slang files containing shaders titled flt_deviceCode.slang and dbl_deviceCode.slang. As of right now I have only really been working on the flt_deviceCode.slang but all of the necessary CMake setup is in place to make use of both slang files for embedding device code.
• I'm not really sure where my shared headers need to be so that they can be used by both host code and device code. As a temporary measure I have just copied the sharedCode.h to sit both in include/xdg/gprt for host code and src/gprt for device code. I'm not sure why a single location isn't working for both host and device.

[Waqar-ukaea]

I've marked this PR as ready for review since I now have GPRTRayTracer somewhat working with a mixed precision method (similar to the EmbreeRayTracer). I.e single precision ray tracing over our acceleration structures of bounding boxes and a final double precision intersection test on potential hits using the Plucker-Triangle intersection tests.

Current status:

• GPRTRayTracer::ray-fire() seems to run okay when using the independent ray-fire tool but I think there are some issues with sense/orientation causing some real hits to be culled by the intersection shader.
• GPRTRayTracer::point_in_volume() seems to be returning true every time. I'm not certain what the problem is here. Potentially an issue of the shader returning the wrong primitives normal?
• No other ray queries have been implemented as of yet.
• GPRTRayTracer::register_volume() does most of the heavy lifting when it comes to setting up geometry/acceleration structures for GPRT. I think this method is mostly fine but I may need to revisit some aspects of it.
• I may need to rebase xdg main into this branch again too.
• Lots of comments/temporary debugging things dotted about which need some cleaning up too.

There are probably more things to note and we may want to address some more of these issues before merging into main. But that being said, this branch is in a pretty workable/stable state now.

[Waqar-ukaea]

A note about passing data between shader stages. I had a lot of trouble with silent hangs when trying to transfer data between raygen and intersection. I'm not sure if I am missing something but this is my understanding from a lot of annoying troubleshooting:

• Custom intersection shaders cannot access payloads or push constants (as far as I can tell)
• Once we are inside the shaders DPMeshData is read only.

Previously in the h5m reader we had explicit gprt::store blocks:

GPRT_RAYGEN_PROGRAM(DPRayGen, (RayGenData, record))
{
...
  // store double precision ray
  gprt::store(record.dpRays, fbOfs * 2 + 0, double4(rayDesc.Origin.x, rayDesc.Origin.y, rayDesc.Origin.z, rayDesc.TMin));
  gprt::store(record.dpRays, fbOfs * 2 + 1, double4(rayDesc.Direction.x, rayDesc.Direction.y, rayDesc.Direction.z, rayDesc.TMax));
...

But I have not been able to recreate this functionality with GPRT anymore. The main issue seems to be that the geometry data is seen as read only once the shader pipeline is started, so any attempts to write to the data causes silent hangs. Instead I have opted to define a buffer for double precision rays on the host, where both RayGenData and DPMeshData point towards the same buffer. That way the buffer can be populated inside GPRTRayTracer::ray_fire() before launching the raygen shader via gprtBufferMap regions:

std::pair<double, MeshID> GPRTRayTracer::ray_fire(TreeID scene,
                                                  const Position& origin,
                                                  const Direction& direction,
                                                  double dist_limit,
                                                  HitOrientation orientation,
                                                  std::vector<MeshID>* const exclude_primitives) 
{
  GPRTAccel volume = tree_to_vol_accel_map.at(scene);
  dblRayGenData* rayGenData = gprtRayGenGetParameters(rayGenProgram_);
  rayGenData->world = gprtAccelGetDeviceAddress(volume);
  rayGenData->orientation = static_cast<int>(orientation); // Set orientation for the ray
  
  gprtBufferMap(rayInputBuffer_); // Update the ray input buffer

  dblRayInput* rayInput = gprtBufferGetHostPointer(rayInputBuffer_);
  rayInput[0].origin = {origin.x, origin.y, origin.z};
  rayInput[0].direction = {direction.x, direction.y, direction.z};
  rayInput[0].tMax = dist_limit;
  rayInput[0].tMin = 0.0;
...
  
  // Launch the ray generation shader with push constants and buffer bindings
  gprtRayGenLaunch1D(context_, rayGenProgram_, 1, pc);
...

This could be pretty easily extended to a function which loops over an array of origins/directions where the buffer rayInput has a size equal to the number of rays we wish to launch and gprtRayGenLaunch1D would also be launched with that number of rays rather than 1.

[Waqar-ukaea]

Latest commit has some debugging prints in place to compare embree vs gprt ray-fires

I have got the GPRT branch to a state where all of the ray-fire tests (on our simple mesh mock) are passing. However there are still problems when trying to fire certain rays against more complex meshes. When trying to run the particle sim on the jezebel.h5m mesh I see a ray which should return a hit returning no intersections. A table summarising the debug outputs I've produced between the two RT libraries for the following ray-fires:

./ray-fire -r GPRT -o 1.3646307217226612 1 -1 -d -0.7778294351116771 0.524847167397148 -0.7778294351116771 ../../tests/test_files/jezebel.h5m 1

./ray-fire -r EMBREE -o 1.3646307217226612 1 -1 -d -0.7778294351116771 0.524847167397148 -0.7778294351116771 ../../tests/test_files/jezebel.h5m 1

Feature	GPRT	Embree
Triangle 1 Vertices	v0 = (-3.025391, 2.503475, -5.034538) v1 = (-3.775899, 2.508778, -4.496172) v2 = (-3.161024, 3.290301, -4.466408)	v0 = (-3.025391, 2.503475, -5.034538) v1 = (-3.161024, 3.290301, -4.466408) v2 = (-2.413324, 3.280428, -4.917480)
Triangle 1 Plücker Coords	plucker_coord0 = -0.262458 plucker_coord1 = 0.564534	plucker_coord0 = 0.0799639 plucker_coord1 = -0.199693
Triangle 1 Result	Rejected: early exit due to differing signs	Rejected: early exit due to differing signs
Triangle 2 Vertices	v0 = (-3.025391, 2.503475, -5.034538) v1 = (-3.161024, 3.290301, -4.466408) v2 = (-2.413324, 3.280428, -4.917480)	v0 = (-3.025391, 2.503475, -5.034538) v1 = (-3.775899, 2.508778, -4.496172) v2 = (-3.161024, 3.290301, -4.466408)
Triangle 2 Plücker Coords	plucker_coord0 = 1.099058 plucker_coord1 = -0.473359	plucker_coord0 = -0.583442 plucker_coord1 = -0.133577 plucker_coord2 = -0.0799639 inverse_sum = -1.25473
Triangle 2 Result	Rejected: early exit due to differing signs	Accepted: intersection found
Intersection Point	—	(-3.19998, 3.08001, -4.56461)
Intersection Distance	1.7976931348623157e+308 (no hit)	7.15247
Surface ID	-1	1

So the vertices' coords match between the two RT implementations but the order of them is different between the two implementations?

A further update on this - apparently moving the ray off of the origin has drastic effects on the results. A ray launched in the same direction as the one above but from the origin produces identical results between embree and gprt:

Feature	GPRT	Embree
Ray Origin	(0, 0, 0)	(0, 0, 0)
Ray Direction	(-0.638187, 0.430622, -0.638187)	(-0.638187, 0.430622, -0.638187)
Triangle Vertices	v0 = (-4.433076, 2.509587, -3.849254)	Same as GPRT
	v1 = (-4.387141, 3.294091, -3.266330)
	v2 = (-3.823750, 3.293800, -3.911108)
Plücker Coordinates	plucker0 = -0.468912	Same as GPRT
	plucker1 = -0.529105
	plucker2 = 0.239059
Winding Check Result	Rejected intersection on some triangles but accepted intersection at primID 342, t=6.360884, u=0.410106, v=0.293020	Accepted intersection at distance 6.36088
Intersection Distance	6.3608844288446438	6.3608844288446438
Surface ID	1	1

But as soon as we move off of the origin they no longer match:

Aspect	GPRT (Origin = 0.5, 0, 0)	Embree (Origin = 0.5, 0, 0)
Ray Origin	(0.5, 0, 0)	(0.5, 0, 0)
Ray Direction	(-0.638187, 0.430622, -0.638187)	(-0.638187, 0.430622, -0.638187)
Triangle Vertices	v0 = (-3.775899, 2.508778, -4.496172)	Same as GPRT
	v1 = (-3.823750, 3.293800, -3.911108)
	v2 = (-3.161024, 3.290301, -4.466408)
Plücker Coordinates	0.330239, -0.530037, (Rejected intersection: winding check fail between edges 0 and 1)	-0.0462276, -0.409358, -0.308746
Intersection Status	Rejected intersection on one triangle edge, but accepted another at t=7.144354, distance=7.144354	Accepted intersection, distance=6.672090
Final Intersection Distance	7.1443541608758006	6.6720905158066888
Surface ID	1	1

What could be causing this?

[Waqar-ukaea]

@pshriwise I think this is in a position where its ready to review. I've rebased all the changes from main and fixed any conflicts which arose because of them. All tests are currently passing except find_element tests with MOAB. I'll dig into that a little bit more. But all of the stuff around GPRT is ready to be reviewed. Let me know if you have any trouble with getting things to build etc. I have been changing some file structures and CMake configs so have perhaps missed something which should have been comitted.

One thing to note as well is that I still never managed to resolve this https://github.com/pshriwise/xdg/pull/94#issuecomment-2796476896 so I will create a separate issue for it.

[Waqar-ukaea]

All tests are currently passing except find_element tests with MOAB

This issue has been resolved with commit pshriwise@9881e08 which updates the test_files subproject commit to check out. Presumably I was still on a version which did not have volumetric elements in the models.

[pshriwise]

I understand that this isn't straightfoward to figure out, but these days we really shouldn't need to set this environment variable for a CMake project. For now I'll create an issue so we don't lose track of it.

[Waqar-ukaea]

I previously assumed this wasn't too much of an issue, more so from a place of ignorance in not really being familiar with RPATHs. But after having done some reading, I agree we should hold off on merging until I figure this one out too.

[pshriwise]

@Waqar-ukaea Thanks so much for all your effort in this PR. It's really cool to see DAGMC geometry running on GPU, even as the first cut. Plenty of room to grow from here!