cuBQL backend

[Waqar-ukaea]

Internal PR for me to track progress on this since it'll be quite an in depth piece of work.

cuBQL is the BVH construction and traversal engine that DPRT is built around. it's designed to be highly flexible and seems to better suit our needs in XDG than DPRT does. Most importantly it has a working OpenMP target offloading backend.

I have made a start on the cuBQL backend here. The nature of the library being quite low level means it requires a little bit more plumbing to get the same functionality working as the Embree and GPRT back ends. Nevertheless, in my prototyping here I think it is definitely doable.

Progress so far

• Successfully compiling cuBQL using llvm/clang-22 with openmp offloading support. This is a huge win for interoperability. GPRT always had a massive question mark above it's head in terms of how we would communicate between OpenMC's OpenMP offload regions and GPRT's vulkan instance. On top of the, I was only able to get DPRT compiling with NVHPC compilers rather than the more vendor agnostic llvm ones.
• Constructing BVH per surface on device via OpenMP target offloading. The BVHs are stored in a vector for later use in the queries. Right now there is no TLAS and these BVHs functionally are BLASs.
• Prototyped an implementation for querying against the BVHs using codex. This is absolutely NOT a final implementation by any stretch of the imagination. But it does serve as a proof of concept.
• In this proof of concept we query against all BVHs in the model and take the closest hit found. This means no TLAS culling of BVHs we don't care about. We explicitly traverse every single one.
• ALL ray-fire tests pass. This is hugely important. With DPRT, handling entering Vs exiting rays was becoming very difficult. But the low level flexible nature of cuBQL has allowed me to handle this pretty easily and in a manner similar to what I do with GPRT.
• Exclude primitives seems pretty simple to implement too.
• Updating the ray-fire tool to also use the cuBQL backend has seen cuBQL producing ray-fire results that match Embree in all cases I've tested (some very minor precision differences in some cases e-15 though). Previously I couldn't get the DPRT implementation to match Embree for different volumes (appeared to be an issue of not properly handling ray orientation).
• cuBQL allows us to define an intersection lambda which is run for every candidate primitive which means we can reuse our plucker ray-triangle intersection - this was impossible with DPRT.

Whats next?

• The build system is a little fragile relying on LD_LIBRARY_PATH pointing to libomptarget.so. Ideally we move to a more robust RPATH setup here.
• Right now all of the traversal logic is prototype code produced by Codex. This needs to be tightened up and cleaned up with an eye for how best to scale out in the future.
• A two level TLAS-BLAS scheme would be nice to employ.
• cuBQL has other traversal templates that will map naturally onto some of the other types of query we want to do with XDG - i.e closest via kNN.
• Currently only played with single ray launches. Need to batch based queries in place alongside a ray-benchmark type of miniapp to test performance.
• Explore whether or not we could use wide BVHs with the OpenMP builder

[Waqar-ukaea]

The latest set of changes have made a start on cleaning up the protyped codex code - namely by removing the duplicated plucker intersection functions for cuBQL. Now we directly use the existing plucker intersection code that the other two ray tracers make use of. Tests are still passing so this looks to be a good change which cuts out around 15 lines of prototype code 🥳

There is an inherent problem with the way I am currently linking libomptarget in this branch. Every test links libxdg.so and libxdg.so is built with the OpenMP target flags so even a test which has nothing to do with cuBQL or any openmp offloading regions gets hit with a fixed initialisation time cost. This pushes up the total time of completing the tests but does not change or affect the results of any. Of course the only tests which actually need the libomptarget offload setup are the ones which make use of the CuBQLRayTracer.

[Waqar-ukaea]

Some notes on BVH traversal from Ingo:

My questions to Ingo:

Hi! Sorry it's been so long. Genesis...

We're working on integrating with CubQL now and so far so good! It's great! Like, exactly what we need. Lambas are wonderful.

We may have API questions down the line but I'll let you know. Primarily right now we're wondering if it might be possible to make the BVH single precision with double precision primitives?

Oh, the other question we had was about best practices for TLAS/BLAS construction in CubQl, seems like a BVH is a BVH and we'll handle that ourselves. Sound about right?[3:16 PM]

His response:

yes you can mix and match: a float bvh is built and traversed only over (boxes of) floats, so to build it you have to give it an array of float boxes - but it's you that generates this array, so if you have double prims you just have to "round out" the doubel boxes to float boxes (there's some round-up round-down stuff somewhere in there already, i've used it before - it may be incomplete btu we can fix that); but once you do that the bvh or traversal doesn't care a fig what you do in your lambdas. so the lambda itself would operate on doubles, just the boxes for the build would have to rounded out to float, and similartly, the return value from the lambda - if it's one that needs a return value - would have to be propertly roudned out. So if you had a double triangle intersection the float traversal would still expect a flaot back from the intersect lambda, but if you round-up that distance all should be good. luckily cuda has very good and fast double2float_ru/rd() or something like that.
re bvh over bvh: again the bvh itself doesn't care what's in a lamba, so "one" way of doing a tlas/blas thing is to really have the tlas use a lambda htat itself calls traverse on a second bvh (the blas). That may not be optimal, though, so for ray traversal i've already added a second traversal lambda that can more directly traverse two bvhes. the way that works is that the traversl knows that there can be ray transforms and another level in the top-level bvh, and calls a separate lambda to do just that. So basically this two-level traversal has two lambdas that the user is asked to provide: one to do the intersection on the blas, and one to do ray transform (and returning which child bvh to traverse) in the tlas. we can/should extend that to other queries, too, i just didn't have any need for it, yet, so only added and tested it for ray traversal.
[3:16 PM]until we add those more specific two-level traversals that might be missing you can always use the 'old' route of a lambda that traverses its own bvh. it's not perfect, but it does the job.(edited)[3:16 PM]Either way re a BVH is a BVH and we'll handle that ourselves that is mostly correct - for the building specificslly a bvh is a bvh, it doesn't matter which level you want to use it in, or whether what you put in is a bounding box of a triangle or the bbox of a (world-transformed) blas. One suggestion though: for the builder you can specify a max leaf threshold; not sure what the default is, but for a TLAS you'll almost certainly want to set that to 1 in the buidl process. For cheaper prims (like points in a kNN query) it's often much faster to have a shallower bvh with multiple prims per leaf - but if "prim" mean "do a full bvh traversal" then you'll amost certainly want only 1 "prim" per leaf 🙂

[Waqar-ukaea]

Handling sense

Sense is an important part of XDG's traversal algorithm allowing traversal to be topology aware. We do so by adjusting normals of primitives on the fly based on a ray's inherent orientation it is launched with along with the volume it is being queried against.

XDG has two distinct types that can refer to a volume:

• MeshID -> The global ID of every geometry entity tracked by XDG
• TreeID -> A unique ID used to track which acceleration structures map to which volumes. A TreeID can also be seen as mapping to a BLAS

Currently XDG doesn't expose TreeIDs to the public API. Rather the public API for any query expects to be given a MeshID corresponding to a volume and internally maps to the underlying TreeID for that volume. This works great with Embree but for the GPU ray tracers it leads to some potentially unnecessary obfuscation. In the intersection shader we need to flip a normal based on the sense relative to the volume being queried against. But this means we also need to still pass the volume MeshID of the volume a ray is being queried against to determine the sense at intersection time.

Primitive normal handling has now been moved directly into the primitive intersection lambda, along with sense handling for normal flipping.

[Waqar-ukaea]

A new omp_device probe has been added to mirror the same logic for GPRT. This means that tests gracefully skip when no GPU device is detected for both omp (cuBQL) and vulkan (GPRT), though this change won't actually be noticeable in CI since we dont have gpu runners and trying to compile an omp_target_offload based code on a system with no offload support seems quite difficult.

[Waqar-ukaea]

Working towards realising a two-level traversal scheme. I have a decent idea of what this should look like in my head and am slowly chipping away at this.

Right now the code is looking more overly complicated to achieve the same structure we already had before of looping over all surfaces. I am introducing some new types:

• CuBQLSurfaceMesh -> Stores surface data (topological metadata and geometry data)
• CuBQLSurfaceBLAS -> Stores an array of CuBQLSurfaceMesh objects as well as an array of local (to the BLAS) primitiveIDs and meshids and the handle of the actual constructed BVH.
• CuBQLSurfaceTLAS -> This has not been implemented yet. This will be a struct which contains an array of BLASs belonging to the TLAS

Some consideration with the current structure I am implementing which follows more directly what DPRT does:

  // BLAS-local primitive reference. This is needed when one BLAS contains
  // triangles from multiple surface meshes: meshID indexes DD::meshes and
  // primID indexes the triangle within that selected surface mesh. meshID is
  // not an xdg MeshID.
  // For example this allows us to group multiple surfaces into a single BLAS
  // TODO - This is what DPRT implements but Iquestion whether or not its required 
  // for XDG. Maybe a more natural alignment would be surface-> BLAS, TLAS-> volume
  struct PrimRef {
    int meshID {0};
    int primID {0};
  };

Once I have CuBQLSurfaceTLAS in place I can start changing the logic in CuBQLRayTracer::ray_fire() to instead use the ShrinkingRayQuery::twoLevel traversal. Which will require defining a new lambda to properly traverse the TLAS into the BLAS.

Will need to be careful to remember Ingo's suggestions:

Either way re a BVH is a BVH and we'll handle that ourselves that is mostly correct - for the building specificslly a bvh is a bvh, it doesn't matter which level you want to use it in, or whether what you put in is a bounding box of a triangle or the bbox of a (world-transformed) blas. One suggestion though: for the builder you can specify a max leaf threshold; not sure what the default is, but for a TLAS you'll almost certainly want to set that to 1 in the buidl process. For cheaper prims (like points in a kNN query) it's often much faster to have a shallower bvh with multiple prims per leaf - but if "prim" mean "do a full bvh traversal" then you'll amost certainly want only 1 "prim" per leaf 🙂

[Waqar-ukaea]

We are now constructing a two-level traversal scheme and successfully ray tracing against it :)

Things are messy and extremely explicit everywhere... but it works as expected with all tests passing including a ray tracer cross test with 1000 queries with cuBQL agreeing with embree. I am slightly concerned about performance but we can tackle that later I suppose.

I opted to go for a structure with three types (like DPRT) with some simplifications since we don't need certain features:

• CuBQLSurfaceMesh owns the device buffers for one topological surface: vertices, triangle indices, and primitive-to-face references.
• CuBQLSurfaceBLAS owns the bottom-level cuBQL BVH built over one CuBQLSurfaceMesh. In DPRT this level would be called a TriangleGroup and allow for multiple surface meshes to be attached to it. I have opted for a single mesh for now since we haven't typically needed the ability for multiple.
• CuBQLVolumeTLAS owns the top-level cuBQL BVH for one volume, plus the surface BLAS instances that bound that volume.

Each owner has a nested DD type, following the DPRT/cuBQL “device data” pattern. The host-side structs own memory and lifetime; the DD structs are compact, device-copyable views used inside OpenMP target traversal.

create_surface_tree() now builds one BLAS per surface, builds a TLAS over the surface bounding boxes, and maps the XDG TreeID directly to the resulting CuBQLVolumeTLAS, mirroring Embree’s TreeID -> scene mapping. It also records per-volume surface sense in SurfaceInstanceDD::reverse_sense, so the same surface can be used by adjacent volumes with opposite orientation.

ray_fire() now uses cuBQL’s two-level traversal path. The TLAS is selected from the XDG tree id, enter_blas maps TLAS instance ids to surface BLASs, and primitive intersection uses the active surface instance’s mesh data. This removes the old host-side loop over surface BLASs and lets cuBQL handle TLAS-to-BLAS traversal.

I think this has been a really interesting problem to solve and has shown that the added verbosity of working directly with the underlying BVH builder and query engine instead of the front facing ray tracer has allowed for a much much more tailor made solution for XDG. Instead of having to constantly fit XDG around the ray tracer (like embree or especially gprt), I have significantly more control over how I want to define parts of the ray tracer around XDG.

[Waqar-ukaea]

Implemented CuBQLRayTracer::point_in_volume(). This functionally behaves identically to CuBQLRayTracer::ray_fire() likewise, calling the same intersection kernel etc. Tests have been extended to now use the cuBQL backend, along with the cross-check test over the pincell also being extended to do a point_in_volume check at various boundary points.

I have also done a little bit of refactoring adding in a new cuBQL\intersection.h and associated implementation file which is where the CuBQLSurfaceHit struct now lives and also a helper for orientation culling during intersection. Any abstracting I do of the shared plucker-tri intersection kernel will probably have its components live in intersection.cpp.