Skip to content

Adding a backend for cuBQL as a RayTracer - #245

Open
Waqar-ukaea wants to merge 21 commits into
xdg-org:mainfrom
Waqar-ukaea:cuBQL-backend
Open

Adding a backend for cuBQL as a RayTracer#245
Waqar-ukaea wants to merge 21 commits into
xdg-org:mainfrom
Waqar-ukaea:cuBQL-backend

Conversation

@Waqar-ukaea

@Waqar-ukaea Waqar-ukaea commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

This PR adds initial support for NVIDIA's cuBQL, a CUDA BVH build-and-query library, as an optional XDG ray tracing backend. The current integration targets cuBQL's OpenMP target-offload path so XDG can build BVHs and run traversal/intersection kernels on an OpenMP target device.

This is still an initial backend implementation. Some OpenMP target allocation, copy, and free handling is intentionally explicit and is expected to be cleaned up in follow-up work.

Build-system and dependency changes

  • Adds cuBQL as an optional git submodule under vendor/cuBQL.
  • Adds XDG_ENABLE_CUBQL as an optional CMake backend switch.
  • Requires OpenMP when cuBQL support is enabled.
  • Adds CMakePresets.json entries for LLVM and NVHPC OpenMP target-offload builds on NVIDIA Ada hardware.
    • The LLVM path has been tested with LLVM 22 on an NVIDIA RTX 2000 Ada.
    • The NVHPC path is present but still needs fuller validation.
  • Extends runtime device probing so XDG can check for:
    • Vulkan ray tracing support for GPRT.
    • OpenMP target device availability for cuBQL.

XDG backend integration

  • Adds RTLibrary::CUBQL and wires it through the standard XDG backend creation paths.
  • Adds a new CuBQLRayTracer implementation.
  • Adds public device batch-ray types:
    • XDGRayHit
    • XDGRayHitBuffer
  • Adds public batch-ray APIs:
    • allocate_ray_hits()
    • free_ray_hits()
    • ray_fire_batch()

cuBQL backend implementation

The backend currently supports surface-based ray queries using a flattened per-volume BVH layout.

  • CuBQLSurfaceMesh owns device buffers for a topological surface's vertices, triangle indices, and primitive IDs.
  • CuBQLVolumeGroup owns one flattened cuBQL BVH per topological volume.
  • Each BVH primitive maps back to its surface and surface-local primitive through CuBQLVolumeGroup::PrimRef.
  • Surface metadata stored alongside the flattened BVH includes:
    • reverse sense,
    • next volume,
    • boundary condition,
    • max parent-volume bounding-box bump.
  • Shared surfaces are reused across volumes, while each volume gets its own flattened BVH over the surfaces it contains.
  • Traversal uses cuBQL's FP32 BVH traversal, with FP64 Plücker triangle intersection used for final hit testing.
  • Scalar support currently includes:
    • ray_fire()
    • point_in_volume()
  • Batch ray_fire_batch() runs on device-resident XDGRayHit arrays and writes hit/crossing payload data back into the same records:
    • hit distance,
    • surface,
    • primitive,
    • point-in-volume state,
    • next volume,
    • boundary condition,
    • surface normal.

Shared geometry/intersection changes

  • Adds a cuBQL-compatible dp_math pathway so the shared Plücker intersection code can compile in OpenMP target code using cuBQL math types.
  • Removes the shared EXIT_EARLY Plücker result object in favour of direct {false, 0.0} returns, which avoids OpenMP device compilation issues.

Tests and tools

  • Extends existing ray-fire and point-in-volume tests to include cuBQL when available.
  • Adds point-in-volume cross-checks across available ray tracing backends.
  • Adds cuBQL batch-ray tests covering:
    • device buffer lifecycle,
    • empty batch handling,
    • batch/scalar agreement on MeshMock,
    • multi-volume payload validation on the pincell model.
  • Updates ray_fire and particle_sim tools so CUBQL can be selected.
  • Updates ray_benchmark with a cuBQL path using device-side ray generation and ray_fire_batch().
  • Since OpenMP target offload is linked to every test file on this branch. Every test undergoes openmp initialisation which takes ~0.19s on my laptop. So moving forwards it would be nice to change this so that it is only linked to tests which require that initialisation. I encountered this very early on and I think that's what I figured out at the time - I may be wrong on what exactly is happening here though :/

Current limitations

  • cuBQL support is currently limited to surface-based ray queries.
  • Element trees are not supported.
  • Global surface and global element trees are not supported.
  • find_element(), closest(), and occluded() are not implemented for cuBQL.
  • Batch queries currently support only a single last_hit_primitive; arbitrary exclusion histories are only supported by scalar queries.
  • Multi-GPU/device selection is not implemented; the backend currently uses OpenMP target device 0.
  • Runtime handling for OpenMP offload libraries/RPATHs still needs refinement.
  • libMesh + cuBQL still needs validation.

Submodule note

cuBQL is included as a git submodule. This PR currently pins cuBQL to 5a651b3, which is the working version for this implementation. A later cuBQL commit was tried but caused XDG ray tracing query tests to fail, so updating the submodule further should be handled separately after investigating the cuBQL-side change.

@Waqar-ukaea
Waqar-ukaea marked this pull request as ready for review June 25, 2026 12:16
@Waqar-ukaea
Waqar-ukaea requested a review from pshriwise June 25, 2026 12:16
- New CuBQLRayTracer class with working create_surface_tree and basic ray queries
- Updated plucker_ray_tri intersect to play nice with openmp target offload regions
- Implemented new structs for cuBQL BVH objects with XDG data
- Implemented mixed precision BVH traversal algorithm for cuBQL
- Updated appropriate constants with new cuBQL specifics
- Ensured cuBQL properly wired through to xdg public API
…tadata

- Boundary condition metadata now stored on surface instances
- Next volume topology also stored on surface instances
- Primitive normal stored on hits to be used in reflective BCs
- Updated rayhit structs throughout to carry this new data
- Replace the two-level BLAS/TLAS scheme with single-level traversal
- Introduce CuBQLSurfaceMesh and CuBQLVolumeGroup following DPRT terminology
- Build flattened primitive references and AABBs in create_surface_tree
- Update scalar and batch intersection paths for flattened traversal
@Waqar-ukaea

Copy link
Copy Markdown
Collaborator Author

Tests now in place and rebased to latest version of main (after Quads+Hex PR)

@Waqar-ukaea

Copy link
Copy Markdown
Collaborator Author

I am unable to update the submodule commit to the latest version for cuBQL due to this issue NVIDIA/cuBQL#43

Another larger problem seemingly exists that is stopping openmp traversal of BVHs being as robust as it should be with subsequent builds occasionally having "bad regions" of the BVH in which ray hits are lost. No issue has been opened for that as of yet as I am still figuring out what the exact cause of that is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant