ShallowWater2D

ShallowWater2D is a mini-app designed to test different parallelization models to simulate the shallow water equations in two dimensions. ShallowWater2D is a lean mini-app (only one source file) written in object-oriented C++. The project is a work in progress.

There are currently 4 versions of the code:

• serial,
• pure openMP using tasks,
• pure MPI using Cartesian virtual topology,
• OpenACC with unified memory.

All versions have been validated against an analytical travelling wave solution in 1D and periodic boundary conditions (using ghost cells).

Equations solved

The shallow water equations solved are:

$$ \frac{\partial }{\partial t} \begin{pmatrix} h \\ m_x \\ m_y \end{pmatrix} + \frac{\partial }{\partial x} \begin{pmatrix} m_x \\ \frac{m_x^2 }{h} + \frac{1}{2} h^2 \\ \frac{m_x m_y }{h} \end{pmatrix} + \frac{\partial }{\partial y} \begin{pmatrix} m_y \\ \frac{m_x m_y }{h} \\ \frac{m_y^2 }{h} + \frac{1}{2} h^2 \end{pmatrix} = \mathbf{S} $$

where $h$ is the column height, $m_x$ and $m_y$ are the discharges in the x,y direction, respectively. They are related to the fluid velocity $\mathbf{u}$ by $\mathbf{u} = \mathbf{m} / h$. The fluid density is assumed constant and units are such that the gravitational acceleration $g=1$.

Numerical scheme

The fluid equations are evolved on a cartesian grid using the finite volume method to achieve second-order accuracy. This is done using a strong stability-preserving 3rd order Runge-Kutta scheme for the timestepping and a Lax-Friedrichs numerical flux for linear interpolation at the cell faces with slope-limiters (minmod, van Leer).

Compilation instructions

• serial:

  g++ main.cpp -O3 -o main.exe
  

• OpenMP

  g++ main_openmp.cpp -O3 -fopenmp -o main_openmp.exe
  

  Run with:

  export OMP_NUM_THREADS=4; ./main_openmp.exe
  

• MPI

  mpic++ main_mpi.cpp -O3 -o main_mpi.exe
  

  Run with:

  mpirun -np 4 ./main_mpi.exe 256 256
  

• OpenACC

  nvc++ -acc -Minfo=accel -gpu=mem:managed -o main_acc.exe main_acc.cpp
  

Profiling

The performance of the different versions of ShallowWater2D can be profiled using different tools. One possibility is with Nvidia Nsight and NVTX annotations.

An example command for profiling is:

nsys profile --trace cuda,nvtx --stats=true ./main.exe 256 256

For the serial code, the timings are:

Time (%)	Total Time (ns)	Instances	Avg (ns)	Med (ns)	Min (ns)	Max (ns)	StdDev (ns)	Style	Range
59.3	20,927,886,491	1,511	13,850,355.1	13,774,741.0	13,559,033	20,144,508	407,534.6	PushPop	advanceTime
19.9	7,033,304,803	4,533	1,551,578.4	1,543,874.0	1,535,097	1,866,331	24,152.5	PushPop	compute_tendencies_x
19.6	6,924,805,648	4,533	1,527,643.0	1,520,053.0	1,512,024	1,831,376	23,371.3	PushPop	compute_tendencies_y
1.0	353,188,751	1,511	233,745.0	232,110.0	231,312	288,900	4,193.6	PushPop	compute_dt
0.0	13,429,268	4,534	2,961.9	3,092.0	1,486	32,613	1,320.7	PushPop	set_halo_values_y
0.0	11,394,816	4,534	2,513.2	2,177.0	1,710	16,334	930.2	PushPop	set_halo_values_x

Acknowledgements

The numerical schemes and the choice of the physical model are based on the project work that the author did for the course "Numerical Methods for Conservation Laws" with Professor Jan S. Hesthaven at the École polytechnique fédérale de Lausanne (EPFL) for the Master in Physics (minor in Computational Science and Engineering), a. 2016-2018.