tripolar grid domain decomposition

[TomMelt]

Decomposing a tripolar grid requires that the north polar split lies exactly along a domain boundary, this is due to the way the halos are shared between processors and how boundary conditions are essentially implemented using MPI.

Although not implemented yet anyway, one thing to consider is that for the tripolar grid you would need at least 2 processes (1 for each side of the boundary). Or, you would have to write custom logic for handling the boundary condition.

The way that Zoltan's RCB method splits the domain is by first bisecting the domain into two equal parts:

In RCB, the computational domain is first divided into two regions by a cutting plane orthogonal to one of the coordinate axes so that half the work load is in each of the sub-regions. The splitting direction is determined by computing in which coordinate direction the set of objects is most elongated, based upon the geometric locations of the objects.

But I think it is possible (although not tested) to force the direction using the option:

RCB_SET_DIRECTIONS = 0 = don't order cuts; 1 = xyz; 2 = xzy; 3 = yzx; 4 = yxz; 5 = zxy; 6 = zyx;

I think we can use either option 1 or 3. (Currently the code sets 1 as the default, but I think we want 3 for our standard setup).

domain_decomp/ZoltanPartitioner.cpp

Line 153 in b554997

_zoltan->Set_Param("RCB_SET_DIRECTIONS", "1");