Get EM back

[Antoinehoff]

This PR is related to the DR #867 and gkylcas#89

Test cases

We report on a few test cases and benchmark, most of them taken from the work of @nmandell (see, e.g., Mandell et al. 2020 and Mandell thesis)

Alfven waves

We create the new associated regression tests.

gyrokinetic/creg/rt_gk_em_alfven_1x2v_p1.c
gyrokinetic/creg/rt_gk_em_alfven_2x2v_p1.c
gyrokinetic/creg/rt_gk_em_alfven_3x2v_p1.c

and verify that the results in Mandell thesis are retrieved.

The scan was performed on Perlmutter using one CPU node and running many Gkeyll session in parallel using the following C input file and python scripting:

• kaw_1x2v.c
• kaw_scan.py
• kaw_analysis.py
• gkeyll_interface.py.

3x2v helical scrape-off layer

We create a new associated regression test

gyrokinetic/creg/rt_gk_em_helical_sol_3x2v_p1.c

and we verify that the results are similar to Mandell et al. 2020.
Note: the geometry is adjusted to pass the right hand check (see PR #933).

Integrated moments

Electron density and perp. magnetic fluctuations

Helical SOL performance benchmark

We benchmark our different electromagnetic implementations with 3x2v shearless helical SOL turbulence, which uses a 3x2v phase-space grid with resolution $N_x=16$, $N_y=32$, $N_z=12$, $N_{v_\parallel}=10$, $N_\mu=5$. The table below summarizes the performance benchmarks for different versions of the solver, using 3 A100 GPUs on Perlmutter.

Version	Throughput
Full upwind, non-optimized	~2.2 steps/s
w. Kernel vpar flux optim.	~2.8 steps/s
w. Ohm's LHS updater (this branch)	~7.7 steps/s
Electrostatic	~9.1 steps/s

this branch

Mandel et al. 2020

Need to smooth $A_\parallel$ along $z$ in tokamak geometry

We now test EM effects on known simulation like TCV simulations. Before smoothing $A_\parallel$, it was not possible to run any TCV simulation for more than O(10) microseconds before observing a violent event breaking the simulation. This phenomenon was observed present in the following cases.

• TCV 2x2v SOL only (24x16x16x8)
• TCV 3x2v SOL only (24x16x12x12x8)
• TCV 2x2v clopen (24x16x16x8)
• TCV 3x2v clopen (24x16x12x12x8)

We tested a version where we apply $\dot A_\parallel \leftarrow \dot A_\parallel \langle \dot A_\parallel \rangle_{fs}$ at each Ohm solve to remove any zonal component of EM fluctuations. Suprisingly it did not even affect the time where the catastrophic event occur.

The simulation crashes approx. 450 mus after the restart. It does not seem that it reached steady state as shown with the evolution of $A_\parallel$ which continuously drops. This is no longer valid and was solved by restoring the bias plane in the potential (lost during some input file editing...).

When $A_\parallel$ is smoothed just after the stepping using $\dot{A}_\parallel$, the TCV PT coarse simulation runs smoothly:

For comparison purposes, this is the same simulation but electrostatic:

and time averaged profile comparisons averaged between 0.5ms and 1ms:

[Antoinehoff]

@JunoRavin, @manauref, I think this is ready for review. I don't think the design will change, the previous results were reproduced and the new tokamak geometry is not stable yet but it is capable. I would like to merge this so that I can stop resolving conflicts every week :)

[Antoinehoff]

Collaborator

@JunoRavin @manauref , a kind reminder now that the simulations are stable.