Cleaning up unit and regression tests

[manauref]

This is a status report of our current unit and regression tests with some action items for me, @JunoRavin, @JonathanGorard @akashukla, @Maxwell-Rosen, @jRoeltgen and maybe others.

Unit tests

1. make check compiles before running the tests, but on some GPU machines (e.g. stellar), one can't compile on the compute node, and you need the compute node to run GPU tests. Maybe make unit just compiles? I didn't check.
2. There are about 130-140 unit test files across core/moments/vlasov/gyrokinetic. Of these
   37 are core
   27 are moments
   18 are vlasov
   40 are GK
   other: _cu files and mctests (parallel). pkpm?
3. It took 4 minutes to run on my Mac, and 8 minutes on Stellar-amd.
4. Failing tests are shown below:

Mac run had the following failures:

../build/moments/unit/ctest_fem_poisson_vareps
  Test test_2x_p1_dirichletx_neumanny_dirichlety... [ FAILED ]
  Test test_2x_p2_neumannx_dirichletx_dirichlety... [ FAILED ]
  Test test_2x_p2_dirichletx_neumanny_dirichlety... [ FAILED ]
  Test test_2x_p2_neumannx_dirichletx_dirichlety... [ FAILED ]
../build/gyrokinetic/unit/ctest_asdex
  Test test_lower...                              Failed to open the eqdsk file: ./data/eqdsk/asdex.geqdsk
  Test ellip...                                   [ FAILED ]
  ctest_gkgeom.c:93: Check gkyl_compare(8.382428377712543, arcL, 1e-12)... failed
  ctest_gkgeom.c:102: Check gkyl_compare(8.172574228918158, arcL, 1e-12)... failed
../build/core/unit/ctest_mirror_grid_gen
  Test wham_no_axis_psi...                        /bin/sh: line 1: 72168 Segmentation fault: 11 ./core/../build/core/unit/ctest_mirror_grid_gen -E
../build/gyrokinetic/unit/ctest_mom_gyrokinetic
  Test test_2x2v_p1...                            [ FAILED ]
../build/gyrokinetic/unit/ctest_rescale_ghost_jacf
  Test test_1x1v_ho...                            [ FAILED ]
  Test test_2x2v_ho...                            [ FAILED ]
../build/gyrokinetic/unit/ctest_step_compare
  Test test_pflo_l...                             Assertion failed: (false), function gkyl_tok_geo_calc_interior, file tok_geo.c, line 972.

Stellar-amd run had the following failures:

./core/unit/ctest_dg_basis_ops.c
  Test cubic_evalf_2d...                          [ FAILED ]
  ctest_dg_basis_ops.c:124: Check gkyl_compare_double(fout[2], 2*sq(xn[0]), 1.0e-15)... failed
./core/unit/ctest_mirror_grid_gen.c
  Test wham_no_axis_psi...                          Test interrupted by SIGSEGV.
  Test wham_with_axis_psi...                        Test interrupted by SIGSEGV.
  Test wham_no_axis_sqrt_psi...                     Test interrupted by SIGSEGV.
  Test wham_with_axis_sqrt_psi...                   Test interrupted by SIGSEGV.
  Test quad_geom_no_axis_psi...                     Test interrupted by SIGSEGV.
  Test quad_geom_with_axis_psi...                   Test interrupted by SIGSEGV.
  Test quad_geom_no_axis_sqrt_psi...                Test interrupted by SIGSEGV.
  Test quad_geom_with_axis_sqrt_psi...              Test interrupted by SIGSEGV.
./core/unit/ctest_tensor_field_ops.c
  Test cu_tensor_field_lower_idx_in_place...      Error:
  CUDA error: an illegal memory access was encountered

  Test cu_tensor_field_raise_idx_in_place...      Error:
  CUDA error: an illegal memory access was encountered

  Test cu_tensor_field_lower_idx_set...           Error:
  CUDA error: an illegal memory access was encountered

  Test cu_tensor_field_raise_idx_set...           Error:
  CUDA error: an illegal memory access was encountered
./moments/unit/ctest_wv_gr_mhd.c
  Test gr_mhd_waves_schwarzschild...              [ FAILED ]
./moments/unit/ctest_wv_gr_mhd_tetrad.c
  Test gr_mhd_tetrad_waves_schwarzschild...       [ FAILED ]
./vlasov/unit/ctest_dg_em_vars.c
  Test test_1x_p1_gpu...                          [ FAILED ]
  Test test_2x_p1_gpu...                          [ FAILED ]
  Test test_1x_p2_gpu...                          [ FAILED ]
  Test test_2x_tensor_p2_gpu...                   ctest_dg_em_vars: zero/dg_bin_ops_cu.cu:447: void gkyl_dg_div_op_range_cu(gkyl_dg_bin_op_mem*, gkyl_basis, int, gkyl_array*, int, const gkyl_array*, int, const gkyl_array*, const gkyl_range*): Assertion `status' failed.
    Test interrupted by SIGABRT.
  Test test_3x_tensor_p2_gpu...                   ctest_dg_em_vars: zero/dg_bin_ops_cu.cu:447: void gkyl_dg_div_op_range_cu(gkyl_dg_bin_op_mem*, gkyl_basis, int, gkyl_array*, int, const gkyl_array*, int, const gkyl_array*, const gkyl_range*): Assertion `status' failed.
    Test interrupted by SIGABRT.
./vlasov/unit/ctest_prim_vlasov.c
or
./vlasov/unit/ctest_dg_lbo_vlasov.c
or
./gyrokinetic/unit/ctest_mom_gyrokinetic.c
  Test test_1x1v_p2_cu...                         Error:
  CUDA error: an illegal memory access was encountered

  Test test_1x2v_p2_cu...                         Error:
  CUDA error: an illegal memory access was encountered
./gyrokinetic/unit/ctest_step_compare.c
or
./gyrokinetic/unit/ctest_asdex.c
  Test test_lower...                              Failed to open the eqdsk file: ./data/eqdsk/asdex.geqdsk
  ctest_asdex: zero/efit.c:33: gkyl_efit_new: Assertion `false' failed.
    Test interrupted by SIGABRT.
  Test test_middle...                             Failed to open the eqdsk file: ./data/eqdsk/asdex.geqdsk
  ctest_asdex: zero/efit.c:33: gkyl_efit_new: Assertion `false' failed.
    Test interrupted by SIGABRT.
  Test test_upper...                              Failed to open the eqdsk file: ./data/eqdsk/asdex.geqdsk
  ctest_asdex: zero/efit.c:33: gkyl_efit_new: Assertion `false' failed.
    Test interrupted by SIGABRT.
./gyrokinetic/unit/ctest_deflated_fem_poisson.c
  Test test_fem_poisson_zind_dd...                ctest_deflated_fem_poisson: zero/fem_poisson.c:337: gkyl_fem_poisson_set_rhs: Assertion `gkyl_array_is_cu_dev(rhsin)' failed.
  Test interrupted by SIGABRT.
./gyrokinetic/unit/ctest_dg_interpolate.c
  Test test_1x1v_gk_ho...                         malloc(): corrupted top size
  Test interrupted by SIGABRT.
  Test test_1x2v_gk_ho...                         malloc(): corrupted top size
  Test interrupted by SIGABRT.
  Test test_2x2v_gk_ho...                         malloc(): corrupted top size
  Test interrupted by SIGABRT.
  Test test_3x2v_gk_ho...                         malloc(): corrupted top size
  Test interrupted by SIGABRT.
  Test test_1x1v_gk_dev...                        free(): invalid next size (fast)
  Test interrupted by SIGABRT.
  Test test_1x2v_gk_dev...                        free(): invalid next size (fast)
  Test interrupted by SIGABRT.
  Test test_2x2v_gk_dev...                        malloc(): invalid size (unsorted)
  Test interrupted by SIGABRT.
  Test test_3x2v_gk_dev...                        free(): invalid next size (fast)
  Test interrupted by SIGABRT.
./gyrokinetic/unit/ctest_gkgeom.c
  Test ellip...                                   [ FAILED ]
  ctest_gkgeom.c:93: Check gkyl_compare(8.382428377712543, arcL, 1e-12)... failed
  ctest_gkgeom.c:102: Check gkyl_compare(8.172574228918158, arcL, 1e-12)... failed
./gyrokinetic/unit/ctest_mom_gyrokinetic.c
  Test test_2x2v_p1...                            [ FAILED ]
  Test test_2x2v_p1_cu...                         [ FAILED ]
./gyrokinetic/unit/ctest_nodal_ops.c
Test test_p1_interior_2x...                       Test interrupted by SIGSEGV.
Test test_p1_interior_3x...                       Test interrupted by SIGSEGV.
Test test_p1_3x...                              [ FAILED ]
./gyrokinetic/unit/ctest_positivity_shift_gyrokinetic.c
  Test test_1x2v_ho...                            malloc(): corrupted top size
  Test interrupted by SIGABRT.
  Test test_1x2v_dev...                           malloc(): corrupted top size
  Test interrupted by SIGABRT.
./gyrokinetic/unit/ctest_rescale_ghost_jacf.c
  Test test_1x1v_ho...                            [ FAILED ]
  Test test_2x2v_ho...                            [ FAILED ]
  Test test_1x1v_dev...                           [ FAILED ]
  Test test_2x2v_dev...                           [ FAILED ]
./gyrokinetic/unit/ctest_step_compare.c
  Test test_core_l...                             ctest_step_compare: zero/tok_geo.c:757: gkyl_tok_geo_calc: Assertion `false' failed.
  Test interrupted by SIGABRT.
  Test test_pflo_l...                             ctest_step_compare: zero/tok_geo.c:972: gkyl_tok_geo_calc_interior: Assertion `false' failed.  Test interrupted by SIGABRT.
./gyrokinetic/unit/ctest_step_outboard.c
  Test test_fixed_z...                            free(): invalid pointer
./test.sh: line 122: 3911843
./cuda-build/gyrokinetic/unit/ctest_step_outboard
 Aborted                 (core dumped)

Warnings:
./gyrokinetic/unit/ctest_dg_rad_gyrokinetic.c
  Test test_1x2v_p1_Li1_lowNe...
  Warning! More densities selected for element 3, charge state 1 than available. Setting to max available.
  [ OK ]
  Test test_1x2v_p1_Li1_midNe...
  Warning! More densities selected for element 3, charge state 1 than available. Setting to max available.
  [ OK ]
  Test test_1x2v_p1_Li1_highNe...
  Warning! More densities selected for element 3, charge state 1 than available. Setting to max available.
  [ OK ]
  Test test_1x2v_p1_L1_midNe_gpu...
  Warning! More densities selected for element 3, charge state 1 than available. Setting to max available.

Favorite test name:
Test triangle...                                [ OK ]

Action items

Here's a list of the people I think are best suited to fix each of these tests. As you go through these, please rename the tests inside the file so they are more unique (to avoid same-named tests in multiple files):
@jRoeltgen
./gyrokinetic/unit/ctest_dg_rad_gyrokinetic.c (what I find weird is that this warning came up on Stellar-amd but not my Mac, so that's the only thing that needs resolving here)
@Maxwell-Rosen
../build/core/unit/ctest_mirror_grid_gen
@JonathanGorard
./moments/unit/ctest_wv_gr_mhd.c
./moments/unit/ctest_wv_gr_mhd_tetrad.c
@JunoRavin
./core/unit/ctest_tensor_field_ops.c
./vlasov/unit/ctest_dg_em_vars.c
./vlasov/unit/ctest_prim_vlasov.c
./vlasov/unit/ctest_dg_lbo_vlasov.c
@manauref
../build/moments/unit/ctest_fem_poisson_vareps
../build/gyrokinetic/unit/ctest_mom_gyrokinetic
../build/gyrokinetic/unit/ctest_rescale_ghost_jacf
./gyrokinetic/unit/ctest_dg_interpolate.c
./gyrokinetic/unit/ctest_positivity_shift_gyrokinetic.c
@akashukla
../build/gyrokinetic/unit/ctest_asdex
../build/gyrokinetic/unit/ctest_step_compare
./core/unit/ctest_dg_basis_ops.c
./gyrokinetic/unit/ctest_deflated_fem_poisson.c
./gyrokinetic/unit/ctest_gkgeom.c
./gyrokinetic/unit/ctest_nodal_ops.c
./gyrokinetic/unit/ctest_step_compare.c
./gyrokinetic/unit/ctest_step_outboard.c
./cuda-build/gyrokinetic/unit/ctest_step_outboard

Regression tests

TBD (I tried running them but I didn't time each, so I'll re-do it and post the report here).

We will require the following regression tests before merging PRs:

Moments

Vlasov

Gyrokinetic

PKPM

[jRoeltgen]

@manauref The warning is supposed to occur in the unit test. Can you check that the radiation_fit_parameters.txt file is the same between your Mac and Stellar.

[akashukla]

For everyone, this may help: On perlmutter, make gyrokinetic-check works on a compute node. I guess you can compile on the compute node there even though I thought that was not possible.

[Maxwell-Rosen]

Multiple unit tests are currently failing on Perlmutter GPU. This issue tracks the failing tests that need to be fixed to ensure reliable CI checks. Additionally, several tests are experiencing crashes (segfaults/aborts) before they can even run.

testing_gk_check_perlmutter.txt

Critical Issues

Tests Crashing Before Execution
These tests are causing segmentation faults or aborts and need immediate attention:

ctest_positivity_shift_gyrokinetic - malloc(): corrupted top size → ABORTED (assigned @manauref)
ctest_ltx_miller - SEGMENTATION FAULT
ctest_dg_interpolate - free(): invalid next size (fast) → ABORTED
ctest_dg_gyrokinetic_kern_tm - SEGMENTATION FAULT

Tests with Numerical Failures

1. ctest_rescale_ghost_jacf - ALL 4 TESTS FAILED
   Status: CRITICAL - Produces excessive output (~20k lines), making debugging difficult
   Failed tests:
   test_1x1v_ho
   test_2x2v_ho
   test_1x1v_dev
   test_2x2v_dev
   Issue: Significant numerical discrepancies between expected and actual values
   Action needed: Reduce verbosity of failure output to single line per test
2. ctest_mom_gyrokinetic - 2 of 7 TESTS FAILED
   Failed tests:
   test_2x2v_p1 (host)
   test_2x2v_p1_cu (GPU)
   Issue: Multiple comparison failures at ctest_mom_gyrokinetic.c:884
3. ctest_gyrokinetic_cross_prim_moms_bgk - 1 of 2 TESTS FAILED
   Failed test: test_1x2v_p1
   Issue: Values differ by ~1% (Expected: 2.834e9, Got: 2.863e9)
   Location: ctest_gyrokinetic_cross_prim_moms_bgk.c:426

Assignments

Based on the previous tally of assignments, here are the individuals allocated to address these shortcomings.

@manauref
ctest_rescale_ghost_jacf (CRITICAL - 4/4 tests failed, excessive output)
All tests failing with numerical discrepancies
Produces ~20k lines of output, needs verbosity fix. Perhaps use a counter to just output one line saying there are 20k errors instead of 20k lines of errors.
ctest_mom_gyrokinetic (2/7 tests failed)
test_2x2v_p1 (host) failing
test_2x2v_p1_cu (GPU) failing
ctest_dg_interpolate (CRASH - free(): invalid next size)
Test aborted before execution
ctest_positivity_shift_gyrokinetic (CRASH - malloc(): corrupted top size)
Test aborted before execution
@akashukla
ctest_ltx_miller (CRASH - Segmentation fault)
Test crashes before execution
ctest_dg_gyrokinetic_kern_tm (CRASH - Segmentation fault)
Test crashes before execution
@JunoRavin
ctest_gyrokinetic_cross_prim_moms_bgk (1/2 tests failed)
test_1x2v_p1 failing with ~1% nume

[manauref]

I think much of this was fixed in PR #1089