DIVERGED_PC_FAILED, FACTOR_OUTMEMORY, DIVERGED_FNORM_NAN in parallel version of Linear and Nonlinear solve only

[YehorYudinIPP]

Issue Description

I am running a TMAP8 simulation at ARCHER2. Here I use singularity to run a docker container (ghcr.io/idaholab/apptainer/tmap8-openmpi-x86_64:2025.12.30-74b8ae5) version of the code.

The serial version of the code starts running succesfully:

Time Step 1, time = 1e-07, dt = 1e-07
|residual|_2 of individual variables:
mobile: 0.415436
trapped: 0
0 Nonlinear |R| = 4.154363e-01
0 Linear |R| = 4.154363e-01
1 Linear |R| = 6.514191e-06
2 Linear |R| = 1.492468e-08
|residual|_2 of individual variables:
mobile: 0.414275
trapped: 1.90977e-27
1 Nonlinear |R| = 4.142748e-01
0 Linear |R| = 4.142748e-01
1 Linear |R| = 6.502131e-06
2 Linear |R| = 1.491681e-08
|residual|_2 of individual variables:
mobile: 0.000608835
trapped: 8.69295e-18
2 Nonlinear |R| = 6.088348e-04
0 Linear |R| = 6.088348e-04
1 Linear |R| = 5.945468e-11
|residual|_2 of individual variables:
mobile: 8.812e-06
trapped: 6.13796e-18
3 Nonlinear |R| = 8.811996e-06
Solve Converged!
Currently Executing
Finished Solving [ 44.28 s] [ 347 MB]

However, when I am launching the same simulation via

singularity exec --pwd=$PWD --bind=$PWD ${SLURM_SUBMIT_DIR}/tmap8-openmpi-x86_64_2025.12.30-74b8ae5.sif \ mpiexec -n ${SLURM_NTASKS} tmap8-opt -i traps_srs_param.i -w

parallelised across 16 cores

Parallelism:
Num Processors: 16
Num Threads: 1

I am encountering a convergence error for both linear and nonlinear solve, with adaptive time-stepping failing eventually:

...
Solve failed, cutting timestep.

Time Step 1, time = 1e-12, dt = 1e-12
|residual|_2 of individual variables:
mobile: 0.415436
trapped: 0
0 Nonlinear |R| = 4.154363e-01
Linear solve did not converge due to DIVERGED_PC_FAILED iterations 0
PC failed due to FACTOR_OUTMEMORY
Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0
Solve Did NOT Converge!
Aborting as solve did not converge

*** ERROR ***
/mnt/lustre/a2fs-work2/work/e05/e05/yyudin01/tmap8_scans/traps_srs_param.i:595.3:
The following occurred in the TimeStepper 'IterationAdaptiveDT' of type IterationAdaptiveDT.

Solve failed and timestep already at dtmin, cannot continue!

The MOOSE troubleshooting pages (https://mooseframework.inl.gov/application_usage/failed_solves.html) recommend implementing a split preconditioning, however, just following the basic example for the method (https://mooseframework.inl.gov/source/preconditioners/FieldSplitPreconditioner.html) did not solve the issues.

I wonder if anyone have any other ideas of how to resolve this issue, or any experience on preconditioning for parallelized problems that might help!

[simopier]

@loganharbour, any suggestions?

[loganharbour]

This piece:

PC failed due to FACTOR_OUTMEMORY

leads me to believe that it's a memory problem.

What are your preconditioner/petsc options?

[YehorYudinIPP]

Thanks for your reply, Logan, and sorry for the tardiness!

For the case I showed here, the preconditioner configuration is:

    type = SMP
    full = true
    solve_type = 'NEWTON'
    petsc_options_iname = '-pc_type -pc_factor_shift_type -pc_factor_shift_amount'
    petsc_options_value = 'lu       NONZERO               1e-10'

No extra PETSC configurations are applied to the solver in the Executioner block. The solver scheme applied here is BDF2.

Here, I requested an entire node, so presumably I should have 256GB of memory per 104802 DOFs in the nonlinear system.

Any recommendations on better preconditioner setup are still very welcome!

[simopier]

@YehorYudinIPP for 104802 DOFs, we would recommend 104802/20000 ~ 5 cores, so using 16 might be a bit too much, but it should be the cause of the issue.

I'll let @loganharbour comment on the preconditioner configuration, but nothings seems obviously bad for me. The NEWTON solve type with the full jacobian and the Lu method will indeed be more memory-hungry, but this isn't a huge problem either.

My recommendation would be to scale your simulation step by step. Can you make it run properly on 1 core with a smaller domain/mesh, and then increase the DOFs by a factor of 2 on 2 cores, etc? I would be curious to hear when that issue arises.