Issue Description
I am running a TMAP8 simulation at ARCHER2. Here I use singularity to run a docker container (ghcr.io/idaholab/apptainer/tmap8-openmpi-x86_64:2025.12.30-74b8ae5) version of the code.
The serial version of the code starts running succesfully:
Time Step 1, time = 1e-07, dt = 1e-07
|residual|_2 of individual variables:
mobile: 0.415436
trapped: 0
0 Nonlinear |R| = 4.154363e-01
0 Linear |R| = 4.154363e-01
1 Linear |R| = 6.514191e-06
2 Linear |R| = 1.492468e-08
|residual|_2 of individual variables:
mobile: 0.414275
trapped: 1.90977e-27
1 Nonlinear |R| = 4.142748e-01
0 Linear |R| = 4.142748e-01
1 Linear |R| = 6.502131e-06
2 Linear |R| = 1.491681e-08
|residual|_2 of individual variables:
mobile: 0.000608835
trapped: 8.69295e-18
2 Nonlinear |R| = 6.088348e-04
0 Linear |R| = 6.088348e-04
1 Linear |R| = 5.945468e-11
|residual|_2 of individual variables:
mobile: 8.812e-06
trapped: 6.13796e-18
3 Nonlinear |R| = 8.811996e-06
Solve Converged!
Currently Executing
Finished Solving [ 44.28 s] [ 347 MB]
However, when I am launching the same simulation via
singularity exec --pwd=$PWD --bind=$PWD ${SLURM_SUBMIT_DIR}/tmap8-openmpi-x86_64_2025.12.30-74b8ae5.sif \ mpiexec -n ${SLURM_NTASKS} tmap8-opt -i traps_srs_param.i -w
parallelised across 16 cores
Parallelism:
Num Processors: 16
Num Threads: 1
I am encountering a convergence error for both linear and nonlinear solve, with adaptive time-stepping failing eventually:
...
Solve failed, cutting timestep.
Time Step 1, time = 1e-12, dt = 1e-12
|residual|_2 of individual variables:
mobile: 0.415436
trapped: 0
0 Nonlinear |R| = 4.154363e-01
Linear solve did not converge due to DIVERGED_PC_FAILED iterations 0
PC failed due to FACTOR_OUTMEMORY
Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0
Solve Did NOT Converge!
Aborting as solve did not converge
*** ERROR ***
/mnt/lustre/a2fs-work2/work/e05/e05/yyudin01/tmap8_scans/traps_srs_param.i:595.3:
The following occurred in the TimeStepper 'IterationAdaptiveDT' of type IterationAdaptiveDT.
Solve failed and timestep already at dtmin, cannot continue!
The MOOSE troubleshooting pages (https://mooseframework.inl.gov/application_usage/failed_solves.html) recommend implementing a split preconditioning, however, just following the basic example for the method (https://mooseframework.inl.gov/source/preconditioners/FieldSplitPreconditioner.html) did not solve the issues.
I wonder if anyone have any other ideas of how to resolve this issue, or any experience on preconditioning for parallelized problems that might help!
Issue Description
I am running a TMAP8 simulation at ARCHER2. Here I use singularity to run a docker container (ghcr.io/idaholab/apptainer/tmap8-openmpi-x86_64:2025.12.30-74b8ae5) version of the code.
The serial version of the code starts running succesfully:
However, when I am launching the same simulation via
parallelised across 16 cores
I am encountering a convergence error for both linear and nonlinear solve, with adaptive time-stepping failing eventually:
The MOOSE troubleshooting pages (https://mooseframework.inl.gov/application_usage/failed_solves.html) recommend implementing a split preconditioning, however, just following the basic example for the method (https://mooseframework.inl.gov/source/preconditioners/FieldSplitPreconditioner.html) did not solve the issues.
I wonder if anyone have any other ideas of how to resolve this issue, or any experience on preconditioning for parallelized problems that might help!