Val-2c/val-2f should perform consistently with 1 or 2 processors.

[simopier]

Reason

Validation cases val-2c and val-2f currently provides slightly different results when using 1 vs 2 processors, when it should provide the same results.

This issue arises later in the simulation close to steady state when concentrations are very small, which makes it acceptable for now, but this is an issue that should be addressed.

Currently, the tests are only run on 1 processor with max_parallel = 1 and this should be removed once this issue is fixed.

Design

• Figure out the source of these differences
• Fix it
• remove max_parallel = 1 from val-2c/tests

Impact

More consistent parallel capabilities.

[lindsayad]

In slack I've raised the issue that tests marked with max_parallel = 1 are very often the tests that cause us headaches in PETSc updates, and this is turning out to be true in idaholab/moose#32404 where it is indeed only the val-2c and val-2f cases that are failing. I'm wondering in the medium to long term whether we might consider alternative constraint approaches, perhaps augmented Lagrange or something like that. I see different active sets even in the first timestep for val-2f when doing serial vs. parallel. In the short term, this is blocking MOOSE moving forward. For SAM tests marked with max_parallel = 1, their CCB suggested marking the tests with skip for the moment until they can make the tests more robust

[simopier]

In slack I've raised the issue that tests marked with max_parallel = 1 are very often the tests that cause us headaches in PETSc updates, and this is turning out to be true in idaholab/moose#32404 where it is indeed only the val-2c and val-2f cases that are failing. I'm wondering in the medium to long term whether we might consider alternative constraint approaches, perhaps augmented Lagrange or something like that. I see different active sets even in the first timestep for val-2f when doing serial vs. parallel. In the short term, this is blocking MOOSE moving forward. For SAM tests marked with max_parallel = 1, their CCB suggested marking the tests with skip for the moment until they can make the tests more robust

Following an offline discussion, let's make sure we tackle this asap and try to strengthen the tests with suggestions provided above. We'll skip as last resort.

@lin-yang-ly, could you try to make those changes as part of #382?
Let's chat if you have any questions.

[lindsayad]

I don't know a priori that augmented Lagrange will fix it but it can provide less constraint oscillations, which we can see currently since the reduced set method is effectively a discontinuous style of constraint enforcement so slight differences in ordering of floating point operations could lead you to one side of the discontinuity or another

[simopier]

#382 is fixing this issue for val-2c. The issue currently persists for val-2f.