Add check_nan_every to CI configs

[hgpeterson]

Closes #1915

[hgpeterson]

Confirmed that the setup that erroneously passed last time correctly failed under these changes.

[szy21]

Note that check nan is very slow. It would be good to check if this significantly increases ci time. If so we can increase check_nan_every for the jobs that run longer.

[hgpeterson]

The timings between the two cases linked above don't seem too different (although for some reason the sims that fail with NaNs take much longer?).

[szy21]

The timings between the two cases linked above don't seem too different (although for some reason the sims that fail with NaNs take much longer?).

cpu jobs can be a bit random as it depends on which node we get. But if it's not too different it's fine. We can always change later if ci becomes too long.

[juliasloan25]

This looks good, thank you! Could you add some info to the "Debugging tips" section of the ClimaCoupler docs explaining that this option exists and what it does?

[juliasloan25]

How does the run that you linked as failing now pass in this PR's CI? Is it because of the fix here?

[hgpeterson]

How does the run that you linked as failing now pass in this PR's CI? Is it because of the fix here?

Yes, that run is fixed due to #1914. But I tested that this PR's changes make CI correctly fail pre-#1914 here.

[akshaysridhar]

Thanks @hgpeterson; following the above comment, if you could add a note to the "debugging" block that addresses this yaml config arg, we can squash and merge this PR. The CI timing seems to be at 1h24m now vs 1h11m prior to this callback frequency change.