precompile CMIP with strict bounds checking

[juliasloan25]

Purpose

#1935 added strict bounds checking to a CMIP run in CI, but that run now takes 2 hours 🤯 This PR adds a run to CI that only runs CMIP for one timestep with strict bounds checking.

If this is still too slow (> 40 minutes?), another solution is to add a run that only initializes CMIP with strict bounds checking, and then add a run to the longrun pipeline that takes a few steps. The bug that was fixed in #1935 would have been caught by the initialization alone.

Update: now each of the CMIP runs takes <= 30 minutes (and are not the longest runs in CI), so I think this is a good solution

Update 2:
#1935 added strict bounds checking to a CMIP run in CI, and the first run of that simulation took 2 hours.
The first run of the new strictly-bounded run took 2 hours (see here), but the second took only ~38 mins (see here). This is because the depot we use to avoid precompiling in each build/run includes the check-bounds flag in its hash. So when we changed that flag for a single run, precompilation got triggered for that run (which you can see in the logs). The second run was much shorter because it was a cache hit and didn't have to precompile again.

I'm keeping the run, but changing both CMIP runs in the CI pipeline to use check-bounds=yes. This way we don't need to precompile CMIP with both check-bounds options, and we can still test the behavior with strict bounds. The runs seem to be about 35% slower with check-bounds=yes, so to offset this slowdown I decreased the run length of the CMIP runs in regular CI. I think this is fine because these are not physically-meaningful runs anyway - their primary purpose is to exercise the software pipes of the simulations.

[hgpeterson]

Works for me. Just wondering how reliable these precompiled caches are... is it likely that we'll still have to precompile from scratch very often? And if so, will it take 2h again?

[juliasloan25]

Works for me. Just wondering how reliable these precompiled caches are... is it likely that we'll still have to precompile from scratch very often? And if so, will it take 2h again?

They're pretty reliable - I would say we clear the depot maybe once a month? It will take a long time but I feel like it's usually more like 1-1.5 hours. But this would be the case with the previous setup anyway because the AMIP and CMIP envs would be precompiled in the init stage.