Skip to content

CI runtime: shard the test batches across parallel jobs when the single job tightens again #730

Description

@lmoresi

build_uw3_and_test.yaml referenced "the CI-runtime issue" as the home for the
durable fix to wall-clock. That issue was never actually filed. This is it, with
the measurements.

Where it stands

The 120-minute cap was set on a 2026-08-15 measurement putting the serial phase
alone at ~55 minutes, before xdist distributed the batches. Measured 2026-09-12,
the five most recent successful runs spent this long in "Run tests":

34m19s   development
35m23s   development
42m32s   feature/particle-demos
44m20s   docs/tier-c-characterisation-tests
47m34s   docs/timestepping-pattern

That headroom is why the previously-deferred bands could simply be switched on
rather than traded against something else.

The fix, for when it is needed

Split the batches across parallel GitHub jobs so wall-clock stops tracking total
test time. The serial phase already falls into natural groups that map onto a
job matrix without restructuring scripts/test.sh much:

  • test_00*-test_03*
  • test_05*-test_08*
  • test_10*-test_14*
  • the --p N mpirun phase, which runs its own batches

Not yet, though

Doing it now would be machinery ahead of need. The single job fits inside its
budget with room, and every job added pays the ~2.5 min build again unless the
build artefact is shared between them — which is the part that needs designing,
not the matrix.

Worth revisiting when "Run tests" passes roughly 70 minutes, or when the build
step can be cached across jobs.

Underworld development team with AI support from Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions