Skip to content

CI: harden the NVHPC jobs (deterministic codegen, retryable image pull) - #10

Merged
sbryngelson merged 2 commits into
masterfrom
ci/nvhpc-bump
Sep 2, 2026
Merged

CI: harden the NVHPC jobs (deterministic codegen, retryable image pull)#10
sbryngelson merged 2 commits into
masterfrom
ci/nvhpc-bump

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Three sources of red builds in MFC NVHPC, none of them ffmt bugs. One commit each.

1. Nondeterministic codegen (4e7f12d)

The cpu job fails intermittently with an nvfortran backend crash while compiling m_model.fpp:

LLVM ERROR: Cannot select: v4f32 = X86ISD::RSQRT14S
  in function '@m_model_s_read_stl_binary_'
nvfortran-Fatal-.../24.5/compilers/share/llvm/bin/llc TERMINATED by signal 6

nvfortran defaults to -tp=native, so it tunes for whichever host the job lands on. GitHub's ubuntu-22.04 pool mixes AVX-512-capable and non-capable CPUs, so on an AVX-512 draw the compiler emits an RSQRT14S its own LLVM backend cannot lower. Hence green on some runs, aborted on others, with no relevant change to ffmt — the same job failed on copilot/fix-failing-github-actions-job-test on 2026-09-01, and its gpu-omp sibling failed on a different run of that branch.

MFC compiles the identical source on 24.5 without trouble because it sets -tp=px. So this pins FFLAGS=-tp=px -Kieee -noswitcherror, CFLAGS=-tp=px, CXXFLAGS=-tp=px, matching MFC's own NVHPC jobs.

2. Stale compiler (4e7f12d)

Container moves 24.5 -> 26.3, the newest release MFC exercises upstream. Its matrix covers every release from 23.11 through 26.3 and all of them are green there, which also confirms the tag resolves. Secondary to the -tp pin — a bump alone would still leave codegen depending on the runner draw.

3. Unretryable image pull (322ca83)

The container: directive pulls the ~25-30 GB cuda_multi image with no retry, and nvcr.io intermittently times out under load (context deadline exceeded). Replaced with an explicit docker pull loop (5 attempts, 30s-per-attempt backoff; pulls resume completed layers, so retries are cheap) plus a long-lived container driven by docker exec, which preserves installed packages and env across steps the way container: did. Again the same approach MFC uses.

The workspace is bind-mounted at /workspace, so git is told to trust it to avoid dubious-ownership noise, and the compiler env moves from the job-level env: block onto docker run -e.

Verification

Local only — the real check is this PR's own run. The workflow parses as YAML, and every run: script plus every inner bash -c body passes bash -n under both matrix targets with the ${{ }} expressions substituted. Nothing outside the mfc-nvhpc job changed.

Commit 2 is separable if you would rather land the flag fix alone.

The MFC NVHPC cpu job fails intermittently with an nvfortran backend
crash while compiling m_model.fpp:

  LLVM ERROR: Cannot select: v4f32 = X86ISD::RSQRT14S
    in function '@m_model_s_read_stl_binary_'
  nvfortran-Fatal-.../24.5/compilers/share/llvm/bin/llc TERMINATED by signal 6

nvfortran defaults to -tp=native, so it tunes for whichever host the job
lands on. GitHub's ubuntu-22.04 pool mixes AVX-512-capable and
non-capable CPUs, so on an AVX-512 draw 24.5 emits an RSQRT14S its own
LLVM backend cannot lower. That is why the job passes on some runs and
aborts on others with no change to ffmt.

Pin -tp=px so codegen no longer depends on the runner draw, matching the
flags MFC sets in its own NVHPC CI, and move the container to 26.3, the
newest release MFC exercises upstream.
The container: directive pulls the ~25-30 GB cuda_multi image with no
retry, and nvcr.io intermittently times out under load ("context
deadline exceeded"), turning a registry hiccup into a red build.

Replace it with an explicit docker pull loop (5 attempts, backing off
30s per attempt; pulls resume completed layers, so retries are cheap)
plus a long-lived container driven by docker exec, which preserves
installed packages and env across steps the way container: did. Same
approach MFC uses in its own NVHPC CI.

The workspace is bind-mounted at /workspace, so git is told to trust it
to avoid dubious-ownership noise. Compiler env moves from the job-level
env: block onto docker run -e.
@sbryngelson sbryngelson changed the title CI: bump NVHPC to 26.3, pin generic x86-64 codegen CI: harden the NVHPC jobs (deterministic codegen, retryable image pull) Sep 2, 2026
@sbryngelson
sbryngelson merged commit 7813d2c into master Sep 2, 2026
9 checks passed
@sbryngelson
sbryngelson deleted the ci/nvhpc-bump branch September 2, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant