CI: harden the NVHPC jobs (deterministic codegen, retryable image pull) - #10
Merged
Conversation
The MFC NVHPC cpu job fails intermittently with an nvfortran backend
crash while compiling m_model.fpp:
LLVM ERROR: Cannot select: v4f32 = X86ISD::RSQRT14S
in function '@m_model_s_read_stl_binary_'
nvfortran-Fatal-.../24.5/compilers/share/llvm/bin/llc TERMINATED by signal 6
nvfortran defaults to -tp=native, so it tunes for whichever host the job
lands on. GitHub's ubuntu-22.04 pool mixes AVX-512-capable and
non-capable CPUs, so on an AVX-512 draw 24.5 emits an RSQRT14S its own
LLVM backend cannot lower. That is why the job passes on some runs and
aborts on others with no change to ffmt.
Pin -tp=px so codegen no longer depends on the runner draw, matching the
flags MFC sets in its own NVHPC CI, and move the container to 26.3, the
newest release MFC exercises upstream.
The container: directive pulls the ~25-30 GB cuda_multi image with no
retry, and nvcr.io intermittently times out under load ("context
deadline exceeded"), turning a registry hiccup into a red build.
Replace it with an explicit docker pull loop (5 attempts, backing off
30s per attempt; pulls resume completed layers, so retries are cheap)
plus a long-lived container driven by docker exec, which preserves
installed packages and env across steps the way container: did. Same
approach MFC uses in its own NVHPC CI.
The workspace is bind-mounted at /workspace, so git is told to trust it
to avoid dubious-ownership noise. Compiler env moves from the job-level
env: block onto docker run -e.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three sources of red builds in
MFC NVHPC, none of them ffmt bugs. One commit each.1. Nondeterministic codegen (
4e7f12d)The
cpujob fails intermittently with an nvfortran backend crash while compilingm_model.fpp:nvfortrandefaults to-tp=native, so it tunes for whichever host the job lands on. GitHub'subuntu-22.04pool mixes AVX-512-capable and non-capable CPUs, so on an AVX-512 draw the compiler emits anRSQRT14Sits own LLVM backend cannot lower. Hence green on some runs, aborted on others, with no relevant change to ffmt — the same job failed oncopilot/fix-failing-github-actions-job-teston 2026-09-01, and itsgpu-ompsibling failed on a different run of that branch.MFC compiles the identical source on 24.5 without trouble because it sets
-tp=px. So this pinsFFLAGS=-tp=px -Kieee -noswitcherror,CFLAGS=-tp=px,CXXFLAGS=-tp=px, matching MFC's own NVHPC jobs.2. Stale compiler (
4e7f12d)Container moves
24.5->26.3, the newest release MFC exercises upstream. Its matrix covers every release from 23.11 through 26.3 and all of them are green there, which also confirms the tag resolves. Secondary to the-tppin — a bump alone would still leave codegen depending on the runner draw.3. Unretryable image pull (
322ca83)The
container:directive pulls the ~25-30 GBcuda_multiimage with no retry, andnvcr.iointermittently times out under load (context deadline exceeded). Replaced with an explicitdocker pullloop (5 attempts, 30s-per-attempt backoff; pulls resume completed layers, so retries are cheap) plus a long-lived container driven bydocker exec, which preserves installed packages and env across steps the waycontainer:did. Again the same approach MFC uses.The workspace is bind-mounted at
/workspace, so git is told to trust it to avoid dubious-ownership noise, and the compiler env moves from the job-levelenv:block ontodocker run -e.Verification
Local only — the real check is this PR's own run. The workflow parses as YAML, and every
run:script plus every innerbash -cbody passesbash -nunder both matrix targets with the${{ }}expressions substituted. Nothing outside themfc-nvhpcjob changed.Commit 2 is separable if you would rather land the flag fix alone.