Stop NVHPC IPO silently miscompiling the six-equation relaxation kernel

[sbryngelson]

NVHPC's two-pass IPO (-Mextract/-Minline) miscompiles m_pressure_relaxation in OpenACC device code. Inlining the equation-of-state chain into the six-equation relaxation kernel drops the internal-energy update. There is no diagnostic — the build succeeds and the answer is wrong.

Symptom

Minimal reproducer: one fluid, plain stiffened gas (eos=2, qv=0), model_eqns=3 — test 2F35A1FE.

variable	golden	OpenACC
cons.1 (αρ)	1.0	1.0 ✅
cons.3 (E)	2.5	2.5 ✅
cons.4 (α)	1.0	1.0 ✅
cons.5 (int_en)	2.5	0.0 ❌

Correct at t=0, destroyed during time stepping, in cells that should be quiescent. Elsewhere the corruption is additive with the fractional digits preserved (JWL: 2.57916258853339 → 0.14916258853359, exactly −2.43), which is a dropped term rather than a perturbed one.

Why PRs looked spuriously broken

This has failed 23 model_eqns=3 tests on the NVHPC OpenACC lane since #1811, and because it lives on master every branch cut after it inherits the failures. PR #1865 changed one file — a CI shell script — and failed eos=jwl -> model_eqns=3. Eight unrelated branches fail the identical 23 tests. Docs and CI PRs are hit hardest because non-.fpp changes bypass coverage selection and run the whole suite.

Identifying the cause

The lane split does it:

lane	IPO	result
NVHPC gpu-acc	on	23 failed
NVHPC gpu-omp	off (MFCTargets.cmake:26)	713 passed
CPU, same machine and compiler	—	passes
Frontier CCE gpu-acc	not NVHPC	passes

Rebuilding OpenACC with IPO disabled makes all six reproducible 1D failures pass. The targeted exclusion here does the same while keeping full IPO on the hot-path files (m_rhs, m_riemann_solvers, m_viscous, m_weno).

Ruled out along the way: a race (results are bit-identical across different GPUs), runtime-sized automatic arrays (applied the existing AMD workaround to NVHPC — no change), stale goldens (CPU passes), and missing device data (eoss, eos_coeffs, any_state_dependent_eos, eqn_idx are all correctly GPU_UPDATEd).

Verification

On A100 / NVHPC 25.11, which reproduces the Phoenix V100 / NVHPC 25.5 failure bit for bit (0.80999999999592 against a golden 0.80999999998385):

• 6/6 reproducers pass
• full 1D sweep: 6 tolerance failures → 0, no new failures
• IPO fully disabled and the narrow exclusion both fix it, so the exclusion is sufficient

Verification is 1D only. The rest of the 23 are 2D and 3D (axisymmetric, capillary, viscous IBM, recovering_sphere) and want a Phoenix run to confirm.

Note for reviewers

Third workaround for this pass, after the -Minline except: list and the m_start_up/m_cbc exclusions — and the first where it produced a wrong answer instead of a compile error. Per-file exclusion works until it doesn't, and the pass still covers the hot-path physics files. Whether to keep patching per-file or gate IPO for OpenACC the way OpenMP already is deserves a separate look.

https://claude.ai/code/session_017zrZooJPhZtZYgg9fJiYhg

[Copilot]

🔵 Needs a closer look

Gate the exclusion to OpenACC so CPU builds retain inlining.

Pull request overview

This pull request prevents an NVHPC IPO miscompilation in the pressure-relaxation kernel by excluding it from cross-file inlining.

Changes:

• Adds m_pressure_relaxation to the NVHPC no-inline exclusions.
• Documents the affected internal-energy corruption.

File summaries

File	Description
cmake/MFCTargets.cmake	Configures the targeted NVHPC IPO exclusion.

Review details

Suppressed comments (2)

cmake/MFCTargets.cmake:77

• NVHPC_USE_TWO_PASS_IPO is enabled for non-Unified NVHPC builds regardless of GPU backend (cmake/GPU.cmake:213-219), so this guard also affects the CPU simulation target. That makes the CPU build compile this hot-path module with -Mnoinline even though the reported miscompile is OpenACC-only, unnecessarily losing inlining; gate the new exclusion on MFC_OpenACC AND ARGS_OpenACC.

This issue also appears on line 73 of the same file.

        foreach(_no_inline_file m_start_up m_cbc m_pressure_relaxation)
            set_source_files_properties(
                "${CMAKE_BINARY_DIR}/fypp/${ARGS_TARGET}/${_no_inline_file}.fpp.f90"
                TARGET_DIRECTORY ${ARGS_TARGET}
                PROPERTIES COMPILE_OPTIONS "-Mnoinline"

cmake/MFCTargets.cmake:73

• The verification described for this compiler workaround covers only the 1D reproducers, while the reported regression set also contains 2D and 3D cases. Since this changes compile flags for the shared simulation module and is intended to fix all 23 failures, please run the remaining failing cases on the Phoenix NVHPC/OpenACC lane before relying on the exclusion; a 1D pass does not validate those execution paths.

        foreach(_no_inline_file m_start_up m_cbc m_pressure_relaxation)

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sbryngelson]

Closing the 1D-only caveat in the description: I have now run the full 2D and 3D suites against this fix on A100 / NVHPC 25.11.

suite	passed	tolerance mismatches
1D	175	0
2D	289	0
3D	224	0

That covers all 23, not just the six I could reproduce in 1D — the 2D half (axisymmetric, capillary, viscous IBM for one and two fluids, isentropicvortex, rayleigh_taylor, riemann_solver=2) and the 3D half (recovering_sphere, capillary, both riemann_solver=2 cases) included.

The 11 failures across those runs (9 in 2D, 2 in 3D) are all Chemistry cases dying at syscheck with exit 132, SIGILL from a stale chemistry-variant build on this machine. They fail identically on an unmodified build, and none of the 23 are among them.

One thing worth recording separately: no benchmark or example case uses model_eqns=3 — all seven benchmarks are model_eqns=2. So s_pressure_relaxation_procedure has correctness coverage (12 test cases) but no performance coverage at all, which is why the cost of this exclusion was not measurable from the existing suite. I am measuring it on a purpose-built 3D 6-equation case and will post the number here; if it is material, the narrower fix is to add the equation-of-state routines to the -Minline except: list instead, which would keep the module itself optimised.