Skip to content

Pin a baseline CPU target for Phoenix NVHPC builds #1864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sbryngelson wants to merge 1 commit into from
+5 −0
Closed

Pin a baseline CPU target for Phoenix NVHPC builds #1864

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions toolchain/modules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ p-gpu OPAL_PREFIX=$NVHPC_ROOT/comm_libs/12.9/hpcx/hpcx-2.22.1/ompi PMIX_MCA_gds=
p-gpu PATH=$NVHPC_ROOT/comm_libs/mpi/bin:$NVHPC_ROOT/cuda/12.9/bin:$PATH
p-gpu LD_LIBRARY_PATH=$NVHPC_ROOT/math_libs/12.9/targets/x86_64-linux/lib:$NVHPC_ROOT/cuda/12.9/targets/x86_64-linux/lib:$NVHPC_ROOT/cuda/12.9/lib64:$NVHPC_ROOT/compilers/lib:$LD_LIBRARY_PATH
p-gpu CC=$NVHPC_ROOT/comm_libs/mpi/bin/mpicc CXX=$NVHPC_ROOT/comm_libs/mpi/bin/mpicxx FC=$NVHPC_ROOT/comm_libs/mpi/bin/mpifort
# Phoenix's compute nodes are heterogeneous and a job's build and run can land on different ones
# (the case-optimization lane submits the prebuild and the run as separate allocations), so a
# natively targeted binary can meet a node that lacks an instruction it uses and die with SIGILL.
# preflight.sh probes with syscheck, which is too small to trip it. Pin a baseline instead.
p-gpu FFLAGS=-tp=px CFLAGS=-tp=px CXXFLAGS=-tp=px

@sbryngelson sbryngelson Sep 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch to check - cmake/GPU.cmake:186-207 does add -march=native to every Fortran target in Release, and nvfortran accepts the flag (rc 0), so CHECK_FORTRAN_COMPILER_FLAG succeeds and it lands on the command line after CMAKE_Fortran_FLAGS. But measured on nvfortran 25.5, -tp wins over a later -march=native rather than the other way round.

Same subroutine, -O3 -fast, object disassembled:

flags AVX-style mnemonics ymm/zmm md5
-tp=px 26 22 ee8f3042
-tp=zen3 16 19 71b4755c
-march=native 16 19 61e15704
-tp=px -march=native 26 22 5f21ba55

-march=native on its own reproduces -tp=zen3 exactly on this Zen 3 host, so it is honoured rather than ignored - but combined with -tp=px the emitted ISA profile is identical to -tp=px alone. So the baseline is not defeated here.

Worth adding what the same measurement says about whether this PR would help at all: neither target emits AVX-512 on this host (zmm count 0 for both), and -tp=px still emits ymm, so "px" is not an SSE2 baseline. It caps the ISA rather than eliminating vectorisation, which is the relevant part - a build host with AVX-512 would emit zmm under native tuning and those are the instructions an older node would fault on.

That said, #1865 excludes the offending node directly, at no performance cost and following the existing node_exclude precedent, so it is the better immediate fix. This PR is the systemic version and carries a real cost (Phoenix Bench numbers will move), so it should probably wait on whether the exclusion holds.

p-gpu-unload xalt

pifx GT Phoenix (IFX)
Expand Down
Loading