GROMACS, libCEED: CPU Arch target not respected

[wspear]

I'm not sure if this is a packaging issue or a Spack core issue, but we've found some packages that don't respect the assigned CPU architecture target. With gromacs and libceed, as specced below, targeting x86_64_v3, the binaries end up with symbols that should only be present with AVX-512 compatibility enabled (this was the case on the machine where these were built, but that shouldn't matter). We need need to be able to install these from the binary repo and run on a ZEN2. They instead give 'illegal instruction' errors on that architecture.

spack find -dvl gromacs
-- linux-ubuntu24.04-x86_64_v3 / %c,cxx=gcc@13.3.0 --------------
2zq2kv3 gromacs@2026.1~cp2k+cuda~cufftmp~cycle_subcounters~double+gmxapi~heffte+hwloc~intel_provided_gcc~ipo~itt~mdrun_only+mpi+nblib~nosuffix~nvshmem~opencl+openmp~relaxed_double_precision+shared~sycl build_system=cmake build_type=Release cuda_arch:=80 generator=make openmp_max_threads=none
k3pbdgj     cmake@3.31.11~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release
lnn2jzl         curl@8.18.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl
f3ioizb             nghttp2@1.67.1 build_system=autotools
jv7fosj                 diffutils@3.12 build_system=autotools
g3gqygu             openssl@3.6.1~docs+shared build_system=generic certs=mozilla
sghkyxj                 ca-certificates-mozilla@2026-03-19 build_system=generic
gixftjv                 perl@5.42.0+cpanm+opcode+open+shared+threads build_system=generic
halbeor                     berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc
fdeyzed                     bzip2@1.0.8~debug~pic+shared build_system=generic
ix4abkw                     gdbm@1.26 build_system=autotools
r67tgvx                         readline@8.3 build_system=autotools patches:=21f0a03
4zop7nq         ncurses@6.6~symlinks+termlib abi=none build_system=autotools patches:=7a351bc
fp3yliw         zlib-ng@2.3.3+compat+new_strategies+opt+pic+shared build_system=autotools
gaqs4ab     compiler-wrapper@1.0 build_system=generic
irnmc53     cuda@12.9.1~allow-unsupported-compilers~dev build_system=generic
cias5wd     fftw@3.3.11+mpi~openmp~pfft_patches+shared build_system=autotools precision:=double,float
mob5yjf     gcc@13.3.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran'
dwcz77v     gcc-runtime@13.3.0 build_system=generic
jj453jr     glibc@2.39 build_system=autotools
27tbysx     gmake@4.4.1~guile build_system=generic
ptficeo     hwloc@2.13.0~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static patches:=b4db98b
tsg6jbt         libpciaccess@0.17 build_system=autotools
p7bulzn             util-macros@1.20.2 build_system=autotools
tpadtin         libxml2@2.15.1+pic~python+shared build_system=autotools
ckvl7de             libiconv@1.18 build_system=autotools libs:=shared,static
g6zrvo4             xz@5.8.3~pic build_system=autotools libs:=shared,static
2bzx5cg     mpich@4.3.2~argobots~cuda+fortran~hwloc+hydra~level_zero+libxml2+pci~rocm+romio~slurm~vci~verbs~wrapperrpath~xpmem build_system=autotools datatype-engine=auto device:=ch4 netmod:=ofi pmi=default
dkhrs2e     openblas@0.3.32~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared~static build_system=makefile symbol_suffix=none threads=openmp
3a4iwvp     pkgconf@2.5.1 build_system=autotools

==> 1 installed package
objdump -d $(spack location -i gromacs)/lib/libgromacs_mpi.so | grep -m5 -E 'zmm|k[0-7],|vpternlog|vpermt2|evex'
 1165371:       62 f1 7d 48 ef c0       vpxord %zmm0,%zmm0,%zmm0
 1165377:       62 f1 7c 48 28 c8       vmovaps %zmm0,%zmm1
 116537d:       62 f1 7c 48 28 d0       vmovaps %zmm0,%zmm2
 1165383:       62 f1 7c 48 28 d8       vmovaps %zmm0,%zmm3
 1165389:       62 f1 7c 48 28 e0       vmovaps %zmm0,%zmm4

spack find -dvl libceed
-- linux-ubuntu24.04-x86_64_v3 / %c,cxx,fortran=gcc@13.3.0 ------
c5q3fsg libceed@0.12.0+cuda~debug~libxsmm~magma~occa~openmp~rocm+shared build_system=makefile commit=4018a20a98d451fac24765d3ddb936861647ce8d cuda_arch:=80
gaqs4ab     compiler-wrapper@1.0 build_system=generic
irnmc53     cuda@12.9.1~allow-unsupported-compilers~dev build_system=generic
mob5yjf     gcc@13.3.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran'
dwcz77v     gcc-runtime@13.3.0 build_system=generic
jj453jr     glibc@2.39 build_system=autotools
27tbysx     gmake@4.4.1~guile build_system=generic

==> 1 installed package

objdump -d $(spack location -i gromacs)/lib/libgromacs_mpi.so | grep -m5 -E 'zmm|k[0-7],|vpternlog|vpermt2|e^Cx'
root@edf5642e5c2c:/# ^C
root@edf5642e5c2c:/# objdump -d $(spack location -i libceed)/lib/libceed.so | grep -m5 -E 'zmm|k[0-7],|vpternlog|vpermt2|evex'
   11250:       62 f1 fd 48 10 bc 30    vmovupd 0x8(%rax,%rsi,1),%zmm7
   1125b:       62 f1 c5 48 59 3c 31    vmulpd (%rcx,%rsi,1),%zmm7,%zmm7
   11284:       62 f3 fd 48 1b f9 01    vextractf64x4 $0x1,%zmm7,%ymm1
   11412:       62 f2 fd 48 19 f0       vbroadcastsd %xmm0,%zmm6
   11420:       62 f1 fd 48 10 8c 30    vmovupd 0x8(%rax,%rsi,1),%zmm1

spack debug report
* **Spack:** 1.1.1 (https://github.com/spack/spack/commit/2e2169d5282d166f63e3ee4db8d4446c43cefa8a)
* **Builtin repo:** https://github.com/spack/spack-packages/commit/cb2b3bc26f9f4ff9d5ef261dda60e395cb88aa04
* **Python:** 3.12.12
* **Platform:** linux-ubuntu24.04-zen2

@eugeneswalker @jedbrown @jeremylt @tzanio @v-dobrev @eirrgang @junghans @mabraham

[jedbrown]

There are a lot of packages that use the likes of -march=native. Does spack have specific guidance on how arch-specific flags should be set, or is the correct thing simply to purge -march=native from all package files?

$  git grep -l march=native | wc
     41      41    2455

[eirrgang]

Are you using target=x86_64_v3 and it isn't doing what it is supposed to?

There might still exist "wrong ways" to propagate CFLAGS that spack can't detect (to uniquely identify specs), but the cflags and cxxflags could be specified explicitly for any or all specs.

I guess it's worth checking that spack is doing something reasonable with that target and that whatever it's doing isn't getting stomped on by either the package recipe or the CMakeLists (which is not uncommon).

[eugeneswalker]

Are you using target=x86_64_v3 and it isn't doing what it is supposed to?

Chiming in here since @wspear and I encountered this issue together. Yes, we are building gromacs and libceed in a Spack environment that requires target=x86_64_v3 for all specs:

• https://github.com/E4S-Project/e4s/blob/master/environments/x86_64/gnu/cpu/spack.yaml#L51
• https://github.com/E4S-Project/e4s/blob/master/environments/x86_64/gnu/_config/packages.yaml#L4-L6
• libceed: https://github.com/E4S-Project/e4s/blob/master/environments/x86_64/gnu/cpu/spack.yaml#L68
• gromacs: https://github.com/E4S-Project/e4s/blob/master/environments/x86_64/gnu/cpu/spack.yaml#L51

We noticed the issue because we were building binaries to push to build cache on one machine architecture, and then testing them on another, which should have worked if x86_64_v3 had been respected. Thank you for looking at this!

[jeremylt]

I think Jed's question is at the core of it. march=native can create issues when compiling on one machine for running on another machine. So what does Spack want here?

[jedbrown]

And whatever the recommended solution is, it should be applied to all (at least 41) packages that invoke -march=native.

[wspear]

I have left a query about the right approach here on the Spack Slack. I haven't dug too deeply yet but it looks like at least a few of those -march=native matches are from patches and filter-file calls that remove that argument from build commands, presumably to enable Spack's own arch specification.