Installation issue: PETSc with optimization flags

[pzehner]

Steps to reproduce the issue

$ spack add petsc@3.25.2 cflags=-O3 cxxflags=-O3 fflags=-O3
$ spack spec -I
 -   petsc@3.25.2 cflags=-O3 cxxflags=-O3 fflags=-O3 ~X~batch~cgns~complex~cuda~debug+double+examples~exodusii~fftw+fortran+fortran-bindings~giflib~hdf5~hpddm~hwloc~hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind~metis~mkl-pardiso~ml~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse~superlu-dist~sycl~tetgen~valgrind~zoltan build_system=generic clanguage=C memalign=none platform=linux os=ubuntu24.04 target=skylake %c,cxx,fortran=gcc@13.3.0
[+]      ^compiler-wrapper@1.1.0 build_system=generic platform=linux os=ubuntu24.04 target=skylake 
[e]      ^diffutils@3.10 build_system=autotools platform=linux os=ubuntu24.04 target=x86_64 
[e]      ^gcc@13.3.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=ubuntu24.04 target=x86_64 
[+]      ^gcc-runtime@13.3.0 build_system=generic platform=linux os=ubuntu24.04 target=skylake 
[e]      ^glibc@2.39 build_system=autotools platform=linux os=ubuntu24.04 target=x86_64 
[e]      ^gmake@4.3~guile build_system=generic platform=linux os=ubuntu24.04 target=x86_64 
[e]      ^mpich@4.2.0~argobots~cuda+fortran~hcoll+hwloc+hydra~level_zero+libxml2+pci~rocm+romio~slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ucx pmi=pmix platform=linux os=ubuntu24.04 target=x86_64 
[+]      ^openblas@0.3.29~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none platform=linux os=ubuntu24.04 target=skylake %c,cxx,fortran=gcc@13.3.0
[+]          ^compiler-wrapper@1.0 build_system=generic platform=linux os=ubuntu24.04 target=skylake 
[+]          ^gcc-runtime@13.3.0 build_system=generic platform=linux os=ubuntu24.04 target=skylake 
[e]      ^python@3.12.3+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~static~tests+tix+tkinter+uuid+zlib build_system=generic platform=linux os=ubuntu24.04 target=x86_64 

Same with fflags=-O2 and fflags=-O1.

Error message

The error comes from too long lines in Fortran files which are generated during the build. As per the Fortran 90 standard, lines are limited 132 characters, when for the Fortran 2023 standard, the limitation is 10000 characters. GCC 13.3.0 does not implement the Fortran 2023 standard.

Error message

  /usr/bin/mpicc -c -O3 -fPIC  -I/tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/include -I/tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include    -MMD -MP /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/src/sys/utils/sorti.c -o arch-linux-c-opt/obj/src/sys/utils/sorti.o
  /usr/bin/mpif90 -c -O3 -fPIC    -I/tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/include -I/tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include    -MMD -MP /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/src/vec/ftn-mod/petscvecmod.F90 -o arch-linux-c-opt/obj/src/vec/ftn-mod/petscvecmod.o -J/tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:559:132:
  
    559 |   subroutine PetscSFCreateByMatchingIndices(layout,numRootIndices,rootIndices,rootLocalIndices,rootLocalOffset,numLeafIndices,leafIndices,leafLocalIndices,leafLocalOffset,sfA,sf, ierr)
        |                                                                                                                                    1
> Error: Line truncated at (1) [-Werror=line-truncation]
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:559:132:
  
    559 |   subroutine PetscSFCreateByMatchingIndices(layout,numRootIndices,rootIndices,rootLocalIndices,rootLocalOffset,numLeafIndices,leafIndices,leafLocalIndices,leafLocalOffset,sfA,sf, ierr)
        |                                                                                                                                    1
> Error: Unexpected junk in formal argument list at (1)
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:560:33:
  
    560 |   use, intrinsic :: ISO_C_binding
        |                                 1
> Error: Unexpected USE statement in INTERFACE block at (1)
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:561:9:
  
    561 |   import tPetscLayout,tPetscSF
        |         1
> Error: IMPORT statement at (1) only permitted in an INTERFACE body
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:562:30:
  
    562 |   PetscLayout :: layout
        |                              1
> Error: Unexpected data declaration statement in INTERFACE block at (1)
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:563:15:
  
    563 |   PetscInt :: numRootIndices
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:564:15:
  
    564 |   PetscInt :: rootIndices(*)
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:565:15:
  
    565 |   PetscInt :: rootLocalIndices(*)
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:566:15:
  
    566 |   PetscInt :: rootLocalOffset
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:567:15:
  
    567 |   PetscInt :: numLeafIndices
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:568:15:
  
    568 |   PetscInt :: leafIndices(*)
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:569:15:
  
    569 |   PetscInt :: leafLocalIndices(*)
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:570:15:
  
    570 |   PetscInt :: leafLocalOffset
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:571:23:
  
    571 |   PetscSF :: sfA
        |                       1
> Error: Unexpected data declaration statement in INTERFACE block at (1)
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:572:22:
  
    572 |   PetscSF :: sf
        |                      1
> Error: Unexpected data declaration statement in INTERFACE block at (1)
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:573:15:
  
    573 |   PetscErrorCode, intent(out) :: ierr
        |               1
> Error: Parameter 'c_int32_t' at (1) has not been declared or is a variable, which does not reduce to a constant expression
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/arch-linux-c-opt/include/../ftn/vec/petscsf.h90:574:5:
  
    574 |   end subroutine
        |     1
> Error: Expecting END INTERFACE statement at (1)
  /tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/src/vec/ftn-mod/petscvecmod.F90:69:7:
  
     69 |   use petscis
        |       1
> Fatal Error: Cannot open module file 'petscis.mod' for reading at (1): No such file or directory
  f951: some warnings being treated as errors
  compilation terminated.
> make[3]: *** [gmakefile:236: arch-linux-c-opt/obj/src/vec/ftn-mod/petscvecmod.o] Error 1
> make[2]: *** [/tmp/<<>>/spack-stage/spack-stage-petsc-3.25.2-pobhebm3xj7t24o5i6qxfx4jo5nzxfnq/spack-src/lib/petsc/conf/rules_doc.mk:5: libs] Error 2
  **************************ERROR*************************************
    Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log
    Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to petsc-maint@mcs.anl.gov
  ********************************************************************
> make[1]: *** [makefile:44: all] Error 1
> make: *** [GNUmakefile:9: all] Error 2
  Command exited with status 2:
      /usr/bin/make V=1

Information on your system

• Spack: 1.2.2
• Builtin repo: d4f7c71
• Python: 3.12.3
• Platform: linux-ubuntu24.04-skylake

Additional information

Note that the problem only occurs when passing the optimization flags, which is weird.

The problem is also related to the fact that PETSc is built without optimization flags by default by Spack. As stated in the install directions of PETSc:

optimized install - spack install petsc cflags='-g -O3 -march=native -mtune=native' fflags='-g -O3 -march=native -mtune=native' cxxflags='-g -O3 -march=native -mtune=native'

This error does not take place for PETSc 3.22.5.

I could circumvent the problem for GCC by adding -ffree-line-length-none to fflags, but this is not portable.

spack-build-env.txt
spack-build-out.txt

@balay, @jczhang07 , @joseeroman

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
• I have uploaded the build log and environment files
• I have searched the issues of this repo and believe this is not a duplicate

[jczhang07]

@balay could you have a look?

[balay]

I could circumvent the problem for GCC by adding -ffree-line-length-none to fflags, but this is not portable.

yeah - but this is the current workaround.

PETSc configure interface is a bit complex - we have FFLAGS (add to - or replace defaults) and FOPTFLAGS (override defaults) - and what you want is to keep the current configure defaults for FFLAGS - but over-ride FOPTFLAGS.

unfortunately - its not clear (to me) how to translate this to spack interface. So currently fflags is the spack interface - that always over-rides the defaults [so are having to add back -ffree-line-length-none for gfortran].

[balay]

my suggestion is to skip listing fflags it won't make a difference (as PETSc library is C, fortran is used a bit to support the fortran interface- primarily compile the module interface sources)

[pzehner]

I guess this is a naive question, but would it be possible to improve the interface between Spack and PETSc?

[balay]

Not sure if this is an improvement - [probably shifts the problem space] - but this change might improve on this use-case

diff --git a/repos/spack_repo/builtin/packages/petsc/package.py b/repos/spack_repo/builtin/packages/petsc/package.py
index 608cd71226..0042dcad64 100644
--- a/repos/spack_repo/builtin/packages/petsc/package.py
+++ b/repos/spack_repo/builtin/packages/petsc/package.py
@@ -453,18 +453,17 @@ class Petsc(Package, CudaPackage, ROCmPackage):
             "--download-hwloc=0",
             "--with-make-exec=%s" % make,
         ]
-        # If 'cflags', 'fflags', and/or 'cxxflags' are not set, let the PETSc
-        # configuration script choose defaults.
+        # If 'cflags', 'fflags', and/or 'cxxflags' are set, add to PETSc configure defaults
         if spec.compiler_flags["cflags"]:
-            options += ["CFLAGS=%s" % " ".join(spec.compiler_flags["cflags"])]
+            options += ["CFLAGS+=%s" % " ".join(spec.compiler_flags["cflags"])]
             if "+debug" not in spec:
                 options += ["COPTFLAGS="]
         if spec.compiler_flags["fflags"]:
-            options += ["FFLAGS=%s" % " ".join(spec.compiler_flags["fflags"])]
+            options += ["FFLAGS+=%s" % " ".join(spec.compiler_flags["fflags"])]
             if "+debug" not in spec:
                 options += ["FOPTFLAGS="]
         if spec.compiler_flags["cxxflags"]:
-            options += ["CXXFLAGS=%s" % " ".join(spec.compiler_flags["cxxflags"])]
+            options += ["CXXFLAGS+=%s" % " ".join(spec.compiler_flags["cxxflags"])]
             if "+debug" not in spec:
                 options += ["CXXOPTFLAGS="]
         options.extend(self.mpi_dependent_options())

[rbberger]

I suggest doing this and remove the CFLAGS, FFLAGS and CXXFLAGS parts and add this:

diff --git a/repos/spack_repo/builtin/packages/petsc/package.py b/repos/spack_repo/builtin/packages/petsc/package.py
index d5652abc9c..fb458cdecb 100644
--- a/repos/spack_repo/builtin/packages/petsc/package.py
+++ b/repos/spack_repo/builtin/packages/petsc/package.py
@@ -443,6 +443,9 @@ class Petsc(Package, CudaPackage, ROCmPackage):
                 compiler_opts.append("--FC_LINKER_FLAGS=-lintlc")
         return compiler_opts

+    def flag_handler(self, name, flags):
+        return (flags, None, None)
+
     def configure_options(self):
         spec = self.spec
         options = [
@@ -452,20 +455,6 @@ class Petsc(Package, CudaPackage, ROCmPackage):
             "--download-hwloc=0",
             "--with-make-exec=%s" % make,
         ]
         # If 'cflags', 'fflags', and/or 'cxxflags' are not set, let the PETSc
         # configuration script choose defaults.
         if spec.compiler_flags["cflags"]:
-           options += ["CFLAGS=%s" % " ".join(spec.compiler_flags["cflags"])]
             if "+debug" not in spec:
                 options += ["COPTFLAGS="]
         if spec.compiler_flags["fflags"]:
-            options += ["FFLAGS=%s" % " ".join(spec.compiler_flags["fflags"])]
             if "+debug" not in spec:
                 options += ["FOPTFLAGS="]
         if spec.compiler_flags["cxxflags"]:
-            options += ["CXXFLAGS=%s" % " ".join(spec.compiler_flags["cxxflags"])]
              if "+debug" not in spec:
                options += ["CXXOPTFLAGS="]
         options.extend(self.mpi_dependent_options())
         options.extend(
             [

[rbberger]

@balay in my testing the += didn't work when building with Spack.

[pzehner]

@rbberger I'm not sure to understand what your proposition would do?

[rbberger]

@pzehner the Spack compiler wrapper injects the flags, the PETSc build system doesn't see them. This leaves the PETSc defaults intact, while allowing you to define additional flags that get directly injected during compilation.