Investigate build in aarch64/arm64 on Apple Silicon

[marcindulak]

The project does not produce (yet?) an image bootable with qemu-system-aarch64 on Apple Silicon. Maybe the errors and notes provided here can be useful later.

Some observations:

1. The first indication that something is off is in cat $LFS/jhalfs/logs/505-gcc-libstdc++-12.2.0 of 505-gcc-libstdc++

   rm: cannot remove '/vagrant/jhalfs/mnt/build_dir/usr/lib/libstdc++.la': No such file or directory
   rm: cannot remove '/vagrant/jhalfs/mnt/build_dir/usr/lib/libstdc++fs.la': No such file or directory
   rm: cannot remove '/vagrant/jhalfs/mnt/build_dir/usr/lib/libsupc++.la': No such file or directory

   It can be notes that the la files are being written into the usr/lib4 folder instead.
   For now the error can be ignored by modifying $LFS/jhalfs/lfs-commands/chapter05/505-gcc-libstdc++

    rm -vf $LFS/usr/lib/lib{stdc++,stdc++fs,supc++}.la

   At the end of the lfs build, the following files may be present

    ls ./usr/lib64/*.la
    ./usr/lib64/libasan.la    ./usr/lib64/libgomp.la    ./usr/lib64/libssp.la            ./usr/lib64/libsupc++.la
    ./usr/lib64/libatomic.la  ./usr/lib64/libhwasan.la  ./usr/lib64/libssp_nonshared.la  ./usr/lib64/libtsan.la
    ./usr/lib64/libcc1.la     ./usr/lib64/libitm.la     ./usr/lib64/libstdc++.la         ./usr/lib64/libubsan.la
   ./usr/lib64/libffi.la     ./usr/lib64/liblsan.la    ./usr/lib64/libstdc++fs.la

2. The next issue appears for 803-glibc-2.36, where it hangs at make -k check.

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root     15626  0.0  0.0   6912  3072 pts/2    Ss+  21:14   0:00 su - vagrant -c source /vagrant/jhalfs/jhalfs.sh && cd $LFS/jhalfs && make mk_CHROOT
    vagrant  15632  0.0  0.0   2700  2060 ?        Ss   21:14   0:00  \_ make mk_CHROOT
    vagrant  19070  0.0  0.0   3872  1292 ?        S    21:50   0:00      \_ /bin/bash -c ( sudo /usr/sbin/chroot /vagrant/jhalfs/mnt/build_dir /usr/bin/env -i HOME=/root TERM="$TERM" PS1='(lfs chroot) \u:\w\$ ' PATH=/usr/bin:/usr/sbin /bin/bash --login -c "cd jhalfs && make BREAKPOINT= CHROOT")
    root     19071  0.0  0.0   7820  3912 ?        S    21:50   0:00          \_ sudo /usr/sbin/chroot /vagrant/jhalfs/mnt/build_dir /usr/bin/env -i HOME=/root TERM=xterm PS1=(lfs chroot) \u:\w\$  PATH=/usr/bin:/usr/sbin /bin/bash --login -c cd jhalfs && make BREAKPOINT= CHROOT
    root     19072  0.0  0.0   2620  1844 ?        S    21:50   0:00              \_ make BREAKPOINT= CHROOT
    root      2118  0.0  0.0   3960  2388 ?        S    21:58   0:00                  \_ /bin/bash -c source envars && \ /jhalfs/lfs-commands/chapter08/803-glibc >>/jhalfs/logs/803-glibc-2.36 2>&1
    root      2119  0.0  0.0   3960  2808 ?        S    21:58   0:00                      \_ /bin/bash /jhalfs/lfs-commands/chapter08/803-glibc
    root     11111  0.0  0.0   2516  1764 ?        S    22:01   0:00                          \_ make -k check
    root     15444  0.0  0.0   4136  3440 pts/1    Ss   21:11   0:00 bash
    root     24377  0.0  0.0   6408  1640 pts/1    R+   22:25   0:00  \_ ps faux
    root         1  0.0  0.0   2308   752 pts/0    Ss+  21:08   0:00 sh -c  sleep infinity
    root         7  0.0  0.0   2204   764 pts/0    S+   21:08   0:00 sleep infinity
    root      2097  0.0  0.0   3960  1948 ?        S    21:58   0:00 /bin/bash -c export BASHBIN=/bin/bash && /bin/bash progress_bar.sh 803-glibc $PPID &
    root      2099  0.3  0.0   3960  2816 ?        S    21:58   0:04  \_ /bin/bash progress_bar.sh 803-glibc 19072
    root     24376  0.0  0.0   2156   236 ?        S    22:25   0:00      \_ /bin/sleep .12
    root      5040  0.0  0.0   2324   220 ?        S    22:07   0:00 /sources/glibc-2.36/build/elf/ld-linux-aarch64.so.1 --library-path /sources/glibc-2.36/build:/sources/glibc-2.36/build/math:/sources/glibc-2.36/build/elf:/sources/glibc-2.36/build/dlfcn:/sources/glibc-2.36/build/nss:/sources/glibc-2.36/build/nis:/sources/glibc-2.36/build/rt:/sources/glibc-2.36/build/resolv:/sources/glibc-2.36/build/mathvec:/sources/glibc-2.36/build/support:/sources/glibc-2.36/build/crypt:/sources/glibc-2.36/build/nptl /sources/glibc-2.36/build/nptl/tst-cancel7 --direct --sem /tmp/tst-cancel7.l14yVI --pidfile /tmp/tst-cancel7-pid-juVafC
    root     29992  0.0  0.0   2324   224 ?        S    22:09   0:00 /sources/glibc-2.36/build/elf/ld-linux-aarch64.so.1 --library-path /sources/glibc-2.36/build:/sources/glibc-2.36/build/math:/sources/glibc-2.36/build/elf:/sources/glibc-2.36/build/dlfcn:/sources/glibc-2.36/build/nss:/sources/glibc-2.36/build/nis:/sources/glibc-2.36/build/rt:/sources/glibc-2.36/build/resolv:/sources/glibc-2.36/build/mathvec:/sources/glibc-2.36/build/support:/sources/glibc-2.36/build/crypt:/sources/glibc-2.36/build/nptl /sources/glibc-2.36/build/nptl/tst-cancelx7 --direct --sem /tmp/tst-cancel7.k5qiF4 --pidfile /tmp/tst-cancel7-pid-AILVrb

   When kill -15 11111 is used, the target proceeds and finishes the without errors.
   The logs cat $LFS/jhalfs/test-logs/803-glibc-2.36 shows the tests finished

   Summary of test results:
       169 FAIL
      4440 PASS
        51 UNSUPPORTED
        16 XFAIL
         2 XPASS
   make[2]: *** [Makefile:647: tests] Error 1
   make[2]: Target 'check' not remade because of errors.
   make[2]: Leaving directory '/sources/glibc-2.36'
   make[1]: *** [Makefile:9: check] Error 2

   and differ slightly vs the corresponding log on x86_64/amd64 (which does not require kill)

   Summary of test results:
         5 FAIL
      5032 PASS
       124 UNSUPPORTED
        16 XFAIL
         4 XPASS
   make[2]: *** [Makefile:647: tests] Error 1
   make[2]: Target 'check' not remade because of errors.
   make[2]: Leaving directory '/sources/glibc-2.36'
   make[1]: *** [Makefile:9: check] Error 2
   make[1]: Leaving directory '/sources/glibc-2.36/build'

3. The next error appears in 814-expect. The build log $LFS/jhalfs/logs/814-expect-5.45.4 shows the build platform $(uname -m) == aarch64) is uknown to expect

   ...
   checking if running Sequent running SVR4... no
   checking build system type... tclconfig/config.guess: unable to guess system type
   
   This script, last modified 2003-10-07, has failed to recognize
   the operating system you are using. It is advised that you
   download the most up to date version of the config scripts from
   
    ftp://ftp.gnu.org/pub/gnu/config/
    
   If the version you run (tclconfig/config.guess) is already up to date, please
   send the following data and any information you think might be
   pertinent to <config-patches@gnu.org> in order to provide the needed
   information to handle your system.
   
   config.guess timestamp = 2003-10-07
   
   uname -m = aarch64
   uname -r = 5.15.64-0-virt
   uname -s = Linux
   uname -v = #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000
   
   /usr/bin/uname -p = unknown
   /bin/uname -X     =
   
   hostinfo               =
   /bin/universe          =
   /usr/bin/arch -k       =
   /bin/arch              =
   /usr/bin/oslevel       =
   /usr/convex/getsysinfo =
   
   UNAME_MACHINE = aarch64
   UNAME_RELEASE = 5.15.64-0-virt
   UNAME_SYSTEM  = Linux
   UNAME_VERSION = #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000
   configure: error: cannot guess build type; you must specify one

   This can be remediated by adjusting $LFS/jhalfs/fs-commands/chapter08/814-expect to use arm (not arm64)

    ./configure
            ...
            --build=arm
            ...

4. The next error happens in 824-gcc, the log cat jhalfs/logs/824-gcc-12.2.0 shows the build fails to find py files under ./user/lib

   ...
   '/usr/lib/bfd-plugins/liblto_plugin.so' -> '../../libexec/gcc/aarch64-unknown-linux-gnu/12.2.0/liblto_plugin.so'
   mkdir: created directory '/usr/share/gdb'
   mkdir: created directory '/usr/share/gdb/auto-load'
   mkdir: created directory '/usr/share/gdb/auto-load/usr'
   mkdir: created directory '/usr/share/gdb/auto-load/usr/lib'
   mv: cannot stat '/usr/lib/*gdb.py': No such file or directory

   Let's ignore the error by commenting out the line mv -v /usr/lib/*gdb.py /usr/share/gdb/auto-load/usr/lib in $LFS/jhalfs/lfs-commands/chapter08/824-gcc and proceed.

5. The next issue appears in 847-libffi, but can only be noticed when trying to build 851-meson, whose log cat $LFS/jhalfs/logs/851-meson-0.63.2 shows

   Processing /sources/meson-0.63.2
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
   
    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [64 lines of output]
        /usr/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
          warnings.warn(msg, warning_class)
        running dist_info
        creating /tmp/pip-modern-metadata-_hbvr78y/meson.egg-info
        ...
          File "/usr/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 26, in <module>
            from .macosx_libfile import calculate_macosx_platform_tag
          File "/usr/lib/python3.10/site-packages/wheel/macosx_libfile.py", line 41, in <module>
            import ctypes
          File "/usr/lib/python3.10/ctypes/__init__.py", line 8, in <module>
            from _ctypes import Union, Structure, Array
        ModuleNotFoundError: No module named '_ctypes'
        [end of output]
   
    note: This error originates from a subprocess, and is likely not a problem with pip.
    error: metadata-generation-failed
   ....

   As a workaroun libffi can to configured with --disable-multi-os-directory flag in $LFS/jhalfs/lfs-commands/chapter08/847-libffi, rebuilt, then python also rebuilt. Meson should then install without any modifications.

6. The next build issue appears for 856-findutils whose build log cat $LFS/jhalfs/logs/856-findutils-4.9.0 reports there is no Makefile

   make[1]: Entering directory '/sources/findutils-4.9.0'
   There seems to be no Makefile in this directory.
   You must run ./configure before running 'make'.
   make[1]: *** [GNUmakefile:108: abort-due-to-no-makefile] Error 1
   make[1]: Leaving directory '/sources/findutils-4.9.0'

   This can be remediated in $LFS/jhalfs/lfs-commands/chapter08/856-findutils which needs the aarch64 platform added x86_64|aarch64) ./configure --prefix=/usr --localstatedir=/var/lib/locate ;;.

7. At the time the error is found in 873-procps-ng-4.0.0 log cat $LFS/jhalfs/logs/873-procps-ng-4.0.0 it becomes clear that many libraries are placed in ./usr/lib64 and this location is not used for configure

   checking for libsystemd-login... no
   configure: error: Package requirements (libsystemd-login) were not met:
   
   No package 'libsystemd-login' found
   
   Consider adjusting the PKG_CONFIG_PATH environment variable if you
   installed software in a non-standard prefix.
   
   Alternatively, you may set the environment variables SYSTEMD_CFLAGS
   and SYSTEMD_LIBS to avoid the need to call pkg-config.
   See the pkg-config man page for more details.

   As a drastic workaround, linking everthing present under ./usr/lib64 can be considered

   cd $LFS/usr/lib
   for file in $(find ../lib64 -maxdepth 1 -type f); do ln -sv $file .; done
   './libtsan.a' -> '../lib64/libtsan.a'
   './libhwasan.so.0.0.0' -> '../lib64/libhwasan.so.0.0.0'
   './libstdc++.la' -> '../lib64/libstdc++.la'
   './libasan.a' -> '../lib64/libasan.a'
   './libasan.so.8.0.0' -> '../lib64/libasan.so.8.0.0'
   './libgcc_s.so' -> '../lib64/libgcc_s.so'
   './libsanitizer.spec' -> '../lib64/libsanitizer.spec'
   './libgomp.a' -> '../lib64/libgomp.a'
   './libasan_preinit.o' -> '../lib64/libasan_preinit.o'
   './libstdc++.so.6.0.30' -> '../lib64/libstdc++.so.6.0.30'
   './libssp_nonshared.la' -> '../lib64/libssp_nonshared.la'
   './libnss_resolve.so.2' -> '../lib64/libnss_resolve.so.2'
   ln: failed to create symbolic link './libffi.so.8.1.1': File exists
   './liblsan.so.0.0.0' -> '../lib64/liblsan.so.0.0.0'
   './libsystemd.so.0.34.0' -> '../lib64/libsystemd.so.0.34.0'
   './libatomic.so.1.2.0' -> '../lib64/libatomic.so.1.2.0'
   './libstdc++fs.a' -> '../lib64/libstdc++fs.a'
   './libtsan.la' -> '../lib64/libtsan.la'
   './libgcc_s.so.1' -> '../lib64/libgcc_s.so.1'
   './liblsan.la' -> '../lib64/liblsan.la'
   './libstdc++.a' -> '../lib64/libstdc++.a'
   './libubsan.so.1.0.0' -> '../lib64/libubsan.so.1.0.0'
   './libhwasan.a' -> '../lib64/libhwasan.a'
   './libtsan_preinit.o' -> '../lib64/libtsan_preinit.o'
   './libcc1.la' -> '../lib64/libcc1.la'
   './libitm.spec' -> '../lib64/libitm.spec'
   './libatomic.a' -> '../lib64/libatomic.a'
   './libtsan.so.2.0.0' -> '../lib64/libtsan.so.2.0.0'
   './libnss_myhostname.so.2' -> '../lib64/libnss_myhostname.so.2'
   './libssp.la' -> '../lib64/libssp.la'
   './libatomic.la' -> '../lib64/libatomic.la'
   './libitm.so.1.0.0' -> '../lib64/libitm.so.1.0.0'
   './libubsan.a' -> '../lib64/libubsan.a'
   './libsupc++.a' -> '../lib64/libsupc++.a'
   './libnss_mymachines.so.2' -> '../lib64/libnss_mymachines.so.2'
   './libssp.so.0.0.0' -> '../lib64/libssp.so.0.0.0'
   './libstdc++fs.la' -> '../lib64/libstdc++fs.la'
   './libubsan.la' -> '../lib64/libubsan.la'
   './libssp_nonshared.a' -> '../lib64/libssp_nonshared.a'
   './libhwasan.la' -> '../lib64/libhwasan.la'
   './libasan.la' -> '../lib64/libasan.la'
   './libnss_systemd.so.2' -> '../lib64/libnss_systemd.so.2'
   './libitm.la' -> '../lib64/libitm.la'
   ln: failed to create symbolic link './libffi.la': File exists
   './libsupc++.la' -> '../lib64/libsupc++.la'
   './libssp.a' -> '../lib64/libssp.a'
   './libstdc++.so.6.0.30-gdb.py' -> '../lib64/libstdc++.so.6.0.30-gdb.py'
   './liblsan.a' -> '../lib64/liblsan.a'
   './libitm.a' -> '../lib64/libitm.a'
   './libcc1.so.0.0.0' -> '../lib64/libcc1.so.0.0.0'
   './libudev.so.1.7.4' -> '../lib64/libudev.so.1.7.4'
   './libgomp.la' -> '../lib64/libgomp.la'
   './liblsan_preinit.o' -> '../lib64/liblsan_preinit.o'
   './libgomp.so.1.0.0' -> '../lib64/libgomp.so.1.0.0'
   './libgomp.spec' -> '../lib64/libgomp.spec'

   cd pkgconfig
   for file in $(find ../../lib64/pkgconfig -maxdepth 1 -type f); do ln -sv $file .; done
   './libsystemd.pc' -> '../../lib64/pkgconfig/libsystemd.pc'
   './libudev.pc' -> '../../lib64/pkgconfig/libudev.pc'

8. The build of 1002-kernel requires changing $LFS/jhalfs/lfs-commands/chapter10/1002-kernel to use Image instead of bzImage cp -v arch/arm64/boot/Image /boot/vmlinuz-5.19.13-lfs-r11.2-125-systemd+.

9. The boot process is unclear yet. The main question may be how to boot a BIOS based qemu image as compared to UEFI one as asked here and here. Currently qemu-system-aarch64 -m 256M -smp 1 -drive file=build_dir.img,format=raw,if=virtio -machine type=virt,accel=hvf,highmem=off -cpu cortex-a72 results only in qemu monitor shown. The surprising fact (?? verify this once more) is that even booting UEFI ubuntu arm64 as descibed in https://gist.github.com/nrjdalal/e70249bb5d2e9d844cc203fd11f74c55 stays at qemu monitor.

[pierre-labastie]

It looks like you are using an x86 book for building on aarch64: this cannot work, although there are not so many changes needed. You can find an aarch64 lfs book at two places:

• https://github.com/cross-lfs/lfs-arm
• in the xry111/arm64 branch of https://git.linuxfromscratch.org/lfs.git
  For the first one, you need to clone the repo, and use it as a working copy for jhalfs. For the second one, you can enter xry111/arm64 as the branch. Then all the build problem should be solved (well, I'm not the maintainer of those branches, so...).

For booting, I cannot help...

[marcindulak]

Thanks, I added notes from an attempted (and failed) aarch64 to https://github.com/cross-lfs/lfs-arm/issues/7. Hopefully I'll get some feedback there.