Support of --sysroot option?

[zhongweiy]

Does ede-compdb support --sysroot option now?

[randomphrase]

@zhongweiy Please try the latest commit - added support for --sysroot=DIR and -isysroot DIR.

[zhongweiy]

@randomphrase , thanks for your update. But my setting with "--sysroot=/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64" still fails. I find the code at line 280, the "argval" is: "=/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64" when debug, which may be not correct, right?

[randomphrase]

OK I think I've fixed it in the latest commit. Thanks for testing.

[zhongweiy]

Sorry about the separate feedback, I just keep seeing new things.

My compile command is:

aarch64-linux-android-g++
-MMD -MF obj/webrtc/modules/audio_coding/codecs/isac/fix/source/iSACFix.spectrum_ar_model_tables.o.d  -DNDEBUG
-Igen -I../../.. -I../../../webrtc/modules/audio_coding/codecs/isac/fix/interface -I../../../webrtc -I../../../webrtc/common_audio/resampler/include -I../../../webrtc/common_audio/signal_processing/include -I../../../webrtc/common_audio/vad/include 
--sysroot=/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64 
-isystem/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//sources/cxx-stl/stlport/stlport   -c ../../../webrtc/modules/audio_coding/codecs/isac/fix/source/spectrum_ar_model_tables.c -o obj/webrtc/modules/audio_coding/codecs/isac/fix/source/iSACFix.spectrum_ar_model_tables.o"

And the generated include-path of compdb-entry is

("gen"
 "../../.."
 "../../../webrtc/modules/audio_coding/codecs/isac/fix/interface"
 "../../../webrtc" "../../../webrtc/common_audio/resampler/include"
 "../../../webrtc/common_audio/signal_processing/include"
 "../../../webrtc/common_audio/vad/include"
 "/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//sources/cxx-stl/stlport/stlport")

Here are 2 issues in above include-path:

• it has not included my toolchain's default including, which can be obtained by running command echo | aarch64-linux-android-g++ -v -E - --sysroot=/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64. In my case, it is:

#include <...> search starts here:

/home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9/include

/home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9/include-fixed

/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64/usr/include

I find semantic-gcc-get-include-paths in semantic/bovine/gcc provides similar function here. But the toolchain is hard-coded, it can't be changed to aarch64-linux-android-g++

• The -isystem/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//sources/cxx-stl/stlport/stlport should not be expanded to /home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//sources/cxx-stl/stlport/stlport as the gcc doc says: "If dir begins with =, then the = will be replaced by the sysroot prefix;"

[randomphrase]

I don't really have a way to test this thoroughly so I appreciate your detailed feedback.

Firstly, with regards to the compiler include path. The way it works currently is that it tries two methods to get the include paths; one which should work for gcc and the other which will work for clang. You can see these two methods in ede-compdb-compiler-include-path which also calls out to semantic/bovine/gcc to help parse the gcc options. With the gcc method we are looking for instances of --with-gxx-include-dir arguments, and with clang we are parsing the include path output as you indicate above.

So, can you send me the output of aarch64-linux-android-g++ -v and aarch64-linux-android-g++ -x c++ -Xclang -v -c - ? It may be that these options need tweaking for your compiler.

Note that it is not really feasible with the current design to include the sysroot argument when doing compiler include path detection. According to the doco it should be possible to simple prepend the sysroot to each of the compiler include paths - although feel free to correct me on this point.

On the second point - I will fix this.

[zhongweiy]

Sorry for late reply, I'm on holidays.

You can see these two methods in ede-compdb-compiler-include-path which also calls out to semantic/bovine/gcc to help parse the gcc options. With the gcc method we are looking for instances of --with-gxx-include-dir arguments, and with clang we are parsing the include path output as you indicate above.

--with-gxx-include-dir is /tmp/ndk-andrewhsieh/build/toolchain/prefix/include/c++/4.9 in my test. I guess it is will not be useful in ede-compdb and is only used in building the toolchain itselft.

According to the doco it should be possible to simple prepend the sysroot to each of the compiler include paths

Yes, sysroot will be added to compiler include paths like /usr/include /usr/local/include etc.

Here is the output of aarch64-linux-android-g++ -v

Using built-in specs.
COLLECT_GCC=/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/aarch64-linux-android-g++
COLLECT_LTO_WRAPPER=/home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/aarch64-linux-android/4.9/lto-wrapper
Target: aarch64-linux-android
Configured with: /s/ndk-toolchain/src/build/../gcc/gcc-4.9/configure --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --target=aarch64-linux-android --host=x86_64-linux-gnu --build=x86_64-linux-gnu --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --with-gmp=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpfr=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpc=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-cloog=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-isl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-ppl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --disable-ppl-version-check --disable-cloog-version-check --disable-isl-version-check --enable-cloog-backend=isl --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --disable-libssp --enable-threads --disable-nls --disable-libmudflap --disable-libgomp --disable-libstdc__-v3 --disable-sjlj-exceptions --disable-shared --disable-tls --disable-libitm --enable-initfini-array --disable-nls --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --with-sysroot=/tmp/ndk-andrewhsieh/build/toolchain/prefix/sysroot --with-binutils-version=2.24 --with-mpfr-version=3.1.1 --with-mpc-version=1.0.1 --with-gmp-version=5.0.5 --with-gcc-version=4.9 --with-gdb-version=7.6 --with-python=/usr/local/google/home/andrewhsieh/mydroid/ndk/prebuilt/linux-x86_64/bin/python-config.sh --with-gxx-include-dir=/tmp/ndk-andrewhsieh/build/toolchain/prefix/include/c++/4.9 --with-bugurl=http://source.android.com/source/report-bugs.html --disable-bootstrap --disable-libquadmath --enable-plugins --enable-libgomp --disable-libsanitizer --enable-gold --enable-graphite=yes --with-cloog-version=0.18.0 --with-isl-version=0.11.1 --enable-eh-frame-hdr-for-static --enable-fix-cortex-a53-835769 --disable-gold --program-transform-name='s&^&aarch64-linux-android-&'
Thread model: posix
gcc version 4.9 20140827 (prerelease) (GCC) 

and aarch64-linux-android-g++ -x c++ -Xclang -v -c -:

Using built-in specs.
COLLECT_GCC=/home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/aarch64-linux-android-g++
aarch64-linux-android-g++: error: unrecognized command line option '-Xclang'
Target: aarch64-linux-android
Configured with: /s/ndk-toolchain/src/build/../gcc/gcc-4.9/configure --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --target=aarch64-linux-android --host=x86_64-linux-gnu --build=x86_64-linux-gnu --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --with-gmp=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpfr=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpc=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-cloog=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-isl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-ppl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --disable-ppl-version-check --disable-cloog-version-check --disable-isl-version-check --enable-cloog-backend=isl --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --disable-libssp --enable-threads --disable-nls --disable-libmudflap --disable-libgomp --disable-libstdc__-v3 --disable-sjlj-exceptions --disable-shared --disable-tls --disable-libitm --enable-initfini-array --disable-nls --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --with-sysroot=/tmp/ndk-andrewhsieh/build/toolchain/prefix/sysroot --with-binutils-version=2.24 --with-mpfr-version=3.1.1 --with-mpc-version=1.0.1 --with-gmp-version=5.0.5 --with-gcc-version=4.9 --with-gdb-version=7.6 --with-python=/usr/local/google/home/andrewhsieh/mydroid/ndk/prebuilt/linux-x86_64/bin/python-config.sh --with-gxx-include-dir=/tmp/ndk-andrewhsieh/build/toolchain/prefix/include/c++/4.9 --with-bugurl=http://source.android.com/source/report-bugs.html --disable-bootstrap --disable-libquadmath --enable-plugins --enable-libgomp --disable-libsanitizer --enable-gold --enable-graphite=yes --with-cloog-version=0.18.0 --with-isl-version=0.11.1 --enable-eh-frame-hdr-for-static --enable-fix-cortex-a53-835769 --disable-gold --program-transform-name='s&^&aarch64-linux-android-&'
Thread model: posix
gcc version 4.9 20140827 (prerelease) (GCC)

BTW, there maybe a typo at line 212 of https://github.com/randomphrase/ede-compdb/blob/master/ede-compdb.el: (concat comp "-v") should be (concat comp " -v").

[randomphrase]

Hi @zhongweiy - I've just pushed some changes which may not entirely fix your issues but are worth testing anyway. Thanks for turning around rapid feedback.

Firstly I've changed the compiler include path detection. I tested with all of the gcc, clang and icc variants that I had at my disposal, and they all worked fine with the -x c++ -v -E - method of displaying compiler include paths. So now I'm just using that.

In addition I fixed the include path processing so that the sysroot prefix is only added for paths starting with =, and to the compiler include paths themselves.

If anything is still outstanding it is likely to be with compiler include paths when --sysroot is used. So let me know if that is still an issue.

Thanks again for the time spent so far debugging and providing feedback.

[zhongweiy]

@randomphrase you are really welcome! And I appreciate your working on ede-compdb. It is nice to have ede-compdb, because I suffer a lot from configuration cedet.

Current master does not work properly with my toolchain. Here are the different output from different toolchain:

Toolchain name	with sysroot	output of include dir
gcc -v -E -	No	#include <...> search starts here: /usr/lib/gcc/x86_64-linux-gnu/4.8/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/include End of search list.
gcc -v -E -	Yes	#include <...> search starts here: /usr/lib/gcc/x86_64-linux-gnu/4.8/include /usr/libgcc/x86_64-linux-gnu/4.8/include-fixed /home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64/usr/include End of search list.
arm-linux-androideabi-gcc -v -E -	No	#include <...> search starts here: /home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/include /home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/include-fixed End of search list.
arm-linux-androideabi-gcc -v -E -	Yes	#include <...> search starts here: /home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/include /home/zhongwei/projects/project/webrtc/src/chromium/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/include-fixed /home/zhongwei/projects/project/webrtc/src/third_party/android_tools/ndk//platforms/android-21/arch-arm64/usr/include End of search list.

The include-dir returns from current get-include-path function is not correct. The system include dir between #include <...> search starts here: and End of search list. needs no additional processing. I think what we need to do is running compiler command with --sysroot option, right?

BTW, you can get the toolchain like arm-linux-androideabi-gcc from https://developer.android.com/tools/sdk/ndk/index.html

[randomphrase]

Hi @zhongweiy, that's great information, thanks.

I wanted to avoid doing the compiler include path discovery process with --sysroot because it would mean doing this every time with different values of sysroot. Because the compiler include paths are cached and shared amongst all projects, it would have been preferable to do some post-processing for sysroot-specific projects. However I guess this is not the case.

I think the next best option is to add an additional level of lookup for the compiler include path caching. In other words, key the cache on both the compiler and the sysroot value. I'll try to get to this soon.

Also FYI, I have been working on integrating compdb with cedet proper. If you pull the latest cedet from sourceforge you'll see the initial commit. For now I'll commit any changes both here and there but eventually - probably at the next cedet release - I'll cut over to cedet only for development.

Thanks yet again for the excellent information and feedback.

[zhongweiy]

I wanted to avoid doing the compiler include path discovery process with --sysroot because it would mean doing this every time with different values of sysroot. Because the compiler include paths are cached and shared amongst all projects

I see.

Also FYI, I have been working on integrating compdb with cedet proper. If you pull the latest cedet from sourceforge you'll see the initial commit. For now I'll commit any changes both here and there but eventually - probably at the next cedet release - I'll cut over to cedet only for development.

Great to know:)